Puncture assistance device and puncture assistance program

Abstract

To provide an operator with a recommended puncture route which is a route to be passed by a puncture needle from the body surface of a subject to a puncture object while avoiding an avoidance tissue to be avoided by the puncture needle.SOLUTION: A volume data acquisition part 52 acquires volume data 46 including the puncture object and the avoidance tissue. The model forming unit 54 forms a subject model 70, which is a three dimensional model including a puncture object model and an avoidance tissue model, based on the volume data 46. The puncture route specifying unit 56 specifies, in the subject model 70, a recommended puncture route 72 through which the puncture needle 18 should pass from the body surface position of the subject E to the puncture target model while avoiding the avoidance tissue model. The display control unit 36 notifies the operator of the specified recommended puncture route 72.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This specification discloses an improvement to a puncture assistance device and a puncture assistance program. [Background technology]

[0002] BACKGROUND ART Ultrasound diagnostic devices are known that transmit ultrasonic waves from an ultrasound probe toward a subject, receive reflected waves from the subject at the ultrasound probe, and perform various processes, such as forming ultrasound tomographic images that represent cross sections inside the subject, based on received signals formed from the reflected waves.

[0003] Conventionally, a puncture needle has been inserted (punctured) into the tissue of a subject for the purpose of extracting tissue (e.g., a tumor) from the subject, etc. Ultrasonic probes equipped with a puncture needle have also been proposed, and with such ultrasonic probes, it is possible to puncture the subject while irradiating the subject with the ultrasonic probe to form and display an ultrasonic tomographic image.

[0004] Patent Document 1 discloses a puncture support system including a puncture needle, a probe guide that holds the puncture needle so that the oscillation angle of the puncture needle can be set and that has a built-in ultrasonic sensor that can perform sector scanning around the axis of the puncture needle, a CCD camera that photographs markers attached to the puncture needle and markers attached to the probe guide, an ultrasonic diagnostic device that forms an ultrasonic tomographic image based on a signal from the ultrasonic sensor built into the probe guide, and a monitor image construction unit that calculates the relationship between the puncture depth of the puncture needle according to the marker attached to the puncture needle in the image photographed by the CCD camera, the oscillation angle of the puncture needle according to the marker attached to the probe guide in the image photographed by the CCD camera, and the position of a target site in the body (e.g., a tumor) obtained from the ultrasonic tomographic image, and displays it on a display unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 10-5223 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, when puncturing a puncture target inside a subject, it may be necessary to puncture the puncture target while avoiding tissues surrounding the puncture target that should not be punctured (such tissues are referred to as "avoidable tissues" in this specification). Avoidable tissues are, for example, tissues that would affect the prognosis of the subject if punctured. Avoidable tissues include, for example, blood vessels and nerves.

[0007] As described above, conventionally, puncture is performed while displaying an ultrasonic tomographic image showing a cross section inside the subject, but the experience and technique of an operator such as a doctor are still required to avoid avoiding tissue and puncture the object to be punctured.

[0008] The purpose of the puncture assistance device disclosed in this specification is to provide the operator with a recommended puncture route, which is the route that the puncture needle should take from the subject's body surface to the object to be punctured while avoiding tissue that should be avoided by the puncture needle. [Means for solving the problem]

[0009] The puncture assistance device disclosed in this specification is characterized by comprising: a probe information acquisition unit that acquires probe information indicating the position and posture of an ultrasound probe; a volume data acquisition unit that acquires volume data based on received signals obtained by transmitting and receiving ultrasound waves from the ultrasound probe to a subject; a model formation unit that forms, based on the volume data, a subject model including a puncture target model that is a three-dimensional model representing the object to be punctured and an avoided tissue model that is a three-dimensional model representing avoided tissue that is tissue that the puncture needle should avoid when puncturing the object to be punctured, the subject model having position information indicating each position of the subject model based on the position and posture of the ultrasound probe when the received signals were acquired; a puncture route identification unit that identifies, in the subject model, a recommended puncture route that is a route that the puncture needle should take from a body surface position of the subject to reach the puncture target model while avoiding the avoided tissue model; and a notification processing unit that notifies an operator of the recommended puncture route.

[0010] The puncture route specifying unit may specify the recommended puncture route so that the shortest distance between the recommended puncture route and the avoided tissue model is equal to or greater than a first threshold distance.

[0011] When the puncture route identification unit identifies multiple candidate puncture routes that reach the puncture target model from the body surface position of the subject while avoiding the avoidance tissue model, it is preferable that the puncture route identification unit identify the candidate puncture route with the shortest length as the recommended puncture route.

[0012] The puncture object is a tumor, the puncture needle is an ablation puncture needle that cauterizes the tumor, and the puncture route identification unit identifies the recommended puncture route so that the end of the recommended puncture route on the tumor model side as the puncture object model is at a second threshold distance or more from a tissue model other than the tumor model.

[0013] The apparatus may further include a recommended position and orientation specification unit that specifies, based on the recommended puncture route, a recommended probe position and orientation, which is the position and orientation of the ultrasound probe, such that the scanning plane of the ultrasound probe includes the recommended puncture route and is parallel to the recommended puncture route, and the notification processing unit may notify an operator of the recommended probe position and orientation.

[0014] The notification processing unit may notify the operator of guide information for transitioning the position and posture of the ultrasound probe from the current position and posture of the ultrasound probe to the recommended probe position and posture based on the difference between the current position and posture of the ultrasound probe and the recommended probe position and posture.

[0015] The apparatus may further include a matching index calculation unit that calculates a matching index, which is an index indicating the degree of match between a reconstructed ultrasound image that includes the recommended puncture route and is cut out from the subject model at a cross section parallel to the recommended puncture route, and a real-time image that is an ultrasound tomographic image formed at the current position and posture of the ultrasound probe, and the notification processing unit may notify an operator of the matching index.

[0016] The device may further include a puncture needle information acquisition unit that acquires puncture needle information indicating the current position and posture of the puncture needle, and a puncture needle deviation determination unit that determines, based on the puncture needle information, that the puncture needle has deviated from the recommended puncture route when the distance between the puncture needle at its current position and the recommended puncture route is equal to or greater than a distance threshold, or when the difference in angle between the current extension direction of the puncture needle and the extension direction of the recommended puncture route is equal to or greater than an angle threshold, and the notification processing unit may notify an operator when it is determined that the puncture needle has deviated from the recommended puncture route.

[0017] The probe information acquisition unit may detect the relative position and orientation of the ultrasound probe with respect to the subject's body surface based on captured images acquired by a camera capturing images of a probe detection marker attached to the ultrasound probe and a body surface detection marker attached to the subject's body surface.

[0018] The probe information acquisition unit detects the position and orientation of the ultrasound probe relative to the body surface of the subject based on an image acquired by a camera photographing a probe detection marker attached to the ultrasound probe and a body surface detection marker attached to the body surface of the subject, and the puncture needle information acquisition unit detects the position and orientation of the puncture needle relative to the body surface of the subject based on an image acquired by a camera photographing the puncture needle detection marker attached to the puncture needle and the body surface detection marker.

[0019] Furthermore, the puncture assistance program disclosed in this specification is characterized in that it causes a computer to function as: a probe information acquisition unit that acquires probe information indicating the position and posture of an ultrasound probe; a volume data acquisition unit that acquires volume data based on received signals obtained by transmitting and receiving ultrasound waves from the ultrasound probe to a subject; a model formation unit that forms, based on the volume data, a subject model including a puncture target model that is a three-dimensional model representing the object to be punctured and an avoided tissue model that is a three-dimensional model representing avoided tissue that is tissue that the puncture needle should avoid when puncturing the object to be punctured, the subject model having position information indicating each position of the subject model based on the position and posture of the ultrasound probe when the received signals were acquired; and a puncture route identification unit that identifies, in the subject model, a recommended puncture route that is a route that the puncture needle should take from a body surface position of the subject to reach the puncture target model while avoiding the avoided tissue model. [Effects of the Invention]

[0020] The puncture assistance device disclosed in this specification can provide the operator with a recommended puncture route, which is the route the puncture needle should take from the subject's body surface to the object to be punctured while avoiding tissue that should be avoided by the puncture needle. [Brief explanation of the drawings]

[0021] [Figure 1]1 is a schematic diagram illustrating the configuration of a puncture support system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of an image captured by a camera. [Figure 3] 1 is a schematic diagram illustrating the configuration of an ultrasound diagnostic apparatus according to the present embodiment. [Figure 4] FIG. 2 is a conceptual diagram illustrating the concept of volume data formation processing. [Figure 5] FIG. 2 is a conceptual diagram illustrating the concept of a subject model formation process. [Figure 6] FIG. 2 is a diagram illustrating an example of a subject model. [Figure 7] FIG. 10 is a diagram showing an example of a recommended puncture route. [Figure 8] 10A and 10B are diagrams showing display examples of a reconstructed ultrasound image and a recommended puncture route image. [Figure 9] FIG. 10 is a first diagram showing an example of notification of a recommended probe position and orientation. [Figure 10] FIG. 10 is a second diagram showing an example of notification of a recommended probe position and orientation. [Figure 11] FIG. 10 is a diagram showing an example of a display of a real-time image that is the same cross section as the reconstructed ultrasound image. [Figure 12] FIG. 10 is a diagram showing the current puncture needle and a recommended puncture route in a model coordinate system. [Figure 13] FIG. 10 is a diagram showing an example of notification that the puncture needle has become displaced. [Figure 14] FIG. 10 is a diagram showing an example of notification that the puncture needle is not displaced. DETAILED DESCRIPTION OF THE INVENTION

[0022] 1 is a schematic diagram of the configuration of a puncture support system 10 according to this embodiment. The puncture support system 10 includes a camera 12, an ultrasound diagnostic device 16 as a puncture support device including an ultrasound probe 14, and a puncture needle 18. In this embodiment, the ultrasound probe 14 and the puncture needle 18 are separate entities, but the ultrasound probe 14 may also be a puncture probe, i.e., the puncture needle 18 may be provided on the ultrasound probe 14. The camera 12 and the ultrasound diagnostic device 16 are connected to each other so that they can communicate with each other.

[0023] In this embodiment, a probe detection marker 20 is attached to the ultrasound probe 14. The probe detection marker 20 is a marker for detecting the position and posture of the ultrasound probe 14. Furthermore, a puncture needle detection marker 22 is attached to the puncture needle 18. The puncture needle detection marker 22 has a different pattern from the probe detection marker 20 and is a mark for detecting the position and posture of the puncture needle 18. Furthermore, a body surface detection marker 24 is attached to the body surface of the subject E. The body surface detection marker 24 has a different pattern from the probe detection marker 20 and the puncture needle detection marker 22 and is a marker for detecting the position and posture of the body surface of the subject E. One example of the probe detection marker 20, the puncture needle detection marker 22, and the body surface detection marker 24 is an AR (Argumented Reality) marker.

[0024] The camera 12 includes a lens, an image sensor, a processor including a CPU (Central Processing Unit), etc., and a communication interface including a network adapter, etc. The camera 12 captures images of the ultrasound probe 14 (specifically, a probe detection marker 20), the puncture needle 18 (specifically, a puncture needle detection marker 22), and the subject E (specifically, a body surface detection marker 24). A captured image is formed by the image sensor of the camera 12, and is transmitted to the ultrasound diagnostic device 16 via the communication interface of the camera 12.

[0025] FIG. 2 is a diagram showing an example of an image 26 captured by the camera 12. As described above, the captured image 26 includes images of the probe detection marker 20, the puncture needle detection marker 22, and the body surface detection marker 24. The ultrasound diagnostic device 16 can detect the position and orientation of the ultrasound probe 14 by analyzing the image of the probe detection marker 20 captured in the captured image 26. Furthermore, the ultrasound diagnostic device 16 can detect the position and orientation of the puncture needle 18 by analyzing the image of the body surface detection marker 24 captured in the captured image 26. The ultrasound diagnostic device 16 can detect the position and orientation of the body surface of the subject E by analyzing the image of the body surface detection marker 24. Details of the detection process for the positions and orientations of the ultrasound probe 14, the puncture needle 18, and the subject E will be described later.

[0026] 3 is a schematic diagram of the configuration of the ultrasonic diagnostic device 16. The ultrasonic diagnostic device 16 is a medical device installed in a medical institution such as a hospital.

[0027] The ultrasonic probe 14 is a device that transmits and receives ultrasonic waves to and from the subject E. The ultrasonic probe 14 has a transducer element array consisting of a plurality of transducer elements. In this embodiment, the transducer element array is formed of a plurality of transducer elements arranged in one direction (array direction). A transmission signal is supplied to each transducer element from a transceiver unit 30, which will be described later, causing each transducer element to generate ultrasonic waves. Specifically, the ultrasonic probe 14 scans an ultrasonic beam in a plane (scanning plane) parallel to the array direction.

[0028] As described above, the ultrasonic probe 14 is provided with a probe detection marker 20 .

[0029] The transmitter / receiver unit 30 transmits a transmission signal to the ultrasound probe 14 (more specifically, to each transducer element of the transducer element array) under the control of the controller 48, which will be described later. This causes the ultrasound beam to scan the scanning plane. The transmitter / receiver unit 30 also receives a reception signal from each transducer element that has received a reflected wave from the target tissue. The transmitter / receiver unit 30 has an adder and a plurality of delay elements corresponding to each transducer element, and performs a phased addition process using the adder and the plurality of delay elements to align and add the phases of the reception signals from each transducer element. This forms a reception beam signal in which information indicating the signal intensity of the reflected wave from the target tissue is aligned in the depth direction of the target tissue.

[0030] The signal processing unit 32 performs various signal processing on the received beam signal from the transmitting / receiving unit 30, including filtering using a band-pass filter and detection processing.

[0031] The image forming unit 34 forms an ultrasonic tomographic image (B-mode image) representing a cross section of the subject E (particularly the scanning plane of the ultrasonic beam) based on the received beam signals that have been signal-processed in the signal processing unit 32.

[0032] The display control unit 36 ​​controls the display of various images, including the ultrasonic tomographic image formed by the image forming unit 34, on the display 38.

[0033] The display 38 serving as a display unit is a display device configured, for example, by a liquid crystal display or an organic EL (Electro Luminescence) display.

[0034] The transmitter / receiver 30, signal processor 32, image generator 34, and display controller 36 of the ultrasound diagnostic device 16 are configured by a processor. The processor includes at least one of a general-purpose processing device (e.g., a CPU) and a dedicated processing device (e.g., a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a programmable logic device). The processor may not be a single processing device, but may be configured by the cooperation of multiple processing devices located at physically separate locations. Furthermore, each of the above units may be realized by the cooperation of hardware such as a processor and software.

[0035] The communication interface 40 is configured by, for example, a network adapter, etc. The communication interface 40 performs the function of communicating with other devices (particularly, the camera 12). In particular, the communication interface 40 receives the captured image 26 from the camera 12.

[0036] The input interface 42 is configured by, for example, buttons, a trackball, a touch panel, etc. The input interface 42 is used to input commands from an operator, such as a doctor, who uses the ultrasound diagnostic apparatus 16 to the ultrasound diagnostic apparatus 16.

[0037] The memory 44 includes a hard disk drive (HDD), a solid state drive (SSD), an embedded multi-media card (eMMC), a read-only memory (ROM), or a random access memory (RAM). The memory 44 stores a puncture assistance program for operating each unit of the ultrasound diagnostic device 16. The puncture assistance program can also be stored in a computer-readable non-transitory storage medium such as a universal serial bus (USB) memory or a CD-ROM. The ultrasound diagnostic device 16 can read and execute the puncture assistance program from such a storage medium. Because the ultrasound diagnostic device 16 performs the functions described below by reading the puncture assistance program, the ultrasound diagnostic device 16 can be considered a computer program product.

[0038] 3, volume data 46 is stored in the memory 44. In this embodiment, the volume data 46 is formed based on a plurality of ultrasonic tomographic images based on received signals acquired while the operator moves the ultrasonic probe 14 in a direction perpendicular to the scanning plane. The volume data 46 will be described in detail later.

[0039] The control unit 48 includes at least one of a general-purpose processor (e.g., a CPU) and a dedicated processor (e.g., a GPU, an ASIC, an FPGA, or a programmable logic device). The control unit 48 may not be a single processing unit, but may be configured by the cooperation of multiple processing units located at physically separate locations. The control unit 48 controls each unit of the ultrasound diagnostic apparatus 16. As shown in FIG. 3 , the control unit 48 fulfills the functions of a probe information acquisition unit 50, a volume data acquisition unit 52, a model formation unit 54, a puncture route identification unit 56, a recommended position and posture identification unit 58, a matching index calculation unit 60, a puncture needle information acquisition unit 62, and a puncture needle displacement determination unit 64 in accordance with a puncture assistance program stored in the memory 44.

[0040] The probe information acquisition unit 50 acquires probe information that indicates the position and posture of the ultrasonic probe 14 .

[0041] In this embodiment, the probe information acquisition unit 50 acquires probe information by analyzing the captured image 26 acquired by the camera 12 to detect the position and orientation of the ultrasound probe 14. As described above, the captured image 26 includes an image of the probe detection marker 20 for detecting the position and orientation of the ultrasound probe 14 (see FIG. 2). The probe information acquisition unit 50 acquires probe information by analyzing the image of the probe detection marker 20 in the captured image 26. The probe information includes position information indicating the position of the ultrasound probe 14 and orientation information indicating the orientation of the ultrasound probe 14. The position information may be, for example, three-dimensional coordinates in a camera coordinate system. The orientation information may be a rotation angle relative to a predetermined axis (e.g., X-axis, Y-axis, Z-axis) in the camera coordinate system. Note that a known method can be used to detect the position and orientation of the ultrasound probe 14 in the camera coordinate system from the image of the probe detection marker 20 included in the captured image 26, and therefore a detailed description thereof will be omitted.

[0042] The probe information acquiring unit 50 may perform calibration before detecting the position and orientation of the ultrasound probe 14 based on the captured image 26. Specifically, the operator sets the position and orientation of the ultrasound probe 14 to a predetermined position and orientation, and inputs a calibration instruction to the ultrasound diagnostic device 16 in that state. The probe information acquiring unit 50 detects the position and orientation of the ultrasound probe 14 at the time the calibration instruction was input based on the probe detection marker 20 included in the captured image 26, and sets this as the reference position and reference orientation. This makes it possible to obtain the relationship between information indicating the position and orientation of the ultrasound probe 14 detected by the probe information acquiring unit 50 (for example, the position and orientation in the camera coordinate system) and the real space coordinate system.

[0043] As described above, the captured image 26 also includes images of the body surface detection markers 24 for detecting the position and posture of the body surface of the subject E (see FIG. 2). The probe information acquisition unit 50 may detect the position and posture of the body surface of the subject E by analyzing the images of the body surface detection markers 24 in the captured image 26. Then, the probe information acquisition unit 50 may detect the position and posture of the ultrasound probe 14 relative to the position and posture of the body surface of the subject E. This makes it possible to obtain the position and posture of the ultrasound probe 14 with respect to the subject E, taking into account fluctuations in the position or posture of the subject E.

[0044] The probe information acquisition unit 50 may detect the position and orientation of the ultrasound probe 14 by a method other than analyzing the captured image 26. For example, the ultrasound probe 14 may be provided with a position and orientation sensor such as a magnetic sensor, and the position and orientation of the ultrasound probe 14 may be detected based on the detection value of the position and orientation sensor.

[0045] The scanning plane, which is the surface scanned by the ultrasonic beam from the ultrasonic probe 14, is determined by the position and orientation of the ultrasonic probe 14. Therefore, it can be said that the probe information acquired by the probe information acquiring unit 50 indicates the position and orientation of the scanning plane.

[0046] The volume data acquisition unit 52 acquires volume data 46 based on received signals obtained by transmitting and receiving ultrasonic waves from the ultrasonic probe 14 to the subject E. In this embodiment, the volume data acquisition unit 52 forms the volume data 46 based on received signals corresponding to a plurality of scanning planes acquired while the operator moves the ultrasonic probe 14 in a direction perpendicular to the scanning planes.

[0047] FIG. 4 is a conceptual diagram illustrating the concept of the process of forming volume data 46. In this embodiment, first, the operator transmits and receives ultrasonic waves to and from the subject E while moving the ultrasonic probe 14 in a direction perpendicular to the scanning plane. Here, it is assumed that the movement path of the scanning plane includes the puncture target inside the subject E and the avoiding tissue surrounding the puncture target. As a result, the transmitting and receiving unit 30 acquires multiple received signals for multiple scanning planes aligned in a direction perpendicular to the scanning plane. Then, the image forming unit 34 forms multiple ultrasonic tomographic images 46a corresponding to the multiple scanning planes based on the multiple received signals. At least some of the multiple ultrasonic tomographic images 46a include an image of the puncture target or an image of the avoiding tissue.

[0048] The probe information acquiring unit 50 detects the position and orientation of the ultrasonic probe 14 when it acquires each received signal corresponding to each ultrasonic tomographic image 46a, and associates probe information indicating the position and orientation of the ultrasonic probe 14 detected by the probe information acquiring unit 50 with each ultrasonic tomographic image 46a. As described above, the probe information may indicate the position and orientation of the ultrasonic probe 14 relative to the subject E. Each position (coordinate) on the ultrasonic tomographic image 46a can be identified based on the position and orientation of the ultrasonic probe 14 when it acquires the received signal corresponding to the ultrasonic tomographic image 46a. In other words, each ultrasonic tomographic image 46a is assigned position information indicating each position (coordinate) of the ultrasonic tomographic image 46a based on the position and orientation of the ultrasonic probe 14 when it acquires the received signal corresponding to the ultrasonic tomographic image 46a.

[0049] The volume data acquisition unit 52 forms volume data 46 based on the multiple ultrasonic tomographic images 46a as received signals. A known method can be used to form the volume data 46 from the multiple ultrasonic tomographic images 46a, and detailed description thereof will be omitted here. As described above, the object to be punctured and the tissue to be avoided are included within the movement path of the scanning plane, and at least a portion of the multiple ultrasonic tomographic images 46a includes an image of the object to be punctured or an image of the tissue to be avoided. Therefore, the volume data 46 is data including the object to be punctured and the tissue to be avoided. Furthermore, as described above, each ultrasonic tomographic image 46a is provided with coordinate information indicating each position of the ultrasonic tomographic image 46a based on the position and orientation of the ultrasonic probe 14 when the received signal corresponding to the ultrasonic tomographic image 46a was acquired. Therefore, the volume data 46 composed of the multiple ultrasonic tomographic images 46a also has coordinate information indicating each position of the volume data 46 based on the position and orientation of the ultrasonic probe 14 when the received signal corresponding to the ultrasonic tomographic image 46a was acquired (referred to as a model coordinate system in this specification).

[0050] In this embodiment, the volume data 46 is composed of a plurality of ultrasonic tomographic images 46a, but if a 2D array probe in which transducer elements are arranged two-dimensionally is used as the ultrasonic probe 14, the volume data acquisition unit 52 may directly form the volume data 46 including the object to be punctured and the tissue to be avoided based on the 23-dimensional received signals from the ultrasonic probe 14. Even in this case, the volume data 46 can be identified based on the position and orientation of the ultrasonic probe 14 when the volume data 46 was formed. That is, even in this case, the volume data 46 has coordinate information indicating each position of the volume data 46 in the model coordinate system.

[0051] The model forming unit 54 forms a subject model, which is a three-dimensional model, based on the volume data 46, including a puncture target model, which is a three-dimensional model representing the puncture target, and an avoided tissue model, which is a three-dimensional model representing the avoided tissue.

[0052] 5 is a conceptual diagram showing the concept of the processing for forming the object model 70. The object model 70 can be formed based on the volume data 46 using a known method, and therefore detailed description thereof will be omitted here. However, the model forming unit 54 forms the object model 70 using a technique such as volume rendering or surface rendering.

[0053] FIG. 6 is a diagram showing an example of an object model 70. In the example of FIG. 6, the object model 70 is a model representing the liver of an object E and its surroundings. The object model 70 is configured to include a plurality of tissue models. In the example of FIG. 6, the object model 70 includes a liver model LV, a vein model VE, an artery model AR, a bile duct model BD, and a tumor model TM. Of these, in this embodiment, since a tumor in the liver is the puncture target, the tumor model TM is the puncture target model, and since the veins, arteries, and bile ducts in the liver are avoided tissues, the vein model VE, the artery model AR, and the bile duct model BD are avoided tissue models. Note that the object model 70 representing the liver and its surroundings would normally also include models of the inferior vena cava, portal vein, vascular region, and other tissues, but these are not shown in FIG. 6.

[0054] The volume data 46 has position information in a model coordinate system, and since the object model 70 is formed from the volume data 46, the object model 70 also has position information indicating each position of the object model 70 (in the model coordinate system) based on the position and orientation of the ultrasound probe 14 when the received signals constituting the volume data 46 were acquired. Since the position of the ultrasound probe 14 when the received signals were acquired represents the body surface position of the object E, the body surface position of the object E in the object model 70 is also known.

[0055] 6 shows three axis directions (X-axis, Y-axis, and Z-axis) of a model coordinate system that represents the position of the object model 70. When the position and orientation of the ultrasound probe 14 are acquired by analyzing the captured image 26 acquired by the camera 12 and the position and orientation of the camera 12 are fixed, the model coordinate system may be the same as the camera coordinate system of the camera 12. Furthermore, the relationship between the model coordinate system and the real space coordinate system can be obtained by the above-mentioned calibration.

[0056] The puncture route specifying unit 56 specifies a recommended puncture route, which is a route along which the puncture needle 18 should pass, in the subject model 70. Fig. 7 is a diagram showing an example of a recommended puncture route 72. In particular, the puncture route specifying unit 56 specifies a recommended puncture route 72 that extends from the body surface position of the subject E to the puncture target model (the tumor model TM in the example of Fig. 7) while avoiding avoidable tissue models (the vein model VE, the artery model AR, and the bile duct model BD in the example of Fig. 7).

[0057] Specifically, the puncture route identifying unit 56 searches for a straight line in the subject model 70 that extends from the puncture target object model toward the body surface position of the subject E (the body surface position is not a single point) and does not pass through the avoided tissue model. If such a straight line is found, the straight line is identified as a candidate for the recommended puncture route 72 (referred to as a "candidate puncture route" in this specification). In particular, the puncture route identifying unit 56 may identify the candidate puncture route so that the shortest distance between the candidate puncture route and the avoided tissue model is equal to or greater than a predetermined first threshold distance. This is to reduce the possibility that the puncture needle 18 will puncture the avoided tissue model when the operator later actually performs puncture along the recommended puncture route 72.

[0058] If only one candidate puncture route is found, the puncture route identifying unit 56 may identify the candidate puncture route as the recommended puncture route 72. However, normally, as shown in Fig. 7, multiple candidate puncture routes 72a to 72c are identified. In this case, the puncture route identifying unit 56 identifies the candidate puncture route with the shortest length (for example, the candidate puncture route 72a) among the multiple candidate puncture routes 72a to 72c as the recommended puncture route 72. This is because, when an operator actually performs puncture along the recommended puncture route 72 later, the shorter the puncture route, the lower the possibility of an unexpected event (such as an operator error) occurring.

[0059] Furthermore, when the puncture target is a tumor, the puncture needle 18 may be an ablation puncture needle that cauterizes the tumor. The ablation puncture needle cauterizes the tumor by passing a high-frequency radio frequency current through the tumor from the tip of the puncture needle 18. In this case, if tissue other than the tumor is present near the tip of the puncture needle 18, the surrounding tissue may be affected when the tumor is cauterized. In consideration of this, when the puncture target is a tumor and the puncture needle 18 is an ablation puncture needle, the puncture route identification unit 56 may identify the recommended puncture route 72 so that the end of the recommended puncture route 72 on the tumor model TM side as the puncture target model is at a predetermined second threshold distance or more from a tissue model other than the tumor model TM.

[0060] The display control unit 36 ​​as a notification processing unit notifies the operator of the recommended puncture route 72 identified by the puncture route identification unit 56. In this embodiment, the display control unit 36 ​​notifies the operator of the recommended puncture route 72 by the method described below.

[0061] First, the image forming unit 34 identifies, in the volume data 46, a cross section that includes the identified recommended puncture route 72 and is parallel to the recommended puncture route 72. As described above, the volume data 46 and the object model 70 have position information in the same model coordinate system, and therefore the image forming unit 34 can identify, in the volume data 46, a cross section that includes the recommended puncture route 72 and is parallel to the recommended puncture route 72, based on the recommended puncture route 72 identified in the object model 70.

[0062] The image forming unit 34 extracts and reconstructs the volume data 46 at the identified cross section to form a reconstructed ultrasound image. Then, the display control unit 36 ​​displays the formed reconstructed ultrasound image together with a recommended puncture route image indicating the recommended puncture route 72 superimposed on the reconstructed ultrasound image on the display 38. In this way, the display control unit 36 ​​notifies the operator of the recommended puncture route 72.

[0063] 8 is a diagram showing a display example of a reconstructed ultrasound image 80 and a recommended puncture route image 82. In the example of FIG. 8, the recommended puncture route image 82 is shown with a dashed line.

[0064] The method of notifying the operator of the recommended puncture route 72 by the display control unit 36 ​​is not limited to the above method. For example, the display control unit 36 ​​may notify the operator of the recommended puncture route 72 by displaying on the display 38 a three-dimensional object model 70 indicating the recommended puncture route 72.

[0065] As described above, according to this embodiment, the puncture route specifying unit 56 specifies a recommended puncture route 72 that reaches the puncture target from the body surface position of the subject E while avoiding the avoidable tissue, and notifies the operator of the recommended puncture route. This allows the operator to easily understand the puncture route for puncturing the puncture target while starting from the avoidable tissue.

[0066] 8, the display control unit 36 ​​may display a real-time image 84, which is an ultrasonic tomographic image formed at the current position and posture of the ultrasound probe 14, together with a reconstructed ultrasound image 80 and a recommended puncture route image 82. When the puncture needle 18 is inserted into the subject E, if the puncture needle 18 passes through the scanning plane of the ultrasound probe 14 (particularly if the puncture needle 18 is parallel to the scanning plane), an image of the puncture needle 18 appears in the real-time image 84. Therefore, if the cross section of the real-time image 84 is the same as that of the reconstructed ultrasound image 80, the operator can perform puncture along the recommended puncture route 72 while comparing the recommended puncture route image 82 on the reconstructed ultrasound image 80 with the image of the puncture needle 18 that appears in the real-time image 84.

[0067] The recommended position and posture specifying unit 58 specifies a recommended probe position and posture, which is the position and posture of the ultrasound probe 14 for forming a real-time image 84 of the same cross section as the reconstructed ultrasound image 80. As described above, the reconstructed ultrasound image 80 represents a cross section in the volume data 46 that includes the recommended puncture route 72 and is parallel to the recommended puncture route 72. Therefore, the recommended position and posture specifying unit 58 also first specifies a cross section in the volume data 46 that includes the recommended puncture route 72 and is parallel to the recommended puncture route 72, based on the recommended puncture route 72. Then, the body surface position included in the specified cross section is the position of the ultrasound probe 14 in the model coordinate system for forming a real-time image 84 of the same cross section as the reconstructed ultrasound image 80, and the posture in which the scanning plane is parallel to the specified cross section is the posture of the ultrasound probe 14 in the model coordinate system for forming a real-time image 84 of the same cross section as the reconstructed ultrasound image 80. Furthermore, the recommended position and posture specifying unit 58 converts the recommended probe position and posture specified in the model coordinate system into a recommended probe position and posture in real space, based on the relationship between the model coordinate system and the real space coordinate system. The recommended probe position and orientation obtained in this way is a position and orientation in which the scanning plane of the ultrasound probe 14 includes the recommended puncture route 72 (in real space) and is parallel to the recommended puncture route 72.

[0068] The display control unit 36 ​​as a notification processing unit notifies the operator of the recommended probe position and orientation identified by the recommended position and orientation identifying unit 58.

[0069] 9 is a first diagram showing an example of notification of the recommended probe position and orientation. For example, as shown in FIG. 9, the display control unit 36 ​​notifies the operator of the recommended probe position and orientation by displaying a body mark 86 indicating the outline of the subject E on the display 38 and indicating the recommended probe position and orientation on the body mark 86.

[0070] In addition, the probe information acquisition unit 50 may detect the current position and posture of the ultrasound probe 14, and the display control unit 36 ​​may notify the operator of guide information for transitioning the position and posture of the ultrasound probe 14 from the current position and posture of the ultrasound probe 14 to the recommended probe position and posture based on the difference between the current position and posture of the ultrasound probe 14 and the recommended probe position and posture.

[0071] Fig. 10 is a second diagram showing an example of notification of the recommended probe position and posture. For example, as shown in Fig. 10, the display control unit 36 ​​may display a body mark 86 indicating the outline of the subject E on the display 38, and may display the current position and posture of the ultrasound probe 14 and the recommended probe position and posture as guide information on the body mark 86. Such guide information may be a moving image showing the transition from the current position and posture of the ultrasound probe 14 to the recommended probe position and posture.

[0072] The notification mode of the recommended probe position and orientation is not limited to the above.

[0073] Based on the notified recommended probe position and orientation, the operator adjusts the position and orientation of the ultrasonic probe 14 so that the cross sections of the reconstructed ultrasonic image 80 and the real-time image 84 are the same.

[0074] The alignment index calculation unit 60 calculates an alignment index, which is an index indicating the degree of match between the reconstructed ultrasound image 80 and the real-time image 84. In this embodiment, the alignment index calculation unit 60 calculates the alignment index by comprehensively evaluating the similarity between the reconstructed ultrasound image 80 and the real-time image 84, the difference between the position indicated by the recommended probe position and orientation and the current position of the ultrasound probe 14, and the difference between the orientation indicated by the recommended probe position and orientation and the current orientation of the ultrasound probe 14. For example, the alignment index calculation unit 60 calculates the alignment index using the following equation 1. (Matching index)=(Similarity between the reconstructed ultrasound image 80 and the real-time image 84)×α1+(Difference between the position indicated by the recommended probe position and orientation and the current position of the ultrasound probe 14)×α2+(Difference between the orientation indicated by the recommended probe position and orientation and the current orientation of the ultrasound probe 14)×α3 (Equation 1) In Equation 1, α1, α2, and α3 are coefficients (weights) that may be set appropriately in advance.

[0075] The display control unit 36 ​​as a notification processing unit notifies the operator of the matching index calculated by the matching index calculation unit 60 .

[0076] 10, the display control unit 36 ​​causes the display 38 to display an alignment index image 88 representing the calculated alignment index together with a reconstructed ultrasound image 80 and a real-time image 84. Note that the method of notifying the alignment index is not limited to this. For example, the audio output unit serving as the notification processing unit may notify the operator of the alignment index by voice.

[0077] FIG. 11 is a diagram showing an example of a display of a real-time image 84, which is the same cross section as the reconstructed ultrasound image 80. By appropriately adjusting the position and orientation of the ultrasound probe 14, the operator can bring the cross section of the real-time image 84 closer to the cross section of the reconstructed ultrasound image 80. In the example of FIG. 11, the alignment index is quite high at 95%. The display control unit 36 ​​may notify the operator that the alignment index has reached a predetermined index threshold or greater. For example, when the alignment index has reached a predetermined index threshold or greater, the display control unit 36 ​​may highlight an alignment index image 88 to notify the operator that the alignment index has reached a predetermined index threshold or greater.

[0078] When a real-time image 84 of the same cross section as the reconstructed ultrasound image 80 can be rendered, the operator starts puncturing the object to be punctured while referring to the recommended puncture route 72 and the real-time image 84. Here, the ultrasound diagnostic device 16 according to this embodiment assists the operator in puncturing along the recommended puncture route 72.

[0079] The puncture needle information acquisition unit 62 acquires puncture needle information that indicates the current position and orientation of the puncture needle 18 in the model coordinate system.

[0080] In this embodiment, the puncture needle information acquisition unit 62 acquires puncture needle information by analyzing the captured image 26 (see FIG. 2 ) acquired by the camera 12 to detect the position and orientation of the puncture needle 18. As described above, the captured image 26 includes an image of the puncture needle detection marker 22 for detecting the position and orientation of the puncture needle 18. The puncture needle information acquisition unit 62 acquires puncture needle information by analyzing the image of the puncture needle detection marker 22 in the captured image 26. The puncture needle information includes position information indicating the position of the puncture needle 18 and orientation information indicating the orientation of the puncture needle 18.

[0081] The position information may be, for example, three-dimensional coordinates in a camera coordinate system. The position of the base of the puncture needle 18 (i.e., the part that is not inserted into the subject E) is detected as the position of the puncture needle 18, but because the shape (e.g., length, etc.) of the puncture needle 18 is known information, the position of the part of the puncture needle 18 that is inserted into the subject E (e.g., the tip of the puncture needle 18) can be detected based on the position information of the puncture needle 18 and the shape of the puncture needle 18.

[0082] Furthermore, the orientation information may be a rotation angle relative to a predetermined axis (for example, X-axis, Y-axis, Z-axis) in the camera coordinate system. The orientation of the puncture needle 18 is, in other words, the extension direction of the puncture needle 18.

[0083] Note that a known method can be used to detect the position and orientation of the puncture needle 18 in the camera coordinate system from the image of the puncture needle detection marker 22 included in the captured image 26, so detailed explanation will be omitted.

[0084] The puncture needle information acquisition unit 62 may perform calibration before detecting the position and orientation of the puncture needle 18 based on the captured image 26. Specifically, the operator sets the position and orientation of the puncture needle 18 to a predetermined position and orientation, and inputs a calibration instruction to the ultrasound diagnostic device 16 in that state. The puncture needle information acquisition unit 62 detects the position and orientation of the puncture needle 18 at the time the calibration instruction was input, based on the puncture needle detection marker 22 included in the captured image 26, and sets this as the reference position and reference orientation. This makes it possible to obtain the relationship between information indicating the position and orientation of the puncture needle 18 detected by the puncture needle information acquisition unit 62 (for example, the position and orientation in the camera coordinate system) and the real space coordinate system.

[0085] As described above, the captured image 26 also includes images of the body surface detection markers 24 for detecting the position and posture of the body surface of the subject E (see FIG. 2). The puncture needle information acquisition unit 62 may detect the position and posture of the body surface of the subject E by analyzing the images of the body surface detection markers 24 in the captured image 26. Then, the puncture needle information acquisition unit 62 may detect the position and posture of the puncture needle 18 relative to the position and posture of the body surface of the subject E. This makes it possible to obtain the position and posture of the puncture needle 18 with respect to the subject E, taking into account any fluctuations in the position or posture of the subject E.

[0086] The puncture needle information acquisition unit 62 may detect the position and orientation of the puncture needle 18 by a method other than analyzing the captured image 26. For example, the puncture needle 18 may be provided with a position and orientation sensor such as a magnetic sensor, and the position and orientation of the puncture needle 18 may be detected based on the detection value of the position and orientation sensor.

[0087] The puncture needle information acquisition unit 62 converts the puncture needle information in the real space coordinate system (or the camera coordinate system) into puncture needle information in the model coordinate system based on the relationship between the real space coordinate system and the model coordinate system.

[0088] The puncture needle deviation determination unit 64 determines that the puncture needle 18 has deviated from the recommended puncture route 72 when the distance between the puncture needle 18 at its current position in the model coordinate system and the recommended puncture route 72 is greater than or equal to a predetermined distance threshold, based on the puncture needle information acquired by the puncture needle information acquisition unit 62.

[0089] 12 is a diagram showing the current puncture needle 18 and the recommended puncture route 72 in the model coordinate system. Because the puncture needle 18 and the recommended puncture route 72 both have elongated shapes, for example, the puncture needle displacement determination unit 64 may determine the maximum distance between the puncture needle 18 and the recommended puncture route 72 (in the example of FIG. 12 , the length of the perpendicular line drawn from the tip 18a of the puncture needle 18 to the recommended puncture route 72) as the distance between the puncture needle 18 at the current position and the recommended puncture route 72.

[0090] In addition, the puncture needle deviation determination unit 64 determines that the puncture needle 18 has deviated from the recommended puncture route 72 when the difference in angle θ (see Figure 12) between the current posture (extension direction) of the puncture needle 18 and the extension direction of the recommended puncture route 72 is greater than or equal to a predetermined angle threshold.

[0091] In this embodiment, the puncture needle misalignment determination unit 64 calculates the distance and the angle difference in the extension direction between the puncture needle 18 at the current position and the recommended puncture route 72 in a model coordinate system, but the puncture needle misalignment determination unit 64 may also perform this calculation in a real space coordinate system.

[0092] Furthermore, as shown in FIG. 11 , when the cross section of the real-time image 84 and the cross section of the reconstructed ultrasound image 80 are the same cross section, the puncture needle displacement determination unit 64 may also take into consideration the comparison result between the image 90 of the puncture needle 18 included in the real-time image 84 and the recommended puncture route image 82, and determine that the puncture needle 18 has deviated from the recommended puncture route 72.

[0093] The display control unit 36 ​​as a notification processing unit notifies the operator when the puncture needle displacement determination unit 64 determines that the puncture needle 18 has deviated from the recommended puncture route 72 .

[0094] 13 is a diagram showing an example of notification that the puncture needle 18 has deviated. When it is determined that the puncture needle 18 has deviated from the recommended puncture route 72, the display control unit 36 ​​notifies the operator that the puncture needle 18 has deviated from the recommended puncture route 72 by, for example, displaying an icon 92a as shown in FIG. 13. The display control unit 36 ​​may change the icon 92a to be displayed depending on the amount of deviation of the puncture needle 18 from the recommended puncture route 72. For example, the display control unit 36 ​​may display a triangle mark as the icon 92a when the puncture needle 18 has deviated from the recommended puncture route 72 but the amount of deviation is relatively small, and may display an cross mark as the icon 92a when the amount of deviation is relatively large.

[0095] 14 is a diagram showing an example of notification that the puncture needle 18 has not deviated from the recommended puncture route 72. Even when the puncture needle 18 has not deviated from the recommended puncture route 72, the display control unit 36 ​​may notify the operator that the puncture needle 18 has not deviated from the recommended puncture route 72 by displaying an icon 92b different from the icon 92a, as shown in FIG.

[0096] The puncture assistance device according to the present disclosure has been described above, but the puncture assistance device according to the present disclosure is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the invention.

[0097] For example, in each of the above embodiments, the puncture support device is the ultrasound diagnostic device 16, and the ultrasound diagnostic device 16 has the functions of the image forming unit 34, the display control unit 36, the probe information acquiring unit 50, the volume data acquiring unit 52, the model forming unit 54, the puncture route identifying unit 56, the recommended position and posture identifying unit 58, the alignment index calculating unit 60, the puncture needle information acquiring unit 62, and the puncture needle displacement determining unit 64. However, these functions do not necessarily have to be performed by the ultrasound diagnostic device 16. For example, these functions may be performed by a server computer or the like communicably connected to the ultrasound diagnostic device 16. Furthermore, instead of all of the above functions being performed by a single device, each of the above functions may be performed by cooperation between multiple devices. [Explanation of symbols]

[0098] 10 Puncture assistance system, 12 camera, 14 ultrasound probe, 16 ultrasound diagnostic device, 18 puncture needle, 20 probe detection marker, 22 puncture needle detection marker, 24 body surface detection marker, 26 captured image, 30 transmitter / receiver, 32 signal processing unit, 34 image formation unit, 36 display control unit, 38 display, 40 communication interface, 42 input interface, 44 memory, 46 volume data, 48 control unit, 50 probe information acquisition unit, 52 volume data acquisition unit, 54 model formation unit, 56 puncture route identification unit, 58 recommended position and posture identification unit, 60 alignment index calculation unit, 62 puncture needle information acquisition unit, 64 puncture needle displacement determination unit, 70 subject model, 72 recommended puncture route, 80 reconstructed ultrasound image, 82 recommended puncture route image, 84 real-time image, 86 body mark, 88 Alignment index image, 90 image, 92a,92b icon.

Claims

1. a probe information acquisition unit that acquires probe information indicating the position and orientation of an ultrasound probe; a volume data acquisition unit that acquires volume data based on received signals obtained by transmitting and receiving ultrasonic waves from the ultrasonic probe to a subject; a model forming unit that forms, based on the volume data, a subject model including a puncture target model that is a three-dimensional model representing the puncture target and an avoided tissue model that is a three-dimensional model representing an avoided tissue that is a tissue that the puncture needle should avoid when puncturing the puncture target, and that has position information indicating each position of the subject model based on the position and orientation of the ultrasound probe when the received signal is acquired; a puncture route specifying unit that specifies a recommended puncture route, which is a route that a puncture needle should take from a body surface position of the subject to the puncture target model while avoiding the avoidable tissue model, in the subject model; a notification processing unit that notifies an operator of the recommended puncture route; A puncture assistance device comprising:

2. the puncture route specifying unit specifies the recommended puncture route so that the shortest distance between the recommended puncture route and the avoided tissue model is equal to or greater than a first threshold distance. The puncture support device according to claim 1 .

3. when a plurality of candidate puncture routes that reach the puncture target model from the body surface position of the subject while avoiding the avoidable tissue model are identified, the puncture route identifying unit identifies the candidate puncture route with the shortest length as the recommended puncture route among the plurality of candidate puncture routes.

3. The puncture support device according to claim 1 or 2.

4. the puncture target is a tumor, the puncture needle is a cauterization puncture needle for cauterizing the tumor, the puncture route specifying unit specifies the recommended puncture route so that an end of the recommended puncture route on a tumor model side as the puncture target model is at a second threshold distance or more from a tissue model other than the tumor model. The puncture support device according to claim 1 .

5. a recommended position and orientation specifying unit that specifies, based on the recommended puncture route, a recommended probe position and orientation, which is a position and orientation of the ultrasound probe, such that a scanning plane of the ultrasound probe includes the recommended puncture route and is parallel to the recommended puncture route; Furthermore, the notification processing unit notifies an operator of the recommended probe position and orientation. The puncture support device according to claim 1 .

6. the notification processing unit notifies an operator of guide information for transitioning the position and orientation of the ultrasound probe from the current position and orientation of the ultrasound probe to the recommended probe position and orientation based on a difference between the current position and orientation of the ultrasound probe and the recommended probe position and orientation. The puncture support device according to claim 5 .

7. a matching index calculation unit that calculates a matching index, which is an index indicating the degree of match between a reconstructed ultrasound image that includes the recommended puncture route and is cut out from the subject model at a cross section parallel to the recommended puncture route, and a real-time image that is an ultrasound tomographic image formed at the current position and orientation of the ultrasound probe; Furthermore, the notification processing unit notifies an operator of the consistency index. The puncture support device according to claim 6 .

8. a puncture needle information acquisition unit that acquires puncture needle information indicating the current position and posture of the puncture needle; a puncture needle deviation determination unit that determines, based on the puncture needle information, that the puncture needle has deviated from the recommended puncture route when the distance between the puncture needle at its current position and the recommended puncture route is equal to or greater than a distance threshold, or when the difference in angle between the extension direction of the current puncture needle and the extension direction of the recommended puncture route is equal to or greater than an angle threshold; Furthermore, the notification processing unit notifies an operator when it is determined that the puncture needle has deviated from the recommended puncture route. The puncture support device according to claim 1 .

9. the probe information acquisition unit detects the relative position and orientation of the ultrasound probe with respect to the body surface of the subject based on a captured image acquired by a camera capturing an image of a probe detection marker attached to the ultrasound probe and a body surface detection marker attached to the body surface of the subject; The puncture support device according to claim 1 or 6.

10. the probe information acquisition unit detects the position and orientation of the ultrasound probe relative to the body surface of the subject based on a captured image acquired by a camera capturing an image of a probe detection marker attached to the ultrasound probe and a body surface detection marker attached to the body surface of the subject; the puncture needle information acquisition unit detects the position and orientation of the puncture needle relative to the body surface of the subject based on a captured image acquired by a camera capturing an image of a puncture needle detection marker attached to the puncture needle and the body surface detection marker; The puncture support device according to claim 8 .

11. Computer, a probe information acquisition unit that acquires probe information indicating the position and orientation of an ultrasound probe; a volume data acquisition unit that acquires volume data based on received signals obtained by transmitting and receiving ultrasonic waves from the ultrasonic probe to a subject; a model forming unit that forms, based on the volume data, a subject model including a puncture target model that is a three-dimensional model representing the puncture target and an avoided tissue model that is a three-dimensional model representing an avoided tissue that is a tissue that the puncture needle should avoid when puncturing the puncture target, and that has position information indicating each position of the subject model based on the position and orientation of the ultrasound probe when the received signal is acquired; a puncture route specifying unit that specifies a recommended puncture route, which is a route that a puncture needle should take from a body surface position of the subject to the puncture target model while avoiding the avoidable tissue model, in the subject model; A puncture assistance program characterized by causing the program to function as a puncture assistance program.

Images