Puncture support apparatus, operation method thereof, and program for operating puncture support apparatus

Abstract

To provide a puncture support apparatus, an operation method thereof, and a program for operating the puncture support apparatus, which can ascertain a positional relationship between a plurality of surgical tools in a body and support an operation of a puncture needle.SOLUTION: Distal end coordinates of a puncture needle 13 are acquired on the basis of a first image. Distal end coordinates of an ultrasound probe 12 are acquired on the basis of the first image. Coordinates of a target region in an ultrasound image acquired by the ultrasound probe 12 are acquired. Support information for supporting a puncture operation of the puncture needle 13 is generated by using at least any one of the distal end coordinates of the puncture needle 13, the distal end coordinates of the ultrasound probe 12, or the coordinates of the target region 29. The support information is displayed on a display 27 by being superimposed on the first image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a puncture support device that supports the operation of a puncture needle used in laparoscopic surgery, etc., a method for operating the same, and an operating program for the puncture support device. [Background technology]

[0002] While open surgery has traditionally been widely performed, less invasive surgeries such as laparoscopic surgery and robotic surgery are becoming more common. These surgical procedures involve creating small port holes in the body and inserting surgical instruments through these holes, which has the advantage of minimizing the amount of incision required in the patient's body.

[0003] Furthermore, surgical tools used in laparoscopic surgery include ultrasound probes for detecting target regions distributed in the depth direction within the body, such as lesions. The tip of the ultrasound probe used in laparoscopic surgery is provided with, for example, a puncture hole or groove, and by passing a puncture needle through the puncture hole or groove of the ultrasound probe, it becomes possible to puncture the organ at any angle (see, for example, Patent Document 1). The puncture needle is used, for example, to inject ICG (Indocyanine green) or blue dye (Indigo carmine) into the portal vein to determine the resection site in liver cancer surgery.

[0004] In Patent Document 2, a magnetic field generator is used to detect a magnetic position sensor built into the tip of the puncture needle, thereby detecting the relative position of the puncture needle and the ultrasound probe, and the position of the puncture needle and the direction in which the puncture needle moves are graphically displayed on the ultrasound tomography. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-80023 [Patent Document 2] U.S. Patent No. 8,688,196 [Non-patent literature]

[0006] [Non-Patent Document 1] Masahiro Hasegawa, et al., "Accuracy of augmented reality with computed tomography-based navigation in total hip arthroplasty", [online], [Retrieved November 6, 2023], Internet,<URL:https: / / www.ncbi.nlm.nih.gov / pmc / articles / PMC10481587 / > Summary of the Invention [Problem to be solved by the invention]

[0007] Although laparoscopic surgery has the above-mentioned advantage of being able to reduce the amount of incision, it is more difficult to grasp the relative positions of multiple surgical instruments from camera images observing the inside of the body compared to open surgery. In particular, when using a puncture needle to identify the surgical site in the liver as described above, it is difficult to properly insert the puncture needle into only the targeted blood vessel among the blood vessels that are spread throughout the liver. Non-Patent Document 1 describes surgical navigation that uses AR markers to identify the relative positions of the pelvis and surgical instruments, but does not describe or suggest how to identify the relative positions of the puncture needle and other surgical instruments, such as an ultrasound probe.

[0008] The present invention aims to provide a puncture assistance device, an operating method thereof, and an operating program for the puncture assistance device that can grasp the positional relationships of multiple surgical tools within the body, such as an ultrasound probe and a puncture needle, and can assist in the operation of the puncture needle. [Means for solving the problem]

[0009] The puncture support device of the present invention includes a processor, which acquires a first image acquired by an imaging device that photographs the puncture needle or ultrasound probe, acquires the tip coordinates of the puncture needle based on the first image, acquires the tip coordinates of the ultrasound probe based on the first image, acquires the coordinates of the target area in the ultrasound image acquired by the ultrasound probe, generates support information to assist in the puncture operation of the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area, and displays the support information on the display by superimposing it on the first image.

[0010] The first image is a rigid endoscope image taken with a rigid endoscope, which is an imaging device, and an internal puncture needle marker is attached to the part of the puncture needle that is inserted into the body, and an internal probe marker is attached to the part of the ultrasound probe that is inserted into the body, and it is preferable that the tip coordinates of the puncture needle are calculated based on the internal puncture needle marker included in the rigid endoscope image, and the tip coordinates of the ultrasound probe are calculated based on the internal probe marker included in the rigid endoscope image.

[0011] The first image is a camera image acquired by a camera installed outside the body, which is an imaging device, and an extracorporeal puncture needle marker is attached to the part of the puncture needle located outside the body, and an extracorporeal probe marker is attached to the part of the ultrasound probe located outside the body, and it is preferable that the tip coordinates of the puncture needle are calculated based on the extracorporeal puncture needle marker included in the camera image, and the tip coordinates of the ultrasound probe are calculated based on the extracorporeal probe marker included in the camera image.

[0012] The processor acquires a first image and a second image different from the first image from a plurality of imaging devices, the first image being a rigid endoscope image taken with a rigid endoscope which is an imaging device, the second image being a camera image acquired with a camera installed outside the body which is an imaging device, and it is preferable that an internal probe marker is attached to a part of the ultrasound probe which is inserted into the body and an external puncture needle marker is attached to a part of the puncture needle which is located outside the body, and the tip coordinates of the ultrasound probe are calculated based on the internal probe marker included in the rigid endoscope image, and the tip coordinates of the puncture needle are calculated based on the external puncture needle marker included in the camera image.

[0013] The processor acquires a first image and a second image different from the first image from multiple imaging devices, the first image being a rigid endoscope image taken with a rigid endoscope as the imaging device, the second image being a camera image taken with a camera installed outside the body as the imaging device, and it is preferable that an internal puncture needle marker is attached to the part of the puncture needle that is inserted into the body and an external probe marker is attached to the part of the ultrasound probe that is located outside the body, the tip coordinates of the ultrasound probe are calculated based on the external probe marker included in the camera image, and the tip coordinates of the puncture needle are calculated based on the internal puncture needle marker included in the rigid endoscope image.

[0014] It is preferable that an extracorporeal rigid endoscope marker is attached to the portion of the rigid endoscope located outside the body, and the coordinates of the tip of the rigid endoscope are calculated based on the extracorporeal rigid endoscope marker included in the camera image.

[0015] The processor preferably generates, as support information, a cross section guide that displays the position of the ultrasound cross section based on the tip coordinates of the ultrasound probe, and an extension line that extends from the tip of the puncture needle based on the tip coordinates of the puncture needle.

[0016] The processor generates a cross-section guide that displays the position of the ultrasound cross-section as support information based on the tip coordinates of the ultrasound probe, and it is preferable that the processor differs in the display manner of the support information depending on whether the support information is in front of at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image, or whether the support information overlaps with or is behind at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image.

[0017] The processor generates, as support information, a cross-section guide that displays the position of the ultrasound cross-section based on the tip coordinates of the ultrasound probe, and generates, as support information, an extension line extending from the tip of the puncture needle based on the tip coordinates of the puncture needle, and it is preferable that the processor differs in the display manner of the extension line depending on whether the extension line is in front of at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image, or whether it overlaps with or is behind at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image.

[0018] The processor preferably generates a puncture needle guide for guiding the puncture needle as support information based on the tip coordinates of the ultrasound probe, in accordance with the position of the needle insertion hole of the ultrasound probe, and superimposes and displays the puncture needle guide on the first image. The first image is preferably a rigid endoscope image taken with a rigid endoscope, which is an imaging device. The first image is preferably a camera image acquired with an externally installed camera, which is an imaging device.

[0019] It is preferable that the processor generates a puncture point, which is a target insertion position of the puncture needle, as the support information based on the coordinates of the target area. It is preferable that the processor generates an extension line extending from the tip of the puncture needle as the support information based on the tip coordinates of the puncture needle, generates the puncture point, which is the target insertion position of the puncture needle, as the support information based on the coordinates of the target area, and when the extension line overlaps with the puncture point, generates the distance from the tip of the puncture needle to the target area as the support information.

[0020] It is preferable that the processor generates a three-dimensional shape model of the target area as support information based on the ultrasound image and the coordinates of the target area, and generates a three-dimensional relationship from the tip of the puncture needle to the target area as support information based on the tip coordinates of the puncture needle and the coordinates of the target area.

[0021] The present invention provides a method for operating a puncture assistance device equipped with a processor, the method comprising the steps of: acquiring a first image acquired by an imaging device that photographs the puncture needle or an ultrasound probe; acquiring the tip coordinates of the puncture needle based on the first image; acquiring the tip coordinates of the ultrasound probe based on the first image; acquiring the coordinates of the target area in the ultrasound image acquired by the ultrasound probe; generating support information to assist the puncture operation with the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area; and superimposing the support information on the first image and displaying it on a display.

[0022] The operating program for the puncture assistance device of the present invention causes a processor to execute the following functions: acquire a first image acquired by an imaging device that photographs the puncture needle or the ultrasound probe; acquire the tip coordinates of the puncture needle based on the first image; acquire the tip coordinates of the ultrasound probe based on the first image; acquire the coordinates of the target area in the ultrasound image acquired by the ultrasound probe; generate support information to assist the puncture operation with the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area; and display the support information on a display by superimposing it on the first image. [Effects of the Invention]

[0023] According to the present invention, it is possible to grasp the positional relationship between a plurality of surgical tools in the body, such as an ultrasound probe or a puncture needle, and to assist in the operation of the puncture needle. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a schematic diagram of a laparoscopic system. [Figure 2] FIG. 1 is an image diagram of an ultrasound image. [Figure 3] FIG. 10 is an explanatory diagram showing a first pattern of acquisition patterns for the tip coordinates of the ultrasonic probe and the tip coordinates of the puncture needle. [Figure 4] FIG. 10 is an explanatory diagram showing a second pattern of acquisition patterns for the tip coordinates of the ultrasonic probe and the tip coordinates of the puncture needle. [Figure 5] FIG. 10 is an explanatory diagram showing a third pattern of acquiring the coordinates of the tip of the ultrasonic probe and the coordinates of the tip of the puncture needle. [Figure 6] FIG. 10 is an explanatory diagram showing a fourth pattern of acquiring the coordinates of the tip of the ultrasonic probe and the coordinates of the tip of the puncture needle. [Figure 7] 10 is an explanatory diagram showing a case where an extension line of a puncture needle overlaps a cross section guide in a first pattern of a generation pattern and a display pattern of support information. FIG. [Figure 8] 10 is an explanatory diagram showing a case where an extension line of the puncture needle is on the near side of the cross section guide in a first pattern of generation patterns and display patterns of support information. FIG. [Figure 9] FIG. 10 is an explanatory diagram showing a second pattern of generation and display patterns of support information. [Figure 10] FIG. 10 is an explanatory diagram showing a third pattern of generation and display patterns of support information. [Figure 11] FIG. 10 is an explanatory diagram showing a case where the extension line of the puncture needle does not overlap the puncture point in the fourth generation pattern and display pattern of the support information. [Figure 12] FIG. 10 is an explanatory diagram showing a case where an extension line of the puncture needle overlaps the puncture point in the fourth generation pattern and display pattern of the support information. [Figure 13] FIG. 10 is an explanatory diagram showing a fifth pattern of generation and display patterns of support information. [Figure 14] 10 is a flowchart showing a series of steps for assisting the manipulation of a puncture needle. [Figure 15] FIG. 10 is an explanatory diagram showing a case where an extension line of the puncture needle is on the front side of the ultrasound probe in the first generation pattern and display pattern of the support information. [Figure 16] FIG. 10 is an explanatory diagram showing a case where an extension line of a puncture needle overlaps with an ultrasound probe in a first generation pattern and display pattern of support information. [Figure 17] FIG. 10 is an explanatory diagram showing a case where an extension line of a puncture needle is on the near side of a cross section of an ultrasound image in a first pattern of generation and display patterns of support information. [Figure 18] 10 is an explanatory diagram showing a case where an extension line of a puncture needle overlaps a cross section of an ultrasound image in a first generation pattern and display pattern of support information. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0025] As shown in FIG. 1, a laparoscopic system 10 uses a rigid endoscope 11 to observe the inside of a body, and an ultrasound probe 12 to detect a target area in the depth direction of the body, such as a lesion. In the laparoscopic system 10, a puncture needle 13 is also inserted into the target area inside the body at an arbitrary angle. The puncture needle 13 is used, for example, to inject ICG (Indocyanine green) or a blue dye (indigo carmine) into the portal vein to determine the resection site in liver cancer surgery. A camera 14 captures images of the rigid endoscope 11, ultrasound probe 12, or puncture needle 13 used during laparoscopic surgery.

[0026] The laparoscope system 10 is provided with a puncture support device 20 for supporting the insertion of a puncture needle 13 into the body. Rigid endoscope images taken by the rigid endoscope 11 are transmitted to the puncture support device 20 by wire or wirelessly. Ultrasound images obtained by the ultrasound probe 12 are transmitted to the puncture support device 20 by wire or wirelessly. Camera images taken by the camera 14 are transmitted to the puncture support device 20 by wire or wirelessly. The camera 14 may be a web camera or the like.

[0027] The puncture support device 20 includes an image acquisition unit 21, a puncture needle coordinate acquisition unit 22, a probe coordinate acquisition unit 23, a target region coordinate acquisition unit 24, a support information generation unit 25, a display control unit 26, and a display 27. The puncture support device 20 is provided with a program memory (not shown) that stores a specific program. A control unit (not shown) that constitutes a processor executes the specific program, thereby realizing the functions of the image acquisition unit 21, the puncture needle coordinate acquisition unit 22, the probe coordinate acquisition unit 23, the target region coordinate acquisition unit 24, the support information generation unit 25, and the display control unit 26.

[0028] The image acquisition unit 21 acquires a first image acquired by an imaging device that captures the puncture needle 13 or the ultrasound probe 12. The puncture needle coordinate acquisition unit 22 acquires the tip coordinates of the puncture needle 13 based on the first image. There are multiple patterns for acquiring the tip coordinates of the puncture needle 13, and these will be described in detail later. The probe coordinate acquisition unit 23 acquires the tip coordinates of the ultrasound probe 12 based on the first image. There are multiple patterns for acquiring the tip coordinates of the ultrasound probe 12, and these will be described in detail later.

[0029] The target area coordinate acquisition unit 24 acquires the coordinates of the target area in the ultrasound image acquired by the ultrasound probe 12. As shown in FIG. 2, the target area coordinate acquisition unit 24 acquires an image of a target area 29 such as a lesion by performing a target area detection process on an ultrasound image 28. The support information generation unit 25 generates support information for supporting the puncture operation of the puncture needle 13 using at least one of the tip coordinates of the puncture needle 13, the tip coordinates of the ultrasound probe 12, and the coordinates of the target area. The display control unit 26 displays the support information on the display by superimposing it on the first image. There are multiple patterns for generating and displaying the support information, and these will be described in detail below.

[0030] The following four patterns will be described for acquiring the tip coordinates of the ultrasound probe 12 and the tip coordinates of the puncture needle 13. The first pattern is a pattern in which the tip coordinates of the ultrasound probe 12 and the tip coordinates of the puncture needle 13 are acquired from an image acquired inside the body. As shown in FIG. 3 , the puncture needle coordinate acquisition unit 22 acquires the tip coordinates of the puncture needle 13 based on a rigid endoscope image, which is the first image. To acquire the tip coordinates of the puncture needle 13, an internal puncture needle marker 30 is attached to the portion of the puncture needle 13 that is inserted into the body. The internal puncture needle marker 30 is provided at a certain distance inside the body from the tip of the puncture needle 13. Therefore, the puncture needle coordinate acquisition unit 22 calculates the tip coordinates of the puncture needle 13 from a known vector VM between the internal puncture needle marker 30 and the tip of the puncture needle 13, and a vector VN between the internal puncture needle marker 30 detected from the rigid endoscope image and the rigid endoscope 11. The internal puncture needle marker 30 is preferably an AR marker.

[0031] Furthermore, the probe coordinate acquisition unit 23 acquires the tip coordinates of the ultrasound probe 12 based on the rigid endoscope image, which is the first image. To acquire the tip coordinates of the ultrasound probe 12, an internal probe marker 31 is attached to the portion of the ultrasound probe 12 that is inserted into the body. The internal probe marker 31 is provided at the tip of the ultrasound probe 12. Therefore, the tip coordinates of the ultrasound probe 12 are calculated from a vector VP between the internal probe marker 31 and the rigid endoscope 11, which is detected from the rigid endoscope image. It is preferable that the internal probe marker 31 is also an AR marker.

[0032] The second pattern is a pattern in which the tip coordinates of the ultrasound probe 12 and the tip coordinates of the puncture needle 13 are acquired from an image acquired outside the body. As shown in FIG. 4, the puncture needle coordinate acquisition unit 22 acquires the tip coordinates of the puncture needle 13 based on a camera image, which is the first image. In order to acquire the tip coordinates of the puncture needle 13, an extracorporeal puncture needle marker 33 is attached to a portion of the puncture needle 13 located outside the body. The extracorporeal puncture needle marker 33 is provided at a portion outside the body that is a certain distance away from the tip of the puncture needle 13. Therefore, the puncture needle coordinate acquisition unit 22 calculates the tip coordinates of the puncture needle 13 from a known vector VQ between the extracorporeal puncture needle marker 33 and the tip of the puncture needle 13, and a vector VR between the extracorporeal puncture needle marker 33 detected from the camera image and the camera 14. The extracorporeal puncture needle marker 33 is preferably an AR marker.

[0033] Furthermore, the probe coordinate acquisition unit 23 acquires the tip coordinates of the ultrasonic probe 12 based on the camera image, which is the first image. To acquire the tip coordinates of the ultrasonic probe 12, an extracorporeal probe marker 34 is attached to a portion of the ultrasonic probe 12 located inside the body. The extracorporeal probe marker 34 is provided at a specific portion outside the body of the ultrasonic probe 12. Therefore, the tip coordinates of the ultrasonic probe 12 are calculated from a known vector VS between the extracorporeal probe marker 34 and the tip of the ultrasonic probe 12, and a vector VT between the extracorporeal probe marker 34 detected from the camera image and the camera 14. It is preferable that the intracorporeal probe marker 31 is also an AR marker.

[0034] The third pattern is a pattern in which the tip coordinates of the ultrasound probe 12 and the tip coordinates of the puncture needle 13 are acquired from images acquired inside and outside the body, and is different from the fourth pattern. As shown in FIG. 5, the puncture needle coordinate acquisition unit 22 calculates the tip coordinates of the puncture needle 13 from the camera image, which is the second image. The method of calculating the tip coordinates of the puncture needle 13 is the same as in the second pattern. Furthermore, it is preferable that the probe coordinate acquisition unit 23 calculates the tip coordinates of the ultrasound probe 12 from the rigid endoscope image, which is the first image. The method of calculating the tip coordinates of the ultrasound probe 12 is the same as in the first pattern. When the tip coordinates of the rigid endoscope 11 are acquired by the camera 14, it is preferable to attach an extracorporeal rigid endoscope marker 36 to a portion of the rigid endoscope 11 located outside the body. In this case, the coordinates of the tip of the rigid endoscope 11 are calculated based on a vector VU between the camera 14 and the external rigid endoscope marker 36 obtained from the camera image and a known vector VW between the external rigid endoscope marker 36 and the tip of the rigid endoscope 11. The external rigid endoscope marker 36 is also preferably an AR marker.

[0035] The fourth pattern is a pattern in which the tip coordinates of the ultrasound probe 12 and the tip coordinates of the puncture needle 13 are acquired from images acquired inside and outside the body, and is different from the third pattern. As shown in FIG. 6 , the puncture needle coordinate acquisition unit 22 calculates the tip coordinates of the puncture needle 13 from the rigid endoscope image, which is the first image. The method of calculating the tip coordinates of the puncture needle 13 is the same as in the first pattern. Furthermore, the probe coordinate acquisition unit 23 calculates the tip coordinates of the ultrasound probe 12 from the camera image, which is the second image. The method of calculating the tip coordinates of the ultrasound probe 12 is the same as in the second pattern. When the tip coordinates of the rigid endoscope 11 are acquired by the camera 14, it is preferable to attach an extracorporeal rigid endoscope marker 36 to a portion of the rigid endoscope 11 located outside the body, as described above. In this case, the coordinates of the tip of the rigid endoscope 11 are calculated based on a vector VU between the camera 14 and the external rigid endoscope marker 36 obtained from the camera image and a known vector VW between the external rigid endoscope marker 36 and the tip of the rigid endoscope 11. The external rigid endoscope marker 36 is also preferably an AR marker.

[0036] The following five patterns of generating and displaying support information will be described. In the first pattern, as shown in FIGS. 7 and 8, the support information generator 25 generates a cross section guide 40 that displays the position of an ultrasound cross section as support information based on the tip coordinates of the ultrasound probe 12. The display controller 26 superimposes an ultrasound image 28 below the ultrasound probe 12 on the rigid endoscope image, and also superimposes the cross section guide 40. The cross section guide 40 is superimposed according to the shape of the tip of the ultrasound probe 12 and is displayed slightly extending from the tip of the ultrasound probe 12. Furthermore, the support information generator 25 generates an extension line extending from the tip of the puncture needle as support information based on the tip coordinates of the puncture needle. The display controller 26 superimposes the extension line 42 on the rigid endoscope image.

[0037] The display control unit 26 changes the display mode of the support information depending on whether the support information is located in front of at least one of the cross sections of the cross section guide 40, the ultrasound probe 12, or the ultrasound image 28 or whether the support information is overlapping or behind at least one of the cross sections of the cross section guide 40, the ultrasound probe 12, or the ultrasound image 28. Specifically, the display control unit 26 changes the display mode of the extension line 42 depending on whether the extension line 42 is located in front of at least one of the cross sections of the cross section guide 40, the ultrasound probe 12, or the ultrasound image 28 or whether the extension line 42 is overlapping or behind at least one of the cross sections of the cross section guide 40, the ultrasound probe 12, or the ultrasound image 28. The display mode of the extension line 42 can be changed by, for example, displaying or hiding a portion of the extension line 42. By displaying or hiding support information such as the extension line 42 in this manner, a sense of depth can be achieved.

[0038] When generating the cross-section guide 40 and the extension line 42, the display control unit 26 changes the display mode of the extension line 42 depending on whether the extension line 42 is located in front of the cross-section guide 40 or whether it overlaps or is located behind the cross-section guide 40. Specifically, when the extension line 42 overlaps or is located behind the cross-section guide 40, the display control unit 26 hides the portion of the extension line 42 that overlaps or is located behind the cross-section guide 40 (displayed as a dotted line in FIG. 8 ), and when the extension line 42 is located in front of the cross-section guide 40, the display control unit 26 displays the extension line (displayed as a solid line in FIG. 7 ). This allows the display mode of the extension line to have a sense of depth. In FIG. 7 , the entire extension line 42 is displayed, so it can be seen that it does not overlap the cross-section guide 40 but is located in front of it. In FIG. 8 , a portion of the extension line is hidden, so it can be seen that the hidden portion overlaps the cross-section guide 40. In addition, the color of the extension line 42 may be different depending on whether it is on the front side of the cross-section guide 40 or whether it overlaps with or is on the back side of the cross-section guide 40 (for example, the color of the extension line 42 may be blue when it overlaps, and red when it does not overlap).

[0039] The display control unit 26 changes the display mode of the extension line 42 depending on whether the extension line 42 is in front of the ultrasonic probe 12 or overlaps the ultrasonic probe 12 or is behind the ultrasonic probe 12. Specifically, as shown in Fig. 15, when the extension line 42 is in front of the ultrasonic probe 12, the display control unit 26 displays the extension line (displayed as a solid line in Fig. 15), and when the extension line 42 overlaps the ultrasonic probe 12 or is behind the ultrasonic probe 12, as shown in Fig. 16, the display control unit 26 hides the portion of the extension line 42 that overlaps the ultrasonic probe 12 or is behind the ultrasonic probe 12 (displayed as a dotted line in Fig. 16).

[0040] Furthermore, the display control unit 26 changes the display mode of the extension line 42 depending on whether the extension line 42 is on the front side of the cross section of the ultrasound image 28 or whether the extension line 42 overlaps with or is on the back side of the cross section of the ultrasound image 28. Specifically, as shown in Fig. 17, when the extension line 42 is on the front side of the cross section of the ultrasound image 28, the display control unit 26 displays the extension line (displayed as a solid line in Fig. 17), and when the extension line 42 overlaps with or is on the back side of the cross section of the ultrasound image 28, as shown in Fig. 18, the display control unit 26 hides (displayed as a dotted line in Fig. 18) the portion of the extension line 42 that overlaps with or is on the back side of the cross section of the ultrasound image 28.

[0041] In the second pattern, as shown in FIG. 9 , the support information generator 25 generates, as support information, a puncture needle guide 46 for guiding the puncture needle 13 in accordance with the position of a needle insertion hole 44 of the ultrasound probe based on the tip coordinates of the ultrasound probe 12. The display controller 26 superimposes an ultrasound image below the ultrasound probe 12 on the rigid endoscope image (first image), and also superimposes the puncture needle guide 46. The puncture needle guide 46 is made up of a plurality of ellipses, each of which is arranged at regular intervals toward the needle insertion hole 44. The user can reliably guide the puncture needle 13 to the needle insertion hole 44 by operating the puncture needle 13 to pass through the ellipses.

[0042] In the third pattern, as shown in Fig. 10, the image of the puncture needle guide 46 superimposed in the second pattern is superimposed on the camera image (first image) instead of the rigid endoscope image. This makes it possible to grasp, outside the body, how much the current position of the puncture needle 13 is displaced from the target insertion position. Note that, since the coordinates of the target region 29 are displayed on the camera image, it is preferable to calculate the coordinates of the target region 29 using an ultrasound image in the third pattern as well.

[0043] In the fourth pattern, as shown in Fig. 11, the support information generator 25 generates a puncture point 48, which is the target insertion position of the puncture needle 13, as support information based on the coordinates of the target region. The display controller 26 superimposes an ultrasound image below the ultrasound probe 12 on the rigid endoscope image, and displays the puncture point 48 in a specific shape (a circle in Fig. 11) in a specific color in the ultrasound image. In addition, by superimposing an extension line 42 of the puncture needle 13, the positional relationship between the puncture needle 13 and the puncture point 48 can be grasped.

[0044] Furthermore, when the support information generating unit 25 generates the puncture point 48 and the extension line 42 of the puncture needle 13, as shown in Fig. 12, if the extension line 42 overlaps with the puncture point 48, the support information generating unit 25 calculates the distance from the tip of the puncture needle 13 to the target area as support information based on the coordinates of the tip of the puncture needle 13. The calculated distance is displayed on the display 27 by the display control unit 26. In Fig. 12, "25 mm" is displayed as the distance, with scale 49 marked at 5 mm intervals.

[0045] In the fifth pattern, the support information generator 25 generates a three-dimensional model of the target region 29 as support information based on the ultrasound image 28 and the coordinates of the target region 29. The display controller 26 displays a shape model 50 representing the target region 29 on the display 27 as the three-dimensional model of the target region 29, as shown in FIG. 13 . The support information generator 25 also generates a three-dimensional relationship from the puncture needle 13 to the target region 29 as support information based on the coordinates of the tip of the puncture needle 13 and the coordinates of the target region 29. The display controller 26 displays a lattice-like rectangular parallelepiped 52 as the three-dimensional relationship, as shown in FIG. 13 . This lattice-like rectangular parallelepiped 52 allows the user to grasp the three-dimensional relationship, such as the distance and direction, between the puncture line 13 and the target region 29, thereby making it easier for the user to operate the puncture needle 13.

[0046] Next, a series of steps for assisting the operation of the puncture needle 13 during liver surgery will be described with reference to the flowchart in FIG. 14. First, the rigid endoscope 11 and the ultrasound probe 12 are inserted into the body through the abdominal cavity using a trocar or the like. The ultrasound probe 12 is provided with either an internal probe marker 31 or an external probe marker 34. An ultrasound image obtained by the ultrasound probe 12 and a rigid endoscope image obtained by the rigid endoscope 11 are separately displayed on the display 27. While observing the display 27, the user places the ultrasound probe 12 on the liver to search for the puncture target. Once the puncture target is found through the search, the user operates a user interface (not shown) connected to the puncture support device 20 to switch to a superimposed display mode in which an ultrasound image is superimposed on the rigid endoscope image. As a result, an image in which the ultrasound image is superimposed on the rigid endoscope image of the puncture target is displayed on the display 27.

[0047] Next, the user inserts the puncture needle 13 into the body while observing the display 27. The puncture needle 13 is provided with either an internal puncture needle marker 30 or an external puncture needle marker 33. The puncture needle coordinate acquisition unit 22 acquires the tip coordinates of the puncture needle 13 based on a first image acquired by an imaging device that captures the puncture needle 13 or the ultrasound probe 12. The probe coordinate acquisition unit 23 acquires the tip coordinates of the ultrasound probe 12 based on the first image. The target area coordinate acquisition unit 24 acquires the coordinates of the target area in the ultrasound image.

[0048] The support information generator 25 generates support information using at least one of the coordinates of the tip of the puncture needle 13, the coordinates of the tip of the ultrasound probe 12, and the coordinates of the target area. The support information is displayed on the display 27, superimposed on the first image. The user operates the puncture needle 13 while observing the support information displayed on the display 27.

[0049] Specifically, when the first image is a rigid endoscope image, the puncture needle 13 is provided with an internal puncture needle marker, and the ultrasound probe 12 is provided with an internal puncture needle marker 30, the tip coordinates of the puncture needle 13 and the tip coordinates of the ultrasound probe 12 are obtained from the rigid endoscope image. Then, based on the tip coordinates of the puncture needle 13, an extension line 42 extending from the tip of the puncture needle 13 is generated as support information and displayed on the display 27. Furthermore, based on the tip coordinates of the ultrasound probe 12, a cross section guide 40 indicating the position of the ultrasound cross section is generated as support information and displayed on the display 27.

[0050] The user operates the puncture needle 13 inside the body while observing the extension line 42 and cross section guide 40 displayed on the display. The extension line 42 moves in accordance with the operation of the puncture needle 13. When the puncture needle 13 moves to an appropriate position, such as when the extension line 42 overlaps the target area, the puncture needle 13 is inserted into the target site.

[0051] In the above embodiment, the hardware structure of processing units that perform various processes, such as the image acquisition unit 21, the puncture needle coordinate acquisition unit 22, the probe coordinate acquisition unit 23, the target region coordinate acquisition unit 24, the support information generation unit 25, and the display control unit 26, is made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0052] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0053] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0054] 10 Laparoscopic System 11 Rigid scope 12 Ultrasound probe 13 Puncture needle 14 Camera 20 Puncture support device 21 Image acquisition unit 22 Puncture needle coordinate acquisition unit 23 Probe coordinate acquisition unit 24 Target area coordinate acquisition unit 25 Support information generation section 26 Display control unit 27 Display 28 Ultrasound images 29 Target Area 30 Internal puncture needle marker 31 Internal probe marker 33 External puncture needle marker 34 External probe marker 36 Extracorporeal rigid endoscope marker 40 Section Guide 42 Extension line 44 Needle insertion hole 46 Puncture needle guide 48 Puncture point 49 scales 50 Shape Models 52 lattice rectangular parallelepiped

Claims

1. a processor; The processor: Acquiring a first image acquired by an imaging device that captures an image of the puncture needle or the ultrasound probe; acquiring a tip coordinate of the puncture needle based on the first image; acquiring a tip coordinate of the ultrasonic probe based on the first image; acquiring coordinates of a target region in an ultrasound image acquired by the ultrasound probe; generating support information for supporting the puncture operation of the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area; The puncture support device displays the support information on a display by superimposing it on the first image.

2. the first image is a rigid endoscope image captured by a rigid endoscope that is the imaging device, an internal puncture needle marker is attached to a portion of the puncture needle that is inserted into the body; an internal probe marker is attached to a portion of the ultrasonic probe that is inserted into the body; the tip coordinates of the puncture needle are calculated based on an internal puncture needle marker included in the rigid endoscope image; The puncture support device according to claim 1 , wherein the coordinates of the tip of the ultrasonic probe are calculated based on an internal probe marker included in the rigid endoscope image.

3. the first image is a camera image acquired by a camera installed outside the body, which is the imaging device; an extracorporeal puncture needle marker is attached to a portion of the puncture needle located outside the body; an extracorporeal probe marker is attached to a portion of the ultrasound probe located outside the body; the tip coordinates of the puncture needle are calculated based on an extracorporeal puncture needle marker included in the camera image; The puncture support device according to claim 1 , wherein the coordinates of the tip of the ultrasonic probe are calculated based on an extracorporeal probe marker included in the camera image.

4. the processor acquires the first image and a second image different from the first image from a plurality of image capture devices; the first image is a rigid endoscope image captured by a rigid endoscope that is the imaging device, the second image is a camera image acquired by an externally installed camera that is the imaging device, an internal probe marker is attached to a portion of the ultrasonic probe that is inserted into the body; An extracorporeal puncture needle marker is attached to a portion of the puncture needle located outside the body, The tip coordinates of the ultrasound probe are calculated based on an internal probe marker included in the rigid endoscope image; The puncture support device according to claim 1 , wherein the coordinates of the tip of the puncture needle are calculated based on an extracorporeal puncture needle marker included in the camera image.

5. the processor acquires the first image and a second image different from the first image from a plurality of image capture devices; the first image is a rigid endoscope image captured by a rigid endoscope as the imaging device, the second image is a camera image acquired by a camera installed outside the body as the imaging device, an internal puncture needle marker is attached to a portion of the puncture needle that is inserted into the body; An extracorporeal probe marker is attached to a portion of the ultrasound probe located outside the body, The tip coordinates of the ultrasound probe are calculated based on an extracorporeal probe marker included in the camera image; The puncture support device according to claim 1 , wherein the coordinates of the tip of the puncture needle are calculated based on an internal puncture needle marker included in the rigid endoscope image.

6. an extracorporeal rigid endoscope marker is attached to a portion of the rigid endoscope located outside the body; The puncture support device according to claim 4 or 5, wherein the coordinates of the tip of the rigid endoscope are calculated based on an extracorporeal rigid endoscope marker included in the camera image.

7. The puncture support device according to claim 1 , wherein the processor generates, as the support information, a cross-section guide that displays the position of an ultrasonic cross section, based on the tip coordinates of the ultrasonic probe.

8. The puncture support device according to claim 1 , wherein the processor generates, as the support information, an extension line extending from the tip of the puncture needle based on the coordinates of the tip of the puncture needle.

9. the processor generates, as the support information, a cross-section guide that displays the position of the ultrasonic cross-section based on the tip coordinates of the ultrasonic probe; The puncture support device according to any one of claims 1 to 6, wherein the processor changes the display mode of the support information depending on whether the support information is located in front of at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image, or whether the support information overlaps with or is located behind at least one of the cross-section guide, the ultrasound probe, or the cross-section of the ultrasound image.

10. the processor generates, as the support information, a cross section guide that displays the position of the ultrasound cross section based on the tip coordinates of the ultrasound probe, and generates, as the support information, an extension line extending from the tip of the puncture needle based on the tip coordinates of the puncture needle; The puncture support device according to any one of claims 1 to 6, wherein the processor changes the display mode of the extension line depending on whether the extension line is in front of the cross-section guide, the ultrasound probe, or at least one of the cross sections of the ultrasound image, or whether the extension line overlaps or is behind the cross-section guide, the ultrasound probe, or at least one of the cross sections of the ultrasound image.

11. 7. The puncture support device according to claim 1, wherein the processor generates, as the support information, a puncture needle guide for guiding the puncture needle in accordance with the position of a needle insertion hole of the ultrasonic probe based on the tip coordinates of the ultrasonic probe, and displays the puncture needle guide superimposed on the first image.

12. The puncture support device according to claim 10 , wherein the first image is a rigid endoscope image captured by a rigid endoscope serving as the imaging device.

13. The puncture support device according to claim 10 , wherein the first image is a camera image acquired by an externally installed camera that is the imaging device.

14. The puncture support device according to claim 1 , wherein the processor generates, as the support information, a puncture point that is a target insertion position of the puncture needle, based on the coordinates of the target region.

15. the processor generates, as the support information, an extension line extending from the tip of the puncture needle based on the tip coordinates of the puncture needle, and generates, as the support information, a puncture point that is a target insertion position of the puncture needle based on the coordinates of the target area; The puncture support device according to claim 1 , wherein the processor generates, as the support information, a distance from the tip of the puncture needle to the target area when the extension line overlaps the puncture point.

16. 7. The puncture support device according to claim 1, wherein the processor generates, as the support information, a three-dimensional shape model of the target area based on the ultrasound image and coordinates of the target area, and generates, as the support information, a three-dimensional relationship from the tip of the puncture needle to the target area based on the tip coordinates of the puncture needle and the coordinates of the target area.

17. A method for operating a puncture assistance device including a processor, comprising: The processor: acquiring a first image acquired by an imaging device that images the puncture needle or the ultrasound probe; acquiring a tip coordinate of the puncture needle based on the first image; acquiring a tip coordinate of the ultrasonic probe based on the first image; acquiring coordinates of a region of interest in an ultrasound image acquired by the ultrasound probe; generating support information for supporting the puncture operation of the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area; and displaying the assistance information on a display by superimposing the assistance information on the first image.

18. a function of acquiring a first image acquired by an imaging device that captures an image of the puncture needle or the ultrasound probe; a function of acquiring a tip coordinate of the puncture needle based on the first image; a function of acquiring a tip coordinate of the ultrasonic probe based on the first image; a function of acquiring coordinates of a target region in an ultrasound image acquired by the ultrasound probe; a function of generating support information for supporting the puncture operation of the puncture needle using at least one of the tip coordinates of the puncture needle, the tip coordinates of the ultrasound probe, and the coordinates of the target area; and a function of displaying the assistance information on a display by superimposing the assistance information on the first image.

Images