Medical support device, actuation method and actuation program of the same, and medical support system

Abstract

To provide a medical support device capable of displaying at a proper position, a puncture route according to a position or a posture of a medical appliance, an actuation method and an actuation program, and a medical support system.SOLUTION: A processor of a medical support device is configured to acquire an in-vivo image imaged by an endoscope for imaging the inside of the body of a subject, the in-vivo image in which an organ pierced by a puncture needle, an insertion part of a medical appliance inserted into the body, and a marker provided on the insertion part and capable of recognizing an image are included in an imaging range, then on the basis of the marker, derive position / posture information including at least one of a position and a posture of the insertion part, then display in a superposed manner, a puncture route of the puncture needle on a position identified in the in-vivo image, on the basis of the position / posture information.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The technology disclosed herein relates to a medical support device, an operation method and an operation program for a medical support device, and a medical support system. [Background technology]

[0002] In laparoscopic surgery, an ultrasound probe is used to obtain ultrasound images of the inside of the liver, and the puncture target, which is the location of the tumor, is identified on the ultrasound image. However, even if the puncture target is displayed on the ultrasound image, it is difficult to know from which position on the body surface and in which direction the puncture needle should be inserted, because the ultrasound image shows the inside of an organ inside the body.

[0003] To solve these problems, technologies to assist puncture are used. For example, Patent Document 1 discloses a technology that uses a magnetic navigation system that uses a puncture needle with a magnetic position sensor at its tip to display the position and posture of the tip of the puncture needle inserted into the body in real time. Patent Document 2 discloses a technology that uses a laser pointer at the tip of an ultrasound probe that has a guide groove for guiding the insertion of the puncture needle, and that irradiates laser light toward the body surface along the inclination of the guide groove. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Publication No. 8,688,196 [Patent Document 2] European Patent Publication No. 3136940 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem with Patent Document 1 is that it requires a large-scale device such as a magnetic navigation system. The problem with Patent Document 2 is that it requires a light source that emits laser light to be built into the tip of a medical device such as an ultrasound probe. To build in a light source, it is necessary to modify the internal structure of the ultrasound probe, which is more time-consuming than modifying the exterior.

[0006] The technology disclosed herein provides a medical support device, an operating method, an operating program, and a medical support system that are simply configured and capable of displaying a puncture path at an appropriate position according to the position or posture of a medical device. [Means for solving the problem]

[0007] A medical support device according to the technology disclosed herein includes a processor, which acquires an in-vivo image captured by an in-vivo camera that captures images of the inside of a subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion, derives position and orientation information including at least one of the position and orientation of the insertion portion based on the marker, and performs display control to superimpose the puncture path of the puncture needle at a position identified in the in-vivo image based on the position and orientation information.

[0008] It is preferable that the processor determines whether or not the change location where the inner wall changes and the puncture path intersect in three-dimensional space inside the body based on two in-body images taken from at least two different viewpoints, in which an operation that causes a change in the inner wall is performed from outside the body and the puncture path is superimposed, and reports the determination result.

[0009] When the change location where a change has occurred intersects with the puncture route displayed in a superimposed manner in two dimensions in both of the two in-vivo images, it is preferable to determine that the change location and the puncture route intersect in three-dimensional space.

[0010] When the operation is performed multiple times, it is preferable that the processor superimposes and displays the history of the change position of each of the multiple operations on the in-vivo image.

[0011] When the viewpoint of the intra-body camera moves during multiple operations, the processor preferably displays a history of the changed position, reflecting the amount of movement of the viewpoint.

[0012] The operation is preferably a pressing operation in which pressure is applied from outside the body to the surface of the subject, and the changed position is preferably a protruding portion where the inner wall is protruded toward the inside of the body by the pressing operation.

[0013] The processor preferably derives an intersection point in three-dimensional space between the inner wall of the body and the puncture path based on internal depth information representing a distribution of depth, which is a distance in the depth direction from a viewpoint of the internal camera.

[0014] It is preferable that a camera capable of estimating depth is used as the internal body camera, and the processor acquires depth information.

[0015] The processor preferably derives the depth information based on two internal body images taken from different viewpoints.

[0016] The processor preferably obtains the depth information using a machine learning model that takes the in-vivo images as input and outputs the depth information.

[0017] When the processor acquires an internal image in which an operation that causes a change in the inner wall of the subject's body cavity is being performed from outside the body, it is preferable that the processor determines, based on the depth information, whether or not the change position where the inner wall changes in the three-dimensional space inside the body intersects with the puncture path, and reports the determination result.

[0018] The medical instrument is preferably a medical probe capable of observing the internal structure of an organ.

[0019] The medical probe is preferably an ultrasound probe.

[0020] The medical probe preferably has an insertion section having a guide groove for engaging with the puncture needle to guide insertion of the puncture needle into a target position inside the organ, and the processor preferably identifies a position for superimposing the puncture path on the internal body image based on the relative positional relationship between the marker and the guide groove.

[0021] It is preferable that the processor identifies a position at which to superimpose the puncture path on the internal image based on a target position inside the organ identified in an internal image of the organ acquired by the medical probe and a correlation between the coordinate system of the internal image and the coordinate system of the internal image derived based on the position and orientation information.

[0022] When a puncture path is set by pre-operative simulation on a three-dimensional image of an organ acquired in advance before surgery, it is preferable that the processor identifies a position at which to superimpose the puncture path on the internal body image by utilizing the correlation between the coordinate system of the internal image of the organ acquired by the medical probe and the coordinate system of the three-dimensional image, and the correlation between the coordinate system of the internal image and the coordinate system of the internal body image derived based on the position and orientation information.

[0023] The processor preferably modifies the display of a portion of the puncture path.

[0024] A part of the puncture path is a part of the inner wall side of the body, and the processor preferably highlights the part of the inner wall side.

[0025] It is preferable that a part of the puncture path is a part on the organ side, and the processor reduces visibility of the part on the organ side compared to the inner wall side.

[0026] It is preferable that the processor detects the insertion length of the insertion portion of the medical instrument, and determines the range for changing the display based on the detected insertion length.

[0027] The processor preferably determines the extent to which the display is to be altered based on a user specification.

[0028] Preferably, the processor is capable of displaying a number of different needle paths.

[0029] A method for operating a medical support device relating to the technology disclosed herein is a method for operating a medical support device having a processor, in which the processor acquires an in-vivo image captured by an in-vivo camera that images the inside of a subject's body, the in-vivo image including within its imaging range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion, derives position and orientation information including at least one of the position and orientation of the insertion portion based on the marker, and performs display control to superimpose and display the puncture path of the puncture needle at a position identified in the in-vivo image based on the position and orientation information.

[0030] An operating program of a medical support device related to the technology disclosed herein is an operating program that causes a computer to function as a medical support device, and causes the computer to execute the following steps: acquiring an in-vivo image captured by an in-vivo camera that captures images inside the subject's body, the in-vivo image including within its capturing range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; deriving position and orientation information including at least one of the position and orientation of the insertion portion based on the marker; and controlling display to superimpose the puncture path of the puncture needle at a position identified in the in-vivo image based on the position and orientation information.

[0031] The medical support system relating to the technology disclosed herein is a medical support system including: a medical instrument having an insertion portion to be inserted into the body of a subject, the insertion portion being provided with a marker capable of image recognition; an in-body camera that captures in-body images whose imaging range includes the insertion portion, the marker, and an organ to be punctured by the puncture needle; and a medical support device having a processor, wherein the processor acquires the in-body images, derives position and orientation information based on the marker, the position and orientation information including at least one of the position and orientation of the insertion portion, and performs display control to superimpose and display the puncture path of the puncture needle at a position identified in the in-body image based on the position and orientation information. Effect of the Invention

[0032] According to the technique of the present disclosure, it is possible to display the puncture path at an appropriate position according to the position or posture of the medical device with a simple configuration. [Brief description of the drawings]

[0033] [Figure 1] 1 is a diagram showing an outline of a medical support system including a medical support device. [Diagram 2] FIG. 1 is a diagram showing the state inside the body during laparoscopic surgery. [Diagram 3] FIG. 4 is an explanatory diagram of a first guide groove of the ultrasonic probe. [Figure 4] FIG. 4 is an explanatory diagram of a second guide groove of the ultrasonic probe. [Diagram 5] 13 is a diagram showing an inserted state of the puncture needle guided by the first guide groove. FIG. [Figure 6] 13 is a diagram showing the inserted state of the puncture needle guided by the second guide groove. FIG. [Figure 7] FIG. 2 is a diagram illustrating a hardware configuration of the medical support device. [Figure 8] 13 is a flowchart showing the overall procedure of medical support processing. [Figure 9] 11A and 11B are diagrams illustrating a method for deriving position and orientation information of an ultrasonic probe based on a marker. [Figure 10] 1A and 1B are diagrams illustrating the relationship between the positions and orientations of a marker and an ultrasound probe. [Figure 11] FIG. 11 is another diagram showing the relationship between the marker and the position and orientation of the ultrasound probe. [Figure 12] 13A and 13B are diagrams showing a method for superimposing and displaying a puncture path based on position and orientation information. [Figure 13] 13A to 13C are diagrams illustrating a pressing operation by a finger FG in the second embodiment. [Figure 14] 13A and 13B are diagrams showing how an insertion position NP is searched for by a pressing operation. [Figure 15] 10 is a flowchart showing the overall procedure of medical support processing according to a second embodiment. [Figure 16]13 is a flowchart showing an example of a procedure for three-dimensional intersection determination. [Figure 17] FIG. 13 is a diagram showing a state in which the protuberance and the first puncture line intersect in both of the two surgical field images. [Figure 18] FIG. 13 is a diagram showing a state in which the protuberance and the first puncture line intersect in only one of the two operative field images. [Figure 19] 13 is a flowchart showing another example of a procedure for three-dimensional intersection determination. [Figure 20] 13A and 13B are diagrams illustrating an example of displaying an operation history of pressing operations. [Figure 21] 11A and 11B are diagrams illustrating an example in which an operation history is made to follow a change in the viewpoint of a camera. [Figure 22] FIG. 22 is a diagram illustrating the processing of a processor in the example shown in FIG. 21. [Figure 23] FIG. 13 is a diagram showing an example in which three-dimensional intersection determination is possible without relying on image analysis. [Figure 24] 13 is a flowchart showing the overall procedure of medical support processing according to a third embodiment. [Diagram 25] FIG. 13 is a diagram illustrating an example of a method for acquiring a depth map. [Figure 26] FIG. 13 illustrates an example of deriving intersection points from a depth map. [Figure 27] FIG. 13 is a diagram showing a modified example of a method for acquiring a depth map. [Figure 28] FIG. 13 is a diagram showing another modified example of the method for acquiring a depth map. [Figure 29] 13 is a flowchart showing an example of a processing procedure for performing a three-dimensional intersection determination between a first puncture line and a protuberance based on a depth map. [Diagram 30] FIG. 13 is a diagram showing an example of a process of determining whether a first puncture line and a protuberance intersect three-dimensionally based on a depth map. [Diagram 31] FIG. 13 is a diagram showing an example of highlighting a part of the inner wall side of the first puncture line. [Diagram 32] 13A and 13B are diagrams showing an example in which visibility of a part of the first puncture line on the organ side is reduced. [Diagram 33] FIG. 13 is a diagram showing an example of displaying a scale on part of the first puncture line. [Diagram 34] FIG. 13 is a diagram showing an example in which the range in which the display of the first puncture line is changed is determined based on the insertion length of the ultrasound probe. [Diagram 35] FIG. 13 is a diagram showing an example of displaying a plurality of first puncture lines. [Diagram 36] FIG. 13 is a diagram illustrating a fourth embodiment. [Figure 37] FIG. 37 is a diagram showing the procedure following FIG. 36. [Figure 38] FIG. 1 is a diagram showing a method for creating a preoperative 3D image. [Figure 39] FIG. 11 is a diagram showing a method for creating preoperative preparation information. [Diagram 40] FIG. 1 is a diagram showing a method for creating an ultrasound 3D image. [Diagram 41] FIG. 13 illustrates a method for utilizing correlation between ultrasound 3D images and preoperative 3D images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] [First embodiment] As shown in FIG. 1 and FIG. 2, the medical support system 10 is used, for example, when performing laparoscopic surgery using an endoscope 13 on a patient PT. Laparoscopic surgery is different from open surgery in that a small hole is made in the body of the patient PT and a medical device such as an endoscope 13 is inserted through the hole. The medical support system 10 provides the medical staff ST, including doctors, with a view of the surgical field inside the body of the patient PT, as well as support information for supporting medical treatment such as surgery and examination. As described later, the support information is specifically the puncture path of the puncture needle 18. Since such a medical support system 10 has a function of providing support information in real time during surgery, it is also called a surgical navigation system. The medical support system 10 includes, for example, a medical support device 11, an endoscope 13, an ultrasonic probe 14, and a display 16. The medical support device 11 is an example of a "medical support device" according to the technology of the present disclosure, and the patient PT is an example of a "subject."

[0035] The medical support device 11 is connected to an endoscope 13, an ultrasonic probe 14, and a display 16 so as to be able to communicate with them. In endoscopic surgery, a portion of the endoscope 13 and the ultrasonic probe 14, including their respective tips, is inserted into the body via a trocar 17. The trocar 17 is an insertion tool having an insertion hole through which the endoscope 13 etc. is inserted and a valve provided in the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and therefore the trocar 17 is used to insert the endoscope 13, the ultrasonic probe 14, etc. into the body.

[0036] In this example, the target site of the operation is the liver LV, and FIG. 2 shows the state in which the endoscope 13 and the ultrasonic probe 14 are inserted into the abdomen of the patient PT. The puncture needle 18 is a treatment tool that punctures a lesion such as a tumor contained in an organ as a puncture target. The puncture needle 18 is, for example, a cauterization puncture needle used to cauterize a lesion. The cauterization puncture needle is provided with an electrode to which a high-frequency voltage is applied at its tip, and when a high-frequency voltage is applied with the electrode punctured into the lesion, the lesion is necrotized by heat generated by the electrode. In this example, an example of a treatment in which a tumor 27 in the liver LV is visualized by an ultrasonic image 22, and the visualized tumor 27 (see also FIG. 5 and FIG. 6) is cauterized by the puncture needle 18 to necrotize the tumor 27 will be described as an example of the operation. The puncture needle 18 has a needle portion 18A and a grip portion 18B provided on the base end side of the needle portion 18A.

[0037] The endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT, and a camera 13B and a light source for illumination (such as an LED (Light Emitting Diode)) are built into the tip of the insertion section 13A. As an example, the endoscope 13 has a rigid endoscope with a hard insertion section 13A, and is also called a laparoscope because it is often used for observing the abdominal cavity. The camera 13B has an image sensor such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor that can capture a color image. The camera 13B is an example of an "intra-body camera" according to the technology of the present disclosure.

[0038] The endoscope 13 optically captures an operative field SF including a target site (liver LV in this example) inside the patient PT using the camera 13B. The operative field SF is a space extending inside a body cavity defined by organs and the body wall inside the body. The endoscope 13 is connected to an image processor for the endoscope (not shown), and the image processor processes the imaging signal output by the image sensor to generate an operative field image 21 of the operative field SF inside the body. The operative field image 21 is an example of an "in-vivo image captured by an in-vivo camera" according to the technology of the present disclosure.

[0039] The illumination light of the endoscope 13 may be special light such as ultraviolet light or infrared light, but visible light such as white light is used. The illumination light of the endoscope 13 may be special light such as ultraviolet light or infrared light. The special light may be light limited to a specific wavelength such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet range. The operative field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via an image processor for the endoscope. In FIG. 2, the symbols Xin and Yin indicate the coordinate system of the operative field image 21.

[0040] A first puncture line NR1 indicating the puncture path of the puncture needle 18 is superimposed on the operative field image 21 as support information for guiding the insertion of the puncture needle 18. The first puncture line NR1 will be described later.

[0041] The ultrasonic probe 14 has an insertion section 14A that is inserted into the body of the patient PT in the same manner as the endoscope 13, and an operation section 14D at the base end side of the insertion section 14A. An ultrasonic transducer 14C is built into the tip section 14B of the insertion section 14A. The ultrasonic transducer 14C transmits ultrasonic waves to a target site and receives reflected waves reflected at the target site. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown). The image processing processor for the ultrasonic probe performs image reconstruction processing based on the reflected waves based on a signal corresponding to the reflected waves received by the ultrasonic probe 14. The image reconstruction processing generates an ultrasonic image 22 that shows the internal structure of the target site scanned by the ultrasonic probe 14. The ultrasonic image 22 is a so-called B (Brightness) mode image that visualizes the internal structure of the target site from the surface layer to the deep layer where the ultrasonic waves reach as brightness information. The ultrasonic image 22 visualizes the internal structure of the target site that cannot be observed in the surgical field image 21 obtained by optical photography. The ultrasound probe 14 is an example of a "medical device" and a "medical probe" according to the technology of the present disclosure.

[0042] A second puncture line NR2 indicating the puncture path of the puncture needle 18 is superimposed on the ultrasound image 22 as support information for guiding the insertion of the puncture needle 18. The second puncture line NR2 will be described later.

[0043] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and obtains a sector-shaped image with the ultrasonic transducer 14C as the base point. By scanning the ultrasonic probe 14, a plurality of ultrasonic images 22 are taken along the scanning direction. The ultrasonic images 22 are transmitted in real time to the medical support device 11 via an image processor for the ultrasonic probe. In FIG. 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0044] In the ultrasonic probe 14, a guide groove 29 is provided in the tip 14B of the insertion section 14A. The guide groove 29 is a guide groove for guiding the insertion of the puncture needle 18 to a target position inside an organ by engaging with the puncture needle 18. The guide groove 29 includes two guide grooves, a first guide groove 29A and a second guide groove 29B. The first guide groove 29A is a guide groove 29 provided on the proximal side of the ultrasonic transducer 14C in the tip 14B. The first guide groove 29B is a guide groove 29 formed at the most end of the tip 14B and provided on the distal side of the ultrasonic transducer 14C. In the following, when there is no need to distinguish between the first guide groove 29A and the second guide groove 29B, both are collectively referred to as the guide groove 29.

[0045] Fig. 3 shows the state of the puncture needle 18 guided by the first guide groove 29A, and Fig. 4 shows the state of the puncture needle 18 guided by the second guide groove 29B. Fig. 5 shows a schematic diagram of the puncture needle 18 inserted into the body being guided by the first guide groove 29A and puncturing a tumor 27 in the liver LV. Fig. 6 shows a schematic diagram of the puncture needle 18 inserted into the body being guided by the first guide groove 29A and puncturing a tumor 27 in the liver LV.

[0046] The first guide groove 29A is inclined at an angle of θ1 with respect to the direction of the axis AX of the tip portion 14B. The first guide groove 29A is inclined backward so that the tip of the puncture needle 18 inserted from the base end side of the tip portion 14B faces the tip side of the tip portion 14B. Conversely, the second guide groove 29B is inclined forward so that the tip of the puncture needle 18 inserted from the tip side of the tip portion 14B faces the base end side of the tip portion 14B.

[0047] The puncture needle 18 is inserted while checking the tumor 27 with the ultrasound image 22. Both the first guide groove 29A and the second guide groove 29B are inclined toward the ultrasound transducer 14C, so that the tip of the puncture needle 18 can be directed toward the area visualized by the ultrasound image 22.

[0048] The tip 14B is provided with a marker M. The marker M is a marker that can be recognized in an optically captured image, that is, an optically detectable marker. The marker M can be captured by the camera 13B of the endoscope 13. The medical support device 11 estimates the position and posture of the guide groove 29 provided in the tip 14B by using the marker M. As will be described later, specifically, the position and posture of the tip 14B of the insertion section 14A are estimated by the marker M. The posture of the tip 14B is represented, for example, by the direction of the axis AX of the tip 14B. Then, based on the estimated direction of the axis AX of the tip 14B, the position and posture of the guide groove 29, the relative positional relationship of which with the tip 14B is known, are estimated.

[0049] As an example, the marker M is configured by a pattern having morphological characteristics such as a grid pattern or a dot pattern. A method for estimating the position and orientation using the marker M will be described later. The marker M is an example of an "image-recognizable marker" according to the technology of the present disclosure. The insertion section 14A is an example of an "insertion section of a medical device" according to the technology of the present disclosure.

[0050] The medical support device 11 acquires a surgical field image 21 from the endoscope 13, and acquires an ultrasound image 22 from the ultrasound probe 14. As shown in Fig. 1, two displays 16 are provided as an example, and two images, a surgical field image 21 and an ultrasound image 22, are displayed on each display 16. The surgical field image 21 and the ultrasound image 22 are each output as a video and displayed on the display 16 as a live view.

[0051] When the ultrasonic probe 14 is inserted into the operative field SF, the ultrasonic probe 14 appears in the operative field image 21. The medical support device 11 outputs the operative field image 21, in which the insertion portion 14A (including the tip portion 14B and the marker M) of the ultrasonic probe 14 appears, to the display 16. A view of the operative field SF inside the body of the patient PT is provided to the medical staff ST through the screen of the display 16.

[0052] 5 and 6, two types of second puncture lines NR2, which respectively indicate a puncture path along the inclination angle θ1 of the first guide groove 29A and a puncture path along the inclination angle θ2 of the second guide groove 29B, can be superimposed and displayed on the ultrasound image 22. The second puncture line NR2 is support information that guides the puncture of the puncture needle 18 to a target position inside an organ in the ultrasound image 22.

[0053] The second puncture line NR2 is information indicating a path passing through the first guide groove 29A or the second guide groove 29B that guides the puncture needle 18. The second puncture line NR2 is generated as an extension line of the guide groove 29 based on a known positional relationship between the ultrasonic transducer 14C and the guide groove 29. The second puncture line NR2 is used as a guide when the puncture needle 18 is punctured into a target position. For example, when the target position to be punctured by the puncture needle 18 is a tumor 27 in the liver LV, the medical staff ST positions the ultrasonic probe 14 so that the second puncture line NR2 and the tumor 27 overlap in the ultrasonic image 22. In this state, the puncture needle 18 is punctured into the tumor 27 using the second puncture line NR2 as a guide. In this way, the puncture needle 18 can be made to reach the target position by passing the puncture needle 18 through the guide groove 29 in a state in which the ultrasonic probe 14 is positioned so that the second puncture line NR2 passes through the target position.

[0054] However, since the ultrasound image 22 is an internal image of an organ, it is difficult to know at what position on the body surface BS and in what posture the puncture needle 18 should be inserted when inserting it into the body from outside the body by simply superimposing the second puncture line NR2 on the ultrasound image 22. Specifically, as shown in Figs. 5 and 6, it is difficult to know the insertion angle of the puncture needle 18 from the body surface BS toward the organ and the insertion position NP on the body surface BS of the patient PT. Therefore, the medical support device 11 superimposes the first puncture line NR1 of the puncture needle 18 on the operative field image 21. This provides the medical staff ST with support information that serves as a guide for the insertion angle and insertion position NP of the puncture needle 18. The first puncture line NR1 is an example of a "puncture path of the puncture needle" according to the technology of the present disclosure.

[0055] In this way, the medical support device 11 superimposes the second puncture line NR2 on the ultrasound image 22 as support information for supporting the puncture of the puncture needle 18 to the target position, and also superimposes the first puncture line NR1 on the operative field image 21. The first puncture line NR1 indicates the puncture path from the body surface BS to the guide groove 29 of the ultrasound probe 14 in the operative field image 21, and the second puncture line NR2 indicates the puncture path from the guide groove 29 to the target position in an organ, such as a tumor 27 in the liver LV, in the ultrasound image 22.

[0056] 7 shows an example of a hardware configuration of the medical support device 11. The medical support device 11 is an example of a "medical support device" and a "computer" according to the technology of the present disclosure, and includes a processor 41, a reception device 42, a display 16, a RAM (Random access memory) 43, a storage 44, a communication I / F 45, and an external I / F 46. Each of these components is connected to a bus 48 and can communicate with each other.

[0057] The medical support apparatus 11 is operated by an operator such as a medical staff ST through a reception device 42. The reception device 42 has a keyboard, a mouse, etc. (not shown) and receives instructions from the operator. The reception device 42 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0058] Examples of the display 16 include an EL (Electro-Luminescence) display, a liquid crystal display, etc. As described above, there are two displays 16, and each display 16 displays various information in addition to the operative field image 21 and the ultrasound image 22.

[0059] The processor 41 is, for example, a CPU (Central Processing Unit), which comprehensively controls each part of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0060] The storage 44 is a non-volatile storage device that stores various programs and various parameters, etc. Examples of the storage 44 include a hard disk drive (HDD) and a solid state drive (SSD). The storage 44 also stores a medical support program 49 for causing the computer to function as the medical support device 11.

[0061] The RAM 43 is a memory that temporarily stores information, and is used as a work memory by the processor 41. The RAM 43 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0062] The communication I / F 45 is connected to a network (not shown) such as a local area network (LAN) and / or a wide area network (WAN), and performs transmission control according to communication protocols defined by various wired or wireless communication standards.

[0063] The external I / F 46 is, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer and a memory card.

[0064] The processor 41 performs medical support processing by reading out a medical support program 49 from the storage 44 and executing the medical support program 49 on the RAM 43. The medical support processing is realized by the processor 41 operating as an image acquisition unit 41A, a position and orientation information derivation unit 41B, an image synthesis unit 41C, and a display control unit 41D. The medical support program 49 is an example of an "operation program of a medical support device" according to the technology of the present disclosure.

[0065] The dimension information 50 includes the dimension information of the ultrasonic probe 14. Specifically, the dimension information of the ultrasonic probe 14 includes information representing the positional relationship of the marker M at the tip 14B of the ultrasonic probe 14 and the positional relationship of the guide groove 29 at the tip 14B. The positional relationship of the marker M at the tip 14B is, for example, information on the position and attitude of the lattice of the lattice pattern of the marker M with respect to the axial direction and circumferential direction of the tip 14B. The positional relationship of the guide groove 29 at the tip 14B is, for example, the linear distance in the axial direction along the axis AX of the tip 14B between the reference point of the tip 14B and the guide groove 29 (including the first guide groove 29A and the second guide groove 29B), the inclination angle θ (including θ1 and θ2) of the guide groove 29 with respect to the axial direction of the tip 14B, and the like.

[0066] The image acquiring unit 41A executes image acquiring processing to acquire the operative field image 21 and the ultrasound image 22. For example, the image acquiring unit 41A acquires the operative field image 21 and / or the ultrasound image 22 from a device including the processor of the endoscope 13 and / or the processor of the ultrasound probe 14 via the external I / F 46 or the communication I / F 45. Note that the medical support device 11 may include the processor of the endoscope 13 and / or the processor of the ultrasound probe 14.

[0067] The position and orientation information derivation unit 41B executes a process of deriving position and orientation information including the position and orientation of the guide groove 29 of the ultrasonic probe 14 based on the marker M.

[0068] The image synthesis unit 41C specifies a position in the operative field image 21 at which the first puncture line NR1 is to be superimposed based on the derived position and orientation information, and superimposes the first puncture line NR1 at the specified position. Here, the position and orientation information is an example of "position and orientation information including at least one of the position and orientation of the insertion part" according to the technology of the present disclosure. In this example, the position and orientation information includes the position and orientation, but if the position at which the first puncture line NR1 is to be superimposed can be specified, the information may include either the position or the orientation.

[0069] The display controller 41D executes control to display the operative field image 21 and the ultrasound image 22 on the display 16. Also, it executes display control to superimpose the first puncture line NR1 on the operative field image 21. The display control to superimpose the first puncture line NR1 is specifically control to generate a composite image in which the first puncture line NR1 is superimposed on the operative field image 21, and display the generated composite image on the display 16. Hereinafter, such display control is also simply referred to as superimposed display.

[0070] The processing executed by the processor 41 of the medical support device 11 of the present disclosure will be specifically described with reference to Fig. 8 to Fig. 12. Fig. 8 is a flowchart showing the overall processing procedure, and Fig. 9 to Fig. 12 explain each step.

[0071] As shown in FIG. 8, in the medical support process, the processor 41 acquires the operative field image 21 captured by the camera 13B of the endoscope 13 in step S1100. Then, in step S1200, the processor 41 detects the marker M of the ultrasonic probe 14 from the operative field image 21. In step S1300, the processor 41 derives position and orientation information of the guide groove 29 of the ultrasonic probe 14 based on the marker M. In step S1400, the processor 41 derives a first puncture line NR1 of the puncture needle 18 in the operative field image 21 based on the position and orientation information of the guide groove 29. Then, in step S1500, the processor 41 displays the first puncture line NR1 superimposed on the operative field image 21. The processor 41 repeats this process until an instruction to end the display is input (step S1600).

[0072] Each processing step will be described in more detail below. First, Fig. 9 is a diagram conceptually showing the detection of the marker M in step S1200 shown in Fig. 8 and the derivation of the position and orientation information of the guide groove 29 of the ultrasonic probe 14 in step S1300. As a supplementary explanation of Fig. 9, the explanation will be given with reference to Figs. 10 and 11 as necessary.

[0073] As shown in FIG. 9, in step S1200, the processor 41 detects the marker M based on the surgical field image 21, for example, using an image processing method such as pattern matching. As an example, the marker M is a marker of a lattice pattern 56 composed of a first line extending in the direction of the axis AX of the tip portion 14B of the ultrasound probe 14 and a second line perpendicular to the axial direction of the tip portion 14B and formed in the circumferential direction along the outer circumferential surface of the tip portion 14B. The lattice interval of the lattice pattern 56 is constant. Reference numeral 56A indicates a plurality of intersections between the first line and the second line. The processor 41 detects the marker M by searching for a morphological feature of the marker M such as the lattice pattern 56 from the surgical field image 21. Of course, instead of a rule-based method such as pattern matching, the marker M may be detected using an AI (Artificial Intelligence) technique using a machine learning model.

[0074] The processor 41 detects the marker M to identify the area in the surgical field image 21 where the marker M exists. Then, in step S1300, the processor 41 first estimates the position and orientation of the tip portion 14B based on morphological features such as the grid pattern 56 of the marker M, and finally derives position and orientation information representing the position and orientation of the guide groove 29. Specifically, the process is as follows.

[0075] 10 and 11 conceptually show the correspondence between the position and orientation of the tip 14B of the ultrasonic probe 14 having a marker M in the operative field image 21 and the position and orientation of the tip 14B in the operative field SF defined as a three-dimensional space. The Zin axis of the operative field SF, which is a three-dimensional space, is a direction parallel to the imaging optical axis of the camera 13B, and the Xin-Yin plane (hereinafter simply referred to as the XY plane) is a plane parallel to the imaging surface of the camera 13B and perpendicular to the imaging optical axis. In FIG. 10 and FIG. 11, the XY plane is parallel to the screen of the operative field image 21. The operative field image 21 is a projection image in which the operative field SF is projected from one viewpoint. Therefore, the processor 41 can estimate the position and orientation of the tip 14B in the operative field SF based on the marker M detected in the operative field image 21 as shown below.

[0076] 10 shows a state in which the axial direction of the tip 14B of the ultrasonic probe 14 is perpendicular to the imaging optical axis of the camera 13B in the operative field SF in three-dimensional space (more specifically, a state in which the axis AX of the tip 14B is parallel to the Xin axis). When the tip 14B of the ultrasonic probe 14 is in such a position, the perpendicular lines of the grid pattern 56 of the marker M in the operative field image 21 are parallel to the Xin axis and the Yin axis, respectively, and the intervals between adjacent intersections 56A are also the same.

[0077] On the other hand, Fig. 11 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13B in the operative field SF in three-dimensional space, but is inclined in the depth direction parallel to the imaging optical axis. The posture shown in Fig. 11 is a state in which the axis AX of the tip 14B is rotated about -15° around the Yin axis from the posture shown in Fig. 10. When the tip 14B is in such a posture, in the operative field image 21, the lines of the grid pattern 56 of the marker M that extend in the circumferential direction become shorter as they are farther from the camera 13B in the depth direction. Furthermore, the distance between adjacent intersections 56A also becomes shorter as the distance between intersections 56A is farther from the camera 13B.

[0078] In this way, the form of the marker M reflected in the operative field image 21 changes depending on the posture of the tip portion 14B. The processor 41 estimates the posture of the tip portion 14B of the ultrasonic probe 14 in the operative field SF based on the form of the marker M in the operative field image 21. Specifically, the direction of the axis AX of the tip portion 14B in the operative field SF, i.e., the axial direction of the tip portion 14B, is detected as the posture of the tip portion 14B.

[0079] Furthermore, if the position of the tip portion 14B changes within the surgical field SF, the position of the marker M reflected in the surgical field image 21 also changes. The processor 41 estimates the position of the tip portion 14B within the surgical field SF based on the position of the marker M. The position of the tip portion 14B is, for example, the position of a reference point of the tip portion 14B, such as the tip position of the tip portion 14B. Furthermore, the shooting distance from the camera 13B to the marker M within the surgical field SF (i.e., the distance in the Zin axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the surgical field image 21. The processor 41 can derive the position coordinates of the reference point of the tip portion 14B within the surgical field SF based on such shooting distance and dimensional information of the tip portion 14B including the known dimensions of the marker M.

[0080] The processor 41 estimates the position and orientation of the guide groove 29 of the ultrasonic probe 14 within the surgical field SF based on the position and orientation of the tip 14B derived in this manner and the positional relationship of the guide groove 29 at the tip 14B contained in the dimensional information of the ultrasonic probe 14.

[0081] 10, for example, the position of the tip 14B of the ultrasonic probe 14 is information such as X01, Y01, and Z01, which are position coordinates of the reference point of the tip 14B in the surgical field SF. The orientation of the tip 14B is defined as the axial direction of the tip 14B in the surgical field SF, and for example, the axis AX of the tip 14B is information that is parallel to the XY plane and the XZ plane and perpendicular to the YZ plane.

[0082] Moreover, the position of the first guide groove 29A is information such as X11, Y11, Z11, which are position coordinates of a reference point in the surgical field SF. The position of the second guide groove 29B is information such as X21, Y21, Z21, which are position coordinates of a reference point in the surgical field SF. The orientations of the first guide groove 29A and the second guide groove 29B are defined as orientations with respect to the axis AX, and these are known inclination angles θ1 and θ2 as dimensional information.

[0083] 11, for example, the position of the tip 14B of the ultrasonic probe 14 is information such as X02, Y02, and Z02, which are position coordinates of the reference point of the tip 14B in the surgical field SF. The orientation of the tip 14B is defined as the axial direction of the tip 14B in the surgical field SF, and for example, the axis AX of the tip 14B is information such as -15° with respect to the XY plane, parallel to the XZ plane, and 75° with respect to the YZ plane.

[0084] Moreover, the position of the first guide groove 29A is information such as X12, Y12, Z12, which are position coordinates of a reference point in the surgical field SF. The position of the second guide groove 29B is information such as X22, Y22, Z22, which are position coordinates of a reference point in the surgical field SF. The orientations of the first guide groove 29A and the second guide groove 29B are defined as orientations with respect to the axis AX, and these are known inclination angles θ1 and θ2 as dimensional information.

[0085] Then, as shown in FIG. 12, in step S1400, the processor 41 derives a first puncture line NR1 of the puncture needle 18 in the operative field image 21 based on the position and orientation of the guide groove 29. For example, the processor 41 derives the first puncture line NR1, which is a straight line along the inclination angle θ1 of the first guide groove 29A in the operative field SF based on the position coordinates of the first guide groove 29A in the operative field SF defined as a three-dimensional space and the inclination angle θ1 of the tip portion 14B with respect to the axis AX. For example, the first puncture line NR1 is a line segment that is angled by the inclination angle θ1 of the first guide groove 29A from the position of the first guide groove 29A as a base point. Then, the processor 41 converts this first puncture line NR1 into the two-dimensional coordinate system of the operative field image 21 to derive the first puncture line NR1 in the operative field image 21. The same applies to the second guide groove 29B.

[0086] Furthermore, the medical staff ST can select by operation or setting whether to display the first puncture line NR1 for the first guide groove 29A or the first puncture line NR1 for the second guide groove 29B. The first puncture line NR1 is derived according to the selected guide groove 29. The example shown in Fig. 12 is an example in which the first puncture line NR1 for the first guide groove 29A is displayed.

[0087] In step S1500, the processor 41 superimposes the derived first puncture line NR1 on the operative field image 21. In this manner, the processor 41 executes display control for superimposing the first puncture line NR1 of the puncture needle 18 at a position specified in the operative field image 21 based on the position and orientation information representing the position and orientation of the tip portion 14B. As a result, the operative field image 21 with the first puncture line NR1 superimposed thereon is displayed on the display 16. The processor 41 repeatedly executes the above process until a display end instruction is input, such as when the medical support device 11 is started up and stopped.

[0088] Since the operative field image 21 is output to the display 16 as a moving image, the position and posture of the first puncture line NR1 in the operative field image 21 displayed on the display 16 are also updated to reflect the positions of the endoscope 13 and the ultrasound probe 14, etc. in real space photographed by the camera 13B.

[0089] The effect of the technique of the present disclosure will be described by taking as an example a case where the puncture needle 18 punctures the liver LV, which is an organ to be punctured, with a tumor 27 in the liver LV as the target position. The medical staff ST adjusts the position and posture of the ultrasonic transducer 14C provided at the tip 14B of the ultrasonic probe 14 while observing the ultrasonic image 22 on which the second puncture line NR2 is superimposed, as shown in, for example, FIG. 1 and FIG. 5. Specifically, the position and posture of the ultrasonic transducer 14C is adjusted to a position where the second puncture line NR2, which is an extension of the guide groove 29, passes through the tumor 27. The position and posture of the first puncture line NR1 superimposed on the operative field image 21 changes according to the change in the position and posture of the ultrasonic probe 14. When the second puncture line NR2 is displayed so as to pass through the tumor 27 in the ultrasonic image 22, the second puncture line NR2 shows an ideal puncture path of the puncture needle 18 in the liver LV. In this state, the position and posture of the ultrasonic probe 14 are fixed while the puncture is being performed.

[0090] In this state, the first puncture line NR1 in the surgical field image 21 indicates the ideal puncture path of the puncture needle 18 within the body cavity from the body surface BS of the patient PT to the surface of the liver LV to be punctured, as shown in Figure 12 as an example.

[0091] While observing the operative field image 21 on which the ideal first puncture line NR1 is superimposed as shown in FIG. 12, the medical staff ST first adjusts the position and posture of the puncture needle 18 outside the body so that the puncture needle 18 is inserted along the ideal first puncture line NR1. Specifically, the medical staff ST finds an approximate insertion position NP corresponding to the first puncture line NR1, and punctures the body surface BS of the patient PT from the insertion position NP. Then, while watching the display of the first puncture line NR1, the medical staff ST inserts the puncture needle 18 toward the first guide groove 29A of the ultrasound probe 14 inside the body cavity while adjusting the position and posture of the puncture needle 18. Since the first puncture line NR1 is displayed with the first guide groove 29A as a base point, the puncture needle 18 can be easily reached to the first guide groove 29A by inserting the puncture needle 18 along the first puncture line NR1.

[0092] After the puncture needle 18 reaches the first guide groove 29A, the puncture needle 18 can be inserted into the liver LV along the second puncture line NR2 while checking the second puncture line NR2 in the ultrasound image 22 as shown in Figures 1, 2, and 5, thereby inserting the puncture needle 18 into the tumor 27, which is the target position. With the puncture needle 18 inserted into the tumor 27, cauterization of the tumor 27 is performed, for example.

[0093] As described above, the medical support device 11 according to the technique of the present disclosure includes a processor 41, and the processor 41 acquires an operative field image 21 (an example of an in-vivo image) captured by a camera 13B (an example of an in-vivo camera) of an endoscope 13 that captures the inside of a patient PT (an example of a subject), the operative field image 21 including within its capture range a liver LV (an example of an organ) punctured by a puncture needle 18, an insertion section 14A of an ultrasound probe 14 (an example of a medical device) inserted into the body, and an image-recognizable marker M provided on the insertion section 14A. Then, the processor 41 derives position and orientation information including at least one of the position and orientation of the insertion section 14A in the operative field image 21 based on the marker M. Furthermore, the processor 41 executes display control to superimpose and display a first puncture line NR1 of the puncture needle 18 at a position specified in the operative field image 21 based on the position and orientation information.

[0094] In the technology of the present disclosure, the hardware configuration for superimposing and displaying the first puncture line NR1 is merely a marker provided in the insertion section of a medical device such as the ultrasound probe 14. Therefore, a large-scale device such as a magnetic system is not required. In addition, even when a conventional medical device is used, it is simpler than a configuration in which a light source for a marker is built-in because only the exterior of the insertion section needs to be modified. Therefore, the medical support device 11 can display the first puncture line NR1 of the puncture needle 18 at an appropriate position according to the position or posture of a medical device such as the ultrasound probe 14 with a simple configuration.

[0095] In the above embodiment, the medical device inserted into the body of the patient PT is an ultrasonic probe 14 (an example of a medical probe) capable of observing the internal structure of an organ such as the liver LV. The procedure of puncturing an internal organ with a puncture needle 18 is often performed using a medical probe capable of observing the internal structure of the organ. Therefore, the technology of the present disclosure is particularly effective when a medical probe is used as a medical device as in the above embodiment. Furthermore, the ultrasonic probe 14 is relatively often used in combination with the puncture needle 18. Therefore, the technology of the present disclosure is even more effective when the ultrasonic probe 14 is used as a medical probe. Note that the medical probe capable of observing the internal structure of an organ may be other than the ultrasonic probe 14, and may be, for example, an optical coherence tomography (OCT) probe or the like.

[0096] The medical device does not have to be a medical probe capable of observing the internal structure of an organ. For example, the medical device may be a treatment tool that does not have a function of observing the internal structure and has only a guide groove 29 for the puncture needle 18 at the tip. For example, when a tumor is present on the surface of an organ and a treatment is performed to puncture the tumor on the surface with the puncture needle 18, even a treatment tool that does not have a function of observing the internal structure can appropriately guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST aligns the guide groove 29 of the treatment tool to a position corresponding to the tumor on the surface of the organ, and in this state, punctures the tumor with the puncture needle 18 through the guide groove 29.

[0097] In the above embodiment, a medical probe such as the ultrasonic probe 14 having a guide groove 29 is used as the medical device. The processor 41 specifies the position at which the first puncture line NR1 is superimposed on the operative field image 21 based on the relative positional relationship between the marker M and the guide groove 29 (e.g., dimensional information of the ultrasonic probe 14). The puncture needle 18 is often used in combination with a medical probe having a guide groove 29 that guides the puncture needle 18. Therefore, the technology of the present disclosure is particularly effective when a medical probe having a guide groove 29 is used as the medical device.

[0098] Furthermore, the medical instrument may be a simple rod-like treatment instrument without the guide groove 29. If there is a tumor on the surface of an organ, even such a treatment instrument can indicate the tumor. Simply superimposing the first puncture line NR1 on the operative field image 21 in which the treatment instrument has indicated the tumor on the surface of the organ can serve as a guide for checking the puncture direction of the puncture needle 18, etc. Therefore, even if the medical instrument is a treatment instrument without the guide groove 29, the technology of the present disclosure is effective.

[0099] [Second embodiment] Even if the first puncture line NR1 is superimposed on the operative field image 21 as shown in Fig. 12 of the first embodiment, the operative field image 21 lacks depth information of the operative field SF, and therefore it may not be possible to accurately grasp the insertion position NP, which is the intersection of the first puncture line NR1 and the body surface BS shown in Fig. 5 and Fig. 6. The second embodiment shown in Figs. 13 to 19 aims to solve such problems of the first embodiment, and provides the user with information for grasping the insertion position NP corresponding to the first puncture line NR1 superimposed on the operative field image 21.

[0100] As an example, consider performing a pressing operation of pressing the body surface BS of a patient PT with a finger FG from outside the body as shown in Fig. 13. As also shown in Fig. 5 and Fig. 6, in laparoscopic surgery, the body cavity is insufflated, so the body surface BS bulges outward from the body cavity in a convex shape. When such a pressing operation is performed on the body surface BS, a part of the inner wall W of the body cavity inside the body surface BS that corresponds to the pressing position bulges outward from the body cavity, creating a bulge PR.

[0101] As shown in FIG. 14, for example, while the operative field image 21 with the first puncture line NR1 superimposed thereon is displayed on the display 16, the medical staff ST performs a pressing operation on the body surface BS multiple times while checking the pressing position by the finger FG on the display 16. Then, as shown in the operative field image 21 in the lower part of FIG. 14, the medical staff ST can find the pressing position where the first puncture line NR1 and the protuberance PR intersect. However, as described above, the operative field image 21 lacks depth information. Therefore, simply confirming that the first puncture line NR1 and the protuberance PR intersect in the operative field SF in the three-dimensional space cannot determine whether the first puncture line NR1 and the protuberance PR intersect three-dimensionally. Therefore, in the second embodiment, a three-dimensional intersection determination is performed to determine whether the first puncture line NR1 and the protuberance PR intersect in the three-dimensional space. A specific description will be given with reference to FIG. 15 to FIG. 19.

[0102] Fig. 15 is an example of a flowchart showing the overall processing procedure of medical support processing when performing three-dimensional intersection detection. In the flowchart shown in Fig. 15, steps S1100 to S1500 and step S1600 are similar to the flowchart of the first embodiment shown in Fig. 8. The following mainly describes the differences. The differences are steps S1510 to S1530, which are indicated by dashed lines.

[0103] 15, the three-dimensional intersection determination is started when an operational instruction is input by the medical staff ST with the first puncture line NR1 superimposed and displayed in step S1500. The medical staff ST inputs an operational instruction for starting the three-dimensional intersection determination with a pressing operation performed on the body surface BS with the finger FG. The processor 41 waits for the operational instruction to be input in step S1510, and when the operational instruction is input (Y in step S1510), the processor 41 proceeds to step S1520 and executes the three-dimensional intersection determination.

[0104] The three-dimensional intersection determination in step S1520 is a three-dimensional intersection determination between the first puncture line NR1 and the protuberance PR based on the operative field images 21 from two different viewpoints.

[0105] As shown in FIG. 16, first, in step S1521, the processor 41 executes a two-dimensional intersection determination based on the operative field images 21 of two viewpoints.

[0106] First, in step S1521A1, the processor 41 acquires two operative field images 21 of the first viewpoint VP1 and the second viewpoint VP2 in different states in which a pressing operation is performed and the first puncture line NR1 is superimposed and displayed.

[0107] 17 and 18, while pressing the body surface BS with a finger FG, the medical staff ST operates the endoscope 13 to change the viewpoint of the camera 13B. Then, as an example, before and after changing from the first viewpoint VP1 to the second viewpoint VP2, the medical staff ST inputs to the processor 41 whether the current viewpoint position is the viewpoint position before the change or the viewpoint position after the change. This allows the processor 41 to obtain two operative field images 21 of the first viewpoint VP1 before the change and the second viewpoint VP2.

[0108] Instead of detecting the viewpoint change of the camera 13B by manual operation, the processor 41 may detect the viewpoint change caused by the movement of the camera 13B by image analysis. In this case, for example, the processor 41 stores a certain number of past frames acquired as a video, and when detecting a viewpoint change, acquires the frames before and after the viewpoint change as two surgical field images 21 of the first viewpoint VP1 and the second viewpoint VP2, respectively.

[0109] The processor 41 then proceeds to step S1521A2, and determines the intersection in the operative field image 21 of each of the first viewpoint VP1 and the second viewpoint VP2.

[0110] In step S1521A2, the processor 41 first detects the protuberance PR in each surgical field image 21. The detection of the protuberance PR is performed, for example, by an image analysis method using pattern matching or a machine learning model. The inner wall W has undulations even when no pressing operation is performed by the finger FG. However, as described above, in laparoscopic surgery, the body cavity is insufflated, so that the inner wall W as a whole bulges outward in a convex shape. Since the pressing operation is performed by the finger FG in this state, the protuberance PR generated by the pressing operation protrudes in the opposite direction to the bulging direction caused by the insufflation. Therefore, the protuberance PR has characteristics in size and shape that are different from natural undulations, so that the protuberance PR can be detected by an image analysis method.

[0111] Of course, other than the method of detecting the protuberance PR using such image analysis, the protuberance PR may be detected by visual confirmation by the medical staff ST. For example, the display 16 may be configured with a touch panel, and the medical staff ST may visually confirm the position of the protuberance PR in the operative field image 21 displayed on the touch panel. Then, the processor 41 may detect the protuberance PR based on an input operation by the medical staff ST to specify the position of the protuberance PR from the touch panel.

[0112] When the protuberance PR is detected, the processor 41 detects an intersection point XP between the protuberance PR and the first puncture line NR1 in each of the operative field images 21 from the different first viewpoint VP1 and second viewpoint VP2. Since the processor 41 holds the position of the first puncture line NR1 in the operative field image 21 as known information, the processor 41 detects the intersection point XP by comparing the position of the detected protuberance PR in the operative field image 21 with the position of the first puncture line NR1.

[0113] Processor 41 performs such a two-dimensional intersection determination in step S1521, and determines a three-dimensional intersection in steps S1522 and after based on the result of the two-dimensional intersection determination.

[0114] 17, when an intersection point XP is detected in both the operative field images 21 from the first viewpoint VP1 and the second viewpoint VP2, it is considered that the protuberance PR and the first puncture line NR1 intersect in the operative field SF, which is a three-dimensional space inside the body. In this case, the intersection point of the first puncture line NR1 and the inner wall W side is the appropriate insertion position NP.

[0115] In step S1522, the processor 41 determines whether or not the protuberance PR and the first puncture line NR1 intersect in both of the operative field images 21 from different first viewpoints VP1 and second viewpoints VP2. If the determination in step S1522 is affirmative (Y in step S1522), the processor 41 proceeds to step S1523 and determines that the protuberance PR and the first puncture line NR1 intersect three-dimensionally. That is, the processor 41 determines that the protuberance PR and the first puncture line NR1 intersect in the operative field SF, which is a three-dimensional space inside the body.

[0116] 18, the intersection point XP is detected in the operative field image 21 from the first viewpoint VP1, but not in the operative field image 21 from the second viewpoint VP2. In this way, when the intersection point XP is not detected from both the first viewpoint VP1 and the second viewpoint VP2, it is considered that the protuberance PR and the first puncture line NR1 do not intersect in the operative field SF, which is the three-dimensional space inside the body. In this case, the intersection point of the first puncture line NR1 and the inner wall W side is not an appropriate insertion position NP.

[0117] If the determination in step S1522 is negative (N in step S1522), the processor 41 proceeds to step S1524 and determines that the protuberance PR and the first puncture line NR1 do not intersect three-dimensionally. That is, the processor 41 determines that the protuberance PR and the first puncture line NR1 do not intersect in the surgical field SF, which is a three-dimensional space inside the body.

[0118] Here, the intersection point XP is, for example, a point where the protuberance PR and the first puncture line NR1 intersect somewhere within the range of the region from the top to the base of the protuberance PR. The intersection point XP may be, for example, a point where the top part of the protuberance XP intersects with the first puncture line NR1. However, in this example, the intersection with the first puncture line NR1 in the region having a width to the base of the protuberance PR is also detected as the intersection point XP. If the range for detecting the intersection point XP is made wider in this way, the insertion position NP to be determined will also have a width, but in practice, this is not a problem. This is because the body surface BS and the inner wall W are flexible, so even if the insertion position NP of the puncture needle 18 is slightly deviated, it is possible to correct the puncture direction of the puncture needle 18.

[0119] After completing the three-dimensional intersection determination in step S1520, the processor 41 notifies the determination result in step S1530 shown in Fig. 15. As a method of notifying the determination result, for example, the first puncture line NR1 on the operative field image 21 may be blinked, the color may be changed, the line may be thickened, or the brightness may be increased. Also, a message indicating that the lines intersect three-dimensionally may be displayed. Also, a method other than such a visual method may be used, for example, the message indicating that the lines intersect three-dimensionally may be notified by voice, or a warning sound may be issued.

[0120] By checking the judgment result of step S1530, the medical staff ST can grasp the pressure position on the body surface BS where the first puncture line NR1 and the raised portion PR intersect in three dimensions, and as a result, can grasp the appropriate insertion position NP of the puncture needle 18.

[0121] The procedure of the two-dimensional intersection determination in step S1521 may be other than the procedure shown in Fig. 16, and may be, for example, the procedure shown in Fig. 19. In the procedure shown in Fig. 16, while a pressing operation by a finger FG is continued, the processor 41 acquires two operative field images 21 of different first and second viewpoints VP1 and VP2, and then performs two-dimensional intersection determination for each of the acquired two operative field images 21. The operative field image 21 of the first viewpoint VP1 and the operative field image 21 of the second viewpoint VP2 are examples of the first in-vivo image and the second in-vivo image acquired while an operation such as a pressing operation is continued.

[0122] 19, the operation includes two operations, a first operation and a second operation, which are performed at different timings on the same position, and an operative field image 21 from a different viewpoint is acquired at each timing. That is, in step S1521B1, the processor 41 first acquires a first operative field image 21 (an example of a first in-vivo image) from a first viewpoint VP1 in a state in which a first operation, which is a first pressing operation, is performed and the first puncture line NR1 is superimposed and displayed. Then, in step S1521B2, the processor 41 determines a two-dimensional intersection between the protuberance PR in the first operative field image 21 and the first puncture line NR1. This determination is an example of a first determination.

[0123] Next, in step S1521B3, the processor 41 acquires a second operative field image 21 (an example of a second in-vivo image) of the second viewpoint VP2 in a state in which a second operation, which is a second pressing operation, is performed and the first puncture line NR1 is superimposed and displayed. Here, the pressing position where the second pressing operation is performed is the same position as the first pressing position. Then, in step S1521B4, a two-dimensional intersection between the protuberance PR in the second operative field image 21 and the first puncture line NR1 is determined. This determination is an example of a second determination. The processing from step S1523 onwards is the same as the procedure in FIG. 16.

[0124] In this way, the procedure for the two-dimensional intersection determination in step S1521 may be the procedure shown in FIG. 16 or the procedure shown in FIG.

[0125] As described above, in the second embodiment, the processor 41 acquires two operative field images 21, which are an example of an in-vivo image in a state in which a pressing operation, which is an example of an operation that causes a change in the inner wall W inside the body, is performed from outside the body and the first puncture line NR1 indicating the puncture path is superimposed and displayed, taken at least from two different first viewpoints VP1 and second viewpoints VP2. Then, based on the two operative field images 21, the processor 41 determines whether the first puncture line NR1 intersects with the protuberance PR, which is an example of a change position where the inner wall W changes in the three-dimensional space inside the body, and notifies the determination result. Therefore, in the second embodiment, it is possible to provide information on the first puncture line NR1 that takes into account the depth information that is lacking in the first embodiment. As a result, the medical staff ST can grasp an appropriate insertion position NP where the first puncture line NR1 intersects with the inner wall W three-dimensionally.

[0126] Furthermore, in the above example, the processor 41 determines that the protuberance PR and the first puncture line NR1 intersect in three-dimensional space when the protuberance PR, which is an example of a position where a change has occurred in the inner wall W, intersects two-dimensionally with the first puncture line NR1, which is a puncture path displayed in a superimposed manner, in both of the two operative field images 21. In this manner, the three-dimensional intersection is determined by determining the two-dimensional intersection in the two operative field images 21, so that the three-dimensional intersection can be determined by simple processing.

[0127] In the second embodiment, the pressing operation is an example of an "operation that causes a change in the inner wall W inside the body", and the protuberance PR is an example of a "change position" where the inner wall W changes. The operation that causes a change in the inner wall W may be an operation other than the pressing operation, for example, light irradiation from outside the body or dye injection. The pressing operation is an operation that causes a shape change in the inner wall W. The light irradiation is an operation that causes a luminance change in the inner wall W part corresponding to the irradiation position by irradiating light to the body surface BS using, for example, a laser pointer. In this case, the position where the luminance change occurs is the "change position". The dye injection is an operation that causes a color change in the inner wall W by injecting a dye into the body surface BS. In this case, the position where the color change occurs is the "change position". For example, the processor 41 can detect the change position where the luminance change or color change occurs based on the pixel value (including luminance information and color information) of the surgical field image 21. In this way, the operation for causing a change in the inner wall W inside the body may be an operation that is performed from outside the body and causes some kind of change in the inner wall W.

[0128] In addition, by performing a pressing operation as the operation for causing a change in the inner wall W, it is possible to cause a change in the inner wall W by a simple operation without using a tool. In addition, since the protuberance PR is a change in shape, it may be easier to recognize the image compared to a change in brightness or color.

[0129] Furthermore, although an example has been described in which three-dimensional intersection determination is performed using operative field images 21 from two viewpoints, operative field images 21 from three or more viewpoints may be used.

[0130] (Modification 1 of the second embodiment: Operation history) 20, when multiple operations that cause changes in the inner wall W, such as pressing operations with a finger FG, are performed, the processor 41 may superimpose and display an operation history, which is a history of the positions of changes in each of the multiple operations, on the operative field image 21. In FIG. 20, the operation history of pressing operations is shown as an example of the history of the positions of changes.

[0131] When the processor 41 detects a protuberance PR, it temporarily records the detection position of the detected protuberance PR, and displays protuberances PR (exemplified as PR1, PR2, and PR3 in FIG. 20) generated by multiple operations performed in the past superimposed on the respective detection positions on the operative field image 21. This allows the medical staff ST to confirm the relative positional relationship between the past pressing positions pressed by the medical staff ST and the first puncture line NR1. This prevents the medical staff ST from repeatedly pressing the same pressing position, improving the efficiency of searching for the insertion position NP.

[0132] (Modification 2 of the second embodiment: Following changes in camera viewpoint) As an example, as shown in Fig. 21, the operation history shown in Fig. 20 may be made to follow a change in the viewpoint of the camera 13B. That is, when the viewpoint of the in-vivo camera moves during a plurality of operations such as a plurality of pressing operations, the processor 41 displays the history of the protuberance PR, which is an example of the history of the changed position, reflecting the amount of movement of the viewpoint.

[0133] 21, when displaying protuberance PR1 and protuberance PR2 as an operation history, consider a case where the viewpoint of camera 13B changes over time from a first viewpoint VP1 to a second viewpoint VP2. In this case, the display positions of protuberance PR1 and protuberance PR2 superimposed on the operative field image 21 from the first viewpoint VP1 acquired at time T1 and the display positions of protuberance PR1 and protuberance PR2 superimposed on the operative field image 21 from the second viewpoint VP2 acquired at time T2 will naturally change. By detecting the amount of movement of the viewpoint of camera 13B, processor 41 corrects the display positions of protuberance PR1 and protuberance PR2, which are past operation history, in accordance with the amount of movement.

[0134] As an example, as shown in FIG. 22, the processor 41 detects the amount of movement of the viewpoint of the camera 13B based on the marker M. For example, as shown in FIG. 5 and FIG. 6, when the puncture needle 18 punctures the tumor 27, the ultrasonic probe 14 is fixed at a position where the tumor 27 and the second puncture line NR2 overlap in the ultrasonic image 22, as an example. When the viewpoint of the camera 13B is moved from the first viewpoint VP1 to the second viewpoint VP2 while the position of the ultrasonic probe 14 is fixed in this way, the appearance of the marker M provided on the tip 14B changes in each of the surgical field image 21 of the first viewpoint VP1 and the surgical field image 21 of the second viewpoint VP2. The processor 41 derives the position and orientation information of the tip 14B of the ultrasonic probe 14 in the manner described with reference to FIG. 10 and FIG. 11. Here, since the position of the tip 14B is fixed, the amount of change between the position and orientation information derived from the operative field image 21 of the first viewpoint VP1 and the position and orientation information derived from the operative field image 21 of the second viewpoint VP2 corresponds to the amount of movement of the viewpoint of the camera 13B. Therefore, the processor 41 can detect the amount of movement of the viewpoint of the camera 13B based on the position and orientation information of the marker M captured in the two operative field images 21 of the different first viewpoint VP1 and second viewpoint VP2. Then, the processor 41 corrects the display position of the operation history according to the detected amount of movement. The operation history is an example of a history of the changed position of the inner wall W, as shown as the display position of the protuberance PR.

[0135] In this way, by displaying the history of the changed position of the inner wall W while reflecting the amount of movement of the viewpoint of the camera 13B, the history of the changed position of the inner wall W can be displayed in an appropriate position even if the viewpoint of the camera 13B changes.

[0136] (Modification 3 of the second embodiment) In the second embodiment, the above example was described as an example in which accurate information about the insertion position NP corresponding to the intersection XP between the changing position of the inner wall W, such as the protuberance PR, and the first puncture line NR1 is presented by image analysis, but there are also cases in which it is possible to present the information without using image analysis.

[0137] As an example, as shown in FIG. 23, consider a case where the axis AX of the tip 14B of the ultrasonic probe 14 and the imaging optical axis AXO of the camera 13B of the endoscope 13 are substantially perpendicular to each other. In this case, even if the pressing position is changed along the axis AX, the position in the depth direction does not change. Therefore, if the pressing position is changed along the axis AX to find an intersection point XP where the protuberance PR and the first puncture line NR1 intersect two-dimensionally in the operative field image 21, the intersection point XP can be determined to be a point where the protuberance PR and the first puncture line NR1 intersect three-dimensionally as well. In this way, in a limited case where the axis AX of the tip 14B of the ultrasonic probe 14 and the imaging optical axis AXO of the camera 13B of the endoscope 13 are substantially perpendicular to each other, it is possible to determine the three-dimensional intersection without image analysis.

[0138] [Third embodiment] 24 to 26, the processor 41 derives an intersection point in three-dimensional space between the inner wall W of the body and the first puncture line NR1 based on an internal depth map DMP representing a distribution of depth, which is a distance in the depth direction from the viewpoint of the camera 13B. The depth map DMP is an example of “depth information” according to the technology of the present disclosure.

[0139] Fig. 24 is a flowchart showing an example of the overall procedure of the medical support process of the third embodiment. The flowchart shown in Fig. 24 is different from the flowchart shown in Fig. 15 of the second embodiment in that steps S1560 to S1580 are added instead of steps S1510 to S1530. That is, in Fig. 24 showing an example of the third embodiment, instead of performing a three-dimensional intersection determination between the first puncture line NR1 and a changed position of the inner wall W (e.g., the raised portion PR) performed in the second embodiment, a three-dimensional intersection point XP between the first puncture line NR1 and the inner wall W is displayed based on the depth map DMP. The third embodiment will be described below, focusing on the differences from the second embodiment.

[0140] In step S1560, the processor 41 receives an instruction to display a three-dimensional intersection point between the first puncture line NR1 and the inner wall W. When an operation instruction to display the intersection point is input (Y in step S1560), the processor 41 derives a three-dimensional intersection point XP between the first puncture line NR1 and the inner wall W based on the depth map DMP. Then, in step S1560, the three-dimensional intersection point XP is displayed superimposed on the operative field image 21.

[0141] FIG. 25 shows an example of a method for acquiring a depth map DMP. In the example shown in FIG. 25, a stereo camera is used as the camera 13B. The camera 13B as a stereo camera shown in FIG. 25 has a plurality of cameras 13B1 and 13B2. The cameras 13B1 and 13B2 are arranged at an interval, and this interval is a base line length L. The focal lengths of the cameras 13B1 and 13B2 are the same. The processor 41 acquires an operative field image 21 captured by a first viewpoint VP1 of the camera 13B1 and an operative field image 21 captured by a second viewpoint VP2 of the camera 13B2. Then, the depth map DMP is generated based on the operative field images 21 of the different first viewpoint VP1 and second viewpoint VP2. The processor 41 generates the depth map DMP based on the focal length of the camera 13B, the base line length L, and the parallax of the same subject captured in both of the operative field images 21 of the two different viewpoints by the principle of triangulation. The depth map DMP indicates the distance in the depth direction of each position in the surgical field SF. The coordinates of the first puncture line NR1 in the surgical field SF can be acquired by a method using the marker M described with reference to Figs. 10 and 11. Therefore, as shown in Fig. 26, the processor 41 can derive the intersection XP between the inner wall W and the first puncture line NR1 based on the depth map DMP.

[0142] In this way, in the third embodiment, the processor 41 derives the intersection point XP in three-dimensional space between the inner wall W and the first puncture line NR1, which is an example of a puncture path, based on the depth map DMP, which is an example of depth information. Therefore, it is possible to derive the three-dimensional intersection point XP between the first puncture line NR1 and the inner wall W without performing an operation that causes a change in the inner wall W, such as a pressing operation, as in the second embodiment.

[0143] The stereo camera in the example shown in FIG. 25 is an example of a "camera capable of estimating depth" for the surgical field SF. In addition to the stereo camera, a camera having a distance sensor may be used as the camera 13B as a camera capable of estimating depth. The distance sensor is, for example, a sensor that uses infrared rays. A distance sensor that uses infrared rays irradiates infrared rays toward a subject and receives the infrared rays reflected by the subject. The intensity of the infrared rays reflected by the subject changes depending on the distance to the subject. The distance sensor measures the distance by detecting a voltage corresponding to the intensity. The processor 41 can generate a depth map DMP based on information obtained by scanning the entire surgical field SF with such a distance sensor.

[0144] (Variation 1 of Depth Map Acquisition Method) The depth map DMP can be obtained by a method other than the method using a camera for depth estimation such as a stereo camera. For example, as shown in FIG. 27, the processor 41 may generate the depth map DMP based on the surgical field image 21 of two different viewpoints, the first viewpoint VP1 and the second viewpoint VP2, acquired by one camera 13B. That is, the method shown in FIG. 27 is a method of acquiring the surgical field image 21 of two different viewpoints by moving one camera 13B instead of using a stereo camera. In this case, the viewpoint is changed by moving the camera 13B. The base length L corresponds to the movement amount of the camera 13B. The movement amount of the camera 13B can be detected based on the marker M in a state in which the tip portion 14B is fixed at one position, as described using FIG. 21.

[0145] In order to detect the amount of movement of the viewpoint of the camera 13B and the parallax of the subject appearing in both of the two operative field images 21 from different viewpoints, another marker may be used in addition to or instead of the marker M of the ultrasonic probe 14. The other marker is fixed, for example, at one position in the operative field SF, and in this state, the two operative field images 21 from different viewpoints are acquired. Since the ultrasonic probe 14 may move, if a marker that is less likely to move than the ultrasonic probe 14 is used, it becomes possible to accurately detect the amount of movement of the viewpoint and the parallax, and it is considered that the accuracy of the depth finally acquired will also be improved.

[0146] The method of generating the depth map DMP from the surgical field images 21 of two viewpoints is similar to the method of generating the depth map DMP using the stereo camera shown in Fig. 25. According to this example, the depth map DMP can be generated without using a stereo camera.

[0147] (Variation 2 of Depth Map Acquisition Method) In addition, the method of obtaining the depth map DMP is not limited to generating it based on the principle of triangulation based on surgical field images 21 from two viewpoints, but it is also possible to obtain it by generating it from one surgical field image 21, as shown in Figure 28 as an example.

[0148] In the example shown in FIG. 28, the processor 41 generates a depth map DMP using a machine learning model LM. The machine learning model LM uses, for example, the operative field image 21 as input data and outputs the depth map DMP. The machine learning model LM learns using teacher data composed of a combination of the operative field image 21 and the depth map DMP serving as correct answer data. For example, a convolutional neural network is used as the machine learning model LM. The convolutional neural network has an encoder that extracts a feature amount of an image, and can output output data in the form of an image according to the feature amount extracted by the encoder. For example, a depth map DMP acquired using a stereo camera is used as the correct answer data of the teacher data.

[0149] (Modification of the third embodiment) Furthermore, the three-dimensional intersection determination between the first puncture line NR1 and the protuberance PR described in the second embodiment with reference to Fig. 15 may be performed using the depth map DMP. The flowchart shown in Fig. 29 is similar to the flowchart shown in Fig. 15 except that step S1520C is added instead of step S1520 shown in Fig. 15. The following description will focus on the differences.

[0150] As described above, the protuberance PR is generated on the inner wall W by a pressing operation. The pressing operation is an example of an operation that causes a change in the inner wall W, and the protuberance PR is an example of a changed position on the inner wall W. The pressing operation causes a shape change on the inner wall W. The processor 41 can grasp depth information of the surgical field SF by the depth map DMP. As described using Figures 10 and 11, the three-dimensional coordinates of the first puncture line NR1 in the surgical field SF can be derived based on the marker M. In step S1520C, the processor 41 performs a three-dimensional intersection determination of the first puncture line NR1 and the protuberance PR based on the depth map DMP.

[0151] As shown in Fig. 30, in step S1520C, the processor 41 first acquires a depth map DMP by the method shown in Fig. 26 etc. Then, the processor 41 performs a three-dimensional intersection determination between the first puncture line NR1 and the protuberance PR by comparing the three-dimensional coordinates of the first puncture line NR1 and the protuberance PR in the depth map DMP. In this way, by using the depth map DMP, it is possible to perform a three-dimensional intersection determination of the first puncture line NR1 and the position of a shape change of the inner wall W.

[0152] In the example of FIG. 30, the pressing operation that changes the shape of the inner wall W has been described as an example of the operation that changes the inner wall W, but the operation that changes the inner wall W does not have to be an operation that changes the shape. For example, as described in the second embodiment, the operation may be an operation such as light irradiation or dye injection that changes the luminance or color of the inner wall W. In this case, for example, the processor 41 detects a change position where a luminance change or color change occurs from the operative field image 21. This allows the processor 41 to derive two-dimensional coordinates of the change position in the operative field image 21. Then, the processor 41 derives three-dimensional coordinates of the change position by extracting depth information corresponding to the two-dimensional coordinates of the change position from the depth map DMP. The processor 41 can perform a three-dimensional intersection determination between the first puncture line NR1 and the luminance or color change position by comparing the three-dimensional coordinates of the first puncture line NR1 with the derived three-dimensional coordinates of the change position.

[0153] [Modification of the method for displaying the first puncture line] 31 and 32, the processor 41 may change the display of a portion of the first puncture line NR1. That is, instead of making the display manner uniform over the entire area from one end of the first puncture line NR1 to the other end, the processor 41 may change the display manner of only a portion of the first puncture line NR1.

[0154] As an example, as shown in FIG. 31, the processor 41 may highlight a part of the first puncture line NR1 on the inner wall W side. In FIG. 31, the range RG1 is the range that is highlighted. The first puncture line NR1 is often required to be displayed immediately after the body surface BS is punctured. This is because, when the tip of the puncture needle 18 approaches the guide groove 29 of the ultrasonic probe 14, the puncture needle 18 can be inserted toward the guide groove 29 of the ultrasonic probe 14, and therefore, at that time, the need to guide the insertion of the puncture needle 18 is relatively low. On the other hand, immediately after the puncture needle 18 is punctured into the body surface BS, the distance to the guide groove 29 of the ultrasonic probe 14 is far. Therefore, it is difficult to grasp the relative positional relationship between the puncture needle 18 and the guide groove 29, and the need to guide the insertion of the puncture needle 18 by the first puncture line NR1 is relatively high. By highlighting a part of the first puncture line NR1 on the inner wall W side, the first puncture line NR1 can be made to stand out in the range RG1 where it is highly necessary.

[0155] The range RG1 is, for example, a range of about 5 cm from the inner wall W side. When observing the inside of an organ with the ultrasound probe 14, the medical staff ST knows the approximate distance from the inner wall W to the organ in the body cavity into which the ultrasound probe 14 is inserted. The approximate distance is, for example, about 10 cm. In this case, if the distance of the range RG1 is about 5 cm, the medical staff ST can know that when the tip of the puncture needle 18 reaches the end of the range RG1 on the organ side, the tip of the puncture needle 18 is located about 5 cm away from the body surface BS toward the organ.

[0156] As an example, as shown in FIG. 32, the visibility of a part of the first puncture line NR1 on the organ side is reduced more than that on the inner wall W side. In the example shown in FIG. 32, the first puncture line NR1 is hidden in a range RG2 of a preset distance from the position of the guide groove 29, which is one end of the first puncture line NR1 on the organ side. When the tip of the puncture needle 18 is approaching the guide groove 29, the medical staff ST inserts the needle tip into the guide groove 29 while checking both the needle tip and the guide groove 29. In this case, the first puncture line NR1 may get in the way when grasping the positional relationship between the needle tip and the guide groove 29. Therefore, by displaying the first puncture line NR1 in a part of the range RG2 on the organ side as shown in FIG. 32, it may be easier to grasp the positional relationship between the needle tip of the puncture needle 18 and the guide groove 29.

[0157] 32, the first puncture line NR1 is hidden in the range RG2, but it does not have to be hidden. For example, the visibility may be reduced by lowering the brightness compared to other areas.

[0158] Furthermore, when displaying the first puncture line NR1, the visibility of the range RG2 does not have to be constantly reduced, and for example, the visibility may be reduced when the tip of the puncture needle 18 approaches an organ after the puncture needle 18 has punctured the body surface BS. The timing to reduce the visibility may be manually specified, or the processor 41 may perform image analysis of the operative field image 21 to detect that the puncture needle 18 has approached the organ or the tip 14B of the ultrasound probe 14.

[0159] 33, scales SC1 may be displayed in a partial range RG2 on the organ side of the first puncture line NR1. The range RG2 in which the scales SC1 are displayed is, for example, a range of about 3 cm from the guide groove 29, and the scales SC1 are displayed in 1 cm increments in the range RG2. By displaying the scales SC1, it is possible to clearly indicate the distance between the tip of the puncture needle 18 and the guide groove 29 when the puncture needle 18 approaches the guide groove 29. The processor 41 can grasp the three-dimensional coordinates of the guide groove 29 in the surgical field SF by the method described using FIGS. 10 and 11, and therefore can derive the scales SC1 based on the three-dimensional coordinates of the guide groove 29.

[0160] As shown in FIG. 34, the processor 41 may detect the insertion length of the insertion section 14A of the ultrasonic probe 14, which is an example of a medical device, and determine the range for changing the display of the range RG1, etc., based on the detected insertion length. The insertion section 14A is provided with, for example, a scale SC2 to serve as a guide for the insertion length of the insertion section 14A. The insertion section 14A is provided with a curved section at the tip section 14B. The scale SC2 is provided at a preset interval (for example, 1 cm interval) from the curved section toward the base end side in the insertion section 14A. The processor 41, for example, performs image recognition of the scale SC2 from the surgical field image 21 to detect the insertion length. In a medical probe such as the ultrasonic probe 14, the tip section 14B is disposed on the surface of an organ. Therefore, the insertion length of the insertion section 14A is an effective clue for grasping the distance from the inner wall W to the surface of the organ. This allows the processor 41 to appropriately derive the range for displaying the range RG1, and to set the range for changing the display of the range RG1, etc., to an appropriate range according to the insertion length.

[0161] Furthermore, the processor 41 may determine the range in which the display of the first puncture line NR1 is changed based on a specification by a user such as the medical staff ST. This allows the range in which the display of the first puncture line NR1 is changed to be an appropriate range according to the user's desire.

[0162] Moreover, as shown in Fig. 35, it is preferable that the processor 41 is capable of displaying a plurality of different first puncture lines NR1. This makes it possible to display various first puncture lines NR1 according to purposes. In the example shown in Fig. 35, a plurality of first puncture lines NR1 corresponding to the two types of first guide grooves 29A and second guide grooves 29B of the ultrasound probe 14 are displayed.

[0163] The processor 41 may display a plurality of first puncture lines NR1 as candidates, have the user, ie, the medical staff ST, select one of them, and hide the remaining candidates. Alternatively, a plurality of first puncture lines NR1 may be displayed, and when the actual puncture needle 18 is inserted, only the first puncture line NR1 that is closer to the position of the puncture needle 18 may be displayed, and the remaining ones may be hidden. The processor 41 may derive the distance between the puncture needle 18 and each of the plurality of first puncture lines NR1 by performing image analysis on the operative field image 21.

[0164] [Fourth embodiment] As an example, in a fourth embodiment shown in Figures 36 and 37, the processor 41 identifies a position at which to superimpose the fourth puncture line NR4 on the operative field image 21 based on a tumor 27 identified in the ultrasound image 22 and a correlation between the coordinate system of the ultrasound image 22 and the coordinate system of the operative field image 21. The tumor 27 is an example of a target position identified in an internal image of an organ acquired by a medical probe. The correlation is an example of a correlation between the coordinate system of the internal image and the coordinate system of the internal body image derived based on the marker M. The fourth puncture line NR4 is an example of a "puncture path" according to the technology of the present disclosure.

[0165] As described with reference to Figs. 10 and 11, the marker M can be used to estimate the position and orientation of the tip 14B of the ultrasonic probe 14. The positional relationship of the ultrasonic transducer 14C in the tip 14B is known from dimensional information. Therefore, if the position and orientation information of the tip 14B can be estimated, it is also possible to estimate the position and orientation of the ultrasonic transducer 14C. Then, the processor 41 can derive the correlation between the coordinate system of the operative field image 21 in which the tip 14B of the ultrasonic probe 14 is captured and the coordinate system of the ultrasonic image 22 acquired by the ultrasonic probe 14.

[0166] In the fourth embodiment, first, as shown in Fig. 35, a case will be considered in which an ultrasound image 22 acquired by an ultrasound probe 14 includes a tumor 27 that is a target position. The second puncture line NR2 is a line that is set with a guide groove 29 of the ultrasound probe 14 as a base point. In the fourth embodiment, the processor 41 derives a third puncture line NR3 that passes through the tumor 27 detected from the ultrasound image 22, in addition to the second puncture line NR2. The third puncture line NR3 is derived with the tumor 27 as a reference, regardless of the guide groove 29.

[0167] The processor 41 converts the coordinate system of the third puncture line NR3 into the coordinate system of the operative field image 21 based on the correlation between the coordinate system of the ultrasound image 22 and the coordinate system of the operative field image 21, and derives a fourth puncture line NR4 which is an extension of the third puncture line NR3. Then, as shown in Fig. 37, the processor 41 displays the derived fourth puncture line NR4 superimposed on the operative field image 21. The fourth puncture line NR4 is an example of a "puncture path" according to the technology of the present disclosure.

[0168] While viewing the operative field image 21, the medical staff ST moves the ultrasound probe 14 to align with the position of the fourth puncture line NR4, that is, so that the first puncture line NR1 extending from the guide groove 29 of the ultrasound probe 14 as a base point coincides with the fourth puncture line NR4. Since the fourth puncture line NR4 is a line that passes through the tumor 27, which is the target position for puncturing the puncture needle 18, aligning the first puncture line NR1 with the fourth puncture line NR4 allows the guide groove 29 of the ultrasound probe 14 to be aligned to an appropriate position. In this state, the medical staff ST can insert the puncture needle 18 along the fourth puncture line NR4 (which is also the first puncture line NR1) to puncture the tumor 27, which is the target position.

[0169] According to the fourth embodiment, an internal image acquired by a medical probe such as the ultrasound probe 14 can be effectively used for positioning the ultrasound probe 14 according to the target position to be punctured by the puncture needle 18.

[0170] [Fifth embodiment] 38 to 41 in a fifth embodiment, a target puncture line NRT prepared before surgery may be superimposed on the operative field image 21. The target puncture line NRT is an example of a "puncture route" according to the technology of the present disclosure.

[0171] (Preparation of preoperative preparation information) In the fifth embodiment, for example, before surgery, a preoperative simulation is performed using 3D (three-dimensional) data of a virtual endoscope, and a target puncture line NRT for the puncture needle 18 is prepared in the preoperative simulation. The target puncture line NRT is an ideal puncture path selected, for example, to avoid sites within an organ that must not be damaged, such as blood vessels, and to pass through sites through which the puncture needle 18 can easily pass, when puncturing with the puncture needle 18. The fifth embodiment is an example in which an ultrasound probe 14 capable of observing the internal structure of an organ is used as a medical device. Hereinafter, a description will be given with reference to Figs. 38 to 41.

[0172] First, a method for creating a target puncture line NRT of the puncture needle 18 to puncture the liver LV in a preoperative simulation using 3D data of a virtual endoscope will be described with reference to Figs. 38 and 39. The 3D data of the virtual endoscope is generated based on a group of tomographic images 132 captured in advance by a tomographic imaging device 131 such as a CT (Computed Tomography) device and an MRI (Magnetic Resonance Imaging) device. When the tomographic imaging device 131 is a CT device, the CT value is acquired while rotating the radiation source and the radiation detector around the body axis of the patient PT. The CT value is acquired at each position in the body axis direction by scanning the radiation source and the radiation detector in the body axis direction of the patient PT. The CT value is a radiation absorption value in the body of the patient PT. The tomographic imaging device 131 generates a tomographic image 132A by performing an image reconstruction process based on the CT values ​​acquired in each direction around the body axis. Each tomographic image 132A is a two-dimensional image generated according to a slice thickness in the body axis direction, and the tomographic image group 132 is a collection of a plurality of tomographic images 132A corresponding to each position in the body axis direction. The tomographic image group 132 is output to an image database 133 such as a PACS (Picture Archiving and Communication System).

[0173] An information processing device (not shown) performs 3D modeling to numerically describe the three-dimensional shape of the body of the patient PT based on the tomographic image group 132 obtained by the tomography device 131, thereby generating a three-dimensional image 134, which is a set of voxel data 134A. The voxel data 134A is a unit of pixel in a three-dimensional space, and has three-dimensional coordinate information and pixel values. The three-dimensional image 134 generated by 3D modeling is also called three-dimensional volume data. In the tomographic image group 132, the pixel interval of each tomographic image 132A and the slice thickness of each tomographic image 132A may differ. In this case, as an example, in the 3D modeling, an interpolation process is performed between adjacent tomographic images 132A to generate a three-dimensional image 134 having isotropic voxel data 134A having equal lengths in each three-dimensional direction. Here, the three-dimensional image 134 generated based on the tomographic image group 132 is information created before surgery, so here, for convenience, it is called a preoperative 3D image 134. The preoperative 3D image 134 is an example of 3D data of a virtual endoscopy, and further, is an example of a "three-dimensional image of an organ acquired in advance before surgery" according to the technique of the present disclosure.

[0174] The preoperative 3D image 134 is an image capable of reproducing the external shape of the patient PT's body, anatomical parts such as internal organs, and their internal structures. The preoperative 3D image 134 shown in Figures 38 and 39 shows data of the liver LV as an example of an anatomical part. The preoperative 3D image 134 also includes data capable of reproducing the vascular structure, which is an example of the internal structure of the liver LV.

[0175] In addition, the preoperative 3D image 134 in this example is a color image, and each pixel value of red (R), green (G), and blue (B) is given as the pixel value of the voxel data 134A. The preoperative 3D image 134 may be a monochrome image. For example, the pixel value of the voxel data 134A may be expressed only by brightness (Y) based on the CT value. Furthermore, the pixel value of the voxel data 134A may be a value obtained by converting the CT value using a preset LUT (Look Up Table) or arithmetic expression. Furthermore, the pixel value of the voxel data 134A may be set to a color associated with each specific part such as an organ or a lesion specified in the preoperative 3D image 134. Furthermore, opacity is also set in the voxel data 134A. The opacity is data used for volume rendering. Rendering is a process of converting a part of the preoperative 3D image 134 into a two-dimensional projected image, and volume rendering is a rendering method that projects internal information of an object included in the preoperative 3D image 134 onto the projected image. By setting the opacity of each voxel data 134A, when performing volume rendering, it becomes possible to selectively express the internal information in the projection image in an opaque manner, a semi-transparent manner, or a transparent manner.

[0176] As shown in Fig. 39, the information 123 including the target puncture line NRT is information that is prepared before surgery, and is therefore referred to as preoperative preparation information below. The preoperative preparation information 123 is generated based on a preoperative 3D image 134. The target puncture line NRT is added to the preoperative 3D image 134 in a surgery simulation performed before surgery using the preoperative 3D image 134. The preoperative preparation information 123 includes a blood vessel structure 137 and a tumor 27 as internal structures of the liver LV to be punctured, in addition to the target puncture line NRT. The preoperative preparation information 123 including the target puncture line NRT and internal structures such as the blood vessel structure 137 and the tumor 27 is extracted from the preoperative 3D image 134.

[0177] In this way, the preoperative preparation information 123 is information that specifies a three-dimensional position in the preoperative 3D image 134. The blood vessel structure 137 included in the preoperative preparation information 123 is used for alignment with the ultrasound image 22 acquired by the ultrasound probe 14.

[0178] (Acquisition of ultrasound 3D images) 40, the processor 41 acquires an ultrasound 3D (three-dimensional) image 151 based on an ultrasound image group 22G, which is a collection of ultrasound images 22 captured by the ultrasound probe 14. The processor 41 uses the ultrasound 3D image 151 acquired during surgery for alignment. The ultrasound 3D image 151 is an example of an "internal image of an organ acquired by a medical probe" according to the technology of the present disclosure.

[0179] As shown in Fig. 40, first, the ultrasonic probe 14 is scanned along the surface of a target site such as the liver LV, and a plurality of ultrasonic images 22 can be acquired at each position on the scanning trajectory S. The processor 41 acquires an ultrasonic image group 22G, and performs 3D modeling based on the acquired ultrasonic image group 22G to generate an ultrasonic 3D image 151. The ultrasonic image 22 is a cross-sectional image similar to the cross-sectional image 132A shown in Fig. 38, and the ultrasonic image group 22G is similar to the cross-sectional image group 132. The ultrasonic image group 22G also depicts the internal structure of the target site, such as the vascular structure 137 of the liver LV.

[0180] The processor 41 generates an ultrasound 3D image 151 by executing a process similar to the 3D modeling described in Fig. 38. The ultrasound 3D image 151 is a collection of voxel data 151A, which are pixels in a three-dimensional space, similar to the preoperative 3D image 134. The voxel data 151A is similar to the voxel data 134A shown in Fig. 38, and three-dimensional coordinates, pixel values, and opacity are set.

[0181] As shown in FIG. 41, the processor 41 derives first positional relationship information 158 which is the correlation between the coordinate system of the operative field image 21 acquired through the endoscope 13 and the coordinate system of the ultrasound 3D image 151, and second positional relationship information 159 which is the correlation between the coordinate system of the ultrasound 3D image 151 and the coordinate system of the preoperative 3D image 134.

[0182] First, a method of deriving the first positional relationship information 158 will be described. The processor 41 estimates the position and orientation of the ultrasonic transducer 14C based on the marker M of the ultrasonic probe 14 shown in the operative field image 21 in the same manner as in the fourth embodiment. Then, the processor 41 estimates the position and orientation of the ultrasonic 3D image 151 in the operative field image 21 based on the estimated position and orientation of the ultrasonic transducer 14C. The position and orientation of the ultrasonic 3D image 151 in the operative field image 21 becomes the first positional relationship information 158. The first positional relationship information 158 is an example of "the correlation between the coordinate system of the internal image and the coordinate system of the in-vivo image derived based on the position and orientation information" according to the technology of the present disclosure.

[0183] Next, a method of deriving the second positional relationship information 159 will be described. The processor 41 compares the blood vessel structure 137 depicted in the ultrasound 3D image 151 and the preoperative 3D image 134. Specifically, a similar structure of the blood vessel structure 137 of both images is searched for, and the ultrasound 3D image 151 and the preoperative 3D image 134 are aligned so that the similar structures of both images match. The search for a similar structure of the blood vessel structure 137 may use a rule-based image analysis method such as pattern matching, or may use a method using an AI technology such as semantic segmentation. The processor 41 can derive the second positional relationship information 159 by performing alignment using such a blood vessel structure 137.

[0184] Then, the processor 41 synthesizes the preoperative preparation information 123 including the target puncture line NRT, which is generated based on the preoperative 3D image 134, with the operative field image 21 based on the first positional relationship information 158 and the second positional relationship information 159. Since the alignment is performed by the vascular structure 137 of the liver LV, the target puncture line NRT is superimposed at an appropriate position on the tumor 27. The first puncture line NR1 based on the guide groove 29 is displayed on the operative field image 21. As shown in FIG. 37 in the fourth embodiment, the medical staff ST moves the ultrasonic probe 14 so that the first puncture line NR1 and the target puncture line NRT coincide with each other. This allows the ultrasonic probe 14 to be aligned at an appropriate position for puncturing the tumor 27.

[0185] In this way, when the target puncture line NRT (an example of a puncture path) is set in the preoperative 3D image 134 (an example of a three-dimensional image of an organ) acquired in advance before surgery by a preoperative simulation, the processor 41 uses the second positional relationship information 159 and the first positional relationship information 158 to superimpose the target puncture line NRT on the operative field image 21. This makes it possible to effectively use the internal image of the medical probe and the results of the preoperative simulation. In the preoperative simulation, for example, it is possible to set an appropriate target puncture line NRT that does not damage the blood vessel structure 137. By providing such a target puncture line NRT in the operative field image 21, it becomes possible to provide more appropriate medical support.

[0186] The shape of the marker M does not have to be a grid-like marker as in this example, and may be a simple figure such as a polygon or a circle, or a combination of a plurality of these figures.Furthermore, it may be a dot pattern arranged in a matrix at equal intervals.

[0187] In the above embodiments, the puncture needle has been described as having a function of cauterization, but the puncture needle may have a function other than cauterization. In addition, the puncture needle has been described as having a function other than the puncture needle, but the puncture needle may have a function other than the puncture needle, such as a treatment tool for injecting a fluorescent agent such as ICG (Indocyanine Green), a biopsy needle or forceps used to collect tissue for a biopsy, etc.

[0188] In addition, in the above embodiments, the inside of the body has been described taking body cavities such as the abdominal cavity and the thoracic cavity as examples, but the inside of the body may be an upper digestive tract such as the esophagus, a lower digestive tract such as the intestines, or a duct such as the bronchi. When the technology of the present disclosure is applied to a surgical field inside a duct, for example, a marker M is provided at the base end of a flexible endoscope inserted into the duct.

[0189] In each of the above embodiments, the medical device having an insertion portion may be the trocar 17 in addition to the endoscope 13 and medical probe (ultrasound probe 14 or OCT probe).

[0190] Furthermore, from the above description, the techniques described in the following supplementary claims can be understood.

[0191] [Additional note 1] A processor is provided. The processor acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; Deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executes display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; Medical support equipment. [Additional note 2] The processor The device is an in-vivo image in a state where an operation that causes a change in an inner wall of the body is performed from outside the body and a puncture path is superimposed and displayed, and based on two in-vivo images taken from at least two different viewpoints, it is determined whether or not a change position where the inner wall changes and the puncture path intersect in a three-dimensional space inside the body, and notifies the determination result. Item 1. A medical support device according to claim 1. [Additional note 3] When a change position where a change has occurred intersects with the puncture path displayed in a superimposed manner in two dimensions in both of the two in-vivo images, it is determined that the change position intersects with the puncture path in a three-dimensional space. Item 3. A medical support device as described in appended item 2. [Additional note 4] When an operation is performed multiple times, The processor A history of the positions of the changes in the multiple operations is superimposed on the in-vivo image. 4. The medical support device according to claim 2 or 3. [Additional note 5] The processor When the viewpoint of the in-vivo camera is moved during multiple operations, a history of the changed position is displayed, reflecting the amount of movement of the viewpoint. 5. The medical support device according to claim 4. [Additional note 6] The operation is a pressing operation of pressing the body surface of the subject from outside the body, The changed position is a protruding portion where the inner wall is protruding toward the inside of the body by the pressing operation. The medical support device according to any one of claims 2 to 5. [Additional note 7] The processor derives an intersection point in a three-dimensional space between an inner wall of the body and the puncture path based on the depth information within the body that represents a distribution of depth, which is a distance in a depth direction from a viewpoint of the internal camera. Item 1. A medical support device according to claim 1. [Additional note 8] As an internal camera, we use a camera capable of estimating depth. The processor obtains the depth information. 8. The medical support device according to claim 7. [Additional note 9] The processor obtains depth information based on two images of the inside of the body taken from different viewpoints. 8. The medical support device according to claim 7. [Additional Note 10] The processor acquires the depth information using a machine learning model that receives an in-vivo image as an input and outputs depth information. 8. The medical support device according to claim 7. [Additional Note 11] The processor When an internal image is acquired while an operation that causes a change in the inner wall of a body cavity of a subject is being performed from outside the body, determining whether or not a change position where the inner wall changes in a three-dimensional space inside the body intersects with the puncture path based on the depth information, and notifying the result of the determination; The medical support device according to any one of claims 7 to 10. [Additional Note 12] The medical device is a medical probe that can observe the internal structure of an organ. The medical support device according to any one of claims 1 to 11. [Additional Note 13] The medical probe is an ultrasound probe. 13. The medical support device according to claim 12. [Additional Note 14] the medical probe is provided in an insertion section and has a guide groove for guiding insertion of the puncture needle to a target position inside the organ by engaging with the puncture needle; The processor specifies a position at which the puncture path is to be superimposed on the in-vivo image based on the relative positional relationship between the marker and the guide groove. 14. The medical support device according to claim 12 or 13. [Additional Note 15] the processor specifies a position at which the puncture path is to be superimposed on the internal image based on a target position inside the organ specified in the internal image of the organ acquired by the medical probe and a correlation between a coordinate system of the internal image and a coordinate system of the internal image derived based on the position and orientation information; The medical support device according to any one of claims 12 to 14. [Additional Note 16] In a case where a puncture path is set by a simulation before surgery on a three-dimensional image of an organ obtained before surgery, the processor uses a correlation between a coordinate system of the internal image of the organ acquired by the medical probe and a coordinate system of the three-dimensional image, and a correlation between the coordinate system of the internal image and a coordinate system of the internal image derived based on the position and orientation information, to specify a position at which the puncture path is to be superimposed on the internal image; The medical support device according to any one of claims 12 to 15. [Additional Note 17] The processor changes the display of a portion of the puncture path. The medical support device according to any one of claims 1 to 16. [Additional Note 18] A part of the puncture path is a part of the inner wall side of the body, The processor highlights a portion of the inner wall side. 18. The medical support device according to claim 17. [Additional Note 19] A part of the puncture path is a part on the organ side, The processor reduces visibility of a portion of the organ side compared to the inner wall side. 19. The medical support device according to claim 17 or 18. [Additional Note 20] The processor detects an insertion length of the insertion portion of the medical device, and determines a range for changing the display based on the detected insertion length. 20. The medical support device according to claim 17, wherein the medical support device is a medical support device for supporting a patient. [Additional Note 21] The processor determines the extent to which the display is to be changed based on the user's designation. The medical support device according to any one of claims 17 to 20. [Additional Note 22] The processor is capable of displaying a plurality of different puncture paths. The medical support device according to any one of claims 1 to 21. [Additional Note 23] A method for operating a medical support device having a processor, comprising: The processor acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; Deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executes display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; A method for operating a medical support device. [Additional note 24] An operating program for causing a computer to function as a medical support device, acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; Deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; performing a display control step of superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; An operating program for a medical support device that causes a computer to execute the above. [Additional note 25] a medical device having an insertion portion to be inserted into a body of a subject, the insertion portion being provided with a marker capable of being image-recognized; an in-vivo camera that captures an in-vivo image whose imaging range includes the insertion portion, the marker, and the organ to be punctured by the puncture needle; A medical support system comprising: a medical support device having a processor, The processor Acquire internal images, Deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executes display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; Medical support system.

[0192] From the above description, the technology described in the following Supplementary Clause 2 can be understood. The technology described in the Supplementary Clause 2 relates to a puncture method performed by the medical staff ST while visually checking the operative field image 21 from the first viewpoint VP1 and the operative field image 21 from the second viewpoint VP2 shown in the above second embodiment.

[0193] [Second additional note 1] While visually checking an in-vivo image captured by an in-vivo camera that captures the inside of a subject's body, the in-vivo image includes within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion, and the in-vivo image has a puncture path of the puncture needle superimposed thereon at a position specified based on position and orientation information including at least one of the position and orientation of the insertion portion derived based on the marker, The manipulation that causes changes to the internal walls of the body is performed from outside the body. While the operation is being performed, it is determined whether or not a change position where an inner wall changes in a three-dimensional space inside the body intersects with the puncture path, based on a first in-vivo image taken from a first viewpoint and a second in-vivo image taken from a second viewpoint different from the first viewpoint; When it is determined that the change position and the puncture path intersect, the puncture needle is inserted into the intersection of the change position and the puncture path. Puncture method. [Second additional note 2] The operation includes two operations, a first operation and a second operation, which are performed at the same position but at different times, visually confirming a first in-vivo image while performing a first operation, and performing a first determination of an intersection between a changed position in the first in-vivo image and the puncture path; Then, while performing the second operation, the second in-vivo image is visually recognized, and a second determination is made to determine an intersection between a changed position in the second in-vivo image and the puncture path; If it is determined that the two lines intersect in both the first and second determinations, it is determined that the two lines intersect in three-dimensional space. A puncture method as described in Attachment 2, Paragraph 1. [Second supplementary note 3] A first in-vivo image is viewed while the operation is being performed, and a first determination is made to determine an intersection between a changed position in the first in-vivo image and the puncture path, and a second in-vivo image is viewed while the operation is being continued, and a second determination is made to determine an intersection between a changed position in the second in-vivo image and the puncture path; If it is determined that the two lines intersect in both the first and second determinations, it is determined that the two lines intersect in three-dimensional space. A puncture method as described in Attachment 2, Paragraph 1.

[0194] In the above embodiment, the hardware structure of the processor 41 that executes various processes such as image acquisition, position and orientation information derivation, image synthesis, and display control may be any of the various processors shown below. The various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) that is a processor having a circuit configuration designed specifically for executing specific processes.

[0195] The above-mentioned various processes may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (e.g., a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). Also, a plurality of processing units may be configured by a single processor. An example of configuring a plurality of processing units by a single processor is a form using a processor that realizes the functions of the entire system including a plurality of processing units by a single IC (Integrated Circuit) chip, such as a System On Chip (SOC).

[0196] In this way, the various processing units are configured as a hardware structure using one or more of the various processors described above.

[0197] Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit combining circuit elements such as semiconductor elements can be used. do.

[0198] In addition to the operating program of the medical support device, the technology disclosed herein also extends to a computer-readable storage medium (such as a USB memory or a DVD (Digital Versatile Disc)-ROM (Read Only Memory)) that non-temporarily stores the operating program of the medical support device.

[0199] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0200] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0201] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0202] 11 Medical Support Equipment 13 Endoscopy 13A Insertion section 13B, 13B1, 13B2 Camera 14 Ultrasound probe 14A Insertion section 14B Tip 14C Ultrasonic Transducer 14D Control section 16 Display 17 Trocar 18 Puncture needle 18A needle part 18B Grip part 21 Surgical field image 22 Ultrasound Images 22G Ultrasound Images 27 Tumor 29 Guide groove 29A Guide groove 29B Guide groove 41 Processors 41A Image acquisition unit 41B Position and orientation information derivation unit 41C Image synthesis unit 41D Display control unit 42 Reception Device 44 Storage 45 Communication I / F 46 External I / F 48 Bus 49 Medical Assistance Program 50 Dimensional Information 56 Lattice Pattern 56A intersection 56A code 123 Preoperative preparation information 131 Tomography equipment 132 Tomographic images 132A Tomographic image 133 Image Database 134 Preoperative 3D images (three-dimensional images) 134A Voxel Data 137 Vascular Structure 151 Ultrasound 3D Images 151A Voxel Data 158 First Positional Information 159 Second location information θ Tilt angle θ1 Tilt angle θ2 Tilt angle AX axis AXO Shooting Optical Axis BS body surface DMP Depth Map FG finger L Baseline length LM Machine Learning Model LV Liver M Marker NP insertion position NR1 First puncture line NR2 Second puncture line NR3 3rd puncture line NR4 4th puncture line NRT target puncture line PR, PR1, PR2 ridges PT patient RG1, RG2 range S Scanning trajectory SF field ST Medical Staff T1, T2 time VP1, VP2 viewpoints W Inner wall XP intersection Xin, Yin, Xpb, Ypb coordinates

Claims

1. A processor is provided. The processor, acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executing a display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; Medical support equipment.

2. The processor, a step of determining whether or not a change position at which the inner wall changes and the puncture path intersect in a three-dimensional space within the body based on two in-body images taken from at least two different viewpoints, the in-body images being in a state in which an operation that causes a change in the inner wall of the body is performed from outside the body and the puncture path is superimposed on the in-body images, and reporting the determination result; The medical support device according to claim 1 .

3. When a change position where the change has occurred and the puncture path displayed in a superimposed manner intersect two-dimensionally in both of the two in-vivo images, it is determined that the change position and the puncture path intersect in the three-dimensional space. The medical support device according to claim 2.

4. When the above operation is performed multiple times, The processor, a history of the change position of each of the multiple operations is displayed in a superimposed manner on the in-vivo image; The medical support device according to claim 2.

5. The processor, When the viewpoint of the internal camera is moved during a plurality of operations, a history of the changed position is displayed, reflecting the amount of movement of the viewpoint. The medical support device according to claim 4.

6. the operation is a pressing operation of pressing a body surface of the subject from outside the body, The changed position is a raised portion where the inner wall is raised toward the inside of the body by the pressing operation. The medical support device according to claim 2.

7. The processor derives an intersection point in a three-dimensional space between an inner wall of the body and the puncture path based on the internal depth information representing a distribution of depth, which is a distance in a depth direction from a viewpoint of the internal camera. The medical support device according to claim 1 .

8. A camera capable of estimating the depth is used as the in-body camera, The processor obtains the depth information. The medical support device according to claim 7.

9. The processor acquires the depth information based on two in-vivo images taken from different viewpoints. The medical support device according to claim 7.

10. The processor acquires the depth information using a machine learning model that receives the in-vivo image as an input and outputs the depth information. The medical support device according to claim 7.

11. The processor, When the in-vivo image is acquired while an operation that causes a change in an inner wall of the body cavity of the subject is being performed from outside the body, determining whether or not a change position at which the inner wall changes in a three-dimensional space inside the body intersects with the puncture path based on the depth information, and notifying a result of the determination; The medical support device according to claim 7.

12. The medical device is a medical probe capable of observing an internal structure of the organ. The medical support device according to claim 1 .

13. The medical probe is an ultrasound probe. The medical support device according to claim 12.

14. the medical probe has a guide groove provided in the insertion section for engaging with the puncture needle to guide insertion of the puncture needle into a target position inside the organ; The processor specifies a position at which the puncture path is to be superimposed on the in-vivo image based on a relative positional relationship between the marker and the guide groove. The medical support device according to claim 12.

15. the processor specifies a position at which to superimpose the puncture path on the in-vivo image based on a target position inside the organ specified in the internal image of the organ acquired by the medical probe and a correlation between a coordinate system of the internal image and a coordinate system of the in-vivo image derived based on the position and orientation information; The medical support device according to claim 12.

16. In a case where the puncture path is set by a simulation before surgery on a three-dimensional image of the organ acquired in advance before surgery, the processor identifies a position at which the puncture path is to be superimposed on the in-vivo image by utilizing a correlation between a coordinate system of the internal image of the organ acquired by the medical probe and a coordinate system of the three-dimensional image, and a correlation between the coordinate system of the internal image and a coordinate system of the in-vivo image derived based on the position and orientation information; The medical support device according to claim 12.

17. The processor changes a display of a portion of the puncture path. The medical support device according to claim 1 .

18. The part of the puncture path is a part on an inner wall side of the body, The processor highlights a portion of the inner wall side. The medical support device according to claim 17.

19. the part of the puncture path is a part on the organ side, The processor reduces visibility of a portion of the organ side compared to an inner wall side. The medical support device according to claim 17.

20. The processor detects an insertion length of the insertion part of the medical device, and determines a range for changing the display based on the detected insertion length. The medical support device according to claim 17.

21. The processor determines an extent to which the display is to be changed based on a user specification. The medical support device according to claim 17.

22. The processor is capable of displaying a plurality of different puncture paths. The medical support device according to claim 1 .

23. A method for operating a medical support device having a processor, comprising: The processor, acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executing a display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; A method for operating a medical support device.

24. An operating program for causing a computer to function as a medical support device, acquiring an in-vivo image captured by an in-vivo camera that captures images of the inside of the subject's body, the in-vivo image including within its capture range an organ to be punctured by the puncture needle, an insertion portion of a medical device to be inserted into the body, and an image-recognizable marker provided on the insertion portion; Deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; performing a display control step of superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; An operating program for a medical support device that causes a computer to execute the above.

25. a medical device having an insertion portion to be inserted into a body of a subject, the insertion portion being provided with a marker that can be image-recognized; an in-vivo camera that captures an in-vivo image whose imaging range includes the insertion section, the marker, and an organ to be punctured by the puncture needle; A medical support system comprising: a medical support device having a processor, The processor, acquiring said in-vivo image; deriving position and orientation information including at least one of a position and an orientation of the insertion unit based on the marker; executing a display control for superimposing and displaying a puncture path of the puncture needle at a position specified in the in-vivo image based on the position and orientation information; Medical support system.

Images