Medical assistance device, medical assistance device operation method, medical assistance device operation program product, and medical assistance system

Abstract

Provided are a medical assistance device, a method for operating a medical assistance device, an operation program product, and a medical assistance system, with which it is possible to display, with a simple configuration, a puncture path at an appropriate position corresponding to the position or posture of a medical instrument. A processor of a medical assistance device performs: a process for acquiring an in-vivo image which is captured by an in-vivo camera for capturing an image in the body of a subject and which includes, in an imaging range, an organ punctured by a puncture needle, an insertion part of a medical device inserted into the body, and a marker provided in the insertion part and capable of recognizing the image; deriving position and posture information including at least one of the position and posture of the insertion part from the mark; and performing display control for superimposing and displaying the puncture path of the puncture needle at the position determined in the in-vivo image according to the position and orientation information.

Description

Technical Field

[0001] The technology of the present invention relates to a medical assistance device, a working method and a working procedure of the medical assistance device, and a medical assistance system. Background Art

[0002] In endoscopic surgery, an ultrasonic probe is used to obtain an ultrasonic image of the liver, and the location of the tumor, i.e., the puncture target, is determined in the ultrasonic image. However, even if the puncture target is shown in the ultrasonic image, it is difficult to determine from which position on the body surface and in which direction the puncture needle should be inserted, because the ultrasonic image is an image showing the inside of an organ in the body.

[0003] In order to solve such problems, puncture-assisted technologies are used. For example, Patent Document 1 discloses a technology that uses a magnetic navigation system using a puncture needle provided with a magnetic position sensor at the front end to display the position and posture of the front end of the puncture needle inserted into the body in real time. Patent Document 2 discloses a technology that uses a laser pointer that irradiates a laser beam toward the body surface along the inclination of the guide groove at the front end of an ultrasonic probe provided with a guide groove for guiding the insertion of the puncture needle.

[0004] Patent Document 1: U.S. Patent No. 8688196

[0005] Patent Document 2: European Patent Gazette No. 3136940

[0006] Patent document 1 has the problem of requiring a large device such as a magnetic navigation system. Patent document 2 requires a light source for emitting a laser beam to be built into the front end of a medical device such as an ultrasonic probe. In order to build in a light source, the internal structure of the ultrasonic probe needs to be modified, which is more laborious than changing the outer packaging. Summary of the invention

[0007] The technology involved in the present invention provides a medical support device capable of displaying a puncture path at an appropriate position corresponding to the position or posture of a medical instrument with a simple structure, an operating method and an operating program of the medical support device, and a medical support system.

[0008] The technology involved in the present invention is a medical assisting device, which has a processor that performs the following processing: acquiring an in-vivo image, which is captured by an in-vivo camera that captures the inside of a subject and includes, within a photographic range, an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the body, and a mark that can identify an image and is provided on the insertion portion; deriving position and posture information including at least one of a position and a posture of the insertion portion based on the mark; and executing display control that overlays and displays the puncture path of the puncture needle at a position determined in the in-vivo image based on the position and posture information.

[0009] Preferably, the processor determines whether the changed position of the inner wall intersects the puncture path in the three-dimensional space inside the body based on the in-vivo image that overlaps and displays the state of the puncture path when the operation is performed from outside the body and the in-vivo image is taken from at least two different viewpoints, and notifies the determination result.

[0010] Preferably, when the changed position in both of the two in-vivo images intersects the superimposed and displayed puncture path two-dimensionally, it is determined that the changed position intersects the puncture path in three-dimensional space.

[0011] Preferably, when a plurality of operations are performed, the processor displays a history of each of the changed positions in the plurality of operations in an overlapping manner on the in-vivo image.

[0012] Preferably, when the viewpoint of the in-vivo camera moves during the multiple operations, the processor reflects the amount of movement of the viewpoint and displays a history of the changed position.

[0013] Preferably, the operation is a pressing operation of pressing the body surface of the subject from outside the body, and the changed position is a bulging portion in which the inner wall bulges toward the inside of the body due to the pressing operation.

[0014] Preferably, the processor derives the intersection point between the inner wall in the body and the puncture path in the three-dimensional space based on the depth information in the body indicating the distribution of the depth, ie, the distance in the depth direction from the viewpoint of the in-body camera.

[0015] Preferably, as an in-vivo camera, using a camera capable of estimating depth, the processor acquires depth information.

[0016] Preferably, the processor obtains depth information based on two in-vivo images captured at different viewpoints.

[0017] The processor preferably obtains the depth information using a machine learning model that takes the in-vivo images as input and outputs the depth information.

[0018] Preferably, when an in-vivo image of a state in which an operation is performed from outside the body to change the inner wall of the subject's body cavity is acquired, the processor determines based on the depth information whether the change position of the inner wall in the three-dimensional space in the body intersects the puncture path and notifies the determination result.

[0019] Preferably, the medical device is a medical probe capable of observing the internal structure of an organ.

[0020] Preferably the medical probe is an ultrasound probe.

[0021] Preferably, the medical probe is arranged at the insertion portion and has a guide groove for guiding the puncture needle to be inserted into a target position inside the organ by engaging with the puncture needle. The processor determines the position for superimposing the puncture path on the in-vivo image based on the relative positional relationship between the mark and the guide groove.

[0022] Preferably, the processor determines a position at which to superimpose the puncture path on the in-vivo image based on a target position inside the organ determined in the internal image of the organ acquired by the medical probe and a correlation between a coordinate system of the internal image derived based on the position and posture information and a coordinate system of the in-vivo image.

[0023] Preferably, when the puncture path is set through a pre-operative simulation in a three-dimensional image of an organ acquired in advance before the operation, the processor determines the position at which the puncture path is superimposed on the in-vivo image by using the correlation between the coordinate system of the internal image of the organ acquired by the medical probe and the coordinate system of the three-dimensional image and the correlation between the coordinate system of the internal image derived based on the position posture information and the coordinate system of the in-vivo image.

[0024] Preferably the processor changes the display of a portion of the puncture path.

[0025] Preferably, a portion of the puncture path is a portion of an inner wall side in the body, and the processor emphatically displays the portion of the inner wall side.

[0026] Preferably, the portion of the puncture path is a portion on the organ side, and the processor makes the visibility of the portion on the organ side lower than that on the inner wall side.

[0027] Preferably, the processor detects an insertion length of an insertion portion of the medical device, and determines a range to change the display based on the detected insertion length.

[0028] Preferably, the processor determines the range to change the display according to a user's designation.

[0029] Preferably the processor is capable of displaying a different plurality of puncture paths.

[0030] In the working method of the medical assistance device involved in the technology of the present invention, the medical assistance device includes a processor, and the processor performs the following processing: obtaining an in-vivo image, which is photographed by an in-vivo camera that photographs the body of a subject and includes an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the body, and a mark that can identify the image and is provided on the insertion portion in the photographing range; deriving position and posture information including at least one of the position and posture of the insertion portion based on the mark; and executing display control to overlap and display the puncture path of the puncture needle at a position determined in the in-vivo image based on the position and posture information.

[0031] In the working procedure of the medical assisting device involved in the technology of the present invention, it enables a computer to function as a medical assisting device, and the working procedure enables the computer to execute the following steps: obtaining an in-vivo image, which is photographed by an in-vivo camera that photographs the body of a subject and includes, within the photographing range, an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the body, and a mark that can identify the image and is provided on the insertion portion; based on the mark, deriving position and posture information including at least one of the position and posture of the insertion portion; and performing display control to overlap and display the puncture path of the puncture needle at a position determined in the in-vivo image based on the position and posture information.

[0032] The technology involved in the present invention is a medical assistance system, which comprises: a medical instrument, having an insertion portion inserted into the body of a subject, and a mark capable of identifying an image is provided on the insertion portion; an in-vivo camera, which captures an in-vivo image that includes the insertion portion, the mark, and an organ punctured by a puncture needle within a photographic range; and a medical assistance device, having a processor, in which the processor performs the following processing: acquiring an in-vivo image; deriving position and posture information including at least one of the position and posture of the insertion portion based on the mark; and executing display control to overlap and display the puncture path of the puncture needle at a position determined in the in-vivo image based on the position and posture information.

[0033] Effects of the Invention

[0034] According to the technique of the present invention, it is possible to display the puncture path at an appropriate position corresponding to the position or posture of the medical instrument with a simple configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a diagram showing an overview of a medical support system including a medical support device.

[0036] Figure 2 It is a figure which shows the state inside the body during endoscopic surgery.

[0037] Figure 3 It is an explanatory diagram of the first guide groove of the ultrasonic probe.

[0038] Figure 4 It is an explanatory diagram of the second guide groove of the ultrasonic probe.

[0039] Figure 5 It is a diagram showing the inserted state of the puncture needle guided by the first guide groove.

[0040] Figure 6 It is a diagram showing the inserted state of the puncture needle guided by the second guide groove.

[0041] Figure 7 This is a diagram showing the hardware configuration of a medical assistance device.

[0042] Figure 8 It is a flowchart showing the overall steps of medical assistance treatment.

[0043] Fig. 9 It is a diagram showing a method of deriving position and posture information of an ultrasonic probe based on a marker.

[0044] Fig.10 It is a diagram showing the relationship between the position and posture of the marker and the ultrasonic probe.

[0045] Fig.11 This is another diagram showing the relationship between the position and posture of the marker and the ultrasonic probe.

[0046] Fig.12 This is a diagram showing a method of superimposing and displaying a puncture path based on position and posture information.

[0047] Fig.13 It is a diagram showing a pressing operation performed by a finger FG according to the second embodiment.

[0048] Fig.14 This is a diagram showing a state in which the insertion position NP is searched for by a pressing operation.

[0049] Fig.15 This is a flowchart showing the overall procedure of the medical support process according to the second embodiment.

[0050] Fig.16 This is a flowchart showing an example of a three-dimensional intersection determination procedure.

[0051] Fig.17 This is a diagram showing a state where the raised portion intersects the first puncture line in both of the two operation field images.

[0052] Fig.18 This is a diagram showing a state in which the raised portion intersects the first puncture line in only one of the two operation field images.

[0053] Fig.19 This is a flowchart showing another example of the three-dimensional intersection determination procedure.

[0054] Fig. 20 This is a diagram showing an example of displaying an operation history of a pressing operation.

[0055] Fig.21 This is a diagram showing an example of making the operation history follow the change of the camera viewpoint.

[0056] Fig. 22 Yes means Fig.21 A diagram of the processing of a processor in the example shown.

[0057] Fig.23This is a diagram showing an example in which three-dimensional intersection determination can be performed without relying on image analysis.

[0058] Fig.24 This is a flowchart showing the overall procedure of the medical support process according to the third embodiment.

[0059] Fig.25 This is a diagram showing an example of a method for acquiring a depth map.

[0060] Fig.26 FIG. 5 is a diagram showing an example of deriving intersection points from a depth map.

[0061] Fig. 27 A diagram showing a modified example of a method for acquiring a depth map.

[0062] Fig.28 This is a diagram showing another variation of the method for acquiring a depth map.

[0063] Fig.29 This is a flowchart showing a processing procedure of an example of performing three-dimensional intersection determination of a first puncture line and a raised portion based on a depth map.

[0064] Fig.30 This is a diagram showing an example of processing for determining the three-dimensional intersection of the first puncture line and the protuberance based on the depth map.

[0065] Fig.31 This is a diagram showing an example in which a portion of the inner wall side of the first puncture line is highlighted and displayed.

[0066] Fig.32 This is a diagram showing an example in which the visibility of a portion of the first puncture line on the organ side is reduced.

[0067] Fig.33 This is a diagram showing an example in which a scale is displayed on a part of the first puncture line.

[0068] Fig.34 This is a diagram showing an example in which the range for changing the display of the first puncture line is determined according to the insertion length of the ultrasonic probe.

[0069] Fig.35 FIG. 1 is a diagram showing an example in which a plurality of first puncture lines are displayed.

[0070] Fig.36 It is a figure which shows the 4th embodiment.

[0071] Fig.37 It means next Fig.36 Diagram of the steps.

[0072] Fig.38 This is a diagram showing a method of creating a preoperative 3D image.

[0073] Fig.39 This is a diagram showing a method of creating preoperative preparation information.

[0074] Fig.40 This is a diagram showing a method for creating an ultrasonic 3D image.

[0075] Fig.41 This is a diagram showing a method of using the correlation between an ultrasonic 3D image and a preoperative 3D image.

[0076] Explanation of symbols

[0077] 11-medical auxiliary device, 13-endoscope, 13A-insertion part, 13B, 13B1, 13B2-camera, 14-ultrasonic probe, 14A-insertion part, 14B-front end part, 14C-ultrasonic transducer, 14D-operating part, 16-display, 17-trocar, 18-puncture needle, 18A-needle part, 18B-grip part, 21-surgical area image, 22-ultrasonic image, 22g-ultrasonic image group, 27-tumor, 29-guide groove, 29A-guide groove , 29B-guide slot, 41-processor, 41A-image acquisition unit, 41B-position and posture information export unit, 41C-image synthesis unit, 41D-display control unit, 42-receiving device, 44-storage device, 45-communication I / F, 46-external I / F, 48-bus, 49-medical auxiliary program, 50-size information, 56-grid pattern, 56A-intersection, 123-preoperative preparation information, 131-tomographic imaging device, 132-tomographic image group, 132A-tomographic image group Layer image, 133-image database, 134-preoperative 3D image (three-dimensional image), 134A-voxel data, 137-vascular structure, 151-ultrasound 3D image, 151A-voxel data, 158-first position relationship information, 159-second position relationship information, θ-tilt angle, θ1-tilt angle, θ2-tilt angle, AX-axis, AXO-photographic optical axis, BS-body surface, DMP-depth map, FG-finger, L-baseline length, LM-machine learning model, LV -liver, M-marker, NP-insertion position, NR1-1st puncture line, NR2-2nd puncture line, NR3-3rd puncture line, NR4-4th puncture line, NRT-target puncture line, PR, PR1, PR2-protuberance, PT-patient, RG1, RG2-range, S-scanning trajectory, SF-surgical field, ST-medical staff, T1, T2-time, VP1, VP2-viewpoint, W-inner wall, XP-intersection, Xin, Yin, Xpb, Ypb-coordinates. DETAILED DESCRIPTION

[0078] [First embodiment]

[0079] like Figure 1 and Figure 2As shown, as an example, a medical assistance system 10 is used when performing endoscopic surgery using an endoscope 13 on a patient PT. Endoscopic surgery is different from laparotomy, and refers to making a small hole in the body of the patient PT, and then inserting a medical device such as an endoscope 13 to perform the surgery. In addition to providing medical personnel ST, including doctors, with a field of view of the surgical field in the patient PT's body, the medical assistance system 10 also provides auxiliary information for assisting surgery and examinations and other medical treatments. As described later, the auxiliary information is specifically the puncture path of the puncture needle 18. Since this medical assistance system 10 has the function of providing auxiliary information in real time during surgery, it is also called a surgical navigation system, etc. As an example, the medical assistance system 10 includes a medical assistance device 11, an endoscope 13, an ultrasonic probe 14, and a display 16. The medical assistance device 11 is an example of a "medical assistance device" involved in the technology of the present invention, and the patient PT is an example of a "subject".

[0080] The medical assisting device 11 is communicatively connected to the endoscope 13, the ultrasonic probe 14, and the display 16. In endoscopic surgery, a portion of the endoscope 13 and the ultrasonic probe 14 including their respective front ends are inserted into the body via a trocar 17. The trocar 17 is an insertion instrument provided with an insertion hole for inserting the endoscope 13 and the like and a valve provided in the insertion hole to prevent gas leakage. In endoscopic surgery, since carbon dioxide gas is injected into the abdominal cavity for pneumoperitoneum, the trocar 17 is used when inserting the endoscope 13 and the ultrasonic probe 14 and the like into the body.

[0081] Furthermore, in this example, the target site of surgery is the liver LV. Figure 2 The state where the endoscope 13 and the ultrasonic probe 14 are inserted into the abdomen of the patient PT is shown. The puncture needle 18 is a treatment instrument for puncturing a lesion such as a tumor contained in an organ as a puncture object. As an example, the puncture needle 18 is a cauterization puncture needle for cauterizing the lesion. The cauterization puncture needle is provided with an electrode for applying a high-frequency voltage at the front end. If the high-frequency voltage is applied while the electrode is puncturing the lesion, the lesion is necrotized due to the heat generated by the electrode. In this example, as a surgery, the following example is explained: a tumor 27 in the liver LV is visualized by an ultrasonic image 22, and the visualized tumor 27 is cauterized by the puncture needle 18 (see also Figure 5 and Figure 6 ), thereby performing treatment to necrotize the tumor 27. The puncture needle 18 includes a needle portion 18A and a grip portion 18B provided on the proximal end side of the needle portion 18A.

[0082] The endoscope 13 has an insertion portion 13A which is inserted into the body of the patient PT, and a camera 13B and a light source for illumination (LED (Light Emitting Diode) etc.) are built into the front end of the insertion portion 13A. As an example, the endoscope 13 is a rigid endoscope whose insertion portion 13A is rigid, and is mostly used for abdominal cavity observation, so it is also called a laparoscope etc. The camera 13B has: image sensors such as CCD (Charge Coupled Device) image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors; and an imaging optical system including a lens that images the subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor that can capture color images. The camera 13B is an example of an "in-vivo camera" involved in the technology of the present invention.

[0083] The endoscope 13 optically photographs the surgical field SF including the target part (liver LV in this example) in the body of the patient PT using the camera 13B. The surgical field SF is a space extending in the body cavity defined by organs and body walls in the body. The endoscope 13 is connected to an image processing processor for the endoscope (not shown), which processes the imaging signal output by the image sensor to generate a surgical field image 21 of the surgical field SF in the body. The surgical field image 21 is an example of an "in-vivo image photographed by an in-vivo camera" involved in the technology of the present invention.

[0084] As the illumination light of the endoscope 13, special light such as ultraviolet light and infrared light is sometimes used, but visible light such as white light can also be used. In addition, special light such as ultraviolet light and infrared light can also be used as the illumination light of the endoscope 13. As special light, short-wavelength narrow-band light obtained by narrowing the band of light in a short-wavelength region such as the ultraviolet region, etc., which is limited to a specific wavelength, can also be used. The surgical field image 21 captured by the endoscope 13 is sent to the medical assistance device 11 in real time via the endoscope image processing processor. Figure 2 , symbols Xin and Yin represent the coordinate system of the surgical field image 21 .

[0085] As auxiliary information for guiding the insertion of the puncture needle 18, a first puncture line NR1 indicating the puncture path of the puncture needle 18 is superimposed and displayed on the operation field image 21. The first puncture line NR1 will be described later.

[0086] The ultrasonic probe 14 has an insertion portion 14A inserted into the body of the patient PT and an operation portion 14D located at the base end side of the insertion portion 14A, similarly to the endoscope 13. An ultrasonic transducer 14C is built into the front end 14B of the insertion portion 14A. The ultrasonic transducer 14C transmits ultrasonic waves to the target part and receives reflected waves reflected at the target part. The ultrasonic probe 14 is connected to an ultrasonic probe image processing processor (not shown). The ultrasonic probe image processing processor performs image reconstruction processing based on the reflected waves according to the signal corresponding to the reflected waves received by the ultrasonic probe 14. An ultrasonic image 22 representing the internal structure of the target part scanned by the ultrasonic probe 14 is generated by the image reconstruction processing. The ultrasonic image 22 is a so-called B (Brightness) mode image that visualizes the internal structure from the surface layer to the deep layer where the ultrasonic wave reaches the target part as brightness information. The ultrasonic image 22 visualizes the internal structure of the target part that cannot be observed in the surgical field image 21 obtained by optically photographing. The ultrasonic probe 14 is an example of a "medical device" and a "medical probe" involved in the technology of the present invention.

[0087] As auxiliary information for guiding the insertion of the puncture needle 18, a second puncture line NR2 indicating the puncture path of the puncture needle 18 is superimposed and displayed on the ultrasonic image 22. The second puncture line NR2 will be described later.

[0088] As an example, the ultrasonic probe 14 is a convex type that transmits ultrasonic waves radially, and obtains a sector-shaped image with the ultrasonic transducer 14C as the base point. By scanning the ultrasonic probe 14, a plurality of ultrasonic images 22 are captured along the scanning direction. The ultrasonic images 22 are transmitted to the medical support device 11 in real time via the ultrasonic probe image processing processor. Figure 2 In FIG. 2 , symbols Xpb and Ypb represent the coordinate system of the ultrasonic image 22 .

[0089] In the ultrasonic probe 14, a guide groove 29 is provided at the front end portion 14B of the insertion portion 14A. The guide groove 29 is a guide groove for guiding the puncture needle 18 to be inserted into a target position inside an organ by engaging with the puncture needle 18. The guide groove 29 has two guide grooves, namely, a first guide groove 29A and a second guide groove 29B. The first guide groove 29A is a guide groove 29 provided at the front end portion 14B on the base side of the ultrasonic transducer 14C. The second guide groove 29B is a guide groove 29 formed at the front end of the front end portion 14B and provided on the front end side of the ultrasonic transducer 14C. Hereinafter, when there is no need to distinguish between the first guide groove 29A and the second guide groove 29B, both are collectively referred to as the guide groove 29.

[0090] Figure 3 FIG. 2 shows the state of the puncture needle 18 guided by the first guide groove 29A. Figure 4The state of the puncture needle 18 guided by the second guide groove 29B is shown. Figure 5 The diagram schematically shows a state in which the puncture needle 18 inserted into the body is guided by the first guide groove 29A and the puncture needle 18 punctures the tumor 27 in the liver LV. Figure 6 The diagram schematically shows a state in which the puncture needle 18 inserted into the body is guided by the first guide groove 29A and the puncture needle 18 punctures the tumor 27 in the liver LV.

[0091] The first guide groove 29A is inclined at an angle of θ1 with respect to the direction of the axis AX of the distal end portion 14B. The first guide groove 29A is inclined backward so that the needle tip of the puncture needle 18 inserted from the proximal end side of the distal end portion 14B faces the distal end side of the distal end portion 14B. Conversely, the second guide groove 29B is inclined forward so that the needle tip of the puncture needle 18 inserted from the proximal end side of the distal end portion 14B faces the proximal end side of the distal end portion 14B.

[0092] The puncture needle 18 is inserted while confirming the tumor 27 on the ultrasonic image 22. Since both the first guide groove 29A and the second guide groove 29B are inclined toward the ultrasonic transducer 14C, the needle tip of the puncture needle 18 can be directed toward the region visualized on the ultrasonic image 22.

[0093] A marker M is provided at the front end portion 14B. The marker M is a marker that is recognizable in an optically captured image, that is, an optically detectable marker. The marker M can be captured by the camera 13B of the endoscope 13. The medical assisting device 11 estimates the position and posture of the guide groove 29 provided at the front end portion 14B by utilizing the marker M. As described later, specifically, the position and posture of the front end portion 14B of the insertion portion 14A are estimated by the marker M. The posture of the front end portion 14B is represented, for example, by the direction of the axis AX of the front end portion 14B. And, based on the estimated direction of the axis AX of the front end portion 14B, the position and posture of the guide groove 29 whose relative positional relationship with the front end portion 14B is known is estimated.

[0094] As an example, the marker M is composed of a pattern having a morphological feature such as a grid pattern or a dot pattern. A method for estimating the position and posture using the marker M will be described later. The marker M is an example of a "marker capable of identifying an image" involved in the technology of the present invention. The insertion portion 14A is an example of an "insertion portion of a medical device" involved in the technology of the present invention.

[0095] The medical assisting device 11 acquires the operation area image 21 from the endoscope 13 and acquires the ultrasonic image 22 from the ultrasonic probe 14. Figure 1 As shown, as an example, two monitors 16 are prepared, and two images, an operation field image 21 and an ultrasonic image 22, are displayed on each monitor 16. The operation field image 21 and the ultrasonic image 22 are output as animations, respectively, and are displayed as real-time previews on the monitors 16.

[0096] When the ultrasonic probe 14 is inserted into the surgical field SF, the ultrasonic probe 14 is reflected in the surgical field image 21. The medical support device 11 outputs the surgical field image 21 reflected by the insertion portion 14A (including the distal end portion 14B and the marker M) of the ultrasonic probe 14 to the display 16. The visual field of the surgical field SF in the body of the patient PT is provided to the medical staff ST through the screen of the display 16.

[0097] like Figure 5 and Figure 6 As shown, two types of second puncture lines NR2 respectively indicating the puncture path along the inclination angle θ1 of the first guide groove 29A and the puncture path along the inclination angle θ2 of the second guide groove 29B can be displayed in a superimposed manner on the ultrasonic image 22. The second puncture line NR2 is auxiliary information for guiding the puncture needle 18 to puncture a target position inside an organ in the ultrasonic image 22.

[0098] The second puncture line NR2 is information indicating a path through the first guide groove 29A or the second guide groove 29B that guides the puncture needle 18. The second puncture line NR2 is generated as an extension of the guide groove 29 based on the known positional relationship between the ultrasonic transducer 14C and the guide groove 29. The second puncture line NR2 is used as a rough target when the puncture needle 18 is punctured to the target position. For example, when the target position for puncturing the puncture needle 18 is a tumor 27 in the liver LV, the medical staff ST positions the ultrasonic probe 14 so that the second puncture line NR2 overlaps the tumor 27 in the ultrasonic image 22. In this state, the puncture needle 18 punctures the tumor 27 with the second puncture line NR2 as a rough target. In this way, the puncture needle 18 is passed through the guide groove 29 in a state where the position of the ultrasonic probe 14 is aligned so that the second puncture line NR2 passes through the target position, so that the puncture needle 18 can reach the target position.

[0099] However, since the ultrasonic image 22 is an internal image of the organ, if only the second puncture line NR2 is superimposed and displayed on the ultrasonic image 22, it is difficult to determine in what posture and at what position on the body surface BS the puncture needle 18 should be inserted when it is inserted from outside the body into the body. Figure 5 and Figure 6 As shown in FIG. 2 , it is difficult to determine the insertion angle of the puncture needle 18 from the body surface BS toward the organ and the insertion position NP on the body surface BS of the patient PT. Therefore, the medical support device 11 displays the first puncture line NR1 of the puncture needle 18 in an overlapping manner in the operation area image 21. Thus, auxiliary information that is a rough guide to the insertion angle and insertion position NP of the puncture needle 18 is provided to the medical staff ST. The first puncture line NR1 is an example of the "puncture path of the puncture needle" involved in the technology of the present invention.

[0100] In this way, as auxiliary information for assisting the puncture needle 18 in puncturing the target position, the medical assisting device 11 not only displays the second puncture line NR2 superimposed on the ultrasonic image 22, but also displays the first puncture line NR1 superimposed on the surgical field image 21. The first puncture line NR1 shows the puncture path from the body surface BS to the guide groove 29 of the ultrasonic probe 14 in the surgical field image 21, and the second puncture line NR2 shows the puncture path from the guide groove 29 to the target position in the organ such as the tumor 27 in the liver LV in the ultrasonic image 22.

[0101] exist Figure 7 1 shows an example of the hardware structure of the medical support device 11. The medical support device 11 is an example of the "medical support device" and "computer" involved in the technology of the present invention, and includes a processor 41, a receiving device 42, a display 16, a RAM (Random access memory) 43, a storage device 44, a communication I / F 45, and an external I / F 46. These components are connected to a bus 48 and can communicate with each other.

[0102] The medical assisting device 11 is operated by an operator such as a medical staff member ST through a receiving device 42. The receiving device 42 has a keyboard and a mouse (not shown) and receives instructions from the operator. The receiving device 42 may be a device that receives touch input such as a touch panel, a device that receives sound input such as a microphone, a device that receives gesture input such as a camera, etc.

[0103] Examples of the display 16 include an FL (Electro-Luminescence) display and a liquid crystal display, etc. As described above, there are two displays 16 , and each display 16 displays various information in addition to the operation field image 21 and the ultrasonic image 22 .

[0104] The processor 41 is, for example, a CPU (Central Processing Unit), and centrally controls each unit of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0105] The storage device 44 is a nonvolatile storage device that stores various programs and various parameters. Examples of the storage device 44 include a HDD (Hard Disk Drive) and an SSD (Solid State Drive). The storage device 44 stores a medical assistance program 49 for causing the computer to function as the medical assistance device 11.

[0106] The RAM 43 is a memory for temporarily storing information, and is used as a working memory by the processor 41. Examples of the RAM 43 include DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0107] The communication I / F 45 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control according to a communication protocol defined by various wired or wireless communication standards.

[0108] The external I / F 46 is, for example, a USB (Universal Serial Bus) interface, and is used for connection with peripheral devices such as a printer and a memory card.

[0109] The processor 41 reads the medical assistance program 49 from the storage device 44 and performs medical assistance processing by executing the medical assistance program 49 on the RAM 43. The medical assistance processing is realized by the processor 41 operating as the image acquisition unit 41A, the position and posture information derivation unit 41B, the image synthesis unit 41C, and the display control unit 41D. The medical assistance program 49 is an example of the "operating program of the medical assistance device" involved in the technology of the present invention.

[0110] The dimension information 50 includes dimension information of the ultrasonic probe 14. Specifically, the dimension information of the ultrasonic probe 14 includes information indicating the positional relationship of the mark M in the front end portion 14B of the ultrasonic probe 14 and the positional relationship of the guide groove 29 in the front end portion 14B, respectively. The positional relationship of the mark M in the front end portion 14B is, for example, information on the position and posture in which the grid of the grid pattern of the mark M is set relative to the axial direction and circumferential direction of the front end portion 14B. The positional relationship of the guide groove 29 in the front end portion 14B is the linear distance between the reference point of the front end portion 14B and the guide groove 29 (including the first guide groove 29A and the second guide groove 29B) along the axial direction of the axis AX of the front end portion 14B, the inclination angle θ (including θ1 and θ2) of the guide groove 29 relative to the axial direction of the front end portion 14B, and the like.

[0111] The image acquisition unit 41A performs image acquisition processing for acquiring the operation area image 21 and the ultrasonic image 22. For example, the image acquisition unit 41A acquires the operation area image 21 and / or the ultrasonic image 22 from a device including a processor of the endoscope 13 and / or a processor of the ultrasonic probe 14 via the external I / F 46 or the communication I / F 45. In addition, the medical support device 11 may also have a processor of the endoscope 13 and / or a processor of the ultrasonic probe 14.

[0112] The position and posture information deriving unit 41B executes processing for deriving position and posture information including the position and posture of the guide groove 29 of the ultrasonic probe 14 based on the marker M.

[0113] The image synthesis unit 41C determines the position at which the first puncture line NR1 is to be overlapped in the surgical field image 21 based on the derived position and posture information, and overlaps the first puncture line NR1 at the determined position. Here, the position and posture information is an example of "position and posture information including at least one of the position and posture of the insertion portion" involved in the technology of the present invention. In this example, the position and posture information includes the position and posture, but in the case where the position at which the first puncture line NR1 is to be overlapped can be determined at either side, it may be information including either the position or the posture.

[0114] The display control unit 41D performs control to display the operation field image 21 and the ultrasonic image 22 on the display 16. In addition, the display control to display the first puncture line NR1 superimposed on the operation field image 21 is performed. Specifically, the display control to display the first puncture line NR1 superimposed is a control to generate a composite image in which the first puncture line NR1 is superimposed on the operation field image 21 and to display the generated composite image on the display 16. Hereinafter, such display control is also simply referred to as superimposed display.

[0115] use Figure 8 to Figure 12 , the processing performed by the processor 41 of the medical assisting device 11 of the present invention is specifically described. Figure 8 is a flowchart showing the overall processing steps. Figures 9 to 12 Each process is described in detail in FIG.

[0116] like Figure 8As shown, in the medical assistance process, the processor 41 acquires the surgical field image 21 captured by the camera 13B of the endoscope 13 in step S1100. Then, in step S1200, the processor 41 detects the mark M of the ultrasonic probe 14 from the surgical field image 21. In step S1300, the processor 41 derives the position and posture information of the guide groove 29 of the ultrasonic probe 14 based on the mark M. In step S1400, the processor 41 derives the first puncture line NR1 of the puncture needle 18 in the surgical field image 21 based on the position and posture information of the guide groove 29. Then, in step S1500, the processor 41 displays the first puncture line NR1 superimposed on the surgical field image 21. The processor 41 repeats such processing until a display end instruction is input (step S1600).

[0117] The following is a more detailed description of the individual processing steps. Fig. 9 It is conceptually shown Figure 8 FIG. 1 is a diagram showing the detection of the marker M in step S1200 and the derivation of the position and posture information of the guide groove 29 of the ultrasonic probe 14 in step S1300. Fig. 9 For additional information, please refer to Fig.10 and Fig.11 Provide explanation.

[0118] like Fig. 9 As shown, in step S1200, the processor 41 detects the marker M based on the surgical field image 21 using an image processing method such as pattern matching. As an example, the marker M is a marker of a grid pattern 56 composed of a first line extending in the axis AX direction of the distal end portion 14B of the ultrasonic probe 14 and a second line orthogonal to the axial direction of the distal end portion 14B and formed in the circumferential direction along the outer peripheral surface of the distal end portion 14B. The grid spacing of the grid pattern 56 is constant. Symbol 56A represents a plurality of intersections of the first line and the second line. The processor 41 detects the marker M by searching for morphological features of the marker M such as the grid pattern 56 from the surgical field image 21. Of course, the marker M can also be detected using AI (Artificial Intelligence) technology using a machine learning model instead of a rule library method such as pattern matching.

[0119] The processor 41 determines the area in the surgical field image 21 where the mark M exists by detecting the mark M. Then, in step S1300, the position and posture of the front end portion 14B are first estimated based on the morphological features such as the grid pattern 56 of the mark M, and finally the position and posture information indicating the position and posture of the guide groove 29 is derived. The details are as follows.

[0120] Fig.10 and Fig.11The correspondence between the position and posture of the distal end portion 14B of the ultrasonic probe 14 having the mark M in the surgical field image 21 and the position and posture of the distal end portion 14B in the surgical field SF defined as a three-dimensional space is conceptually shown. The Zin axis of the surgical field SF as a three-dimensional space is parallel to the photographic optical axis of the camera 13B, and the Xin-Yin plane (hereinafter referred to as the XY plane) is parallel to the imaging plane of the camera 13B and is orthogonal to the photographic optical axis. Fig.10 and Fig.11 In the figure, the XY plane is parallel to the screen of the surgical field image 21. The surgical field image 21 is a projection image of the surgical field SF projected from one viewpoint. Therefore, as shown below, the processor 41 can estimate the position and posture of the distal end portion 14B in the surgical field SF based on the marker M detected from the surgical field image 21.

[0121] exist Fig.10 , the state in which the axial direction of the front end portion 14B of the ultrasonic probe 14 is orthogonal to the photographic optical axis of the camera 13B in the surgical field SF in three-dimensional space (more specifically, the state in which the axis AX of the front end portion 14B is parallel to the Xin axis) is shown. When the front end portion 14B of the ultrasonic probe 14 is in such a posture, in the surgical field image 21, the orthogonal lines of the grid pattern 56 marked M are respectively parallel to the Xin axis and the Yin axis, and the intervals between adjacent intersections 56A are also the same.

[0122] On the other hand, Fig.11 2 shows a state in which the axial direction of the distal end portion 14B of the ultrasonic probe 14 is not orthogonal to the imaging optical axis of the camera 13B but is inclined in the depth direction parallel to the imaging optical axis in the surgical field SF in the three-dimensional space. Fig.11 The posture shown is that the axis AX of the front end portion 14B is Fig.10 The posture shown is a state rotated about -15° around the Yin axis. When the distal end portion 14B is in this posture, in the surgical field image 21, the lines extending in the circumferential direction of the grid pattern 56 marked with M become shorter as they move away from the camera 13B in the depth direction. In addition, the intervals between adjacent intersections 56A are shortened by the intervals between the intersections 56A located at a position away from the camera 13B.

[0123] In this way, the form of the mark M reflected in the operation field image 21 changes according to the posture of the distal end portion 14B. The processor 41 estimates the posture of the distal end portion 14B of the ultrasonic probe 14 in the operation field SF based on the form of the mark M in the operation field image 21. Specifically, the direction of the axis AX of the distal end portion 14B in the operation field SF, that is, the axial direction of the distal end portion 14B is detected as the posture of the distal end portion 14B.

[0124] Furthermore, if the position of the front end portion 14B changes within the surgical field SF, the position of the marker M reflected in the surgical field image 21 will also change. The processor 41 estimates the position of the front end portion 14B within the surgical field SF based on the position of the marker M. The position of the front end portion 14B is, for example, the position of the reference point of the front end portion 14B, such as the front end position of the front end portion 14B. Furthermore, the photographic distance from the camera 13B to the marker M within the surgical field SF (i.e., the distance in the Zin axis direction parallel to the photographic optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the surgical field image 21. The processor 41 can derive the position coordinates of the reference point of the front end portion 14B within the surgical field SF based on the size information of the front end portion 14B including such a photographic distance and the known size of the marker M.

[0125] The processor 41 estimates the position and posture of the guide groove 29 of the ultrasonic probe 14 in the operation field SF based on the position and posture of the distal end portion 14B derived in this way and the positional relationship of the guide groove 29 in the distal end portion 14B included in the size information of the ultrasonic probe 14 .

[0126] In the position and posture information of the example of FIG. 10 , for example, the position of the distal end portion 14B of the ultrasonic probe 14 is information such as the position coordinates of the reference point of the distal end portion 14B in the surgical field SF, namely, X01, Y01, and Z01. The posture of the distal end portion 14B is defined as the axial direction of the distal end portion 14B in the surgical field SF, for example, the axis AX of the distal end portion 14B is parallel to the XY plane and the XZ plane and is orthogonal to the YZ plane.

[0127] The position of the first guide groove 29A is information such as the position coordinates of the reference point in the surgical field SF, namely, X11, Y11, and Z11. The position of the second guide groove 29B is information such as the position coordinates of the reference point in the surgical field SF, namely, X21, Y21, and Z21. The postures of the first guide groove 29A and the second guide groove 29B are respectively defined as postures with respect to the axis AX, which are the inclination angles θ1 and θ2 known as dimensional information.

[0128] exist Fig.11 In the position and posture information of the example, for example, the position of the distal end portion 14B of the ultrasonic probe 14 is information such as the position coordinates of the reference point of the distal end portion 14B in the surgical field SF, namely, X02, Y02, and Z02. The posture of the distal end portion 14B is defined as the axial direction of the distal end portion 14B in the surgical field SF, for example, the axis AX of the distal end portion 14B is -15° relative to the XY plane, parallel to the XZ plane, and 75° relative to the YZ plane.

[0129] The position of the first guide groove 29A is information such as the position coordinates of the reference point in the surgical field SF, namely, X12, Y12, and Z12. The position of the second guide groove 29B is information such as the position coordinates of the reference point in the surgical field SF, namely, X22, Y22, and Z22. The postures of the first guide groove 29A and the second guide groove 29B are respectively defined as postures with respect to the axis AX, which are the inclination angles θ1 and θ2 known as dimensional information.

[0130] And, if Fig.12 As shown, in step S1400, the processor 41 derives the first puncture line NR1 of the puncture needle 18 in the surgical field image 21 based on the position and posture of the guide groove 29. For example, the processor 41 derives the first puncture line NR1 that is a straight line along the inclination angle θ1 of the first guide groove 29A in the surgical field SF defined as a three-dimensional space based on the position coordinates of the first guide groove 29A in the surgical field SF defined as a three-dimensional space and the inclination angle θ1 relative to the axis AX of the distal end portion 14B. For example, with the position of the first guide groove 29A as a base point, a line segment that makes an angle of the inclination angle θ1 of the first guide groove 29A from the position is set as the first puncture line NR1. Then, the processor 41 derives the first puncture line NR1 in the surgical field image 21 by converting the first puncture line NR1 into the two-dimensional coordinate system of the surgical field image 21. The same is true for the second guide groove 29B.

[0131] Furthermore, the medical staff ST can select, by operation or setting, whether to display the first puncture line NR1 on the first guide groove 29A or on the second guide groove 29B. The first puncture line NR1 is derived from the selected guide groove 29. The example shown in FIG. 12 is an example in which the first puncture line NR1 is displayed on the first guide groove 29A.

[0132] In step S1500, the processor 41 displays the derived first puncture line NR1 superimposed on the surgical field image 21. In this way, the processor 41 performs display control to display the first puncture line NR1 of the puncture needle 18 superimposed at the position determined in the surgical field image 21 based on the position and posture information indicating the position and posture of the distal end portion 14B. As a result, the surgical field image 21 superimposed with the first puncture line NR1 is displayed on the display 16. The processor 41 repeatedly performs the above-mentioned processing until a display end instruction such as the end of activation of the medical assisting device 11 is input.

[0133] The operation area image 21 is output as an animation to the display 16 , so that the positions of the endoscope 13 and the ultrasonic probe 14 in the real space captured by the camera 13B are reflected, and the position and posture of the first puncture line NR1 in the operation area image 21 displayed on the display 16 are also updated.

[0134] The effect of the technology of the present invention will be described by taking the case where the puncture needle 18 is punctured into the liver LV, which is the organ to be punctured, as the target position. Figure 1 and Figure 5 As shown, for example, the medical staff ST adjusts the position and posture of the ultrasonic transducer 14C provided at the front end portion 14B of the ultrasonic probe 14 while observing the ultrasonic image 22 on which the second puncture line NR2 is superimposed. Specifically, the position and posture of the ultrasonic transducer 14C are adjusted at the position where the second puncture line NR2, which is an extension of the guide groove 29, passes through the tumor 27. The position and posture of the first puncture line NR1 superimposed on the surgical field image 21 changes in accordance with such changes in the position and posture of the ultrasonic probe 14. When the second puncture line NR2 is displayed in the ultrasonic image 22 in a manner passing through the tumor 27, the second puncture line NR2 shows the ideal puncture path of the puncture needle 18 in the liver LV. In this state, the position and posture of the ultrasonic probe 14 are fixed during the puncture.

[0135] And, for example, as an example, Fig.12 As shown, the first puncture line NR1 in the operation field image 21 in this state shows an ideal puncture path of the puncture needle 18 in the body cavity from the body surface BS of the patient PT to the surface of the liver LV as the puncture target.

[0136] When observing the overlap, it is shown as Fig.12 While viewing the operation area image 21 of the ideal first puncture line NR1 shown in the figure, the medical staff ST first adjusts the position and posture of the puncture needle 18 outside the body so that the puncture needle 18 is inserted along the ideal first puncture line NR1. Specifically, the medical staff ST finds the approximate insertion position NP corresponding to the first puncture line NR1, and punctures the puncture needle 18 into the body surface BS of the patient PT from the insertion position NP. Then, while observing the display of the first puncture line NR1, the medical staff ST adjusts the position and posture of the puncture needle 18, and inserts the puncture needle 18 into the body cavity toward the first guide groove 29A of the ultrasonic probe 14. The first puncture line NR1 is displayed with the first guide groove 29A as the base point, so by inserting the puncture needle 18 along the first puncture line NR1, it is possible to easily reach the first guide groove 29A.

[0137] After the puncture needle 18 reaches the first guide groove 29A, Figure 1 , Figure 2 and Figure 5 The puncture needle 18 can puncture the target tumor 27 while puncturing the liver LV along the second puncture line NR2 shown in the ultrasonic image 22. As an example, the tumor 27 is cauterized while the puncture needle 18 punctures the tumor 27.

[0138] As described above, the medical support device 11 according to the technology of the present invention includes a processor 41, and the processor 41 acquires the operation area image 21 (an example of an in-vivo image), which is captured by the camera 13B (an example of an in-vivo camera) of the endoscope 13 that captures the inside of the patient PT (an example of a subject), and includes the liver LV (an example of an organ) punctured by the puncture needle 18, the insertion portion 14A of the ultrasonic probe 14 (an example of a medical device) inserted into the body, and the mark M that can identify the image provided in the insertion portion 14A. Then, the processor 41 derives the position and posture information including at least one of the position and posture of the insertion portion 14A in the operation area image 21 based on the mark M. In addition, the processor 41 performs display control to display the first puncture line NR1 of the puncture needle 18 superimposed at the position determined in the operation area image 21 based on the position and posture information.

[0139] In the technology of the present invention, the hardware structure for overlapping and displaying the first puncture line NR1 only needs to set a mark on the insertion part of the medical device such as the ultrasonic probe 14. Therefore, there is no need for a large device such as a magnetic system. In addition, even in the case of misappropriating existing medical devices, it is only necessary to change the outer packaging of the insertion part, so it is simpler even compared with the structure of the built-in marking light source. Therefore, the medical auxiliary device 11 can display the first puncture line NR1 of the puncture needle 18 at an appropriate position corresponding to the position or posture of the medical device such as the ultrasonic probe 14 with a simple structure.

[0140] In the above-mentioned embodiment, the medical device inserted into the body of the patient PT is an ultrasonic probe 14 (an example of a medical probe) capable of observing the internal structure of organs such as the liver LV. Puncture treatment of organs in the body using a puncture needle 18 is mostly performed using a medical probe capable of observing the internal structure of the organ. Therefore, as in the above-mentioned embodiment, the technology of the present invention is particularly effective when a medical probe is used as a medical device. Moreover, the ultrasonic probe 14 and the puncture needle 18 are often used in combination. Therefore, the technology of the present invention is more effective when the ultrasonic probe 14 is used as a medical probe. In addition, as a medical probe capable of observing the internal structure of an organ, a probe other than the ultrasonic probe 14 may be used, for example, an optical coherence tomography (OCT: Optical Coherence Tomography) probe, etc.

[0141] In addition, the medical instrument may not be a medical probe capable of observing the internal structure of an organ. For example, the medical instrument may be a treatment instrument that does not have the function of observing the internal structure and has only the guide groove 29 for the puncture needle 18 at the front end. For example, when a tumor exists on the surface of an organ and the puncture needle 18 is used to puncture the tumor on the surface, even a treatment instrument that does not have such an internal structure observation function can appropriately guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST positions the guide groove 29 of the treatment instrument at a position corresponding to the tumor on the surface of the organ, and in this state, the puncture needle 18 is punctured into the tumor through the guide groove 29.

[0142] Furthermore, in the above-mentioned embodiment, as a medical instrument, a medical probe such as an ultrasonic probe 14 having a guide groove 29 is used. Furthermore, the processor 41 determines the position of the first puncture line NR1 overlapped in the surgical field image 21 based on the relative positional relationship between the marker M and the guide groove 29 (as an example, the size information of the ultrasonic probe 14). The puncture needle 18 is often used in combination with a medical probe having a guide groove 29 for guiding the puncture needle 18. Therefore, the technology of the present invention is particularly effective when a medical probe having a guide groove 29 is used as a medical instrument.

[0143] Furthermore, as a medical instrument, a simple rod-shaped treatment instrument without the guide groove 29 may be used. In the case where a tumor exists on the surface of an organ, even such a treatment instrument can be directed toward the tumor. Even if the first puncture line NR1 is displayed superimposed on the surgical field image 21 in which the treatment instrument is directed toward the tumor on the surface of the organ, it serves as a rough target for confirming the puncture direction of the puncture needle 18. Therefore, even if the medical instrument is a treatment instrument without the guide groove 29, the technology of the present invention is effective.

[0144] [Second embodiment]

[0145] As in the first embodiment Fig.12 As shown, even if the first puncture line NR1 is superimposed and displayed in the operation field image 21, the depth information of the operation field SF in the operation field image 21 is insufficient, so it is sometimes impossible to accurately grasp the depth of the operation field SF. Figure 5 and Figure 6 The intersection point of the first puncture line NR1 shown in FIG. 5 and the body surface BS is the insertion position NP. Figures 13 to 19 The second embodiment shown is a method for providing the user with information for understanding the insertion position NP corresponding to the first puncture line NR1 displayed superimposed on the operation field image 21 in order to solve the problem of the first embodiment.

[0146] As an example, Fig.13As shown in FIG. 1 , consider a pressing operation of pressing the body surface BS of the patient PT with the finger FG from outside the body. Figure 5 and Figure 6 As shown in , during endoscopic surgery, due to pneumoperitoneum in the body cavity, the body surface BS bulges outward from the body cavity in a convex shape. If such a pressing operation is performed on the body surface BS, the portion of the inner wall W of the body cavity inside the body surface BS corresponding to the pressing position bulges inward toward the body, generating a bulge PR.

[0147] And, if Fig.14 As shown, for example, when the operation area image 21 with the first puncture line NR1 superimposed thereon is displayed on the display 16, the medical staff ST performs a plurality of pressing operations on the body surface BS while confirming the pressing position with the finger FG on the display 16. Fig.14 As shown in the lower section of the operating field image 21, the medical staff ST can find the pressing position where the first puncture line NR1 intersects the raised portion PR. However, as described above, the depth information in the operating field image 21 is insufficient. Therefore, by only confirming that the first puncture line NR1 intersects the raised portion PR in the operating field image 21 at one viewpoint, it is impossible to determine whether the first puncture line NR1 and the raised portion PR intersect in three dimensions within the operating field SF in three-dimensional space. Therefore, in the second embodiment, a three-dimensional intersection determination is performed to determine whether the first puncture line NR1 and the raised portion PR intersect in three-dimensional space. Reference Figure 15 to Figure 19 Provide specific instructions.

[0148] Fig.15 This is an example of a flowchart showing the overall processing steps of medical support processing when performing three-dimensional intersection determination. Fig.15 In the flowchart shown, steps S1100 to S1500 and step S1600 are Figure 8 The flowchart of the first embodiment is the same as that shown in the figure. The following description will focus on the differences. The differences are steps S1510 to S1530 indicated by dotted lines.

[0149] like Fig.15 As shown in the example, in step S1500, when the operation instruction of the medical staff ST is input in the state where the first puncture line NR1 is displayed in an overlapping manner, the three-dimensional intersection determination is started. The medical staff ST inputs the operation instruction for starting the three-dimensional intersection determination while performing the pressing operation on the body surface BS with the finger FG. In step S1510, the processor 41 waits for the input of the operation instruction, and when the operation instruction is input ("Yes" in step S1510), it transfers to step S1520 to perform the three-dimensional intersection determination.

[0150] The three-dimensional intersection determination in step S1520 is a three-dimensional intersection determination of the first puncture line NR1 and the raised portion PR in the surgical field image 21 based on two different viewpoints.

[0151] like Fig.16 As shown, first, in step S1521 , the processor 41 performs a two-dimensional intersection determination of the operation field image 21 based on two viewpoints.

[0152] First, in step S1521A1, the processor 41 performs a pressing operation to acquire the surgical field image 21 of two different first and second viewpoints VP1 and VP2 in which the first puncture line NR1 is displayed in a superimposed manner.

[0153] like Fig.17 and Fig.18 As schematically shown in FIG. 1 , while pressing the body surface BS with the finger FG, the medical staff ST changes the viewpoint of the camera 13B by operating the endoscope 13. And, as an example, before and after the change from the first viewpoint VP1 to the second viewpoint VP2, the medical staff ST inputs whether the current viewpoint position is the viewpoint position before the change or the viewpoint position after the change to the processor 41. Thus, the processor 41 can obtain two surgical field images 21 of the first viewpoint VP1 and the second viewpoint VP2 before the change.

[0154] In addition, instead of detecting the viewpoint change of the camera 13B by manual operation, the processor 41 may detect the viewpoint change caused by the movement of the camera 13B by image analysis. In this case, for example, the processor 41 stores a certain number of past frames acquired as animation, and when a viewpoint change is detected, the frames before and after the frames are acquired as two surgical field images 21 of the first viewpoint VP1 and the second viewpoint VP2.

[0155] The processor 41 then proceeds to step S1521A2 and determines whether the first viewpoint VP1 and the second viewpoint VP2 intersect in the surgical field image 21 .

[0156] In step S1521A2, the processor 41 first detects the protrusion PR in each surgical field image 21. The detection of the protrusion PR is performed, for example, by an image analysis method using a pattern matching or machine learning model. The inner wall W has undulations even in a state where the pressing operation by the finger FG is not performed, but as described above, pneumoperitoneum is formed in the body cavity during endoscopic surgery, so the inner wall W as a whole bulges convexly toward the outside of the body. In this state, the pressing operation is performed by the finger FG, so the protrusion PR generated by the pressing operation protrudes in a direction opposite to the bulging direction caused by the pneumoperitoneum. Therefore, the protrusion PR has characteristics different from natural undulations in size and shape, so the protrusion PR can be detected by an image analysis method.

[0157] Of course, in addition to the method of detecting the protrusion PR by such image analysis, the protrusion PR may be detected by visual confirmation by the medical staff ST. For example, the display 16 is constituted by a touch panel, and the medical staff ST visually confirms the position of the protrusion PR in the surgical field image 21 displayed on the touch panel. Furthermore, the processor 41 may detect the protrusion PR based on the input of the operation of the medical staff ST specifying the position of the protrusion PR from the touch panel.

[0158] When the raised portion PR is detected, the processor 41 detects the intersection XP of the raised portion PR and the first puncture line NR1 in the surgical field images 21 at the different first viewpoint VP1 and second viewpoint VP2. The processor 41 holds the position of the first puncture line NR1 in the surgical field image 21 as known information, and thus detects the intersection XP by comparing the position of the detected raised portion PR in the surgical field image 21 with the position of the first puncture line NR1.

[0159] The processor 41 performs the two-dimensional intersection determination in step S1521 , and determines the three-dimensional intersection after step S1522 based on the determination result of the two-dimensional intersection determination.

[0160] like Fig.17 As shown, when the intersection XP is detected in both the surgical field images 21 of the first viewpoint VP1 and the second viewpoint VP2, it is considered that the raised portion PR intersects the first puncture line NR1 in the surgical field SF, which is a three-dimensional space in the body. In this case, the intersection point between the first puncture line NR1 and the inner wall W side becomes the appropriate insertion position NP.

[0161] In step S1522, the processor 41 determines whether the raised portion PR intersects the first puncture line NR1 in the surgical field images 21 at the first viewpoint VP1 and the second viewpoint VP2. Then, if the determination in step S1522 is affirmative ("Yes" in step S1522), the processor 41 transfers to step S1523 and determines that the raised portion PR intersects the first puncture line NR1 in three dimensions. That is, the processor 41 determines that the raised portion PR intersects the first puncture line NR1 in the three-dimensional space in the body, that is, the surgical field SF.

[0162] On the other hand, Fig.18In the example shown, the intersection XP is detected in the surgical field image 21 of the first viewpoint VP1, but the intersection XP is not detected in the surgical field image 21 of the second viewpoint VP2. In this way, when the intersection XP is not detected in both the first viewpoint VP1 and the second viewpoint VP2, it is considered that the raised portion PR and the first puncture line NR1 do not intersect in the surgical field SF, which is the three-dimensional space in the body. In this case, the intersection of the first puncture line NR1 and the inner wall W side does not become the appropriate insertion position NP.

[0163] If the determination in step S1522 is negative ("No" in step S1522), the processor 41 moves to step S1524 and determines that the raised portion PR and the first puncture line NR1 do not intersect three-dimensionally. That is, the processor 41 determines that the raised portion PR and the first puncture line NR1 do not intersect in the three-dimensional space in the body, that is, the surgical field SF.

[0164] Here, as an example, the intersection XP is a point where the protrusion PR intersects with the first puncture line NR1 somewhere within the range of the area from the apex of the protrusion PR to the periphery of the bottom. The intersection XP may be, for example, a point where the apex of the protrusion XP intersects with the first puncture line NR1. However, in this example, the intersection with the first puncture line NR1 in the area having a width up to the periphery of the bottom of the protrusion PR is also detected as the intersection XP. In this way, if there is a width within the range of detecting the intersection XP, a width will also be generated at the determined insertion position NP, but there is no problem in practice. This is because the body surface BS and the inner wall W have flexibility, so even if the insertion position NP of the puncture needle 18 is slightly deviated, the puncture direction of the puncture needle 18 can be corrected.

[0165] After completing the three-dimensional intersection determination in step S1520, the processor 41 Fig.15 The determination result is notified in step S1530 shown in the figure. As a method of notifying the determination result, for example, a method of flashing the first puncture line NR1, changing the color, thickening the line, or increasing the brightness on the surgical field image 21 can be considered. In addition, a message indicating a three-dimensional intersection can also be displayed. In addition, a method other than such a visual method can also be used, such as notifying a message indicating a three-dimensional intersection with sound, or issuing a warning sound.

[0166] By checking the determination result of step S1530 , the medical staff ST can grasp the pressing position on the body surface BS where the first puncture line NR1 and the raised portion PR intersect three-dimensionally, and as a result, can grasp the appropriate insertion position NP of the puncture needle 18 .

[0167] In addition, as a two-dimensional intersection determination step of step S1521, in addition to Fig.16 In addition to the steps shown, it is also possible to Fig.19 Steps shown. Fig.16 In the steps shown, while the pressing operation by the finger FG is continued, the processor 41 acquires two surgical field images 21 of different first viewpoints VP1 and second viewpoints VP2, and then performs two-dimensional intersection determination on each of the acquired two surgical field images 21. The surgical field image 21 of the first viewpoint VP1 and the surgical field image 21 of the second viewpoint VP2 are examples of the first in-vivo image and the second in-vivo image acquired while the pressing operation or the like is continued.

[0168] In contrast, Fig.19 In the steps shown, the operation includes two operations, namely, the first operation and the second operation, performed on the same position at different timings, and the surgical field image 21 of different viewpoints is acquired at each timing. That is, the processor 41 first performs the first pressing operation, namely the first operation, in step S1521B1, and acquires the first surgical field image 21 (an example of the first in-vivo image) of the first viewpoint VP1 in a state where the first puncture line NR1 is superimposed and displayed. Then, in step S1521B2, the processor 41 determines the two-dimensional intersection of the protrusion PR in the first surgical field image 21 and the first puncture line NR1. This determination is an example of the first determination.

[0169] Next, the processor 41 performs a second pressing operation, i.e., the second operation, in step S1521B3, and obtains the second surgical field image 21 (an example of a second in-vivo image) of the second viewpoint VP2 in which the first puncture line NR1 is superimposed and displayed. Here, the pressing position of the second pressing operation is the same as the first pressing position. Then, in step S1521B4, the two-dimensional intersection of the raised portion PR in the second surgical field image 21 and the first puncture line NR1 is determined. This determination is an example of the second determination. The processing after step S1523 is the same as that after step S1524. Fig.16 The steps are the same.

[0170] Thus, the two-dimensional intersection determination step of step S1521 can be Fig.16 The steps shown can also be Fig.19 Steps shown.

[0171] As described above, in the second embodiment, the processor 41 acquires two surgical field images 21 captured from at least two different first viewpoints VP1 and second viewpoints VP2, and the surgical field images 21 are examples of in-vivo images in which a pressing operation, which is an example of an operation that causes a change in the inner wall W in the body, is performed from the outside of the body, and a state of the first puncture line NR1 indicating the puncture path is superimposed and displayed. Furthermore, the processor 41 determines whether the protrusion PR, which is an example of a changed position where the inner wall W changes in the three-dimensional space in the body, intersects with the first puncture line NR1 based on the two surgical field images 21, and notifies the determination result. Therefore, in the second embodiment, information on the first puncture line NR1 can be provided in consideration of the lack of depth information in the first embodiment. As a result, the medical staff ST can grasp the appropriate insertion position NP where the first puncture line NR1 intersects with the inner wall W in three dimensions.

[0172] Furthermore, in the above example, when the raised portion PR, which is an example of a changed position where the inner wall W changes in both of the two surgical field images 21, intersects the first puncture line NR1, which is a puncture path displayed in an overlapping manner, in two dimensions, the processor 41 determines that the raised portion PR intersects the first puncture line NR1 in three-dimensional space. In this way, the three-dimensional intersection is determined by determining the two-dimensional intersection in the two surgical field images 21, so the three-dimensional intersection can be determined by simple processing.

[0173] In the above-mentioned second embodiment, the pressing operation is an example of "an operation that causes a change in the inner wall W in the body", and the raised portion PR is an example of a "change position" where the inner wall W changes. The operation that causes the inner wall W to change may also be an operation other than the pressing operation, for example, it may be an operation such as light irradiation or pigment injection from outside the body. The pressing operation is an operation that causes the inner wall W to change in shape. Light irradiation is, for example, an operation that causes a brightness change in the portion of the inner wall W corresponding to the irradiation position by irradiating light to the body surface BS using a laser pointer or the like. In this case, the position where the brightness change occurs is called the "change position". Pigment injection is an operation that causes the inner wall W to change in color by injecting pigment into the body surface BS. In this case, the position where the color change occurs is called the "change position". For example, the processor 41 can detect the change position where the brightness change or color change occurs based on the pixel value (including brightness information and color information) of the surgical field image 21. In this way, the operation that causes the inner wall W in the body to change from outside the body can be any operation that causes some kind of change in the inner wall W.

[0174] Furthermore, by using a pressing operation to change the inner wall W, the inner wall W can be changed by a simple operation without using a tool. Also, since the raised portion PR changes in shape, it may be easier to recognize the image than changes in brightness or color.

[0175] Furthermore, although the example in which the three-dimensional intersection determination is performed using the surgical field images 21 of two viewpoints has been described, the surgical field images 21 of three or more viewpoints may be used.

[0176] (Variation 1 of the Second Embodiment: Operation History)

[0177] As an example, Fig. 20 As shown, for example, when multiple pressing operations such as pressing operations by fingers FG are performed to change the inner wall W, the processor 41 can overlay and display the history of the position changes of each of the multiple operations, i.e., the operation history, on the surgical field image 21. Fig. 20 , as an example of the history of position changes, the operation history of the pressing operation is shown.

[0178] When the protrusion PR is detected, the processor 41 temporarily records the detection position of the detected protrusion PR, and the protrusion PR generated by the multiple operations performed in the past (at Fig. 20 The first puncture line NR1 is superimposed on the respective detection positions of the operation area image 21 (exemplified as PR1, PR2 and PR3). Thus, the medical staff ST can confirm the relative positional relationship between the past pressed positions pressed by the medical staff ST and the first puncture line NR1. Therefore, since the situation of repeatedly pressing the same pressed position is suppressed, the search efficiency of the insertion position NP is improved.

[0179] (Variation 2 of the Second Embodiment: Following Changes in Camera Viewpoint)

[0180] And, as an example, Fig.21 As shown, you can also use Fig. 20 The operation history shown follows the viewpoint change of the camera 13B. That is, when the viewpoint of the in-vivo camera moves during multiple operations such as multiple pressing operations, the processor 41 reflects the amount of viewpoint movement and displays an example of the history of the changed position, that is, the history of the ridge PR.

[0181] As an example, Fig.21 As shown, when the ridges PR1 and PR2 are displayed as the operation history, it is considered that the viewpoint of the camera 13B changes from the first viewpoint VP1 to the second viewpoint VP2 over time. In this case, the display positions of the ridges PR1 and PR2 displayed in an overlapping manner in the surgical field image 21 of the first viewpoint VP1 acquired at time T1 and the display positions of the ridges PR1 and PR2 displayed in an overlapping manner in the surgical field image 21 of the second viewpoint VP2 acquired at time T2 will of course change. The processor 41 detects the amount of movement of the viewpoint of the camera 13B and corrects the display positions of the ridges PR1 and PR2, which are the past operation history, according to the amount of movement.

[0182] As an example, Fig. 22 As shown, the processor 41 detects the movement amount of the viewpoint of the camera 13B based on the mark M. For example, Figure 5 and Figure 6 As shown, when the puncture needle 18 is punctured into the tumor 27, as an example, the ultrasonic probe 14 is fixed at a position where the tumor 27 overlaps with the second puncture line NR2 in the ultrasonic image 22. In this way, when the viewpoint of the camera 13B is moved from the first viewpoint VP1 to the second viewpoint VP2 while the position of the ultrasonic probe 14 is fixed, the projection mode of the mark M set at the front end portion 14B in the surgical field image 21 of the first viewpoint VP1 and the surgical field image 21 of the second viewpoint VP2 are changed respectively. The processor 41 uses Fig.10 and Fig.11 The position and posture information of the front end portion 14B of the ultrasonic probe 14 is derived according to the description. Here, since the position of the front end portion 14B is fixed, the change in the position and posture information derived from the surgical field image 21 of the first viewpoint VP1 and the position and posture information derived from the surgical field image 21 of the second viewpoint VP2 corresponds to the movement amount of the viewpoint of the camera 13B. Therefore, the processor 41 can detect the movement amount of the viewpoint of the camera 13B based on the position and posture information of the marker M projected in the two surgical field images 21 of the first viewpoint VP1 and the second viewpoint VP2. Then, the processor 41 corrects the display position of the operation history based on the detected movement amount. As shown as the display position of the raised portion PR, the operation history is an example of the history of the change position of the inner wall W.

[0183] In this way, by reflecting the movement amount of the viewpoint of the camera 13B to display the history of the changed position of the inner wall W, even if the viewpoint of the camera 13B changes, the history of the changed position of the inner wall W can be displayed at an appropriate position.

[0184] (Variation 3 of the Second Embodiment)

[0185] In the second embodiment, in the above example, accurate information of the insertion position NP corresponding to the intersection XP of the first puncture line NR1 and the changed position of the inner wall W of the protrusion PR is presented by image analysis. However, the information may be presented without using image analysis.

[0186] As an example, Fig.23As shown, consider the case where the axis AX of the front end portion 14B of the ultrasonic probe 14 is approximately orthogonal to the photographic optical axis AXO of the camera 13B of the endoscope 13. In this case, even if the pressing position is changed in the direction of the axis AX, the position in the depth direction will not change. Therefore, if the pressing position is changed in the direction of the axis AX, if the intersection XP where the raised portion PR and the first puncture line NR1 intersect in two dimensions is found in the surgical field image 21, it can be determined that the intersection XP is also a point of intersection in three dimensions. In this way, under the limitation that the axis AX of the front end portion 14B of the ultrasonic probe 14 is approximately orthogonal to the photographic optical axis AXO of the camera 13B of the endoscope 13, the three-dimensional intersection can be determined without relying on image analysis.

[0187] [Third Embodiment]

[0188] As an example, in Figure 24 to Figure 26 In the third embodiment shown, the processor 41 derives the intersection of the inner wall W in the body and the first puncture line NR1 in three-dimensional space based on the depth map DMP in the body that represents the depth distribution in the depth direction from the viewpoint of the camera 13B. The depth map DMP is an example of "depth information" involved in the technology of the present invention.

[0189] Fig.24 FIG. 1 is a flowchart showing an example of the overall steps of the medical assistance process according to the third embodiment. Fig.24 In the flowchart shown in the second embodiment, Fig.15 The flowchart shown in FIG. 1 is different in that steps S1560 to S1580 are added instead of steps S1510 to S1530. Fig.24 In the embodiment, the three-dimensional intersection XP of the first puncture line NR1 and the inner wall W is displayed based on the depth map DMP, instead of the three-dimensional intersection determination of the change position (for example, the raised portion PR) of the first puncture line NR1 and the inner wall W performed in the second embodiment. The third embodiment will be described below, centering on the differences from the second embodiment.

[0190] In step S1560, the processor 41 receives an instruction to display the three-dimensional intersection point between the first puncture line NR1 and the inner wall W. When the operation instruction to display the intersection point is input ("Yes" in step S1560), the processor 41 derives the three-dimensional intersection point XP between the first puncture line NR1 and the inner wall W based on the depth map DMP. Then, in step S1560, the three-dimensional intersection point XP is superimposed and displayed on the surgical field image 21.

[0191] Fig.25 An example of a method for acquiring a depth map DMP is shown. Fig.25 In the example shown, a stereo camera is used as the camera 13B. Fig.25 The camera 13B of the stereo camera shown has a plurality of cameras 13B1 and 13B2. The camera 13B1 and the camera 13B2 are arranged at a distance from each other, and the distance becomes a baseline length L. Furthermore, the focal lengths of the cameras 13B1 and 13B2 are the same. The processor 41 acquires the surgical field image 21 captured by the first viewpoint VP1 of the camera 13B1 and the surgical field image 21 captured by the second viewpoint VP2 of the camera 13B2. Then, a depth map DMP is generated based on the respective surgical field images 21 of the different first viewpoint VP1 and second viewpoint VP2. The processor 41 generates the depth map DMP based on the principle of triangulation based on the focal length of the camera 13B, the baseline length L, and the parallax of the same subject reflected in both surgical field images 21 of the two different viewpoints. The depth map DMP shows the distance in the depth direction of each position within the surgical field SF. Furthermore, the depth map DMP can be obtained by utilizing the following: Fig.10 and Fig.11 The coordinates of the first puncture line NR1 in the surgical field SF are obtained by using the marking method described in the above description. Fig.26 As shown, the processor 41 can derive the intersection point XP between the inner wall W and the first puncture line NR1 based on the depth map DMP.

[0192] Thus, in the third embodiment, the processor 41 derives the intersection point XP in the three-dimensional space between the inner wall W and the first puncture line NR1, which is an example of the puncture path, based on the depth map DMP, which is an example of the depth information. Therefore, the three-dimensional intersection point XP between the first puncture line NR1 and the inner wall W can be derived without performing an operation such as a pressing operation to change the inner wall W as in the second embodiment.

[0193] Fig.25 The stereo camera of the example shown is an example of a "camera capable of estimating depth" for the operating field SF. As a camera capable of estimating depth, in addition to a stereo camera, a camera having a distance sensor as camera 13B can also be used. The distance sensor is, for example, a sensor using infrared rays. A distance sensor using infrared rays irradiates infrared rays toward a subject and receives infrared rays reflected in the subject. The intensity of the infrared rays reflected in the subject varies depending on the distance to the subject. The distance sensor measures the distance by detecting a voltage corresponding to the intensity. The processor 41 can generate a depth map DMP based on information obtained by scanning the entire operating field SF with such a distance sensor.

[0194] (Variation 1 of the Depth Map Acquisition Method)

[0195] In addition to the method of estimating depth using a camera such as a stereo camera, other methods can also be used to obtain a depth map DMP. Fig. 27As shown, the processor 41 can generate a depth map DMP based on the surgical field images 21 of two different viewpoints, namely, the first viewpoint VP1 and the second viewpoint VP2, acquired by one camera 13B. Fig. 27 The method shown is a method of acquiring the surgical field image 21 from two different viewpoints by moving one camera 13B instead of using a stereo camera. In this case, the viewpoint is changed by moving the camera 13B. The baseline length L corresponds to the amount of movement of the camera 13B. Fig.21 As described above, the movement amount of the camera 13B can be detected based on the mark M in the state where the tip portion 14B is fixed at one position.

[0196] In addition, in order to detect the amount of movement of the viewpoint of the camera 13B and the parallax of the subject reflected in the two surgical field images 21 at different viewpoints, other markers may be used in addition to or in place of the marker M of the ultrasonic probe 14. For example, the other marker is fixed at one position of the surgical field SF, and the two surgical field images 21 at different viewpoints are acquired in this state. Since the ultrasonic probe 14 may move, it is considered that if a marker that is less likely to move than the ultrasonic probe 14 is used, the amount of movement of the viewpoint and the parallax can be accurately detected, and the accuracy of the depth finally acquired will also be improved.

[0197] Method for generating a depth map DMP based on surgical area images 21 from two viewpoints Fig.25 The same is true for the case of the stereo camera shown. According to this example, the depth map DMP can be generated without using a stereo camera.

[0198] (Variation 2 of the Depth Map Acquisition Method)

[0199] Furthermore, the method for acquiring the depth map DMP is not limited to the method of generating the depth map DMP based on the operation area image 21 of two viewpoints by the principle of triangulation. Fig.28 As shown, it can also be acquired by generating it based on an operating area image 21 .

[0200] exist Fig.28 In the example shown, the processor 41 generates a depth map DMP using a machine learning model LM. The machine learning model LM, for example, takes the surgical field image 21 as input data and outputs a depth map DMP. The machine learning model LM uses teacher data consisting of a combination of the surgical field image 21 and the depth map DMP that is correct answer data for learning. As the machine learning model LM, for example, a convolutional neural network can be used. The convolutional neural network has an encoder that extracts feature quantities of an image, and is capable of outputting output data in the form of an image based on the feature quantities extracted by the encoder. As the depth map DMP of correct answer data used as teacher data, for example, a depth map DMP acquired using a stereo camera can be used.

[0201] (Variation of the third embodiment)

[0202] Furthermore, the depth map DMP may be used to perform the operation described in the second embodiment. Fig.15 The three-dimensional intersection determination of the first puncture line NR1 and the raised portion PR will be described. Fig.29 In the flowchart shown, step S1520C is added to replace Fig.15 In addition to step S1520 shown in FIG. Fig.15 The flowcharts shown are the same. The following description will focus on the differences.

[0203] As described above, the protrusion PR is generated on the inner wall W by the pressing operation. The pressing operation is an example of an operation that causes a change in the inner wall W, and the protrusion PR is an example of a changed position of the inner wall W. The inner wall W is changed in shape by the pressing operation. The processor 41 can obtain information about the depth of the surgical field SF based on the depth map DMP. Fig.10 and Fig.11 As described above, the three-dimensional coordinates of the first puncture line NR1 in the surgical field SF can be derived from the marker M. In step S1520C, the processor 41 performs three-dimensional intersection determination of the first puncture line NR1 and the raised portion PR based on the depth map DMP.

[0204] like Fig.30 As shown, in step S1520C, the processor 41 first passes Fig.26 The processor 41 then compares the first puncture line NR1 with the three-dimensional coordinates of the raised portion PR in the depth map DMP to determine the three-dimensional intersection of the first puncture line NR1 and the raised portion PR. In this way, by using the depth map DMP, the three-dimensional intersection determination of the first puncture line NR1 and the position of the shape change of the inner wall W can be performed.

[0205] In addition, Fig.30In the example of the embodiment, as the operation to change the inner wall W, a pressing operation to change the shape of the inner wall W is used as an example, but the operation to change the inner wall W may not be an operation to change the shape. For example, as described in the second embodiment, it may be an operation such as light irradiation or pigment injection to change the brightness or color of the inner wall W. In this case, for example, the processor 41 detects the change position where the brightness change or color change occurs from the surgical field image 21. Thus, the processor 41 can derive the two-dimensional coordinates of the change position in the surgical field image 21. In addition, the processor 41 derives the three-dimensional coordinates of the change position by extracting the depth information corresponding to the two-dimensional coordinates of the change position from the depth map DMP. The processor 41 can perform a three-dimensional intersection determination between the first puncture line NR1 and the change position of brightness or color by comparing the three-dimensional coordinates of the first puncture line NR1 with the derived three-dimensional coordinates of the change position.

[0206] [Modification of the method of displaying the first puncture line]

[0207] like Fig.31 and Fig.32 As shown, the processor 41 may change the display of a part of the first puncture line NR1. That is, instead of making the display mode the same in the entire region from one end to the other end of the first puncture line NR1, the display mode of a part may be changed.

[0208] As an example, Fig.31 As shown, the processor 41 may highlight a portion of the inner wall W side of the first puncture line NR1. Fig.31 In the figure, range RG1 is the highlighted range. With regard to the first puncture line NR1, the necessity of displaying it is relatively high mostly just after puncturing the body surface BS. This is because the state in which the needle tip of the puncture needle 18 is close to the guide groove 29 of the ultrasonic probe 14 is a state in which the puncture needle 18 can be inserted toward the guide groove 29 of the ultrasonic probe 14, and therefore the necessity of guiding the insertion of the puncture needle 18 at this point in time is relatively low. On the other hand, just after the puncture needle 18 punctures the body surface BS, the distance to the guide groove 29 of the ultrasonic probe 14 is long. Therefore, it is difficult to grasp the relative positional relationship between the puncture needle 18 and the guide groove 29, and the necessity of guiding the insertion of the puncture needle 18 through the first puncture line NR1 is relatively high. By highlighting a portion of the inner wall W side of the first puncture line NR1, the first puncture line NR1 can be made conspicuous within the range RG1 where the necessity is high.

[0209] The range RG1 is, for example, a range of about 5 cm from the inner wall W. When observing the inside of an organ with the ultrasonic probe 14, the medical staff ST knows the approximate distance from the inner wall W to the organ in the body cavity into which the ultrasonic probe 14 is inserted. The approximate distance is, for example, about 10 cm. In this case, if the distance of the range RG1 is set to about 5 cm, when the needle tip of the puncture needle 18 reaches the end of the range RG1 on the organ side, the medical staff ST can know that the needle tip of the puncture needle 18 is located about 5 cm from the body surface BS toward the organ.

[0210] And, as an example, Fig.32 As shown in FIG. 1 , the visibility of a portion of the first puncture line NR1 on the organ side is made lower than that on the inner wall W side. Fig.32 In the example shown, the first puncture line NR1 is set not to be displayed for a pre-set distance range RG2 from the position of the guide groove 29, which is one end of the first puncture line NR1 on the organ side. When the needle tip of the puncture needle 18 is close to the guide groove 29, the medical staff ST inserts the needle tip into the guide groove 29 while checking both the needle tip and the guide groove 29. In this case, when grasping the positional relationship between the needle tip and the guide groove 29, the first puncture line NR1 may become an obstacle. Therefore, if Fig.32 As shown, by displaying the first puncture line NR1 in a partial range RG2 on the organ side, the positional relationship between the needle tip of the puncture needle 18 and the guide groove 29 may be easily grasped.

[0211] In addition, Fig.32 In the example shown, the first puncture line NR1 is not displayed in the range RG2, but it does not have to be displayed. For example, the visibility may be reduced by lowering the brightness compared to other areas.

[0212] Furthermore, when the first puncture line NR1 is displayed, the visibility of the range RG2 may not always be reduced. For example, the visibility may be reduced when the needle tip of the puncture needle 18 approaches an organ after the puncture needle 18 punctures the body surface BS. The timing for reducing the visibility may be manually specified, or the processor 41 may perform image analysis on the surgical field image 21 to detect when the puncture needle 18 approaches an organ or the front end portion 14B of the ultrasonic probe 14.

[0213] And, if Fig.33As shown, the scale SC1 may be displayed in a partial range RG2 on the organ side of the first puncture line NR1. The range Rg2 where the scale SC1 is displayed is, for example, a range of about 3 cm from the guide groove 29, and the scale SC1 is displayed every 1 cm in the range RG2. By displaying the scale SC1, the distance between the needle tip of the puncture needle 18 and the guide groove 29 can be clearly displayed at the position where the puncture needle 18 approaches the guide groove 29. The processor 41 can use the three-dimensional coordinates of the guide groove 29 in the surgical field SF to Fig.10 and Fig.11 Since the method described above is understood, the scale SC1 can be derived from the three-dimensional coordinates of the guide groove 29 .

[0214] like Fig.34 As shown, the processor 41 can detect the insertion length of the insertion portion 14A of the ultrasonic probe 14, which is an example of a medical device, and determine the displayed range of the change range RG1, etc. according to the detected insertion length. For example, in order to serve as an approximate target of the insertion length of the insertion portion 14A, the insertion portion 14A is provided with a scale SC2. On the insertion portion 14A, a curved portion is provided at the front end 14B. In the insertion portion 14A, a scale SC2 is provided at a predetermined interval (for example, 1 cm interval) from the curved portion toward the base end side. The processor 41, for example, performs image recognition of the scale SC2 from the surgical field image 21 and detects the insertion length. The front end 14B of a medical probe such as the ultrasonic probe 14 is arranged on the surface of an organ. Therefore, the insertion length of the insertion portion 14A becomes an effective clue for grasping the distance from the inner wall W to the surface of the organ. Thus, the processor 41 can appropriately derive the range of the displayed range RG1, or can set the displayed range of the change range RG1, etc. to an appropriate range corresponding to the insertion length.

[0215] Furthermore, the processor 41 may determine the range in which the display of the first puncture line NR1 is changed according to the designation of the user such as the medical staff ST. Thus, the range in which the display of the first puncture line NR1 is changed can be set to an appropriate range in accordance with the user's request.

[0216] And, if Fig.35 As shown in FIG. 1 , the processor 41 is preferably capable of displaying a plurality of different first puncture lines NR1. Thus, various first puncture lines NR1 corresponding to the purpose can be displayed. Fig.35 In the illustrated example, a plurality of first puncture lines NR1 corresponding to two types of first guide grooves 29A and second guide grooves 29B included in the ultrasonic probe 14 are displayed.

[0217] In addition, the processor 41 may display a plurality of first puncture lines NR1 as candidates, and allow the user, i.e., the medical staff ST, to select one of them, and set the remaining candidates to be non-displayed. Alternatively, a plurality of first puncture lines NR1 may be displayed, and when the actual puncture needle 18 is inserted, only the first puncture line NR1 close to the position of the puncture needle 18 is displayed, and the remaining portions are set to be non-displayed. The processor 41 can derive the respective distances between the puncture needle 18 and the plurality of first puncture lines NR1 by performing image analysis on the surgical field image 21.

[0218] [Fourth embodiment]

[0219] As an example, in Fig.36 and Fig.37 In the fourth embodiment shown, the processor 41 determines the position of the fourth puncture line NR4 superimposed on the surgical field image 21 based on the tumor 27 determined in the ultrasonic image 22 and the correlation between the coordinate system of the ultrasonic image 22 and the coordinate system of the surgical field image 21. The tumor 27 is an example of a target position determined in the internal image of the organ acquired by the medical probe. The correlation is an example of a correlation between the coordinate system of the internal image derived from the marker M and the coordinate system of the in-vivo image. The fourth puncture line NR4 is an example of a "puncture path" involved in the technology of the present invention.

[0220] If used Fig.10 and Fig.11 As described above, by using the marker M, the position and posture of the front end portion 14B of the ultrasonic probe 14 can be estimated. The positional relationship of the ultrasonic transducer 14C in the front end portion 14B is known based on the size information. Therefore, if the positional posture information of the front end portion 14B can be estimated, the position and posture of the ultrasonic transducer 14C can also be estimated. In this way, the processor 41 can derive the correlation between the coordinate system of the surgical field image 21 projected by the front end portion 14B of the ultrasonic probe 14 and the coordinate system of the ultrasonic image 22 acquired by the ultrasonic probe 14.

[0221] In the fourth embodiment, first, Fig.35 As shown, consider a case where a tumor 27, which is a target position, is included in an ultrasonic image 22 acquired by an ultrasonic probe 14. The second puncture line NR2 is a line set with the guide groove 29 of the ultrasonic probe 14 as a base point. In the fourth embodiment, the processor 41 derives a third puncture line NR3 passing through the tumor 27 detected from the ultrasonic image 22 separately from the second puncture line NR2. The third puncture line NR3 is derived with the tumor 27 as a reference, regardless of the guide groove 29.

[0222] The processor 41 converts the coordinate system of the third puncture line NR3 into the coordinate system of the surgical field image 21 according to the correlation between the coordinate system of the ultrasonic image 22 and the coordinate system of the surgical field image 21, and derives the fourth puncture line NR4 which is the extension of the third puncture line NR3. Fig.37 As shown, the processor 41 displays the derived fourth puncture line NR4 in a superimposed manner on the operation field image 21. The fourth puncture line NR4 is an example of a "puncture path" involved in the technique of the present invention.

[0223] The medical staff ST moves the ultrasonic probe 14 according to the position of the fourth puncture line NR4, i.e., the first puncture line NR1 extending from the guide groove 29 of the ultrasonic probe 14 as a base point, while observing the surgical field image 21. The fourth puncture line NR4 is a line through which the puncture needle 18 punctures the target position, i.e., the tumor 27. Therefore, if the first puncture line NR1 and the fourth puncture line NR4 are aligned, the guide groove 29 of the ultrasonic probe 14 can be aligned to an appropriate position. In this state, the medical staff ST can insert the puncture needle 18 along the fourth puncture line NR4 (also the first puncture line NR1), thereby puncturing the target position, i.e., the tumor 27.

[0224] According to the fourth embodiment, the internal image acquired by a medical probe such as the ultrasonic probe 14 can be effectively used for positioning the ultrasonic probe 14 corresponding to the target position to be punctured by the puncture needle 18 .

[0225] [Fifth embodiment]

[0226] As an example, Figures 38 to 41 As shown in the fifth embodiment, the target puncture line NRT prepared in advance before the operation can be displayed superimposed on the operation field image 21. The target puncture line NRT is an example of the "puncture path" involved in the technique of the present invention.

[0227] (Creation of preoperative preparation information)

[0228] In the fifth embodiment, for example, before surgery, a preoperative simulation is performed using 3D (three-dimensional: 3Dimension) data of a virtual endoscope, and a target puncture line NRT of the puncture needle 18 is prepared in the preoperative simulation. The target puncture line NRT is an ideal puncture path that is selected to avoid blood vessels and other parts that must not be scratched inside the organ or to pass through a part that is easy for the puncture needle 18 to pass through when the puncture needle 18 punctures. In addition, the fifth embodiment is an example of using an ultrasonic probe 14 that can observe the internal structure of an organ as a medical device. Below, refer to Figures 38 to 41 Provide explanation.

[0229] First, refer to Fig.38 and Fig.39 , a method for creating a target puncture line NRT of a puncture needle 18 that punctures a liver LV in a preoperative simulation using 3D data of a virtual endoscope is described. In addition, the 3D data of the virtual endoscope is generated based on a group of tomographic images 132 that are previously captured by a tomographic imaging device 131 such as a CT (Computed Tomography) device and an MRI (Magnetic Resonance Imaging) device. When the tomographic imaging device 131 is a CT device, a CT value is acquired while rotating a radiation source and a radiation detector around the body axis of the patient PT. By scanning the radiation source and the radiation detector along the body axis direction of the patient PT, CT values ​​are acquired at each position in the body axis direction. The CT value is a radiation absorption value in the body of the patient PT. The tomographic imaging device 131 performs image reconstruction processing based on the CT values ​​acquired in each direction around the body axis, thereby generating a tomographic image 132A. Each tomographic image 132A is a two-dimensional image generated according to the slice thickness in the body axis direction, and the tomographic image group 132 is a collection of a plurality of tomographic images 132A corresponding to each position in the body axis direction. The tomographic image group 132 is output to an image database 133 such as a PACS (Picture Archiving and Communication System).

[0230] The information processing device (not shown) performs 3D modeling to numerically describe the three-dimensional shape of the patient PT body based on the tomographic image group 132 obtained by the tomographic imaging device 131, thereby generating a three-dimensional image 134 which is a collection of voxel data 134A. The voxel data 134A is a unit of pixel in a three-dimensional space, and has three-dimensional coordinate information and pixel value. The three-dimensional image 134 generated by 3D modeling is also called three-dimensional volume data, etc. In the tomographic image group 132, the pixel interval of each tomographic image 132A and the slice thickness of each tomographic image 132A may be different. In this case, as an example, in 3D modeling, by performing interpolation processing between adjacent tomographic images 132A, a three-dimensional image 134 having isotropic voxel data 134A with equal length in each three-dimensional direction is generated. Here, the three-dimensional image 134 generated based on the tomographic image group 132 is information created before surgery, and is therefore referred to as a preoperative 3D image 134 for convenience. The preoperative 3D image 134 is an example of 3D data of a virtual endoscope, and is an example of “a three-dimensional image of an organ acquired in advance before surgery” according to the technique of the present invention.

[0231] The preoperative 3D image 134 is an image that can reproduce the external shape of the patient PT's body, anatomical parts such as organs in the body, and their internal structures. Fig.38 and Fig.39The preoperative 3D image 134 shown shows data of the liver LV as an example of the anatomical part. The preoperative 3D image 134 also includes data of a vascular structure that can reproduce an example of the internal structure of the liver LV.

[0232] Furthermore, the preoperative 3D image 134 of this example is a color image, and each pixel value of red (R), green (G), and blue (B) is assigned as the pixel value of the voxel data 134A. In addition, the preoperative 3D image 134 may also be a monochrome image. For example, the pixel value of the voxel data 134A may be represented by only the brightness (Y) based on the CT value. Furthermore, the pixel value of the voxel data 134A may also be a value obtained by converting the CT value using a pre-set LUT (Look Up Table) or an arithmetic expression. Furthermore, the pixel value of the voxel data 134A may also be set to a color correspondingly associated with each specific part such as an organ or lesion determined in the preoperative 3D image 134. Furthermore, opacity is also set in the voxel data 134A. Opacity is data used for volume rendering. Rendering is a process of converting a part of the preoperative 3D image 134 into a two-dimensional projection image, and volume rendering is a rendering method that projects the internal information of the object contained in the preoperative 3D image 134 onto the projection image. By setting the opacity for each voxel data 134A, when performing volume rendering, it is possible to distinguish between expressions such as opaque projection, semi-transparent projection, and transparent projection on the projection image among the internal information used.

[0233] like Fig.39 As shown, the information 123 including the target puncture line NRT is information prepared in advance before the operation, and is therefore referred to as preoperative preparation information hereinafter. The preoperative preparation information 123 is generated based on the preoperative 3D image 134. The target puncture line NRT is assigned to the preoperative 3D image 134 in the surgical simulation performed before the operation using the preoperative 3D image 134. In addition to the target puncture line NRT, the preoperative preparation information 123 also includes a blood vessel structure 137 and a tumor 27 as the internal structure of the liver LV to be punctured. The preoperative preparation information 123 including such a target puncture line NRT and the internal structures such as the blood vessel structure 137 and the tumor 27 is extracted from the preoperative 3D image 134.

[0234] As described above, the preoperative preparation information 123 is information that specifies the three-dimensional position in the preoperative 3D image 134. The blood vessel structure 137 included in the preoperative preparation information 123 is used for positional alignment with the ultrasonic image 22 acquired by the ultrasonic probe 14.

[0235] (Acquisition of Ultrasonic 3D Images)

[0236] like Fig.40As shown, the processor 41 acquires an ultrasonic 3D (three-dimensional) image 151 based on an ultrasonic image group 22g, which is a collection of ultrasonic images 22 taken by the ultrasonic probe 14. The processor 41 uses the ultrasonic 3D image 151 acquired during surgery for positioning. The ultrasonic 3D image 151 is an example of an "internal image of an organ acquired by a medical probe" involved in the technology of the present invention.

[0237] like Fig.40 As shown, first, if the ultrasonic probe 14 is scanned along the surface of the target part such as the liver LV, a plurality of ultrasonic images 22 at various positions on the scanning trajectory S can be acquired. The processor 41 acquires the ultrasonic image group 22g, and performs 3D modeling based on the acquired ultrasonic image group 22g, thereby generating an ultrasonic 3D image 151. The ultrasonic image 22 and Fig.38 The tomographic image 132A shown is also a tomographic image, and the ultrasonic image group 22g is the same as the tomographic image group 132. The ultrasonic image group 22g also depicts the internal structure of the target site such as the blood vessel structure 137 of the liver LV.

[0238] Processor 41 executes Fig.38 The ultrasonic 3D image 151 is generated by the same process as the 3D modeling described in the previous section. Similar to the preoperative 3D image 134, the ultrasonic 3D image 151 is a collection of pixels in a three-dimensional space, namely, voxel data 151A. The voxel data 151A is also Fig.38 The voxel data 134A shown is the same, and three-dimensional coordinates, pixel values, and opacity are set.

[0239] like Fig.41 As shown, the processor 41 derives the first positional relationship information 158, which is the correlation between the coordinate system of the surgical field image 21 acquired by the endoscope 13 and the coordinate system of the ultrasonic 3D image 151, and the second positional relationship information 159, which is the correlation between the coordinate system of the ultrasonic 3D image 151 and the coordinate system of the preoperative 3D image 134.

[0240] First, the method for deriving the first positional relationship information 158 is described. The processor 41 estimates the position and posture of the ultrasonic transducer 14C based on the mark M of the ultrasonic probe 14 shown in the surgical field image 21 in the same manner as shown in the fourth embodiment. Furthermore, the processor 41 estimates the position and posture of the ultrasonic 3D image 151 in the surgical field image 21 based on the estimated position and posture of the ultrasonic transducer 14C. The position and posture of the ultrasonic 3D image 151 in the surgical field image 21 becomes the first positional relationship information 158. The first positional relationship information 158 is an example of "the correlation between the coordinate system of the internal image derived from the position and posture information and the coordinate system of the in-vivo image" involved in the technology of the present invention.

[0241] Next, the method for deriving the second positional relationship information 159 is described. The processor 41 compares the vascular structures 137 depicted in the ultrasonic 3D image 151 and the preoperative 3D image 134. Specifically, the ultrasonic 3D image 151 and the preoperative 3D image 134 are aligned in such a manner that the similar structures of the two vascular structures 137 are consistent with each other. The search for the similar structure of the vascular structure 137 can use an image analysis method using a rule base such as pattern matching, or a method using an AI technology such as semantic segmentation. The processor 41 can derive the second positional relationship information 159 by performing the alignment using the vascular structure 137.

[0242] Then, the processor 41 synthesizes the preoperative preparation information 123 including the target puncture line NRT generated based on the preoperative 3D image 134 and the surgical field image 21 according to the first positional relationship information 158 and the second positional relationship information 159. Since the vascular structure 137 of the liver LV is used for alignment, the target puncture line NRT overlaps the tumor 27 at an appropriate position. The first puncture line NR1 with the guide groove 29 as the base point is displayed in the surgical field image 21. In the fourth embodiment, as shown in FIG. Fig.37 As shown, the medical staff ST moves the ultrasonic probe 14 so that the first puncture line NR1 coincides with the target puncture line NRT. This allows the position of the ultrasonic probe 14 to be aligned with an appropriate position for puncturing the tumor 27 .

[0243] In this way, when the target puncture line NRT (an example of a puncture path) is set in the preoperative 3D image 134 (an example of a three-dimensional image of an organ) acquired in advance before the operation through the preoperative simulation, the processor 41 uses the second positional relationship information 159 and the first positional relationship information 158 to overlap the target puncture line NRT with the surgical field image 21. In this way, the internal image of the medical probe and the result of the preoperative simulation can be effectively used. In the preoperative simulation, for example, an appropriate target puncture line NRT that does not scratch the vascular structure 137 can be set. By providing such a target puncture line NRT in the surgical field image 21, more appropriate medical assistance can be performed.

[0244] The mark M may be in a form other than a grid-shaped mark as in this example, and may be a simple polygon or circle or a combination of these shapes. Furthermore, it may be a dot pattern arranged in a matrix at equal intervals.

[0245] Furthermore, in the above-mentioned embodiments, the puncture needle is described as an example of cauterization, but the puncture needle may also be a function other than cauterization. Furthermore, the treatment instrument is described as an example of the puncture needle, but other than the puncture needle, a treatment instrument for injecting a fluorescent agent such as ICG (Indocyanine Green), a biopsy needle for collecting tissue for biopsy, a forceps, etc. may also be used.

[0246] Furthermore, in the above-mentioned embodiments, the body cavity such as the abdominal cavity and the thoracic cavity is described as an example, but the body may also be the upper digestive tract such as the esophagus, the lower digestive tract such as the intestine, and the ducts such as the bronchi. When the technology of the present invention is applied to the surgical area in the duct, for example, a mark M is set at the base end of the flexible endoscope inserted into the duct.

[0247] Furthermore, in the above-described embodiments, the medical instrument having an insertion portion may be the trocar 17 in addition to the endoscope 13 and the medical probe (ultrasonic probe 14 or OCT probe).

[0248] Furthermore, based on the above description, the techniques described in the following supplementary notes can be mastered.

[0249] [Supplementary Item 1]

[0250] A medical assisting device includes a processor, wherein the processor performs the following processing:

[0251] Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of the subject, and including within a photographic range an organ punctured by the puncture needle, an insertion portion of a medical device inserted into the body, and a mark that enables image recognition and is provided at the insertion portion;

[0252] deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and

[0253] Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image specified based on the position and posture information.

[0254] [Supplementary Item 2]

[0255] The medical assisting device according to Supplementary Item 1, wherein:

[0256] The processor determines whether the changed position of the inner wall intersects the puncture path in the three-dimensional space inside the body based on the in-vivo image of the state in which the operation is performed from outside the body to change the inner wall inside the body and the puncture path is overlapped, that is, two in-vivo images taken from at least two different viewpoints, and notifies the determination result.

[0257] [Supplementary item 3]

[0258] The medical assisting device according to Supplementary Item 2, wherein:

[0259] When the changed position at which a change has occurred in both of the two in-vivo images intersects the superimposed and displayed puncture path two-dimensionally, it is determined that the changed position intersects the puncture path in three-dimensional space.

[0260] [Supplementary Item 4]

[0261] The medical assisting device according to Supplementary Item 2 or Supplementary Item 3, wherein:

[0262] When multiple operations are performed, the processor displays the history of each changed position of the multiple operations in an overlapping manner on the in-vivo image.

[0263] [Supplementary Item 5]

[0264] The medical assisting device according to Supplementary Item 4, wherein:

[0265] When the viewpoint of the in-vivo camera moves during multiple operations, the processor reflects the amount of viewpoint movement and displays a history of the changed position.

[0266] [Supplementary Item 6]

[0267] A medical assisting device according to any one of Supplementary Items 2 to 5, wherein:

[0268] The operation is to press the surface of the subject from outside the body.

[0269] The changed position is a raised portion where the inner wall is raised toward the inside of the body by a pressing operation.

[0270] [Supplementary Item 7]

[0271] The medical assisting device according to Supplementary Item 1, wherein:

[0272] The processor derives the intersection point between the inner wall in the body and the puncture path in the three-dimensional space based on the in-body depth information indicating the distribution of the depth, that is, the distance in the depth direction from the viewpoint of the in-body camera.

[0273] [Supplementary Item 8]

[0274] The medical assisting device according to supplementary item 7, wherein:

[0275] As an in-body camera, using a camera that can estimate depth,

[0276] The processor obtains the depth information.

[0277] [Supplementary Item 9]

[0278] The medical assisting device according to supplementary item 7, wherein:

[0279] The processor obtains depth information based on two in-vivo images taken from different viewpoints.

[0280] [Supplementary Item 10]

[0281] The medical assisting device according to supplementary item 7, wherein:

[0282] The processor acquires the depth information using a machine learning model that takes in-vivo images as input and outputs depth information.

[0283] [Supplementary Note 11]

[0284] A medical assisting device according to any one of supplementary items 7 to 10, wherein:

[0285] When an in-vivo image of a state in which an operation is performed from outside the body to change the inner wall of the body cavity of the subject is acquired,

[0286] The processor determines, based on the depth information, whether the changed position of the inner wall in the three-dimensional space in the body intersects the puncture path, and notifies the determination result.

[0287] [Supplementary Item 12]

[0288] A medical assisting device according to any one of supplementary items 1 to 11, wherein:

[0289] Medical instruments are medical probes that can observe the internal structure of organs.

[0290] [Supplementary Item 13]

[0291] The medical assisting device according to supplementary item 12, wherein:

[0292] The medical probe is an ultrasonic probe.

[0293] [Supplementary Item 14]

[0294] The medical assisting device according to Supplementary Item 12 or 13, wherein:

[0295] The medical probe is provided at the insertion portion and has a guide groove for guiding the puncture needle to be inserted into a target position inside an organ by engaging with the puncture needle.

[0296] The processor determines a position at which the puncture path is superimposed on the in-vivo image based on the relative positional relationship between the mark and the guide groove.

[0297] [Supplementary Item 15]

[0298] A medical assisting device according to any one of supplementary items 12 to 14, wherein:

[0299] The processor determines a position at which to superimpose the puncture path on the in-vivo image based on a target position inside the organ determined in the in-vivo image acquired by the medical probe and a correlation between a coordinate system of the in-vivo image and a coordinate system of the in-vivo image derived based on the position and posture information.

[0300] [Supplementary Item 16]

[0301] A medical assisting device according to any one of supplementary items 12 to 15, wherein:

[0302] When the puncture path is set by preoperative simulation using a three-dimensional image of the organ acquired before the operation,

[0303] The processor determines the position at which the puncture path is superimposed on the in-vivo image by using the correlation between the coordinate system of the internal image of the organ acquired by the medical probe and the coordinate system of the three-dimensional image and the correlation between the coordinate system of the internal image derived based on the position and posture information and the coordinate system of the in-vivo image.

[0304] [Supplementary Item 17]

[0305] A medical assisting device according to any one of supplementary items 1 to 16, wherein:

[0306] The processor changes the display of a portion of the puncture path.

[0307] [Supplementary Item 18]

[0308] The medical assisting device according to Supplementary Item 17, wherein:

[0309] A portion of the puncture path is a portion of the inner wall side of the body,

[0310] The processor highlights a portion of the inner wall side.

[0311] [Supplementary Item 19]

[0312] The medical assisting device according to Supplementary Item 17 or 18, wherein:

[0313] Part of the puncture path is part of the organ side,

[0314] The processor makes a portion of the organ side less visible than an inner wall side.

[0315] [Supplementary item 20]

[0316] A medical assisting device according to any one of supplementary items 17 to 19, wherein:

[0317] The processor detects an insertion length of an insertion portion of the medical instrument and determines a range to change a display based on the detected insertion length.

[0318] [Supplementary item 21]

[0319] A medical assisting device according to any one of supplementary items 17 to 20, wherein:

[0320] The processor determines the range to change the display according to the user's designation.

[0321] [Supplementary item 22]

[0322] A medical assisting device according to any one of supplementary items 1 to 21, wherein:

[0323] The processor is capable of displaying different multiple puncture paths.

[0324] [Supplementary item 23]

[0325] A working method of a medical assisting device, wherein the medical assisting device comprises a processor, and the processor performs the following processing:

[0326] Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of the subject, and including within a photographic range an organ punctured by the puncture needle, an insertion portion of a medical device inserted into the body, and a mark that enables image recognition and is provided at the insertion portion;

[0327] deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and

[0328] Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image specified based on the position and posture information.

[0329] [Supplementary item 24]

[0330] A working program of a medical assisting device, which enables a computer to function as a medical assisting device, wherein the working program enables the computer to execute the following steps:

[0331] Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of the subject, and including within a photographic range an organ punctured by the puncture needle, an insertion portion of a medical device inserted into the body, and a mark that enables image recognition and is provided at the insertion portion;

[0332] deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and

[0333] Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position specified in the in-vivo image based on the position and posture information.

[0334] [Supplementary item 25]

[0335] A medical assistance system comprising:

[0336] A medical device having an insertion portion inserted into a body of a subject, wherein a mark capable of recognizing an image is provided on the insertion portion;

[0337] an in-vivo camera that captures in-vivo images including the insertion portion, the marker, and the organ punctured by the puncture needle within a photographic range; and

[0338] A medical assisting device having a processor,

[0339] In the medical assistance system, the processor performs the following processing:

[0340] Acquire in vivo images;

[0341] deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and

[0342] Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image specified based on the position and posture information.

[0343] Furthermore, according to the above description, it is possible to grasp the technology described in the following supplementary item 2. The technology described in the supplementary item 2 is a technology related to a puncture method performed by a medical staff ST while visually recognizing the surgical field image 21 of the first viewpoint VP1 and the surgical field image 21 of the second viewpoint VP2 shown in the above-mentioned second embodiment.

[0344] [Supplementary Note 2 1]

[0345] A puncture method, wherein:

[0346] An operation for causing a change in the inner wall of the body is performed from outside the body while visually recognizing an in-vivo image, the in-vivo image being captured by an in-vivo camera capturing the inside of the body of the subject, the in-vivo image including, within a photographic range, an organ punctured by the puncture needle, an insertion portion of a medical device inserted into the body, and an image-recognizable marker provided at the insertion portion, and a puncture path of the puncture needle being displayed in a superimposed manner at a position determined based on position and posture information including at least one of a position and a posture of the insertion portion derived based on the marker,

[0347] In the state of performing the operation, it is determined whether the changed position of the inner wall in the three-dimensional space in the body intersects the puncture path based on the first in-vivo image captured at the first viewpoint and the second in-vivo image captured at the second viewpoint different from the first viewpoint,

[0348] If the determination result shows an intersection, the puncture needle is caused to puncture the intersection of the change position and the puncture path.

[0349] [Supplementary Note 2]

[0350] The puncture method according to Supplementary Item 2, wherein:

[0351] The operation includes two operations, namely, a first operation and a second operation performed at different timings on the same position.

[0352] performing a first determination, wherein the first in-vivo image is visually recognized while the first operation is being performed, and determining an intersection between the changed position in the first in-vivo image and the puncture path,

[0353] Then, a second determination is performed, wherein the second in-vivo image is visually recognized while the second operation is being performed, and an intersection between the changed position in the second in-vivo image and the puncture path is determined.

[0354] When it is determined as an intersection in both the first determination and the second determination, it is determined as an intersection in the three-dimensional space.

[0355] [Supplementary Note 2 Item 3]

[0356] The puncture method according to Supplementary Item 2, wherein:

[0357] A first determination and a second determination are performed, wherein the first determination is performed by visually recognizing a first in-vivo image while the operation is being performed, and determining whether the changed position in the first in-vivo image intersects with the puncture path; and the second determination is performed by visually recognizing a second in-vivo image while the operation is being performed, and determining whether the changed position in the second in-vivo image intersects with the puncture path.

[0358] When it is determined as an intersection in both the first determination and the second determination, it is determined as an intersection in the three-dimensional space.

[0359] Furthermore, in the above-mentioned embodiment, for example, as the hardware structure of the processor 41 that performs various processes such as image acquisition, position and posture information derivation, image synthesis, and display control, various processors (Processer) shown below can be used. As various processors, in addition to general-purpose processors that execute software (programs) and function as various processing units, namely CPUs, there are also processors such as FPGAs (Field-Programmable Gate Arrays) and PLDs (Programmable Logic Devices) whose circuit structures can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits) and other processors having circuit structures specially designed to perform specific processes, namely dedicated circuits.

[0360] Furthermore, the various processes described above may be performed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and a CPU and an FPGA, etc.). Furthermore, multiple processing units may be constituted by one processor. As an example of a plurality of processing units being constituted by one processor, there is a method of using a processor that implements the functions of the entire system including multiple processing units by one IC (Integrated Circuit) chip, such as a system on a chip (System 0n Chip: SOC).

[0361] In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware configuration.

[0362] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit (Circuitry) formed by combining circuit elements such as semiconductor elements can be used.

[0363] Furthermore, the technology of the present invention relates to a storage medium (USB memory or DVD (Digital Versatile Disc)-ROM (Read Only Memory), etc.) in addition to the working program of the medical assistance device, which can be read by a computer that non-temporarily stores the working program of the medical assistance device.

[0364] The records and illustrations shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only examples of the technology of the present invention. For example, the description of the above-mentioned structure, function, action and effect is a description of an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, it is of course possible to delete unnecessary parts, or add or replace new elements in the records and illustrations shown above without departing from the main purpose of the technology of the present invention. In addition, in order to avoid complexity and facilitate understanding of the parts involved in the technology of the present invention, in the records and illustrations shown above, descriptions related to technical common sense that does not need to be specifically explained are omitted on the basis of being able to implement the technology of the present invention.

[0365] In this specification, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Moreover, in this specification, when three or more situations are expressed in conjunction with "and / or", the same concept as "A and / or B" also applies.

[0366] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A medical assisting device comprising a processor, wherein: The processor performs the following processing: Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of a subject, and including within a photographic range an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the subject's body, and a mark that enables image recognition and is disposed on the insertion portion; deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image determined based on the position and posture information.

2. The medical assisting device according to claim 1, wherein: The processor determines whether the changed position of the inner wall intersects the puncture path in the three-dimensional space within the body based on the in-vivo image, i.e., two in-vivo images taken from at least two different viewpoints, in which an operation is performed from outside the body to change the inner wall within the body and the state of the puncture path is overlapped and displayed, and notifies the determination result.

3. The medical assisting device according to claim 2, wherein: When the changed position where the change occurs in both of the two in-vivo images intersects the puncture path displayed in a superimposed manner in two dimensions, it is determined that the changed position intersects the puncture path in the three-dimensional space.

4. The medical assisting device according to claim 2, wherein: When the operation is performed a plurality of times, the processor displays a history of the changed positions of each of the plurality of operations in a superimposed manner on the in-vivo image.

5. The medical assisting device according to claim 4, wherein: In a case where the viewpoint of the in-vivo camera moves during a plurality of the operations, the processor reflects the amount of movement of the viewpoint and displays a history of the changed position.

6. The medical assisting device according to claim 2, wherein: The operation is a pressing operation of pressing the body surface of the subject from outside the body, The changed position is a raised portion caused by the pressing operation to cause the inner wall to swell toward the inside of the body.

7. The medical assisting device according to claim 1, wherein: The processor derives an intersection point between the inner wall of the body and the puncture path in a three-dimensional space based on the depth information in the body indicating the distribution of depths, that is, the distance in the depth direction from the viewpoint of the in-body camera.

8. The medical assisting device according to claim 7, wherein: As the in-vivo camera, a camera capable of estimating the depth is used, The processor obtains the depth information.

9. The medical assisting device according to claim 7, wherein: The processor acquires the depth information based on the two in-vivo images captured at different viewpoints.

10. The medical assisting device according to claim 7, wherein: The processor obtains the depth information using a machine learning model that takes the in-vivo image as input and outputs the depth information.

11. The medical assisting device according to claim 7, wherein: When the in-vivo image is acquired in a state where an operation is performed from outside the body to change the inner wall of the body cavity of the subject, The processor determines, based on the depth information, whether a changed position of the inner wall in the three-dimensional space within the body intersects the puncture path, and notifies the determination result.

12. The medical assisting device according to claim 1, wherein: The medical instrument is a medical probe capable of observing the internal structure of the organ.

13. The medical assisting device according to claim 12, wherein: The medical probe is an ultrasonic probe.

14. The medical assisting device according to claim 12, wherein: The medical probe is disposed in the insertion portion and has a guide groove for guiding the puncture needle to be inserted into a target position inside the organ by engaging with the puncture needle. The processor determines a position at which the puncture path is superimposed on the in-vivo image based on a relative positional relationship between the mark and the guide groove.

15. The medical assisting device according to claim 12, wherein: The processor determines a position at which the puncture path is superimposed on the in-vivo image based on a target position inside the organ determined in the internal image of the organ acquired by the medical probe and a correlation between a coordinate system of the internal image derived based on the position and posture information and a coordinate system of the in-vivo image.

16. The medical assisting device according to claim 12, wherein: When the puncture path is set by a preoperative simulation in a three-dimensional image of the organ acquired before the operation, The processor determines the position at which the puncture path is superimposed on the in-vivo image by using the correlation between the coordinate system of the internal image of the organ acquired by the medical probe and the coordinate system of the three-dimensional image and the correlation between the coordinate system of the internal image derived based on the position and posture information and the coordinate system of the in-vivo image.

17. The medical assisting device according to claim 1, wherein: The processor changes the display of a portion of the puncture path.

18. The medical assisting device according to claim 17, wherein: A portion of the puncture path is a portion of the inner wall side of the body, The processor highlights and displays a portion of the inner wall side.

19. The medical assisting device according to claim 17, wherein: A portion of the puncture path is a portion of the organ side, The processor makes the visibility of a portion of the organ side lower than that of the inner wall side.

20. The medical assisting device according to claim 17, wherein: The processor detects an insertion length of the insertion portion of the medical instrument, and determines to change a range of the display based on the detected insertion length.

21. The medical assisting device according to claim 17, wherein: The processor determines a range to change the display according to a user's designation.

22. The medical assisting device according to claim 1, wherein: The processor is capable of displaying different multiple puncture paths.

23. A method for operating a medical assisting device, wherein: The medical assisting device includes a processor, and the processor performs the following processing: Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of a subject, and including within a photographic range an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the subject's body, and a mark that enables image recognition and is disposed on the insertion portion; deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image determined based on the position and posture information.

24. A working program product for a medical assisting device, the working program product causing a computer to function as a medical assisting device, the working program product causing the computer to execute the following processing: Acquiring an in-vivo image, the in-vivo image being captured by an in-vivo camera that captures the in-vivo of a subject, and including within a photographic range an organ punctured by a puncture needle, an insertion portion of a medical device inserted into the subject's body, and a mark that enables image recognition and is disposed on the insertion portion; deriving position and posture information including at least one of a position and a posture of the insertion portion based on the marker; and Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image specified based on the position and posture information.

25. A medical assistance system comprising: A medical device having an insertion portion inserted into a body of a subject, wherein a mark capable of recognizing an image is provided on the insertion portion; an in-vivo camera that captures an in-vivo image including the insertion portion, the marker, and an organ punctured by the puncture needle within a photographic range; and A medical assisting device having a processor, In the medical assistance system, the processor performs the following processing: acquiring the in-vivo image; deriving position and posture information including at least one of the position and posture of the insertion portion based on the marker; and Display control is performed to display the puncture path of the puncture needle in a superimposed manner at a position in the in-vivo image determined based on the position and posture information.

Images