Medical assistance device, medical assistance method, and medical assistance program

Abstract

This medical assistance device comprises a processor. The processor controls live view display of each of a surgical field image which is obtained by optically capturing, by means of a camera, an image of a surgical field including a subject part of a body and a medical instrument inserted into the body and an internal image which shows the internal structure of the subject part. Upon receiving an instruction to record the surgical field image as a still image during the live view display, the processor also records a still image of the internal image at the timing of the instruction in addition to the still image of the surgical field image.

Description

Medical support device, medical support method, and medical support program

[0001] The present disclosure relates to a medical support device, a medical support method, and a medical support program.

[0002] Conventionally, there is known a technique for identifying the position and orientation of a medical instrument inserted into a body and supporting medical treatment such as surgery and examination based on the identified position and orientation of the medical instrument. For example, International Publication No. 2023 / 162657 discloses a technique for estimating the position and orientation of an ultrasound probe by performing image analysis on an optically captured surgical field image including an ultrasound probe, and for displaying preparation information corresponding to the estimated position and orientation superimposed on the surgical field image.

[0003] In laparoscopic surgery, the internal structure is grasped by observing the surgical field image optically captured by the endoscope and the ultrasound image while adjusting the position and orientation of the ultrasound probe in the surgical field. However, because the surgical field image and the ultrasound image have different viewpoints, imaging ranges, and drawing methods, it is difficult to appropriately adjust the position and orientation of the ultrasound probe by comparing them, and this has required advanced skills.

[0004] The present disclosure provides a medical support device, a medical support method, and a medical support program that can further improve convenience.

[0005] A first aspect of the present disclosure is a medical support device comprising a processor that controls live view display of a surgical field image optically captured by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body, and an internal image showing the internal structure of the target area, and when an instruction to record the surgical field image as a still image is received during the live view display, the processor records a still image of the internal image at the time of the instruction in addition to the still image of the surgical field image.

[0006] The processor may perform control so that a still image of the internal image at the timing is displayed on the screen that displays the surgical field image in live view.

[0007] The processor may derive position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, and may control the display of the position indicated by the position and orientation information at the timing on the surgical field image being displayed in live view.

[0008] When the processor receives instructions for multiple timings, it may record still images of the internal image at each of the multiple timings, and may control the display so that the correspondence between the still images of the internal image and the positions indicated by the position and orientation information is identifiable for each timing.

[0009] The processor may perform control so that the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information is displayed in an identifiable manner using at least one of text and graphics.

[0010] The processor may perform control so that the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information is displayed in a distinguishable manner by changing the display color.

[0011] The processor may derive position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, and record a superimposed image in which a still image of the internal image at the timing is superimposed on a still image of the surgical field image at the timing, and the display manner of the still image of the internal image is adjusted based on the position and orientation information at the timing.

[0012] The processor may perform control so that the superimposed image is displayed on a screen that displays a live view of the surgical field image.

[0013] When the processor receives instructions for multiple timings, it may record superimposed images at each of the multiple timings and control the display of the superimposed images at each of the multiple timings side by side on a screen that displays a live view of the surgical field image.

[0014] The processor may control the display of the position indicated by the position and orientation information at the timing on the surgical field image being displayed in live view, and may further control the display of the correspondence between the superimposed image and the position indicated by the position and orientation information in a distinguishable manner for each timing.

[0015] The processor may perform control so that the correspondence between the superimposed image at each timing and the position indicated by the position and orientation information is displayed in an identifiable manner using at least one of text and graphics.

[0016] The processor may perform control so that the correspondence between the superimposed image at each timing and the position indicated by the position and orientation information is displayed in a distinguishable manner by changing the display color.

[0017] The processor may perform control based on the position and orientation information to display a scale indicating the size of the internal structure included in the internal image on the superimposed image.

[0018] The processor may perform control based on the position and orientation information so that the puncture path of a puncture needle inserted into the body using a medical instrument is displayed on the superimposed image.

[0019] Markers configured with optically detectable patterns may be provided on the outer peripheral surface of the medical instrument, and the processor may derive the position and orientation information based on the markers included in the surgical field image.

[0020] The medical instrument may be an ultrasound probe that transmits ultrasound waves to a target area and detects electrical signals corresponding to ultrasound echoes reflected from the target area, and the internal image may be an ultrasound image generated in response to the electrical signals.

[0021] The processor may use an image obtained by removing blank areas in the ultrasound image that do not show the internal structure of the target region as the internal image.

[0022] The processor may derive position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, acquire a three-dimensional image of the target area, and extract an internal image from the three-dimensional image based on the position and orientation information.

[0023] A second aspect of the present disclosure is a medical support method in which a computer controls live view display of a surgical field image obtained by optically photographing a surgical field including a target area inside the body and a medical instrument to be inserted into the body using a camera, and an internal image showing the internal structure of the target area, and when an instruction to record the surgical field image as a still image is received during the live view display, the computer executes a process to record a still image of the internal image at the time of the instruction in addition to the still image of the surgical field image.

[0024] A third aspect of the present disclosure is a medical support program that controls live view display of a surgical field image optically captured by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body, and an internal image showing the internal structure of the target area, and when an instruction to record the surgical field image as a still image is received during the live view display, causes a computer to execute processing to record a still image of the internal image at the time of the instruction in addition to the still image of the surgical field image.

[0025] According to the above aspects, the medical support device, medical support method, and medical support program of the present disclosure can further improve convenience.

[0026] 1 is a diagram illustrating an overview of a medical support system including a medical support device. FIG. 1 is a diagram illustrating the state inside the body during laparoscopic surgery. FIG. 2 is a diagram illustrating the insertion state of a puncture needle guided by a guide groove. FIG. 2 is a diagram illustrating an example of the hardware configuration of a medical support device. FIG. 3 is a diagram illustrating an example of the functional configuration of a medical support device. FIG. 4 is a diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. FIG. 5 is a diagram illustrating an example of a superimposed image. FIG. 6 is a diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. FIG. 7 is a diagram illustrating an example of a superimposed image. FIG. 8 is an example of a screen displayed on a display. FIG. 9 is an example of a screen displayed on a display. FIG. 10 is an example of a screen displayed on a display. FIG. 11 is a flowchart illustrating an example of medical support processing. FIG. 12 is a diagram illustrating an example of a scale. FIG. 13 is a diagram illustrating an example of a puncture path.

[0027] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are designated by the same reference numerals, and duplicate descriptions will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0028] An example of a medical support system 10 incorporating a medical support device 11 according to this embodiment will be described with reference to FIGS. 1 to 3 . As an example, the medical support system 10 is used when performing endoscopic surgery on a patient PT using an endoscope 13. Unlike open surgery, endoscopic surgery involves drilling a small hole in the patient PT's body and inserting a medical instrument such as the endoscope 13 through the hole. The medical support system 10 provides medical staff ST, including physicians, with a view of the surgical field within the patient PT's body, as well as support information to support medical treatment such as surgery and examinations. The support information, as described below, may be a superimposed image 26 for displaying an ultrasound image 22 superimposed on a surgical field image 21. Because this medical support system 10 has the function of providing support information in real time during surgery, it is also referred to as a surgical navigation system.

[0029] 1 , the medical support system 10 includes a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is connected to the endoscope 13, the ultrasound probe 14, and the display 16 so as to be able to communicate with each other.

[0030] 2 shows the state in which an endoscope 13 and an ultrasonic probe 14 are being inserted into the abdomen of a patient PT. In endoscopic surgery, a portion of the endoscope 13 and the ultrasonic probe 14, including their respective distal ends, is inserted into the body via a trocar 17. The trocar 17 is an insertion tool that has an insertion hole through which the endoscope 13 and the like are inserted and a valve that is provided within the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and therefore the trocar 17 is used to insert the endoscope 13, the ultrasonic probe 14, and the like into the body.

[0031] The puncture needle 18 is a treatment tool used to puncture a lesion, such as a tumor, contained in an organ. Specifically, the puncture needle 18 has a needle portion 18A and a grip portion 18B provided on the proximal end of the needle portion 18A. The puncture needle 18 is, for example, an ablation puncture needle used to cauterize a lesion. The ablation puncture needle has an electrode to which a high-frequency voltage is applied at its tip. When the high-frequency voltage is applied while the electrode is inserted into the lesion, the heat generated by the electrode necrotizes the lesion. In this embodiment, as an example, a tumor 27 in the liver LV is visualized using an ultrasound image 22, and the visualized tumor 27 is cauterized with the puncture needle 18, thereby performing a treatment to necrotize the tumor 27.

[0032] The endoscope 13 optically captures an operative field SF, including a target site (the liver LV in this example) inside the body of the patient PT, using a camera 13B. The operative field SF is a space extending inside a body cavity defined by organs and the body wall. Specifically, the endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT. The tip of the insertion section 13A is equipped with a camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)). As an example, the insertion section 13A of the endoscope 13 is a rigid endoscope, and is also called a laparoscope because it is often used for observing the abdominal cavity.

[0033] The camera 13B includes an image sensor, such as a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images. The endoscope 13 is connected to an endoscope image processor (not shown). The image processor processes the image signal output by the image sensor to generate an operative field image 21 of the operative field SF within the body. The operative field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via the endoscope image processor. In FIG. 2, the symbols Xin and Yin indicate the coordinate system of the operative field image 21. The camera 13B is an example of a "camera" in the present disclosure.

[0034] The illumination light for the endoscope 13 is, for example, visible light such as white light. Alternatively, special light such as ultraviolet light or infrared light may be used as the illumination light for the endoscope 13. The special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet range. The surgical field image 21 is an image of the surgical field SF illuminated by the illumination light, and more specifically, an image based on light reflected from the surface of the surgical field SF. Therefore, while the surgical field image 21 can depict structures present near the surface of the target area, it is difficult to observe the internal structure.

[0035] The ultrasonic probe 14 transmits ultrasonic waves to a target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area. Specifically, the ultrasonic probe 14 has an insertion section 14A that is inserted into the body of the patient PT and an operation section 14D on the proximal end side of the insertion section 14A. An ultrasonic transducer 14C is built into a tip section 14B of the insertion section 14A. The ultrasonic probe 14 is an example of a "medical instrument" of the present disclosure.

[0036] The ultrasonic transducer 14C transmits ultrasonic waves to the target region and receives ultrasonic echoes reflected from the target region. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown). This image processing processor performs image reconstruction processing based on electrical signals corresponding to the ultrasonic echoes received by the ultrasonic transducer 14C. The image reconstruction processing generates an ultrasonic image 22 showing the internal structure of the target region scanned by the ultrasonic probe 14. The ultrasonic image 22 captured by the ultrasonic probe 14 is transmitted to the medical support device 11 in real time via the image processing processor for the ultrasonic probe. In FIG. 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0037] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and acquires a fan-shaped ultrasonic image 22 centered on the ultrasonic transducer 14C. By scanning the ultrasonic probe 14, multiple ultrasonic images 22 are captured along the scanning direction. The ultrasonic images 22 are so-called B (brightness) mode images that visualize the internal structure of the target area, from the superficial layer to the deep layer where the ultrasonic waves reach, as brightness information. The ultrasonic images 22 visualize the internal structure of the target area that cannot be observed in the surgical field image 21 obtained by optical imaging.

[0038] A guide groove 29 is provided in the distal end portion 14B of the insertion section 14A. The guide groove 29 is a groove that engages with the puncture needle 18 to guide insertion of the puncture needle 18 to a target position inside an organ. FIG. 3 schematically shows the puncture needle 18 inserted into the body puncturing a tumor 27 in the liver LV while being guided by the guide groove 29. The guide groove 29 is provided, for example, on the proximal side of the distal end portion 14B relative to the ultrasonic transducer 14C, and is inclined at an angle θ with respect to the direction of the axis AX of the distal end portion 14B. The first guide groove 29 is inclined backward so that the tip of the puncture needle 18 inserted from the proximal side of the distal end portion 14B faces the distal end side of the distal end portion 14B.

[0039] The puncture needle 18 is inserted while checking the tumor 27 using the ultrasound image 22. Because the area visualized by the ultrasound image 22 is a radial area originating from the ultrasound transducer 14C, the inclined guide grooves 29 make it possible to point the tip of the puncture needle 18 toward the tumor 27 visualized by the ultrasound image 22. Note that the insertion section 14A may be provided with multiple guide grooves 29 that are angled differently relative to the direction of the axis AX. In this case, the medical staff ST can select an appropriate guide groove 29 to insert the puncture needle 18.

[0040] Furthermore, a marker M is attached to the outer peripheral surface of the tip 14B of the insertion section 14A. The marker M is a marker that can be recognized from the operative field image 21 optically captured by the camera 13B of the endoscope 13, that is, an optically detectable marker. The marker M is used by the medical support device 11 to estimate the position and orientation of the ultrasound probe 14, more specifically, the position and orientation of the tip 14B of the insertion section 14A in the operative field SF. A method of estimating the position and orientation using the marker M will be described later.

[0041] As an example, the marker M is a marker of a grid pattern 62 that is configured by a first line extending in the direction of the axis AX of the tip portion 14B of the ultrasonic probe 14 and a second line that is perpendicular to the axial direction of the tip portion 14B and formed in the circumferential direction along the outer circumferential surface of the tip portion 14B. Furthermore, the intersections in the grid pattern 62 are given circular symbols 64 or rectangular symbols 66, respectively.

[0042] The marker M only needs to include at least the symbols 64, 66 arranged at each intersection of the grid pattern 62, and does not need to include the line segments representing the first and second lines as shown in Fig. 3 etc. Furthermore, the symbols 64, 66 in the marker M are not limited to circles and rectangles and may have any shape, and may be represented by, for example, figures such as triangles, polygons, stars, and various marks, or may be represented by letters etc. Furthermore, in each drawing, the illustration of the marker M may be simplified or omitted to avoid cluttering the drawing.

[0043] The medical support device 11 acquires a surgical field image 21 from the endoscope 13 and acquires an ultrasound image 22 from the ultrasound probe 14. The medical support device 11 also generates a superimposed image 26 by superimposing the ultrasound image 22 on the surgical field image 21, and controls the display 16 to display the superimposed image 26. The superimposed image 26 displayed on the display 16 provides the medical staff ST with the field of view of the surgical field SF inside the body of the patient PT and the internal structure of the target site (liver LV).

[0044] Next, the configuration of the medical support device 11 will be described. Fig. 4 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a display 16, a processor 41, a RAM (Random Access Memory) 42, a storage 43, a reception device 46, a communication I / F (Interface) 47, and an external I / F 48. Each of these components is connected to a bus 49, such as a system bus and a control bus, and can communicate with each other.

[0045] The display 16 displays a screen D containing various information, in addition to the surgical field image 21, the ultrasound image 22, and the superimposed image 26. Examples of the display 16 include a liquid crystal display and an EL (Electro-Luminescence) display. The number of displays 16 may be at least one as shown in FIG. 1, but may also be multiple.

[0046] The processor 41 is, for example, a CPU (Central Processing Unit), which comprehensively controls each part of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0047] The RAM 42 is a memory that temporarily stores information and is used as a work memory by the processor 41. Examples of the RAM 42 include a dynamic random access memory (DRAM) and a static random access memory (SRAM).

[0048] The storage 43 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 43 include a hard disk drive (HDD) and a solid state drive (SSD). The storage 43 stores a medical support program 44 that causes the computer to function as the medical support device 11.

[0049] The storage 43 also stores dimension information 45. The dimension information 45 includes dimensions of the ultrasonic probe 14, specifically, information representing the relative positional relationships of the marker M, ultrasonic transducer 14C, guide groove 29, and the like in the ultrasonic probe 14. The positional relationship of the marker M is, for example, information on the positions and orientations of the reference symbols 64 and 66 constituting the marker M relative to the axial and circumferential directions of the tip portion 14B. The positional relationship of the ultrasonic transducer 14C is represented by, for example, the linear distance between a reference point of the tip portion 14B and the ultrasonic transducer 14C, and the inclination angle of the ultrasonic transducer 14C relative to the axis AX of the tip portion 14B. The positional relationship of the guide groove 29 is represented by, for example, the linear distance between a reference point of the tip portion 14B and the guide groove 29, and the inclination angle θ of the guide groove 29 relative to the axis AX of the tip portion 14B.

[0050] The reception device 46 has a keyboard, a mouse, etc. (not shown) and receives instructions from an operator. That is, the medical support apparatus 11 is operated by an operator such as a medical staff member ST through the reception device 46. The reception device 46 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0051] The communication I / F 47 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control in accordance with communication protocols defined by various wired or wireless communication standards.

[0052] The external I / F 48 is, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer, a memory card, etc. The medical support device 11 may be, for example, a server computer, a personal computer, a smartphone, a tablet terminal, a wearable terminal, or the like, as appropriate.

[0053] Next, the functional configuration of the medical support device 11 will be described. Fig. 5 is a block diagram showing an example of the functional configuration of the medical support device 11. The medical support device 11 includes an acquisition unit 50, a derivation unit 52, a recording control unit 54, and a display control unit 56. The processor 41 reads the medical support program 44 from the storage 43 and executes the medical support program 44 on the RAM 42, causing the processor 41 to function as the functional units of the acquisition unit 50, the derivation unit 52, the recording control unit 54, and the display control unit 56. The processor 41 operates as the functional units of the acquisition unit 50, the derivation unit 52, the recording control unit 54, and the display control unit 56, thereby realizing medical support processing.

[0054] The acquisition unit 50 acquires a surgical field image 21 obtained by optically capturing an operative field SF including a target site within the body and a medical instrument inserted into the body using the camera 13B. For example, the acquisition unit 50 acquires the surgical field image 21 from a device including a processor of the endoscope 13 via the external I / F 48 or the communication I / F 47. As shown in FIG. 2 , when the ultrasound probe 14 is inserted into the surgical field SF, the ultrasound probe 14, more specifically, the tip 14B of the insertion section 14A, appears in the surgical field image 21.

[0055] The acquisition unit 50 also acquires an internal image showing the internal structure of the target area. For example, the acquisition unit 50 acquires an ultrasound image 22 from a device including a processor of the ultrasound probe 14 via the external I / F 48 or the communication I / F 47. The ultrasound image 22 is an example of an "internal image" in the present disclosure. Note that the processor of the endoscope 13 and / or the processor of the ultrasound probe 14 may be included in the medical support device 11.

[0056] The derivation unit 52 derives position and orientation information indicating the position and orientation of a medical instrument (e.g., the ultrasound probe 14) in the surgical field SF based on the surgical field image 21. Specifically, the derivation unit 52 derives the position and orientation information based on a marker M included in the surgical field image 21. For example, the derivation unit 52 may detect the marker M by searching for morphological features of the marker M, such as the reference numerals 64 and 66, in the surgical field image 21. For example, the derivation unit 52 may detect the marker M using an image processing method such as pattern matching.

[0057] Furthermore, for example, instead of a rule-based method such as pattern matching, an AI (Artificial Intelligence) technique using a machine learning model may be used to detect the marker M. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) that is trained in advance to input the surgical field image 21 and output the area of ​​the marker M in the input surgical field image 21 can be applied.

[0058] The display control unit 56 controls the display of the superimposed image 26, which is obtained by superimposing an internal image (e.g., an ultrasound image 22) on the operative field image 21, and which adjusts the display mode of the internal image based on the position and orientation information. As an example, the display control unit 56 generates the superimposed image 26, which indicates the imaging range (e.g., position, orientation, and size) of the ultrasound image 22 in the operative field SF, by superimposing the ultrasound image 22 on an area on the operative field image 21 that corresponds to the imaging range of the ultrasound image 22.

[0059] Specific examples of the superimposed image 26 will be described with reference to Figures 6 to 9. Figures 6 and 8 are diagrams conceptually showing the position and orientation of the tip 14B of the ultrasonic probe 14 within the surgical field SF defined as a three-dimensional space, each assuming a different position and orientation. Figures 7 and 9 show examples of the superimposed image 26 generated according to the position and orientation of the ultrasonic probe 14 within the surgical field SF. Figure 7 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 6. Figure 9 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 8.

[0060] In the three-dimensional operative field SF shown in Figures 6 and 8, the Z axis is parallel to the imaging optical axis of the camera 13B of the endoscope 13. Furthermore, in the three-dimensional operative field SF, the X-Y plane is parallel to the imaging plane of the camera 13B and is perpendicular to the imaging optical axis. In other words, the X-Y plane of the three-dimensional operative field SF is parallel to the screen (Xin-Yin plane) of the operative field image 21. The operative field image 21 is a projected image of the operative field SF projected from one viewpoint. Note that, in Figures 6 and 8, of the reference symbols 64 and 66 that constitute the marker M, those that appear in the operative field image 21 are indicated in dark colors, and those that do not appear in the operative field image 21 are indicated in light colors.

[0061] 6 shows a state in which the axis AX of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13B (more specifically, the axis AX of the tip 14B is parallel to the X-axis) within the surgical field SF in three-dimensional space. In this case, as shown in FIG. 7, the marker M reflected in the surgical field image 21 has orthogonal lines of the grid pattern 62 parallel to the X-axis and Y-axis, respectively. Furthermore, the reference numerals 64 and 66 appear at equal intervals in the surgical field image 21.

[0062] On the other hand, Fig. 8 shows a state in which, within the surgical field SF in three-dimensional space, the direction of the axis AX of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13B, but is tilted in the depth direction parallel to the imaging optical axis. The attitude shown in Fig. 8 is a state in which the axis AX of the tip 14B is rotated approximately -25 degrees around the Y axis from the attitude shown in Fig. 6. In this case, as shown in Fig. 9, the marker M appearing in the surgical field image 21 appears such that the circumferentially extending lines of the grid pattern 62 become shorter and the interval between the reference symbols 64 and 66 becomes shorter as the marker M becomes farther from the camera 13B in the depth direction.

[0063] In this way, the shape of the marker M that appears in the operative field image 21 changes depending on the posture of the tip portion 14B. The derivation unit 52 estimates the posture of the tip portion 14B of the ultrasound probe 14 in the operative field SF based on the posture of the marker M in the operative field image 21. Specifically, the derivation unit 52 detects the direction of the axis AX of the tip portion 14B in the operative field SF as the posture of the tip portion 14B.

[0064] Furthermore, if the position of the tip portion 14B changes within the operative field SF, the position of the marker M reflected in the operative field image 21 also changes. The derivation unit 52 estimates the position of the tip portion 14B within the operative field SF based on the position of the marker M. The position of the tip portion 14B is detected as the position of a reference point of the tip portion 14B, which is provided, for example, at the tip position of the tip portion 14B. Furthermore, the shooting distance from the camera 13B to the marker M within the operative field SF (i.e., the distance in the Z-axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the operative field image 21. The derivation unit 52 derives the position coordinates of the reference point of the tip portion 14B within the operative field SF based on the shooting distance and dimension information 45 of the tip portion 14B, which includes the known dimensions of the marker M.

[0065] 6, the position of the tip 14B of the ultrasonic probe 14 is estimated as information such as the position coordinates (X01, Y01, Z01) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as information that it is parallel to the XY plane and the XZ plane and perpendicular to the YZ plane.

[0066] 8, the position of the tip portion 14B of the ultrasound probe 14 is estimated as information such as the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B within the surgical field SF. The orientation of the tip portion 14B is defined as the direction of the axis AX of the tip portion 14B within the surgical field SF. For example, the axis AX of the tip portion 14B is estimated as information such as -25° with respect to the XY plane, parallel to the XZ plane, and 65° with respect to the YZ plane.

[0067] When the position and orientation of the tip portion 14B in the surgical field SF is estimated by the derivation unit 52, the display control unit 56 estimates the position and orientation of the ultrasonic transducer 14C in the surgical field SF based on the estimated position and orientation of the tip portion 14B and the dimension information 45. As described above, the linear distance between the reference point of the tip portion 14B and the ultrasonic transducer 14C, and the tilt angle of the ultrasonic transducer 14C with respect to the direction of the axis AX of the tip portion 14B are known information defined as the dimension information 45. By using this dimension information 45, the position and orientation of the ultrasonic transducer 14C in the surgical field SF can be estimated based on the position and orientation of the tip portion 14B in the surgical field SF.

[0068] 6, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X11, Y11, Z11) obtained by correcting the position coordinates (X01, Y01, Z01) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The attitude of the ultrasonic transducer 14C is defined as the attitude of the tip portion 14B with respect to the axis AX, and the tilt angle is known as the dimension information 45.

[0069] 8, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X12, Y12, Z12) obtained by correcting the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The orientation of the ultrasonic transducer 14C is defined as the orientation of the tip portion 14B with respect to the axis AX, and this becomes a tilt angle known as the dimension information 45.

[0070] The display control unit 56 also estimates the imaging range of the ultrasound image 22 in the surgical field SF based on the estimated position and orientation of the ultrasound transducer 14C. The imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer 14C. Therefore, if the position and orientation of the ultrasound transducer 14C in the surgical field SF are known, the imaging range of the ultrasound image 22 in the surgical field SF can also be estimated. For example, the imaging range of the ultrasound image 22 captured by the convex-type ultrasound transducer 14C is a sector-shaped range that spreads radially from the ultrasound transducer 14C as the base point. Figures 6 and 8 illustrate an imaging range 22R of the ultrasound image 22 in the surgical field SF.

[0071] The display controller 56 then superimposes the ultrasound image 22 on a region of the operative field image 21 (Xin-Yin plane) that corresponds to the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF, thereby generating a superimposed image 26. In the examples of Figures 6 and 7, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). On the other hand, in the examples of Figures 8 and 9, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is not parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). Taking into account the difference in coordinate systems between the operative field image 21 and the ultrasound image 22, the display control unit 56 performs transformations such as projective transformation, affine transformation, movement, rotation, enlargement, and reduction on the ultrasound image 22, and then generates a superimposed image 26 that is superimposed on the operative field image 21.

[0072] The superimposed image 26 makes it easier to grasp the position and orientation of the ultrasonic transducer 14C even if it is difficult to visually recognize the position and orientation of the ultrasonic transducer 14C in the surgical field image 21 due to, for example, obstruction by an organ or the positional relationship with the camera 13B. Therefore, it becomes easier to adjust the position and orientation of the ultrasonic probe 14 so that a desired area (e.g., a tumor 27) is captured in the ultrasonic image 22, which contributes to improving the operability of the ultrasonic probe 14.

[0073] As shown in FIG. 2 , a margin region 22B may be added to the ultrasound image 22 to make the overall shape of the ultrasound image 22 rectangular. In this case, the display control unit 56 preferably generates the superimposed image 26 using the image from which the margin region 22B has been removed as an internal image. As described above, when the ultrasound probe 14 is a convex type, the ultrasound image 22 is fan-shaped. The margin region 22B is used to convert this fan-shaped ultrasound image 22 into a rectangular shape for standardization of image processing. In other words, the margin region 22B does not show information necessary for medical treatment, such as the internal structure of the target area in the ultrasound image 22. By not including the margin region 22B in the superimposed image 26, it is possible to prevent the surgical field image 21 from becoming difficult to view in the superimposed image 26. In FIG. 7 and other figures, the superimposed image 26 from which the margin region 22B has been removed is illustrated.

[0074] The display control unit 56 controls the live view display of the operative field image 21 and the internal image. Live view display refers to the real-time display of moving images generated at a predetermined frame rate based on signals output by an imaging device for imaging the target. The imaging device includes, for example, an image sensor included in the endoscope 13 that optically captures images of the target, and an ultrasonic transducer 14C that uses ultrasound to image the target.

[0075] The display control unit 56 may also perform control to display the superimposed image 26 in a live view. In this case, the derivation unit 52 and the display control unit 56 repeatedly derive the position and orientation information and generate the superimposed image 26 each time the operative field image 21 and the ultrasound image 22 are acquired, or at predetermined time intervals. By displaying the superimposed image 26 in a live view, the operative field image 21 and the internal image can be displayed together in a live view.

[0076] In laparoscopic surgery, an ultrasound probe 14 is scanned to search for a region of interest, such as a tumor 27, contained within the internal structure of the target region. If there are multiple regions of interest, ultrasound images 22 capturing each of the regions of interest are acquired. In this case, there is a need to later check the ultrasound images 22 capturing each region of interest and the surgical field image 21 at the time the ultrasound images 22 were acquired. However, images acquired in live view display are not recorded in a reproducible manner. Therefore, if a user wishes to re-observe an image previously observed in live view display, the user must search again to reproduce the image.

[0077] Therefore, when the recording control unit 54 receives an instruction to record the operative field image 21 as a still image during live view display, the recording control unit 54 records a still image of the operative field image 21 at the timing of the instruction. Recording a still image means recording a captured image in a reproducible manner. In other words, an image that is recorded in a reproducible manner is called a still image. For example, the recording control unit 54 stores the recorded still image in the storage 43, thereby enabling the still image to be read from the storage 43.

[0078] Furthermore, when the recording control unit 54 receives an instruction to record the operative field image 21 as a still image, it also records a still image of the internal image (ultrasound image 22) at the timing of the instruction in addition to the still image of the operative field image 21. According to this configuration, the operative field image 21 and the internal image at the same timing can be recorded in association with each other, which improves the convenience of checking the still image later.

[0079] Furthermore, the recording control unit 54 may record a still image of the superimposed image 26 at the timing of the instruction. Specifically, the still image of the superimposed image 26 at the timing of the instruction is a superimposed image 26 in which a still image of the internal image at the timing of the instruction is superimposed on a still image of the operative field image 21 at the timing of the instruction, and the display mode of the still image of the internal image is adjusted based on the position and orientation information at the timing of the instruction.

[0080] The recording control unit 54 may also receive instructions at multiple times. In this case, the recording control unit 54 may record still images of the internal image at each of the multiple times. The recording control unit 54 may also record the superimposed image 26 at each of the multiple times.

[0081] The display control unit 56 may perform control so that at least one of a still image of the internal image at the timing of the instruction and a still image of the superimposed image 26 is displayed on the screen D that displays the operative field image 21 in live view.

[0082] 10 to 12 show an example of screen D displayed on the display 16 by the display control unit 56. FIGS. 10 to 12 are diagrams showing how screen D transitions. At a certain timing ta, screen D1 in FIG. 10 is displayed. At timing tb, which is after timing ta, screen D2 in FIG. 11 is displayed. At timing tc, which is after timing tb, screen D3 in FIG. 13 is displayed. Hereinafter, when there is no need to distinguish between screens D1 to D3, they will simply be referred to as screen D.

[0083] The operative field images 21 displayed in live view at times ta, tb, and tc are referred to as operative field images 21a, 21b, and 21c. Similarly, the ultrasound images 22 displayed in live view at times ta, tb, and tc are referred to as ultrasound images 22a, 22b, and 22c. Similarly, the superimposed images 26 displayed in live view at times ta, tb, and tc are referred to as superimposed images 26a, 26b, and 26c.

[0084] The screen D includes an area 90 for live view display and areas 96a and 96b for displaying still images. The area 90 displays any one of the operative field image 21, the ultrasound image 22, and the superimposed image 26 in live view. The image to be displayed in live view in the area 90 may be arbitrarily selected by the user, for example. That is, the display control unit 56 may receive a designation of an image to be displayed in live view on the display 16 from the operative field image 21, the ultrasound image 22, and the superimposed image 26, and perform control so that the designated image is displayed in live view.

[0085] As an example, screen D includes buttons 93 for selecting an image to be displayed in the live view display area 90. When the "operative field image" button 93 is selected on screen D, the operative field image 21 is displayed in live view in the area 90. When the "ultrasound image" button 93 is selected, the ultrasound image 22 is displayed in live view in the area 90. When the "superimposed image" button 93 is selected, the superimposed image 26 is displayed in live view in the area 90.

[0086] On screen D1 at timing ta, the "Superimposed Image" button 93 is selected, and superimposed image 26a is displayed in area 90. On screen D2 at timing tb, the "Superimposed Image" button 93 is selected, and superimposed image 26b is displayed in area 90. On screen D3 at timing tc, the "Operative Field Image" button 93 is selected, and operative field image 21c is displayed in area 90.

[0087] Screen D also includes a "still image recording" button 94 for specifying the timing for recording a still image during live view display. The user observes the image displayed in the live view display area 90 and selects the button 94 at the timing when the user desires to record a still image. When the button 94 is selected, the recording control unit 54 records a still image of the operative field image 21, a still image of the ultrasound image 22, and a still image of the superimposed image 26 at that timing.

[0088] When a still image is recorded at timing ta, the display control unit 56 controls the still image display area 96a to display a still image of the ultrasound image 22a at timing ta or a still image of the superimposed image 26a at timing ta. In the example of Fig. 11, the still image of the superimposed image 26a at timing ta is displayed in the area 96a. In this way, by displaying the live view display and the still image at the specified timing ta side by side, it is possible to improve convenience when checking the still image later.

[0089] Furthermore, it is preferable that the display control unit 56 controls the display so that the position indicated by the position and orientation information at the timing of the instruction is displayed on the operative field image 21 during live view display. In Fig. 11, a mark 98a indicating the position indicated by the position and orientation information at the timing ta of the instruction (i.e., the position of the ultrasound probe 14) is displayed on the operative field image 21b during live view display (more specifically, on the operative field image 21b included in the superimposed image 26b). This configuration makes it easy to see at which position in the operative field image 21b the still image was recorded, improving convenience when checking the still image later.

[0090] Furthermore, when a still image is recorded at timing tb, the display controller 56 controls the still image display area 96b to display a still image of the ultrasound image 22b at timing tb or a still image of the superimposed image 26b at timing tb. In the example of Fig. 12, the still image of the superimposed image 26b at timing tb is displayed in the area 96b. In this way, by controlling the superimposed images 26 at each of the multiple timings to be displayed side by side on the screen D that displays the operative field image 21 in live view, it is possible to easily compare the superimposed images 26 at each timing.

[0091] Furthermore, it is preferable that the display control unit 56 controls the display so that the correspondence between the still image of the internal image and the position indicated by the position and orientation information is identifiable for each instruction timing.Similarly, it is preferable that the display control unit 56 controls the display so that the correspondence between the superimposed image 26 and the position indicated by the position and orientation information is identifiable for each instruction timing.

[0092] The means for displaying the correspondence so as to be distinguishable is not particularly limited. For example, the display control unit 56 may use at least one of text and graphics to distinguishably display the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information, and / or the correspondence between the superimposed image 26 at each timing and the position indicated by the position and orientation information. For example, different text may be used for each timing, or different graphics (e.g., a circle, a rectangle, a star, etc.) may be used for each timing.

[0093] Furthermore, for example, the display control unit 56 may change the display color to distinguishably display the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information, and / or the correspondence between the superimposed image 26 at each timing and the position indicated by the position and orientation information. For example, the display control unit 56 may change the color of a mark indicating the position indicated by the position and orientation information.

[0094] 12, the letter "A" is assigned to mark 98a indicating the position indicated by the position and orientation information at timing ta of instruction, and the caption "A" is also assigned to area 96a where superimposed image 26a is displayed at timing ta. On the other hand, the letter "B" is assigned to mark 98b indicating the position indicated by the position and orientation information at timing tb of instruction, and the caption "B" is also assigned to area 96b where superimposed image 26b is displayed at timing tb.

[0095] According to this configuration, even if multiple still images are recorded at multiple times (i.e., multiple positions), the position at which each still image was recorded can be identified in the surgical field image 21b, thereby improving convenience when checking the still images later.

[0096] The types of still images displayed in the regions 96a and 96b may be the same or different. For example, the display control unit 56 may display a still image of the superimposed image 26a at timing ta in the region 96a and a still image of the ultrasound image 22b at timing tb in the region 96b. The types of still images displayed in the regions 96a and 96b may be arbitrarily set by the user, for example.

[0097] Furthermore, the display control unit 56 may display multiple types of still images at the same time on the screen D. As an example, Fig. 13 shows a screen D3P on which multiple types of still images are displayed at the same time. The screen D3P is a modified example of the screen D3, in which the area 96a on the screen D3 is divided into an area 91a and an area 92a, and the area 96b on the screen D3 is divided into an area 91b and an area 92b.

[0098] In the area 91a, a still image of the operative field image 21a at timing ta is displayed. In the area 92a, a still image of the ultrasound image 22a at timing ta is displayed. In the area 91b, a still image of the operative field image 21b at timing tb is displayed. In the area 92b, a still image of the ultrasound image 22a at timing tb is displayed. The type of still image to be displayed on the screen D may be arbitrarily set by the user, for example.

[0099] Next, the operation of the medical support device 11 according to this embodiment will be described with reference to Fig. 14. In the medical support device 11, the processor 41 executes the medical support program 44, thereby executing the medical support process shown in Fig. 14. This process is executed, for example, when a user issues an instruction to start execution via the reception device 46.

[0100] In step S10, the acquisition unit 50 acquires an operative field image obtained by optically capturing an operative field SF including a target site within the body and a medical instrument to be inserted into the body using a camera. The acquisition unit 50 also acquires an internal image showing the internal structure of the target site. In step S12, the derivation unit 52 derives position and orientation information showing the position and orientation of the medical instrument in the operative field SF based on the operative field image acquired in step S10.

[0101] In step S14, the display control unit 56 controls the display of a live view of a superimposed image obtained by superimposing an internal image on the operative field image acquired in step S10, the superimposed image having the display mode of the internal image adjusted based on the position and orientation information derived in step S12. In step S16, the recording control unit 54 waits until an instruction to record the operative field image being displayed in live view as a still image is received. If no instruction is input (N in step S16), the process returns to step S10, and the live view display of the superimposed image continues.

[0102] On the other hand, if an instruction has been received (Y in step S16), the process proceeds to step S18. In step S18, the recording control unit 54 records a still image of the internal image at the timing of the instruction received in step S16, in addition to a still image of the operative field image at the timing of the instruction. In step S20, the display control unit 56 controls the live view display screen to display the still image of the internal image recorded in step S18, and then the process ends.

[0103] As described above, the medical support device 11 according to this embodiment includes the processor 41. The processor 41 controls the live view display of a surgical field image obtained by optically capturing an surgical field SF, including a target region within the body and a medical instrument to be inserted into the body, using the camera 13B, and an internal image showing the internal structure of the target region. Furthermore, when the processor 41 receives an instruction to record the surgical field image as a still image during the live view display, it records a still image of the internal image at the time of the instruction in addition to the still image of the surgical field image.

[0104] That is, the medical support device 11 according to the present embodiment can record the operative field image 21 and the internal image at the same time in association with each other, thereby improving the convenience of checking the still images later.

[0105] In the above embodiment, the operative field image 21, the internal image (ultrasound image 22), and / or the superimposed image 26 are each displayed on one screen D. However, the present invention is not limited to this. For example, when there are multiple displays 16, the display control unit 56 may control the operative field image 21, the internal image, and the superimposed image 26 to be displayed on different displays 16. Furthermore, for example, the display control unit 56 may control the operative field image 21, the internal image, and the superimposed image 26 to be displayed in a subwindow different from the screen D.

[0106] In the above embodiment, the screen D that displays the operative field image 21, the internal image (ultrasound image 22), and / or the superimposed image 26 in a live view is controlled to display the recorded still image, but this is not limiting. For example, when there are multiple displays 16, the display control unit 56 may separate the display 16 that displays the live view from the display 16 that displays the recorded still image. Also, for example, the display control unit 56 may control the recorded still image to be displayed in a subwindow different from the screen D.

[0107] Furthermore, in the above embodiment, the area 90 for live view display and the areas 96 a, 96 b for still image display do not overlap on the screen D, but this is not limiting, and the area 90 may overlap with the areas 96 a, 96 b. In other words, the area 90 may overlap with the areas 96 a, 96 b, not limiting to an area where the image displayed in live view and the still image do not overlap, but may overlap with the still image.

[0108] In the above embodiment, an instruction to record the operative field image 21 as a still image is given by selecting the "record still image" button 94, but this is not limiting. Other methods for inputting instructions may be used, such as touch input using a touch panel, voice input using a microphone, and gesture input using a camera.

[0109] Alternatively, for example, an operation unit (e.g., a physical button) having a function similar to that of the "record still image" button 94 may be provided on the endoscope 13 or connected to an image processor for the endoscope, and instructions may be received using the operation unit. In this case, when the operation unit is operated, a signal indicating that an instruction to record a still image of the operative field image 21 has been issued is transmitted to the medical support device 11 via the image processor for the endoscope. In response to the signal, the medical support device 11 records a still image of the operative field image 21, a still image of the ultrasound image 22, and a still image of the superimposed image 26 at the timing of the instruction.

[0110] Next, modifications that can be applied to the above embodiment will be described. Note that the modifications described below can be combined in part or in whole as appropriate.

[0111] 15, the display control unit 56 may perform control based on the position and orientation information to display a scale 32 indicating the size of an internal structure (e.g., a tumor 27) included in an internal image (e.g., an ultrasound image 22) on the superimposed image 26. The size of the scale 32 can be calculated based on the distance from the camera 13B in the operative field SF to the display surface of the internal image (Xpb-Ypb plane) (i.e., the distance in the Z-axis direction parallel to the imaging optical axis). Note that, while the scale 32 is displayed on the ultrasound image 22 in FIG. 16, the present invention is not limited to this, and the scale 32 may be displayed at any position within the superimposed image 26.

[0112] 16 , based on the position and orientation information, the display control unit 56 may perform control so as to display a puncture path 30 of the puncture needle 18 inserted into the body using a medical instrument on the superimposed image 26. The puncture path 30 can be generated as an extension of the guide groove 29 based on the known positional relationship between the position indicated by the position and orientation information (e.g., the position of the reference point of the tip 14B of the ultrasound probe 14) and the guide groove 29.

[0113] The puncture path 30 is used as a guide when inserting the puncture needle 18 into the target position. For example, if the target position to be inserted with the puncture needle 18 is a tumor 27 in the liver LV, the medical staff ST positions the ultrasound probe 14 so that the puncture path 30 overlaps the tumor 27 in the ultrasound image 22. In this state, the puncture needle 18 is inserted into the tumor 27 using the puncture path 30 as a guide. In this way, by passing the puncture needle 18 through the guide groove 29 with the ultrasound probe 14 positioned so that the puncture path 30 passes through the target position, the puncture needle 18 can be made to reach the target position.

[0114] The display control unit 56 may display the puncture path 30 on the operative field image 21 in addition to or instead of displaying the puncture path 30 on the ultrasound image 22. In this case, the puncture path 30 indicates the path of the puncture needle 18 from the insertion position NP on the body surface BS of the patient PT to the guide groove 29 of the ultrasound probe 14.

[0115] In the above embodiment, the ultrasound image 22 is used as an example of an internal image showing the internal structure of the target region, but this is not limiting. In the medical support device 11, an internal image expressed in another way can be used in addition to or instead of the ultrasound image 22.

[0116] For example, a tomographic image of a target region extracted from a three-dimensional image taken by a tomography device such as a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device may be used as the internal image. Also, for example, a vascular image showing the vascular structure of a target region extracted from a three-dimensional image taken by a tomography device such as a CT device may be used as the internal image.

[0117] Specifically, prior to endoscopic surgery using the endoscope 13, ultrasound probe 14, etc., a three-dimensional image of the patient PT is captured in advance using a CT scanner or the like and stored in the storage 43, etc. The acquisition unit 50 acquires this three-dimensional image of the target area captured in advance. The display control unit 56 extracts internal images from the three-dimensional images based on the position and orientation information. For example, the display control unit 56 extracts, from the three-dimensional images, tomographic images and / or vascular images showing the internal structure in a predetermined region corresponding to the position and orientation of the medical instrument indicated by the position and orientation information. The display control unit 56 then generates the superimposed image 26 using the extracted internal images (tomographic images and / or vascular images).

[0118] According to this embodiment, internal images based on CT images or the like can be confirmed even during surgery, making it easier to grasp the internal structure in three dimensions, thereby improving convenience.

[0119] As described above, the medical support device 11 of the present disclosure can apply multiple types of internal images that are expressed in different ways, such as the ultrasound image 22, a tomographic image, and a vascular image. Therefore, the medical support device 11 may be able to switch the type of internal image to be displayed on the superimposed image 26.

[0120] Specifically, the acquisition unit 50 acquires a plurality of types of internal images expressed by different methods, including, for example, at least one of the ultrasound image 22, a tomographic image extracted from a three-dimensional image of the target region (such as a CT image or an MRI image), and a vascular image showing the vascular structure of the target region extracted from the three-dimensional image of the target region.

[0121] The display control unit 56 receives, for example, via the reception device 46, a designation of the type of internal image to be included in the superimposed image 26. Then, the display control unit 56 generates the superimposed image 26 using the internal image of the designated type.

[0122] Furthermore, when two or more types of internal images are specified, the display control unit 56 may generate a superimposed image 26 in which each of the specified internal images is superimposed on the operative field image 21. For example, when an ultrasound image 22 and a blood vessel image are specified, the display control unit 56 may generate a superimposed image 26 in which a semi-transparent blood vessel image is superimposed on the ultrasound image 22. In this way, the display control unit 56 may combine the internal images using a specific superimposition ratio.

[0123] In the above embodiment, the marker M is attached to the tip 14B of the insertion section 14A of the ultrasonic probe 14, but this is not limiting. The marker M may be attached to a portion of the medical instrument that is inserted into the body of the patient PT, and may be attached, for example, to the middle of the insertion section 14A or to the base end side of the insertion section 14A.

[0124] In the above embodiment, the position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF is derived using the marker M, but the present invention is not limited to this. For example, the position and orientation information may be derived by detecting the characteristic shape of the medical instrument from the surgical field image 21 by image analysis.

[0125] In the above embodiment, the medical instrument inserted into the body of the patient PT is an ultrasound probe 14 (an example of a medical probe) capable of observing the internal structure of an organ. Puncturing an internal organ with a puncture needle 18 is often performed using a medical probe capable of observing the internal structure of the organ. Therefore, the technology of the present disclosure is particularly effective when a medical probe is used as the medical instrument, as in the above embodiment.

[0126] Furthermore, the ultrasound probe 14 is relatively often used in combination with the puncture needle 18. Therefore, the technology of the present disclosure is even more effective when the ultrasound probe 14 is used as a medical probe. Note that the medical probe capable of observing the internal structure of an organ may be other than the ultrasound probe 14, and may be, for example, an optical coherence tomography (OCT) probe or the like.

[0127] Note that the medical instrument may be anything other than a medical probe capable of observing the internal structure of an organ. For example, the medical instrument may be a treatment instrument that does not have the function of observing the internal structure and only has a guide groove 29 for the puncture needle 18 at its tip. For example, if a tumor is present on the surface of an organ and the puncture needle 18 is to be inserted into the surface tumor, even a treatment instrument that does not have the function of observing the internal structure can properly guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST aligns the guide groove 29 of the treatment instrument with a position corresponding to the tumor on the surface of the organ and then inserts the puncture needle 18 into the tumor through the guide groove 29.

[0128] Furthermore, the medical instrument may be a simple rod-like treatment instrument without a guide groove 29. If a tumor is present on the surface of an organ, even such a treatment instrument can be used to indicate the tumor. Simply superimposing the puncture path 30 on the surgical field image 21 in which the treatment instrument has indicated a tumor on the surface of the organ can serve as a guide for checking the puncture direction of the puncture needle 18, etc. Therefore, the technology of the present disclosure is effective even if the medical instrument does not have a guide groove 29.

[0129] In the above embodiment, cauterization has been described as an example of the function of the puncture needle 18, but the function of the puncture needle 18 is not limited to this. In addition, the puncture needle 18 has been described as an example of a treatment tool, but other tools such as a treatment tool for injecting a fluorescent agent such as ICG (Indocyanine Green), a biopsy needle or forceps used for collecting tissue for a biopsy, etc. may also be used.

[0130] In addition, in the above embodiment, the inside of the body has been described using body cavities such as the abdominal cavity and the thoracic cavity as examples, but the inside of the body may also be the inside of an upper digestive tract such as the esophagus, a lower digestive tract such as the intestines, or a duct such as the bronchi. When the technology of the present disclosure is applied to a surgical field inside a duct, for example, a marker M is provided at the base end of a flexible endoscope that is inserted into the duct.

[0131] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 50, derivation unit 52, recording control unit 54, and display control unit 56. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0132] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0133] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0134] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0135] In the above embodiment, the medical support program 44 is pre-stored (installed) in the storage 43, but this is not limiting. The medical support program 44 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The medical support program 44 may also be downloaded from an external device via a network.

[0136] The present disclosure can also be applied to programs and program products. Specifically, the medical support program 44 in the above embodiment may be provided as a program product. The program product includes any type of product for providing a program. For example, the program product includes a program provided via a network such as the Internet, as well as a computer-readable recording medium that non-temporarily stores a program.

[0137] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0138] The following supplementary notes are further disclosed with respect to each of the above embodiments. [Supplementary Note 1] A medical support device including a processor, which controls live view display of a surgical field image obtained by optically capturing an internal surgical field including a target site within the body and a medical instrument to be inserted into the body using a camera, and an internal image showing the internal structure of the target site, and when an instruction to record the surgical field image as a still image is received during the live view display, records a still image of the internal image at the timing of the instruction in addition to the still image of the surgical field image. [Supplementary Note 2] The medical support device according to Supplementary Note 1, wherein the processor controls to display the still image of the internal image at the timing on a screen that displays the surgical field image in the live view. [Supplementary Note 3] The medical support device according to Supplementary Note 1 or Supplementary Note 2, wherein the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, and controls to display the position indicated by the position and orientation information at the timing on the surgical field image during the live view display. [Supplementary Note 4] The medical support device according to Supplementary Note 3, wherein the processor, when receiving instructions for a plurality of the timings, records still images of the internal image at each of the plurality of timings, and controls to identifiably display, for each timing, a correspondence between the still image of the internal image and the position indicated by the position and orientation information. [Supplementary Note 5] The medical support device according to Supplementary Note 4, wherein the processor controls to identifiably display, using at least one of text and graphics, the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information. [Supplementary Note 6] The medical support device according to Supplementary Note 4 or Supplementary Note 5, wherein the processor controls to identifiably display, by changing a display color, the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information.[Supplementary Note 7] The medical support device according to any one of Supplementary Notes 1 to 6, wherein the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the operative field image, and records a superimposed image in which a still image of the internal image at the timing is superimposed on a still image of the operative field image at the timing, the superimposed image being obtained by adjusting a display manner of the still image of the internal image based on the position and orientation information at the timing. [Supplementary Note 8] The medical support device according to Supplementary Note 7, wherein the processor controls to display the superimposed image on a screen that displays the operative field image in live view. [Supplementary Note 9] The medical support device according to Supplementary Note 8, wherein the processor, when receiving instructions for a plurality of the timings, records the superimposed image at each of the plurality of timings, and controls to display the superimposed images at each of the plurality of timings side by side on the screen that displays the operative field image in live view. [Supplementary Note 10] The medical support device according to Supplementary Note 9, wherein the processor controls to display the position indicated by the position and orientation information at the timing on the surgical field image being displayed in the live view, and further controls to display, for each timing, a correspondence between the superimposed image and the position indicated by the position and orientation information in a identifiable manner. [Supplementary Note 11] The medical support device according to Supplementary Note 10, wherein the processor controls to display, for each timing, a correspondence between the superimposed image and the position indicated by the position and orientation information in a identifiable manner using at least one of text and graphics. [Supplementary Note 12] The medical support device according to Supplementary Note 10 or Supplementary Note 11, wherein the processor controls to display, for each timing, a correspondence between the superimposed image and the position indicated by the position and orientation information in a identifiable manner by changing a display color. [Supplementary Note 13] The medical support device according to any one of Supplements 7 to 12, wherein the processor controls to display, on the superimposed image, a scale indicating the size of the internal structure included in the internal image based on the position and orientation information.[Supplementary Note 14] The medical support device according to any one of Supplements 7 to 13, wherein the processor controls to display on the superimposed image, based on the position and orientation information, a puncture path of a puncture needle inserted into the body using the medical instrument. [Supplementary Note 15] The medical support device according to any one of Supplements 3 to 14, wherein a marker configured with an optically detectable pattern is provided on an outer peripheral surface of the medical instrument, and the processor derives the position and orientation information based on the marker included in the surgical field image. [Supplementary Note 16] The medical support device according to any one of Supplements 1 to 15, wherein the medical instrument is an ultrasound probe that transmits ultrasound to the target site and detects electrical signals corresponding to ultrasound echoes reflected from the target site, and the internal image is an ultrasound image generated in accordance with the electrical signals. [Supplementary Note 17] The medical support device according to Supplementary Note 16, wherein the processor uses, as the internal image, an image from which a margin area not showing the internal structure of the target site in the ultrasound image has been removed. [Supplementary Note 18] The medical support device according to any one of Supplementary Note 1 to Supplementary Note 17, wherein the processor: derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquires a three-dimensional image of the target site; and extracts the internal image from the three-dimensional image based on the position and orientation information. [Supplementary Note 19] A medical support method in which a computer executes processes: controlling live view display of a surgical field image obtained by optically capturing, by a camera, a surgical field including a target site inside the body and the medical instrument to be inserted into the body, and an internal image showing the internal structure of the target site, and when an instruction to record the surgical field image as a still image is received during the live view display, recording a still image of the internal image at the timing of the instruction in addition to the still image of the surgical field image.[Supplementary Note 20] A medical support program that causes a computer to execute a process of controlling live view display of a surgical field image obtained by optically photographing a surgical field including a target part inside the body and a medical instrument to be inserted into the body using a camera, and an internal image showing the internal structure of the target part, and when an instruction to record the surgical field image as a still image is received during the live view display, recording a still image of the internal image at the timing of the instruction in addition to the still image of the surgical field image.

[0139] The disclosure of Japanese Patent Application No. 2024-114344, filed on July 17, 2024, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A medical support device comprising a processor that controls live view display of an operative field image optically captured by a camera of a operative field including a target area inside the body and a medical instrument to be inserted into the body, and an internal image showing the internal structure of the target area, and when an instruction to record the operative field image as a still image is received during the live view display, the processor records a still image of the internal image at the time of the instruction in addition to the still image of the operative field image.

2. The medical support device according to claim 1, wherein the processor controls the display of a still image of the internal image at the timing on a screen that displays the surgical field image in live view.

3. The medical support device according to claim 1, wherein the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, and controls the display of the position indicated by the position and orientation information at the timing on the surgical field image during the live view display.

4. The medical support device according to claim 3, wherein when the processor receives instructions for a plurality of said timings, it records still images of the internal image at each of the plurality of said timings, and controls the display so that, for each said timing, the correspondence between the still image of the internal image and the position indicated by the position and orientation information is identifiable.

5. The medical support device according to claim 4, wherein the processor controls the display so that the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information is identifiable using at least one of text and graphics.

6. The medical support device according to claim 4, wherein the processor controls the display so that the correspondence between the still image of the internal image at each timing and the position indicated by the position and orientation information is displayed in a distinguishable manner by changing the display color.

7. The medical support device according to claim 1, wherein the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, and records a superimposed image in which a still image of the internal image at the timing is superimposed on a still image of the surgical field image at the timing, the superimposed image having a display mode of the still image of the internal image adjusted based on the position and orientation information at the timing.

8. The medical support device according to claim 7, wherein the processor controls the display of the superimposed image on a screen that displays the surgical field image in live view.

9. The medical support device according to claim 8, wherein when the processor receives instructions for a plurality of said timings, the processor records the superimposed images at each of said plurality of timings, and controls the superimposed images at each of said plurality of timings to be displayed side by side on the screen that displays the surgical field image in live view.

10. The medical support device according to claim 9, wherein the processor controls the display of the position indicated by the position and orientation information at the timing on the surgical field image during the live view display, and further controls the display of the correspondence between the superimposed image and the position indicated by the position and orientation information in a distinguishable manner for each timing.

11. The medical support device according to claim 10, wherein the processor controls the display so that the correspondence between the superimposed image at each timing and the position indicated by the position and orientation information is identifiable using at least one of text and graphics.

12. The medical support device according to claim 10, wherein the processor controls the display so that the correspondence between the superimposed image for each timing and the position indicated by the position and orientation information is displayed in a distinguishable manner by changing the display color.

13. The medical support device according to claim 7, wherein the processor controls the display of a scale indicating the size of the internal structure included in the internal image on the superimposed image based on the position and orientation information.

14. The medical support device according to claim 7, wherein the processor controls the display of the puncture path of a puncture needle inserted into the body using the medical instrument on the superimposed image based on the position and orientation information.

15. The medical support device according to claim 7, wherein the medical instrument has a marker formed of an optically detectable pattern on its outer surface, and the processor derives the position and orientation information based on the marker included in the surgical field image.

16. The medical support device according to claim 1, wherein the medical instrument is an ultrasound probe that transmits ultrasound to the target area and detects electrical signals corresponding to ultrasound echoes reflected from the target area, and the internal image is an ultrasound image generated in response to the electrical signals.

17. The medical support device according to claim 16, wherein the processor uses an image obtained by removing blank areas in the ultrasound image that do not show the internal structure of the target area as the internal image.

18. The medical support device of claim 1, wherein the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, acquires a three-dimensional image of the target area, and extracts the internal image from the three-dimensional image based on the position and orientation information.

19. A medical support method in which a computer executes a process to control the live view display of an operative field image, which is an optical image of an operative field including a target area inside the body and a medical instrument to be inserted into the body, taken by a camera, and an internal image showing the internal structure of the target area, and when an instruction to record the operative field image as a still image is received during the live view display, a still image of the internal image at the time of the instruction is also recorded in addition to the still image of the operative field image.

20. A medical support program that causes a computer to execute a process that controls the live view display of an operative field image, which is an optical image of an operative field including a target area inside the body and a medical instrument to be inserted into the body, taken by a camera, and an internal image showing the internal structure of the target area, and when an instruction to record the operative field image as a still image is received during the live view display, records a still image of the internal image at the time of the instruction in addition to the still image of the operative field image.

Images