Medical support device, medical support method, and medical support program

Abstract

A medical support device according to the present invention includes a processor. The processor acquires a surgical field image obtained by optically shooting a surgical field, including a subject site in a body and a medical instrument inserted into the body, by a camera, derives position and orientation information indicating a position and an orientation of the medical instrument in the surgical field on the basis of the surgical field image, acquires a first internal image indicating an internal structure of the subject site, controls display of a superimposed image obtained by superimposing the first internal image on the surgical field image and adjusting a display mode of the first internal image on the basis of the position and orientation information, and, furthermore, changes the display mode of the first internal image in accordance with a direction of the first internal image in the superimposed image, the direction changing in accordance with the orientation of the medical instrument.

Description

Medical support device, medical support method, and medical support program

[0001] The present disclosure relates to a medical support device, a medical support method, and a medical support program.

[0002] Conventionally, there is known a technique for identifying the position and orientation of a medical instrument inserted into a body and supporting medical treatment such as surgery and examination based on the identified position and orientation of the medical instrument. For example, International Publication No. 2023 / 162657 discloses a technique for estimating the position and orientation of an ultrasound probe by performing image analysis on an optically captured surgical field image including an ultrasound probe, and for displaying preparation information corresponding to the estimated position and orientation superimposed on the surgical field image.

[0003] In recent years, there has been a demand for technology that can further improve convenience when superimposing and displaying information on an operative field image according to the position and posture of a medical instrument inserted inside the body.

[0004] The present disclosure provides a medical support device, a medical support method, and a medical support program that can further improve convenience.

[0005] A first aspect of the present disclosure is a medical support device comprising a processor, which acquires a surgical field image optically photographed by a camera of a surgical field including a target area within the body and a medical instrument to be inserted into the body, derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, acquires a first internal image showing the internal structure of the target area, controls the display of a superimposed image in which the first internal image is superimposed on the surgical field image, and adjusts the display mode of the first internal image based on the position and orientation information, and further changes the display mode of the first internal image according to the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

[0006] The processor may change the display mode of the first internal image according to an angle representing the orientation of the first internal image, which is the angle at which the camera's shooting optical axis is least parallel to the display surface of the first internal image and the angle at which the shooting optical axis is most normal to the display surface of the first internal image.

[0007] The processor may perform control to display a superimposed image when the angle is equal to or greater than a predetermined threshold, and may perform control to display a surgical field image instead of the superimposed image when the angle is less than the threshold.

[0008] The processor may record position and orientation information of the medical instrument over time, and if the angle is less than a threshold, control the display of the movement trajectory of the medical instrument on the surgical field image based on the position and orientation information recorded over time.

[0009] The processor may control the visibility of the movement trajectory to decrease over time.

[0010] The processor may control the movement trajectory to be displayed as a point cloud.

[0011] Regarding the orientation of the first internal image, if the display surface of the first internal image and the camera's shooting optical axis are perpendicular to each other, the front is defined as the direction in which the image is viewed. If the orientation of the first internal image determined based on the position and orientation information does not satisfy a predetermined condition, the processor may change the orientation of the first internal image to the front.

[0012] If the orientation of the first internal image does not satisfy the condition, the processor may superimpose a first internal image facing forward at a position within the superimposed image determined based on the position and orientation information.

[0013] If the orientation of the first internal image does not satisfy the condition, the processor may superimpose a first internal image facing forward at a preset position on the periphery of the superimposed image.

[0014] The medical instrument may be an ultrasound probe that transmits ultrasound to a target area and detects electrical signals corresponding to ultrasound echoes reflected from the target area, and the first internal image may be an ultrasound image generated in response to the electrical signals.

[0015] The processor may generate the superimposed image by using, as the first internal image, an image obtained by removing blank areas in the ultrasound image that do not show the internal structure of the target region.

[0016] The processor acquires a three-dimensional image of the target area, and based on the position and orientation information, extracts from the three-dimensional image a second internal image that shows the internal structure of the target area using a different representation method than the first internal image.If the angle is greater than or equal to a predetermined threshold, the processor controls to display a first superimposed image in which the second internal image is superimposed on the surgical field image in addition to the first internal image, and if the angle is less than the threshold, the processor controls to display a second superimposed image in which the second internal image is superimposed on the surgical field image instead of the first internal image.

[0017] The processor may be configured to control the visibility of the second internal image in at least one of the first and second superimposed images to be changed depending on the distance from the camera to the medical instrument.

[0018] The processor may perform control to change the visibility of the second internal image in at least one of the first superimposed image and the second superimposed image depending on the depth in the direction from the surface layer to the deep layer of the target area.

[0019] The second internal image may be a tomographic image of the target area.

[0020] The second internal image may be a blood vessel image showing the vascular structure of the target area.

[0021] The processor may perform control based on the position and orientation information to display a scale indicating the size of the internal structure included in the first internal image on the superimposed image.

[0022] The processor may perform control based on the position and orientation information so that the puncture path of a puncture needle inserted into the body using a medical instrument is displayed on the superimposed image.

[0023] A second aspect of the present disclosure is a medical support method in which a computer executes a process to acquire a surgical field image optically photographed by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body, derive position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, acquire a first internal image indicating the internal structure of the target area, superimpose the first internal image on the surgical field image, control the display of the superimposed image in which the display mode of the first internal image is adjusted based on the position and orientation information, and further change the display mode of the first internal image according to the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

[0024] A third aspect of the present disclosure is a medical support program that causes a computer to perform a process of acquiring a surgical field image optically photographed by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body, deriving position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image, acquiring a first internal image showing the internal structure of the target area, controlling the display of a superimposed image in which the first internal image is superimposed on the surgical field image, with the display mode of the first internal image adjusted based on the position and orientation information, and further changing the display mode of the first internal image according to the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

[0025] According to the above aspects, the medical support device, medical support method, and medical support program of the present disclosure can further improve convenience.

[0026] 1 is a diagram illustrating an overview of a medical support system including a medical support device. FIG. 1 is a diagram illustrating the state inside the body during laparoscopic surgery. FIG. 2 is a diagram illustrating an insertion state of a puncture needle guided by a guide groove. FIG. 2 is a diagram illustrating an example of the hardware configuration of a medical support device. FIG. 3 is a diagram illustrating an example of the functional configuration of a medical support device. FIG. 4 is a diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. FIG. 5 is a diagram illustrating an example of a superimposed image. FIG. 6 is a diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. FIG. 7 is a diagram illustrating an example of a superimposed image. FIG. 8 is a diagram illustrating an example of a superimposed image in a distanced state. FIG. 9 is a diagram illustrating an example of an operative field image in a distanced state. FIG. 10 is a diagram for explaining the visibility of the superimposed image. A flowchart illustrating an example of medical support processing according to the first embodiment. A diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. A diagram illustrating an example of a superimposed image. A diagram illustrating the relationship between the position and posture of a marker and an ultrasound probe. A diagram illustrating an example of a superimposed image. A diagram illustrating an example of an operative field image. A diagram illustrating an example of a movement trajectory. A flowchart illustrating an example of medical support processing according to a second embodiment. A diagram illustrating an example of a superimposed image. A diagram illustrating an example of a superimposed image. A diagram illustrating an example of a superimposed image. A diagram illustrating an example of a scale. A diagram illustrating an example of a puncture route. A diagram illustrating an example of a superimposed image with changed visibility. FIG. 1 is a diagram showing an example of a superimposed image with changed visibility; FIG. 2 is a diagram showing an example of a superimposed image using a tomographic image; FIG. 3 is a diagram showing an example of a superimposed image using a blood vessel image; FIG. 4 is a diagram for explaining target coordinates; FIG. 5 is a diagram for explaining target coordinates; FIG. 6 is a diagram for explaining target coordinates.

[0027] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are designated by the same reference numerals, and duplicate descriptions will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0028] First Embodiment An example of a medical support system 10 incorporating a medical support device 11 according to this embodiment will be described with reference to FIGS. 1 to 3 . As an example, the medical support system 10 is used when performing endoscopic surgery on a patient PT using an endoscope 13. Unlike open surgery, endoscopic surgery is a procedure in which a small hole is made in the patient PT's body and a medical instrument, such as an endoscope 13, is inserted through the hole. The medical support system 10 not only provides medical staff ST, including physicians, with a view of the surgical field within the patient PT's body, but also provides support information to support medical treatment, such as surgery and examinations. The support information may be, for example, a superimposed image 26 for displaying an ultrasound image 22 overlaid on a surgical field image 21, as will be described later. Because this medical support system 10 has the function of providing support information in real time during surgery, it is also referred to as a surgical navigation system.

[0029] 1 , the medical support system 10 includes a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is connected to the endoscope 13, the ultrasound probe 14, and the display 16 so as to be able to communicate with each other.

[0030] 2 shows the state in which an endoscope 13 and an ultrasonic probe 14 are being inserted into the abdomen of a patient PT. In endoscopic surgery, a portion of the endoscope 13 and the ultrasonic probe 14, including their respective distal ends, is inserted into the body via a trocar 17. The trocar 17 is an insertion tool that has an insertion hole through which the endoscope 13 and the like are inserted and a valve that is provided within the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and therefore the trocar 17 is used to insert the endoscope 13, the ultrasonic probe 14, and the like into the body.

[0031] The puncture needle 18 is a treatment tool used to puncture a lesion, such as a tumor, contained in an organ. Specifically, the puncture needle 18 has a needle portion 18A and a grip portion 18B provided on the proximal end of the needle portion 18A. The puncture needle 18 is, for example, an ablation puncture needle used to cauterize a lesion. The ablation puncture needle has an electrode to which a high-frequency voltage is applied at its tip. When the high-frequency voltage is applied while the electrode is inserted into the lesion, the heat generated by the electrode necrotizes the lesion. In this embodiment, as an example, a tumor 27 in the liver LV is visualized using an ultrasound image 22, and the visualized tumor 27 is cauterized with the puncture needle 18, thereby performing a treatment to necrotize the tumor 27.

[0032] The endoscope 13 optically captures an operative field SF, including a target site (the liver LV in this example) inside the body of the patient PT, using a camera 13B. The operative field SF is a space extending inside a body cavity defined by organs and the body wall. Specifically, the endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT. The tip of the insertion section 13A is equipped with a camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)). As an example, the insertion section 13A of the endoscope 13 is a rigid endoscope, and is also called a laparoscope because it is often used for observing the abdominal cavity.

[0033] The camera 13B includes an image sensor, such as a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images. The endoscope 13 is connected to an endoscope image processor (not shown). The image processor processes the image signal output by the image sensor to generate an operative field image 21 of the operative field SF within the body. The operative field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via the endoscope image processor. In FIG. 2, the symbols Xin and Yin indicate the coordinate system of the operative field image 21. The camera 13B is an example of a "camera" in the present disclosure.

[0034] The illumination light for the endoscope 13 is, for example, visible light such as white light. Alternatively, special light such as ultraviolet light or infrared light may be used as the illumination light for the endoscope 13. The special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet range. The surgical field image 21 is an image of the surgical field SF illuminated by the illumination light, and more specifically, an image based on light reflected from the surface of the surgical field SF. Therefore, while the surgical field image 21 can depict structures present near the surface of the target area, it is difficult to observe the internal structure.

[0035] The ultrasonic probe 14 transmits ultrasonic waves to a target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area. Specifically, the ultrasonic probe 14 has an insertion section 14A that is inserted into the body of the patient PT and an operation section 14D on the proximal end side of the insertion section 14A. An ultrasonic transducer 14C is built into a tip section 14B of the insertion section 14A. The ultrasonic probe 14 is an example of a "medical instrument" of the present disclosure.

[0036] The ultrasonic transducer 14C transmits ultrasonic waves to the target region and receives ultrasonic echoes reflected from the target region. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown). This image processing processor performs image reconstruction processing based on electrical signals corresponding to the ultrasonic echoes received by the ultrasonic transducer 14C. The image reconstruction processing generates an ultrasonic image 22 showing the internal structure of the target region scanned by the ultrasonic probe 14. The ultrasonic image 22 captured by the ultrasonic probe 14 is transmitted to the medical support device 11 in real time via the image processing processor for the ultrasonic probe. In FIG. 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0037] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and acquires a fan-shaped ultrasonic image 22 centered on the ultrasonic transducer 14C. By scanning the ultrasonic probe 14, multiple ultrasonic images 22 are captured along the scanning direction. The ultrasonic images 22 are so-called B (brightness) mode images that visualize the internal structure of the target area, from the superficial layer to the deep layer where the ultrasonic waves reach, as brightness information. The ultrasonic images 22 visualize the internal structure of the target area that cannot be observed in the surgical field image 21 obtained by optical imaging.

[0038] A guide groove 29 is provided in the distal end portion 14B of the insertion section 14A. The guide groove 29 is a groove that engages with the puncture needle 18 to guide insertion of the puncture needle 18 to a target position inside an organ. FIG. 3 schematically shows the puncture needle 18 inserted into the body puncturing a tumor 27 in the liver LV while being guided by the guide groove 29. The guide groove 29 is provided, for example, on the proximal side of the distal end portion 14B relative to the ultrasonic transducer 14C, and is inclined at an angle θ with respect to the direction of the axis AX of the distal end portion 14B. The first guide groove 29 is inclined backward so that the tip of the puncture needle 18 inserted from the proximal side of the distal end portion 14B faces the distal end side of the distal end portion 14B.

[0039] The puncture needle 18 is inserted while checking the tumor 27 using the ultrasound image 22. Because the area visualized by the ultrasound image 22 is a radial area originating from the ultrasound transducer 14C, the inclined guide grooves 29 make it possible to point the tip of the puncture needle 18 toward the tumor 27 visualized by the ultrasound image 22. Note that the insertion section 14A may be provided with multiple guide grooves 29 that are angled differently relative to the direction of the axis AX. In this case, the medical staff ST can select an appropriate guide groove 29 to insert the puncture needle 18.

[0040] Furthermore, a marker M is attached to the tip 14B of the insertion section 14A. The marker M is a marker that can be recognized from the operative field image 21 optically captured by the camera 13B of the endoscope 13, that is, an optically detectable marker. The marker M is used by the medical support device 11 to estimate the position and orientation of the ultrasound probe 14, more specifically, the position and orientation of the tip 14B of the insertion section 14A in the operative field SF. A method for estimating the position and orientation using the marker M will be described later.

[0041] As an example, the marker M is a marker of a grid pattern 62 that is configured by a first line extending in the direction of the axis AX of the tip portion 14B of the ultrasonic probe 14 and a second line that is perpendicular to the axial direction of the tip portion 14B and formed in the circumferential direction along the outer circumferential surface of the tip portion 14B. Furthermore, the intersections in the grid pattern 62 are given circular symbols 64 or rectangular symbols 66, respectively.

[0042] The marker M only needs to include at least the symbols 64, 66 arranged at each intersection of the grid pattern 62, and does not need to include the line segments representing the first and second lines as shown in Fig. 3 etc. Furthermore, the symbols 64, 66 in the marker M are not limited to circles and rectangles and may have any shape, and may be represented by, for example, figures such as triangles, polygons, stars, and various marks, or may be represented by letters etc. Furthermore, in each drawing, the illustration of the marker M may be simplified or omitted to avoid cluttering the drawing.

[0043] The medical support device 11 acquires a surgical field image 21 from the endoscope 13 and acquires an ultrasound image 22 from the ultrasound probe 14. The medical support device 11 also generates a superimposed image 26 by superimposing the ultrasound image 22 on the surgical field image 21, and controls the display 16 to display the superimposed image 26. The superimposed image 26 displayed on the display 16 provides the medical staff ST with the field of view of the surgical field SF inside the body of the patient PT and the internal structure of the target site (liver LV).

[0044] Next, the configuration of the medical support device 11 will be described. Fig. 4 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a display 16, a processor 41, a RAM (Random Access Memory) 42, a storage 43, a reception device 46, a communication I / F (Interface) 47, and an external I / F 48. Each of these components is connected to a bus 49, such as a system bus and a control bus, and can communicate with each other.

[0045] The display 16 displays various information in addition to the superimposed image 26. Examples of the display 16 include a liquid crystal display and an EL (Electro-Luminescence) display. The number of displays 16 may be at least one as shown in FIG. 1, but may also be multiple.

[0046] The processor 41 is, for example, a CPU (Central Processing Unit), which comprehensively controls each part of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0047] The RAM 42 is a memory that temporarily stores information and is used as a work memory by the processor 41. Examples of the RAM 42 include a dynamic random access memory (DRAM) and a static random access memory (SRAM).

[0048] The storage 43 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 43 include a hard disk drive (HDD) and a solid state drive (SSD). The storage 43 stores a medical support program 44 that causes the computer to function as the medical support device 11.

[0049] The storage 43 also stores dimension information 45. The dimension information 45 includes dimensions of the ultrasonic probe 14, specifically, information representing the relative positional relationships of the marker M, ultrasonic transducer 14C, guide groove 29, and the like in the ultrasonic probe 14. The positional relationship of the marker M is, for example, information on the positions and orientations of the reference symbols 64 and 66 constituting the marker M relative to the axial and circumferential directions of the tip portion 14B. The positional relationship of the ultrasonic transducer 14C is represented by, for example, the linear distance between a reference point of the tip portion 14B and the ultrasonic transducer 14C, and the inclination angle of the ultrasonic transducer 14C relative to the axis AX of the tip portion 14B. The positional relationship of the guide groove 29 is represented by, for example, the linear distance between a reference point of the tip portion 14B and the guide groove 29, and the inclination angle θ of the guide groove 29 relative to the axis AX of the tip portion 14B.

[0050] The reception device 46 has a keyboard, a mouse, etc. (not shown) and receives instructions from an operator. That is, the medical support apparatus 11 is operated by an operator such as a medical staff member ST through the reception device 46. The reception device 46 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0051] The communication I / F 47 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control in accordance with communication protocols defined by various wired or wireless communication standards.

[0052] The external I / F 48 is, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer, a memory card, etc. The medical support device 11 may be, for example, a server computer, a personal computer, a smartphone, a tablet terminal, a wearable terminal, or the like, as appropriate.

[0053] Next, the functional configuration of the medical support device 11 will be described. Fig. 5 is a block diagram showing an example of the functional configuration of the medical support device 11. The medical support device 11 includes an acquisition unit 50, a derivation unit 52, a determination unit 54, and a display control unit 56. The processor 41 reads the medical support program 44 from the storage 43 and executes the medical support program 44 on the RAM 42, causing the processor 41 to function as the functional units of the acquisition unit 50, the derivation unit 52, the determination unit 54, and the display control unit 56. The processor 41 operates as the functional units of the acquisition unit 50, the derivation unit 52, the determination unit 54, and the display control unit 56, thereby realizing medical support processing.

[0054] The acquisition unit 50 acquires a surgical field image 21 obtained by optically capturing an operative field SF including a target site within the body and a medical instrument inserted into the body using the camera 13B. For example, the acquisition unit 50 acquires the surgical field image 21 from a device including a processor of the endoscope 13 via the external I / F 48 or the communication I / F 47. As shown in FIG. 2 , when the ultrasound probe 14 is inserted into the surgical field SF, the ultrasound probe 14, more specifically, the tip 14B of the insertion section 14A, appears in the surgical field image 21.

[0055] The acquisition unit 50 also acquires an internal image showing the internal structure of the target area. For example, the acquisition unit 50 acquires an ultrasound image 22 from a device including a processor of the ultrasound probe 14 via the external I / F 48 or the communication I / F 47. The ultrasound image 22 is an example of an "internal image" in the present disclosure. Note that the processor of the endoscope 13 and / or the processor of the ultrasound probe 14 may be included in the medical support device 11.

[0056] The derivation unit 52 derives position and orientation information indicating the position and orientation of a medical instrument (e.g., the ultrasound probe 14) in the surgical field SF based on the surgical field image 21. Specifically, the derivation unit 52 detects the marker M by searching for morphological features of the marker M, such as the reference numerals 64 and 66, in the surgical field image 21. For example, the derivation unit 52 may detect the marker M using an image processing method such as pattern matching.

[0057] Furthermore, for example, instead of a rule-based method such as pattern matching, an AI (Artificial Intelligence) technique using a machine learning model may be used to detect the marker M. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) that is trained in advance to input the surgical field image 21 and output the area of ​​the marker M in the input surgical field image 21 can be applied.

[0058] The display control unit 56 controls the display of the superimposed image 26, which is obtained by superimposing an internal image (e.g., an ultrasound image 22) on the operative field image 21, and which adjusts the display mode of the internal image based on the position and orientation information. As an example, the display control unit 56 generates the superimposed image 26, which indicates the imaging range (e.g., position, orientation, and size) of the ultrasound image 22 in the operative field SF, by superimposing the ultrasound image 22 on an area on the operative field image 21 that corresponds to the imaging range of the ultrasound image 22.

[0059] Specific examples of the superimposed image 26 will be described with reference to Figures 6 to 9. Figures 6 and 8 are diagrams conceptually showing the position and orientation of the tip 14B of the ultrasonic probe 14 within the surgical field SF defined as a three-dimensional space, each assuming a different position and orientation. Figures 7 and 9 show examples of the superimposed image 26 generated according to the position and orientation of the ultrasonic probe 14 within the surgical field SF. Figure 7 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 6. Figure 9 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 8.

[0060] In the three-dimensional operative field SF shown in Figures 6 and 8, the Z axis is parallel to the imaging optical axis of the camera 13B of the endoscope 13. Furthermore, in the three-dimensional operative field SF, the X-Y plane is parallel to the imaging plane of the camera 13B and is perpendicular to the imaging optical axis. In other words, the X-Y plane of the three-dimensional operative field SF is parallel to the screen (Xin-Yin plane) of the operative field image 21. The operative field image 21 is a projected image of the operative field SF projected from one viewpoint. Note that, in Figures 6 and 8, of the reference symbols 64 and 66 that constitute the marker M, those that appear in the operative field image 21 are indicated in dark colors, and those that do not appear in the operative field image 21 are indicated in light colors.

[0061] 6 shows a state in which the axis AX of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13B (more specifically, the axis AX of the tip 14B is parallel to the X-axis) within the surgical field SF in three-dimensional space. In this case, as shown in FIG. 7, the marker M reflected in the surgical field image 21 has orthogonal lines of the grid pattern 62 parallel to the X-axis and Y-axis, respectively. Furthermore, the reference numerals 64 and 66 appear at equal intervals in the surgical field image 21.

[0062] On the other hand, Fig. 8 shows a state in which, within the surgical field SF in three-dimensional space, the direction of the axis AX of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13B, but is tilted in the depth direction parallel to the imaging optical axis. The attitude shown in Fig. 8 is a state in which the axis AX of the tip 14B is rotated approximately -25 degrees around the Y axis from the attitude shown in Fig. 6. In this case, as shown in Fig. 9, the marker M appearing in the surgical field image 21 appears such that the circumferentially extending lines of the grid pattern 62 become shorter and the interval between the reference symbols 64 and 66 becomes shorter as the marker M becomes farther from the camera 13B in the depth direction.

[0063] In this way, the shape of the marker M that appears in the operative field image 21 changes depending on the posture of the tip portion 14B. The derivation unit 52 estimates the posture of the tip portion 14B of the ultrasound probe 14 in the operative field SF based on the posture of the marker M in the operative field image 21. Specifically, the derivation unit 52 detects the direction of the axis AX of the tip portion 14B in the operative field SF as the posture of the tip portion 14B.

[0064] Furthermore, if the position of the tip portion 14B changes within the operative field SF, the position of the marker M reflected in the operative field image 21 also changes. The derivation unit 52 estimates the position of the tip portion 14B within the operative field SF based on the position of the marker M. The position of the tip portion 14B is detected as the position of a reference point of the tip portion 14B, which is provided, for example, at the tip position of the tip portion 14B. Furthermore, the shooting distance from the camera 13B to the marker M within the operative field SF (i.e., the distance in the Z-axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the operative field image 21. The derivation unit 52 derives the position coordinates of the reference point of the tip portion 14B within the operative field SF based on the shooting distance and dimension information 45 of the tip portion 14B, which includes the known dimensions of the marker M.

[0065] 6, the position of the tip 14B of the ultrasonic probe 14 is estimated as information such as the position coordinates (X01, Y01, Z01) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as information that it is parallel to the XY plane and the XZ plane and perpendicular to the YZ plane.

[0066] 8, the position of the tip portion 14B of the ultrasound probe 14 is estimated as information such as the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B within the surgical field SF. The orientation of the tip portion 14B is defined as the direction of the axis AX of the tip portion 14B within the surgical field SF. For example, the axis AX of the tip portion 14B is estimated as information such as -25° with respect to the XY plane, parallel to the XZ plane, and 65° with respect to the YZ plane.

[0067] When the position and orientation of the tip portion 14B in the surgical field SF is estimated by the derivation unit 52, the display control unit 56 estimates the position and orientation of the ultrasonic transducer 14C in the surgical field SF based on the estimated position and orientation of the tip portion 14B and the dimension information 45. As described above, the linear distance between the reference point of the tip portion 14B and the ultrasonic transducer 14C, and the tilt angle of the ultrasonic transducer 14C with respect to the direction of the axis AX of the tip portion 14B are known information defined as the dimension information 45. By using this dimension information 45, the position and orientation of the ultrasonic transducer 14C in the surgical field SF can be estimated based on the position and orientation of the tip portion 14B in the surgical field SF.

[0068] 6, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X11, Y11, Z11) obtained by correcting the position coordinates (X01, Y01, Z01) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The attitude of the ultrasonic transducer 14C is defined as the attitude of the tip portion 14B with respect to the axis AX, and the tilt angle is known as the dimension information 45.

[0069] 8, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X12, Y12, Z12) obtained by correcting the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The orientation of the ultrasonic transducer 14C is defined as the orientation of the tip portion 14B with respect to the axis AX, and this becomes a tilt angle known as the dimension information 45.

[0070] The display control unit 56 also estimates the imaging range of the ultrasound image 22 in the surgical field SF based on the estimated position and orientation of the ultrasound transducer 14C. The imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer 14C. Therefore, if the position and orientation of the ultrasound transducer 14C in the surgical field SF are known, the imaging range of the ultrasound image 22 in the surgical field SF can also be estimated. For example, the imaging range of the ultrasound image 22 captured by the convex-type ultrasound transducer 14C is a sector-shaped range that spreads radially from the ultrasound transducer 14C as the base point. Figures 6 and 8 illustrate an imaging range 22R of the ultrasound image 22 in the surgical field SF.

[0071] The display controller 56 then superimposes the ultrasound image 22 on a region of the operative field image 21 (Xin-Yin plane) that corresponds to the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF, thereby generating a superimposed image 26. In the examples of Figures 6 and 7, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). On the other hand, in the examples of Figures 8 and 9, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is not parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). Taking into account the difference in coordinate systems between the operative field image 21 and the ultrasound image 22, the display control unit 56 performs transformations such as projective transformation, affine transformation, movement, rotation, enlargement, and reduction on the ultrasound image 22, and then generates a superimposed image 26 that is superimposed on the operative field image 21.

[0072] The superimposed image 26 makes it easier to grasp the position and orientation of the ultrasonic transducer 14C even if it is difficult to visually recognize the position and orientation of the ultrasonic transducer 14C in the surgical field image 21 due to, for example, obstruction by an organ or the positional relationship with the camera 13B. Therefore, it becomes easier to adjust the position and orientation of the ultrasonic probe 14 so that a desired area (e.g., a tumor 27) is captured in the ultrasonic image 22, which contributes to improving the operability of the ultrasonic probe 14.

[0073] As shown in FIG. 2 , a margin region 22B may be added to the ultrasound image 22 to make the overall shape of the ultrasound image 22 rectangular. In this case, the display control unit 56 preferably generates the superimposed image 26 using the image from which the margin region 22B has been removed as an internal image. As described above, when the ultrasound probe 14 is a convex type, the ultrasound image 22 is fan-shaped. The margin region 22B is used to convert this fan-shaped ultrasound image 22 into a rectangular shape for standardization of image processing. In other words, the margin region 22B does not show information necessary for medical treatment, such as the internal structure of the target area in the ultrasound image 22. By not including the margin region 22B in the superimposed image 26, it is possible to prevent the surgical field image 21 from becoming difficult to view in the superimposed image 26. In FIG. 7 and other figures, the superimposed image 26 from which the margin region 22B has been removed is illustrated.

[0074] The display control unit 56 controls the display 16 to display the generated superimposed image 26. The display control unit 56 may also control the display 16 to display the operative field image 21 and the ultrasound image 22. In this case, the display control unit 56 may control the operative field image 21, the ultrasound image 22, and the superimposed image 26 to be displayed together on one display 16. In addition, when there are multiple displays 16, the display control unit 56 may control the operative field image 21, the ultrasound image 22, and the superimposed image 26 to be displayed on different displays 16. In addition, the display control unit 56 may receive designation of an image to be displayed on the display 16 from the operative field image 21, the ultrasound image 22, and the superimposed image 26, and control the designated image to be displayed on at least one display 16.

[0075] The operative field image 21 and the ultrasound image 22 are each output as a moving image. The display control unit 56 may control the display 16 to display at least one of the operative field image 21, the ultrasound image 22, and the superimposed image 26 in a live view. Live view display refers to displaying, as a moving image in real time, images generated at a predetermined frame rate based on signals output by an imaging device for imaging the target. The imaging device may include, for example, an image sensor included in the endoscope 13 that optically captures images of the target, and an ultrasound transducer 14C that uses ultrasound to image the target. When displaying the superimposed image 26 in a live view, the derivation unit 52 and the display control unit 56 repeatedly derive the position and orientation information and generate the superimposed image 26 each time the operative field image 21 and the ultrasound image 22 are acquired, or at predetermined time intervals.

[0076] However, as shown in FIG. 10 , if the ultrasound transducer 14C is not properly contacted with the target region (e.g., the liver LV), the ultrasound image 22 may not be an accurate image. For example, if the ultrasound transducer 14C is separated from the target region by air, the air reflects and attenuates the ultrasound, preventing the detection of ultrasound echoes from the target region. In this case, the ultrasound image 22 does not display information about the internal structure of the target region, and displaying the superimposed image 26 does not adequately assist in medical treatment. Furthermore, if the superimposed image 26 is displayed while the ultrasound probe 14 is being positioned, the blank ultrasound image 22 may interfere with the observation of the surgical field image 21, potentially reducing convenience.

[0077] Therefore, the medical support device 11 according to this embodiment has a function of changing the display mode of the internal image (e.g., ultrasound image 22) in the superimposed image 26 depending on the contact state or separation state of the medical instrument (e.g., ultrasound probe 14). The display mode refers to the position and orientation when the internal image is displayed in the superimposed image 26, as well as whether or not it is displayed. As described above, the basic display mode of the internal image in the superimposed image 26 is the position and orientation according to the imaging range of the internal image.

[0078] The determination unit 54 determines whether the medical instrument is in a contact state, in which it is in contact with the target site, or in a separated state, in which it is separated from the target site. Specifically, the determination unit 54 determines whether a predetermined portion of the medical instrument (e.g., the ultrasonic transducer 14C of the ultrasonic probe 14) is in contact with the target site. Note that the "contact state" in the present disclosure is not limited to a state in which the medical instrument and the target site are in close contact with each other, and may include a gap between the medical instrument and the target site that is generally acceptable in the technical field to which the technology of the present disclosure belongs.

[0079] The specific method for determining whether the device is in contact or separated is not particularly limited. Specific examples of the determination method are shown below, but the method is not limited to these. Any known method can be applied as a method for determining whether the device is in contact or separated. The determination unit 54 may combine some or all of the following determination methods as appropriate.

[0080] As an example, a sensor such as a contact sensor or a proximity sensor may be used to determine whether the medical device is in contact with the target site. For example, by providing a sensor on a part of the medical device that may come into contact with the target site, the sensor can determine whether the medical device is in contact with the target site. Examples of such sensors include a photosensor, a contact sensor using a piezoelectric element, a capacitance proximity sensor, and a microswitch.

[0081] As another example, the determination of whether the surgical field image 21 is in contact with the target site may be performed by image analysis based on the surgical field image 21. For example, the determination unit 54 may derive the distance between the target site and a portion of the medical instrument that may be in contact with the target site from the surgical field image 21, and determine the contact state if the distance is less than a predetermined threshold, or determine the separation state if the distance is equal to or greater than the predetermined threshold. Furthermore, the determination of whether the surgical field image 21 is in contact with the target site may be performed using AI technology employing a machine learning model. For example, a neural network model such as CNN may be used as such a machine learning model, which is trained in advance to input the surgical field image 21 and output a determination result of the contact state or separation state based on the input surgical field image 21. Furthermore, for example, the determination of whether the surgical field image 21 is in contact with the target site may be performed by inserting a camera separate from the endoscope 13 into the body and analyzing images captured by the separate camera in addition to or instead of the surgical field image 21.

[0082] As yet another example, when the medical instrument is an ultrasound probe 14 and the internal image is an ultrasound image 22, the determination of whether the state is in contact or in a separate state may be based on at least one of an electrical signal corresponding to an ultrasound echo detected by the ultrasound probe 14 and the ultrasound image 22. For example, the determination unit 54 may monitor the frequency components of the detected electrical signal over time, and determine that the state has switched between contact and separate when there is a change in the frequency components. Alternatively, for example, frequency components that can be detected in a contact state may be stored in advance, and the state may be determined to be in contact when the frequency components are included in the detected electrical signal.

[0083] For example, the determination unit 54 may monitor numerical information such as an average brightness value of the ultrasound image 22 over time, and determine that a switch between the contact state and the separation state has occurred when there is a change in the numerical information. For example, the determination unit 54 may detect a structure from the ultrasound image 22, and determine that a switch between the contact state and the separation state has occurred when there is a change in the shape of the structure over time (for example, appearance or disappearance). For example, numerical information and / or the shape of the structure that can be observed in the contact state may be stored in advance, and a contact state may be determined when the numerical information and / or the structure are included in the ultrasound image 22.

[0084] The display control unit 56 changes the display mode of the internal image in the superimposed image 26 depending on whether the medical instrument is in contact or distant. Specifically, the display control unit 56 controls the display so that the superimposed image 26 is displayed in the contact state. On the other hand, the display control unit 56 controls the display so that the surgical field image 21 is displayed instead of the superimposed image 26 in the distant state. For example, the display control unit 56 controls the display so that the superimposed image 26 shown in FIG. 7 is displayed in the contact state, and the surgical field image 21 shown in FIG. 11 is displayed in the distant state. According to this configuration, when the medical instrument is not in contact with the target site, the internal image is not displayed on the surgical field image 21, thereby avoiding interference with observation of the surgical field image 21.

[0085] The display controller 56 may also perform control to increase the visibility of the internal image in the superimposed image 26 over time from the point in time when the state is switched from the distant state to the contact state. Figure 12 shows how the visibility of the ultrasound image 22 in the superimposed image 26 changes during a period including the point in time Tp when the state is switched from the distant state to the contact state. In Figure 12, the visibility is schematically shown as the density of the ultrasound image 22, with darker colors indicating higher visibility. Figure 12 also shows the surgical field image 21 that is displayed in place of the superimposed image 26 in the distant state.

[0086] For example, the display control unit 56 may control the display to gradually increase the opacity of the internal image in the superimposed image 26 from time Tp when the state switches from the distant state to the contact state. That is, immediately after time Tp, the internal image may be displayed semi-transparently, and the opacity of the internal image may be gradually increased (the transparency may be decreased) to improve visibility. In this manner, if the medical staff ST erroneously applies the ultrasound probe 14 to the target area for only a short time, the ultrasound image 22 in the superimposed image 26 is displayed semi-transparently, thereby preventing the ultrasound image 22 from interfering with the surgical field image 21.

[0087] Furthermore, for example, the display control unit 56 may perform control to reduce noise in the internal image in the superimposed image 26 over time from time Tp when the state switches from the separated state to the contact state. For example, the display control unit 56 may obtain a clear image with less noise by extracting common (i.e., overlapping) portions in multiple ultrasound images 22 acquired consecutively in chronological order. In this case, as the time elapsed from time Tp increases and the number of samples increases, noise in the ultrasound image 22 is reduced and structures become clearer, thereby improving visibility. Note that the method for reducing noise is not limited to this, and for example, a known image processing technique for reducing noise may be applied when a predetermined period of time has elapsed since time Tp.

[0088] Furthermore, for example, the display control unit 56 may control the resolution of the internal image in the superimposed image 26 to increase over time from time Tp when the state switches from the distant state to the contact state. For example, the display control unit 56 may display the superimposed ultrasound image 22 at a lower resolution immediately after time Tp, and increase the resolution of the ultrasound image 22 as the time elapses from time Tp increases. In general, the resolution (resolution) of the ultrasound image 22 is a trade-off with penetration. By using a low frequency immediately after time Tp, the resolution is lowered to prioritize rough search and positioning of the tumor 27, and by increasing the frequency as the time elapses, the resolution is increased to enable more detailed positioning, thereby improving convenience.

[0089] In this embodiment, as will be described later, a tomographic image extracted from a three-dimensional image can also be used as the internal image. In this case, the display control unit 56 may increase the resolution when extracting the tomographic image from the three-dimensional image as the elapsed time from time Tp increases.

[0090] The display control unit 56 may apply a combination of the above-described methods for improving visibility. For example, the display control unit 56 may control the internal image in the superimposed image 26 so as to increase the opacity over time from the time Tp when the state switches from the distant state to the contact state, while reducing noise.

[0091] Next, the operation of the medical support device 11 according to this embodiment will be described with reference to Fig. 13. In the medical support device 11, the processor 41 executes the medical support program 44, thereby executing the medical support process shown in Fig. 13. This process is executed, for example, when a user issues an instruction to start execution via the reception device 46.

[0092] In step S10, the acquisition unit 50 acquires an operative field image obtained by optically capturing an operative field SF including the target site inside the body and the medical instrument inserted into the body using a camera. The acquisition unit 50 also acquires an internal image showing the internal structure of the target site. In step S12, the determination unit 54 determines whether the medical instrument is in a contact state where it is in contact with the target site, or a separated state where it is separated from the target site.

[0093] If the determination result in step S12 is a contact state, step S14 is affirmative, and the process proceeds to step S16. In step S16, the derivation unit 52 derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF based on the surgical field image acquired in step S10. In step S18, the display control unit 56 controls the display 16 to display a superimposed image in which an internal image is superimposed on the surgical field image acquired in step S10, with the display mode of the internal image adjusted based on the position and orientation information derived in step S16.

[0094] On the other hand, if the determination result in step S12 is the separated state, step S14 is negative, and the process proceeds to step S20. In step S20, the display control unit 56 controls the display of the operative field image acquired in step S10 instead of the superimposed image. When step S18 or step S20 is completed, this process ends.

[0095] As described above, the medical support device 11 according to this embodiment includes a processor 41. The processor 41 acquires an operative field image, which is an optical image of the operative field SF, including the target site within the body and the medical instrument inserted therein, captured by the camera 13B. The processor 41 also derives position and orientation information indicating the position and orientation of the medical instrument in the operative field SF based on the operative field image. The processor 41 also acquires an internal image indicating the internal structure of the target site. The processor 41 also determines whether the medical instrument is in contact with the target site or in a separated state, where the medical instrument is separated from the target site. The processor 41 also controls the display of a superimposed image, in which the internal image is superimposed on the operative field image, with the display mode of the internal image adjusted based on the position and orientation information, and further changes the display mode of the internal image in the superimposed image depending on the contact state or the separated state.

[0096] That is, according to the medical support device 11 of this embodiment, the support information displayed on the display 16 can be changed between the contact state and the distanced state. For example, by displaying the superimposed image 26 in the contact state, the correspondence between the surgical field image and the internal image can be made clear, while by hiding the internal image in the distanced state, it is possible to avoid interfering with the observation of the surgical field image 21. Therefore, convenience can be further improved.

[0097] In the above embodiment, the surgical field image 21 is displayed in the distant state, and the internal image is not displayed, but this is not limiting. The medical support device 11 may be configured to change the display mode of the internal image between the contact state and the distant state.

[0098] For example, the internal image in the superimposed image 26 may be displayed semi-transparently in the distant state, and may be displayed opaquely in the contact state. In this case, it is possible to prevent the ultrasound image 22 from interfering with the observation of the surgical field image 21, while making it easier to see that nothing is visible in the ultrasound image 22, i.e., that the ultrasound image 22 is in the distant state. Furthermore, since the display mode of the internal image in the superimposed image 26 changes depending on the position and orientation of the medical instrument, it may be easier to see the position and orientation of the medical instrument by always displaying the internal image in the superimposed image 26, even if nothing is visible.

[0099] [Second embodiment] Figure 14 shows a state in which, within the operative field SF in three-dimensional space, the direction of the axis AX of the tip 14B of the ultrasound probe 14 is further rotated around the Y axis from the attitude shown in Figure 8 and is nearly parallel to the imaging optical axis of the camera 13B. In this case, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF and the Xin-Yin plane of the operative field image 21 are nearly perpendicular to each other. Therefore, as shown in Figure 15, the ultrasound image 22 superimposed on the operative field image 21 has a nearly linear shape, making it difficult to observe.

[0100] FIG. 16 also shows a state in which the tip 14B of the ultrasonic probe 14 is rotated approximately 80 degrees around the X-axis from the orientation shown in FIG. 6 within the surgical field SF in three-dimensional space. In this case, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the surgical field SF and the Xin-Yin plane of the surgical field image 21 are nearly perpendicular to each other. Therefore, as shown in FIG. 17, the ultrasound image 22 superimposed on the surgical field image 21 has a nearly linear shape, making it difficult to observe. In this example, the tip 14B is located between the camera 13B and the ultrasound transducer 14C, and occlusion by the tip 14B also makes the ultrasound image 22 difficult to observe. In FIG. 17, the portions of the ultrasound transducer 14C and the ultrasound image 22 hidden by the tip 14B are indicated by dashed lines. The basic illustration methods of FIGS. 14 and 16 are the same as those of FIG. 6, and the basic illustration methods of FIGS. 15 and 17 are the same as those of FIG. 7.

[0101] In the superimposed image 26 shown in FIGS. 15 and 17 , it is difficult to observe both the ultrasound image 22 and the surgical field image 21. Therefore, the medical support device 11 according to this embodiment changes the display mode of the internal image according to the orientation of the internal image in the superimposed image 26, which changes according to the posture of the medical instrument. The functions and configuration of the medical support device 11 according to this embodiment will be described below. However, functions and configurations similar to those of the medical support device 11 according to the first embodiment will be denoted by the same reference numerals and detailed description will be omitted. Furthermore, the hardware configuration of the medical support device 11 according to this embodiment is similar to the hardware configuration of the medical support device 11 according to the first embodiment. Furthermore, the medical support device 11 according to this embodiment can be applied to the medical support system 10, just like the medical support device 11 according to the first embodiment.

[0102] The acquisition unit 50 acquires a surgical field image 21 obtained by optically capturing an operative field SF including a target site within the body and a medical instrument to be inserted into the body using the camera 13B. The acquisition unit 50 also acquires an internal image showing the internal structure of the target site. The internal image is, for example, an ultrasound image 22. The internal image is an example of a "first internal image" in the present disclosure.

[0103] The derivation unit 52 derives position and orientation information indicating the position and orientation of a medical instrument (for example, the ultrasound probe 14) in the surgical field SF based on the surgical field image 21.

[0104] The display control unit 56 controls the display of the superimposed image 26, which is an internal image superimposed on the surgical field image 21 and in which the display mode of the internal image is adjusted based on the position and orientation information. In this case, the display control unit 56 changes the display mode of the internal image according to the orientation of the internal image in the superimposed image 26, which changes according to the orientation of the medical instrument.

[0105] Specifically, the angle θ1 representing the orientation of the internal image is set to be the angle that is minimum when the imaging optical axis of camera 13B is parallel to the display surface of the internal image and maximum when the imaging optical axis is normal to the display surface of the internal image. The angle θ1 representing the orientation of the internal image in superimposed image 26 can be derived based on the position and orientation information and dimension information 45.

[0106] As an example, FIGS. 6, 8, 14, and 16 show the angle θ1 corresponding to each state of the ultrasound probe 14. In these examples, the angle θ1 representing the orientation of the ultrasound image 22 is the angle between the display surface (Xpb-Ypb plane) of the ultrasound image 22 and the Z axis, which is the imaging optical axis of the camera 13B. In FIG. 6, the Z axis is normal to the display surface of the ultrasound image 22, and the angle θ1 is at its maximum (90 degrees). Compared to this state, the angle θ1 is slightly smaller in FIG. 8, and is even smaller in FIGS. 14 and 16. Note that for ease of understanding, FIGS. 6, 8, 14, and 16 illustrate a straight line Zp that is parallel to the Z axis and passes through the display surface of the ultrasound image 22.

[0107] The display control unit 56 changes the display mode of the internal image depending on the angle θ1 indicating the orientation of the internal image. For example, when the ultrasound probe 14 is in the state shown in FIGS. 6 and 8 and the angle θ1 is equal to or greater than a predetermined threshold, the display control unit 56 may control the display to display the superimposed image 26. On the other hand, when the ultrasound probe 14 is in the state shown in FIGS. 14 and 16 and the angle θ1 is less than the threshold, the display control unit 56 may control the display to display the operative field image 21 instead of the superimposed image 26. For example, when the ultrasound probe 14 is in the state shown in FIG. 14, the display control unit 56 may display the operative field image 21 shown in FIG. 18 instead of the superimposed image 26 shown in FIG. 15.

[0108] The display control unit 56 may also record the position and orientation information of the medical instrument derived by the derivation unit 52 over time, and, when the angle θ1 is less than a threshold value, control the display of a movement trajectory 34 of the medical instrument on the operative field image 21 based on the position and orientation information recorded over time. For example, as shown in Fig. 19, the display control unit 56 may control the display of the movement trajectory 34 as a point cloud. This configuration makes it possible to clearly view the movement of the position and orientation of the ultrasound probe 14 in the operative field SF while avoiding the display of unnecessary ultrasound images 22 that are difficult to view due to the small angle θ1.

[0109] The display control unit 56 may also display only the movement trajectory 34 for a predetermined period from the current time, and may not display the movement trajectory 34 from before that period. The display control unit 56 may also control the visibility of the movement trajectory 34 to decrease over time. For example, the display control unit 56 may decrease the opacity (i.e., increase the transparency) of the movement trajectory 34 as the portion of the movement trajectory 34 becomes older, thereby decreasing the visibility of the older portion. For example, the display control unit 56 may decrease the visibility of the older portion of the movement trajectory 34 by using a lighter color or thinner lines as the portion of the movement trajectory 34 becomes older. For example, if the movement trajectory 34 is a point cloud, the display control unit 56 may decrease the visibility of the older portion of the movement trajectory 34 by decreasing the density of the point cloud or by decreasing the size of the points as the portion of the movement trajectory 34 becomes older.

[0110] The display control unit 56 may also perform control to display the movement trajectory 34 so that the portion where the medical instrument has come into contact with the target site and the portion where the medical instrument has not come into contact can be distinguished from each other. For example, the display control unit 56 may display only the portion of the movement trajectory 34 where the medical instrument has come into contact with the target site, or may highlight the portion where the medical instrument has come into contact with the target site. Whether the medical instrument has come into contact with the target site can be determined by the determination unit 54, as in the first embodiment.

[0111] Next, the operation of the medical support device 11 according to this embodiment will be described with reference to Fig. 20. In the medical support device 11, the processor 41 executes the medical support program 44, thereby executing the medical support process shown in Fig. 20. This process is executed, for example, when a user issues an instruction to start execution via the reception device 46.

[0112] In step S50, the acquisition unit 50 acquires an operative field image obtained by optically capturing an operative field SF including a target site within the body and a medical instrument to be inserted into the body using a camera. The acquisition unit 50 also acquires an internal image showing the internal structure of the target site. In step S52, the derivation unit 52 derives position and orientation information showing the position and orientation of the medical instrument in the operative field SF based on the operative field image acquired in step S10.

[0113] In step S54, the display control unit 56 determines whether the angle θ1, which is the orientation of the internal image acquired in step S50, is equal to or greater than a predetermined threshold. The angle θ1 is the angle that is minimum when the imaging optical axis of the camera 13B is parallel to the display surface of the internal image and maximum when the imaging optical axis is normal to the display surface. Specifically, the display control unit 56 derives the angle θ1 based on the position and orientation information derived in step S52 and the dimension information 45.

[0114] If the angle θ1 is equal to or greater than the threshold value and the result of step S54 is affirmative, the process proceeds to step S56. In step S56, the display control unit 56 controls the display 16 to display a superimposed image in which the internal image is superimposed on the operative field image acquired in step S50, and in which the display mode of the internal image has been adjusted based on the position and orientation information derived in step S52.

[0115] On the other hand, if the angle θ1 is less than the threshold value and the result of step S54 is negative, the process proceeds to step S58. In step S58, the display control unit 56 controls the display so that the operative field image acquired in step S50 is displayed instead of the superimposed image. When step S56 or step S58 is completed, this process ends.

[0116] As described above, the medical support device 11 according to this embodiment includes a processor 41. The processor 41 acquires an operative field image obtained by optically capturing an operative field SF, including a target region within the body and a medical instrument inserted into the body, using the camera 13B. The processor 41 also derives position and orientation information indicating the position and orientation of the medical instrument in the operative field SF based on the operative field image. The processor 41 also acquires a first internal image indicating the internal structure of the target region. The processor 41 also controls the display of a superimposed image in which the first internal image is superimposed on the operative field image, with the display mode of the first internal image adjusted based on the position and orientation information. The processor 41 also changes the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes depending on the orientation of the medical instrument.

[0117] That is, with the medical support device 11 according to this embodiment, the support information displayed on the display 16 can be changed depending on the orientation of the first internal image in the superimposed image. For example, when the orientation of the first internal image in the superimposed image is easy to see, the correspondence between the surgical field image and the internal image can be made clear by displaying the superimposed image 26. Furthermore, when the orientation of the first internal image in the superimposed image is difficult to see, the first internal image can be hidden to avoid interfering with the observation of the surgical field image 21. Therefore, convenience can be further improved.

[0118] In the above embodiment, when the angle θ1 is less than the threshold value, the operative field image 21 is displayed instead of the superimposed image 26, thereby hiding the internal image. However, the present invention is not limited to this. The medical support device 11 may change the display mode of the internal image depending on the orientation of the internal image in the superimposed image.

[0119] For example, the orientation of the internal image is defined as the front when the display surface (Xpb-Ypb plane) of the internal image is perpendicular to the Z axis, which is the imaging optical axis of camera 13B. In this case, if the orientation of the internal image determined based on the position and orientation information does not satisfy a preset condition, the display control unit 56 may control the orientation of the internal image to change to the front. The preset condition is, for example, when the angle θ1 is equal to or greater than a predetermined threshold value (e.g., 10 degrees).

[0120] For example, if the orientation of the internal image does not satisfy a preset condition, the display control unit 56 may superimpose a front-facing internal image at a position in the superimposed image 26 determined based on the position and orientation information. Figure 21 shows the superimposed image 26 in this case. In Figure 21, an ultrasound image 22 facing front is superimposed at a position based on the ultrasound transducer 14C in the surgical field image 21. According to this configuration, although the superimposed image 26 does not accurately reflect the position and orientation of the internal image in the surgical field SF, the visibility of the internal image is improved, which may improve convenience.

[0121] Furthermore, for example, if the orientation of the internal image does not satisfy a preset condition, the display control unit 56 may superimpose a front-facing internal image at a preset position on the periphery of the superimposed image 26. Figure 22 shows the superimposed image 26 in this case. In Figure 22, an ultrasound image 22 facing front is superimposed in an area 28 provided in the lower left of the superimposed image 26. In this configuration, although the superimposed image 26 does not accurately reflect the position and orientation of the internal image in the surgical field SF, the visibility of the internal image is improved, which may improve convenience.

[0122] Furthermore, for example, the display control unit 56 may perform control so that the internal image in the superimposed image 26 is displayed semi-transparently when the angle θ1 is less than a threshold value, and so that the internal image in the superimposed image 26 is displayed opaquely when the angle θ1 is equal to or greater than the threshold value. According to this configuration, for example, the position and orientation of the ultrasound probe 14 may be made easier to understand while preventing the ultrasound image 22 from interfering with observation of the operative field image 21.

[0123] For example, when the angle θ1 becomes less than a threshold value, the display control unit 56 may control the display of the superimposed image 26 generated immediately before the angle θ1 becomes less than a threshold value. FIG. 23 shows the superimposed image 26 in this case. In FIG. 23 , the ultrasound image 22 at the time when the angle θ1 becomes a threshold value (e.g., 10 degrees) is superimposed on the operative field image 21 at a position relative to the ultrasound transducer 14C. That is, the display control unit 56 may fix the shape of the ultrasound image 22 to be superimposed on the superimposed image 26 when the angle θ1 becomes less than the threshold value to the shape of the ultrasound image 22 at the time when the angle θ1 became the threshold value. This configuration may allow the superimposed image 26 to reflect the position and orientation of the internal image in the operative field SF to some extent while ensuring the visibility of the internal image, thereby improving convenience in some cases.

[0124] Next, modifications that can be applied to each of the above-described embodiments will be described. Note that the modifications described below can be combined in part or in whole as appropriate.

[0125] 24, the display control unit 56 may perform control based on the position and orientation information to display a scale 32 indicating the size of an internal structure (e.g., a tumor 27) included in an internal image (e.g., an ultrasound image 22) on the superimposed image 26. The size of the scale 32 can be calculated based on the distance from the camera 13B within the operative field SF to the display surface of the internal image (Xpb-Ypb plane) (i.e., the distance in the Z-axis direction parallel to the imaging optical axis). Note that, while the scale 32 is displayed on the ultrasound image 22 in FIG. 24, the present invention is not limited to this, and the scale 32 may be displayed at any position within the superimposed image 26.

[0126] 25 , based on the position and orientation information, the display control unit 56 may perform control so as to display a puncture path 30 of the puncture needle 18 inserted into the body using a medical instrument on the superimposed image 26. The puncture path 30 can be generated as an extension of the guide groove 29 based on the known positional relationship between the position indicated by the position and orientation information (e.g., the position of the reference point of the tip 14B of the ultrasound probe 14) and the guide groove 29.

[0127] The puncture path 30 is used as a guide when inserting the puncture needle 18 into the target position. For example, if the target position to be inserted with the puncture needle 18 is a tumor 27 in the liver LV, the medical staff ST positions the ultrasound probe 14 so that the puncture path 30 overlaps the tumor 27 in the ultrasound image 22. In this state, the puncture needle 18 is inserted into the tumor 27 using the puncture path 30 as a guide. In this way, by passing the puncture needle 18 through the guide groove 29 with the ultrasound probe 14 positioned so that the puncture path 30 passes through the target position, the puncture needle 18 can be made to reach the target position.

[0128] The display control unit 56 may display the puncture path 30 on the operative field image 21 in addition to or instead of displaying the puncture path 30 on the ultrasound image 22. In this case, the puncture path 30 indicates the path of the puncture needle 18 from the insertion position NP on the body surface BS of the patient PT to the guide groove 29 of the ultrasound probe 14.

[0129] [Third Modification] As shown in Fig. 26 , the display controller 56 may control the display so that the visibility of the internal image (e.g., ultrasound image 22) in the superimposed image 26 increases with increasing depth from the superficial layer to the deeper layer of the target region. In Fig. 26 , the visibility is schematically shown as the density of the ultrasound image 22, with darker colors indicating higher visibility. The same methods as those described in the first embodiment can be used to change the visibility of the internal image. For example, the display controller 56 may increase the opacity, reduce noise, or increase the resolution to increase the visibility.

[0130] Furthermore, the display control unit 56 may apply the visibility change process to all or part of the internal image. For example, the display control unit 56 may apply the visibility change process according to depth only to the peripheral part of the internal image, and then generate the superimposed image 26.

[0131] In this way, by changing the rendering attributes of the internal image according to the depth of the target region, the correspondence between the operative field image 21 and the internal image in the depth direction (Ypb axis direction) becomes intuitively easy to understand. In other words, even from the two-dimensional superimposed image 26, it becomes easy to imagine the three-dimensional position and orientation of the internal image in the operative field SF. Therefore, convenience can be improved.

[0132] 27 , the display control unit 56 may perform control to change the visibility of the internal image in the superimposed image 26 depending on the distance from the camera 13B to the medical instrument. For example, the display control unit 56 may perform control to increase the visibility of the internal image in the superimposed image 26 as the distance from the camera 13B to the medical instrument becomes shorter. Conversely, the display control unit 56 may perform control to increase the visibility of the internal image in the superimposed image 26 as the distance from the camera 13B to the medical instrument becomes longer.

[0133] 27, visibility is schematically shown as the density of the ultrasound image 22, with darker colors indicating higher visibility. The same methods as those described in the first embodiment can be applied to change the visibility of the internal image. For example, the display control unit 56 may increase the opacity, reduce noise, or increase the resolution to improve visibility.

[0134] Furthermore, the display control unit 56 may apply visibility change processing to the entire internal image or to part of the internal image. For example, the display control unit 56 may apply visibility change processing to only the peripheral portion of the internal image in accordance with the distance from the camera 13B to the medical instrument, and then generate the superimposed image 26.

[0135] In this way, by changing the rendering attributes of the internal images according to the distance from the camera 13B to the medical instrument, the correspondence between the surgical field image 21 and the internal images in the depth direction (Z-axis direction) becomes intuitively understandable. In other words, it becomes easier to imagine the three-dimensional position and orientation of the internal images in the surgical field SF even from the two-dimensional superimposed image 26. Therefore, convenience can be improved.

[0136] [Fifth Modification] In the above embodiments, the ultrasound image 22 is used as an example of an internal image showing the internal structure of a target region, but this is not limiting. In the medical support device 11, an internal image expressed in another way can be used in addition to or instead of the ultrasound image 22.

[0137] For example, a tomographic image of the target area extracted from a three-dimensional image taken by a tomography device such as a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device may be used as the internal image. Fig. 28 shows a superimposed image 26 in which a tomographic image 23 is superimposed on a surgical field image 21 instead of an ultrasound image 22.

[0138] Furthermore, for example, a vascular image showing the vascular structure of the target region extracted from a three-dimensional image captured by a tomography device such as a CT device may be used as the internal image. Fig. 29 shows a superimposed image 26 in which a vascular image 25 is superimposed on the surgical field image 21 in addition to the ultrasound image 22. When superimposing multiple internal images in this manner, the display control unit 56 may combine the internal images at a specific superimposition rate.

[0139] Specifically, prior to endoscopic surgery using the endoscope 13, ultrasound probe 14, etc., a three-dimensional image of the patient PT is captured in advance using a CT scanner or the like and stored in the storage 43, etc. The acquisition unit 50 acquires this three-dimensional image of the target area captured in advance. The display control unit 56 extracts an internal image from the three-dimensional image based on the position and orientation information. For example, the display control unit 56 extracts from the three-dimensional image a tomographic image 23 and / or a vascular image 25 that shows the internal structure in a predetermined region corresponding to the position and orientation of the medical instrument indicated by the position and orientation information. The display control unit 56 then generates a superimposed image 26 using the extracted internal image (the tomographic image 23 and / or the vascular image 25).

[0140] According to this embodiment, since internal images based on CT images or the like can be confirmed even during surgery, it becomes easier to grasp the internal structure in three dimensions, thereby improving convenience.

[0141] The display control unit 56 may also control the tomographic image 23 and the blood vessel image 25 so as to change the visibility according to the depth direction and / or the depth direction (see the third and fourth modifications). In Fig. 29, the visibility of the blood vessel image 25 as an internal image is changed according to the depth direction.

[0142] As described above, the medical support device 11 of the present disclosure can apply multiple types of internal images that are expressed in different ways, such as the ultrasound image 22, the tomographic image 23, and the blood vessel image 25. Therefore, the medical support device 11 may be able to switch the type of internal image to be displayed on the superimposed image 26.

[0143] Specifically, the acquisition unit 50 acquires a plurality of types of internal images expressed by different methods, including, for example, at least one of an ultrasound image 22, a tomographic image 23 extracted from a three-dimensional image (such as a CT image or an MRI image) of the target region, and a blood vessel image 25 showing the blood vessel structure of the target region extracted from the three-dimensional image of the target region.

[0144] The display control unit 56 receives, for example, via the reception device 46, a designation of the type of internal image to be included in the superimposed image 26. Then, the display control unit 56 generates the superimposed image 26 using the internal image of the designated type.

[0145] Furthermore, when two or more types of internal images are specified, the display control unit 56 may generate a superimposed image 26 in which each of the specified internal images is superimposed on the operative field image 21. For example, when an ultrasound image 22 and a blood vessel image 25 are specified, the display control unit 56 may generate a superimposed image 26 as shown in Fig. 29. In this case, the display control unit 56 may combine the internal images at a specific superimposition ratio, for example, by superimposing a semi-transparent tomographic image 23 on the ultrasound image 22.

[0146] [Seventh Modification] Furthermore, when a preset condition is satisfied, the medical support device 11 may perform control so as to switch the type of internal image to be displayed on the superimposed image 26. For example, in the first embodiment, the display control unit 56 may switch the type of internal image to be superimposed on the superimposed image 26 depending on whether the patient is in contact or separated.

[0147] Furthermore, for example, in the second embodiment, the internal image superimposed on the superimposed image 26 may be switched according to the angle θ1 representing the orientation of the internal image. Specifically, the display control unit 56 extracts, from the three-dimensional image, at least one of a tomographic image 23 and a blood vessel image 25 that show the internal structure of the target region using a representation method different from that of the ultrasound image 22, based on the position and orientation information. The ultrasound image 22 is an example of a first internal image of the present disclosure. At least one of the tomographic image 23 and the blood vessel image 25 is an example of a second internal image of the present disclosure.

[0148] When the angle θ1 is equal to or greater than a predetermined threshold, the display control unit 56 may perform control to display, as the superimposed image 26, a first superimposed image in which the tomographic image 23 and / or the vascular image 25 are superimposed on the operative field image 21 in addition to the ultrasound image 22. On the other hand, when the angle θ1 is less than the threshold, the display control unit 56 may perform control to display, as the superimposed image 26, a second superimposed image in which the tomographic image 23 and / or the vascular image 25 are superimposed on the operative field image 21 instead of the ultrasound image 22.

[0149] That is, the display control unit 56 may always display the tomographic image 23 and / or the blood vessel image 25 regardless of the angle θ1, and when the angle θ1 is equal to or greater than a threshold, may also display the ultrasound image 22. According to this configuration, when the ultrasound image 22 is easily visible, the combination of the ultrasound image 22 and the tomographic image 23 and / or the blood vessel image 25 makes it easier to grasp the internal structure in more detail. Furthermore, when the ultrasound image 22 is difficult to view, the ultrasound image 22 is hidden, thereby avoiding interference with observation of the surgical field image 21, and the tomographic image 23 and / or the blood vessel image 25 can assist in grasping the internal structure.

[0150] As in the third modified example, the display control unit 56 may perform control to change the visibility of the tomographic image 23 and / or the blood vessel image 25 in at least one of the first superimposed image and the second superimposed image depending on the depth in the direction from the superficial layer to the deep layer of the target region. Furthermore, whether or not to change the visibility of the tomographic image 23 and / or the blood vessel image 25 may be switched depending on whether the first superimposed image or the second superimposed image is displayed (i.e., whether the ultrasound image 22 is displayed in a superimposed manner).

[0151] Furthermore, similarly to the fourth modification, the display control unit 56 may perform control to change the visibility of the tomographic image 23 and / or the blood vessel image 25 in at least one of the first superimposed image and the second superimposed image depending on the distance from the camera 13B to the medical instrument. Furthermore, whether or not to change the visibility of the tomographic image 23 and / or the blood vessel image 25 may be switched depending on whether the first superimposed image or the second superimposed image is being displayed (i.e., whether the ultrasound image 22 is being displayed in a superimposed manner).

[0152] 30 and 31 , the medical support device 11 may receive a designation of a target coordinate T in the internal image, and perform control so as to display information indicating the position of the target coordinate T on the superimposed image 26. In FIGS. 30 and 31 , the target coordinate T is indicated by a star. FIG. 30 shows a state in which the ultrasonic transducer 14C is positioned near the center of the surgical field image 21. FIG. 31 shows a state in which the ultrasonic transducer 14C has moved in the direction of the Xin axis (toward the right on the paper) from the state in FIG. 30 , changing the imaging range of the internal image.

[0153] The position of the target coordinate T is represented by three-dimensional position coordinates (Xt, Yt, Zt) in the surgical field SF. Therefore, for example, even if the ultrasonic transducer 14C moves from the state shown in FIG. 30 to the state shown in FIG. 31, the position of the target coordinate T does not change. The target coordinate T can be specified, for example, by the medical staff ST inputting a point on the internal image via the reception device 46. By making it possible to display such target coordinate T on the superimposed image 26, convenience can be improved.

[0154] The display control unit 56 may also switchably control whether or not to display information indicating the position of the target coordinates T on the superimposed image 26. For example, the medical staff member ST may specify, via the reception device 46, whether or not to display the information indicating the position of the target coordinates T.

[0155] Furthermore, for example, when a preset condition is satisfied, the display control unit 56 may perform control so that information indicating the position of the target coordinates T is displayed on the superimposed image 26. For example, in the first embodiment, the display control unit 56 may switch whether or not to display the information indicating the position of the target coordinates T depending on the contact state or the separation state. Furthermore, for example, in the second embodiment, the display control unit 56 may switch whether or not to display the information indicating the position of the target coordinates T depending on the orientation of the internal image.

[0156] Furthermore, when the type of internal image included in the superimposed image 26 is changed as in the sixth and seventh modifications, the display control unit 56 may perform control so that information indicating the position of the target coordinate T is displayed on the superimposed image 26 both before and after the change. For example, suppose that after the target coordinate T is specified in the tomographic image 23 on the superimposed image 26 (see FIG. 30 ), the internal image to be included in the superimposed image 26 is changed to the ultrasound image 22. In this case, the display control unit 56 may also display information (a star mark) indicating the position of the target coordinate T on the ultrasound image 22, as shown in FIG. 32 .

[0157] Further, for example, the display control unit 56 may display information indicating the position of the target coordinates T on the surgical field image 21, not limited to the internal image.

[0158] In the above-described embodiments, the marker M is attached to the tip 14B of the insertion section 14A of the ultrasonic probe 14, but this is not limiting. The marker M may be attached to a portion of the medical instrument that is inserted into the body of the patient PT, and may be attached, for example, to the middle of the insertion section 14A or to the base end side of the insertion section 14A.

[0159] In addition, in each of the above embodiments, the position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF is derived using the marker M, but this is not limiting. For example, the position and orientation information may be derived by detecting the characteristic shape of the medical instrument from the surgical field image 21 by image analysis.

[0160] In each of the above embodiments, the medical instrument inserted into the body of the patient PT is an ultrasound probe 14 (an example of a medical probe) capable of observing the internal structure of an organ. Puncturing an internal organ with a puncture needle 18 is often performed using a medical probe capable of observing the internal structure of the organ. Therefore, the technology of the present disclosure is particularly effective when a medical probe is used as the medical instrument, as in the above embodiments.

[0161] Furthermore, the ultrasound probe 14 is relatively often used in combination with the puncture needle 18. Therefore, the technology of the present disclosure is even more effective when the ultrasound probe 14 is used as a medical probe. Note that the medical probe capable of observing the internal structure of an organ may be other than the ultrasound probe 14, and may be, for example, an optical coherence tomography (OCT) probe or the like.

[0162] Note that the medical instrument may be anything other than a medical probe capable of observing the internal structure of an organ. For example, the medical instrument may be a treatment instrument that does not have the function of observing the internal structure and only has a guide groove 29 for the puncture needle 18 at its tip. For example, if a tumor is present on the surface of an organ and the puncture needle 18 is to be inserted into the surface tumor, even a treatment instrument that does not have the function of observing the internal structure can properly guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST aligns the guide groove 29 of the treatment instrument with a position corresponding to the tumor on the surface of the organ and then inserts the puncture needle 18 into the tumor through the guide groove 29.

[0163] Furthermore, the medical instrument may be a simple rod-like treatment instrument without a guide groove 29. If a tumor is present on the surface of an organ, even such a treatment instrument can be used to indicate the tumor. Simply superimposing the puncture path 30 on the surgical field image 21 in which the treatment instrument has indicated a tumor on the surface of the organ can serve as a guide for checking the puncture direction of the puncture needle 18, etc. Therefore, the technology of the present disclosure is effective even if the medical instrument does not have a guide groove 29.

[0164] In addition, in each of the above embodiments, cauterization has been described as an example of the function of the puncture needle 18, but the function of the puncture needle 18 is not limited to this. In addition, although the puncture needle 18 has been described as an example of a treatment tool, other than the puncture needle 18, treatment tools for injecting a fluorescent agent such as ICG (Indocyanine Green), biopsy needles and forceps used for collecting tissue for biopsy, etc. may also be applied.

[0165] In addition, in the above embodiments, the inside of the body has been described using body cavities such as the abdominal cavity and the thoracic cavity as examples, but the inside of the body may also be the inside of an upper digestive tract such as the esophagus, a lower digestive tract such as the intestines, or a duct such as the bronchi. When the technology of the present disclosure is applied to a surgical field inside a duct, for example, a marker M is provided at the base end of a flexible endoscope that is inserted into the duct.

[0166] Furthermore, in each of the above embodiments, the following various processors can be used as the hardware structure of the processing units that perform various processes, such as the acquisition unit 50, the derivation unit 52, the determination unit 54, and the display control unit 56. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0167] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0168] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0169] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0170] In addition, in each of the above embodiments, the medical support program 44 is described as being pre-stored (installed) in the storage 43, but this is not limiting. The medical support program 44 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The medical support program 44 may also be downloaded from an external device via a network.

[0171] The present disclosure can also be applied to programs and program products. Specifically, the medical support program 44 in the above embodiment may be provided as a program product. The program product includes any type of product for providing a program. For example, the program product includes a program provided via a network such as the Internet, as well as a computer-readable recording medium that non-temporarily stores a program.

[0172] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0173] The following supplementary notes are further disclosed with respect to each of the above embodiments. [Supplementary Note 1] A medical support device comprising a processor, the processor: acquires a surgical field image optically captured by a camera of a surgical field including a target site within the body and a medical instrument to be inserted into the body; derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquires a first internal image showing an internal structure of the target site; controls display of a superimposed image in which the first internal image is superimposed on the surgical field image, the superimposed image having a display manner of the first internal image adjusted based on the position and orientation information; and changes the display manner of the first internal image according to an orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument. [Supplementary Note 2] The medical support device according to Supplementary Note 1, wherein the processor changes the display manner of the first internal image according to an angle representing the orientation of the first internal image, the angle being the smallest when the imaging optical axis of the camera is parallel to the display surface of the first internal image and the largest when the imaging optical axis is normal to the display surface of the first internal image. [Supplementary Note 3] The medical support device according to Supplementary Note 2, wherein the processor controls to display the superimposed image when the angle is equal to or greater than a predetermined threshold, and controls to display the surgical field image instead of the superimposed image when the angle is less than the threshold. [Supplementary Note 4] The medical support device according to Supplementary Note 3, wherein the processor records the position and orientation information of the medical instrument over time, and when the angle is less than the threshold, controls to display a movement trajectory of the medical instrument on the surgical field image based on the position and orientation information recorded over time. [Supplementary Note 5] The medical support device according to Supplementary Note 4, wherein the processor controls to decrease the visibility of the movement trajectory over time. [Supplementary Note 6] The medical support device according to Supplementary Note 4 or Supplementary Note 5, wherein the processor controls to display the movement trajectory as a point cloud.[Supplementary Note 7] The medical support device of any one of Supplementary Notes 1 to 6, wherein, with respect to the orientation of the first internal image, the front is defined as the state where the display surface of the first internal image and the imaging optical axis of the camera are orthogonal to each other, and the processor changes the orientation of the first internal image to the front side if the orientation of the first internal image determined based on the position and orientation information does not satisfy a preset condition. [Supplementary Note 8] The medical support device of Supplementary Note 7, wherein, if the orientation of the first internal image does not satisfy the condition, the processor superimposes the first internal image facing the front side at a position in the superimposed image determined based on the position and orientation information. [Supplementary Note 9] The medical support device of Supplementary Note 7, wherein, if the orientation of the first internal image does not satisfy the condition, the processor superimposes the first internal image facing the front side at a preset position on the periphery of the superimposed image. [Supplementary Note 10] The medical support device according to any one of Supplementary Notes 1 to 9, wherein the medical instrument is an ultrasound probe that transmits ultrasound to the target site and detects an electrical signal corresponding to an ultrasound echo reflected from the target site, and the first internal image is an ultrasound image generated in response to the electrical signal. [Supplementary Note 11] The medical support device according to Supplementary Note 10, wherein the processor generates the superimposed image by using, as the first internal image, an image obtained by removing a margin area in the ultrasound image that does not show the internal structure of the target site. [Supplementary Note 12] The medical support device described in Supplementary Note 2, wherein the processor: acquires a three-dimensional image of the target area; extracts from the three-dimensional image a second internal image showing the internal structure of the target area using a representation method different from that of the first internal image based on the position and orientation information; when the angle is equal to or greater than a predetermined threshold, controls to display, as the superimposed image, a first superimposed image in which the second internal image is superimposed on the surgical field image in addition to the first internal image; and when the angle is less than the threshold, controls to display, as the superimposed image, a second superimposed image in which the second internal image is superimposed on the surgical field image instead of the first internal image.[Supplementary Note 13] The medical support device according to Supplementary Note 12, wherein the processor controls to change the visibility of the second internal image in at least one of the first superimposed image and the second superimposed image depending on the distance from the camera to the medical instrument. [Supplementary Note 14] The medical support device according to Supplementary Note 12 or Supplementary Note 13, wherein the processor controls to change the visibility of the second internal image in at least one of the first superimposed image and the second superimposed image depending on the depth of the target region in a direction from the superficial layer to the deep layer. [Supplementary Note 15] The medical support device according to any one of Supplementary Notes 12 to 14, wherein the second internal image is a tomographic image of the target region. [Supplementary Note 16] The medical support device according to any one of Supplementary Notes 12 to 15, wherein the second internal image is a vascular image showing the vascular structure of the target region. [Supplementary Note 17] The medical support device according to any one of Supplementary Notes 1 to 16, wherein the processor controls, based on the position and orientation information, to display, on the superimposed image, a scale indicating the size of the internal structure included in the first internal image. [Supplementary Note 18] The medical support device according to any one of Supplementary Notes 1 to 17, wherein the processor controls, based on the position and orientation information, to display, on the superimposed image, a puncture path of a puncture needle inserted into the body using the medical instrument. [Supplementary Note 19] A medical support method in which a computer executes the following processes: acquiring a surgical field image obtained by optically photographing a surgical field including a target area inside the body and a medical instrument to be inserted into the body using a camera; deriving position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquiring a first internal image indicating the internal structure of the target area; controlling the display of a superimposed image in which the first internal image is superimposed on the surgical field image, the superimposed image having a display mode of the first internal image adjusted based on the position and orientation information; and changing the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes depending on the orientation of the medical instrument.[Supplementary Note 20] A medical support program that causes a computer to execute the following processes: acquiring a surgical field image obtained by optically photographing a surgical field including a target area inside the body and a medical instrument to be inserted into the body using a camera; deriving position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquiring a first internal image indicating the internal structure of the target area; controlling the display of a superimposed image in which the first internal image is superimposed on the surgical field image, the superimposed image having a display mode of the first internal image adjusted based on the position and orientation information; and changing the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes depending on the orientation of the medical instrument.

[0174] The disclosure of Japanese Patent Application No. 2024-114346, filed on July 17, 2024, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A medical support device comprising a processor that: acquires an operative field image optically photographed by a camera of a operative field including a target area inside the body and a medical instrument to be inserted into the body; derives position and orientation information indicating the position and orientation of the medical instrument in the operative field based on the operative field image; acquires a first internal image showing the internal structure of the target area; controls the display of a superimposed image in which the first internal image is superimposed on the operative field image, the superimposed image having the display mode of the first internal image adjusted based on the position and orientation information; and further changes the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

2. The medical support device of claim 1, wherein the processor changes the display mode of the first internal image according to an angle representing the orientation of the first internal image, the angle at which the camera's imaging optical axis is least likely to be parallel to the display surface of the first internal image and the angle at which the imaging optical axis is most likely to be normal to the display surface of the first internal image.

3. The medical support device according to claim 2, wherein the processor controls the superimposed image to be displayed when the angle is equal to or greater than a predetermined threshold, and controls the surgical field image to be displayed instead of the superimposed image when the angle is less than the threshold.

4. The medical support device according to claim 3, wherein the processor records the position and orientation information of the medical instrument over time, and when the angle is less than the threshold value, controls the display of the movement trajectory of the medical instrument on the surgical field image based on the position and orientation information recorded over time.

5. The medical support device according to claim 4, wherein the processor controls the visibility of the movement trajectory to decrease over time.

6. The medical support device according to claim 4, wherein the processor controls the display of the movement trajectory as a point cloud.

7. The medical support device of claim 1, wherein, with respect to the orientation of the first internal image, the front is defined as the state in which the display surface of the first internal image and the optical axis of the camera are perpendicular to each other, and the processor changes the orientation of the first internal image to the front side if the orientation of the first internal image determined based on the position and orientation information does not satisfy a predetermined condition.

8. The medical support device according to claim 7, wherein, if the orientation of the first internal image does not satisfy the condition, the processor superimposes the first internal image facing forward at a position within the superimposed image determined based on the position and orientation information.

9. The medical support device of claim 7, wherein the processor superimposes the first internal image facing forward at a predetermined position on the periphery of the superimposed image if the orientation of the first internal image does not satisfy the condition.

10. The medical support device according to claim 1, wherein the medical instrument is an ultrasonic probe that transmits ultrasonic waves to the target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area, and the first internal image is an ultrasonic image generated in response to the electrical signals.

11. The medical support device according to claim 10, wherein the processor generates the superimposed image by using an image obtained by removing blank areas in the ultrasound image that do not show the internal structure of the target area as the first internal image.

12. The medical support device of claim 2, wherein the processor: acquires a three-dimensional image of the target area; extracts from the three-dimensional image, based on the position and orientation information, a second internal image showing the internal structure of the target area using a representation method different from that of the first internal image; and, when the angle is equal to or greater than a predetermined threshold, controls to display, as the superimposed image, a first superimposed image in which the second internal image is superimposed on the surgical field image in addition to the first internal image; and, when the angle is less than the threshold, controls to display, as the superimposed image, a second superimposed image in which the second internal image is superimposed on the surgical field image instead of the first internal image.

13. The medical support device according to claim 12, wherein the processor controls to change the visibility of the second internal image in at least one of the first superimposed image and the second superimposed image depending on the distance from the camera to the medical instrument.

14. The medical support device described in claim 12, wherein the processor controls to change the visibility of the second internal image in at least one of the first superimposed image and the second superimposed image depending on the depth from the surface layer to the deep layer of the target area.

15. The medical support device according to claim 12, wherein the second internal image is a tomographic image of the target area.

16. The medical support device according to claim 12, wherein the second internal image is a vascular image showing the vascular structure of the target area.

17. The medical support device according to claim 1, wherein the processor controls the display of a scale indicating the size of the internal structure included in the first internal image on the superimposed image based on the position and orientation information.

18. The medical support device according to claim 1, wherein the processor controls the display of the puncture path of a puncture needle inserted into the body using the medical instrument on the superimposed image based on the position and orientation information.

19. A medical support method in which a computer executes the following processes: acquiring a surgical field image optically photographed by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body; deriving position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquiring a first internal image showing the internal structure of the target area; controlling the display of a superimposed image in which the first internal image is superimposed on the surgical field image, the superimposed image having the display mode of the first internal image adjusted based on the position and orientation information; and changing the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

20. A medical support program that causes a computer to perform the following processes: acquiring a surgical field image optically photographed by a camera of a surgical field including a target area inside the body and a medical instrument to be inserted into the body; deriving position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the surgical field image; acquiring a first internal image showing the internal structure of the target area; controlling the display of a superimposed image in which the first internal image is superimposed on the surgical field image, the superimposed image having the display mode of the first internal image adjusted based on the position and orientation information; and changing the display mode of the first internal image in accordance with the orientation of the first internal image in the superimposed image, which changes according to the orientation of the medical instrument.

Images