Medical support device, medical support method, and medical support program

Abstract

To provide a medical support device, a medical support method, and a medical support program capable of supporting observation of an internal image.SOLUTION: The medical support device 11 includes a processor, and the processor acquires a three dimensional model indicating a surface shape of a target site in a body, acquires an internal image indicating an internal structure of the target site, acquires position / posture information indicating a position and a posture of a medical instrument in a surgical field including the target site and the medical instrument inserted into the body, and controls display of a first superimposed image in which the internal image is superimposed on the three dimensional model and a display mode of the internal image is adjusted based on the position / posture information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a medical support device, a medical support method, and a medical support program. [Background technology]

[0002] Conventionally, there is known a technique for identifying the position and orientation of a medical instrument inserted into a body and supporting medical treatment such as surgery and examination based on the identified position and orientation of the medical instrument. For example, Patent Document 1 discloses a technique for estimating the position and orientation of an ultrasound probe by performing image analysis on an optically captured surgical field image including an ultrasound probe, and for displaying preparation information corresponding to the estimated position and orientation by superimposing it on the surgical field image.

[0003] Furthermore, for example, Patent Document 2 discloses a method of acquiring a first image and a second image of a marker provided on an object from different viewpoints, detecting the area of ​​the marker from each of the images, and acquiring the three-dimensional coordinates of the marker by a stereo method. Furthermore, for example, Non-Patent Document 1 discloses a method of acquiring parallax images by attaching two cameras to the tip of a rigid endoscope, recognizing the depth of the organ surface by a stereo method, and reconstructing a three-dimensional model of the organ surface. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 162657 [Patent Document 2] Patent Publication No. 2021-085751 [Non-patent literature]

[0005] [Non-Patent Document 1] Stoyanov, et.al., "Dense 3D Depth Recovery for Soft Tissue Deformation During Robotically Assisted Laparoscopic Surgery", Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2004. Summary of the Invention [Problem to be solved by the invention]

[0006] In laparoscopic surgery, the internal structure is grasped by observing the surgical field image optically captured by the endoscope and the ultrasound image while adjusting the position and orientation of the ultrasound probe in the surgical field. However, because the surgical field image and the ultrasound image have different viewpoints, imaging ranges, and drawing methods, it is difficult to appropriately adjust the position and orientation of the ultrasound probe by comparing them, and this has required advanced skills.

[0007] The present disclosure provides a medical support device, a medical support method, and a medical support program that can support the observation of internal images. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a medical support device comprising a processor, which acquires a three-dimensional model showing the surface shape of a target area inside the body, acquires an internal image showing the internal structure of the target area, acquires position and orientation information showing the position and orientation of a medical instrument in a surgical field including the target area and the medical instrument to be inserted into the body, and controls the display of a first superimposed image in which the internal image is superimposed on the three-dimensional model, with the display mode of the internal image adjusted based on the position and orientation information.

[0009] The processor may acquire an operative field image optically photographed by a camera, and if a preset condition regarding the display of the first superimposed image is met, control the display of the first superimposed image; if the condition is not met, control the display of a second superimposed image in which an internal image is superimposed on the operative field image instead of the first superimposed image, the second superimposed image having the display mode of the internal image adjusted based on the position and orientation information.

[0010] The processor may determine that the condition is met if the angle representing the orientation of the internal image, which is the smallest when the camera's shooting optical axis is parallel to the display surface of the internal image and the largest when the shooting optical axis is normal to the display surface of the internal image, is less than a predetermined threshold.

[0011] The processor may determine that the condition is satisfied when a user instructs to display the first superimposed image.

[0012] The processor may control information indicating the position of the camera to be displayed on the first superimposed image.

[0013] The camera may be a camera provided at the tip of an endoscope used in laparoscopic surgery.

[0014] Markers configured with optically detectable patterns may be provided on the outer peripheral surface of the medical instrument, and the processor may derive the position and orientation information based on the markers included in the surgical field image.

[0015] The processor may receive a specification of the orientation of the three-dimensional model in the first superimposed image, and generate the first superimposed image based on the specified orientation and position and orientation information of the three-dimensional model.

[0016] With regard to the orientation of the internal image, if the viewpoint of the first superimposed image is located on a perpendicular line to the display surface of the internal image, the front is defined as the front, and the processor may specify the orientation of the three-dimensional model so that the orientation of the internal image determined based on the position and orientation information is the front side.

[0017] The processor may control the three-dimensional model included in the first superimposed image so as to increase the transparency of a portion located between the viewpoint of the first superimposed image and the display surface of the internal image.

[0018] The medical instrument may be an ultrasound probe that transmits ultrasound waves to a target area and detects electrical signals corresponding to ultrasound echoes reflected from the target area, and the internal image may be an ultrasound image generated in response to the electrical signals.

[0019] The processor may acquire a three-dimensional image including the internal structure of the target area, extract a second internal image corresponding to the display surface of the ultrasound image from the three-dimensional image based on the position and orientation information, and generate a first superimposed image by superimposing the ultrasound image and the second internal image on the three-dimensional model.

[0020] The second internal image may be a tomographic image of the target area that includes the display surface of the ultrasound image.

[0021] The second internal image may be a blood vessel image showing the vascular structure of the target area passing through the display plane of the ultrasound image.

[0022] The processor may generate the first superimposed image by using, as the internal image, an image obtained by removing a margin area in the ultrasound image that does not show the internal structure of the target region.

[0023] The processor may acquire a three-dimensional image of the target region and extract an internal image from the three-dimensional image based on the position and orientation information.

[0024] The processor may acquire the three-dimensional model using a Time of Flight (ToF) camera.

[0025] The processor may acquire the three-dimensional model using a stereo camera.

[0026] A second aspect of the present disclosure is a medical support method in which a computer executes a process of acquiring a three-dimensional model showing the surface shape of a target part inside the body, acquiring an internal image showing the internal structure of the target part, acquiring position and orientation information showing the position and orientation of a medical instrument in a surgical field including the target part and the medical instrument to be inserted into the body, and controlling the display of a first superimposed image in which the internal image is superimposed on the three-dimensional model, and the display mode of the first superimposed image is adjusted based on the position and orientation information.

[0027] A third aspect of the present disclosure is a medical support program that causes a computer to execute a process of acquiring a three-dimensional model showing the surface shape of a target part inside the body, acquiring an internal image showing the internal structure of the target part, acquiring position and orientation information showing the position and orientation of a medical instrument in a surgical field including the target part and the medical instrument to be inserted into the body, and controlling the display of a first superimposed image in which the internal image is superimposed on the three-dimensional model, and in which the display mode of the internal image is adjusted based on the position and orientation information. [Effects of the Invention]

[0028] According to the above aspects, the medical support device, medical support method, and medical support program of the present disclosure can support the observation of internal images. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram illustrating an outline of a medical support system including a medical support device. [Figure 2] 1 is a diagram showing the state of the body during laparoscopic surgery. FIG. [Figure 3] 10A and 10B are diagrams showing the insertion state of the puncture needle guided by the guide groove. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a medical support device. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a medical support device. [Figure 6] 10A and 10B are diagrams illustrating the relationship between the positions and orientations of a marker and an ultrasound probe. [Figure 7] FIG. 10 is a diagram illustrating an example of a superimposed image. [Figure 8] 10A and 10B are diagrams illustrating the relationship between the positions and orientations of a marker and an ultrasound probe. [Figure 9] FIG. 10 is a diagram illustrating an example of a superimposed image. [Figure 10] FIG. 10 is a diagram illustrating an example of a three-dimensional superimposed image. [Figure 11] FIG. 10 is a diagram illustrating an example of a three-dimensional superimposed image. [Figure 12] 10A and 10B are diagrams illustrating the relationship between the positions and orientations of a marker and an ultrasound probe. [Figure 13] FIG. 10 is a diagram illustrating an example of a superimposed image. [Figure 14] FIG. 10 is a diagram illustrating an example of a three-dimensional superimposed image. [Figure 15] FIG. 10 is a diagram illustrating an example of a three-dimensional superimposed image. [Figure 16] FIG. 10 is a diagram illustrating an example of a method for generating a three-dimensional model. [Figure 17] FIG. 10 is a diagram for explaining another example of a method for generating a three-dimensional model. [Figure 18] 10 is a flowchart illustrating an example of medical support processing. [Figure 19] 10 is a flowchart illustrating an example of a three-dimensional model generation process. DETAILED DESCRIPTION OF THE INVENTION

[0030] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are designated by the same reference numerals, and duplicate descriptions will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0031] An example of a medical support system 10 to which a medical support device 11 according to this embodiment is applied will be described with reference to FIGS. 1 to 3. As an example, the medical support system 10 is used when performing endoscopic surgery on a patient PT using an endoscope 13. Unlike open surgery, endoscopic surgery is a surgery performed by drilling a small hole in the patient PT's body and inserting medical instruments such as the endoscope 13 through the hole. The medical support system 10 not only provides medical staff ST, including doctors, with a view of the surgical field inside the patient PT's body, but also provides support information to support medical treatment such as surgery and examinations. As will be described later, the support information may be a superimposed image 26 for displaying an ultrasound image 22 superimposed on a surgical field image 21. Because such a medical support system 10 has the function of providing support information in real time during surgery, it is also referred to as a surgical navigation system.

[0032] 1, the medical support system 10 includes a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is connected to the endoscope 13, the ultrasound probe 14, and the display 16 so as to be able to communicate with each other.

[0033] 2 shows the state in which an endoscope 13 and an ultrasound probe 14 are being inserted into the abdomen of a patient PT. In endoscopic surgery, a portion of the endoscope 13 and the ultrasound probe 14, including their respective tips, is inserted into the body via a trocar 17. The trocar 17 is an insertion tool that has an insertion hole through which the endoscope 13 and the like are inserted and a valve that is provided within the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and therefore the trocar 17 is used to insert the endoscope 13, the ultrasound probe 14, and the like into the body.

[0034] The puncture needle 18 is a treatment tool that punctures a lesion, such as a tumor, contained in an organ. Specifically, the puncture needle 18 has a needle portion 18A and a grip portion 18B provided on the proximal end side of the needle portion 18A. The puncture needle 18 is, for example, a cauterization puncture needle used to cauterize a lesion. The cauterization puncture needle has an electrode to which a high-frequency voltage is applied at its tip. When the high-frequency voltage is applied while the electrode is inserted into the lesion, the heat generated by the electrode necrotizes the lesion. In this embodiment, as an example, a tumor 27 in the liver LV is visualized using an ultrasound image 22, and the visualized tumor 27 is cauterized with the puncture needle 18, thereby performing a treatment to necrotize the tumor 27.

[0035] The endoscope 13 optically captures an operative field SF including a target site (liver LV in this example) inside the body of the patient PT using a camera 13B. The operative field SF is a space extending inside a body cavity defined by organs and the body wall inside the body. Specifically, the endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT. A camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)) are built into the tip of the insertion section 13A. As an example, the insertion section 13A of the endoscope 13 is a rigid endoscope, and is also called a laparoscope because it is often used to observe the abdominal cavity.

[0036] The camera 13B has an image sensor, such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images. The endoscope 13 is connected to an image processing processor for the endoscope (not shown). This image processing processor performs signal processing on the image signal output by the image sensor to generate an operative field image 21 of the operative field SF inside the body. The operative field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via the image processing processor for the endoscope. In FIG. 2, the symbols Xin and Yin indicate the coordinate system of the operative field image 21. The camera 13B is an example of the "camera" and "camera provided at the tip of an endoscope used in endoscopic surgery" in the present disclosure.

[0037] The illumination light for the endoscope 13 is, for example, visible light such as white light. Alternatively, special light such as ultraviolet light or infrared light may be used as the illumination light for the endoscope 13. The special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet range. The surgical field image 21 is an image of the surgical field SF illuminated by the illumination light, and more specifically, is an image based on light reflected from the surface of the surgical field SF. Therefore, while the surgical field image 21 can depict structures present near the surface of the target area, it is difficult to observe the internal structure.

[0038] The ultrasonic probe 14 transmits ultrasonic waves to a target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area. Specifically, the ultrasonic probe 14 has an insertion section 14A that is inserted into the body of the patient PT and an operation section 14D on the proximal end side of the insertion section 14A. An ultrasonic transducer 14C is built into a distal end section 14B of the insertion section 14A. The ultrasonic probe 14 is an example of a "medical instrument" of the present disclosure.

[0039] The ultrasonic transducer 14C transmits ultrasonic waves to the target area and receives ultrasonic echoes reflected from the target area. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown). This image processing processor performs image reconstruction processing based on electrical signals corresponding to the ultrasonic echoes received by the ultrasonic transducer 14C. The image reconstruction processing generates an ultrasonic image 22 showing the internal structure of the target area scanned by the ultrasonic probe 14. The ultrasonic image 22 captured by the ultrasonic probe 14 is transmitted to the medical support device 11 in real time via the image processing processor for the ultrasonic probe. In FIG. 2, symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0040] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and acquires a fan-shaped ultrasonic image 22 centered on the ultrasonic transducer 14C. By scanning the ultrasonic probe 14, multiple ultrasonic images 22 are captured along the scanning direction. The ultrasonic images 22 are so-called B (brightness) mode images that visualize the internal structure of the target area, from the surface to the deep layers reached by the ultrasonic waves, as brightness information. The ultrasonic images 22 visualize the internal structure of the target area that cannot be observed in the surgical field image 21 obtained by optical imaging.

[0041] A guide groove 29 is provided in the tip portion 14B of the insertion section 14A. The guide groove 29 is a groove that engages with the puncture needle 18 to guide insertion of the puncture needle 18 to a target position inside an organ. FIG. 3 schematically shows the puncture needle 18, which has been inserted into the body from an insertion position NP on the body surface BS of the patient PT, being guided by the guide groove 29 to puncture a tumor 27 in the liver LV. The guide groove 29 is provided, for example, on the proximal side of the tip portion 14B relative to the ultrasonic transducer 14C, and is inclined at an angle θ with respect to the direction of the axis AX of the tip portion 14B. The first guide groove 29 is inclined backward so that the tip of the puncture needle 18, which is inserted from the proximal side of the tip portion 14B, faces the distal side of the tip portion 14B.

[0042] The puncture needle 18 is inserted while checking the tumor 27 using the ultrasound image 22. Because the area visualized by the ultrasound image 22 is a radial area from the ultrasound transducer 14C as the base point, the inclined guide grooves 29 make it possible to point the tip of the puncture needle 18 toward the tumor 27 visualized by the ultrasound image 22. Note that the insertion section 14A may be provided with multiple guide grooves 29 that are angled differently relative to the direction of the axis AX. In this case, the medical staff ST can select an appropriate guide groove 29 to insert the puncture needle 18.

[0043] Furthermore, a marker M is attached to the outer peripheral surface of the tip 14B of the insertion section 14A. The marker M is a marker that can be recognized from the surgical field image 21 optically captured by the camera 13B of the endoscope 13, that is, an optically detectable marker. The marker M is used by the medical support device 11 to estimate the position and orientation of the ultrasonic probe 14, more specifically, the position and orientation of the tip 14B of the insertion section 14A in the surgical field SF. A method for estimating the position and orientation using the marker M will be described later.

[0044] As an example, the marker M is a marker of a grid pattern 62 that is configured by a first line that extends in the direction of the axis AX of the tip portion 14B of the ultrasonic probe 14 and a second line that is perpendicular to the axial direction of the tip portion 14B and is formed in the circumferential direction along the outer circumferential surface of the tip portion 14B. Furthermore, the intersections in the grid pattern 62 are given circular symbols 64 or rectangular symbols 66, respectively.

[0045] The marker M is required to include at least the symbols 64, 66 arranged at each intersection of the grid pattern 62, and may not include the line segments indicating the first and second lines as shown in Fig. 3 etc. Furthermore, the symbols 64, 66 in the marker M are not limited to circles and rectangles and may have any shape, and may be represented by, for example, figures such as triangles, polygons, stars, and various marks, or may be represented by letters etc. Furthermore, in each drawing, the illustration of the marker M may be simplified or omitted to avoid cluttering the drawing.

[0046] The medical support device 11 acquires an operative field image 21 from the endoscope 13, and acquires an ultrasound image 22 from the ultrasound probe 14. The medical support device 11 also generates a superimposed image 26 by superimposing the ultrasound image 22 on the operative field image 21, and controls the display 16 to display the superimposed image 26. The superimposed image 26 displayed on the display 16 provides the medical staff ST with the field of view of the operative field SF inside the body of the patient PT and the internal structure of the target site (liver LV).

[0047] Next, the configuration of the medical support device 11 will be described. Fig. 4 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a display 16, a processor 41, a RAM (Random Access Memory) 42, a storage 43, a reception device 46, a communication I / F (Interface) 47, and an external I / F 48. Each of these components is connected to a bus 49 such as a system bus and a control bus, and can communicate with each other.

[0048] The display 16 displays various information in addition to the superimposed image 26. Examples of the display 16 include a liquid crystal display and an EL (Electro-Luminescence) display. The number of displays 16 may be at least one as shown in FIG. 1, but may also be multiple.

[0049] The processor 41 is, for example, a CPU (Central Processing Unit), which comprehensively controls each part of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0050] The RAM 42 is a memory that temporarily stores information and is used as a work memory by the processor 41. Examples of the RAM 42 include a dynamic random access memory (DRAM) and a static random access memory (SRAM).

[0051] The storage 43 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 43 include a hard disk drive (HDD) and a solid state drive (SSD). The storage 43 stores a medical support program 44 that causes the computer to function as the medical support device 11.

[0052] The storage 43 also stores dimension information 45. The dimension information 45 includes information representing the dimensions of the ultrasonic probe 14, specifically, the relative positional relationship between the marker M, the ultrasonic transducer 14C, the guide groove 29, etc. in the ultrasonic probe 14. The positional relationship of the marker M is, for example, information on the positions and attitudes of the reference symbols 64 and 66 constituting the marker M relative to the axial and circumferential directions of the tip portion 14B.

[0053] The positional relationship of the ultrasonic transducer 14C is represented by the linear distance between the reference point of the tip portion 14B and the ultrasonic transducer 14C, and the inclination angle of the ultrasonic transducer 14C relative to the direction of the axis AX of the tip portion 14B, etc. The positional relationship of the guide groove 29 is represented by the linear distance between the reference point of the tip portion 14B and the guide groove 29, and the inclination angle θ of the guide groove 29 relative to the direction of the axis AX of the tip portion 14B, etc.

[0054] The reception device 46 has a keyboard, a mouse, etc. (not shown) and receives instructions from an operator. That is, the medical support apparatus 11 is operated by an operator such as a medical staff member ST through the reception device 46. The reception device 46 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0055] The communication I / F 47 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control in accordance with communication protocols defined by various wired or wireless communication standards.

[0056] The external I / F 48 is, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer, a memory card, etc. The medical support device 11 may be, for example, a server computer, a personal computer, a smartphone, a tablet terminal, a wearable terminal, or the like, as appropriate.

[0057] Next, the functional configuration of the medical support device 11 will be described. Fig. 5 is a block diagram showing an example of the functional configuration of the medical support device 11. The medical support device 11 includes an acquisition unit 50, a derivation unit 52, a display control unit 54, and a generation unit 56. The processor 41 reads the medical support program 44 from the storage 43 and executes the medical support program 44 on the RAM 42, causing the processor 41 to function as the functional units of the acquisition unit 50, the derivation unit 52, the display control unit 54, and the generation unit 56. The processor 41 operates as the functional units of the acquisition unit 50, the derivation unit 52, the display control unit 54, and the generation unit 56, thereby realizing medical support processing.

[0058] The acquisition unit 50 acquires a surgical field image 21 obtained by optically capturing an operative field SF including a target site within the body and a medical instrument inserted into the body using the camera 13B. For example, the acquisition unit 50 acquires the surgical field image 21 from a device including a processor of the endoscope 13 via the external I / F 48 or the communication I / F 47. As shown in FIG. 2, when the ultrasonic probe 14 is inserted into the surgical field SF, the ultrasonic probe 14, more specifically, the tip 14B of the insertion unit 14A, appears in the surgical field image 21.

[0059] The acquisition unit 50 also acquires an internal image showing the internal structure of the target region. For example, the acquisition unit 50 acquires an ultrasound image 22 from a device including a processor of the ultrasound probe 14 via the external I / F 48 or the communication I / F 47. The ultrasound image 22 is an example of an "internal image" in the present disclosure. Note that the processor of the endoscope 13 and / or the processor of the ultrasound probe 14 may be included in the medical support device 11.

[0060] The derivation unit 52 derives position and orientation information indicating the position and orientation of a medical instrument (e.g., the ultrasound probe 14) in the surgical field SF based on the surgical field image 21, more specifically, on the marker M included in the surgical field image 21. Specifically, the derivation unit 52 detects the marker M by searching for morphological features of the marker M, such as the reference numerals 64 and 66, in the surgical field image 21. For example, the derivation unit 52 may detect the marker M using an image processing method such as pattern matching.

[0061] Furthermore, for example, instead of a rule-based method such as pattern matching, an AI (Artificial Intelligence) technique using a machine learning model may be used to detect the marker M. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) that is trained in advance to input the surgical field image 21 and output the area of ​​the marker M in the input surgical field image 21 can be applied.

[0062] The display control unit 54 controls the display of the superimposed image 26, which is obtained by superimposing an internal image (e.g., an ultrasound image 22) on the operative field image 21, and in which the display mode of the internal image is adjusted based on the position and orientation information. As an example, the display control unit 54 generates the superimposed image 26, which indicates the imaging range (e.g., position, orientation, and size) of the ultrasound image 22 in the operative field SF, by superimposing the ultrasound image 22 on an area on the operative field image 21 that corresponds to the imaging range of the ultrasound image 22. The display mode refers to the position and orientation when the internal image is displayed in the superimposed image 26, as well as whether or not it is displayed. The superimposed image 26 is an example of a "second superimposed image" in the present disclosure.

[0063] Specific examples of the superimposed image 26 will be described with reference to Figures 6 to 9. Figures 6 and 8 are diagrams conceptually showing the position and orientation of the tip 14B of the ultrasonic probe 14 within the surgical field SF defined as a three-dimensional space, each assuming a different position and orientation. Figures 7 and 9 show examples of the superimposed image 26 generated according to the position and orientation of the ultrasonic probe 14 within the surgical field SF. Figure 7 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 6. Figure 9 shows the superimposed image 26 in which the position and orientation of the tip 14B of the ultrasonic probe 14 are displayed in the state shown in Figure 8.

[0064] In the three-dimensional operative field SF shown in Figures 6 and 8, the Z axis is parallel to the imaging optical axis of the camera 13B of the endoscope 13. In addition, in the three-dimensional operative field SF, the XY plane is parallel to the imaging plane of the camera 13B and is perpendicular to the imaging optical axis. That is, the XY plane of the three-dimensional operative field SF is parallel to the screen (Xin-Yin plane) of the operative field image 21. The operative field image 21 is a projected image of the operative field SF projected from one viewpoint. Note that in Figures 6 and 8, of the reference symbols 64 and 66 that make up the marker M, those that appear in the operative field image 21 are indicated in dark colors, and those that do not appear in the operative field image 21 are indicated in light colors.

[0065] 6 shows a state in which the direction of the axis AX of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13B within the surgical field SF in three-dimensional space (more specifically, a state in which the axis AX of the tip 14B is parallel to the X-axis). In this case, as shown in FIG. 7, the marker M reflected in the surgical field image 21 has orthogonal lines of the grid pattern 62 parallel to the X-axis and Y-axis, respectively. Furthermore, the reference numerals 64 and 66 appear at equal intervals in the surgical field image 21.

[0066] On the other hand, Fig. 8 shows a state in which, within the surgical field SF in three-dimensional space, the direction of the axis AX of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13B, but is tilted in the depth direction parallel to the imaging optical axis. The attitude shown in Fig. 8 is a state in which the axis AX of the tip 14B is rotated approximately -25 degrees around the Y axis from the attitude shown in Fig. 6. In this case, as shown in Fig. 9, the marker M appearing in the surgical field image 21 appears so that the circumferentially extending lines of the grid pattern 62 become shorter and the interval between the reference symbols 64 and 66 becomes shorter as the marker M becomes farther from the camera 13B in the depth direction.

[0067] In this way, the shape of the marker M reflected in the operative field image 21 changes depending on the posture of the tip portion 14B. The derivation unit 52 estimates the posture of the tip portion 14B of the ultrasound probe 14 in the operative field SF based on the posture of the marker M in the operative field image 21. Specifically, the derivation unit 52 detects the direction of the axis AX of the tip portion 14B in the operative field SF as the posture of the tip portion 14B.

[0068] Furthermore, if the position of the tip portion 14B changes within the operative field SF, the position of the marker M reflected in the operative field image 21 also changes. The derivation unit 52 estimates the position of the tip portion 14B within the operative field SF based on the position of the marker M. The position of the tip portion 14B is detected as the position of a reference point of the tip portion 14B, which is provided at the tip position of the tip portion 14B, for example. Furthermore, the shooting distance from the camera 13B to the marker M within the operative field SF (i.e., the distance in the Z-axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the operative field image 21. The derivation unit 52 derives the position coordinates of the reference point of the tip portion 14B within the operative field SF based on the shooting distance and dimensional information 45 of the tip portion 14B, which includes the known dimensions of the marker M.

[0069] 6, the position of the tip 14B of the ultrasonic probe 14 is estimated as information such as the position coordinates (X01, Y01, Z01) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as information that it is parallel to the XY plane and the XZ plane and perpendicular to the YZ plane.

[0070] 8, the position of the tip 14B of the ultrasonic probe 14 is estimated as information such as the position coordinates (X02, Y02, Z02) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as information such as -25° with respect to the XY plane, parallel to the XZ plane, and 65° with respect to the YZ plane.

[0071] When the position and orientation of the tip portion 14B in the surgical field SF is estimated by the derivation unit 52, the display control unit 54 estimates the position and orientation of the ultrasonic transducer 14C in the surgical field SF based on the estimated position and orientation of the tip portion 14B and the dimension information 45. As described above, the linear distance between the reference point of the tip portion 14B and the ultrasonic transducer 14C, and the tilt angle of the ultrasonic transducer 14C with respect to the direction of the axis AX of the tip portion 14B are known information defined as the dimension information 45. By using this dimension information 45, the position and orientation of the ultrasonic transducer 14C in the surgical field SF can be estimated based on the position and orientation of the tip portion 14B in the surgical field SF.

[0072] 6, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X11, Y11, Z11) obtained by correcting the position coordinates (X01, Y01, Z01) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The attitude of the ultrasonic transducer 14C is defined as the attitude of the tip portion 14B with respect to the axis AX, and this becomes a tilt angle known as the dimension information 45.

[0073] 8, the position of the ultrasonic transducer 14C is estimated as information such as position coordinates (X12, Y12, Z12) obtained by correcting the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B in the surgical field SF using the dimension information 45. The attitude of the ultrasonic transducer 14C is defined as the attitude of the tip portion 14B with respect to the axis AX, and this becomes a tilt angle known as the dimension information 45.

[0074] The display control unit 54 also estimates the imaging range of the ultrasound image 22 in the surgical field SF based on the estimated position and orientation of the ultrasound transducer 14C. The imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer 14C. Therefore, if the position and orientation of the ultrasound transducer 14C in the surgical field SF are known, the imaging range of the ultrasound image 22 in the surgical field SF can also be estimated. For example, the imaging range of the ultrasound image 22 captured by the convex-type ultrasound transducer 14C is a sector-shaped range that spreads radially from the ultrasound transducer 14C as the base point. FIGS. 6 and 8 illustrate an imaging range 22R of the ultrasound image 22 in the surgical field SF.

[0075] The display controller 54 then generates a superimposed image 26 by superimposing the ultrasound image 22 on a region of the operative field image 21 (Xin-Yin plane) that corresponds to the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF. In the examples of FIGS. 6 and 7, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). On the other hand, in the examples of FIGS. 8 and 9, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the operative field SF is not parallel to the screen of the operative field image 21 in the operative field SF (Xin-Yin plane). Taking into account the difference in coordinate systems between the operative field image 21 and the ultrasound image 22, the display controller 54 performs transformations such as projective transformation, affine transformation, translation, rotation, enlargement, and reduction on the ultrasound image 22, and then generates a superimposed image 26 by superimposing the ultrasound image 22 on the operative field image 21.

[0076] The superimposed image 26 makes it easy to grasp the position and orientation of the ultrasonic transducer 14C even if it is difficult to visually recognize the position and orientation of the ultrasonic transducer 14C in the surgical field image 21 due to, for example, obstruction by an organ or the positional relationship with the camera 13B. Therefore, it becomes easier to adjust the position and orientation of the ultrasonic probe 14 so that a desired area (for example, a tumor 27) is captured in the ultrasonic image 22, which contributes to improving the operability of the ultrasonic probe 14.

[0077] As shown in FIG. 2, a margin region 22B may be added to the ultrasound image 22 to make the overall shape of the ultrasound image 22 rectangular. In this case, the display control unit 54 preferably generates the superimposed image 26 by using the image from which the margin region 22B has been removed as the internal image. As described above, when the ultrasound probe 14 is a convex type, the ultrasound image 22 is fan-shaped. The margin region 22B is used to convert this fan-shaped ultrasound image 22 into a rectangle for standardized image processing. In other words, the margin region 22B does not show information necessary for medical treatment, such as the internal structure of the target area in the ultrasound image 22. By not including the margin region 22B in the superimposed image 26, it is possible to prevent the surgical field image 21 from becoming difficult to view in the superimposed image 26. FIG. 7 and other figures illustrate the superimposed image 26 from which the margin region 22B has been removed.

[0078] As described above, the superimposed image 26 is generated using the operative field image 21 captured by the endoscope 13. The operative field image 21 is a projected image of the operative field SF projected from a single viewpoint, and therefore does not include depth information. Therefore, depending on the position and posture of the ultrasound probe 14, the superimposed image 26 based on the operative field image 21 may make it difficult to understand the positional relationship (particularly the front-to-back relationship) between the target area in the operative field SF and the imaging range of the ultrasound image 22. Furthermore, depending on the position and posture of the ultrasound probe 14, it may be difficult to confirm the ultrasound image 22.

[0079] Therefore, in the medical support device 11 according to this embodiment, a three-dimensional superimposed image 24 is generated by superimposing an internal image (for example, an ultrasound image 22) on a three-dimensional model 28 showing the surface shape of a target part inside the body, as shown in Fig. 10. This makes it possible to clearly present the positional relationship between the target part in the surgical field SF and the imaging range of the ultrasound image 22.

[0080] Specifically, the acquisition unit 50 acquires a three-dimensional model 28 that indicates the surface shape of the target region inside the body. The three-dimensional model 28 is a three-dimensional image that includes at least depth information of the surface of the target region. Furthermore, the acquisition unit 50 preferably acquires a real-time three-dimensional model 28 that reflects the current state of the target region over time. A method for acquiring (generating) the real-time three-dimensional model 28 will be described later.

[0081] The display control unit 54 controls the display of the three-dimensional superimposed image 24, which is obtained by superimposing an internal image (e.g., ultrasound image 22) on the three-dimensional model 28, and in which the display mode of the internal image is adjusted based on the position and orientation information. Specifically, the display control unit 54 generates the three-dimensional superimposed image 24, which indicates the imaging range (e.g., position, orientation, and size) of the ultrasound image 22 in the surgical field SF, by superimposing the ultrasound image 22 on an area on the three-dimensional model 28 that corresponds to the imaging range of the ultrasound image 22. The three-dimensional superimposed image 24 is an example of a "first superimposed image" in the present disclosure.

[0082] The display control unit 54 may also control the three-dimensional model 28 included in the three-dimensional superimposed image 24 to increase the transparency of a portion located between the viewpoint of the three-dimensional superimposed image 24 and the display surface of the internal image. Increasing the transparency means making a portion of the three-dimensional model 28 semi-transparent or transparent. Fig. 11 shows an example in which the three-dimensional model 28 between the viewpoint of the three-dimensional superimposed image 24 and the display surface (Xpb-Ypb plane) of the ultrasound image 22 is transparent.

[0083] That is, in this case, the three-dimensional superimposed image 24 shows a cross section 28U of the three-dimensional model 28 in the same plane as the display surface (Xpb-Ypb plane) of the ultrasound image 22. This configuration makes it possible to more clearly present the positional relationship in the depth direction between the three-dimensional model 28 and the ultrasound image 22. Note that when the three-dimensional model 28 shows only the surface shape, the inside of the cross section 28U is hollow.

[0084] The display control unit 54 may also perform control so as to switch between displaying the three-dimensional superimposed image 24 based on the three-dimensional model 28 and the superimposed image 26 based on the surgical field image 21 on the display 16. For example, the display control unit 54 may perform control so as to display the three-dimensional superimposed image 24 when a preset condition related to the display of the three-dimensional superimposed image 24 is satisfied, and may perform control so as to display the superimposed image 26 instead of the three-dimensional superimposed image 24 when the condition is not satisfied.

[0085] FIG. 12 shows a state in which the direction of the axis AX of the tip 14B of the ultrasonic probe 14 is nearly parallel to the imaging optical axis of the camera 13B within the surgical field SF in three-dimensional space. In this case, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the surgical field SF and the Xin-Yin plane of the surgical field image 21 are nearly perpendicular to each other. Therefore, as shown in FIG. 13, the ultrasound image 22 superimposed on the surgical field image 21 has a nearly linear shape, making it difficult to observe. Furthermore, in such a superimposed image 26, both the ultrasound image 22 and the surgical field image 21 are difficult to observe. Note that the illustration method of FIG. 12 is the same as that of FIG. 6, and the illustration method of FIG. 13 is the same as that of FIG. 7.

[0086] That is, the smaller the angle θ1, which represents the orientation of the internal image and is the smallest when the imaging optical axis of camera 13B is parallel to the display surface of the internal image and the largest when the imaging optical axis is normal to the display surface of the internal image, the more difficult it is to observe the superimposed image 26. Therefore, the display control unit 54 may determine that the predetermined condition for displaying the 3D superimposed image 24 is satisfied when the angle θ1 is less than a predetermined threshold. That is, when the angle θ1 is small and the superimposed image 26 is difficult to observe, the display control unit 54 may switch to and display the 3D superimposed image 24.

[0087] Furthermore, for example, the display control unit 54 may determine that the condition is satisfied when the user instructs to display the three-dimensional superimposed image 24. In other words, the display control unit 54 may switch between the three-dimensional superimposed image 24 and the superimposed image 26 to be displayed in response to an instruction from the user.

[0088] The display control unit 54 may also receive a specification of the orientation of the three-dimensional model 28 in the three-dimensional superimposed image 24, and generate the three-dimensional superimposed image 24 based on the specified orientation and position and orientation information of the three-dimensional model 28. FIG. 14 shows the three-dimensional superimposed image 24 in a state where the orientation of the three-dimensional model 28 is rotated 90 degrees around the Y axis from the state in FIG. 10. In this manner, the orientation of the three-dimensional model 28 (i.e., the viewpoint of the three-dimensional superimposed image 24) can be changed arbitrarily. Therefore, even if the ultrasound image 22 (see FIG. 13) is oriented in a way that makes it difficult to observe when aligned with the viewpoint of the surgical field image 21, it can be displayed in a direction that makes it easy to observe while ensuring consistency with the three-dimensional model 28.

[0089] The orientation of the three-dimensional model 28 may be arbitrarily specified by the user, or may be specified in advance, such as a viewpoint shifted by 90 degrees around the Y axis from the viewpoint of the surgical field image 21.

[0090] Furthermore, for example, the display control unit 54 may specify the orientation of the three-dimensional model 28 so that the internal image superimposed on the three-dimensional model 28 faces forward. Specifically, the orientation of the internal image is defined as the front when the viewpoint of the first superimposed image is positioned on a line perpendicular to the display surface of the internal image. In this case, the display control unit 54 may specify the orientation of the three-dimensional model 28 so that the orientation of the internal image determined based on the position and orientation information faces forward.

[0091] Furthermore, the display control unit 54 may perform control so that information indicating the position of the camera 13B (i.e., the viewpoint of the operative field image 21) is displayed on the three-dimensional superimposed image 24. FIG. 14 illustrates a rectangle 13R with an arrow pointing along the imaging optical axis of the camera 13B as an example of information indicating the position of the camera 13B. Note that the information indicating the position of the camera 13B is not limited to this, and may be displayed, for example, in text. According to this configuration, even if the orientation of the three-dimensional model 28 is changed in the three-dimensional superimposed image 24, the orientation of the camera 13B actually inserted inside the body can be known, making it easier to adjust the position and posture of the endoscope 13, the ultrasound probe 14, etc.

[0092] Furthermore, the display control unit 54 may apply an internal image expressed by another method as an internal image to be superimposed on the three-dimensional superimposed image 24, in addition to or instead of the ultrasound image 22. For example, the display control unit 54 may apply, as the internal image, a tomographic image of the target region extracted from a three-dimensional image taken by a tomography device such as a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device. Also, for example, the display control unit 54 may apply, as the internal image, a blood vessel image showing the vascular structure of the target region extracted from a three-dimensional image taken by a tomography device such as a CT device. The tomographic image and the blood vessel image are examples of the "internal image" and "second internal image" of the present disclosure.

[0093] Specifically, before endoscopic surgery using the endoscope 13, ultrasound probe 14, etc., a three-dimensional image of the patient PT is captured in advance using a CT device or the like and stored in the storage 43, etc. The acquisition unit 50 acquires this pre-captured three-dimensional image including the internal structure of the target area. The display control unit 54 extracts an internal image from the three-dimensional image based on the position and orientation information.

[0094] For example, the display control unit 54 may extract, from the three-dimensional image, a tomographic image and / or a blood vessel image corresponding to the display surface of the ultrasound image 22 based on the position and orientation information. That is, the extracted tomographic image may be a tomographic image of the target region including the display surface of the ultrasound image 22. Furthermore, the extracted blood vessel image may be a blood vessel image showing the blood vessel structure of the target region passing through the display surface of the ultrasound image 22.

[0095] The display control unit 54 uses the internal image (cross-sectional image and / or blood vessel image) extracted from the three-dimensional image to generate the three-dimensional superimposed image 24. For example, the display control unit 54 may generate the three-dimensional superimposed image 24 by superimposing the ultrasound image 22 and the internal image extracted from the three-dimensional image on the three-dimensional model 28.

[0096] FIG. 15 shows an example of a three-dimensional superimposed image 24 in which a blood vessel image 25 is superimposed on a three-dimensional model 28 in addition to an ultrasound image 22. When superimposing multiple internal images in this manner, the display control unit 54 may combine each internal image at a specific superimposition ratio. This configuration allows the internal image based on a CT image or the like to be confirmed even during surgery, making it easier to understand the internal structure of the target area. This improves convenience.

[0097] The display control unit 54 controls the display 16 to display at least one of the superimposed image 26 and the three-dimensional superimposed image 24. The display control unit 54 may also control the display 16 to display the operative field image 21 and the ultrasound image 22. In this case, the display control unit 54 may control the display 16 to display the operative field image 21, the ultrasound image 22, the superimposed image 26, and the three-dimensional superimposed image 24 all together.

[0098] Furthermore, when there are multiple displays 16, the display control unit 54 may control the operative field image 21, the ultrasound image 22, the superimposed image 26, and the three-dimensional superimposed image 24 to be displayed on different displays 16. Furthermore, the display control unit 54 may receive a designation of an image to be displayed on the display 16 from among the operative field image 21, the ultrasound image 22, the superimposed image 26, and the three-dimensional superimposed image 24, and control the designated image to be displayed on at least one display 16.

[0099] The operative field image 21 and the ultrasound image 22 are each output as a moving image. The display controller 54 may control the display 16 to display at least one of the operative field image 21, the ultrasound image 22, the superimposed image 26, and the three-dimensional superimposed image 24 as a live view. Live view display refers to the real-time display of moving images generated at a predetermined frame rate based on signals output by an imaging device for imaging the target. The imaging device includes, for example, an image sensor included in the endoscope 13 that optically captures images of the target, and an ultrasound transducer 14C that uses ultrasound to image the target.

[0100] When the superimposed image 26 and / or the three-dimensional superimposed image 24 are displayed as a live view, the derivation unit 52 repeatedly derives the position and orientation information every time a surgical field image 21 is acquired or at predetermined time intervals. Correspondingly, the display control unit 54 repeatedly generates the superimposed image 26 and / or the three-dimensional superimposed image 24 so that changes in the position and orientation information are reflected.

[0101] [3D model generation] It is preferable that the current state of the target region be reflected in the three-dimensional model 28 used in the three-dimensional superimposed image 24. Therefore, the generation unit 56 generates the three-dimensional model 28 in real time by capturing images of the target region during endoscopic surgery.

[0102] As an example, the generation unit 56 may use a stereo camera to generate (acquire) the three-dimensional model 28. A stereo camera is a camera that simultaneously captures images of an object from two different directions and measures the distance to the object from the parallax of each image.

[0103] Specifically, the generating unit 56 acquires a first image and a second image obtained by optically capturing, from different viewpoints, a surgical field SF including a target site (e.g., the liver LV) within the body using a first camera and a second camera, respectively. For example, the first camera may be provided in a trocar 17 for inserting a medical instrument into the body. Alternatively, for example, the first camera may be provided in a body wall within the body. Alternatively, for example, the second camera may be a camera 13B provided at the tip of an insertion section 13A of an endoscope 13 used in laparoscopic surgery.

[0104] The generating unit 56 also detects, from at least the first image, a marker configured with an optically detectable pattern that is placed in the surgical field SF. The generating unit 56 then determines the relative positional relationship between the viewpoint of the first camera and the viewpoint of the second camera based on the detected marker. In a method using a stereo camera, it is necessary to determine the positional relationship between the two cameras in order to calculate the parallax. This marker is provided at a position that can be photographed by at least the first camera, as a reference when determining the positional relationship between the two cameras.

[0105] Then, the generation unit 56 generates a three-dimensional model 28 that indicates the surface shape of the target region based on the identified relative positional relationship between the viewpoints of the first and second cameras and the first and second images. A specific method for generating the three-dimensional model 28 by the generation unit 56 will be described below with reference to an example.

[0106] (First Example) An example of a configuration for generating a three-dimensional model 28 using a stereo camera will be described with reference to Fig. 16. Fig. 16 shows the state in which an endoscope 13 and an ultrasound probe 14 are inserted into the abdomen of a patient PT. In this embodiment, a camera 80 is provided as the first camera in a trocar 17 for inserting the ultrasound probe 14 into the body. A camera 13B provided at the tip of the insertion section 13A of the endoscope 13 is used as the second camera. In Fig. 16, the viewpoint of camera 80 is indicated by P, and the imaging range of camera 80 is indicated by a dotted line. Furthermore, the viewpoint of camera 13B is indicated by Q, and the imaging range of camera 13B is indicated by a dashed line.

[0107] Marker 13M is attached to the outer peripheral surface of camera 13B (more specifically, the distal end of insertion section 13A of endoscope 13) as a reference marker for identifying the relative positional relationship (i.e., vector PQ) between viewpoint P of camera 80 and viewpoint Q of camera 13B. Marker 13M is a marker that can be recognized from the first image captured by camera 80, that is, an optically detectable marker. The positional relationship between viewpoint Q of camera 13B and position R of marker 13M is known and is stored in advance in storage 43 as, for example, dimension information 45. Note that, although marker 13M is shown as five dots in FIG. 16, the shape of marker 13M is not particularly limited.

[0108] The generation unit 56 identifies the relative positional relationship between the viewpoint P of the camera 80 and the viewpoint Q of the camera 13B based on the marker 13M in the first image captured by the camera 80. Specifically, the generation unit 56 first calculates the relative positional relationship (i.e., vector PR) between the viewpoint P of the camera 80 and the position R of the marker 13M based on the position and shape of the marker 13M in the first image. As described above, the positional relationship (i.e., vector QR) between the viewpoint Q of the camera 13B and the position R of the marker 13M is known. The generation unit 56 can calculate the vector PQ, which is the relative positional relationship between the viewpoint P of the camera 80 and the viewpoint Q of the camera 13B, from the difference between the vector PR and the vector QR.

[0109] The generation unit 56 generates a three-dimensional model 28 that indicates the surface shape of the target region (for example, the liver LV) based on the relative positional relationship between the viewpoint P of the camera 80 and the viewpoint Q of the camera 13B, and the first and second images. A known technique using a stereo method can be appropriately applied as a method for generating the three-dimensional model 28 using the first and second images.

[0110] In this embodiment, an example has been described in which the first camera 80 is provided on the trocar 17, but this is not limiting. The camera 80 may be provided at a position where it can photograph the marker 13M, and may be provided on the body wall inside the body, for example.

[0111] (Second Example) Referring to Figure 17, another example of a configuration for generating a three-dimensional model 28 using a stereo camera will be described. Figure 17 shows the state in which an endoscope 13 and an ultrasound probe 14 are inserted into the abdomen of a patient PT. In this embodiment, a camera 81 is provided on the body wall inside the body as the first camera. A camera 13B provided at the tip of the insertion section 13A of the endoscope 13 is used as the second camera. In Figure 17, the viewpoint of camera 81 is indicated by P, and the imaging range of camera 81 is indicated by a dotted line. Furthermore, the viewpoint of camera 13B is indicated by Q, and the imaging range of camera 13B is indicated by a dashed line.

[0112] A marker M is attached to the medical instrument (ultrasound probe 14) to be inserted into the body as a reference marker for identifying the relative positional relationship (i.e., vector PQ) between the viewpoint P of camera 81 and the viewpoint Q of camera 13B. The marker M is photographed by each of camera 81 and camera 13B. As described above, the marker M is an optically detectable marker.

[0113] The generation unit 56 identifies the relative positional relationship between the viewpoint P of the camera 81 and the viewpoint Q of the camera 13B based on the marker M in the first image captured by the camera 81 and the marker M in the second image captured by the camera 13B. Specifically, first, the generation unit 56 calculates the relative positional relationship between the viewpoint P of the camera 81 and the position R of the marker M (i.e., vector PR) based on the position and shape of the marker M in the first image. Next, the generation unit 56 calculates the relative positional relationship between the viewpoint Q of the camera 13B and the position R of the marker M (i.e., vector QR) based on the position and shape of the marker M in the second image. The generation unit 56 can calculate the vector PQ, which is the relative positional relationship between the viewpoint P of the camera 81 and the viewpoint Q of the camera 13B, from the difference between the vector PR and the vector QR.

[0114] The generation unit 56 generates a three-dimensional model 28 that indicates the surface shape of the target region (for example, the liver LV) based on the relative positional relationship between the viewpoint P of the camera 81 and the viewpoint Q of the camera 13B, and the first and second images. A known technique using a stereo method can be appropriately applied as a method for generating the three-dimensional model 28 using the first and second images.

[0115] In this embodiment, an example in which the marker is attached to the ultrasonic probe 14 has been described, but the present invention is not limited to this. The marker may be provided at a position that can be photographed by each of the camera 81 and the camera 13B, and may be provided on, for example, a treatment tool that is separately inserted into the body.

[0116] The generation unit 56 may generate the three-dimensional model 28 by other methods than the method using a stereo camera.

[0117] For example, the generation unit 56 may generate (acquire) the three-dimensional model 28 using a ToF (Time of Flight) camera. The ToF camera is a camera that irradiates a target with light such as infrared light and measures the distance between the ToF camera and the target based on the time it takes for the reflected light to be received or the phase change between the emitted light and the received light. Specifically, an image captured by the ToF camera has distance information that indicates the distance between the ToF camera and the target for each pixel.

[0118] Therefore, for example, a ToF camera may be inserted into the body together with the endoscope 13 and the ultrasound probe 14, thereby generating a three-dimensional model 28 that shows the surface shape of the target area as seen from the ToF camera. The ToF camera may be provided, for example, in a trocar 17 used to insert the endoscope 13 or the ultrasound probe 14. The acquisition unit 50 may also generate the three-dimensional model 28 by combining a ToF camera with a visible light camera (for example, camera 13B of the endoscope 13).

[0119] Furthermore, for example, the acquisition unit 50 may acquire the three-dimensional model 28 based on three-dimensional images captured by a tomography device such as a CT device or an MRI device. In this case, for example, a three-dimensional image of the patient PT may be captured in advance and stored in the storage 43 or the like before endoscopic surgery. The acquisition unit 50 may acquire this previously captured three-dimensional image and apply a known three-dimensional reconstruction process to acquire the three-dimensional model 28. In this case, it is difficult to generate the three-dimensional model 28 in real time, but it is possible to generate a more accurate three-dimensional model 28 that includes the internal structure.

[0120] Furthermore, for example, the generation unit 56 may generate the three-dimensional model 28 by combining the above methods. For example, in a method using a stereo camera, it may be impossible to capture the back side of the target part as seen from the viewpoint of the camera, resulting in a loss of part of the three-dimensional model 28. Therefore, for example, the generation unit 56 may basically generate the three-dimensional model 28 in real time using the stereo camera, but reconstruct the part that cannot be generated by the stereo camera from a three-dimensional image that has been captured in advance.

[0121] Next, the operation of the medical support device 11 according to this embodiment will be described with reference to Fig. 18. In the medical support device 11, the processor 41 executes the medical support program 44, thereby executing the medical support process shown in Fig. 18. This process is executed, for example, when a user issues an instruction to start execution via the reception device 46.

[0122] In step S10, the generator 56 performs a process of generating a three-dimensional model that represents the surface shape of a target part inside the body. The three-dimensional model generation process will be described later.

[0123] In step S20, the acquisition unit 50 acquires the three-dimensional model generated in step S10. The acquisition unit 50 also acquires an internal image showing the internal structure of the target region. The acquisition unit 50 also acquires an operative field image obtained by optically capturing, with a camera, an operative field SF including the target region inside the body and the medical instrument to be inserted into the body.

[0124] In step S22, the derivation unit 52 acquires position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF. In step S24, the display control unit 54 determines whether the angle θ1, which is the orientation of the internal image acquired in step S20, is less than a predetermined threshold. The angle θ1 is the angle that is minimum when the imaging optical axis of camera 13B is parallel to the display surface of the internal image and maximum when the imaging optical axis is normal to the display surface of the internal image. Specifically, the display control unit 54 derives the angle θ1 based on the position and orientation information derived in step S12 and the dimension information 45.

[0125] If the angle θ1 is less than the threshold value and the result of step S24 is YES, the process proceeds to step S26. In step S26, the display control unit 54 controls the display 16 to display a first superimposed image in which an internal image is superimposed on the three-dimensional model acquired in step S20, the first superimposed image being the first superimposed image in which the display manner of the internal image has been adjusted based on the position and orientation information derived in step S22.

[0126] On the other hand, if the angle θ1 is equal to or greater than the threshold value and the result of step S24 is negative, the process proceeds to step S28. In step S28, the display control unit 54 controls the display 16 to display a second superimposed image in which an internal image is superimposed on the operative field image acquired in step S20, and in which the display manner of the internal image has been adjusted based on the position and orientation information derived in step S22. When step S26 or step S28 is completed, this process ends.

[0127] Next, the three-dimensional model generation process executed in step S10 will be described with reference to Fig. 19. In step S12, the generation unit 56 acquires a first image and a second image of the surgical field SF including the target region inside the body, optically captured by a first camera and a second camera from different viewpoints, respectively. In step S14, the generation unit 56 detects, from at least the first image, a marker configured with an optically detectable pattern to be placed in the surgical field SF.

[0128] In step S16, the generation unit 56 identifies the relative positional relationship between the viewpoint of the first camera and the viewpoint of the second camera based on the marker detected in step S14. In step S18, the generation unit 56 generates a three-dimensional model 28 that indicates the surface shape of the target region based on the relative positional relationship between the viewpoint of the first camera and the viewpoint of the second camera identified in step S16 and the first and second images acquired in step S12.

[0129] As described above, the medical support device 11 according to one aspect of the present embodiment includes the processor 41. The processor 41 acquires a three-dimensional model showing the surface shape of a target region inside the body and acquires an internal image showing the internal structure of the target region. The processor 41 also acquires position and orientation information showing the position and orientation of a medical instrument in a surgical field SF including the target region and the medical instrument to be inserted into the body. The processor 41 also controls the display of a first superimposed image in which the internal image is superimposed on the three-dimensional model, and in which the display mode of the internal image is adjusted based on the position and orientation information.

[0130] The medical support device 11 according to this embodiment can generate a three-dimensional superimposed image 24 in which an internal image is superimposed on a three-dimensional model 28. Therefore, the positional relationship between the target area in the surgical field SF and the imaging range of the internal image can be presented in an easy-to-understand manner, thereby supporting the observation of the internal image.

[0131] Furthermore, a medical support device 11 according to another aspect of this embodiment includes a processor 41. The processor 41 acquires a first image and a second image of a surgical field including a target region within the body, which are optically captured by a first camera and a second camera from different viewpoints, respectively. The processor 41 also detects, from at least the first image, a marker configured with an optically detectable pattern that is placed in the surgical field SF. The processor 41 also generates a three-dimensional model showing the surface shape of the target region based on the first image and the second image, which is a relative positional relationship between the viewpoints of the first camera and the second camera and is identified based on the marker.

[0132] According to the medical support device 11 of this embodiment, the positional relationship between two cameras whose positional relationship is unknown can be easily determined even inside a body cavity, facilitating the generation of a three-dimensional model 28 in real time using a stereo method. Furthermore, in the stereo method, the accuracy of the three-dimensional model 28 tends to decrease if the parallax is too small. However, according to the medical support device 11 of this embodiment, the degree of freedom in the positions of the two cameras can be ensured, thereby preventing a decrease in the accuracy of the three-dimensional model 28. In other words, since a three-dimensional model 28 with excellent real-time performance and accuracy can be generated, when applied to a three-dimensional superimposed image 24, it can assist in the observation of internal images.

[0133] In the above embodiment, the marker M is attached to the tip 14B of the insertion section 14A of the ultrasonic probe 14, but the present invention is not limited to this. The marker M may be attached to a portion of the various medical instruments that is inserted into the body of the patient PT, and may be attached to, for example, the middle part of the insertion section 14A or the base end side of the insertion section 14A.

[0134] In the above embodiment, the position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF is derived using the marker M, but the present invention is not limited to this. For example, the position and orientation information may be derived by detecting the characteristic shape of the medical instrument from the surgical field image 21 through image analysis. Alternatively, for example, the shape of the medical instrument may be identified by photographing the medical instrument using a ToF camera or a stereo camera, and the position and orientation information may be derived based on the identified shape.

[0135] In the above embodiment, the operative field image 21 is captured by the camera 13B of the endoscope 13, but this is not limiting. For example, an image captured by a camera attached to another instrument inserted into the body, such as the trocar 17, may be used, or an image captured by a camera attached to the body wall may be used.

[0136] Furthermore, in the above embodiment, the medical instrument inserted into the body of the patient PT is an ultrasound probe 14 (an example of a medical probe) capable of observing the internal structure of an organ. The procedure of puncturing an internal organ with a puncture needle 18 is often performed using a medical probe capable of observing the internal structure of the organ. Therefore, when a medical probe is used as the medical instrument, as in the above embodiment, the technology of the present disclosure is particularly effective.

[0137] Furthermore, the ultrasound probe 14 is relatively often used in combination with a puncture needle 18. Therefore, the technology of the present disclosure is even more effective when the ultrasound probe 14 is used as a medical probe. Note that the medical probe capable of observing the internal structure of an organ may be other than the ultrasound probe 14, such as an optical coherence tomography (OCT) probe. Alternatively, for example, a drop-in type probe may be used.

[0138] Note that the medical instrument may be anything other than a medical probe capable of observing the internal structure of an organ. For example, the medical instrument may be a treatment instrument that does not have the function of observing the internal structure and only has a guide groove 29 for the puncture needle 18 at its tip. For example, when a tumor is present on the surface of an organ and a treatment is performed in which the puncture needle 18 is inserted into the surface tumor, even a treatment instrument that does not have the function of observing the internal structure can properly guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST aligns the guide groove 29 of the treatment instrument with a position corresponding to the tumor on the surface of the organ, and then inserts the puncture needle 18 into the tumor through the guide groove 29.

[0139] Furthermore, the medical instrument may be a simple rod-like treatment instrument without a guide groove 29. If a tumor is present on the surface of an organ, even such a treatment instrument can be used to indicate the tumor. Simply superimposing the puncture path 30 on the surgical field image 21 in which the treatment instrument has been used to indicate the tumor on the surface of the organ can serve as a guide for checking the puncture direction of the puncture needle 18, etc. Therefore, the technology of the present disclosure is effective even if the medical instrument does not have a guide groove 29.

[0140] Furthermore, in the above embodiment, cauterization has been described as an example of the function of the puncture needle 18, but the function of the puncture needle 18 is not limited to this. Furthermore, although the puncture needle 18 has been described as an example of a treatment tool, other than the puncture needle 18, treatment tools for injecting a fluorescent agent such as ICG (Indocyanine Green), biopsy needles and forceps used for collecting tissue for biopsy, etc. may also be applied.

[0141] In addition, in the above embodiment, the inside of the body has been described using body cavities such as the abdominal cavity and the thoracic cavity as examples, but the inside of the body may also be the inside of an upper digestive tract such as the esophagus, a lower digestive tract such as the intestines, or a duct such as the bronchi. When the technology of the present disclosure is applied to a surgical field inside a duct, for example, a marker M is provided at the base end of a flexible endoscope inserted into the duct.

[0142] In the above embodiment, the medical support device 11 is used in laparoscopic surgery, but the present invention is not limited to this. The medical support device 11 can also be used in robotic surgery, for example.

[0143] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 50, derivation unit 52, display control unit 54, and generation unit 56. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0144] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0145] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0146] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0147] In the above embodiment, the medical support program 44 is pre-stored (installed) in the storage 43, but the present invention is not limited to this. The medical support program 44 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The medical support program 44 may also be downloaded from an external device via a network.

[0148] The present disclosure can also be applied to programs and program products. Specifically, the medical support program 44 in the above embodiment may be provided as a program product. The program product includes any type of product for providing a program. For example, the program product includes a program provided via a network such as the Internet, as well as a computer-readable recording medium that non-temporarily stores a program.

[0149] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0150] The following additional notes are further disclosed regarding the above embodiment. [Appendix 1] a processor, the processor comprising: Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image obtained by superimposing the internal image on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; Medical support equipment. [Appendix 2] The processor: Acquire an operative field image optically photographed by a camera of the operative field; When a preset condition regarding the display of the first superimposed image is satisfied, control is performed to display the first superimposed image; If the condition is not satisfied, a second superimposed image in which the internal image is superimposed on the operative field image, and in which the display mode of the internal image is adjusted based on the position and orientation information, is displayed instead of the first superimposed image. 2. The medical support device of claim 1. [Appendix 3] The processor determines that the condition is satisfied when an angle representing the orientation of the internal image, the angle being the smallest when the imaging optical axis of the camera is parallel to the display surface of the internal image and the largest when the imaging optical axis is normal to the display surface of the internal image, is less than a predetermined threshold. 3. The medical support device according to claim 2. [Appendix 4] The processor determines that the condition is satisfied when a user instructs the display of the first superimposed image. 4. The medical support device according to claim 2 or 3. [Appendix 5] The processor controls to display information indicating the position of the camera on the first superimposed image. 5. A medical support device according to any one of claims 2 to 4. [Appendix 6] The camera is a camera provided at the tip of an endoscope used in laparoscopic surgery. 6. A medical support device according to any one of claims 2 to 5. [Appendix 7] a marker configured with an optically detectable pattern is provided on an outer peripheral surface of the medical instrument; The processor derives the position and orientation information based on the markers included in the surgical field image. 7. A medical support device according to any one of claims 2 to 6. [Appendix 8] The processor: accepting a designation of an orientation of the three-dimensional model in the first superimposed image; The first superimposed image is generated based on the specified orientation of the three-dimensional model and the position and orientation information. 8. A medical support device according to any one of claims 1 to 7. [Appendix 9] Regarding the orientation of the internal image, when the viewpoint of the first superimposed image is located on a line perpendicular to the display surface of the internal image, the front is defined as: The processor specifies the orientation of the three-dimensional model so that the orientation of the internal image determined based on the position and orientation information faces the front side. 9. The medical support device of claim 8. [Appendix 10] The processor controls the three-dimensional model included in the first superimposed image so as to increase the transparency of a portion located between a viewpoint of the first superimposed image and a display surface of the internal image. 10. A medical support device according to any one of claims 1 to 9. [Appendix 11] the medical instrument is an ultrasonic probe that transmits ultrasonic waves to the target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area; The internal image is an ultrasound image generated in response to the electrical signals. 11. A medical support device according to any one of claims 1 to 10. [Appendix 12] The processor: acquiring a three-dimensional image including the internal structure of the target region; extracting a second internal image corresponding to a display surface of the ultrasound image from the three-dimensional image based on the position and orientation information; generating a first superimposed image by superimposing the ultrasound image and the second internal image on the three-dimensional model; 12. The medical support device of claim 11. [Appendix 13] The second internal image is a tomographic image of the target area including the display surface of the ultrasound image. 13. The medical support device of claim 12. [Appendix 14] The second internal image is a blood vessel image showing the blood vessel structure of the target area passing through the display surface of the ultrasound image. 13. The medical support device of claim 12. [Appendix 15] The processor generates the first superimposed image by using, as the internal image, an image obtained by removing a margin area in the ultrasound image that does not show the internal structure of the target region. 12. The medical support device of claim 11. [Appendix 16] The processor: acquiring a three-dimensional image of the target area; Extracting the internal image from the three-dimensional image based on the position and orientation information. 16. A medical support device according to any one of claims 1 to 15. [Appendix 17] The processor acquires the three-dimensional model using a time-of-flight (ToF) camera. 17. A medical support device according to any one of claims 1 to 16. [Appendix 18] The processor acquires the three-dimensional model using a stereo camera. 18. A medical support device according to any one of claims 1 to 17. [Appendix 19] Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image obtained by superimposing the internal image on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; A medical support method in which processing is performed by a computer. [Appendix 20] Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image obtained by superimposing the internal image on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; A medical support program that causes a computer to perform processing. [Explanation of symbols]

[0151] 10 Medical Support System 11 Medical support equipment 13 Endoscopy 13A, 14A insertion section 13B, 80, 81 Camera 13M marker 13R rectangle 14 Ultrasound probe 14B Tip 14C Ultrasonic Transducer 14D Control unit 16 Display 17 Trocar 18 Puncture needle 18A needle part 18B Gripping part 21 Surgical field image 22 Ultrasound images 22B Margin area 22R shooting range 24 3D superimposed images 25 Blood Vessel Images 26 Superimposed Images 27 Tumor 28 Three-dimensional model 28U cross section 29 Guide groove 30 Puncture Path 41 processors 42 RAM 43 Storage 44 Medical Assistance Program 45 Dimensional Information 46 Reception Device 47 Communication I / F 48 External I / F 49 Bus 50 Acquisition Department 52 Derivation part 54 Display control unit 56 Generation part 62 Grid Pattern 64, 66 sign θ Tilt angle AX axis BS body surface LV liver M marker NP insertion position PT patient SF field ST Medical Staff X, Xin, Xpb, Y, Yin, Ypb, Z coordinates

Claims

1. a processor, the processor comprising: Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image in which the internal image is superimposed on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; Medical support equipment.

2. The processor: Acquire an operative field image optically photographed by a camera of the operative field; When a preset condition regarding the display of the first superimposed image is satisfied, control is performed to display the first superimposed image; If the condition is not satisfied, control is performed so that a second superimposed image in which the internal image is superimposed on the operative field image, and in which a display mode of the internal image is adjusted based on the position and orientation information, is displayed instead of the first superimposed image. The medical support device according to claim 1 .

3. The processor determines that the condition is satisfied when an angle representing the orientation of the internal image, the angle being the smallest when the imaging optical axis of the camera is parallel to the display surface of the internal image and the largest when the imaging optical axis is normal to the display surface of the internal image, is less than a predetermined threshold. The medical support device according to claim 2 .

4. The processor determines that the condition is satisfied when a user instructs the display of the first superimposed image. The medical support device according to claim 2 .

5. The processor controls to display information indicating the position of the camera on the first superimposed image. The medical support device according to claim 2 .

6. The camera is a camera provided at the tip of an endoscope used in laparoscopic surgery. The medical support device according to claim 2 .

7. a marker configured with an optically detectable pattern is provided on an outer peripheral surface of the medical instrument; The processor derives the position and orientation information based on the markers included in the surgical field image. The medical support device according to claim 2 .

8. The processor: Accepting a designation of an orientation of the three-dimensional model in the first superimposed image; The first superimposed image is generated based on the specified orientation of the three-dimensional model and the position and orientation information. The medical support device according to claim 1 .

9. Regarding the orientation of the internal image, when the viewpoint of the first superimposed image is located on a line perpendicular to the display surface of the internal image, the orientation is defined as the front. The processor specifies the orientation of the three-dimensional model so that the orientation of the internal image determined based on the position and orientation information faces the front side. The medical support device according to claim 8.

10. The processor controls the three-dimensional model included in the first superimposed image so as to increase the transparency of a portion located between a viewpoint of the first superimposed image and a display surface of the internal image. The medical support device according to claim 1 .

11. the medical instrument is an ultrasonic probe that transmits ultrasonic waves to the target area and detects electrical signals corresponding to ultrasonic echoes reflected from the target area; The internal image is an ultrasound image generated in response to the electrical signals. The medical support device according to claim 1 .

12. The processor: acquiring a three-dimensional image including the internal structure of the target region; extracting a second internal image corresponding to a display surface of the ultrasound image from the three-dimensional image based on the position and orientation information; generating a first superimposed image by superimposing the ultrasound image and the second internal image on the three-dimensional model; The medical support device according to claim 11.

13. The second internal image is a tomographic image of the target region including the display surface of the ultrasound image. The medical support device according to claim 12.

14. The second internal image is a blood vessel image showing the blood vessel structure of the target area passing through the display surface of the ultrasound image. The medical support device according to claim 12.

15. The processor generates the first superimposed image by using, as the internal image, an image obtained by removing a margin area in the ultrasound image that does not show the internal structure of the target region. The medical support device according to claim 11.

16. The processor: acquiring a three-dimensional image of the target area; Extracting the internal image from the three-dimensional image based on the position and orientation information. The medical support device according to claim 1 .

17. The processor acquires the three-dimensional model using a time-of-flight (ToF) camera. The medical support device according to claim 1 .

18. The processor acquires the three-dimensional model using a stereo camera. The medical support device according to claim 1 .

19. Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image in which the internal image is superimposed on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; A medical support method in which processing is performed by a computer.

20. Obtain a three-dimensional model showing the surface shape of the target area inside the body; acquiring an internal image showing the internal structure of the target region; acquiring position and orientation information indicating a position and orientation of the medical instrument in a surgical field including the target site and the medical instrument to be inserted into the body; a first superimposed image in which the internal image is superimposed on the three-dimensional model, the first superimposed image being a first superimposed image in which a display mode of the internal image is adjusted based on the position and orientation information; A medical support program that causes a computer to perform processing.

Images