Medical assistance device and program

Abstract

This medical assistance device comprises a processor. The processor acquires a surgical field image including an organ and an ultrasound probe that have been imaged, and an organ model representing the three-dimensional shape of the organ. The processor displays position orientation information and the organ model on a display. The position orientation information relates to a first position and a first orientation that are the position and the orientation of the ultrasonic probe with respect to the organ model corresponding to an ultrasonic image. The first position and the first orientation are derived on the basis of: organ structure information that is structure information of the organ; and structure information of the organ model corresponding to the inside of an imaging estimation range of the organ determined by the ultrasonic probe. The organ structure information is extracted from the ultrasonic image. The imaging estimation range is derived on the basis of: a second position that is the position of the ultrasound probe; a second orientation that is the orientation of the ultrasound probe; and predetermined range information. The second position and the second orientation are derived on the basis of the surgical field image.

Description

Medical support device and program

[0001] The present disclosure relates to a medical support device and a program.

[0002] When an ultrasound probe is inserted into the body to scan a target organ, it is more difficult to grasp the position and orientation of the ultrasound probe relative to the organ than when the ultrasound probe is scanned from the body surface.

[0003] Therefore, a method is disclosed in which an ultrasound imaging diagnostic device that generates a three-dimensional ultrasound image as shown in Non-Patent Document 1 allows a user to grasp the position and orientation of an ultrasound probe more accurately than by referring to a two-dimensional ultrasound image.

[0004] Furthermore, Non-Patent Document 2 discloses a method in which a magnetic field is generated by a magnetic transmitter, and the position and posture of an ultrasound probe inserted into the body are determined from the amount of change in the strength of the magnetic field detected by a magnetic sensor attached to the ultrasound probe.

[0005] GE Healthcare Japan, Ultrasound Imaging Diagnostic Devices for Obstetrics and Gynecology [Retrieved April 26, 2025], Internet, <URL: https: / / www.gehealthcare.co.jp / products / ultrasound / voluson> Tetsuya Kawagishi, "Achieving New 3D Ultrasound Images and High Resolution with Aplio (Registered Trademark) i800," Japan Medical Imaging Systems Industry Association Technical Report No. 52, 2017

[0006] However, ultrasound diagnostic imaging devices that generate three-dimensional ultrasound images are more expensive than ultrasound diagnostic imaging devices that generate two-dimensional ultrasound images, and the range of images that can be made three-dimensional is narrower.

[0007] Furthermore, when the position and orientation of an ultrasonic probe are determined using a magnetic sensor, a magnetic field is generated by a magnetic transmitter, which may adversely affect peripheral devices.

[0008] The present disclosure aims to provide a medical support device and program that enable a user to grasp the position and orientation of an ultrasound probe scanned along an organ inside the body without using an ultrasound imaging diagnostic device or magnetic sensor that generates three-dimensional ultrasound images.

[0009] A medical support device of a first aspect of the present disclosure includes a processor, which acquires an operative field image including an organ photographed by an internal camera and an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ, estimates the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the operative field image, extracts structural information of the organ from the ultrasound image of the organ, calculates an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the operative field image and predetermined range information, compares the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ, identifies the position and orientation of the ultrasound probe in the organ model corresponding to the ultrasound image of the organ, and displays information regarding the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

[0010] A medical support device of a second aspect according to the present disclosure is the medical support device of the first aspect, wherein the processor estimates the orientation of the organ from the surgical field image and calculates the estimated imaging range of the organ using the estimated orientation of the organ.

[0011] A medical support device of a third aspect according to the present disclosure is a medical support device according to the second aspect, in which the processor translates the structural information of the organ within the estimated imaging range of the organ in the organ model, and identifies, as the position and orientation of the ultrasound probe in the organ model, the position and orientation of the ultrasound probe that will result in an ultrasound image of the organ whose structural information is closest to the structural information of the organ model.

[0012] A fourth aspect of the medical support device according to the present disclosure is the medical support device according to the third aspect, in which the processor positions the orientation of the organ model in the same orientation as the orientation of the organ estimated from the surgical field image, and then determines the position and orientation of the ultrasound probe in the organ model.

[0013] A fifth aspect of the medical support device according to the present disclosure is the medical support device according to the first aspect, wherein the processor uses structural information of the organ obtained from multiple ultrasound images of the organ to identify the position and orientation of the ultrasound probe in an organ model corresponding to the ultrasound images of the organ.

[0014] A medical support device of a sixth aspect according to the present disclosure is the medical support device according to the first aspect, wherein the processor generates an organ model from a plurality of ultrasound images of the organ.

[0015] A seventh aspect of the medical support device according to the present disclosure is the medical support device according to the first aspect, wherein the structural information of the organ includes information regarding the shape of the organ, the position and shape of blood vessels and tumors contained in the organ, and the external appearance of the organ.

[0016] A medical support device of an eighth aspect of the present disclosure is a medical support device according to any one of the first to twelfth aspects, wherein the processor displays a probe image representing the ultrasound probe according to the identified position and orientation of the ultrasound probe by superimposing it on an organ model.

[0017] A ninth aspect of the medical support device according to the present disclosure is a medical support device according to the eighth aspect, wherein the processor displays at least one of an ultrasound image of an organ photographed by an ultrasound probe represented by a probe image, a plate surface representing the imaging range of the ultrasound probe, and a contour line representing the outline of the imaging range of the ultrasound probe, superimposed on the organ model.

[0018] A medical support device of a tenth aspect of the present disclosure is the medical support device of the ninth aspect, wherein the processor changes the position and orientation of the probe image, and the display position of at least one of the ultrasound image of the organ, the plate surface, and the contour line, in accordance with the movement of the ultrasound probe, and displays them on the organ model.

[0019] A medical support device of an eleventh aspect of the present disclosure is a medical support device of the ninth aspect, in which, when displaying a plate surface, the processor adjusts the transparency of the plate surface to be displayed superimposed on the organ model in accordance with a user's instructions.

[0020] A medical support device of a twelfth aspect of the present disclosure is a medical support device of the eighth aspect, in which the processor displays an organ model with the degree of transparency of structures constituting the organ model that exist at each distance adjusted according to the distance from the cross section of the organ model represented by the ultrasound image of the organ.

[0021] A medical support device of a thirteenth aspect of the present disclosure is a medical support device of the twelfth aspect, in which the processor displays an organ model in which the transparency of structures constituting the organ model at each distance increases as the distance from the cross section of the organ model increases.

[0022] A medical support device of a fourteenth aspect according to the present disclosure is a medical support device according to any one of the second to fourth aspects, wherein the processor displays the organ model so that, when the orientation of the organ in the surgical field image changes, the orientation of the organ model is the same as the orientation of the organ in the surgical field image.

[0023] A medical support device of a fifteenth aspect of the present disclosure is a medical support device of the fourteenth aspect, in which, when a user allows a change in the orientation of the organ model, the processor displays the organ model so that the orientation of the organ model is the same as the orientation of the organ in the surgical field image.

[0024] A medical support program of a sixteenth aspect according to the present disclosure is a program for causing a computer to execute a process of acquiring an operative field image including an organ photographed by an internal camera and an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ, estimating the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the operative field image, extracting structural information of the organ from the ultrasound image of the organ, calculating an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the operative field image and predetermined range information, comparing the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ, identifying the position and orientation of the ultrasound probe in the organ model corresponding to the ultrasound image of the organ, and displaying information related to the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

[0025] A seventeenth aspect of the present disclosure is a medical support device that includes a processor, wherein the processor acquires an operative field image including an organ and an ultrasound probe that receives reflected waves reflected by the organ by transmitting ultrasound to the organ, and an organ model representing the three-dimensional shape of the organ, and displays position and orientation information and the organ model on a display, wherein the position and orientation information is information relating to a first position and a first orientation that are the position and orientation of the ultrasound probe with respect to the organ model corresponding to the ultrasound image generated based on the reflected waves, and the first position and the first orientation are derived based on organ structure information that is structural information of the organ and structural information of the organ model that corresponds to an estimated imaging range of the organ by the ultrasound probe, the organ structure information is extracted from the ultrasound image generated based on the reflected waves, and the estimated imaging range is derived based on a second position that is the position of the ultrasound probe, a second orientation that is the orientation of the ultrasound probe, and predetermined range information, and the second position and second orientation are derived based on the operative field image.

[0026] An eighteenth aspect of the present disclosure is a medical support device according to the seventeenth aspect, in which the orientation of an organ is derived based on an operative field image, and an estimated imaging range of the organ is derived based on the orientation of the organ.

[0027] A 19th aspect of the present disclosure is a medical support device according to the 18th aspect, in which the position and orientation of the ultrasound probe that obtains an ultrasound image of the organ whose structural information is closest to the structural information of the organ model by translating the structural information of the organ within the estimated imaging range of the organ in the organ model is derived as the position and orientation of the ultrasound probe in the organ model.

[0028] A twentieth aspect of the present disclosure is a medical support device according to the nineteenth aspect, in which the orientation of the organ model and the orientation of the organ derived based on the surgical field image are arranged in the same orientation, and then the position and orientation of the ultrasound probe in the organ model are derived.

[0029] A 21st aspect of the present disclosure is a medical support device according to the 17th aspect, in which the position and orientation of the ultrasound probe in an organ model corresponding to an ultrasound image of the organ are derived based on structural information of the organ obtained from multiple ultrasound images representing the organ.

[0030] A twenty-second aspect of the present disclosure is the medical support device according to the seventeenth aspect, in which the organ model is generated based on a plurality of ultrasound images representing the organ.

[0031] A 23rd aspect of the present disclosure is a medical support device according to the 17th aspect, in which the structural information of the organ includes information regarding the shape of the organ, blood vessels contained in the organ, the position of a tumor contained in the organ, the shape of the tumor, and / or the appearance of the organ.

[0032] A 24th aspect of the present disclosure is a medical support device according to the 17th aspect, in which a probe image representing an ultrasound probe in a first position and a first posture is displayed on a display superimposed on an organ model.

[0033] A 25th aspect of the present disclosure is a medical support device according to the 24th aspect, in which at least one of an ultrasound image obtained by imaging with an ultrasound probe represented by a probe image, a plate surface representing the imaging range of the ultrasound probe, and a contour line representing the outline of the imaging range of the ultrasound probe are displayed on a display superimposed on an organ model.

[0034] A 26th aspect of the present disclosure is a medical support device according to the 25th aspect, in which the position and orientation of the probe image, and the display position of at least one of the ultrasound image of the organ, the plate surface, and the contour line change in accordance with the movement of the ultrasound probe.

[0035] A 27th aspect of the present disclosure is a medical support device according to the 25th aspect, in which, when the plate surface is displayed on the display, the transparency of the plate surface displayed superimposed on the organ model is adjusted according to given instructions.

[0036] A 28th aspect of the present disclosure is a medical support device according to the 24th aspect, in which, when an organ model is displayed on a display, the degree of transparency of structures constituting the organ model that exist at each distance from a cross-section of the organ model represented by an ultrasound image of the organ is adjusted according to the distance.

[0037] A 29th aspect of the present disclosure is a medical support device according to the 28th aspect, in which, when an organ model is displayed on a display, the transparency of structures constituting the organ model at each distance from the cross section of the organ model increases as the distance from the cross section increases.

[0038] A 30th aspect of the present disclosure is a medical support device according to the 18th aspect, in which, when the orientation of an organ in the surgical field image displayed on the display changes, the orientation of the organ model displayed on the display becomes the same as the orientation of the organ in the surgical field image.

[0039] A thirty-first aspect of the present disclosure is a medical support device according to the thirtieth aspect, in which, when an instruction to allow a change in the orientation of the organ model is given, the orientation of the organ model displayed on the display becomes the same as the orientation of the organ in the surgical field image.

[0040] A 32nd aspect of the present disclosure is a program for causing a computer to execute processing, the processing including acquiring an operative field image including an organ and an ultrasound probe that receives reflected waves reflected by the organ by transmitting ultrasound to the organ, and an organ model representing the three-dimensional shape of the organ, and displaying position and orientation information and the organ model on a display, wherein the position and orientation information is information regarding a first position and a first orientation that are the position and orientation of the ultrasound probe with respect to the organ model corresponding to the ultrasound image generated based on the reflected waves, the first position and the first orientation are derived based on organ structure information that is structural information of the organ and structural information of the organ model that corresponds to an estimated imaging range of the organ by the ultrasound probe, the organ structure information is extracted from the ultrasound image generated based on the reflected waves, the estimated imaging range is derived based on a second position that is the position of the ultrasound probe, a second orientation that is the orientation of the ultrasound probe, and predetermined range information, and the second position and second orientation are derived based on the operative field image.

[0041] A thirty-third aspect of the present disclosure is a medical support device that includes a processor, which acquires an operative field image showing an organ and an ultrasound probe that transmits ultrasound waves to the organ and receives reflected waves from the organ, and an organ model that represents the three-dimensional shape of the organ, derives ultrasound probe information regarding the position and posture of the ultrasound probe based on the operative field image, and displays the ultrasound probe information and the organ model on a display.

[0042] According to the present disclosure, it is possible to allow a user to grasp the position and orientation of an ultrasound probe that has been scanned along an organ inside the body without using an ultrasound diagnostic imaging device that generates a three-dimensional ultrasound image or a magnetic sensor.

[0043] 1 is a diagram illustrating an example of the configuration of a medical support system. FIG. 1 is a diagram illustrating an example of an endoscope surgery. FIG. 2 is a diagram illustrating an example of an organ model. FIG. 3 is a diagram illustrating an example of the hardware configuration of a medical support device according to the first embodiment. FIG. 4 is a flowchart illustrating an example of the flow of medical support processing according to the first embodiment. FIG. 4 is a diagram illustrating an example of a state of an ultrasound probe perpendicular to the imaging optical axis. FIG. 5 is a diagram illustrating an example of a state of an ultrasound probe tilted with respect to the imaging optical axis. FIG. 6 is a diagram illustrating an example of an organ model image on which a probe image is superimposed. FIG. 7 is a diagram illustrating an example of an organ model image on which a plate surface is superimposed. FIG. 8 is a diagram illustrating an example of an organ model image on which a contour line representing the contour of the plate surface is superimposed. FIG. 9 is a diagram illustrating an example of an organ model image on which an ultrasound image is superimposed. FIG. 10 is a diagram illustrating an example of an organ model image on which a probe image and a plate surface are superimposed on an organ model. FIG. 11 is a diagram illustrating an example of an organ model image on which a texture image is applied. FIG. 12 is a conceptual diagram illustrating an example of an aspect in which the transparency within a cross section when an organ model is virtually cut by an imaging plane is lowest and the transparency increases with increasing distance from the imaging plane. FIG. 13 is a diagram illustrating an example of a display of an organ model in which the transparency of a structure has been adjusted. FIG. 14 is a diagram illustrating an example of the hardware configuration of a medical support device according to the second embodiment. FIG. 15 is a flowchart illustrating an example of the flow of medical support processing according to the second embodiment.

[0044] The present embodiment will be described below with reference to the drawings. Note that the same components and processes are denoted by the same reference numerals throughout the drawings, and redundant descriptions will be omitted. The dimensional proportions in the drawings are exaggerated for ease of explanation and may differ from the actual proportions. Furthermore, in this specification, imaging using an ultrasonic probe refers to a series of processes in which the ultrasonic probe receives reflected waves obtained when ultrasonic waves transmitted from the ultrasonic probe are reflected at a target site, and an image processing processor for the ultrasonic probe at a subsequent stage performs imaging based on the received signals. In other words, imaging using an ultrasonic probe can be described as a series of operations in which a probe equipped with an ultrasonic transducer transmits ultrasonic waves toward a target site, receives reflected waves, and an ultrasonic image is generated by an image processing processor for the ultrasonic probe based on the reflected waves (i.e., a series of processes in which an ultrasonic image is generated using signals obtained via the ultrasonic probe). Furthermore, in this specification, an image captured by an ultrasonic probe refers to an ultrasonic image reconstructed by an image processing processor for the ultrasonic probe based on signals transmitted and received by the ultrasonic probe (in other words, an ultrasonic image generated based on reflected waves received by the ultrasonic probe).

[0045] First Embodiment FIG. 1 is a diagram showing an example of the configuration of a medical support system 10 according to the present disclosure.

[0046] As an example, the medical support system 10 is used when performing endoscopic surgery on a patient PT using an endoscope 13. Unlike open surgery, endoscopic surgery is a surgery performed by making a small hole in the patient PT's body and inserting a medical device such as an endoscope 13 through the hole. The medical support system 10 provides the medical staff ST, including doctors, with a view of the surgical field. Specifically, the medical support system 10 provides the medical staff ST with an image of the state of the inside of the patient PT's body using the endoscope 13.

[0047] Furthermore, the medical support system 10 provides support information for supporting medical treatment such as surgery and examinations, in addition to the field of view of the surgical field. Such a medical support system 10 has the function of providing support information in real time during surgery, and is therefore also called a "surgical navigation system."

[0048] The medical support system 10 includes, for example, a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is an example of the "medical support device" and "computer" according to the present disclosure.

[0049] The medical support device 11 is communicatively connected to the endoscope 13, the ultrasonic probe 14, the display 16, and the storage 44. The ultrasonic probe 14 is an example of an "ultrasonic probe" according to the present disclosure. In endoscopic surgery, a portion of the endoscope 13 and the ultrasonic probe 14, including their respective distal ends, is inserted into the body via a trocar 17. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and the trocar 17 is used to insert the endoscope 13 and the ultrasonic probe 14 into the body. The trocar 17 is an insertion tool that has an insertion hole through which the endoscope 13 and / or the ultrasonic probe 14 is inserted and a valve that is provided in the insertion hole to prevent gas leakage.

[0050] FIG. 2 shows the state in which an endoscope 13 and an ultrasonic probe 14 are inserted into the abdomen of a patient PT.

[0051] Hereinafter, an example will be described in which endoscopic surgery is performed while visualizing the inside of the liver (LV) using an ultrasound image 22. The liver (LV) is an example of an "organ" according to the present disclosure. Note that the target site of endoscopic surgery is not limited to the liver (LV) and may be another organ.

[0052] The endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT, and a camera 13C and a light source for illumination (such as an LED (Light Emitting Diode)) are built into a tip section 13B of the insertion section 13A. The endoscope 13 is, for example, a rigid endoscope with a hard insertion section 13A, and is also called a laparoscope because it is often used for observing the abdominal cavity. The camera 13C has an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface (in other words, the "light receiving surface") of the image sensor. The image sensor is, for example, an image sensor that is capable of capturing images to obtain color images. The camera 13C is an example of an "intracorporeal camera" according to the present disclosure.

[0053] The endoscope 13 optically captures an internal surgical field SFin including a target site (hepatic LV in this example) inside the patient PT using a camera 13C. The endoscope 13 is connected to an endoscope image processor (not shown), which processes the captured image signal output by the image sensor to generate an internal surgical field image 21 of the internal surgical field SFin. The internal surgical field image 21 is an example of a "surgical field image" according to the present disclosure.

[0054] The illumination light for the endoscope 13 may be special light such as ultraviolet or infrared light, but visible light such as white light is also used. The special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet region. An internal surgical field image 21 obtained by capturing an image of the internal surgical field SFin with the endoscope 13 is transmitted in real time to the medical support device 11 via an image processor for the endoscope. The internal surgical field image 21 is displayed as a moving image on the display 16. In FIG. 2 , the symbols Xin and Yin indicate the coordinate system of the internal surgical field image 21.

[0055] The ultrasound probe 14 has an insertion section 14A inserted into the body of the patient PT, similar to the endoscope 13, and an operation section 14D at the proximal end of the insertion section 14A. An ultrasound transducer 14C is built into the distal end 14B of the insertion section 14A. The ultrasound transducer 14C transmits ultrasound waves to the target region and receives reflected waves from the target region. The ultrasound probe 14 is connected to an ultrasound probe image processing processor (not shown). The ultrasound probe image processing processor performs image reconstruction processing based on signals corresponding to the reflected waves received by the ultrasound probe 14. The image reconstruction processing generates a two-dimensional ultrasound image 22 showing the internal structure of the target region scanned by the ultrasound probe 14. The ultrasound image 22 is a so-called B (brightness) mode image that visualizes the internal structure of the target region, from the superficial to the deep layers reached by ultrasound waves, as brightness information. The ultrasound image 22 visualizes the internal structure of the target region that cannot be observed in the internal surgical field image 21 obtained by optical imaging.

[0056] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and acquires a fan-shaped image with the ultrasonic transducer 14C as the base point. By scanning the target area with the ultrasonic probe 14, an ultrasonic image 22 corresponding to each position along the scanning direction is acquired by an image processing processor (not shown) for the ultrasonic probe. The ultrasonic image 22 is transmitted in real time to the medical support device 11 via the image processing processor for the ultrasonic probe. The ultrasonic image 22 is displayed as a moving image on the display 16. In FIG. 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0057] An image-recognizable marker M1 is provided on the tip 14B of the ultrasound probe 14. The marker M1 is photographed by the camera 13C of the endoscope 13. As an example, the marker M1 is configured by a pattern having predetermined morphological characteristics. As will be described later, the medical support device 11 uses the marker M1 to estimate the position and orientation of the tip 14B of the ultrasound probe 14 that scans the surface of the liver LV. Hereinafter, estimating the position and orientation of the tip 14B of the ultrasound probe 14 that scans the surface of the liver LV will simply be referred to as "estimating the position and orientation of the ultrasound probe 14."

[0058] Meanwhile, as shown in FIG. 1 , a storage 44 for storing data is connected to the medical support device 11. The storage 44 is configured, for example, by a hard disk drive (HDD) and / or a solid state drive (SSD). As an example, the storage 44 stores an organ model 20 representing the three-dimensional shape of a target region (the liver LV in this example) captured in an ultrasound image 22 acquired by imaging using the ultrasound probe 14. The organ model 20 is an example of a three-dimensional model of an organ. The organ model 20 is generated in advance by capturing images of the organ using, for example, computed tomography (CT) and / or magnetic resonance imaging (MRI) before performing endoscopic surgery. FIG. 3 shows an example of the organ model 20. In FIG. 3 , the symbols X, Y, and Z indicate a three-dimensional coordinate system of the organ model 20. The three-dimensional coordinate system of the organ model 20 is referred to as a "model coordinate system."

[0059] The organ model 20 represents structural information of the structures that make up the organ. The structural information of the organ includes, for example, the shape of the organ, the blood vessels V contained in the organ, the position of a tumor contained in the organ, the shape of the tumor contained in the organ, and information about the organ's appearance. The information about the organ's appearance includes, for example, information about the pattern (hereinafter referred to as "texture") that appears on the surface of the organ due to the state of the blood vessels V and / or tissue. The structural information of the structures that make up the organ is an example of "organ structural information" according to the present disclosure.

[0060] The medical support device 11 acquires the organ model 20 of the target area shown in the ultrasound image 22 from the storage 44, and displays the organ model image 23, which is a three-dimensional representation of the organ model 20, on the display 16 (see FIG. 1).

[0061] As will be described later, the organ model image 23 also displays information about the ultrasound probe 14 whose position and orientation relative to the organ have been estimated.

[0062] The medical support device 11 acquires an endoscopic surgical field image 21 (i.e., an endoscopic surgical field image 21 showing the liver LV and the ultrasound probe 14) including the liver LV captured by the camera 13C of the endoscope 13 and the ultrasound probe 14 scanning the surface of the liver LV to generate an ultrasound image 22 of the liver LV in an ultrasound probe image processor. The medical support device 11 then determines, from the endoscopic surgical field image 21, a relative positional relationship RP2 (see FIG. 2 ) between the camera 13C and a marker M1 provided on the tip 14B of the ultrasound probe 14. For example, the medical support device 11 determines the relative positional relationship RP2 between the camera 13C and the marker M1 by performing AI and / or non-AI processing on the endoscopic surgical field image 21. For example, AI processing refers to processing in which the internal surgical field image 21 is input to a trained model obtained by performing machine learning using teacher data including an example image corresponding to the internal surgical field image 21 and correct answer data corresponding to the positional relationship RP2, and the trained model derives the positional relationship RP2. Furthermore, non-AI processing refers to processing in which image recognition processing such as template matching is performed on the internal surgical field image 21 to recognize the camera 13C and the marker M1, and the positional relationship RP2 is derived based on the recognition result.

[0063] The medical support device 11 estimates the position and orientation of the ultrasound probe 14 scanning the surface of the liver LV based on the positional relationship RP2. The medical support device 11 superimposes information about the ultrasound probe 14, the position and orientation of which has been estimated relative to the liver LV, on the organ model 20 and displays it on the display 16 as an organ model image 23 (see FIG. 1 ).

[0064] This allows the display 16 to provide the medical staff ST with the position and orientation of the ultrasound probe 14 that has scanned along the liver LV inside the body.

[0065] Next, the functions of the medical support device 11 will be described in detail.

[0066] 4 is a diagram showing an example of the hardware configuration of the medical support device 11. The medical support device 11 is configured, for example, by a computer. The medical support device 11 includes a display 16, a processor 41, a reception device 42, a RAM (Random Access Memory) 43, a storage 44, a communication I / F (Interface) 45, and an external I / F 46. Each of these components is connected to a bus 48 and can communicate with each other.

[0067] The medical support device 11 is operated by an operator such as a medical staff member ST through a reception device 42. The reception device 42 has a keyboard, a mouse, etc. (not shown) and receives instructions from the operator. The reception device 42 may be, for example, a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0068] The display 16 may be, for example, an organic electroluminescence (EL) display or a liquid crystal display. As shown in Fig. 1, the medical support system 10 has three displays 16, and each display 16 displays an internal surgical field image 21, an ultrasound image 22, and an organ model image 23, respectively.

[0069] The processor 41 is, for example, a CPU (Central Processing Unit), and controls each part of the medical support device 11 in an integrated manner according to a medical support program 49. The medical support program 49 is an example of a "program" according to the present disclosure.

[0070] The storage 44 is a non-volatile storage device that stores a medical support program 49 for causing the computer to function as the medical support device 11, various parameters 50, and the like.

[0071] The RAM 43 is a memory that temporarily stores information and is used as a work memory by the processor 41. The RAM 43 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0072] The communication I / F 45 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control in accordance with communication protocols defined by various wired or wireless communication standards.

[0073] The external I / F 46 includes, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer and a memory card.

[0074] The processor 41 reads the medical support program 49 from the storage 44 and executes the medical support program 49 on the RAM 43 to perform medical support processing.

[0075] The medical support processing is realized by the processor 41 executing the medical support program 49, causing the processor 41 to operate as an image acquisition unit 41A, an organ model acquisition unit 41B, a position and orientation information estimation unit 41C, a structural information extraction unit 41D, an estimated shooting range calculation unit 41E, a position and orientation identification unit 41F, and a display unit 41G.

[0076] The parameters 50 include, for example, information about the marker M1. The parameters 50 include, for example, information about how the marker M1 appears when the marker M1 is photographed by the camera 13C of the endoscope 13 from various angles and distances.

[0077] The image acquisition unit 41A acquires an internal surgical field image 21 from the endoscope 13, and acquires an ultrasound image 22 of the liver LV from the ultrasound probe 14. The image acquisition unit 41A acquires the internal surgical field image 21 and the ultrasound image 22 via, for example, the external I / F 46 or the communication I / F 45.

[0078] The organ model acquisition unit 41B acquires the organ model 20 of the liver LV of the patient PT, which is the subject of the acquired ultrasound image 22, from the storage 44. Similar to the image acquisition unit 41A, the organ model acquisition unit 41B acquires the organ model 20 via, for example, the external I / F 46 or the communication I / F 45.

[0079] The position and orientation information estimating unit 41C derives the position and orientation of the ultrasound probe 14 scanning the surface of the liver LV based on the internal surgical field image 21 acquired by the image acquiring unit 41A. For example, the position and orientation information estimating unit 41C estimates the position and orientation of the ultrasound probe 14 scanning the surface of the liver LV based on the display status of the marker M1 included in the internal surgical field image 21 acquired by the image acquiring unit 41A. Hereinafter, information regarding the position and orientation of the ultrasound probe 14 will be referred to as "position and orientation information." Note that the position estimated by the position and orientation information estimating unit 41C is an example of a "second position" according to the present disclosure, and the orientation estimated by the position and orientation information estimating unit 41C is an example of a "second orientation" according to the present disclosure.

[0080] The structural information extraction unit 41D extracts structural information of the liver LV from the ultrasound image 22 generated by the image processing processor for the ultrasound probe based on the reflected waves received by the ultrasound probe 14 (i.e., the reflected waves from the liver LV due to ultrasound transmitted from the ultrasound probe 14 to the liver LV).

[0081] The estimated imaging range calculation unit 41E derives an estimated imaging range of the liver LV by the ultrasound probe 14. For example, the estimated imaging range calculation unit 41E calculates an estimated imaging range of the liver LV by the ultrasound probe 14 using the position and orientation of the ultrasound probe 14 estimated from the internal surgical field image 21 and range information set in advance for the estimated position and orientation of the ultrasound probe 14.

[0082] The position and orientation identification unit 41F derives the position and orientation of the ultrasound probe 14 relative to the organ model 20. For example, the position and orientation identification unit 41F compares structural information of the liver LV extracted by the structural information extraction unit 41D with structural information of the organ model 20 corresponding to the estimated imaging range of the liver LV calculated by the estimated imaging range calculation unit 41E, and identifies the position and orientation of the ultrasound probe 14 relative to the organ model 20 based on the comparison result. The structural information of the organ model 20 is an example of "structural information of an organ model" according to the present disclosure. The position identified by the position and orientation identification unit 41F is an example of a "first position" according to the present disclosure, and the orientation identified by the position and orientation identification unit 41F is an example of a "first orientation" according to the present disclosure.

[0083] The display unit 41G displays the position and orientation information identified by the position and orientation identifying unit 41F by superimposing it on the organ model 20. The position and orientation information identified by the position and orientation identifying unit 41F is an example of the "position and orientation information" according to the present disclosure.

[0084] Next, the operation of the medical support device 11 will be described in detail.

[0085] FIG. 5 is a flowchart showing an example of the flow of medical support processing executed by the medical support device 11 when the processor 41 receives, via the reception device 42, an instruction from the medical staff ST to start navigation of the ultrasound probe 14 inserted into the body of the patient PT.

[0086] It is assumed that an organ model 20 of the liver LV, which is the target site for surgery, is generated in advance and stored in the storage 44 before the navigation start instruction is received.

[0087] In step S10, the processor 41 acquires an internal surgical field image 21 generated by capturing an image of the internal surgical field SFin using the camera 13C of the endoscope 13. The internal surgical field image 21 includes the liver LV and the ultrasound probe 14 scanning the surface of the liver LV (i.e., the internal surgical field image 21 shows the liver LV and the ultrasound probe 14).

[0088] In step S20, the processor 41 acquires from the storage 44 the organ model 20 of the liver LV of the patient PT undergoing surgery.

[0089] In step S30, the processor 41 detects a marker M1 provided on the tip 14B of the ultrasound probe 14 from the internal surgical field image 21 acquired by the processing of step S10. For example, the processor 41 detects the marker M1 by performing AI or non-AI processing on the internal surgical field image 21.

[0090] In step S40, the processor 41 estimates position and orientation information of the ultrasound probe 14 based on the appearance of the pattern of the marker M1 detected by the processing in step S30. For example, the position and orientation information is derived by a mathematical method (e.g., geometric analysis using a perspective projection model and transformation processing between coordinate systems) based on a known geometric pattern of the marker M1 captured in the internal surgical field image 21 (e.g., the grid pattern 56 shown in FIGS. 6 and 7 ). In particular, the position and orientation information is derived by solving an inverse problem based on the positional relationship of geometric feature points when the known three-dimensional pattern is projected onto a two-dimensional image (i.e., deformation due to perspective projection). Here, the position and orientation information is derived by performing coordinate transformation between coordinate systems (i.e., between the coordinate system defined for the marker M1 and the camera coordinate system) and geometric analysis using a perspective projection model.

[0091] 6 and 7 show the correspondence relationship between the position and orientation of the ultrasound probe 14 having a marker M1 at its tip 14B and the position and orientation of the ultrasound probe 14 within the internal surgical field SFin defined as a three-dimensional space. The Zin axis of the internal surgical field SFin, which is a three-dimensional space, is parallel to the imaging optical axis of the camera 13C of the endoscope 13, and the Xin-Yin plane is a plane parallel to the imaging surface of the camera 13C and perpendicular to the imaging optical axis. The Xin-Yin plane is parallel to the screen of the internal surgical field image 21. The three-dimensional coordinate system in the internal surgical field SFin is called the "internal coordinate system."

[0092] As shown in FIGS. 6 and 7, the marker M1 is represented by, for example, a grid pattern 56 with predetermined intervals, but may also be a dot pattern with predetermined intervals between adjacent dots.

[0093] 6 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13C in the internal surgical field SFin in three-dimensional space, more specifically, a state in which the axial direction of the tip 14B is parallel to the Xin axis. When the tip 14B of the ultrasound probe 14 is in this position, the orthogonal lines of the grid pattern 56 of the marker M1 in the internal surgical field image 21 are parallel to the Xin axis and the Yin axis, respectively, and the distances between adjacent intersections 56A are also the same.

[0094] On the other hand, Figure 7 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13C within the internal surgical field SFin in three-dimensional space, but is tilted toward the Zin axis. For example, the posture of the ultrasound probe 14 shown in Figure 7 represents a state in which the tip 14B is rotated approximately -15° around the Y axis from the posture shown in Figure 6. When the tip 14B is in this posture, in the internal surgical field image 21, the lines of the grid pattern 56 of the marker M1 that extend in the circumferential direction become shorter as the distance from the camera 13C in the Zin axis direction increases. Furthermore, the distance between adjacent intersections 56A in the axial direction of the tip 14B also becomes shorter as the distance from the camera 13C in the Zin axis direction increases.

[0095] In this way, the shape of the marker M1 reflected in the internal surgical field image 21 changes depending on the position and orientation of the ultrasound probe 14. The processor 41 estimates the position and orientation information of the ultrasound probe 14 based on the shape of the marker M1 in the internal surgical field image 21.

[0096] 6, the position and orientation information indicates, for example, that the axial direction of the tip portion 14B of the ultrasound probe 14 is parallel to the Xin-Yin plane and the Xin-Zin plane and perpendicular to the Yin-Zin plane. Also, the coordinates of the position of the reference point of the tip portion 14B (for example, the tip of the tip portion 14B) are information such as X1, Y1, and Z1.

[0097] 7, the position and orientation information indicates, for example, that the axial direction of the tip 14B of the ultrasound probe 14 is at 75° with respect to the Xin-Yin plane, parallel to the Xin-Zin plane, and at −15° with respect to the Yin-Zin plane. Also, the coordinates of the position of the reference point of the tip 14B are, for example, X2, Y2, and Z2.

[0098] In this way, the processor 41 estimates the position and orientation information of the ultrasound probe 14 based on the appearance of the pattern of the marker M1.

[0099] 5, the processor 41 acquires an ultrasound image 22 of the liver LV from the ultrasound probe 14 and extracts structural information of the liver LV from the ultrasound image 22. To extract the structural information from the ultrasound image 22, for example, a known extraction method is used to extract feature amounts of structures that make up the liver LV from the image.

[0100] The processing of step S40 estimates the position and orientation of the ultrasound probe 14 relative to the liver LV, but the orientation of the liver LV in the internal surgical field image 21 does not necessarily match the orientation of the organ model 20. Therefore, even if the ultrasound probe 14 is placed on the organ model 20 according to the estimated position and orientation, the position and orientation of the ultrasound probe 14 relative to the organ model 20 may differ from the position and orientation of the ultrasound probe 14 relative to the liver LV in the internal surgical field image 21.

[0101] However, since the position and orientation of the ultrasound probe 14 relative to the liver LV in the internal surgical field SFin are estimated, the relative difference in the position and orientation of the ultrasound probe 14 relative to the organ model 20 is limited to a certain range.

[0102] Therefore, in step S60, the processor 41 sets a specified range for the position and orientation of the ultrasound probe 14 estimated by the process of step S40, and calculates an estimated imaging range of the ultrasound probe 14.

[0103] The specified range is, for example, a three-dimensional region set based on the position and orientation of the ultrasound probe 14, and indicates the range of a three-dimensional region having a predetermined size that includes the ultrasound transducer 14C of the ultrasound probe 14. The specified range is set to a range smaller than the size of the liver LV, and data representing the specified range is stored in advance in the storage 44 as an example of parameters 50. The specified range is an example of "predetermined range information" according to the present disclosure.

[0104] Note that the model coordinate system in the organ model 20 is different from the internal body coordinate system in the internal surgical field SFin. Therefore, in step S70, the processor 41 converts the position and orientation of the ultrasound probe 14 and the specified range in the internal body coordinate system into the space of the model coordinate system. The range on the model coordinate system corresponding to the specified range represents the estimated imaging range of the ultrasound image 22.

[0105] In step S80, the processor 41 compares, within the estimated imaging range, the structural information of the liver LV extracted from the ultrasound image 22 by the processing of step S50 with the structural information of the organ model 20. The processor 41 specifies, in the model coordinate system, the position and orientation of the ultrasound probe 14 that will obtain an ultrasound image 22 in which the structural information of the liver LV is closest to the structural information of the organ model 20, as the position and orientation information of the ultrasound probe 14 in the organ model 20. That is, the processor 41 specifies the position and orientation information of the ultrasound probe 14 in the organ model 20 from the similarity between the thickness of the blood vessel V, the position of the blood vessel V, the size of the tumor, the position of the tumor, and the shape of the liver LV in the ultrasound image 22 and the organ model 20.

[0106] Since the estimated imaging range of the ultrasound image 22 is set in the organ model 20 based on the position and orientation information of the ultrasound probe 14 relative to the liver LV, the range for comparing the structural information of the liver LV with the structural information of the organ model 20 is limited to the estimated imaging range. Therefore, the time required to identify the position and orientation information of the ultrasound probe 14 in the organ model 20 can be shortened compared to when comparing the structural information of the liver LV with the structural information of the organ model 20 for the entire liver LV.

[0107] As already explained, in the internal surgical field image 21, the distance between the lines extending in the circumferential direction of the grid pattern 56 of the marker M1 on the ultrasound probe 14 and the distance between the intersections 56A adjacent in the axial direction becomes shorter as the distance from the camera 13C in the Zin axis direction increases. In other words, if the appearance of the grid pattern 56 of the marker M1 is associated in advance with the distance from the camera 13C to the marker M1 in the Zin axis direction, the distance from the camera 13C to the marker M1 in the Zin axis direction can be obtained from the appearance of the grid pattern 56 of the marker M1.

[0108] If the focal length of the camera 13C in the endoscope 13 is kept constant, the distance from the camera 13C to the marker M1 in the Zin axis direction, i.e., the subject distance, can be used to obtain a rough magnification ratio of the liver LV shown in the internal surgical field image 21 relative to the actual size of the liver LV.

[0109] Therefore, based on the scaling ratio of the liver LV, the processor 41 can display the organ model 20 in the organ model image 23 in a size that matches the size of the liver LV depicted in the anatomical field image 21. This makes it easier to understand the correspondence between the positional relationship between the liver LV in the anatomical field image 21 and the organ model 20 in the organ model image 23 than when the size of the organ model 20 is displayed without being adjusted to the size of the liver LV depicted in the anatomical field image 21.

[0110] In step S90, the processor 41 displays on the display 16 an organ model image 23 in which the position and orientation information of the ultrasound probe 14 in the organ model 20 identified by the processing in step S80 is superimposed on the organ model 20.

[0111] 8 to 15 show examples of displaying the position and orientation information of the ultrasound probe 14 in the organ model image 23. FIG.

[0112] 8 is a diagram showing an example of an organ model image 23 in which an image representing the ultrasound probe 14, i.e., a probe image 14X, is superimposed on the organ model 20 in accordance with the identified position and orientation information of the ultrasound probe 14. Since the probe image 14X is displayed on the organ model 20 in accordance with the position and orientation information of the ultrasound probe 14, it is possible to provide the medical staff ST with information indicating at what position and in what orientation the ultrasound probe 14 is scanning the liver LV.

[0113] 9 is a diagram showing an example of an organ model image 23 on which a plate surface 22A, i.e., a surface representing the imaging range of the ultrasound image 22 of the ultrasound probe 14 (i.e., the range of the cross section represented by the ultrasound image 22), is superimposed in accordance with the identified position and orientation information of the ultrasound probe 14. If the ultrasound probe 14 is a convex type that transmits ultrasound waves radially, the plate surface 22A has a trapezoidal shape that extends radially from the axial width of the ultrasound transducer 14C. The plate surface 22A is displayed, for example, so as to intersect with the organ model 20, and can notify the medical staff ST of which surface of the organ model 20 the ultrasound image 22 represents.

[0114] The processor 41 may display the plate surface 22A on the organ model image 23 with a transparency sufficient to allow the blood vessels V and other structures of the organ model 20 behind the plate surface 22A to be visible. The transparency of the plate surface 22A can be changed by the medical staff ST. The higher the transparency, the easier it is to see the structures of the organ model 20 behind the plate surface 22A. By making the plate surface 22A transparent, structural information of the organ model 20 on the imaging plane of the ultrasound image 22 can be provided to the medical staff ST. Here, the structural information of the organ model 20 on the imaging plane of the ultrasound image 22 refers to structural information of the region within the entire region of the organ model 20 within the cross section represented by the ultrasound image 22 (in other words, structural information of the organ model 20 that exists on the same plane as the cross section represented by the ultrasound image 22 (e.g., structures such as blood vessels and / or tumors)). The imaging plane refers to a plane within the organ model 20 that coincides with the ultrasound image 22. Since the ultrasound image 22 is a cross-sectional image, a cross section that overlaps with the ultrasound image 22 is extracted from the organ model 20, and information about the structures present on the extracted cross section is displayed on the display 16 as "structural information" and provided to the medical staff ST.

[0115] 10 is a diagram showing an example of an organ model image 23 in which only a contour line 22B representing the contour of the plate surface 22A is superimposed in accordance with the identified position and orientation information of the ultrasound probe 14. Because only the imaging range of the ultrasound image 22 is displayed by the contour line 22B, it is easier to confirm the imaging range of the ultrasound image 22 in the organ model 20 compared to when the plate surface 22A is displayed.

[0116] 11 is a diagram showing an example of an organ model image 23 on which an ultrasound image 22 generated by an image processing processor for the ultrasound probe based on reflected waves received by the ultrasound probe 14 in accordance with the identified position and orientation information of the ultrasound probe 14 is superimposed. By superimposing the ultrasound image 22 on the organ model 20, structural information about the liver LV, which is difficult to grasp from the ultrasound image 22 alone, can be provided to the medical staff ST.

[0117] When superimposing the ultrasound image 22 on the organ model 20, the processor 41 may superimpose a transparent ultrasound image 22 on the organ model image 23, similar to the display of the plate surface 22A. The transparency of the ultrasound image 22 can be changed by the medical staff ST.

[0118] The processor 41 may also display on the display 16 an organ model image 23 in which at least one of the probe image 14X, the plate surface 22A, the contour line 22B, and the ultrasound image 22 described above is superimposed on the organ model 20. In this case, more information can be provided to the medical staff ST in a comprehensive manner compared to when the probe image 14X, the plate surface 22A, the contour line 22B, and the ultrasound image 22 are displayed individually.

[0119] FIG. 12 is a diagram showing an example of an organ model image 23 in which the probe image 14X and the plate surface 22A are superimposed on the organ model 20.

[0120] When the organ model image 23 is displayed on the display 16 , the processor 41 may display the texture of the liver LV on the surface of the organ model 20 .

[0121] 13 is a diagram showing an example in which a texture image 25 representing the texture of the liver LV is displayed on the surface of the organ model 20. The texture of the liver LV serves as a marker indicating the location of the liver LV. By displaying the texture image 25 of the liver LV on the surface of the organ model 20, it becomes easier to grasp the position and orientation information of the ultrasound probe 14 relative to the liver LV compared to when the texture image 25 of the liver LV is not displayed on the surface of the organ model 20.

[0122] The texture image 25 to be displayed on the surface of the organ model 20 is obtained, for example, by imaging using CT, MRI, etc., but the processor 41 may also obtain the texture image 25 to be displayed on the surface of the organ model 20 from the internal surgical field image 21.

[0123] In this way, the processor 41 displays the organ model 20 and ultrasound image information on the organ model image 23. Ultrasound image information is a general term for information including at least one of the probe image 14X, the plate surface 22A, the contour line 22B, the ultrasound image 22, and the texture image 25, and includes information that is displayed superimposed on the organ model 20. Information that is displayed superimposed on the organ model 20 does not necessarily mean information whose display position overlaps with the organ model 20, but rather means information that is displayed on the same screen as the organ model 20 or on a different screen.

[0124] When the processor 41 displays the ultrasound image 22 or the plate surface 22A for the organ model 20, the processor 41 may display the organ model 20 on the organ model image 23 with the degree of transparency of the structures of the organ model 20 present at each distance adjusted according to the distance in the normal direction (e.g., the direction perpendicular to the cross section represented by the ultrasound image 22) relative to the imaging surface of the organ model 20 represented by the ultrasound image 22 or the plate surface 22A (i.e., the surface indicating which cross section of the organ model 20 is visualized by the ultrasound image 22 or the plate surface 22A simulating the range of the ultrasound image 22).

[0125] Specifically, the processor 41 displays the organ model 20 on the organ model image 23 with the degree of transparency of the structures of the organ model 20 present at each distance increasing as the distance in the normal direction to the imaging plane of the organ model 20 (i.e., the plate surface 22A (in other words, a virtual cross section in which the same plane as the cross section represented by the ultrasound image 22 is reproduced on the organ model 20)) increases. Note that the imaging plane of the organ model 20 can also be said to be a display surface on which visual information (blood vessels and / or tumors, etc.) is superimposed and structures are explicitly displayed along the virtual cross section in which the same plane as the cross section represented by the ultrasound image 22 is reproduced on the organ model 20.

[0126] 14 is a diagram showing an example of the display of the organ model 20 in which the transparency of blood vessels V, which are an example of a structure of the organ model 20, has been adjusted. As shown in FIG. 14 as an example, the closer the blood vessels V are to the imaging plane of the organ model 20 (i.e., the ultrasound image 22 or the plate surface 22A), the lower the transparency is displayed, and the further the blood vessels are from the imaging plane of the organ model 20, the higher the transparency is displayed, thereby emphasizing the perspective of the blood vessels V displayed in the organ model image 23. That is, of the entire region of the organ model 20, the region within the cross section 20A where the organ model 20 is virtually cut by the imaging plane has the lowest transparency, and of the entire region of the organ model 20, the further the region from the imaging plane, the higher the transparency is displayed, making it easier for the medical staff ST to grasp the perspective of the blood vessels V.

[0127] The processor 41 may set multiple sections in relation to the distance in the normal direction relative to the imaging surface of the organ model 20, and display the organ model 20 so that the transparency of structures included in the same section is the same. For example, the transparency of the blood vessel V included in section A, which is closest to the imaging surface of the organ model 20, may be set to 0%, i.e., opaque, and the transparency of the blood vessel V included in section B, which is next closest to the imaging surface of the organ model 20 after section A, may be set to 50%.

[0128] 15 , for example, when the processor 41 displays an organ model image 23 in which a transparent ultrasound image 22 is superimposed on an organ model 20, the processor 41 may highlight the cross section of a blood vessel V constituting the organ model 20 in the imaging plane of the ultrasound image 22 (i.e., the transverse cut of the blood vessel V present in the organ model 20 in the cross section that coincides with the ultrasound image 22) so that it is easier to see than the blood vessel V at other positions. Typically, the blood vessel V extends three-dimensionally within the organ, but the ultrasound image 14 visualizes a cut-out portion of the blood vessel V in a planar form. By highlighting the cross section of a structure constituting the organ model 20 in the imaging plane of the ultrasound image 22, such as the blood vessel V, the structure of the organ model 20 can be more easily seen even when the organ model 20 and the ultrasound image 22 are overlapping. That is, the blood vessel V is cut transversely, making it appear as a cross-section, and the highlighting of the transverse cut of the blood vessel V creates a visual effect useful for diagnosis.

[0129] On the other hand, in the laparoscopic surgery, when scanning of the liver LV with the ultrasound probe 14 is no longer necessary, the medical staff ST issues an instruction to end the display of the organ model image 23 through the reception device 42. Therefore, in step S100 of Fig. 5, the processor 41 determines whether or not an instruction to notify the end of the display of the organ model image 23 has been received.

[0130] If not, the process proceeds to step S10, and steps S10 to S100 are repeatedly executed until it is determined in the determination process of step S100 that an instruction to end display has been received.

[0131] Since a new internal surgical field image 21 is acquired in the processing of step S10, if the ultrasound probe 14 has moved, position and orientation information after the movement of the ultrasound probe 14 is estimated. The processor 41 displays ultrasound image information of the ultrasound probe 14, the position and orientation of which have been changed based on the newly estimated position and orientation information, on the organ model image 23. Therefore, the processor 41 displays ultrasound image information, the display position of which has been changed in accordance with the movement of the ultrasound probe 14, on the organ model image 23.

[0132] On the other hand, if it is determined in the determination process of step S100 that an instruction to end the display has been received, the medical support process shown in FIG. 5 is ended.

[0133] By executing the medical support process, the processor 41 can show the medical staff ST where in the liver LV the ultrasound probe 14, which cannot be seen directly from outside the body, is scanning and in what position.

[0134] Second Embodiment In the medical support device 11 according to the first embodiment, structural information of the liver LV extracted from the ultrasound image 22 is compared with structural information of the organ model 20 within an estimated imaging range set for the organ model 20, and the position and orientation information of the ultrasound probe 14 in the organ model 20 is identified based on the comparison result. In this case, even though the comparison range between the structural information of the liver LV and the structural information of the organ model 20 is limited to the estimated imaging range, the structural information of the liver LV is compared with the structural information of the organ model 20 while being moved and rotated. Specifically, the processor 41 searches for a location that matches the structural information of the organ model 20 while rotating and moving the shape of the structure obtained from the structural information of the liver LV around the X-axis, the Y-axis, and the Z-axis at each position.

[0135] In contrast, in the second embodiment, a medical support device 11A is described in which structural information of the liver LV is translated within the estimated imaging range and compared with structural information of the organ model 20, and the position and orientation information of the ultrasound probe 14 in the organ model 20 is determined based on the comparison result.

[0136] In the second embodiment, in the medical support system 10 shown in FIG. 1, the medical support device 11 is replaced with a medical support device 11A.

[0137] Fig. 16 is a diagram showing an example of the hardware configuration of a medical support device 11A. The medical support device 11A shown in Fig. 16 differs from the hardware configuration of the medical support device 11 according to the first embodiment shown in Fig. 4 in that the processor 41 also operates as an orientation estimation unit 41H. In addition, the position and orientation identification unit 41F in the hardware configuration of the medical support device 11 according to the first embodiment shown in Fig. 4 is replaced with a position and orientation identification unit 41Fα.

[0138] The orientation estimation unit 41H estimates the orientation of the liver LV from the internal surgical field image 21 acquired by the image acquisition unit 41A, and positions the orientation of the organ model 20 in the model coordinate system in the same orientation as the estimated orientation of the liver LV.

[0139] The position and orientation specifying unit 41Fα compares the structural information of the liver LV extracted by the structural information extracting unit 41D with the structural information of the organ model 20 corresponding to the estimated imaging range of the liver LV calculated by the estimated imaging range calculating unit 41E, and specifies the position and orientation of the ultrasound probe 14 relative to the organ model 20 based on the comparison result. When comparing the structural information of the liver LV with the structural information of the organ model 20, the position and orientation specifying unit 41Fα compares the structural information of the liver LV with the structural information of the organ model 20 by translating the structural information of the liver LV within the estimated imaging range of the liver LV.

[0140] The other hardware configurations of the medical support device 11A are the same as those of the medical support device 11 according to the first embodiment, and therefore will not be described.

[0141] Next, the operation of the medical support device 11A will be described in detail.

[0142] FIG. 17 is a flowchart showing an example of the flow of medical support processing executed by the medical support device 11A when the processor 41 receives, via the reception device 42, an instruction from the medical staff ST to start navigation of the ultrasound probe 14 inserted into the body of the patient PT.

[0143] The flowchart shown in Fig. 17 differs from the flowchart of the medical support processing according to the first embodiment shown in Fig. 5 in that steps S55, S72, and S74 are added, and the processing of step S80 is replaced with step S80A. Therefore, the following description will focus on the differences from the medical support processing according to the first embodiment.

[0144] It is assumed that an organ model 20 of the liver LV, which is the target site for surgery, is generated in advance and stored in the storage 44 before the navigation start instruction is received.

[0145] After extracting structural information of the liver LV from the ultrasound image 22 by the processing of step S50 in Figure 17, in step S55, the processor 41 estimates the orientation of the liver LV from the internal surgical field image 21 and adjusts the orientation of the organ model 20 to match the orientation of the liver LV in the internal surgical field image 21.

[0146] Therefore, the processor 41 estimates an orientation matrix 31 that converts the orientation of the liver LV in the internal surgical field image 21 into an orientation in the model coordinate system.

[0147] 18 is a diagram showing an example of a configuration for estimating the posture matrix 31. To estimate the posture matrix 31, a learning model 30 is used, which is machine-learned using learning data in which internal surgical field images 21 collected in advance for learning are used as input data and a correct posture matrix 31 for the internal surgical field images 21 is used as training data. The internal surgical field images 21 actually obtained from the endoscope 13 are input to the learning model 30, and the posture matrix 31 is output from the learning model 30. Instead of the learning model 30, the processor 41 may estimate the posture matrix 31 using multiple models, such as a model that extracts features from the internal surgical field images 21 and a model that estimates the posture matrix 31 from the features.

[0148] In order to improve the accuracy of the orientation of the organ model 20, multiple endoscopic surgical field images 21 obtained at different times may be used to estimate the orientation matrix 31. For example, multiple endoscopic surgical field images 21 that change over time are input to the learning model 30, which then outputs the orientation matrix 31. In this case, the learning model 30 performs machine learning using training data in which multiple endoscopic surgical field images 21 that change over time and that have been collected in advance are used as input data, and the correct orientation matrix for the last endoscopic surgical field image 21 in the time series is used as training data. This improves the accuracy of estimating the orientation of the organ model 20 compared to when estimating the orientation of the organ model 20 using only endoscopic surgical field images 21 at a single time point.

[0149] The processor 41 rotates the orientation of the organ model 20 in accordance with the orientation matrix 31, and adjusts the orientation of the organ model 20 in the organ model image 23 so that it is the same as the orientation of the liver LV in the internal surgical field image 21. Note that, because the estimation of the orientation matrix 31 contains errors, the orientation of the organ model 20 in the adjusted organ model image 23 and the orientation of the liver LV in the internal surgical field image 21 may not necessarily match.

[0150] After the position, orientation, and prescribed range of the ultrasound probe 14 in the internal body coordinate system are converted into the model coordinate system by the process of step S70 in FIG. 17, step S72 is executed.

[0151] In step S72, the processor 41 compares, within the estimated imaging range, the structural information of the liver LV extracted from the ultrasound image 22 by the processing of step S50 with the structural information of the organ model 20, and determines whether the degree of agreement of the structural information is at a maximum based on the comparison result. Note that, for example, the degree of agreement of the structural information indicates a state in which the structural information matches more as the value increases.

[0152] If the degree of match of the structural information is not the highest, the process proceeds to step S74. In this case, it can be considered that the structural information does not match, and the processor 41 searches for a position and orientation of the ultrasonic probe 14 that will result in an ultrasonic image 22 with the highest degree of match of the structural information in the model coordinate system.

[0153] Meanwhile, the processing in step S55 aligns the orientation of the organ model 20 with the orientation of the liver LV in the internal surgical field image 21. Therefore, by translating the structural information of the liver LV relative to the organ model 20, the processor 41 can search for the position and orientation of the ultrasound probe 14 that will result in an ultrasound image 22 with the highest degree of agreement of the structural information in the model coordinate system.

[0154] For the above reasons, in step S74, the processor 41 translates the structural information of the liver LV within the estimated imaging range, and then proceeds to step S72.

[0155] Therefore, the processor 41 continues to compare the structural information of the liver LV with the structural information of the organ model 20 at each position to which the structural information of the liver LV is translated until the degree of agreement of the structural information is maximized.

[0156] If it is determined in step S72 that the degree of match of the structural information is the highest, the process proceeds to step S80A.

[0157] In the determination process of step S72, the processor 41 may determine whether the degree of match of the structural information is equal to or greater than a threshold ε, and proceed to step S80A if the degree of match is equal to or greater than the threshold ε, or proceed to step S74 if the degree of match is less than the threshold ε. The threshold ε is an example of a parameter 50 that, if equal to or greater than this value, can be considered to indicate that the structural information of the liver LV and the structural information of the organ model 20 match. The threshold ε can be changed by the medical staff ST.

[0158] Furthermore, in step S74, the processor may translate the structural information of the liver LV and rotate the structural information of the liver LV within a predetermined range, and compare the structural information of the liver LV with the structural information of the organ model 20. In a situation where there is an error between the orientation of the liver LV in the internal surgical field image 21 and the orientation of the organ model 20, the accuracy of estimating the position and orientation information of the ultrasound probe 14 relative to the organ model 20 can be improved compared to when the structural information of the liver LV is only translated. Furthermore, because the range in which the structural information of the liver LV is rotated is limited, the time required to estimate the position and orientation information of the ultrasound probe 14 relative to the organ model 20 can be reduced compared to when the rotation range is not limited.

[0159] In step S80A, the processor 41 identifies the position and orientation of the ultrasound probe 14 in the organ model 20 as the position and orientation information of the ultrasound probe 14 in the model coordinate system that will result in an ultrasound image 22 with the highest degree of conformance to the structural information.

[0160] In step S90, the processor 41 displays on the display 16 an organ model image 23 in which the position and orientation information of the ultrasound probe 14 in the identified organ model 20 is superimposed on the organ model 20 oriented to match the orientation of the liver LV in the internal surgical field image 21.

[0161] The processor 41 continues to acquire new internal surgical field images 21 through the processing of step S10 until it is determined in the determination processing of step S100 that an instruction to end display has been received. Therefore, if the orientation of the liver LV in the internal surgical field image 21 changes, the processor 41 displays, in the organ model image 23, the organ model 20 whose orientation has been changed in conjunction with the orientation of the liver LV so that it has the same orientation as the changed orientation of the liver LV. The operation of displaying, on the display 16, the organ model image 23 whose orientation has been changed in conjunction with the change in orientation of the liver LV is referred to as "linked display of the organ model 20."

[0162] In some cases, a situation may arise in which it is not desired to change the orientation of the organ model 20 in the organ model image 23 even if the orientation of the liver LV in the internal surgical field image 21 changes. Therefore, the processor 41 may execute the linked display of the organ model 20 only when the medical staff ST permits the linked display of the organ model 20. If the linked display of the organ model 20 is not permitted, the processor 41 prevents the orientation of the organ model 20 in the organ model image 23 from being changed even if the orientation of the liver LV in the internal surgical field image 21 changes.

[0163] In this way, according to the medical support device 11A of the second embodiment, the orientation of the organ model 20 in the model coordinate system is aligned with the orientation of the liver LV in the internal surgical field image 21, and then the position and orientation information of the ultrasound probe 14 relative to the organ model 20 is estimated.

[0164] In addition, in the medical support devices 11 and 11A, if the organ model 20 is not stored in the storage 44 before receiving a navigation start instruction, the processor 41 may generate the organ model 20 from multiple ultrasound images 22 of the liver LV and start the medical support processing.

[0165] For example, the processor 41 may display a notification on the display 16 or provide an audio notification instructing the medical staff ST to move the ultrasound probe 14 back and forth and left and right. This results in multiple ultrasound images 22 generated by capturing images of the liver LV from various positions and postures. Meanwhile, position and posture information for each ultrasound probe 14 in the internal body coordinate system is obtained from the appearance of the marker M1 in the internal surgical field image 21. Therefore, the processor 41 generates the organ model 20 by constructing a three-dimensional image of the liver LV from the multiple acquired ultrasound images 22 using the position and posture information of the ultrasound probe 14. For example, AI (artificial intelligence) may be used as a method for constructing a three-dimensional image from two-dimensional images such as the ultrasound images 22.

[0166] If an organ model 20 is generated from a plurality of ultrasound images 22, it is possible to omit the process of generating an organ model 20 of the target area for surgery in advance using CT, MRI, etc.

[0167] When both an organ model 20 generated from CT, MRI, etc. and an organ model 20 generated from multiple ultrasound images 22 exist, the processor 41 may select the organ model 20 to be used for medical support processing in accordance with the instructions of the medical staff ST.

[0168] Furthermore, in the medical support devices 11 and 11A, when the processor 41 compares the structural information of the liver LV with the structural information of the organ model 20, it may compare a plurality of pieces of structural information of the liver LV extracted from a plurality of ultrasound images 22 with the structural information of the organ model 20. In this case, the processor 41 may, for example, calculate the degree of agreement between each piece of structural information of the liver LV and the structural information of the organ model 20 within the estimated imaging range, and use the sum of the degrees of agreement as the final degree of agreement. By comparing a plurality of pieces of structural information of the liver LV with the structural information of the organ model 20, the processor 41 can improve the estimation accuracy of the position and orientation information of the ultrasound probe 14 relative to the organ model 20 compared to when comparing with structural information of the liver LV extracted from a single ultrasound image 22.

[0169] Furthermore, in the medical support devices 11 and 11A, the processor 41 may display the internal surgical field image 21, the ultrasound image 22, and the organ model image 23 side by side on the same display 16. By displaying the internal surgical field image 21, the ultrasound image 22, and the organ model image 23 side by side on the same display 16, the processor 41 allows the medical staff ST to operate the ultrasound probe 14 without moving their line of sight, compared to when each image is displayed on a different display 16.

[0170] Furthermore, in the present disclosure, an example has been described in which the medical staff ST operates the ultrasound probe 14, but for example, the ultrasound probe 14 may be operated by a medical robot following instructions from the medical staff ST.

[0171] While one form of the medical support system 10 has been described above using the embodiment, the disclosed form of the medical support system 10 is merely an example, and the form of the medical support system 10 is not limited to the scope described in the embodiment. Various changes or improvements can be made to the embodiment without departing from the gist of the present disclosure, and forms incorporating such changes or improvements are also included in the technical scope of the disclosure.

[0172] For example, the internal processing order in the flowcharts of the medical support processing shown in FIGS. 5 and 17 may be changed without departing from the gist of the present disclosure.

[0173] In addition, in each of the above embodiments, an example has been given in which the processor 41 (e.g., the position and orientation information estimation unit 41C shown in FIG. 4) derives position and orientation information based on the internal surgical field image 21, but this is merely one example. For example, the position and orientation information may be derived based on the internal surgical field image 21 by a processor different from the processor 41. An example of a processor different from the processor 41 is a processor installed in a device (e.g., a personal computer or a server) different from the medical support devices 11 and 11A.

[0174] Furthermore, in each of the above embodiments, an example has been given in which the processor 41 (for example, the position and orientation identifying unit 41F shown in FIG. 4 or the position and orientation identifying unit 41Fα shown in FIG. 16) derives the position and orientation of the ultrasound probe 14 relative to the organ model 20 based on structural information of the liver LV and structural information of the organ model 20 corresponding to the estimated imaging range of the liver LV, but this is merely one example. For example, the position and orientation of the ultrasound probe 14 relative to the organ model 20 may be derived by a processor different from the processor 41 based on structural information of the liver LV and structural information of the organ model 20 corresponding to the estimated imaging range of the liver LV.

[0175] Furthermore, in each of the above embodiments, an example was given in which structural information of the liver LV is extracted from the ultrasound image 22 by the processor 41 (for example, the structural information extraction unit 41D shown in FIG. 4), but this is merely one example, and structural information of the liver LV may be extracted from the ultrasound image 22 by, for example, a processor different from the processor 41.

[0176] Furthermore, in each of the above embodiments, an example has been given in which the processor 41 (for example, the estimated imaging range calculation unit 41E shown in FIG. 4 ) derives the estimated imaging range of the liver LV by the ultrasound probe 14 based on the position of the ultrasound probe 14, the orientation of the ultrasound probe 14, and range information preset for the position and orientation of the ultrasound probe 14, but this is merely one example. For example, the estimated imaging range of the liver LV by the ultrasound probe 14 may be derived by a processor different from the processor 41 based on the position of the ultrasound probe 14, the orientation of the ultrasound probe 14, and range information preset for the position and orientation of the ultrasound probe 14.

[0177] In addition, in each of the above embodiments, various images (e.g., the internal surgical field image 21, the ultrasound image 22, and the organ model image 23) are displayed on the display 16 by the processor 41 (e.g., the display unit 41G shown in FIG. 4), but this is merely one example. For example, various images may be transmitted to a device (e.g., a personal computer or a server) different from the medical support device 11 or 11A, and the various images may be displayed on the display 16 and / or a display different from the display 16 by a processor of the device different from the medical support device 11 or 11A.

[0178] 5 and 17 , an example has been given in which the processor 41 displays on the display 16 an organ model image 23 in which the position and orientation information of the ultrasound probe 14 in the organ model 20 identified by the processing of step S80 is superimposed on the organ model 20, but the position and orientation information estimated in step S40 and the organ model 20 may also be displayed on the display 16. In this case, it is preferable that the organ model 20 is displayed on the display 16, and that the position and orientation information is displayed on the display 16 in a state where it is superimposed on the organ model 20.

[0179] Here, the position and orientation information displayed on the display 16 together with the organ model 20 may be derived based on the endoscopic surgical field image 21 acquired by the image acquisition unit 41A. In the above-described embodiments, an example was given in which the position and orientation information was derived based on the display status of the marker M1 included in the endoscopic surgical field image 21, but this is merely an example. For example, the endoscopic surgical field image 21 may be input into a trained model (e.g., a trained neural network) generated by performing machine learning on the model (e.g., a neural network) using training data, which is a data set of a sample image corresponding to a surgical field image in which the marker M1 appears and information regarding the position and orientation of an ultrasound probe having the marker M1 appearing in the sample image, and correct answer data. The position and orientation information derived by the trained model may then be displayed on the display 16 superimposed on the organ model 20.

[0180] In the above embodiments, the medical support process is implemented by software processing. However, the same process as shown in the flowchart of the medical support process may be implemented by hardware. In this case, the processing speed can be increased compared to when the medical support process is implemented by software processing.

[0181] In each of the above embodiments, the processor 41 refers to a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU) and / or a dedicated processor (e.g., a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, and a programmable logic device, etc.).

[0182] Furthermore, the operations of the processor 41 in each of the above embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor 41 is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0183] In the above embodiments, an example has been described in which the medical support program 49 is stored in the storage 44. However, the storage destination of the medical support program 49 is not limited to the storage 44. The medical support program 49 can also be provided in a form recorded on a computer-readable storage medium.

[0184] For example, the medical support program 49 can be stored on a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a Blu-ray Disc.

[0185] The medical support program 49 may be provided in a form recorded on an optical disc such as a DVD. The medical support program 49 may also be provided in a form recorded on a portable semiconductor memory such as a USB memory or a memory card. The storage 44, a CD-ROM, a DVD-ROM, a Blu-ray disc, a USB memory, and a memory card are examples of non-transitory storage media.

[0186] Furthermore, the medical support device 11 may download a medical support program 49 from an external device connected to the network via the communication I / F 45, and store the downloaded medical support program 49 in the storage 44. The present disclosure may also be applied to programs and program products.

[0187] The following additional notes are further disclosed regarding the above embodiment.

[0188] (Supplementary Note 1) A medical support device comprising a processor, which: acquires an operative field image including an organ photographed by an intracorporeal camera and an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ; estimates the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the operative field image; extracts structural information of the organ from the ultrasound image of the organ; calculates an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the operative field image and predetermined range information; compares the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ, identifies the position and orientation of the ultrasound probe in the organ model that corresponds to the ultrasound image of the organ; and displays information relating to the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

[0189] (Supplementary Note 2) The medical support device according to Supplementary Note 1, wherein the processor estimates an orientation of the organ from the surgical field image, and calculates an estimated imaging range of the organ using the estimated orientation of the organ.

[0190] (Supplementary Note 3) The medical support device described in Supplementary Note 2, wherein the processor translates structural information of the organ within an estimated imaging range of the organ in the organ model, and identifies, as the position and orientation of the ultrasound probe in the organ model, a position and orientation of the ultrasound probe that results in an ultrasound image of the organ whose structural information is closest to the structural information of the organ model.

[0191] (Supplementary Note 4) The medical support device according to Supplementary Note 3, wherein the processor positions the organ model in the same orientation as the orientation of the organ estimated from the surgical field image, and then identifies the position and orientation of the ultrasound probe in the organ model.

[0192] (Supplementary Note 5) The medical support device described in any one of Supplementary Notes 1 to 4, wherein the processor identifies the position and orientation of the ultrasound probe in the organ model corresponding to the ultrasound images of the organ using structural information of the organ obtained from multiple ultrasound images of the organ.

[0193] (Supplementary Note 6) The medical support device according to any one of Supplementary Notes 1 to 5, wherein the processor generates the organ model from a plurality of ultrasound images of the organ.

[0194] (Supplementary Note 7) The medical support device according to any one of Supplementary Notes 1 to 6, wherein the structural information of the organ includes information on the shape of the organ, the positions and shapes of blood vessels and tumors contained in the organ, and the external appearance of the organ.

[0195] (Supplementary Note 8) The medical support device according to any one of Supplementary Notes 1 to 7, wherein the processor displays a probe image representing the ultrasound probe according to the identified position and orientation of the ultrasound probe by superimposing it on the organ model.

[0196] (Supplementary Note 9) The medical support device described in Supplementary Note 8, wherein the processor displays at least one of an ultrasound image of the organ captured by the ultrasound probe represented by the probe image, a plate surface representing the imaging range of the ultrasound probe, and a contour line representing the outline of the imaging range of the ultrasound probe, superimposed on the organ model.

[0197] (Supplementary Note 10) The medical support device according to Supplementary Note 9, wherein the processor changes the position and orientation of the probe image, and the display position of at least one of the ultrasound image of the organ, the plate surface, and the contour line in accordance with the movement of the ultrasound probe, and displays them on the organ model.

[0198] (Supplementary Note 11) The medical support device according to Supplementary Note 9, wherein when the plate surface is displayed, the processor adjusts a degree of transparency of the plate surface to be displayed superimposed on the organ model in accordance with an instruction from a user.

[0199] (Supplementary Note 12) The medical support device described in any one of Supplementary Notes 8 to 11, wherein the processor displays the organ model with the degree of transparency of structures constituting the organ model that exist at each distance adjusted depending on the distance from the cross-section of the organ model represented by the ultrasound image of the organ.

[0200] (Supplementary Note 13) The medical support device according to Supplementary Note 12, wherein the processor displays the organ model with increasing transparency of structures constituting the organ model that are present at each distance from the cross section of the organ model as the distance from the cross section increases.

[0201] (Supplementary Note 14) The medical support device described in any one of Supplementary Notes 2 to 4, wherein when the orientation of the organ in the surgical field image changes, the processor displays the organ model so that the orientation of the organ model is the same as the orientation of the organ in the surgical field image.

[0202] (Supplementary Note 15) The medical support device according to Supplementary Note 14, wherein, when a user allows a change in orientation of the organ model, the processor displays the organ model so that the orientation of the organ model is the same as the orientation of the organ in the surgical field image.

[0203] (Supplementary Note 16) A medical support program for causing a computer to execute a process of: acquiring an operative field image including an organ photographed by an internal camera and an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ; estimating the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the operative field image; extracting structural information of the organ from the ultrasound image of the organ; calculating an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the operative field image and predetermined range information; comparing the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ; identifying the position and orientation of the ultrasound probe in the organ model that corresponds to the ultrasound image of the organ; and displaying information regarding the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

[0204] (Supplementary Note 17) A non-transitory storage medium storing a program executable by a computer to execute medical support processing, the medical support processing including the steps of: acquiring a surgical field image including an organ photographed by an internal camera and an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ; estimating the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the surgical field image; extracting structural information of the organ from the ultrasound image of the organ; calculating an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the surgical field image and predetermined range information; comparing the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ, and identifying the position and orientation of the ultrasound probe in the organ model that corresponds to the ultrasound image of the organ; and displaying information regarding the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

[0205] (Supplementary Note 18) A computer program product including a program for executing a process of acquiring an organ photographed by an intracorporeal camera, a surgical field image including an ultrasound probe that photographs two-dimensional ultrasound images, and an organ model representing the three-dimensional shape of the organ, estimating the position and orientation of the ultrasound probe that photographs the ultrasound image of the organ from the surgical field image, extracting structural information of the organ from the ultrasound image of the organ, calculating an estimated imaging range of the organ by the ultrasound probe using the position and orientation of the ultrasound probe estimated from the surgical field image and predetermined range information, comparing the structural information of the organ with structural information of the organ model corresponding to the estimated imaging range of the organ, identifying the position and orientation of the ultrasound probe in the organ model that corresponds to the ultrasound image of the organ, and displaying information relating to the position and orientation of the ultrasound probe in the identified organ model, as well as the organ model.

Claims

1. A medical support device comprising a processor, which acquires an operative field image including an organ and an ultrasound probe that transmits ultrasound to the organ and receives reflected waves reflected by the organ, and an organ model representing the three-dimensional shape of the organ, and displays position and orientation information and the organ model on a display, wherein the position and orientation information is information relating to a first position and a first orientation which are the position and orientation of the ultrasound probe with respect to the organ model corresponding to an ultrasound image generated based on the reflected waves, and the first position and the first orientation are derived based on organ structure information which is structural information of the organ and structural information of the organ model corresponding to an estimated imaging range of the organ by the ultrasound probe, and the organ structure information is extracted from the ultrasound image generated based on the reflected waves, and the estimated imaging range is derived based on a second position which is the position of the ultrasound probe, a second orientation which is the orientation of the ultrasound probe, and predetermined range information, and the second position and the second orientation are derived based on the operative field image.

2. The medical support device according to claim 1, wherein the orientation of the organ is derived based on the surgical field image, and the estimated imaging range of the organ is derived based on the orientation of the organ.

3. The medical support device according to claim 2, wherein the position and orientation of the ultrasound probe that obtains an ultrasound image of the organ whose structural information is closest to the structural information of the organ model by translating the structural information of the organ within the estimated imaging range of the organ in the organ model is derived as the position and orientation of the ultrasound probe in the organ model.

4. The medical support device according to claim 3, wherein the organ model and the organ derived based on the surgical field image are aligned in the same orientation, and then the position and orientation of the ultrasound probe in the organ model are derived.

5. The medical support device according to claim 1, wherein the position and orientation of the ultrasound probe in the organ model corresponding to the ultrasound image of the organ are derived based on structural information of the organ obtained from multiple ultrasound images representing the organ.

6. The medical support device according to claim 1, wherein the organ model is generated based on a plurality of ultrasound images representing the organ.

7. The medical support device according to claim 1, wherein the structural information of the organ includes information on the shape of the organ, blood vessels contained in the organ, the position of a tumor contained in the organ, the shape of the tumor, and / or the external appearance of the organ.

8. The medical support device according to claim 1, wherein a probe image representing the ultrasound probe in the first position and the first posture is displayed on the display in a state where it is superimposed on the organ model.

9. The medical support device according to claim 8, wherein at least one of the ultrasound image obtained by imaging with the ultrasound probe represented by the probe image, the plate surface representing the imaging range of the ultrasound probe, and the contour line representing the outline of the imaging range of the ultrasound probe are displayed on the display in a state superimposed on the organ model.

10. The medical support device according to claim 9, wherein the position and orientation of the probe image, and the display position of at least one of the ultrasound image of the organ, the plate surface, and the contour line change in accordance with the movement of the ultrasound probe.

11. A medical support device according to claim 9, wherein when the plate surface is displayed on the display, the transparency of the plate surface displayed superimposed on the organ model is adjusted according to a given instruction.

12. A medical support device as described in claim 8, wherein when the organ model is displayed on the display, the transparency of structures constituting the organ model that exist at each distance from the cross section of the organ model represented by the ultrasound image of the organ is adjusted according to the distance.

13. A medical support device according to claim 12, wherein when the organ model is displayed on the display, the transparency of structures constituting the organ model at each distance increases as the distance from the cross section of the organ model increases.

14. A medical support device as described in claim 2, wherein when the orientation of the organ in the surgical field image displayed on the display changes, the orientation of the organ model displayed on the display becomes the same as the orientation of the organ in the surgical field image.

15. The medical support device according to claim 14, wherein, when an instruction to permit a change in the orientation of the organ model is given, the orientation of the organ model displayed on the display becomes the same as the orientation of the organ in the surgical field image.

16. A program for causing a computer to execute processing, the processing including: acquiring an operative field image including an organ and an ultrasound probe that receives reflected waves from the organ by transmitting ultrasound to the organ, and an organ model representing the three-dimensional shape of the organ; and displaying position and orientation information and the organ model on a display, wherein the position and orientation information is information relating to a first position and a first orientation which are the position and orientation of the ultrasound probe with respect to the organ model corresponding to the ultrasound image generated based on the reflected waves, and the first position and the first orientation are derived based on organ structure information which is structural information of the organ and structural information of the organ model corresponding to the estimated imaging range of the organ by the ultrasound probe, the organ structure information is extracted from the ultrasound image generated based on the reflected waves, and the estimated imaging range is derived based on a second position which is the position of the ultrasound probe, a second orientation which is the orientation of the ultrasound probe, and predetermined range information, and the second position and the second orientation are derived based on the operative field image.

Images