Medical assistance device, medical assistance method, and medical assistance program

Abstract

Provided is a medical assistance device comprising a processor. The processor acquires a first image obtained by using a camera to optically image a surgical field including a target site in a body and a medical instrument that is inserted into the body and that images an internal structure of the target site in a contact state in which the medical instrument is in contact with the target site. The processor derives, on the basis of the first image, position and orientation information which indicates the position and orientation of the medical instrument in the surgical field, and acquires a second image imaged by the medical instrument. The processor controls, on the basis of the position and orientation information, the display of a superimposed image in which the second image is superimposed on the first image, and stops the display of the superimposed image if a separation state, which is a state in which the medical instrument is separated from the target site, continues for a predetermined period.

Description

Medical Support Device, Medical Support Method, and Medical Support Program

[0001] The present disclosure relates to a medical support device, a medical support method, and a medical support program.

[0002] Conventionally, techniques for identifying the position and orientation of a medical instrument inserted into the body and supporting medical treatments such as surgeries and examinations based on the identified position and orientation of the medical instrument are known. For example, Japanese Patent Application Publication No. 2019-517291 discloses controlling the fusion between an endoscopic image and an ultrasonic image based on image conversion between the ultrasonic image space of a laparoscopic ultrasonic probe and the endoscopic image space of an endoscope.

[0003] Also, for example, Japanese Patent Application Laid-Open No. 58-218951 discloses discriminating between multiple reflection echoes obtained from a non-contact part of a scanning probe with a living body and biological echoes obtained from a contact part with the living body, and supplying or stopping an ultrasonic video signal to a monitor based on the output.

[0004] In an ultrasonic image, if the ultrasonic probe is not appropriately contacted with the target site, an appropriate image may not be obtained. For example, when the ultrasonic probe is separated from the target site and air exists therebetween, the ultrasonic waves are reflected and attenuated by the air, so that the ultrasonic echoes from the target site cannot be detected. In this case, since information such as the internal structure of the target site does not appear in the ultrasonic image, even if the ultrasonic image is superimposed and displayed on the endoscopic image as described in Japanese Patent Application Publication No. 2019-517291, medical treatment cannot be appropriately supported.

[0005] On the other hand, a method of superimposing and displaying an ultrasonic image only when the ultrasonic probe is appropriately contacted with the target site can be considered. However, for example, when the ultrasonic probe 14 is operated so as to rub the surface of a target site with irregularities, a contact state is formed at the convex part and a separated state is formed at the concave part. In such a case, if the presence or absence of the superimposed display is switched according to the contact state or the separated state, there is a problem that the visibility deteriorates due to flickering when temporarily passing through the concave part, and the convenience is rather reduced.

[0006] This disclosure provides a medical support device, a medical support method, and a medical support program that can further improve convenience.

[0007] A first aspect of this disclosure is a medical support device comprising a processor, the processor optically captures a first image of a surgical field including a target area in the body and a medical instrument inserted into the body and in contact with the target area, which images the internal structure of the target area, using a camera, the processor derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the first image, acquires a second image captured by the medical instrument, controls the display of a superimposed image by superimposing the second image on the first image based on the position and orientation information, and stops displaying the superimposed image if a separation state, in which the medical instrument is separated from the target area, continues for a predetermined period of time.

[0008] If the separation state continues for a predetermined period of time, the processor may display the first image instead of the superimposed image.

[0009] The processor may display the superimposed image during the specified period.

[0010] The processor may display a superimposed image within the period, which is a second image from a past point in time prior to the point of switching from a contact state to a separation state, superimposed on the first image from the current time.

[0011] The processor may gradually increase the transparency of the second image within the superimposed image over time from the point of switching.

[0012] If the processor switches from a separated state to a contact state within the specified period, it may display a superimposed image by superimposing the current second image onto the current first image.

[0013] The processor may, within the specified period, display a superimposed image in which the current second image and a past second image from before the point of switching from a contact state to a separation state are superimposed on the current first image at a predetermined ratio.

[0014] The processor may, over time from the switching point, increase the transparency of the second image at past points in time within the superimposed image.

[0015] The processor may, over time from the switching point, increase the transparency of the second image at the current time and the second image at past time points within the superimposed image.

[0016] If the processor switches from a separated state to a contact state within the specified period, it may display a superimposed image by superimposing the current second image onto the current first image.

[0017] The processor may notify that it is in a detached state during the specified period.

[0018] The processor may display at least one of text and / or graphics indicating that it is in a detached state during the specified period.

[0019] The processor may highlight the frame of the second image within the superimposed image during the specified period.

[0020] The processor may, within the specified period, emit at least one of text and / or audio instructing the device to enter a contact state.

[0021] The medical device is an ultrasound probe that transmits ultrasound to a target area and detects an electrical signal corresponding to the ultrasound echo reflected from the target area. The second image is an ultrasound image generated in response to the electrical signal. The processor may determine whether the area is in contact or separated based on at least one of the electrical signal and the ultrasound image.

[0022] A second aspect of this disclosure is a medical support method, wherein a computer performs the following processes: first, optically captures a surgical field with a camera, which includes a target area inside the body and a medical device inserted into the body and in contact with the target area, and which images the internal structure of the target area; second, optically captures a surgical field with a camera; first, optically captures a surgical field with a surgical field with a surgical device

[0023] A third aspect of this disclosure is a medical support program which optically captures a first image of a surgical field, including a target area inside the body and a medical device inserted into the body and in contact with the target area, and which images the internal structure of the target area, using a camera; derives position and orientation information indicating the position and orientation of the medical device in the surgical field based on the first image; acquires a second image captured by the medical device; controls the display of a superimposed image, which is obtained by superimposing the second image on the first image, based on the position and orientation information; and causes a computer to execute a process to stop displaying the superimposed image if the separated state, in which the medical device is separated from the target area, continues for a predetermined period of time.

[0024] According to the above embodiments, the medical support device, medical support method, and medical support program of this disclosure can be made more convenient.

[0025] This figure shows an overview of a medical support system including medical support devices. This figure shows the inside of the body during laparoscopic surgery. This figure shows the insertion state of a puncture needle guided by a guide groove. This figure shows an example of the hardware configuration of a medical support device. This figure shows an example of the functional configuration of a medical support device. This figure shows the relationship between the position and orientation of the marker and the ultrasound probe. This figure shows an example of a superimposed image. This figure shows the relationship between the position and orientation of the marker and the ultrasound probe. This figure shows an example of a superimposed image. This figure shows an example of display control of a superimposed image in a separated state. This figure shows an example of display control of a superimposed image in a separated state. This figure shows an example of display control of a superimposed image in a separated state. This figure shows an example of display control of a superimposed image in a separated state. This figure shows an example of display control of a superimposed image in a separated state. This is a flowchart of an example of medical support processing. This figure shows an example of a scale. This figure shows an example of a puncture route.

[0026] The following describes an example of an embodiment of the disclosed technology with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals, and redundant descriptions are omitted. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0027] Referring to Figures 1 to 3, an example of a medical support system 10 to which the medical support device 11 according to this embodiment is applied will be described. As an example, the medical support system 10 is used when performing endoscopic surgery using an endoscope 13 on a patient PT. Unlike open surgery, endoscopic surgery is a type of surgery in which a small hole is made in the patient PT's body and a medical instrument such as an endoscope 13 is inserted through it. The medical support system 10 provides medical staff, including physicians, with a field of view of the surgical field inside the patient PT's body, as well as support information to assist in surgery and examinations. The support information includes, as will be described later, superimposed images 26 for displaying an ultrasound image 22 superimposed on the surgical field image 21. Since such a medical support system 10 has the function of providing support information in real time during surgery, it is also called a surgical navigation system.

[0028] As shown in Figure 1, the medical support system 10 comprises a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is communicated with the endoscope 13, the ultrasound probe 14, and the display 16.

[0029] Figure 2 shows the insertion of an endoscope 13 and an ultrasound probe 14 into the abdomen of a patient (PT). In endoscopic surgery, a portion of the endoscope 13 and ultrasound probe 14, including their respective tips, is inserted into the body via a trocar 17. The trocar 17 is an insertion device equipped with an insertion hole for inserting the endoscope 13, etc., and a valve provided within the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is performed by injecting carbon dioxide into the abdominal cavity, so the trocar 17 is used to insert the endoscope 13 and ultrasound probe 14, etc., into the body.

[0030] The puncture needle 18 is a treatment instrument used to puncture lesions such as tumors contained in organs. Specifically, the puncture needle 18 has a needle portion 18A and a gripping portion 18B provided on the proximal end side of the needle portion 18A. The puncture needle 18 is, for example, a cauterizing puncture needle used to cauterize lesions. The cauterizing puncture needle has an electrode at its tip to which a high-frequency voltage is applied. When the electrode is inserted into a lesion and a high-frequency voltage is applied, the heat generated by the electrode causes the lesion to necrose. In this embodiment, as an example, a treatment method is described in which a tumor 27 in the liver LV is visualized using an ultrasound image 22, and the visualized tumor 27 is cauterized with the puncture needle 18 to cause necrosis of the tumor 27.

[0031] The endoscope 13 optically images the surgical field SF, which includes the target area inside the patient's PT (in this example, the liver LV), using the camera 13B. The surgical field SF is the space within the body cavity defined by organs and the body wall. Specifically, the endoscope 13 has an insertion section 13A that is inserted into the patient's PT. The tip of the insertion section 13A contains a camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)). As an example, the endoscope 13 is a rigid endoscope with a rigid insertion section 13A, and is often used for abdominal cavity observation, hence it is also called a laparoscope.

[0032] Camera 13B includes an image sensor such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of the subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images. The endoscope 13 is connected to an image processing processor for the endoscope (not shown). This image processing processor performs signal processing on the imaging signal output by the image sensor to generate a surgical field image 21 of the surgical field SF inside the body. The surgical field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via the image processing processor for the endoscope. In Figure 2, the symbols Xin and Yin indicate the coordinate system of the surgical field image 21. Camera 13B is an example of a "camera" in this disclosure.

[0033] For illumination of the endoscope 13, visible light such as white light is used, for example. However, special light such as ultraviolet light and infrared light may also be used as illumination of the endoscope 13. Special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the bandwidth of light in the short-wavelength region such as the ultraviolet region. The surgical field image 21 is an image of the surgical field SF illuminated by the illumination light, and more specifically, it is an image based on reflected light from the illumination light reflected near the surface of the surgical field SF. Therefore, in the surgical field image 21, structures present near the surface of the target area can be depicted, but internal structures are difficult to observe.

[0034] The ultrasound probe 14 transmits ultrasound to a target site and detects an electrical signal corresponding to the ultrasound echo reflected from the target site. Specifically, the ultrasound probe 14 has an insertion part 14A that is inserted into the patient's body and an operating part 14D on the proximal end of the insertion part 14A. An ultrasound transducer 14C is built into the tip 14B of the insertion part 14A. The ultrasound probe 14 is an example of a "medical device" as disclosed herein.

[0035] The ultrasonic transducer 14C transmits ultrasound to the target area and receives the ultrasound echoes reflected from the target area. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown). This image processing processor performs image reconstruction processing based on the electrical signals corresponding to the ultrasound echoes received by the ultrasonic transducer 14C so that an ultrasonic image corresponding to the electrical signals is generated. The image reconstruction processing generates an ultrasonic image 22 showing the internal structure of the target area scanned by the ultrasonic probe 14. The ultrasonic image 22 captured by the ultrasonic probe 14 is transmitted in real time to the medical support device 11 via the image processing processor for the ultrasonic probe. In Figure 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0036] The ultrasound probe 14 is, for example, a convex type that transmits ultrasound radially, and a fan-shaped ultrasound image 22 is acquired with the ultrasound transducer 14C as the base point. By scanning the ultrasound probe 14, multiple ultrasound images 22 are taken along the scanning direction. The ultrasound image 22 is a so-called B (Brightness) mode image that visualizes the internal structure from the surface to the deep layers reached by the ultrasound in the target area as brightness information. The ultrasound image 22 visualizes the internal structure of the target area that cannot be observed with the surgical field image 21 obtained by optical imaging.

[0037] A guide groove 29 is provided at the tip 14B of the insertion portion 14A. The guide groove 29 is a groove that, by engaging with the puncture needle 18, guides the insertion of the puncture needle 18 to a target position inside the organ. Figure 3 schematically shows how the puncture needle 18, inserted into the body from the insertion position NP on the body surface BS of the patient PT, punctures a tumor 27 in the liver LV while being guided by the guide groove 29. The guide groove 29 is provided, for example, at the tip 14B on the proximal side of the ultrasonic transducer 14C and is inclined at an angle θ with respect to the direction of the axis AX of the tip 14B. The first guide groove 29 is tilted backward so that the needle tip of the puncture needle 18 inserted from the proximal side of the tip 14B faces towards the tip side of the tip 14B.

[0038] The insertion of the puncture needle 18 is performed while confirming the tumor 27 using an ultrasound image 22. Since the area visualized by the ultrasound image 22 is a radial area originating from the ultrasound transducer 14C, the inclination of the guide groove 29 makes it possible to direct the tip of the puncture needle 18 towards the tumor 27 visualized by the ultrasound image 22. The insertion section 14A may be provided with multiple guide grooves 29 that have different angles relative to the direction of the axis AX. In this case, the medical staff ST can select an appropriate guide groove 29 to insert the puncture needle 18.

[0039] Furthermore, a marker M is attached to the tip 14B of the insertion section 14A. The marker M is a marker that can be recognized from the surgical field image 21 optically captured by the camera 13B of the endoscope 13, that is, an optically detectable marker. The marker M is used by the medical support device 11 to estimate the position and orientation of the ultrasound probe 14, more specifically, the position and orientation of the tip 14B of the insertion section 14A in the surgical field SF. The method for estimating the position and orientation using the marker M will be described later.

[0040] As an example, marker M is a marker of a grid pattern 62, which is composed of a first line extending in the direction of the axis AX of the tip portion 14B of the ultrasonic probe 14, and a second line perpendicular to the axial direction of the tip portion 14B and formed circumferentially along the outer surface of the tip portion 14B. Furthermore, the intersections in the grid pattern 62 are each assigned either a circular reference numeral 64 or a rectangular reference numeral 66.

[0041] Note that the marker M only needs to include reference numerals 64 and 66 placed at each intersection of the grid pattern 62, and does not need to include line segments indicating the first and second lines as shown in Figure 3, etc. Also, reference numerals 64 and 66 in the marker M are not limited to circles and rectangles, but can be any shape, and may be represented by figures such as triangles, polygons, stars, and various marks, or by letters, etc. Furthermore, in some drawings, the illustration of the marker M may be simplified or omitted to avoid making the drawing complicated.

[0042] The medical support device 11 acquires a surgical field image 21 from an endoscope 13 and also acquires an ultrasonic image 22 from an ultrasonic probe 14. Further, the medical support device 11 generates a superimposed image 26 in which the ultrasonic image 22 is superimposed on the surgical field image 21, and controls to display it on a display 16. The superimposed image 26 displayed on the display 16 provides the medical staff ST with the visual field of the surgical field SF inside the patient PT's body and the internal structure of the target site (liver LV).

[0043] Next, the configuration of the medical support device 11 will be described. FIG. 4 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a display 16, a processor 41, a RAM (Random Access Memory) 42, a storage 43, a reception device 46, a communication I / F (InterFace) 47, and an external I / F 48. These respective units are connected to a bus 49 such as a system bus and a control bus, and can communicate with each other.

[0044] In addition to the surgical field image 21, the ultrasonic image 22, and the superimposed image 26, various kinds of information are displayed on the display 16. Examples of the display 16 include a liquid crystal display and an EL (Electro-Luminescence) display. Note that the number of displays 16 may be at least one as shown in FIG. 1, and a plurality of displays may be provided.

[0045] The processor 41 is, for example, a CPU (Central Processing Unit), and comprehensively controls each unit of the medical support device 11 according to a control program and executes various processes according to various application programs.

[0046] The RAM 42 is a memory in which information is temporarily stored, and is used as a work memory by the processor 41. Examples of the RAM 42 include a DRAM (Dynamic Random Access Memory) and a SRAM (Static Random Access Memory).

[0047] Storage 43 is a non-volatile storage device that stores various programs, various parameters, and the like. Examples of the storage 43 include an HDD (Hard Disk Drive) and an SSD (Solid State Drive). A medical support program 44 for causing the computer to function as the medical support device 11 is stored in the storage 43.

[0048] Further, dimension information 45 is stored in the storage 43. The dimension information 45 includes information representing the dimensions of the ultrasonic probe 14, specifically, the relative positional relationships of the marker M, the ultrasonic transducer 14C, and the guide groove 29 in the ultrasonic probe 14. The positional relationship of the marker M is, for example, information on how the codes 64 and 66 constituting the marker M are provided in terms of position and orientation with respect to the axial direction and the circumferential direction of the distal end portion 14B. The positional relationship of the ultrasonic transducer 14C is represented by the linear distance between the reference point of the distal end portion 14B and the ultrasonic transducer 14C, and the tilt angle of the ultrasonic transducer 14C with respect to the direction of the axis AX of the distal end portion 14B. The positional relationship of the guide groove 29 is represented by the linear distance between the reference point of the distal end portion 14B and the guide groove 29, and the tilt angle θ of the guide groove 29 with respect to the direction of the axis AX of the distal end portion 14B.

[0049] The reception device 46 has a keyboard and a mouse (not shown) and receives instructions from the operator. That is, the medical support device 11 is operated by an operator such as medical staff ST through the reception device 46. The reception device 46 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0050] The communication I / F 47 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control according to a communication protocol defined by various wired or wireless communication standards.

[0051] External I / F 48 is, for example, a USB (Universal Serial Bus) interface and is used to connect to peripheral devices such as printers and memory cards. The medical support device 11 can be, for example, a server computer, a personal computer, a smartphone, a tablet terminal, or a wearable device, as appropriate.

[0052] Next, the functional configuration of the medical support device 11 will be described. Figure 5 is a block diagram showing an example of the functional configuration of the medical support device 11. The medical support device 11 includes an acquisition unit 50, an output unit 52, a determination unit 54, and a display control unit 56. The processor 41 reads the medical support program 44 from the storage 43 and executes the medical support program 44 on the RAM 42, thereby allowing the processor 41 to function as the acquisition unit 50, the output unit 52, the determination unit 54, and the display control unit 56. Medical support processing is realized by the processor 41 operating as the acquisition unit 50, the output unit 52, the determination unit 54, and the display control unit 56.

[0053] The acquisition unit 50 acquires a surgical field image 21, which is optically captured by the camera 13B, of the surgical field SF, which includes the target area inside the body and the ultrasound probe 14 that images the internal structure of the target area in a contact state. For example, the acquisition unit 50 acquires the surgical field image 21 from a device including the processor of the endoscope 13 via an external I / F 48 or a communication I / F 47. As shown in Figure 2, when the ultrasound probe 14 is inserted into the surgical field SF, the surgical field image 21 includes the ultrasound probe 14, more specifically the tip 14B of the insertion part 14A. The surgical field image 21 is an example of the "first image" of this disclosure.

[0054] Furthermore, the acquisition unit 50 acquires the ultrasound image 22 captured by the ultrasound probe 14. For example, the acquisition unit 50 acquires the ultrasound image 22 from the device including the processor of the ultrasound probe 14 via an external I / F 48 or a communication I / F 47. The ultrasound image 22 is an example of the "second image" of this disclosure. Note that the medical support device 11 may have the processor of the endoscope 13 and / or the processor of the ultrasound probe 14.

[0055] The derivation unit 52 derives positional and orientation information indicating the position and orientation of the ultrasound probe 14 in the surgical field SF based on the surgical field image 21. Specifically, the derivation unit 52 detects markers M by searching for morphological features of markers M, such as symbols 64 and 66, within the surgical field image 21. For example, the derivation unit 52 may detect markers M using image processing techniques such as pattern matching.

[0056] Alternatively, instead of rule-based methods such as pattern matching, marker M may be detected using AI (Artificial Intelligence) technology with a machine learning model. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) can be applied, which is pre-trained to take a surgical field image 21 as input and output the region of marker M in the input surgical field image 21.

[0057] The display control unit 56 controls the display of the superimposed image 26, which is created by superimposing the ultrasound image 22 onto the surgical field image 21, based on position and orientation information. Specifically, the display control unit 56 adjusts the display mode of the ultrasound image 22 in the superimposed image 26 based on position and orientation information. As an example, the display control unit 56 generates a superimposed image 26 that shows the imaging range (e.g., position, orientation, and size) of the ultrasound image 22 in the surgical field SF by superimposing the ultrasound image 22 onto an area on the surgical field image 21 corresponding to the imaging range of the ultrasound image 22.

[0058] Specific examples of superimposed images 26 will be explained with reference to Figures 6 to 9. Figures 6 and 8 are conceptual diagrams showing the position and orientation of the tip 14B of the ultrasound probe 14 within the surgical field SF, which is defined as a three-dimensional space, and each shows a different position and orientation. Figures 7 and 9 show examples of superimposed images 26 generated according to the position and orientation of the ultrasound probe 14 in the surgical field SF. Figure 7 is a superimposed image 26 displayed when the position and orientation of the tip 14B of the ultrasound probe 14 is as shown in Figure 6. Figure 9 is a superimposed image 26 displayed when the position and orientation of the tip 14B of the ultrasound probe 14 is as shown in Figure 8.

[0059] In the three-dimensional surgical field SF shown in Figures 6 and 8, the Z-axis is parallel to the optical axis of the camera 13B of the endoscope 13. Also, in the three-dimensional surgical field SF, the X-Y plane is parallel to the imaging plane of the camera 13B and is perpendicular to the optical axis. That is, the X-Y plane of the three-dimensional surgical field SF is parallel to the screen of the surgical field image 21 (Xin-Yin plane). The surgical field image 21 is a projected image obtained by projecting the surgical field SF from one viewpoint. In Figures 6 and 8, among the reference numerals 64 and 66 that constitute the marker M, those that are visible in the surgical field image 21 are shown in a dark color, and those that are not visible in the surgical field image 21 are shown in a light color.

[0060] In Figure 6, the axis AX of the tip 14B of the ultrasound probe 14 is perpendicular to the optical axis of the camera 13B within the three-dimensional surgical field SF (more specifically, the axis AX of the tip 14B is parallel to the X axis). In this case, as shown in Figure 7, the markers M that appear in the surgical field image 21 have orthogonal lines in the grid pattern 62 that are parallel to the X and Y axes, respectively. Also, reference numerals 64 and 66 appear in the surgical field image 21 at equal intervals.

[0061] On the other hand, Figure 8 shows a state in which, within the three-dimensional surgical field SF, the axis AX of the tip 14B of the ultrasound probe 14 is not perpendicular to the optical axis of the camera 13B, but is tilted in the depth direction parallel to the optical axis. The posture shown in Figure 8 is a state in which the axis AX of the tip 14B has been rotated by approximately -25 degrees around the Y axis from the posture shown in Figure 6. In this case, as shown in Figure 9, the marker M that appears in the surgical field image 21 appears such that the lines extending in the circumferential direction of the grid pattern 62 become shorter and the spacing between reference numerals 64 and 66 becomes shorter the further away it is from the camera 13B in the depth direction.

[0062] In this way, the shape of the marker M that appears in the surgical field image 21 changes depending on the orientation of the tip portion 14B. Based on the shape of the marker M in the surgical field image 21, the input unit 52 estimates the orientation of the tip portion 14B of the ultrasound probe 14 in the surgical field SF. Specifically, the input unit 52 detects the direction of the axis AX of the tip portion 14B in the surgical field SF as the orientation of the tip portion 14B.

[0063] Furthermore, if the position of the tip portion 14B changes within the surgical field SF, the position of the marker M captured in the surgical field image 21 also changes. The derivation unit 52 estimates the position of the tip portion 14B within the surgical field SF based on the position of the marker M. The position of the tip portion 14B is detected as the position of a reference point of the tip portion 14B, for example, which is provided at the tip position of the tip portion 14B. In addition, the shooting distance from the camera 13B to the marker M within the surgical field SF (i.e., the distance in the Z-axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M captured in the surgical field image 21. Based on this shooting distance and the dimensional information 45 of the tip portion 14B, which includes the dimensions of the known marker M, the derivation unit 52 derives the position coordinates of the reference point of the tip portion 14B within the surgical field SF.

[0064] In the example shown in Figure 6, the position of the tip 14B of the ultrasound probe 14 is estimated as information such as the position coordinates (X01, Y01, Z01) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as information such as being parallel to the XY plane and the XZ plane, and perpendicular to the YZ plane.

[0065] In the example shown in Figure 8, the position of the tip 14B of the ultrasound probe 14 is estimated as information such as the position coordinates (X02, Y02, Z02) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the direction of the axis AX of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is estimated as -25° with respect to the XY plane, parallel to the XZ plane, and 65° with respect to the YZ plane.

[0066] Once the position and orientation of the tip portion 14B within the surgical field SF are estimated by the derivation unit 52, the display control unit 56 estimates the position and orientation of the ultrasonic transducer 14C within the surgical field SF based on the estimated position and orientation of the tip portion 14B and the dimensional information 45. As described above, the straight-line distance between the reference point of the tip portion 14B and the ultrasonic transducer 14C, and the inclination angle of the ultrasonic transducer 14C with respect to the axis AX of the tip portion 14B are known information defined as dimensional information 45. By using this dimensional information 45, the position and orientation of the ultrasonic transducer 14C within the surgical field SF can be estimated based on the position and orientation of the tip portion 14B within the surgical field SF.

[0067] In the example shown in Figure 6, the position of the ultrasonic transducer 14C is estimated as information such as the position coordinates (X11, Y11, Z11) obtained by correcting the position coordinates (X01, Y01, Z01) of the reference point of the tip portion 14B within the surgical field SF using dimensional information 45. The orientation of the ultrasonic transducer 14C is defined as the orientation of the tip portion 14B with respect to the axis AX, and is a known inclination angle as dimensional information 45.

[0068] In the example shown in Figure 8, the position of the ultrasonic transducer 14C is estimated as information such as the position coordinates (X12, Y12, Z12) obtained by correcting the position coordinates (X02, Y02, Z02) of the reference point of the tip portion 14B within the surgical field SF using dimensional information 45. The orientation of the ultrasonic transducer 14C is defined as the orientation of the tip portion 14B with respect to the axis AX, which is a known inclination angle as dimensional information 45.

[0069] Furthermore, the display control unit 56 estimates the imaging range of the ultrasound image 22 in the surgical field SF based on the estimated position and orientation of the ultrasound transducer 14C. The imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer 14C. Therefore, if the position and orientation of the ultrasound transducer 14C in the surgical field SF are known, the imaging range of the ultrasound image 22 in the surgical field SF can also be estimated. For example, the imaging range of the ultrasound image 22 captured by a convex-type ultrasound transducer 14C is a fan-shaped area that radiates outward from the ultrasound transducer 14C. Figures 6 and 8 illustrate the imaging range 22R of the ultrasound image 22 in the surgical field SF.

[0070] The display control unit 56 then generates a superimposed image 26 by superimposing the ultrasound image 22 onto the region within the surgical field image 21 (Xin-Yin plane) that corresponds to the imaging range of the ultrasound image 22 in the surgical field SF (Xpb-Ypb plane). In the examples of Figures 6 and 7, the imaging range of the ultrasound image 22 in the surgical field SF (Xpb-Ypb plane) and the screen of the surgical field image 21 in the surgical field SF (Xin-Yin plane) are parallel. On the other hand, in the examples of Figures 8 and 9, the imaging range of the ultrasound image 22 in the surgical field SF (Xpb-Ypb plane) and the screen of the surgical field image 21 in the surgical field SF (Xin-Yin plane) are not parallel. The display control unit 56 takes into account the difference in coordinate systems between the surgical field image 21 and the ultrasound image 22, and applies transformations such as projection transformation, affine transformation, translation, rotation, enlargement, and reduction to the ultrasound image 22, and then generates a superimposed image 26 by superimposing it on the surgical field image 21.

[0071] According to the superimposed image 26, even if the position and orientation of the ultrasound transducer 14C are difficult to see in the surgical field image 21 due to reasons such as obstruction by organs, etc., and the positional relationship with the camera 13B, it becomes easier to grasp its position and orientation. Therefore, it becomes easier to adjust the position and orientation of the ultrasound probe 14 so that a desired area (e.g., tumor 27) is captured in the ultrasound image 22, which contributes to improving the operability of the ultrasound probe 14.

[0072] As shown in Figure 2, the ultrasound image 22 may have a margin area 22Q added to make the overall shape of the ultrasound image 22 rectangular. In this case, it is preferable for the display control unit 56 to generate the superimposed image 26 using the image with the margin area 22Q removed as the internal image. As described above, when the ultrasound probe 14 is of the convex type, the ultrasound image 22 is fan-shaped. The margin area 22Q is used to make this fan-shaped ultrasound image 22 rectangular in order to standardize image processing. In other words, the margin area 22Q does not show information necessary for medical treatment, such as the internal structure of the target area in the ultrasound image 22. By not including this margin area 22Q in the superimposed image 26, it is possible to avoid the surgical field image 21 in the superimposed image 26 becoming difficult to see. Figure 7, etc., shows the superimposed image 26 with the margin area 22Q removed.

[0073] The display control unit 56 controls the display 16 to display the generated superimposed image 26. The display control unit 56 may also control the display 16 to display the surgical field image 21 and the ultrasound image 22. In this case, the display control unit 56 may control the display to show the surgical field image 21, the ultrasound image 22, and the superimposed image 26 together or switchably on a single display 16. Furthermore, if there are multiple displays 16, the display control unit 56 may control the display to show the surgical field image 21, the ultrasound image 22, and the superimposed image 26 on different displays 16. The display control unit 56 may also accept a request to specify which images from the surgical field image 21, the ultrasound image 22, and the superimposed image 26 should be displayed on the display 16, and control the display to show the specified images on at least one display 16.

[0074] Furthermore, the surgical field image 21 and the ultrasound image 22 are output as videos. The display control unit 56 may control the display 16 to display at least one of the surgical field image 21, the ultrasound image 22, and the superimposed image 26 in live view. Live view display means displaying images generated at a predetermined frame rate based on signals output by an imaging device for imaging the target as a video in real time. The imaging device includes, for example, an image sensor that optically images the target including the endoscope 13, and an ultrasound transducer 14C that images the target using ultrasound. When the superimposed image 26 is displayed in live view, the output unit 52 and the display control unit 56 repeat the output of position and orientation information and the generation of the superimposed image 26 each time the surgical field image 21 and the ultrasound image 22 are acquired, or at predetermined time intervals.

[0075] Incidentally, if the ultrasound transducer 14C is not in proper contact with the target site (e.g., liver LV), a proper ultrasound image 22 may not be obtained. For example, if the ultrasound transducer 14C is separated from the target site and air is present between them, the ultrasound will be reflected and attenuated by the air, making it impossible to detect ultrasound echoes from the target site. In this case, the ultrasound image 22 will not show information such as the internal structure of the target site, so even if the superimposed image 26 is displayed, it will not be able to properly support medical treatment. Furthermore, if the superimposed image 26 is displayed while the ultrasound probe 14 is being operated, the ultrasound image 22, which shows nothing, may interfere with the observation of the surgical field image 21, and may actually reduce convenience.

[0076] In response to this, a method has been considered in which the ultrasound image 22 is superimposed on the surgical field image 21 only when the ultrasound transducer 14C is in proper contact with the target area. However, if the ultrasound probe 14 is operated in a manner that rubs against the surface of a target area with irregularities, contact will occur at the convex areas and separation at the concave areas. In such cases, switching the presence or absence of superimposed display according to the contact or separation state would result in a deterioration of visibility due to flickering when passing through concave areas, which would actually reduce convenience.

[0077] Therefore, the medical support device 11 according to this embodiment has a function to switch the presence or absence of superimposed display depending on the contact state or separation state of the medical instrument (e.g., ultrasonic probe 14) while ensuring visibility.

[0078] The determination unit 54 determines whether the ultrasonic probe 14 is in a contact state, meaning it is in contact with the target area, or whether it is in a separated state, meaning it is separated from the target area. Specifically, the determination unit 54 determines whether a predetermined part of the medical device, such as the ultrasonic transducer 14C of the ultrasonic probe 14, is in contact with the target area. It should be noted that the term "contact state" in this disclosure is not limited to a state in which the medical device and the target area are in close contact, and there may be a gap between the medical device and the target area that is generally acceptable in the art field to which the technology of this disclosure belongs.

[0079] The specific method for determining whether the contact state or separation state is present is not particularly limited. Specific examples of determination methods are given below, but the method is not limited to these; known methods can be appropriately applied as methods for determining whether the contact state or separation state is present. The determination unit 54 may combine some or all of the following determination methods as appropriate.

[0080] For example, the determination unit 54 may determine whether the device is in contact or separated state based on at least one of the electrical signal corresponding to the ultrasonic echo detected by the ultrasonic probe 14 and the ultrasonic image 22. For example, the determination unit 54 may monitor the frequency components of the detected electrical signal over time and determine that the device has switched between contact and separated states when there is a change in the frequency components. Alternatively, for example, the determination unit 54 may store in advance the frequency components that can be detected in the contact state and determine that the device is in contact state when these frequency components are included in the detected electrical signal.

[0081] For example, the determination unit 54 may monitor numerical information such as the average brightness value of the ultrasonic image 22 over time and determine that the contact state has switched to the separation state when there is a change in the numerical information. For example, the determination unit 54 may detect a structure from the ultrasonic image 22 and determine that the contact state has switched to the separation state when there is a change in the shape of the structure over time (e.g., appearance or disappearance). For example, the numerical information and / or the shape of the structure that can be observed in the contact state may be stored in advance, and the contact state may be determined when the numerical information and / or the structure are included in the ultrasonic image 22.

[0082] As another example, the determination unit 54 may use a sensor such as a contact sensor or proximity sensor to determine whether the device is in contact or separation. For example, by providing a sensor on a part of the medical device that can come into contact with the target area, the sensor can determine whether the device is in contact or separation. Examples of such sensors include photosensors, contact sensors using piezoelectric elements, capacitive proximity sensors, and microswitches.

[0083] As another example, the determination unit 54 may determine whether the device is in contact or separated from the target area by image analysis based on the surgical field image 21, etc. For example, the determination unit 54 may derive the distance between the part of the medical device that can come into contact with the target area and the target area from the surgical field image 21, determine that the device is in contact if the distance is less than a predetermined threshold, and determine that the device is separated from the target area if the distance is equal to or greater than a predetermined threshold.

[0084] Alternatively, for example, AI technology using a machine learning model may be used to determine the contact or separation state from the surgical field image 21. As such a machine learning model, for example, a neural network model such as a CNN can be applied, which is pre-trained to take the surgical field image 21 as input and output the result of determining the contact or separation state in the input surgical field image 21. Alternatively, for example, a camera separate from the endoscope 13 may be inserted into the body, and the contact or separation state may be determined by analyzing the image taken by the other camera in addition to or instead of the surgical field image 21.

[0085] The display control unit 56 controls the display of the superimposed image 26 to stop if the ultrasonic probe 14 remains separated from the target area for a predetermined period Tx. The period Tx is, for example, 0.1 seconds to several seconds. This control will be explained below with reference to an example.

[0086] (Example 1) Figure 10 shows an example of each image displayed on the display 16 when the ultrasonic probe 14 is operated and the contact state and separation state are switched according to the elapsed time. In this example, the initial state is the contact state, the state switches from the contact state to the separation state at switching time t0, and the state switches from the separation state to the contact state at switching time t2. Time t1 is defined as the time when a period Tx has elapsed from switching time t0.

[0087] When describing the surgical field image 21, ultrasound image 22, and superimposed image 26, which change over time, each will be followed by a letter A to E. For example, the surgical field image 21, ultrasound image 22, and superimposed image 26 in the initial state will be designated as surgical field image 21A, ultrasound image 22A, and superimposed image 26A, respectively. Also, the surgical field image 21, ultrasound image 22, and superimposed image 26 immediately after the switching time t0 will be designated as surgical field image 21B, ultrasound image 22B, and superimposed image 26B, respectively.

[0088] As shown before the switching time t0 and after the switching time t2, the display control unit 56 controls the display to show the superimposed image 26 when in contact. Also, as shown between time t1 and the switching time t2, if the separation state continues for a predetermined period Tx, the display control unit 56 controls the display to show the surgical field image 21 instead of the superimposed image 26. This suppresses flicker caused by frequent switching of the display and ensures visibility, compared to, for example, switching the display from the superimposed image 26 to the surgical field image 21 the moment the separation state is entered without setting a period Tx. Furthermore, by stopping the display of the superimposed image 26 when the separation state continues for the period Tx, it is possible to avoid the continuous superimposition of an ultrasound image 22 with nothing in it on the surgical field image 21, thus avoiding interference with the observation of the surgical field image 21.

[0089] Furthermore, as shown between switching time t0 and time t1, the display control unit 56 may be controlled to display the superimposed image 26 during the period Tx. That is, during the period Tx, the superimposed image 26 including the ultrasound image 22 may be displayed regardless of whether or not the internal structure is visible in the ultrasound image 22. This makes it easier for the operator of the ultrasound probe 14 to understand whether or not the ultrasound transducer 14C is in contact with the target area, and can be used as a reference for adjusting the position and orientation of the ultrasound probe 14.

[0090] (Example 2) When an ultrasound image 22 showing nothing during the period Tx is superimposed, as in the ultrasound image 22B of Figure 10, the ultrasound image 22 itself may flicker intensely, reducing visibility, for example, when the contact state and separation state are frequently switched.

[0091] Therefore, the display control unit 56 may control the system to display a superimposed image 26, which is obtained by superimposing the ultrasound image 22 from a past point in time prior to the switching point t0 from the contact state to the separation state, onto the current surgical field image 21 within the period Tx. Figure 11 shows an example of each image displayed on the display 16 in this case. The method of illustration in Figure 11 and the progression of the contact state and separation state of the ultrasound probe 14 are the same as in the example in Figure 10.

[0092] As shown between switching time t0 and time t1, the display control unit 56 may control the display of the superimposed image 26, which includes the ultrasound image 22 acquired before switching time t0 (for example, immediately before), during the period Tx. That is, during the period Tx, the ultrasound image 22 to be superimposed is the ultrasound image 22A from a past time, and the position and orientation of the ultrasound image 22A in the superimposed image 26B may be adjusted based on the position and orientation information derived from the surgical field image 21B at the current time. This makes it possible to avoid flickering of the ultrasound image 22 itself, for example, when the contact state and separation state are frequently switched, thereby ensuring visibility.

[0093] (Example 3) As in Example 2, if an ultrasound image 22 from a past point in time rather than the present time is superimposed within the period Tx, there is a possibility that the observer may misinterpret the contact and separation states of the ultrasound probe 14 at the present time.

[0094] Therefore, when the display control unit 56 overlays an ultrasound image 22 from a past point in time within the period Tx, it may control the display control unit 56 to increase the transparency of the ultrasound image 22 in the overlaid image 26 over time from the switching point t0. Figure 12 shows an example of each image displayed on the display 16 in this case. In Figure 12, the transparency of the ultrasound image 22 is schematically shown as density, with darker colors indicating lower transparency. The other illustration methods in Figure 12, and the progress of the contact and separation states of the ultrasound probe 14, are the same as in the example in Figure 10.

[0095] Specifically, as shown between switching time t0 and time t1, the display control unit 56 may control the transparency of the ultrasound image 22A from the past time to be relatively low immediately after switching time t0, and then gradually increase the transparency of the ultrasound image 22A. In other words, the ultrasound image 22A in the superimposed image 26 may be faded out during the period Tx. This allows the observer to distinguish whether the ultrasound image 22 displayed in the superimposed image 26 is real-time or from the past, thereby improving convenience. Furthermore, by displaying the ultrasound image 22A from the past time in a semi-transparent state, the visibility of the surgical field image 21 can be improved.

[0096] (Example 4) As in Example 2, if an ultrasound image 22 from a past point in time rather than the present time is superimposed within the period Tx, there is a possibility that the observer may misinterpret the contact and separation states of the ultrasound probe 14 at the present time.

[0097] Therefore, the display control unit 56 may control the display to show a superimposed image 26 within the period Tx, which is obtained by superimposing the current ultrasound image 22 and the ultrasound image 22 from a past time point before the switching time t0 on the current surgical field image 21 at a predetermined ratio. Figure 13 shows an example of each image displayed on the display 16 in this case. The method of illustration in Figure 13 and the progression of the contact state and separation state of the ultrasound probe 14 are the same as in the example in Figure 10.

[0098] Specifically, as shown between switching time t0 and time t1, the display control unit 56 may be controlled to blend and superimpose the ultrasound image 22A from a past time and the ultrasound image 22B from the current time. This allows the observer to distinguish whether the ultrasound image 22 displayed in the superimposed image 26 is real-time or from the past, thereby improving convenience.

[0099] Furthermore, the display control unit 56 may control the transparency of the blended ultrasound image 22 in the superimposed image 26 over time from the switching time t0. For example, the display control unit 56 may control the transparency of the ultrasound image 22A from a past time in the superimposed image 26 over time from the switching time t0. That is, the visibility of the ultrasound image 22A from a past time may be gradually reduced. Figure 13 shows this configuration, where the visibility of the ultrasound image 22A from a past time is reduced in the superimposed image 26C compared to the superimposed image 26B, and an image closer to the current ultrasound image 22C (see Figure 10) is superimposed.

[0100] Alternatively, for example, the display control unit 56 may control the display control unit 56 to increase the transparency of the ultrasound image 22 at the current time and the ultrasound image 22A at past time points within the superimposed image 26 over time from the switching time t0. That is, similar to Embodiment 3, the blended ultrasound image 22 within the superimposed image 26 may be faded out during the period Tx.

[0101] These configurations make it easier to determine whether the ultrasound image 22 displayed on the superimposed image 26 is real-time or historical, thereby improving usability. Furthermore, displaying the ultrasound image 22A from a past point in time in a semi-transparent state improves the visibility of the surgical field image 21.

[0102] In each of the above embodiments 1 to 4, the display control unit 56 controls the system to display a superimposed image 26, which is the current ultrasound image 22 superimposed on the current surgical field image 21, if the system switches from a separated state to a contact state within the period Tx. That is, if the system returns to a contact state again within the period Tx, the system displays a superimposed image 26 using the real-time surgical field image 21 and ultrasound image 22. In this case, the real-time ultrasound image 22 may be displayed at the moment the system switches from a separated state to a contact state, or the real-time ultrasound image 22 may be gradually faded in from the moment the system switches from a separated state to a contact state. Also, if the system returns to a contact state again within the period Tx, the count for the period Tx is reset.

[0103] Furthermore, the display control unit 56 may accept the user's selection as to which of the above embodiments 1 to 4 to apply.

[0104] Furthermore, in each of the above embodiments 1 to 4, it is preferable that the display control unit 56 notifies that the ultrasonic probe 14 is in a separated state during the period Tx. Figure 14 shows an example of various information used to indicate that the probe is in a separated state.

[0105] For example, the display control unit 56 may be controlled to display at least one of text and / or a graphic indicating that the ultrasonic probe 14 is in a detached state during the period Tx. In Figure 14, an exclamation mark 90, which is displayed when the ultrasonic probe 14 is in a detached state, is displayed on the superimposed image 26.

[0106] Alternatively, for example, the display control unit 56 may indicate that the ultrasonic probe 14 is in a separated state by highlighting the frame of the ultrasonic image 22 within the superimposed image 26 during the period Tx. Examples of highlighting methods include differentiating the frame color, line type, and thickness from those used in the contact state. In Figure 14, the frame of the ultrasonic image 22, which was displayed as a solid line in the contact state (as shown in Figure 7), is now a dotted line, indicating that it is being highlighted.

[0107] Alternatively, for example, the display control unit 56 may be controlled to emit at least one of text and / or audio instructing the ultrasonic probe 14 to be in contact during the period Tx. In Figure 14, text 92 indicating the instruction "Please make contact" with the ultrasonic probe 14 is displayed on the superimposed image 26.

[0108] Next, the operation of the medical support device 11 according to this embodiment will be explained with reference to Figure 15. In the medical support device 11, the medical support process shown in Figure 15 is executed when the processor 41 executes the medical support program 44. This process is executed, for example, when the user gives an instruction to start execution via the reception device 46.

[0109] In step S10, the acquisition unit 50 acquires a surgical field image 21, which is an optical image of the surgical field SF including the target area inside the body and the ultrasound probe 14, captured by the camera 13B. In step S12, the output unit 52 derives position and orientation information indicating the position and orientation of the ultrasound probe 14 in the surgical field SF based on the surgical field image 21 acquired in step S10.

[0110] In step S14, the acquisition unit 50 acquires the ultrasound image 22 captured by the ultrasound probe 14. In step S16, the determination unit 54 determines whether the ultrasound probe 14 is in contact with the target area or whether the ultrasound probe 14 is separated from the target area.

[0111] If the determination result in step S16 is a contact state, step S18 is affirmed and the process proceeds to step S20. In step S20, the display control unit 56 controls the display of the superimposed image 26, which is obtained by superimposing the ultrasound image 22 obtained in step S14 onto the surgical field image 21 obtained in step S10, based on the position and orientation information derived in step S12.

[0112] On the other hand, if the determination result in step S16 is a separated state, step S18 is negated and the process proceeds to step S22. In step S22, the display control unit 56 determines whether the separated state has continued for a predetermined period Tx. If the separated state has not continued for the period Tx, step S22 is negated and the process proceeds to step S20.

[0113] On the other hand, if the separation state continues for a period Tx, step S22 is affirmed and the process proceeds to step S24. In step S24, the display control unit 56 controls the display of the superimposed image 26 to be stopped. When step S20 or step S24 is completed, this process ends.

[0114] As described above, the medical support device 11 according to this embodiment includes a processor 41. The processor 41 acquires a first image (surgical field image 21) optically captured by a camera 13B of the surgical field SF, which includes a target area inside the body and a medical instrument (ultrasound probe 14) inserted into the body and in contact with the target area to image the internal structure of the target area. The processor 41 also derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field SF based on the first image. The processor 41 also acquires a second image (ultrasound image 22) captured by the medical instrument. The processor 41 also controls the display of a superimposed image, which is obtained by superimposing the second image on the first image, based on the position and orientation information, and stops displaying the superimposed image if the separated state, in which the medical instrument is separated from the target area, continues for a predetermined period of time.

[0115] In other words, the medical support device 11 according to this embodiment can switch between having an overlay display or not depending on the contact state or separation state, and can avoid the deterioration of visibility due to flickering when the separation state occurs, for example by temporarily passing through a recess. Therefore, convenience can be further improved.

[0116] Next, we will describe some modifications applicable to the above embodiment. Note that some or all of the modifications shown below can be combined as appropriate.

[0117] As shown in Figure 16, the display control unit 56 may control the display of a scale 32 on the superimposed image 26 to indicate the size of internal structures (e.g., tumors 27) included in the ultrasound image 22, based on positional information. The size of the scale 32 can be calculated based on the distance from the camera 13B within the surgical field SF to the display plane of the internal image (Xpb-Ypb plane) (i.e., the distance in the Z-axis direction parallel to the optical axis). This scale 32 may or may not be displayed during the period Tx. In Figure 16, the scale 32 is displayed on the ultrasound image 22, but it is not limited to this, and the scale 32 may be displayed at any position within the superimposed image 26.

[0118] As shown in Figure 17, the display control unit 56 may control the display to show the puncture path 30 of the puncture needle 18 inserted into the body via the ultrasound probe 14 on the superimposed image 26, based on position and orientation information. The puncture path 30 can be generated as an extension of the guide groove 29, based on the known positional relationship between the position indicated by the position and orientation information (for example, the position of the reference point of the tip 14B of the ultrasound probe 14) and the guide groove 29.

[0119] The puncture route 30 is used as a guide when inserting the puncture needle 18 into the target position. For example, if the target position for inserting the puncture needle 18 is a tumor 27 in the liver LV, the ultrasound probe 14 is positioned by the medical staff ST so that the puncture route 30 and the tumor 27 overlap in the ultrasound image 22. In this state, the puncture needle 18 is inserted into the tumor 27 using the puncture route 30 as a guide. In this way, by positioning the ultrasound probe 14 so that the puncture route 30 passes through the target position and then passing the puncture needle 18 through the guide groove 29, the puncture needle 18 can reach the target position.

[0120] In addition to displaying the puncture route 30 on the ultrasound image 22, the display control unit 56 may also display the puncture route 30 on the surgical field image 21. In this case, the puncture route 30 indicates the path of the puncture needle 18 from the insertion position NP on the patient's PT body surface BS to the guide groove 29 of the ultrasound probe 14. These puncture routes 30 may or may not be displayed during the period Tx.

[0121] In the above embodiment, a configuration in which a marker M is attached to the tip 14B of the insertion portion 14A of the ultrasound probe 14 has been described, but the embodiment is not limited to this. The marker M can be attached to the portion of various medical devices that is inserted into the patient's body, for example, to the middle portion of the insertion portion 14A and to the proximal end of the insertion portion 14A.

[0122] In the above embodiment, a method was described in which positional and orientation information indicating the position and orientation of a medical instrument in the surgical field SF is derived using a marker M, but the embodiment is not limited to this. For example, positional and orientation information may be derived by detecting the characteristic shape of the medical instrument from the surgical field image 21 through image analysis.

[0123] In the above embodiment, an example of a medical device used to image the internal structure of a target area in a contact state, which is inserted into the patient's body and in contact with the target area, was described, but the embodiment is not limited to this. For example, a medical probe such as an optical coherence tomography (OCT) probe may be used as such a medical device.

[0124] In the above embodiment, cauterization was used as an example of the function of the puncture needle 18, but the function of the puncture needle 18 is not limited to this. Also, although the puncture needle 18 was used as an example of a treatment instrument, other instruments such as a treatment instrument for injecting a fluorescent agent such as ICG (Indocyanine Green), a biopsy needle used for tissue collection for biopsy, and forceps may be used instead.

[0125] Furthermore, although the above embodiments described body cavities such as the abdominal cavity and thoracic cavity as examples of internal body parts, internal body parts may also include the upper digestive tract such as the esophagus, the lower digestive tract such as the intestines, and ducts such as the bronchi. When applying the technology of this disclosure to a surgical field within a duct, for example, a marker M is provided at the proximal end of a flexible endoscope inserted into the duct.

[0126] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In this case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. The execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0127] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). The type of hardware may also be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these components may reside in physically separate devices or in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0128] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0129] Furthermore, although the above embodiment describes a configuration in which the medical support program 44 is pre-stored (installed) in the storage 43, the invention is not limited to this configuration. The medical support program 44 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the medical support program 44 may be provided in the form of a download from an external device via a network.

[0130] The technology disclosed herein extends to all program products. A program product includes all forms of products for providing programs. For example, a program product includes programs provided via a network such as the Internet, as well as non-temporary computer-readable recording media such as CD-ROMs, DVD-ROMs, and USB memory sticks on which programs are stored.

[0131] The technology of this disclosure can also be appropriately combined with the above-described embodiments and modifications. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above descriptions of the configuration, function, operation, and effect are examples of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not deviate from the spirit of the technology of this disclosure.

[0132] With respect to the above embodiment, the following additional notes are disclosed. [Addendum 1] A medical support device comprising a processor, wherein the processor optically captures a first image of a surgical field including a target area in the body and a medical instrument inserted into the body and in contact with the target area, which images the internal structure of the target area, using a camera; derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the first image; acquires a second image captured by the medical instrument; controls the display of a superimposed image obtained by superimposing the second image on the first image based on the position and orientation information; and discontinues the display of the superimposed image if the separation state, in which the medical instrument is separated from the target area, continues for a predetermined period of time. [Addendum 2] The medical support device according to Addendum 1, wherein the processor displays the first image instead of the superimposed image if the separation state continues for a predetermined period of time. [Addendum 3] The medical support device according to Addendum 1 or Addendum 2, wherein the processor displays the superimposed image within the period of time. [Note 4] The medical support device according to any one of Notes 1 to 3, wherein the processor displays a superimposed image, which is obtained by superimposing the second image at a past time prior to the time of switching from the contact state to the separation state onto the first image at the current time, during the period. [Note 5] The medical support device according to Note 4, wherein the processor increases the transparency of the second image in the superimposed image over time from the time of switching. [Note 6] The medical support device according to Note 4 or 5, wherein the processor displays a superimposed image, which is obtained by superimposing the second image at the current time onto the first image at the current time, when the system switches from the separation state to the contact state during the period. [Note 7] The medical support device according to any one of Notes 1 to 6, wherein the processor displays a superimposed image in which the current second image and the second image at a past time prior to the time of switching from the contact state to the separation state are superimposed on the current first image within the period, in a predetermined ratio. [Note 8] The medical support device according to Note 7, wherein the processor increases the transparency of the second image at the past time in the superimposed image over time from the time of switching.[Note 9] The medical support device according to Note 7 or Note 8, wherein the processor increases the transparency of the second image at the current time and the second image at past time in the superimposed image over time from the switching point. [Note 10] The medical support device according to any one of Notes 7 to 9, wherein the processor, when switching from the separation state to the contact state within the period, displays the superimposed image obtained by superimposing the second image at the current time onto the first image at the current time. [Note 11] The medical support device according to any one of Notes 1 to 10, wherein the processor notifies that the separation state is in place within the period. [Note 12] The medical support device according to Note 11, wherein the processor displays at least one of text and a graphic indicating that the separation state is in place within the period. [Note 13] The medical support device according to any one of Notes 1 to 12, wherein the processor highlights the frame of the second image in the superimposed image during the period. [Note 14] The medical support device according to any one of Notes 1 to 13, wherein the processor emits at least one of text and / or voice instructing to enter the contact state during the period. [Note 15] The medical device according to any one of Notes 1 to 14, wherein the medical instrument is an ultrasonic probe that transmits ultrasonic waves to the target area and detects an electrical signal corresponding to an ultrasonic echo reflected from the target area, the second image is an ultrasonic image generated in response to the electrical signal, and the processor determines whether the state is in contact or separation based on at least one of the electrical signal and the ultrasonic image.[Note 16] A medical support method comprising: acquiring a first image by optically capturing a surgical field with a camera, which includes a target area inside the body and a medical device inserted into the body and in contact with the target area, which images the internal structure of the target area; deriving position and orientation information indicating the position and orientation of the medical device in the surgical field based on the first image; acquiring a second image captured by the medical device; controlling the display of a superimposed image by superimposing the second image on the first image based on the position and orientation information; and stopping the display of the superimposed image if the separation state, in which the medical device is separated from the target area, continues for a predetermined period of time. [Note 17] A medical support program that optically captures a first image of a surgical field including a target area inside the body and a medical device inserted into the body and in contact with the target area to image the internal structure of the target area using a camera; derives position and orientation information indicating the position and orientation of the medical device in the surgical field based on the first image; acquires a second image captured by the medical device; controls the display of a superimposed image obtained by superimposing the second image on the first image based on the position and orientation information; and causes a computer to execute a process to stop displaying the superimposed image if the medical device remains separated from the target area for a predetermined period of time.

[0133] The disclosure of Japanese Patent Application No. 2024-171063, filed on 30 September 2024, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. A medical support device comprising a processor, the processor optically captures a first image of a surgical field including a target area in the body and a medical instrument inserted into the body and in contact with the target area, which images the internal structure of the target area, using a camera; derives position and orientation information indicating the position and orientation of the medical instrument in the surgical field based on the first image; acquires a second image captured by the medical instrument; controls the display of a superimposed image obtained by superimposing the second image on the first image based on the position and orientation information; and stops displaying the superimposed image if the medical instrument remains separated from the target area for a predetermined period of time.

2. The medical support device according to claim 1, wherein the processor displays the first image instead of the superimposed image if the separation state continues for a predetermined period of time.

3. The medical support device according to claim 1, wherein the processor displays the superimposed image within the specified period.

4. The medical support device according to claim 1, wherein the processor displays a superimposed image obtained by superimposing the second image from a past point in time prior to the point in time when the contact state is switched to the separation state onto the first image at the present time within the period.

5. The medical support device according to claim 4, wherein the processor increases the transparency of the second image in the superimposed image over time from the switching point.

6. The medical support device according to claim 4, wherein the processor switches from the separation state to the contact state during the period, and displays the superimposed image obtained by superimposing the current second image onto the current first image.

7. The medical support device according to claim 1, wherein the processor displays a superimposed image obtained by superimposing the current second image and the second image at a past time prior to the time of switching from the contact state to the separation state on the current first image within the period, at a predetermined ratio.

8. The medical support device according to claim 7, wherein the processor increases the transparency of the second image at the past time in the superimposed image over time from the switching point.

9. The medical support device according to claim 7, wherein the processor increases the transparency of the second image at the current time and the second image at past time within the superimposed image over time from the switching point.

10. The medical support device according to claim 7, wherein the processor switches from the separation state to the contact state during the period, and displays the superimposed image obtained by superimposing the current second image onto the current first image.

11. The medical support device according to claim 1, wherein the processor notifies that the separation state is in place during the period.

12. The medical support device according to claim 11, wherein the processor displays at least one of text and / or a graphic indicating the separation state during the period.

13. The medical support device according to claim 1, wherein the processor highlights the frame of the second image within the superimposed image during the period.

14. The medical support device according to claim 1, wherein the processor emits at least one of text and / or voice instructing the contact state to be established within the specified period.

15. The medical device is an ultrasonic probe that transmits ultrasonic waves to the target area and detects an electrical signal corresponding to an ultrasonic echo reflected from the target area, the second image is an ultrasonic image generated in accordance with the electrical signal, and the processor determines whether the area is in contact or separated based on at least one of the electrical signal and the ultrasonic image.

16. A medical support method comprising: acquiring a first image by optically capturing a surgical field with a camera, which includes a target area inside the body and a medical device inserted into the body and in contact with the target area, thereby imaging the internal structure of the target area; deriving position and orientation information indicating the position and orientation of the medical device in the surgical field based on the first image; acquiring a second image captured by the medical device; controlling the display of a superimposed image by superimposing the second image on the first image based on the position and orientation information; and, if the separation state in which the medical device is separated from the target area continues for a predetermined period of time, the computer performs a process to stop displaying the superimposed image.

17. A medical support program that optically captures a first image of a surgical field, including a target area inside the body and a medical device inserted into the body and in contact with the target area, which images the internal structure of the target area, using a camera; derives position and orientation information indicating the position and orientation of the medical device in the surgical field based on the first image; acquires a second image captured by the medical device; controls the display of a superimposed image, which is obtained by superimposing the second image onto the first image, based on the position and orientation information; and causes a computer to execute a process to stop displaying the superimposed image if the medical device remains separated from the target area for a predetermined period of time.

Images