Medical support device, medical support method, and medical support program

Abstract

This invention provides a medical support device, a medical support method, and a medical support program that can assist in the operation of an ultrasound probe. [Solution] The medical support device includes a processor, which acquires a surgical field image captured by a camera showing the ultrasound probe scanning the surgical field, derives posture information of the ultrasound probe based on the surgical field image, and adjusts the display mode of the ultrasound image captured by the ultrasound probe based on the posture information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a medical support device, a medical support method, and a medical support program.

BACKGROUND ART

[0002] Conventionally, there is known a technology that identifies the position and orientation of a medical instrument, and supports medical practices such as surgery and examination based on the identified position and orientation of the medical instrument.

[0003] For example, Patent Document 1 discloses deriving position and orientation information of a medical device in a captured image based on markers included in the captured image, and displaying a composite image in which medical support information is superimposed on a position specified in the captured image based on the position and orientation information. Further, for example, Patent Document 2 discloses performing at least rotation processing on at least one of a radiation image and an ultrasound image such that the region of interest captured in the radiation image and the region of interest captured in the ultrasound image have the same orientation as each other.

PRIOR ART DOCUMENTS

PATENT DOCUMENTS

[0004]

PATENT DOCUMENT 1

PATENT DOCUMENT 2

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] Generally, when an ultrasound probe is placed on the body surface for imaging, the operator and the patient, who is lying supine, face each other, meaning the operator's gaze is directed from the patient's tail towards their head. In this case, the correspondence between the top, bottom, left, and right of the ultrasound image displayed on the screen appears natural to the operator. Specifically, the right side of the patient's body becomes the left side of the ultrasound image, and the left side of the patient's body becomes the right side of the ultrasound image. This makes it easier to operate the ultrasound probe as the direction in which the ultrasound image moves intuitively matches the direction in which the ultrasound probe moves from side to side.

[0006] In recent years, in procedures such as laparoscopic surgery and robotic surgery, ultrasound probes are manipulated via surgical field images captured by a camera. In such surgeries, there may be constraints on the position and orientation of the ultrasound probe, making it impossible to determine a consistent orientation from the surgeon's viewpoint, i.e., the camera's viewpoint. Consequently, the direction of movement of the ultrasound probe on the surgical field image may differ from the direction of movement of the ultrasound image, making intuitive manipulation difficult.

[0007] This disclosure provides a medical support device, a medical support method, and a medical support program that can assist in the operation of an ultrasound probe. [Means for solving the problem]

[0008] A first aspect of this disclosure is a medical support device comprising a processor, the processor acquiring a surgical field image captured by a camera showing the scanning of an ultrasound probe in the surgical field, deriving posture information of the ultrasound probe based on the surgical field image, and adjusting the display mode of the ultrasound image captured by the ultrasound probe based on the posture information.

[0009] The processor may calculate the angle between the orientation of the ultrasound probe indicated by the posture information and a predetermined reference direction, and adjust the display mode depending on whether the angle meets the predetermined standard.

[0010] The processor may be controlled to display the ultrasound image inverted horizontally or vertically if the angle exceeds a predetermined threshold.

[0011] The reference direction is a predetermined direction in a three-dimensional coordinate system, and the processor may derive three-dimensional orientation information of the ultrasound probe based on the surgical field image.

[0012] The reference direction is a predetermined direction in the two-dimensional coordinate system on the surgical field image. The processor may derive two-dimensional pose information of the ultrasound probe based on the surgical field image.

[0013] The processor may control the display of information indicating the orientation of the ultrasound probe, based on posture information, around the ultrasound image or superimposed on the ultrasound image.

[0014] The processor may determine over time whether the ultrasound probe is in contact with the target area being imaged by the ultrasound probe, adjust the display mode when the ultrasound probe is separated from the target area, and fix the display mode when the ultrasound probe is in contact with the target area.

[0015] The processor may determine whether the ultrasound probe is in a separated state or in contact with the target site based on at least one of the electrical signal corresponding to the reflected ultrasound waves transmitted by the ultrasound probe to the target site and the ultrasound image.

[0016] The processor may derive posture information and position information of the ultrasound probe in the surgical field based on the surgical field image, and control the display of a superimposed image in which the ultrasound image is superimposed on the surgical field image, with the display mode and display position of the ultrasound image adjusted based on the posture information and position information.

[0017] The camera is an endoscope, and the ultrasound probe may be inserted into the body and in contact with the target area to photograph the internal structure of the target area.

[0018] A second aspect of the present disclosure is a medical support method, wherein a computer executes processes of acquiring a surgical field image captured by a camera showing a state where an ultrasonic probe is being scanned in a surgical field, deriving posture information of the ultrasonic probe based on the surgical field image, and adjusting a display mode of an ultrasonic image captured by the ultrasonic probe based on the posture information.

[0019] A third aspect of the present disclosure is a medical support program that causes a computer to execute processes of acquiring a surgical field image captured by a camera showing a state where an ultrasonic probe is being scanned in a surgical field, deriving posture information of the ultrasonic probe based on the surgical field image, and adjusting a display mode of an ultrasonic image captured by the ultrasonic probe based on the posture information. Effects of the Invention

[0020] According to the above aspects, the medical support device, the medical support method, and the medical support program of the present disclosure can support the operation of the ultrasonic probe. Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing the state inside the body in robotic surgery. [Figure 2] It is a diagram showing an example of the hardware configuration of the medical support device. [Figure 3] It is a diagram showing an example of the functional configuration of the medical support device. [Figure 4] It is a diagram showing the positional and posture relationship between a marker and an ultrasonic probe. [Figure 5] It is a diagram showing an example of a surgical field image. [Figure 6] It is a diagram showing the positional and posture relationship between a marker and an ultrasonic probe. [Figure 7] It is a diagram showing an example of a surgical field image. [Figure 8] It is a diagram for explaining the correspondence relationship of the movement directions between the ultrasonic probe and the ultrasonic image. [Figure 9]This diagram illustrates the correspondence between the movement direction of the ultrasound probe and the ultrasound image. [Figure 10] This diagram illustrates the correspondence between the movement direction of the ultrasound probe and the movement direction of the horizontally inverted ultrasound image. [Figure 11] This figure shows an example of a screen displayed on a display. [Figure 12] This is a diagram illustrating the processing of the determination unit. [Figure 13] This is a flowchart showing an example of medical support processing. [Figure 14] This figure shows an example of a screen displayed on the display in the second embodiment. [Figure 15] This figure shows an example of a screen displayed on the display in the second embodiment. [Figure 16] This figure shows a modified example of the screen displayed on the display in the second embodiment. [Figure 17] This figure shows an example of a screen displayed on the display in the third embodiment. [Figure 18] This figure shows an example of a screen displayed on the display in the third embodiment. [Modes for carrying out the invention]

[0022] The following describes an example of an embodiment of the disclosed technology with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals, and redundant descriptions are omitted. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0023] [First Embodiment] Referring to Figure 1, an example of a medical support system 10 to which the medical support device 11 according to this embodiment is applied will be described. The medical support system comprises the medical support device 11, an endoscope 13, and an ultrasound probe 14. The medical support device 11 is communicated with the endoscope 13 and the ultrasound probe 14.

[0024] As an example, medical support systems are used when performing surgery on a patient PT using an endoscopic surgical support robot (hereinafter referred to as robotic surgery). Robotic surgery is a procedure in which various medical instruments connected to robotic arms are inserted through multiple small incisions in the patient PT's body, and the surgeon remotely controls the system via a dedicated console or similar device.

[0025] The medical support system 10 provides medical staff, including the surgeon, with a field of view of the surgical field SF inside the patient's body, as well as support information to assist in medical treatment such as surgery and examinations. As described later, the support information includes ultrasound images 22 corresponding to the orientation of the ultrasound probe 14 on the surgical field image 21. Because such a medical support system 10 has the function of providing support information in real time during surgery, it is also called a surgical navigation system.

[0026] Figure 1 shows an endoscope 13, an ultrasound probe 14, and forceps 18 being inserted into the abdomen of patient PT. These various medical instruments are inserted into the body via a trocar 17. In addition, pneumoperitoneum is performed by injecting carbon dioxide into the abdominal cavity. The trocar 17 is an insertion device equipped with an insertion port for inserting various medical instruments and a valve provided within the insertion port to prevent gas leakage.

[0027] In the example shown in Figure 1, the forceps 18 grasps the ultrasound probe 14. The position and orientation of the ultrasound probe 14 are controlled by a robotic arm (not shown) connected to the forceps 18. Note that the medical instruments used in the medical support system 10 are not limited to the endoscope 13, ultrasound probe 14, and forceps 18. For example, a puncture needle, biopsy needle, injection needle, snare, electrosurgical unit, and high-frequency knife may also be used.

[0028] The endoscope 13 optically images the surgical field SF, which includes the target area inside the patient's body (in this example, the liver LV), using a camera. The surgical field SF is the space within the body cavity defined by organs and the body wall. Specifically, the endoscope 13 has an insertion section that is inserted into the patient's body. The tip of the insertion section contains a camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)). The endoscope 13 is, for example, a rigid endoscope with a rigid insertion section, and is often used for abdominal cavity observation, hence it is also called a laparoscope.

[0029] Camera 13B includes an image sensor such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of the subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images. The endoscope 13 is connected to an image processing processor for the endoscope (not shown). This image processing processor performs signal processing on the imaging signal output by the image sensor to generate a surgical field image 21 of the surgical field SF inside the body. The surgical field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via the image processing processor for the endoscope. Camera 13B of the endoscope 13 is an example of a "camera" in this disclosure.

[0030] For illumination of the endoscope 13, visible light such as white light is used, for example. Alternatively, special light such as ultraviolet and infrared light may be used as illumination for the endoscope 13. The special light may be limited to a specific wavelength, such as short-wavelength narrowband light obtained by narrowing the bandwidth of light in the short-wavelength region, such as the ultraviolet region. The surgical field image 21 is an image of the surgical field SF illuminated by the illumination light, and more specifically, it is an image based on reflected light from the illumination light reflected near the surface of the surgical field SF. Therefore, while structures near the surface of the target area can be depicted in the surgical field image 21, internal structures are difficult to observe.

[0031] The ultrasound probe 14, while in contact with the target area inside the body, transmits ultrasound to the target area and detects electrical signals corresponding to the ultrasound echoes reflected from the target area, thereby capturing images of the internal structure of the target area. Specifically, the ultrasound probe 14 has an ultrasound transducer. The ultrasound transducer transmits ultrasound to the target area and receives ultrasound echoes reflected from the target area. The ultrasound probe 14 is connected to an image processing processor for the ultrasound probe (not shown). This image processing processor performs image reconstruction processing based on the electrical signals corresponding to the ultrasound echoes received by the ultrasound transducer, so that an ultrasound image corresponding to the electrical signals is generated. The image reconstruction processing generates an ultrasound image 22 showing the internal structure of the target area scanned by the ultrasound probe 14. The ultrasound image 22 captured by the ultrasound probe 14 is transmitted in real time to the medical support device 11 via the image processing processor for the ultrasound probe.

[0032] The ultrasound probe 14 is, for example, a convex type that transmits ultrasound radially, and a fan-shaped ultrasound image 22 is acquired with the ultrasound transducer as the base point. By scanning the ultrasound probe 14, multiple ultrasound images 22 are taken along the scanning direction. The ultrasound image 22 is a so-called B (Brightness) mode image that visualizes the internal structure from the surface to the deep layers reached by the ultrasound in the target area as brightness information. The ultrasound image 22 visualizes the internal structure of the target area that cannot be observed with the surgical field image 21 obtained by optical imaging.

[0033] The medical support device 11 acquires surgical field images 21 from the endoscope 13 and ultrasound images 22 from the ultrasound probe 14. The view of the inside of the patient's body is provided to the medical staff through the display screen. In robotic surgery, the surgeon performs surgery and examinations by referring to these images and operating the robotic arms via a dedicated console.

[0034] The detailed configuration of the medical support device 11 is described below. Figure 2 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a processor 41, RAM (Random Access Memory) 42, storage 43, display 45, reception device 46, communication I / F (Interface) 47, and external I / F 48. Each of these parts is connected to a bus 49, such as a system bus and a control bus, and can communicate with each other.

[0035] The processor 41 is, for example, a CPU (Central Processing Unit) that comprehensively controls each part of the medical support device 11 according to a control program and executes various processes according to various application programs. The processor 41 is an example of the "processor" in this disclosure.

[0036] RAM42 is a memory that temporarily stores information and is used as work memory by the processor 41. Examples of RAM42 include DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0037] The storage 43 is a non-volatile memory device that stores various programs and parameters. Examples of storage 43 include HDDs (Hard Disk Drives) and SSDs (Solid State Drives). The storage 43 stores medical support programs 44 that enable the computer to function as a medical support device 11.

[0038] Furthermore, storage 43 stores dimensional information 59. The dimensional information 59 includes information about the geometric size and shape of the medical device (ultrasound probe 14 and forceps 18, etc.) to which the markers described later are attached. This includes, for example, information indicating the length, width, diameter, height, thickness, and structural characteristics (such as straight or curved) of the medical device as a whole and / or a specific part of the medical device (such as the tip). The dimensional information 59 also includes information about the shape of the markers and the placement of the markers on the medical device. This includes, for example, information indicating how the markers are positioned relative to the axial and circumferential directions of the medical device.

[0039] The display 45 displays various types of information, including the surgical field image 21 and the ultrasound image 22. Examples of the display 45 include liquid crystal displays and EL (Electro-Luminescence) displays. The number of displays 45 is not particularly limited and may be one or more.

[0040] The reception device 46 has a keyboard and mouse (not shown) and receives instructions from the operator. In other words, the medical support device 11 is operated by an operator such as medical staff through the reception device 46. The reception device 46 may be a device that accepts touch input such as a touch panel, a device that accepts voice input such as a microphone, and a device that accepts gesture input such as a camera.

[0041] Communication I / F47 is connected to a network such as a LAN (Local Area Network) and / or a WAN (Wide Area Network) (not shown), and performs transmission control according to communication protocols specified by various wired or wireless communication standards. External I / F48 is, for example, a USB (Universal Serial Bus) interface and is used to connect to peripheral devices such as printers and memory cards. As the medical support device 11, for example, a server computer, personal computer, smartphone, tablet terminal, and wearable terminal can be appropriately applied.

[0042] Next, the functional configuration of the medical support device 11 will be described. Figure 3 is a block diagram showing an example of the functional configuration of the medical support device 11. The medical support device 11 includes an acquisition unit 50, an output unit 52, a display control unit 54, and a determination unit 56 as functional units. The processor 41 reads the medical support program 44 from the storage 43 and executes the medical support program 44 on the RAM 42, thereby allowing the processor 41 to function as each functional unit. Medical support processing is realized through the operation of the processor 41 as each functional unit.

[0043] The acquisition unit 50 acquires a surgical field image 21, which is captured by the camera 13B as the ultrasound probe 14 is scanning the surgical field SF. For example, the acquisition unit 50 acquires the surgical field image 21 from a device including the processor of the endoscope 13 via an external I / F 48 or a communication I / F 47. The processor of the endoscope 13 may be located in the medical support device 11.

[0044] Furthermore, the acquisition unit 50 acquires the ultrasound image 22 captured by the ultrasound probe 14. For example, the acquisition unit 50 acquires the ultrasound image 22 from the device including the processor of the ultrasound probe 14 via an external I / F 48 or a communication I / F 47. Note that the processor of the ultrasound probe 14 may be located in the medical support device 11.

[0045] The derivation unit 52 derives positional information of the ultrasound probe 14 based on the surgical field image 21. Specifically, the derivation unit 52 derives three-dimensional positional information of the ultrasound probe 14 based on the surgical field image 21. Furthermore, the derivation unit 52 may also derive positional information of the ultrasound probe 14 in the surgical field SF based on the surgical field image 21.

[0046] For example, the input unit 52 inputs posture information and position information of the ultrasound probe 14 based on the marker 14M included in the surgical field image 21. The marker 14M is a marker that can be recognized from the surgical field image 21 optically captured by the camera 13B and is attached to the ultrasound probe 14. In addition, the shape of the marker 14M and information regarding the placement of the marker 14M on the ultrasound probe 14 are known and are pre-stored in the storage 43 as dimensional information 59.

[0047] Specifically, the derivation unit 52 detects the marker 14M by searching for its morphological features in the surgical field image 21. For example, the derivation unit 52 may detect the marker 14M using image processing techniques such as pattern matching. Alternatively, the derivation unit 52 may detect the marker 14M using AI (Artificial Intelligence) technology with a machine learning model. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) that has been pre-trained to take the surgical field image 21 as input and output the region of the marker 14M in the input surgical field image 21 can be applied.

[0048] The derivation unit 52 then derives posture information and position information of the ultrasound probe 14 based on the detected marker 14M. The method for deriving posture information and position information will be explained with reference to Figures 4 to 7. Figures 4 and 6 are conceptual diagrams showing the tip of the ultrasound probe 14 in the surgical field SF defined as a three-dimensional space, and each shows a different position and posture. Figure 5 is a surgical field image 21 taken when the position and posture of the tip of the ultrasound probe 14 are as shown in Figure 4. Figure 7 is a surgical field image 21 taken when the position and posture of the tip of the ultrasound probe 14 are as shown in Figure 6.

[0049] As an example, marker 14M is a marker of a grid pattern composed of a first line extending in the direction of the axis AX of the tip of the ultrasonic probe 14 and a second line perpendicular to the direction of the axis AX of the tip and formed circumferentially along the outer surface of the tip. Each intersection in the grid pattern is assigned either a circular or rectangular symbol.

[0050] The marker 14M may include at least one of the lines forming the grid pattern and the symbols placed at each intersection of the grid pattern. The symbols in the marker 14M can be of any shape, and may be represented by figures such as triangles, polygons, stars, and various marks, or by letters. Furthermore, two-dimensional codes and geometric patterns (e.g., ArUco and ChArUco) may be used as the marker 14M.

[0051] In the three-dimensional surgical field SF shown in Figures 4 and 6, the Z-axis is parallel to the optical axis of the camera 13B. Furthermore, the XY plane in the surgical field SF is parallel to the imaging plane (Xin-Yin plane) of the camera 13B and perpendicular to the optical axis. That is, the surgical field image 21 corresponds to a projected image obtained by projecting the surgical field SF from a single viewpoint O. In Figures 4 and 6, among the symbols constituting the marker 14M, those that appear in the surgical field image 21 are shown in darker colors, while those that do not appear in the surgical field image 21 are shown in lighter colors.

[0052] The orientation of the ultrasound probe 14 is detected, for example, as the direction of the axis AX of the tip of the ultrasound probe 14 within the surgical field SF. The direction of axis AX is expressed, for example, as the inclination angle with respect to the X axis, Y axis, and Z axis, respectively.

[0053] For example, Figure 4 shows a state in which the axis AX of the tip of the ultrasound probe 14 is parallel to the X axis within the surgical field SF in three-dimensional space. In this case, as shown in Figure 5, the marker 14M that appears in the surgical field image 21 has orthogonal lines of the grid pattern parallel to the X axis and Y axis, respectively. Also, the symbols forming the grid pattern appear at equal intervals in the surgical field image 21.

[0054] On the other hand, Figure 6 shows a state in which the axis AX of the tip of the ultrasound probe 14 is tilted with respect to the depth direction parallel to the optical axis of imaging within the surgical field SF in three-dimensional space. In this case, as shown in Figure 7, the marker 14M captured in the surgical field image 21 has shorter lines extending in the circumferential direction of the grid pattern and the spacing between symbols forming the grid pattern becomes narrower the further it is from the camera 13B in the depth direction.

[0055] In this way, the shape of the marker 14M that appears in the surgical field image 21 changes depending on the orientation of the ultrasound probe 14. Based on the shape of the marker 14M (projected grid pattern, etc.) that changes in this way within the surgical field image 21, the output unit 52 outputs the direction of the axis AX of the tip of the ultrasound probe 14 as orientation information.

[0056] Furthermore, if the position of the ultrasound probe 14 changes within the surgical field SF, the position of the marker 14M captured in the surgical field image 21 also changes. The derivation unit 52 may derive positional information of the ultrasound probe 14 in the surgical field SF based on the position of the marker 14M. The position of the ultrasound probe 14 is detected, for example, as the positional coordinates (X, Y, Z) of a predetermined reference point (e.g., tip position) on the ultrasound probe 14.

[0057] The shooting distance (i.e., the coordinate in the Z-axis direction) from camera 13B to marker 14M can be calculated based on the focal length of camera 13B and the size of marker 14M as it appears in the surgical field image 21. Specifically, by associating the size of marker 14M detected from the surgical field image 21 with the information of marker 14M contained in the dimensional information 59, the change in projection scale can be estimated, and thus the shooting distance can be calculated. For example, the further the tip is from camera 13B, the smaller the apparent size of marker 14M appears, and the closer it is, the larger it appears. The derivation unit 52 derives the shooting distance using this geometric relationship.

[0058] Figures 4 and 6 also show the imaging range 22R of the ultrasound image 22 in the surgical field SF, and the angle θ representing the orientation of the ultrasound image 22. The angle θ is smallest when the imaging optical axis (Z axis) of the camera 13B is parallel to the display plane (Xpb-Ypb plane) of the ultrasound image 22, and largest when the imaging optical axis is the normal. In Figures 4 and 6, for clarity, a straight line Zp parallel to the Z axis passing through the display plane of the ultrasound image 22 is shown. The angle θ can be derived based on the posture information of the ultrasound probe 14 and the dimensional information 59.

[0059] Incidentally, the correspondence between the up, down, left, and right directions of the ultrasound image 22 and the orientation of the ultrasound probe 14 is predetermined. Specifically, the horizontal direction (Xpb direction) of the ultrasound image 22 is associated with the longitudinal or short-length direction of the ultrasound probe 14. For example, the longitudinal tip of the ultrasound probe 14 is associated with the right side of the ultrasound image 22, and the longitudinal proximal end of the ultrasound probe 14 is associated with the left side of the ultrasound image 22. In this case, if the ultrasound probe 14 is moved left or right in an orientation such that its longitudinal direction is visible in the surgical field image 21, the ultrasound image 22 will also move left or right.

[0060] Here, as shown in Figure 8, consider the case where the surgeon moves the ultrasound probe 14 to the right (Xin direction, shown by a thick arrow) on the surgical field image 21 while referring to the surgical field image 21 from time T1 to T3. As in the example in Figure 8, if the tip of the ultrasound probe 14 is positioned to the right of the surgical field image 21, the imaging range of the ultrasound image 22 also moves to the right (Xpb direction, shown by a thick arrow), making the operation easier to perform intuitively. Note that in Figure 8, for ease of understanding, the position of the tumor 27 included in the ultrasound image 22 is also shown on the surgical field image 21 with a dotted line. Also, the illustration of the forceps 18 is omitted.

[0061] On the other hand, Figure 9 shows an example where the position of the ultrasound probe 14 is different from that in Figure 8. Similar to the above, consider the case where the operator moves the ultrasound probe 14 to the right (Xin direction, shown by a thick arrow) on the surgical field image 21 from time T1 to T3. As in the example in Figure 9, when the tip of the ultrasound probe 14 is positioned to the left of the surgical field image 21, the imaging range of the ultrasound image 22 moves to the left (minus Xpb direction, shown by a thick arrow). That is, the direction of movement of the ultrasound probe 14 and the movement of the ultrasound image 22 are opposite, making intuitive operation difficult.

[0062] Therefore, the display control unit 54 adjusts the display mode of the ultrasound image 22 captured by the ultrasound probe 14 based on the orientation information of the ultrasound probe 14. "Adjusting the display mode" includes changing the appearance of the ultrasound image 22 itself, such as inverting, rotating, enlarging, shrinking, and changing the display position on the screen. It also includes, for example, maintaining the ultrasound image 22 itself while adding information indicating the orientation of the ultrasound probe 14 around it. This enables intuitive operation regardless of the orientation of the ultrasound probe 14.

[0063] Specifically, the display control unit 54 calculates the angle φ between the orientation of the ultrasonic probe 14 indicated by the posture information and a predetermined reference direction. The orientation of the ultrasonic probe 14 is represented, for example, by a vector pointing from the proximal end to the tip on the axis AX of the tip of the ultrasonic probe 14. The reference direction is a predetermined direction in a three-dimensional coordinate system. In the examples of Figures 8 and 9, the angle φ is illustrated with the X-axis of the surgical field SF, that is, the direction of the Xin axis on the surgical field image 21 (from left to right on the page), as the reference direction in a three-dimensional coordinate system (XYZ coordinate system) with the viewpoint O of the camera 13B as the origin. In Figure 8, the angle φ is approximately 15 degrees, and in Figure 9, the angle φ is approximately 180 degrees.

[0064] The display control unit 54 then adjusts the display mode of the ultrasound image 22 depending on whether the angle φ meets a predetermined standard. For example, the display control unit 54 may control the display to flip the ultrasound image 22 horizontally if the angle φ is greater than or equal to a predetermined threshold. The threshold can be set arbitrarily; for example, setting it to 90 degrees allows for fine adjustment of the display mode according to the orientation of the ultrasound probe 14. Alternatively, setting the threshold to around 91 to 120 degrees allows for horizontal flipping only when the orientation of the ultrasound probe 14 is significantly reversed.

[0065] Figure 10 shows the ultrasound image 22 from the example in Figure 9 reversed horizontally. In this case, the longitudinal tip of the ultrasound probe 14 corresponds to the left side of the reversed ultrasound image 22F, and the longitudinal proximal end of the ultrasound probe 14 corresponds to the right side of the reversed ultrasound image 22F. As a result, when the operator moves the ultrasound probe 14 to the right (Xin direction, shown by a thick arrow) on the surgical field image 21 from time T1 to T3, the direction of movement of the ultrasound probe 14 coincides with the movement of the reversed ultrasound image 22F. Therefore, intuitive operation becomes possible.

[0066] Furthermore, it is preferable for the display control unit 54 to notify the user if the ultrasound image 22 is horizontally inverted. Figure 11 shows an example of screen D1 displayed on the display 45 by the display control unit 54. Screen D1 includes the surgical field image 21, the horizontally inverted ultrasound image 22F, and a notification 90 that says "Horizontal inversion in progress". By enabling the operator to immediately understand whether or not the ultrasound image 22 is horizontally inverted, more appropriate operation becomes possible.

[0067] Furthermore, the display control unit 54 may adjust the display mode of the ultrasonic image 22 in steps according to the angle φ. For example, the display control unit 54 may control the display to show the ultrasonic image 22 as is if the angle φ is 0 to 90 degrees, to show a warning and then show the ultrasonic image 22 as is if the angle is 91 to 100 degrees, and to show the ultrasonic image 22F after horizontal inversion if the angle is 101 degrees or more.

[0068] Furthermore, the display control unit 54 may individually set a threshold for initiating the left-right inversion of the ultrasonic image 22 with respect to the angle φ, and a threshold for releasing the left-right inversion. In this case, it is preferable that the initiation threshold is larger than the release threshold. By providing hysteresis in this way, it is possible to prevent the display mode from being frequently changed when the angle φ is fluctuating around the threshold.

[0069] Furthermore, the display control unit 54 may refrain from horizontal inversion when the angle φ is changing rapidly, i.e., when the time derivative of the angle φ is large, and perform horizontal inversion of the ultrasound image 22 when the time derivative of the angle φ falls below a predetermined threshold. This prevents the display from being frequently changed in response to temporary movements such as hand tremors.

[0070] Furthermore, depending on the insertion direction of the ultrasound probe 14 into the surgical field SF and its positional relationship with the camera 13B, it may be effective to invert the image vertically. The display can be adjusted for vertical inversion in the same way as for horizontal inversion. That is, the display control unit 54 may control the display to invert the ultrasound image 22 vertically when the angle φ is greater than or equal to a predetermined threshold.

[0071] The correspondence between the top, bottom, left, and right of the ultrasound image 22 and the orientation of the ultrasound probe 14 described above is just one example, and any setting can be made according to the usage and the operator's preference. For example, the longitudinal tip of the ultrasound probe 14 may be set to correspond to the left side of the ultrasound image 22, and the longitudinal proximal end of the ultrasound probe 14 may be set to correspond to the right side of the ultrasound image 22. Alternatively, for example, the right side in the short direction when viewed from the proximal end of the ultrasound probe 14 may be set to correspond to the right side of the ultrasound image 22, and the left side in the short direction when viewed from the proximal end of the ultrasound probe 14 may be set to correspond to the left side of the ultrasound image 22.

[0072] Furthermore, the above-mentioned reference direction is merely an example, and any setting is possible depending on the usage and the operator's preference. Specifically, the three-dimensional coordinate system for representing the position and orientation of the camera 13B and the ultrasound probe 14 can be set arbitrarily. For example, it may be set according to at least one of the position and orientation of the camera 13B or the ultrasound probe 14, or a world coordinate system may be set. The reference direction is defined as a predetermined direction vector in these three-dimensional coordinate systems and is compared with the orientation of the ultrasound probe 14 represented in the said three-dimensional coordinate system.

[0073] Specifically, the three-dimensional coordinate system can be a local coordinate system set according to the position and orientation of the ultrasound probe 14 (for example, a coordinate system with the tip of the ultrasound probe 14 as the origin, the longitudinal direction as the X-axis, the transverse direction as the Y-axis, and the irradiation direction as the Z-axis). In this case, the angle φ between the orientation of the ultrasound probe 14 and the reference direction can be calculated by converting and projecting the reference direction defined in the three-dimensional coordinate system (camera coordinate system) with the viewpoint O of the camera 13B as the origin to the local coordinate system. Furthermore, even if the reference direction is a fixed direction in the world coordinate system (for example, the direction of gravity and the horizontal direction), the angle φ between the orientation of the ultrasound probe 14 and the reference direction can be calculated by converting and projecting the said reference direction to the camera coordinate and / or local coordinate system.

[0074] (modified version) As shown in Figure 12, imaging may be performed on a single tumor 27 while changing the orientation of the ultrasound probe 14 between times T1 and T5. In such continuous imaging situations, horizontally reversing the ultrasound image 22 between times T3 and T4 may actually reduce visibility. In particular, when the direction of the axis AX of the tip of the ultrasound probe 14 and the imaging optical axis of the camera 13B are nearly parallel, the angle φ may fluctuate near the threshold, potentially causing the control of horizontal reversal to switch frequently, which may make it difficult for the operator to understand. Therefore, the medical support device 11 may choose not to adjust the display mode of the ultrasound image 22 while imaging is being performed by the ultrasound probe 14.

[0075] Specifically, the determination unit 56 may determine over time whether the ultrasound probe 14 is in contact with the target area being imaged by the ultrasound probe 14. That is, the determination unit 56 may sequentially determine whether the ultrasound probe 14 is in a separated state, separated from the target area, or in a contact state, in contact with the target area. It should be noted that "contact" in this disclosure is not limited to a state in which the ultrasound probe 14 and the target area are in close contact, and there may be a gap between the ultrasound probe 14 and the target area of ​​a degree that is generally acceptable in the art to which the technology of this disclosure belongs.

[0076] The specific method for determining whether the contact state or separation state is present is not particularly limited. Specific examples of determination methods are given below, but the method is not limited to these; known methods can be appropriately applied as methods for determining whether the contact state or separation state is present. The determination unit 56 may combine some or all of the following determination methods as appropriate.

[0077] For example, the determination unit 56 may determine whether the ultrasonic probe 14 is in a separated state or a contact state based on at least one of the electrical signal corresponding to the reflected ultrasonic waves transmitted by the ultrasonic probe 14 to the target site and the ultrasonic image 22. For example, the determination unit 56 may monitor the frequency components of the detected electrical signal over time and determine that the contact state and separated state have switched when there is a change in the frequency components. Alternatively, for example, the unit may store in advance the frequency components that can be detected in the contact state and determine that the contact state is present when those frequency components are included in the detected electrical signal.

[0078] For example, the determination unit 56 may monitor numerical information such as the average brightness value of the ultrasonic image 22 over time and determine that the contact state has switched to the separation state when there is a change in the numerical information. For example, the determination unit 56 may detect a structure from the ultrasonic image 22 and determine that the contact state has switched to the separation state when there is a change in the shape of the structure over time (e.g., appearance or disappearance). For example, the numerical information and / or the shape of the structure that can be observed in the contact state may be stored in advance, and the contact state may be determined when the numerical information and / or the structure are included in the ultrasonic image 22.

[0079] As another example, the determination unit 56 may use a sensor such as a contact sensor or proximity sensor to determine whether the device is in contact or separation. For example, by providing a sensor on a part of the ultrasonic probe 14 that can come into contact with the target area, the sensor can determine whether the device is in contact or separation. Examples of such sensors include photosensors, contact sensors using piezoelectric elements, capacitive proximity sensors, and microswitches.

[0080] As another example, the determination unit 56 may determine whether the device is in contact or separation state by image analysis based on the surgical field image 21, etc. For example, the determination unit 56 may derive the distance between the ultrasound probe 14 and the target site from the surgical field image 21, determine that the device is in contact state if the distance is less than a predetermined threshold, and determine that the device is in separation state if the distance is equal to or greater than a predetermined threshold. Alternatively, for example, AI technology using a machine learning model may be used to determine whether the device is in contact or separation state from the surgical field image 21. As such a machine learning model, for example, a neural network model such as a CNN that has been pre-trained to take the surgical field image 21 as input and output the result of determining whether the device is in contact or separation state in the input surgical field image 21 can be applied. Alternatively, for example, a camera other than the endoscope 13 may be inserted into the body, and the device may determine whether the device is in contact or separation state by analyzing the image taken by the other camera in addition to or instead of the surgical field image 21.

[0081] The display control unit 54 may adjust the display mode of the ultrasound image 22 when the ultrasound probe 14 is separated from the target area. In other words, the display control unit 54 may fix the display mode of the ultrasound image 22 when the ultrasound probe 14 is in contact with the target area, and update the display mode only when the device is separated. For example, in Figure 12, the display control unit 54 may continuously apply the display mode (without horizontal inversion) determined at time T1 to times T2 to T5.

[0082] Next, the operation of the medical support device 11 according to this embodiment will be explained with reference to Figure 13. In the medical support device 11, the medical support process shown in Figure 13 is executed by the processor 41 executing the medical support program 44. This process is executed, for example, when the user gives an instruction to start execution via the reception device 46.

[0083] In step S10, the acquisition unit 50 acquires the surgical field image 21 and the ultrasound image 22. In step S12, the output unit 52 derives the orientation information of the ultrasound probe 14 based on the surgical field image 21. In step S14, the display control unit 54 calculates the angle φ between the orientation of the ultrasound probe 14 indicated by the orientation information and a predetermined reference direction.

[0084] In step S16, the display control unit 54 determines whether the angle φ meets a predetermined criterion. If step S16 is determined to be positive, the process proceeds to step S18, where the display control unit 54 controls the display to flip the ultrasound image 22 horizontally (or vertically) and then terminates the process. If step S16 is determined to be negative, the process proceeds to step S20, where the display control unit 54 controls the display to display the ultrasound image 22 as is (without horizontal or vertical flipping) and then terminates the process.

[0085] As described above, the medical support device 11 according to this embodiment includes a processor, which acquires a surgical field image 21 captured by the camera 13B showing the ultrasound probe 14 scanning the surgical field SF. Based on the surgical field image 21, the processor derives posture information of the ultrasound probe 14. Based on the posture information, the processor adjusts the display mode of the ultrasound image 22 captured by the ultrasound probe 14.

[0086] According to the medical support device 11 of this embodiment, the direction of movement of the ultrasound probe 14 on the surgical field image 21 can be correlated with the direction of movement of the ultrasound image 22. That is, the correspondence between the position inside the organ being observed in the ultrasound image 22 and the position of the ultrasound probe 14 in the surgical field SF can be represented simply and intuitively, improving the operability of the ultrasound probe 14 via the camera 13B. Therefore, the operation of the ultrasound probe 14 can be supported.

[0087] [Second Embodiment] In this embodiment, as another method for adjusting the display mode of the ultrasound image 22, a configuration in which information indicating the orientation of the ultrasound probe 14 is displayed will be described. In the following description, some explanations that overlap with the first embodiment will be omitted.

[0088] The display control unit 54 controls the display to show information indicating the orientation of the ultrasound probe 14 around the ultrasound image 22, based on the orientation information of the ultrasound probe 14. Figures 14 and 15 show examples of screens D2 and D3 displayed on the display 45 by the display control unit 54.

[0089] Screen D2 contains a surgical field image 21, an ultrasound image 22 that is not horizontally or vertically flipped, and an orientation mark 92. The orientation mark 92 is an example of information indicating the orientation of the ultrasound probe 14, and consists of symbols and figures that indicate the tip (or proximal) end of the ultrasound probe 14. In the example in Figure 14, the orientation mark 92 is a star-shaped figure that indicates the longitudinal tip end of the ultrasound probe 14. In addition to or instead of the orientation mark 92, the top, bottom, left, and right directions in the surgical field image 21 may be indicated by text.

[0090] Screen D3 includes a surgical field image 21, an ultrasound image 22 that is not horizontally or vertically inverted, and a schematic diagram 94 of the ultrasound probe 14. Schematic diagram 94 is an example of information indicating the orientation of the ultrasound probe 14, and is a diagram intended to intuitively show the orientation of the ultrasound probe 14.

[0091] According to the orientation mark 92 and / or schematic diagram 94, it is visually clear which direction on the screen the tip of the ultrasound probe 14 is pointing. This makes it easier to see that, for example, if the ultrasound probe 14 is moved to the right on the surgical field image 21, the ultrasound image 22 moves to the left, but it is clear that the ultrasound image 22 is moving towards the proximal end of the ultrasound probe 14. In other words, the correspondence between the position inside the organ being observed in the ultrasound image 22 (e.g., tumor 27) and the position of the ultrasound probe 14 in the surgical field SF can be represented simply and intuitively, improving the operability of the ultrasound probe 14 via the camera 13B. Therefore, it is possible to support the operation of the ultrasound probe 14.

[0092] The display control unit 54 may also control the display to superimpose information indicating the orientation of the ultrasound probe 14 onto the ultrasound image 22 based on the orientation information of the ultrasound probe 14. For example, in screen D2 of Figure 14, the orientation mark 92 may be displayed superimposed on the ultrasound image 22.

[0093] Furthermore, the first and second embodiments may be combined. That is, the display control unit 54 may, when the angle φ is greater than or equal to a predetermined threshold, display the ultrasound image 22 inverted horizontally, while simultaneously displaying information indicating the orientation of the ultrasound probe 14 around the ultrasound image 22. Figure 16 shows an example of a screen D4 displayed on the display 45 by the display control unit 54. Screen D4 includes a surgical field image 21, a horizontally inverted ultrasound image 22, and a schematic diagram 94 of the ultrasound probe 14. Unlike the example in Figure 15, the schematic diagram 94 is horizontally inverted to match the horizontally inverted ultrasound image 22. With this configuration, the correspondence of the position of the ultrasound probe 14 in the surgical field SF becomes easier to grasp visually, and the operability of the ultrasound probe 14 via the camera 13B can be improved.

[0094] [Third Embodiment] In this embodiment, as another method for adjusting the display mode of the ultrasound image 22, a form in which the ultrasound image 22 is superimposed on the surgical field image 21 will be described. In the following description, some descriptions that overlap with the first or second embodiment will be omitted.

[0095] The display control unit 54 controls the display to show a superimposed image 26, which is an ultrasound image 22 superimposed on a surgical field image 21, with the display mode and position of the ultrasound image 22 adjusted based on the posture information and position information of the ultrasound probe 14. Figures 17 and 18 show examples of screens D4 and D6 displayed on the display 45 by the display control unit 54.

[0096] Screens D5 and D6 each contain a superimposed image 26 and a notification 90 that reads "Ultrasound image being reversed left to right." The position and orientation of the ultrasound probe 14 differ between screens D5 and D6. Screen D5 corresponds to the state shown in Figures 4 and 5. Screen D6 corresponds to the state shown in Figures 6 and 7.

[0097] Specifically, the display control unit 54 estimates the position and orientation of the ultrasonic transducer within the surgical field SF based on the posture information and position information of the ultrasonic probe 14, as well as the dimensional information 59. The straight-line distance between the reference point of the position indicated by the position information and the ultrasonic transducer, and the inclination angle of the ultrasonic transducer with respect to the direction of axis AX indicated by the posture information, are known information defined as dimensional information 59. By using this dimensional information 59, the position and orientation of the ultrasonic transducer within the surgical field SF can be estimated from the posture information and position information of the ultrasonic probe 14.

[0098] The display control unit 54 estimates the imaging range of the ultrasound image 22 in the surgical field SF based on the estimated position and orientation of the ultrasound transducer. The imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer. For example, the imaging range of the ultrasound image 22 acquired by a convex-type ultrasound transducer is a fan-shaped area that radiates outward from the ultrasound transducer (see Figures 4 and 6).

[0099] The display control unit 54 then generates a superimposed image 26 by superimposing the ultrasound image 22 onto the region within the surgical field image 21 (Xin-Yin plane) that corresponds to the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the surgical field SF. In the example in Figure 17, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the surgical field SF and the screen of the surgical field image 21 in the surgical field SF (Xin-Yin plane) are parallel. On the other hand, in the example in Figure 18, the imaging range (Xpb-Ypb plane) of the ultrasound image 22 in the surgical field SF and the screen of the surgical field image 21 in the surgical field SF (Xin-Yin plane) are not parallel. The display control unit 54 takes into account the difference in coordinate systems between the surgical field image 21 and the ultrasound image 22, and generates a superimposed image 26 by applying transformations such as projection transformation, affine transformation, translation, rotation, enlargement, and reduction to the ultrasound image 22 and then superimposing it onto the surgical field image 21.

[0100] Furthermore, the display control unit 54 adjusts the display mode of the ultrasound image 22 according to whether the angle φ between the orientation of the ultrasound probe 14 indicated by the posture information and a predetermined reference direction satisfies a predetermined standard. For example, in the examples of Figures 17 and 18, the ultrasound image 22 is horizontally inverted, similar to the first embodiment. In addition, similar to the second embodiment, the system may be controlled to display information indicating the orientation of the ultrasound probe 14 around the ultrasound image 22 or to superimpose it on the ultrasound image 22.

[0101] According to the superimposed image 26, even if the position and orientation of the ultrasound probe 14 cannot be seen or are difficult to see in the surgical field image 21 due to reasons such as obstruction by organs, etc., and the positional relationship with the camera 13B, its position and orientation can be easily determined. Therefore, it becomes easier to adjust the position and orientation of the ultrasound probe 14 so that the desired area (e.g., tumor 27) is captured in the ultrasound image 22, contributing to improved operability.

[0102] In the embodiments described above, the posture information of the ultrasound probe 14 has been described as three-dimensional posture information, but it is not limited to this. The posture information of the ultrasound probe 14 may also be two-dimensional. For example, the derivation unit 52 may derive two-dimensional posture information of the ultrasound probe 14 based on the surgical field image 21. In this case, the "reference direction" used to calculate the angle φ may be a predetermined direction in the two-dimensional coordinate system (Xin-Yin plane) on the surgical field image 21.

[0103] In the embodiments described above, the position and orientation of the ultrasound probe 14 in the surgical field SF were derived using a marker 14M, but the invention is not limited to this. For example, the position and orientation may be derived by detecting the characteristic shape of the ultrasound probe 14 from the surgical field image 21 through image analysis. Alternatively, the position and orientation of the ultrasound probe 14 may be derived using AI technology with a machine learning model. As such a machine learning model, for example, a neural network model such as a CNN that has been pre-trained to take the surgical field image 21 as input and output the position and orientation of the ultrasound probe 14 in the input surgical field image 21 can be applied.

[0104] In the embodiments described above, a configuration was described in which the scanning of the ultrasound probe 14 in the surgical field SF is captured by a camera 13B on a single endoscope 13. However, the number of cameras 13B may be multiple. For example, multiple endoscopes 13 may be inserted into the surgical field SF (at different positions and / or angles), or cameras 13B may be provided on the trocar 17 and the body wall inside the body. In this case, the surgical field image 21 used for each process may be specified by the user, or the processor 41 may switch them automatically.

[0105] Specifically, the acquisition unit 50 acquires the surgical field images 21 captured by each camera 13B, and accepts the user's specification of which surgical field image 21 to use. The derivation unit 52 derives the posture information of the ultrasound probe 14 based on the specified surgical field image 21. The display control unit 54 adjusts the display mode of the ultrasound image 22 captured by the ultrasound probe 14 based on the posture information of the ultrasound probe 14 in the specified surgical field image 21. If the specified surgical field image 21 is changed, each function unit performs its respective processing based on the changed surgical field image 21.

[0106] Furthermore, if the processor 41 automatically switches which surgical field image 21 to use, it is conceivable that it would select a surgical field image 21 in which the ultrasound probe 14 is easily visible. The acquisition unit 50 acquires the surgical field images 21 captured by each camera 13B. The derivation unit 52 derives the posture information of the ultrasound probe 14 based on each surgical field image 21. The display control unit 54 selects a surgical field image 21 based on the posture information of each ultrasound probe 14 and adjusts the display mode of the ultrasound image 22 captured by the ultrasound probe 14 based on the posture information of the ultrasound probe 14 in the selected surgical field image 21. For example, the smaller the angle θ (see Figures 4 and 6) representing the orientation of the ultrasound image 22, the more difficult it becomes to grasp the position and orientation of the ultrasound probe 14 in the surgical field image 21. Therefore, the display control unit 56 may select a surgical field image 21 in which the angle θ representing the orientation of the ultrasound image 22 is close to vertical.

[0107] For example, the display control unit 56 may select a surgical field image 21 captured by a camera 13B that is close to the ultrasound probe 14, based on the position information of the ultrasound probe 14. This improves operability because the ultrasound probe 14 is displayed larger in the surgical field image 21. Alternatively, the display control unit 56 may combine the above examples to select a surgical field image 21 based on the posture information and position information of the ultrasound probe 14. If the positional relationship between the ultrasound probe 14 and each camera 13B changes due to the movement of the ultrasound probe 14 and / or camera 13B, the display control unit 54 may re-select the surgical field image 21.

[0108] In the embodiments described above, the surgical field SF was described using body cavities such as the abdominal cavity and thoracic cavity as examples, but it is not limited to these. For example, the surgical field SF may be the upper digestive tract such as the esophagus, the lower digestive tract such as the intestines, or the inside of a duct such as the bronchi.

[0109] While the embodiments described above describe how the medical support system 10 of this disclosure is applied to robotic surgery, it is not limited to these embodiments. For example, the medical support system 10 of this disclosure may be applied to laparoscopic surgery. Alternatively, for example, the medical support system 10 of this disclosure may be applied to remote surgery via the body surface by using a webcam as the camera and an ultrasound probe that scans the body surface as the ultrasound probe.

[0110] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0111] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0112] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0113] Furthermore, although the above embodiment describes a configuration in which the medical support program 44 is pre-stored (installed) in the storage 43, the invention is not limited to this configuration. The medical support program 44 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the medical support program 44 may be provided in the form of a download from an external device via a network.

[0114] The technology disclosed herein extends to all program products. A program product includes all forms of products for providing programs. For example, a program product includes programs provided via a network such as the Internet, as well as non-temporary computer-readable recording media such as CD-ROMs, DVD-ROMs, and USB memory sticks on which programs are stored.

[0115] The technology of this disclosure can also be appropriately combined with the above-described embodiments and modifications. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above descriptions of the configuration, function, operation, and effect are examples of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not deviate from the spirit of the technology of this disclosure.

[0116] The following additional information is disclosed regarding the above embodiments. [Note 1] The processor comprises, We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. Medical support device. [Note 2] The aforementioned processor, The angle between the orientation of the ultrasonic probe indicated by the posture information and a predetermined reference direction is calculated. The display mode is adjusted depending on whether the angle meets a predetermined standard. Medical support device as described in Appendix 1. [Note 3] The processor controls the display of the ultrasound image by flipping it horizontally or vertically when the angle is greater than or equal to a predetermined threshold. Medical support devices as described in Appendix 2. [Note 4] The aforementioned reference direction is a predetermined direction in the three-dimensional coordinate system. The processor derives three-dimensional orientation information of the ultrasound probe based on the surgical field image. Medical support devices as described in Appendix 2 or Appendix 3. [Note 5] The aforementioned reference direction is a predetermined direction in the two-dimensional coordinate system on the surgical field image. The processor derives two-dimensional pose information of the ultrasound probe based on the surgical field image. Medical support devices as described in Appendix 2 or Appendix 3. [Note 6] The processor controls the display of information indicating the orientation of the ultrasound probe, based on the posture information, to display it around the ultrasound image or to superimpose it on the ultrasound image. A medical support device as described in any one of the appendices 1 through 5. [Note 7] The aforementioned processor, The ultrasound probe is determined over time to determine whether or not it is in contact with the area to be imaged by the ultrasound probe. When the ultrasonic probe is separated from the target area, the display mode is adjusted. The display mode is fixed when the ultrasonic probe is in contact with the target area. A medical support device as described in any one of the appendices 1 through 6. [Note 8] The processor determines whether the ultrasonic probe is in the separated state or the contact state based on at least one of the electrical signal corresponding to the reflected ultrasound waves transmitted by the ultrasonic probe to the target area and the ultrasonic image. Medical support devices as described in Appendix 7. [Note 9] The aforementioned processor, Based on the surgical field image, the posture information and the position information of the ultrasound probe in the surgical field are derived. A superimposed image obtained by superimposing the ultrasound image onto the surgical field image, wherein the superimposed image is controlled to display the ultrasound image with adjusted display mode and display position based on the posture information and position information. A medical support device as described in any one of the appendices 1 through 8. [Note 10] The aforementioned camera is an endoscope, The ultrasound probe is inserted into the body and, while in contact with the target area, captures images of the internal structure of the target area. A medical support device as described in any one of the appendices 1 through 9. [Note 11] We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. A medical support method in which a computer performs the processing. [Note 12] We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. A medical support program that uses computers to perform processing. [Explanation of Symbols]

[0117] 10. Medical support systems 11 Medical support devices 13 Endoscopy 13B Camera 14. Ultrasound probe 14M Marker 15 Drive unit 16. Auxiliary endoscope 17 Trocar 18 forceps 21 Surgical field image 22 Ultrasound images 22F Ultrasound image after left-right reversal 22R shooting range 26 Superimposed images 27 Tumors 41 processors 42 RAM 43 Storage 44 Medical support programs 45 displays 46 Reception devices 47 Communication I / F 48 External I / F 49 bus 50 Acquisition Department 52 Derivation part 54 Display Control Unit 56 Judgment section 59 Dimensional Information 90 notifications 92 Orientation Mark 94 Schematic diagram AX axis Screens D1-D6 LV Liver PT patient SF field

Claims

1. The processor comprises, We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. Medical support device.

2. The aforementioned processor, The angle between the orientation of the ultrasonic probe indicated by the posture information and a predetermined reference direction is calculated. The display mode is adjusted depending on whether the angle meets a predetermined standard. The medical support device according to claim 1.

3. The processor controls the display of the ultrasound image by flipping it horizontally or vertically when the angle is greater than or equal to a predetermined threshold. The medical support device according to claim 2.

4. The aforementioned reference direction is a predetermined direction in the three-dimensional coordinate system. The processor derives three-dimensional orientation information of the ultrasound probe based on the surgical field image. The medical support device according to claim 2.

5. The aforementioned reference direction is a predetermined direction in the two-dimensional coordinate system on the surgical field image. The processor derives two-dimensional pose information of the ultrasound probe based on the surgical field image. The medical support device according to claim 2.

6. The processor controls the display of information indicating the orientation of the ultrasound probe, based on the posture information, to display it around the ultrasound image or to superimpose it on the ultrasound image. The medical support device according to claim 1.

7. The aforementioned processor, The ultrasound probe is determined over time to determine whether or not it is in contact with the area to be imaged by the ultrasound probe. When the ultrasonic probe is separated from the target area, the display mode is adjusted. The display mode is fixed when the ultrasonic probe is in contact with the target area. The medical support device according to claim 1.

8. The processor determines whether the ultrasonic probe is in the separated state or the contact state based on at least one of the electrical signal corresponding to the reflected ultrasound waves transmitted by the ultrasonic probe to the target area and the ultrasonic image. The medical support device according to claim 7.

9. The aforementioned processor, Based on the surgical field image, the posture information and the position information of the ultrasound probe in the surgical field are derived. A superimposed image obtained by superimposing the ultrasound image onto the surgical field image, wherein the superimposed image is controlled to display the ultrasound image with adjusted display mode and display position based on the posture information and position information. The medical support device according to claim 1.

10. The aforementioned camera is an endoscope, The ultrasound probe is inserted into the body and, while in contact with the target area, captures images of the internal structure of the target area. The medical support device according to claim 1.

11. We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. A medical support method in which a computer performs the processing.

12. We acquire surgical field images by taking pictures with a camera of the ultrasound probe scanning the surgical field. Based on the surgical field image, the positional information of the ultrasound probe is derived, Based on the posture information, the display mode of the ultrasound image captured by the ultrasound probe is adjusted. A medical support program that uses computers to perform processing.

Images