Medical assistance device and medical assistance method

Abstract

The present application provides a medical assistance device and a medical assistance method. The medical assistance device has one or more processors configured to: acquire a medical image, and generate a guide display for superimposition on the medical image, the guide display indicating a boundary between sections in which a degree of recommendation of a medical treatment in the medical image is different.

Description

Technical Field

[0001] This disclosure relates to medical assist devices and methods for assisting medical procedures such as laparoscopic surgery. Background Technology

[0002] Surgery still relies heavily on the surgeon's anatomical knowledge. This tendency is even stronger in laparoscopic surgery, where surgeons lack the tactile feedback to understand the structures inside the patient's body and rely solely on visual information.

[0003] Laparoscopic cholecystectomy is a surgical procedure aimed at removing the gallbladder. Therefore, it is necessary to identify and sequentially remove the cystic duct and cystic artery. In anatomically challenging situations, the surgeon must be careful not to misidentify anatomical structures. The surgeon must be particularly careful not to misidentify the common bile duct and cystic duct. Laparoscopic cholecystectomy is one of the procedures performed by less experienced surgeons after training, and may be performed by surgeons whose necessary anatomical knowledge and experience with general anatomical changes are insufficient.

[0004] To compensate for physicians' lack of anatomical knowledge and experience, surgical assistance systems are being considered to guide physicians to recommended treatment areas. For example, a surgical assistance system has been proposed to assist physicians in laparoscopic surgery by overlaying virtual images of the resection plane onto laparoscopic images (e.g., see Patent Document 1).

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2005-278888

[0008] As mentioned above, in laparoscopic cholecystectomy, in order to compensate for the surgeon's lack of anatomical knowledge and experience, there is a need to further improve the existing surgical support system. Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] This disclosure was made in view of the following circumstances, and its purpose is to provide a technique that enables the surgeon to recognize and treat highly recommended areas during surgery.

[0011] Methods for solving problems

[0012] To address the aforementioned issues, a medical assistive device according to one aspect of this disclosure has one or more processors configured to: acquire a medical image; and, based on the medical image, generate a guidance display for overlaying on the medical image, the guidance display representing the boundaries between segments in the medical image with varying degrees of recommendation for medical treatment.

[0013] Another aspect of the medical assistance method disclosed herein includes the steps of: acquiring a medical image; and generating a guidance display overlaid on the medical image, the guidance display representing the boundaries between segments in the medical image with different levels of recommendation for medical treatment.

[0014] Furthermore, any combination of the above-mentioned constituent elements, or any transformation of the present disclosure into methods, apparatus, systems, recording media, computer programs, etc., is also valid as a form of the present disclosure. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the structure of a medical assistance system according to an embodiment.

[0016] Figure 2 This is a diagram that roughly illustrates the overall process of laparoscopic cholecystectomy.

[0017] Figure 3 This is a diagram illustrating the method for generating a machine learning model in an embodiment.

[0018] Figure 4 This is an example of a laparoscopic image bearing the B-SAFE landmark.

[0019] Figure 5 This is an example of a laparoscopic image showing the Rouviere groove and the hilar plate of the liver.

[0020] Figure 6 This is a flowchart illustrating the basic operation of the medical assistive device in an embodiment.

[0021] Figure 7 This is an example of an image obtained by overlaying lines on a laparoscopic image taken during laparoscopic cholecystectomy.

[0022] Figure 8 This is an example of an image obtained by overlaying an unsafe area onto a laparoscopic image taken during laparoscopic cholecystectomy.

[0023] Figure 9 This is an example of an image obtained by overlaying a safety zone pattern on a laparoscopic image taken during laparoscopic cholecystectomy.

[0024] Figure 10 This is another example of an image obtained by overlaying a laparoscopic image taken during laparoscopic cholecystectomy with a guided display.

[0025] Figure 11This is an example of a laparoscopic image taken during a laparoscopic cholecystectomy, showing four subjects being examined.

[0026] Figure 12 It shows that Figure 11 The image shown is a magnified version of the laparoscopic image.

[0027] Figure 13 This is a diagram illustrating an example of the changes in laparoscopic images caused by the displacement of the laparoscope.

[0028] Figures 14A to 14B This is an example of an image obtained by overlaying the resection area, representing the location of the cystic duct to be removed, onto a laparoscopic image taken during laparoscopic cholecystectomy.

[0029] Figure 15 This is an example of an image obtained by overlaying three lines onto a laparoscopic image taken during laparoscopic cholecystectomy.

[0030] Figure 16 This is another example of an image obtained by overlaying three lines onto a laparoscopic image taken during laparoscopic cholecystectomy.

[0031] Figure 17 This is an example of an image obtained by overlaying animated graphics onto laparoscopic images taken during laparoscopic cholecystectomy.

[0032] Symbol Explanation

[0033] 1 Medical Assistance System, 2 Laparoscope, 3 Treatment Instruments, 10 Medical Assistance Devices, 11 Medical Image Acquisition Unit, 12 Segment Information Generation Unit, 13 Treatment Instrument Recognition Unit, 14 Learning Model Holding Unit, 15 Display Generation Unit, 20 Video Processor, 30 Monitor, GB Gallbladder, DU Duodenum, CD Cystic Duct, HD Hepatic Duct, HA Hepatic Artery, CBD Common Bile Duct, RS Rouviere Groove, S4b Lower Margin of Hepatic S4. Detailed Implementation

[0034] Figure 1 The structure of the medical assistance system 1 according to an embodiment is shown. The medical assistance system 1 is a system for assisting laparoscopic surgery used in the surgical department. In laparoscopic surgery, an abdominal surgical endoscope (hereinafter referred to as laparoscope 2) and a treatment device 3 are inserted through multiple openings in the patient's abdomen. The laparoscope 2 is not a flexible endoscope used for examinations of the stomach or colon, but a rigid endoscope made of metal. The treatment device 3 includes forceps, cannulas, energy devices, etc.

[0035] The medical assistance system 1 is installed in the operating room and includes a medical assistance device 10, a video processor 20, and a monitor 30. The medical assistance device 10 includes a medical image acquisition unit 11, a segment information generation unit 12, a treatment instrument recognition unit 13, a learning model holding unit 14, and a display generation unit 15.

[0036] The structure of the medical assistive device 10 can be implemented in hardware using any processor (e.g., CPU, GPU), memory, auxiliary storage device (e.g., HDD, SSD), other LSI, and in software using a program loaded into memory, etc., but functional blocks implemented through their cooperation are described herein. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms, either solely in hardware, solely in software, or through a combination thereof.

[0037] The laparoscope 2 has a light guide for transmitting illumination light supplied from a light source device to illuminate the patient's body. At its front end are: an illumination window for projecting the illumination light transmitted by the light guide onto the subject; and an imaging unit that captures images of the subject at predetermined intervals and outputs the image signals to a video processor 20. The imaging unit includes a solid-state imaging element (e.g., a CCD image sensor or a CMOS image sensor) that converts incident light into electrical signals.

[0038] The video processor 20 performs image processing on the camera signal converted from photoelectric image by the solid-state camera element of the laparoscope 2 to generate a laparoscopic image. In addition to conventional image processing such as A / D conversion and noise removal, the video processor 20 can also perform effect processing such as emphasis display.

[0039] The laparoscopic image generated by the video processor 20 is output to the medical assist device 10, and after being overlaid and guided by the medical assist device 10, it is displayed on the monitor 30. Additionally, as a failover function, it is also possible to bypass the medical assist device 10 and output the laparoscopic image directly from the video processor 20 to the monitor 30. Furthermore, in Figure 1 The diagram shows the structure where the medical assistive device 10 and the video processor 20 are separate devices, but the medical assistive device 10 and the video processor 20 can also be combined into one device.

[0040] Figure 2This diagram roughly illustrates the overall procedure for laparoscopic cholecystectomy. First, equipment is prepared in the operating room (P1). Medical support system 1 is activated in this step. Next, the patient is brought into the operating room (P2). Then, the anesthesiologist administers general anesthesia to the patient (P3). Next, the entire surgical team performs a timeout (P4). Specifically, the team confirms the patient, surgical site, and surgical procedure. Next, the surgeon sets up the access port to the abdominal cavity (P5). This involves making a small incision in the abdomen and inserting a cannula to create the access port. The laparoscope, forceps, and energy devices are inserted into the abdominal cavity using a tracker. Carbon dioxide is delivered into the abdominal cavity from the pneumoperitoneum device and the cavity is filled with gas to ensure sufficient space for the surgical procedure.

[0041] Next, the surgeon decompresses the gallbladder (P6) and retracts the gallbladder with forceps (P7). Then, the surgeon uses dissecting forceps to peel away the peritoneum covering the gallbladder (P8). Next, the surgeon exposes the cystic duct and cystic artery (P9), and then the gallbladder bed (P10). Next, the surgeon uses clamps to fix the cystic duct and cystic artery (P11) and uses scissors to cut them apart (P12). Finally, the surgeon uses dissecting forceps to detach the gallbladder from the gallbladder bed and places the detached gallbladder into a recovery bag for collection.

[0042] In steps P8 to P11, the medical assist device 10 involved in the embodiment assists the surgeon in confirming the safe area by overlaying the guidance display on the laparoscopic image displayed in real time on the monitor 30.

[0043] In the medical assistance system 1 of the embodiment, a machine learning model trained to identify important anatomical landmarks from real-time images during surgery is used.

[0044] Figure 3 This diagram illustrates the method for generating a machine learning model in an embodiment. In a simple case, annotators with expertise, such as doctors, annotate anatomical landmarks mapped onto multiple laparoscopic images. The AI / machine learning system uses a taught dataset of laparoscopic images annotated with anatomical landmarks as training data to perform machine learning, generating a machine learning model for landmark detection. For example, as machine learning, CNN, RNN, LSTM, etc., as a type of deep learning, can be used.

[0045] In challenging cases, a machine learning model is generated to detect safe regions in laparoscopic images of complex cases where anatomical landmarks are not visible. Safe regions within multiple laparoscopic images without visible anatomical landmarks are annotated by highly specialized annotators, such as skilled physicians. For example, safe region lines are drawn within the laparoscopic images. Additionally, the estimated locations of anatomical landmarks obscured by adipose tissue or similar structures can be annotated. The AI / machine learning system uses a taught dataset of laparoscopic images annotated with safe region associations as training data to generate a machine learning model for safe region detection.

[0046] A machine learning model for detecting at least one of the anatomical landmarks and safety areas generated as described above is registered in the learning model holding unit 14 of the medical assistive device 10.

[0047] In laparoscopic images of the gallbladder or surrounding tissues taken during laparoscopic cholecystectomy, the B-SAFE landmark can be used as anatomical landmarks. The B-SAFE landmarks include the bile duct (B), Rouviere groove (S), the lower border of hepatic S4 (S), the hepatic artery (A), the umbilical fissure (F), and intestinal structures (duodenum) (E). Furthermore, the hilar plate can also be used as an anatomical landmark. Figure 4 This is an example of a laparoscopic image bearing the B-SAFE logo. Figure 5 This is an example of a laparoscopic image showing the Rouviere groove and the hilar plate of the liver. Additionally, a laparoscopic image showing the surrounding tissues of the gallbladder refers to a laparoscopic image showing any one of the aforementioned B-SAFE landmarks, or at least one of the cystic duct or cystic artery.

[0048] Figure 6 This is a flowchart illustrating the basic operation of the medical assistance device 10 according to an embodiment. The medical image acquisition unit 11 acquires a medical image (in this embodiment, a laparoscopic image showing the gallbladder or a laparoscopic image showing the tissue surrounding the gallbladder) from the video processor 20 (S10). The segment information generation unit 12 generates segment information that defines the level of safety of a medical procedure (in this embodiment, laparoscopic cholecystectomy) by region based on the medical image acquired by the medical image acquisition unit 11 (S20). The display generation unit 15 generates a guide display for overlaying on the medical image based on the segment information generated by the segment information generation unit 12 (S30). The display generation unit 15 overlays the generated guide display on the medical image input from the video processor 20 (S40). Hereinafter, a detailed description will be provided.

[0049] The segment information generation unit 12 estimates the positions of two or more anatomical landmarks based on the medical image input from the video processor 20. Using a learning model read from the learning model holding unit 14, the segment information generation unit 12 detects the positions of two or more anatomical landmarks from the input medical image. Based on the detected positions of the two or more anatomical landmarks (specifically, the relative positions of the two or more anatomical landmarks to each other), the segment information generation unit 12 generates segment information. Segment information defines the safety level of the area reflected in the medical image during the implementation of a medical procedure.

[0050] The display generation unit 15 can generate line graphics representing the boundaries between multiple segments defined by segment information, which are then overlaid on the medical image as a guide display. The display generation unit 15 overlays the generated line graphics as an OSD (On Screen Display) onto the medical image.

[0051] Figure 7 This is an example of an image obtained by overlaying the line pattern L1 on a laparoscopic image taken during laparoscopic cholecystectomy. Figure 7 In the example shown, the segment information generation unit 12 detects Rouviere groove RS and the lower edge S4b of liver S4 as anatomical landmarks from the laparoscopic image reflecting the gallbladder GB. Alternatively, the hilar plate or umbilicial fissure located near the lower edge S4b of liver S4 can be used instead of the lower edge S4b of liver S4.

[0052] During laparoscopic cholecystectomy, the cystic duct (CD), which connects to the gallbladder (GB), needs to be removed before it merges with the common hepatic duct (the duct formed by the confluence of the right and left hepatic ducts) that connects to the liver. If the cystic duct (CD) is mistakenly removed from the common bile duct (CBD) after its confluence with the common hepatic duct, bile cannot flow from the liver to the duodenum (DU).

[0053] The segment information generation unit 12 generates a line (hereinafter appropriately referred to as the R4U line) that passes through the detected Rouviere groove RS and the lower edge S4b of liver S4. Segments above the R4U line are designated as safe area segments, and segments below the R4U line are designated as unsafe area segments. The display generation unit 15 generates a line graph L1 representing the boundary (R4U line) between the safe and unsafe area segments and overlays it onto the laparoscopic image.

[0054] The display generation unit 15 can generate a surface graphic representing a region of a segment defined by segment information, which is then overlaid on the medical image as a guide display. The display generation unit 15 overlays the generated surface graphic as an OSD onto the medical image.

[0055] Figure 8 This is an example of an image obtained by overlaying an unsafe region pattern (USZ) onto a laparoscopic image taken during laparoscopic cholecystectomy. Figure 9 This is an example of an image obtained by overlaying a safety region pattern SZ onto a laparoscopic image taken during laparoscopic cholecystectomy.

[0056] exist Figure 8 In the example shown, the generation unit 15 generates an unsafe region graphic USZ on a segment lower than the R4U line and overlays it onto the laparoscopic image. Alternatively, the unsafe region graphic USZ and the line graphic L1 can be overlaid on the laparoscopic image. The unsafe region graphic USZ is preferably generated using a color that draws the physician's attention (e.g., red). Figure 8 In the design, the unsafe area pattern (USZ) is formed by rectangular faces, but the upper edge (top edge) of the face can follow the R4U line. It can also be formed by parallelogram faces or faces containing curves. The size of the unsafe area pattern (USZ) is set to at least cover the size of the common bile duct (CBD).

[0057] exist Figure 9 In the example shown, the display generator 15 generates a safety zone graphic SZ on the segment above the R4U line and overlays it onto the laparoscopic image. Note that the safety zone graphic SZ and the line graphic L1 can both be overlaid on the laparoscopic image. The safety zone graphic SZ is preferably generated using a color (e.g., green) that reminds the physician of the recommended area where treatment should be performed. Figure 9 In the diagram, the safe area shape SZ is formed by a rectangular face, but the lower edge (bottom edge) of the face can be along the R4U line. It can also be formed by a parallelogram face or a face containing a curve.

[0058] In addition, when the unsafe area graphic USZ and the safe area graphic SZ are displayed in different colors, the display generation unit 15 can also overlay the unsafe area graphic USZ and the safe area graphic SZ on the laparoscopic image.

[0059] Figure 10 This is another example of an image obtained by overlaying a guided view onto a laparoscopic image taken during laparoscopic cholecystectomy. The surgeon performs the cholecystectomy in the area above the line graph L1, representing the R4U line, while simultaneously observing the laparoscopic image.

[0060] Figure 1The device recognition unit 13 shown identifies the device 3 based on medical images. The device recognition unit 13 detects the device 3 from the laparoscopic images by comparing them with template images of the device 3 taken during laparoscopic surgery. As template images, multiple images are prepared for each device, each with different orientations, protrusion lengths, and open / closed states. Furthermore, for devices with asymmetrical shapes whose shapes change with rotation in the images, multiple images with different rotation angles are prepared.

[0061] The instrument recognition unit 13 generates an edge image that emphasizes the edges of the laparoscopic image, and detects line segment shapes from the edge image using template matching, Hough transform, etc. The instrument recognition unit 13 compares the detected line segment shapes with the template image, and selects the instrument with the highest consistency with the template image as the detection result. Alternatively, pattern detection algorithms utilizing features such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) can be used to identify the instrument 3. Furthermore, the instrument 3 can also be identified using a machine learning model obtained by annotating the position (edges, display area) of the instrument.

[0062] The display generation unit 15 generates different guidance displays based on the relative positional relationship between the treatment device 3 identified by the treatment device identification unit 13 and the segment defined by the segment information. The display generation unit 15 generates alarm-like guidance displays based on the identified relative positional relationship between the treatment device 3 and the unsafe area segment.

[0063] For example, if the protrusion at the front end of the identified treatment device 3 is contained within the unsafe area segment, the display generation unit 15 generates a more emphasized (e.g., a darker red) unsafe area graphic USZ as an alarm-like guidance display.

[0064] exist Figures 7-9 In the example shown, scissors 3a and gripping forceps 3b are detected from the laparoscopic image. When the protrusion at the tip of the scissors 3a is contained within the unsafe area segment, the display generation unit 15 generates a more emphasized unsafe area graphic USZ.

[0065] Furthermore, when the protrusion at the front end of the handling device 3 is near the R4U line, the alarm frequently switches between on and off. As a countermeasure, an R4U area can be created by extending the R4U line in the width direction, designating the R4U area as a non-sensitive area. The display generation unit 15 stops the alarm from on / off while the protrusion at the front end of the handling device 3 is within the R4U area. The display generation unit 15 turns off the alarm when the protrusion at the front end of the handling device 3 moves out of the R4U area into a safe zone, and turns on the alarm when it moves out of the unsafe zone.

[0066] Furthermore, the alarm display based on the identification of the treatment device 3 is not a necessary function and can be omitted. In this case, the treatment device identification unit 13 of the medical assistance device 10 can be omitted. Additionally, the display generation unit 15 may not overlay the guidance display on the laparoscopic image under normal conditions, but may overlay an alarm-like guidance display when an alarm state is predicted or is about to occur. In this case, the guidance display only appears when the treatment device 3 approaches an unsafe area.

[0067] The segment information generation unit 12 can extract two or more feature points from medical images, in addition to the aforementioned two or more anatomical landmarks. The segment information generation unit 12 detects feature quantities such as SIFT and SURF, and extracts feature points from the medical images. The feature points can be any points that the segment information generation unit 12 can track, such as patient-specific scars or points with distinctive colors.

[0068] The segment information generation unit 12 generates segment information based on the positions of two or more anatomical landmarks and two or more feature points. The segment information generation unit 12 defines a specific segment based on the relative positional relationship of these four or more detection objects.

[0069] Figure 11 This image illustrates an example of four objects examined within a laparoscopic image taken during laparoscopic cholecystectomy. The four objects are Rouviere groove (RS), the lower border of liver S4 (S4b), the first feature point (FA), and the second feature point (FB).

[0070] When the detection of at least one anatomical landmark is interrupted, the segment information generation unit 12 generates segment information after the interruption of the detection of at least one anatomical landmark based on the positions of at least two feature points. The display generation unit 15 changes the guided display superimposed on the medical image according to the segment information after the interruption of the detection of at least one anatomical landmark. That is, even if at least one anatomical landmark is deviated from the field of view of the laparoscope 2 due to magnification or movement of the laparoscope 2, the segment information generation unit 12 can estimate the relative positional relationship between two or more anatomical landmarks based on the positions of at least two feature points.

[0071] Figure 12 It shows that Figure 11 The image shown is a magnified version of the laparoscopic image. Figure 12 In the laparoscopic image shown, the Rouviere groove (RS), which is one of the anatomical landmarks, deviates from the field of view of the laparoscope 2. In this case, the segment information generation unit 12 can also generate the R4U line based on the relative positional relationship of the lower edge S4b of the liver S4, the first feature point FA, and the second feature point FB.

[0072] When the field of view of the laparoscopy 2 changes, the segment information generation unit 12 can create an R4U line by tracking the movement of at least two feature points instead of their relative positional relationship. In this case, the segment information generation unit 12 uses the KLT (Kanade-Lucas-Tomasi Feature Tracker) method, Mean-Shift search, etc., to track the position of two or more feature points extracted from the medical image.

[0073] When the detection of at least one anatomical landmark is interrupted, the segment information generation unit 12 estimates the positional change of the segment defined by the segment information after the detection of at least one anatomical landmark is interrupted, based on the positional changes of at least two feature points. The display generation unit 15 adjusts the positional change of the guide display superimposed on the medical image according to the positional change of the segment after the detection of at least one anatomical landmark is interrupted.

[0074] When using a laparoscopic surgery assistive robot, the medical assist device 10 can obtain position change information (hereinafter referred to as motion information) of the laparoscope 2 mounted on the robot arm based on the mobility of each drive joint of the robot arm. In this case, the segment information generation unit 12 obtains the motion information of the laparoscope 2 from the robot system when the detection of at least one anatomical landmark is interrupted. Based on the obtained motion information of the laparoscope 2, the segment information generation unit 12 estimates the position change of the segment defined by the segment information after the detection of at least one anatomical landmark is interrupted. The display generation unit 15 adjusts the position change of the guide display superimposed on the medical image based on the position change of the segment after the detection of at least one anatomical landmark is interrupted.

[0075] Figure 13 This diagram illustrates an example of the change in a laparoscopic image caused by the displacement of the laparoscope 2. The field of view F1 of the laparoscope 2 changes to the field of view F1' due to the displacement of the laparoscope 2 caused by the robotic arm. In the field of view F1', the Rouviere groove RS, which is one of the anatomical landmarks, is not visible. The segment information generation unit 12 determines the positional change of the R4U line based on the motion information MV of the laparoscope 2. As a result, the display generation unit 15 enables the line graph L1 based on the R4U line to follow the displacement of the laparoscope 2.

[0076] Without using a laparoscopic surgical robot, the movement of the laparoscope 2 is detected by tracking specific markers within the laparoscopic image using image processing. The segment information generation unit 12 can also use the motion information of the laparoscope 2 detected by this image processing to estimate the displacement of segments within the laparoscopic image. Furthermore, when using a laparoscopic surgical robot, more accurate motion and manipulation information of the laparoscope 2 can be obtained, thus allowing for a more accurate estimation of the positions of anatomical landmarks outside the field of view.

[0077] When multiple feature points are detected from the laparoscopic image, the desired segment information generation unit 12 selects feature points located along or near the R4U line. If two feature points can be detected along the R4U line inside two anatomical landmarks, the R4U line can be easily created even when anatomical landmarks are not visible from the field of view of the laparoscopy 2 due to magnification operations by the physician. Furthermore, tracking the movement of the laparoscopy 2 can improve the estimation accuracy of the R4U line. The positions of anatomical landmarks located outside the field of view (e.g., Rouviere groove RS, lower edge S4b of liver S4) can also be easily estimated.

[0078] The segment information generation unit 12 can track the movement of tissue mapped in a medical image. This tissue can also be tissue from a resected object (e.g., the cystic duct). If the resected object's tissue has been learned in a learning model, the segment information generation unit 12 uses that model to detect the resected object's tissue. If the resected object's tissue is registered in other dictionary data, the segment information generation unit 12 uses that dictionary data to detect the resected object's tissue.

[0079] The segment information generation unit 12 can track the detected tissue of the excised object through motion tracking. The display generation unit 15 generates a guided display that follows the movement of the tissue based on the tracking results of the tissue's movement.

[0080] Figures 14A to 14B This is an example of an image obtained by overlaying a resection region Acd, representing the position of the cystic duct CD to be removed, onto a laparoscopic image taken during laparoscopic cholecystectomy. The display generation unit 15 changes the position of the resection region Acd, representing the position of the cystic duct CD, according to the movement of the cystic duct CD in the laparoscopic image. The resection region Acd is, for example, overlaid on the laparoscopic image with a green marker.

[0081] Furthermore, the segment information generation unit 12 can also track the movement of two or more detected anatomical landmarks through motion tracking. Although in Figures 14A-14BAlthough not shown in the diagram, the display generation unit 15 can change the guided display (e.g., line graph L1) based on the tracking results of the movement of the two or more anatomical landmarks.

[0082] As a guide display to be overlaid on a medical image, the display generation unit 15 can generate multiple line graphics that represent the boundaries between multiple segments defined by segment information in stages. The display generation unit 15 overlays the generated multiple line graphics onto the medical image. These multiple line graphics can also be three or more line graphics located at different positions from each other, each linked by two anatomical landmarks. Preferably, these three or more line graphics are generated with different colors from each other.

[0083] Figure 15 This is an example of an image obtained by overlaying three lines L1a to L1c onto a laparoscopic image taken during laparoscopic cholecystectomy. Figure 15 In the example shown, the segment information generation unit 12 detects Rouviere groove RS and the lower edge S4b of liver S4 as anatomical landmarks. The display generation unit 15 overlaps different colored tiles on Rouviere groove RS and the lower edge S4b of liver S4, respectively.

[0084] exist Figure 15 In the example shown, the display generation unit 15 superimposes a green first line pattern L1a passing through the upper end of the Rouviere groove RS and the upper end of the lower edge S4b of the liver S4, a yellow third line pattern L1c passing through the lower end of the Rouviere groove RS and the lower edge S4b of the liver S4, and a yellow-green second line pattern L1b passing between the first line pattern L1a and the third line pattern L1c onto the laparoscopic image.

[0085] Figure 16 This is another example of an image obtained by overlaying three lines L1a to L1c onto a laparoscopic image taken during laparoscopic cholecystectomy. Figure 15 The example shown illustrates three line graphs, L1a to L1c, formed by straight lines. However, as... Figure 16 As shown, the three-line graphs L1a to L1c can also be formed by curves.

[0086] exist Figures 15-16 In the example shown, the portion above the first line graphic L1a is the first segment (safe zone segment), and the portion below the third line graphic L1c is the second segment (unsafe zone segment), which is less safe than the first segment. The first and second segments are divided in stages by the three line graphics L1a to L1c.

[0087] The display generation unit 15 can also generate multiple line graphics in the boundary area between the first segment and the second segment, which are animated graphics that move from the second segment to the first segment as a guide display.

[0088] Figure 17 This is an example of an image obtained by overlaying animated graphics onto laparoscopic images taken during laparoscopic cholecystectomy. Although in Figure 17 Although not shown in the diagram, the segment information generation unit 12 detects the Rouviere groove RS and umbilical fissure as anatomical landmarks.

[0089] exist Figure 17 In the example shown, the display generation unit 15 sequentially overlays three lines—red (Lr), yellow (Ly), and green (Lg)—on the laparoscopic image, starting from the bottom, in the boundary region between the first and second segments. The three lines move from the bottom to the top of the boundary region. Specifically, the red line Lr emerges from the bottom of the boundary region and changes color from red to yellow to green as it moves towards the top, disappearing when it reaches the top of the boundary region. Alternatively, four or more lines can be displayed in the boundary region.

[0090] By displaying animated graphics of multiple lines moving like waves in the boundary area in the direction from the unsafe area to the safe area, doctors can be visually guided to leave from the unsafe area to the safe area.

[0091] In the embodiments described above, if the segment information generation unit 12 cannot detect the R4U line, the display generation unit 15 can also display an alarm message on the monitor 30 to draw the attention of the doctor in the process. Using this alarm message as an opportunity, a less experienced doctor can consult a more experienced doctor, or the experienced doctor can take over the treatment, helping to minimize the risk of damage to the common bile duct.

[0092] As described above, according to this embodiment, when performing laparoscopic surgery, the surgeon can fully identify which parts are safer and which are less safe. The surgeon can safely perform laparoscopic surgery by performing the procedure on the first segment. By overlaying the various guiding displays described above onto the laparoscopic image, it is possible to prevent the common bile duct from being mistakenly removed during laparoscopic cholecystectomy, and to facilitate the removal of the cystic duct and cystic artery in the appropriate location. For example, by displaying the R4U line as a guiding display, even inexperienced surgeons can always be aware of the R4U line and perform the surgery at a position higher than the R4U line.

[0093] The present disclosure has been described above based on several embodiments. These embodiments are illustrative, and those skilled in the art will understand that various modifications can be made to the combination of these constituent elements and processes, and such modifications are also within the scope of this disclosure.

[0094] In the above embodiments, a machine learning model is used to detect anatomical landmarks from the captured laparoscopic images. Alternatively, features such as SIFT, SURF, edges, and corners can be detected based on the captured laparoscopic images, and anatomical landmarks can be detected based on the feature descriptions of the laparoscopic images.

[0095] Furthermore, when a machine learning model for safe region detection is prepared, as in the challenging situations described above, segment information can be generated directly without the detection of anatomical landmarks. In this case, even when anatomical landmarks are not reflected in the laparoscopic image, safe and unsafe regions within the laparoscopic image can be identified.

[0096] Industrial availability

[0097] This disclosure can be used in the field of displaying laparoscopic images.

Claims

1. A medical assistive device, wherein, This medical assistive device has more than one processor. The processor is configured as follows: Obtain medical images, Based on the medical image, a guided display is generated for overlay on the medical image. This guided display represents the boundaries between segments in the medical image that indicate different levels of recommendation for medical procedures. The processor generates multiple line graphics that represent the boundaries between the segments in stages as the guidance display. In addition, in the boundary region between the first segment and the second segment, whose recommendation for medical treatment is lower than that of the first segment, the processor generates a movement animation in which the positions of the multiple line graphics are changed sequentially from the second segment toward the first segment as the guidance display.

2. The medical auxiliary device according to claim 1, wherein, The processor estimates the locations of more than two anatomical landmarks based on the medical images. The processor generates the guidance display based on the estimated positions of the two or more anatomical landmarks.

3. The medical auxiliary device according to claim 2, wherein, The processor acquires the medical image containing the gallbladder. Of the two or more anatomical landmarks mentioned, at least one is included: Rouviere groove, lower border of liver S4, hilar plate, hepatic artery, and umbilical fissure.

4. The medical auxiliary device according to claim 1, wherein, The processor generates at least one of a line graph and a surface as the guide display, wherein the line graph represents the boundary between the segments, the surface represents the region of a segment, and the edge of the surface represents the boundary between the segments.

5. The medical auxiliary device according to claim 2, wherein, The processor generates the guide display based on the relative positions of the two or more anatomical landmarks.

6. The medical auxiliary device according to claim 1, wherein... The processor identifies the treatment device from the medical image. The processor changes the presence or display mode of the guidance display based on the relative position of the treatment device and the segments with different levels of recommendation for the medical treatment.

7. The medical auxiliary device according to claim 6, wherein, The processor generates the guidance display as an alarm based on the relative position of the treatment device and the segments with different levels of recommendation for the medical treatment.

8. The medical auxiliary device according to claim 2, wherein, The processor also extracts more than two feature points from the medical image. The processor generates the guided display based on the positions of the two or more anatomical landmarks and the positions of the two or more feature points.

9. The medical auxiliary device according to claim 8, wherein, When the detection of at least one of the anatomical landmarks is interrupted, the processor causes the guidance display to change according to the positions of at least two or more feature points after the detection of at least one of the anatomical landmarks is interrupted.

10. The medical auxiliary device according to claim 2, wherein, The processor extracts two or more feature points from the medical image, tracks the positional changes of the two or more feature points, and when the detection of at least one of the anatomical landmarks is interrupted, estimates the positional changes of segments indicating different levels of recommendation for medical treatment after the interruption of the detection of at least one of the anatomical landmarks, based on the positional changes of the two or more feature points. The processor changes the position of the guidance display based on the varying degrees of recommendation for medical treatment following the detection of at least one of the anatomical landmarks.

11. The medical auxiliary device according to claim 2, wherein, The processor acquires the medical images captured by the endoscope. When the detection of at least one of the anatomical landmarks is interrupted, the processor acquires the motion information of the endoscope. The processor changes the position of the guide display based on the motion information of the endoscope after the detection of at least one of the anatomical landmarks.

12. The medical auxiliary device according to claim 1, wherein, The processor tracks the movement of tissues reflected in the medical images. The processor generates a guided display that follows the movement of the tissue based on the tracking results of the tissue's movement.

13. The medical auxiliary device according to claim 1, wherein, The processor displays the medical image, which is superimposed on the guidance display, on a monitor connected to the medical assistive device.

14. The medical auxiliary device according to claim 4, wherein, The processor acquires the medical image showing the gallbladder or surrounding tissue. The edge of the line graphic or the surface passes through the Rouviere groove and the lower edge of liver S4 reflected in the medical image.

15. The medical auxiliary device according to claim 4, wherein, The processor acquires the medical image showing the gallbladder or surrounding tissue. The edge of the line graphic or the surface is reflected in the medical image through the Rouviere groove and umbilical fissure.

16. The medical auxiliary device according to claim 1, wherein, The multiple line figures are three or more line figures that are each connected by two anatomical landmarks and located in different positions from each other.

17. The medical auxiliary device according to claim 16, wherein, The colors of the three or more line graphics are different from each other.

18. The medical auxiliary device according to claim 1, wherein, The medical image shows the gallbladder or surrounding tissue.

Images