Medical support device and method for operating medical support device

Abstract

To cause a medical doctor to recognize an area whose recommendation level for medical treatment is high, in execution of a surgery.SOLUTION: A medical support apparatus has one processor or more, and the processor is configured to: acquire a medical image; and based on the medical image, generate a guidance display that indicates a boundary between segments whose recommendation levels for medical treatment are different in the medical image and that is to be superimposed on the medical image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a medical support device and a medical support for supporting medical procedures such as laparoscopic surgery. Operation of the device It concerns the method. [Background technology]

[0002] Today, surgery still relies heavily on a surgeon's knowledge of anatomy, and this is even more so during laparoscopic surgery, where doctors rely solely on vision without tactile feedback to understand the structures inside a patient's body.

[0003] Laparoscopic cholecystectomy is a surgical procedure aimed at removing the gallbladder. To do so, it is necessary to identify two structures, the cystic duct and the cystic artery, and then sequentially resect them. In difficult anatomical situations, physicians must be careful not to misidentify anatomical structures. In particular, physicians must be careful not to mistake the common bile duct for the cystic duct. Laparoscopic cholecystectomy is one of the procedures performed by physicians with little experience after training, and may be performed by physicians who do not yet have sufficient anatomical knowledge or experience with common anatomical variations.

[0004] In order to compensate for the lack of anatomical knowledge and experience of doctors, it is conceivable to use a surgery support system to guide doctors to the recommended treatment area. For example, a surgery support system has been proposed that supports the doctor's procedure by superimposing a virtual image of the resection surface on a laparoscopic image during laparoscopic surgery (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2005-278888 A Summary of the Invention [Problem to be solved by the invention]

[0006] As mentioned above, in laparoscopic cholecystectomy, there is a need to further improve existing surgical support systems to compensate for doctors' lack of anatomical knowledge and experience.

[0007] The present disclosure has been made in consideration of these circumstances, and has as its objective to provide a technology that allows a doctor to recognize areas where treatment is highly recommended when performing surgery. [Means for solving the problem]

[0008] In order to solve the above problems, a medical support device of one embodiment of the present disclosure has one or more processors, and the processors are configured to acquire a medical image and generate, based on the medical image, a guidance display for superimposing on the medical image, indicating boundaries between segments in the medical image that differ in the degree to which medical treatment is recommended.

[0009] A medical support method of another aspect of the present disclosure comprises acquiring a medical image and generating, based on the medical image, a guidance display for superimposing on the medical image, the guidance display indicating boundaries between segments in the medical image that differ in the degree to which medical treatment is recommended.

[0010] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc. are also valid aspects of the present disclosure. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a configuration of a medical support system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an overall flow of laparoscopic cholecystectomy. [Diagram 3] FIG. 1 is a diagram for explaining a method for generating a machine learning model according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of a laparoscopic image showing B-SAFE landmarks. [Diagram 5] FIG. 13 is a diagram showing an example of a laparoscopic image showing the Rouvierre groove and the hepatic hilar plate. [Figure 6] 4 is a flowchart showing a basic operation of the medical support device according to the embodiment. [Figure 7] FIG. 13 is a diagram showing an example of an image in which a line graphic is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. [Figure 8] FIG. 13 shows an example of an image in which an unsafe zone graphic is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. [Figure 9] FIG. 13 shows an example of a laparoscopic image taken during a laparoscopic cholecystectomy with a safe zone graphic superimposed thereon. [Figure 10] FIG. 13 is a diagram showing another example of an image in which a guidance display is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. [Figure 11] FIG. 1 shows an example of four detection targets in a laparoscopic image taken during a laparoscopic cholecystectomy. [Figure 12] FIG. 12 is a diagram showing a laparoscopic image obtained by zooming in on the laparoscopic image shown in FIG. 11. [Figure 13] FIG. 13 is a diagram showing an example of a change in a laparoscopic image due to a displacement of a laparoscope. [Figure 14] 14(A)-(B) are diagrams showing an example of an image in which a resection area indicating the position of the cystic duct to be resected is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. [Figure 15] FIG. 13 is a diagram showing an example of an image in which three line graphics are superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. [Figure 16] FIG. 13 shows another example of a laparoscopic image taken during a laparoscopic cholecystectomy with three line graphics superimposed thereon. [Figure 17] FIG. 13 shows an example of an image in which animated graphics are superimposed on a laparoscopic image taken during a laparoscopic cholecystectomy. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 shows the configuration of a medical support system 1 according to an embodiment. The medical support system 1 is a system for supporting laparoscopic surgery used in the surgical department. In laparoscopic surgery, an endoscope for abdominal surgery (hereinafter referred to as laparoscope 2) and treatment tool 3 are inserted through several holes made in the patient's abdomen. The laparoscope 2 is not a flexible scope used for examinations of the stomach or large intestine, but is composed of a rigid metal scope. The treatment tool 3 may be a forceps, a trocar, an energy device, or the like.

[0013] The medical support system 1 is installed in an operating room, and includes a medical support device 10, a video processor 20, and a monitor 30. The medical support device 10 includes a medical image acquisition unit 11, a segment information generation unit 12, a treatment tool recognition unit 13, a learning model storage unit 14, and a display generation unit 15.

[0014] The medical support device 10 can be realized in hardware by any processor (e.g., CPU, GPU), memory, auxiliary storage device (e.g., HDD, SSD), or other LSI, and in software by a program loaded into memory, but here, functional blocks realized by cooperation of these are illustrated. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of these.

[0015] The laparoscope 2 has a light guide for transmitting illumination light supplied from a light source device to illuminate the inside of the patient's body, and at its tip, an illumination window for emitting the illumination light transmitted by the light guide to a subject, and an imaging unit for capturing images of the subject at a predetermined period and outputting the image capture signal to the video processor 20. The imaging unit includes a solid-state imaging element (e.g., a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

[0016] The video processor 20 generates a laparoscopic image by performing image processing on the imaging signal photoelectrically converted by the solid-state imaging element of the laparoscope 2. The video processor 20 can also perform effect processing such as highlighting in addition to normal image processing such as A / D conversion and noise removal.

[0017] The laparoscopic image generated by the video processor 20 is output to the medical support device 10, and after a guidance display is superimposed by the medical support device 10, it is displayed on the monitor 30. Note that as a failback function, the laparoscopic image can be output directly from the video processor 20 to the monitor 30, bypassing the medical support device 10. Also, although FIG. 1 shows a configuration in which the medical support device 10 and the video processor 20 are separated into separate devices, the medical support device 10 and the video processor 20 may be integrated into a single device.

[0018] Figure 2 is a diagram showing the overall flow of laparoscopic cholecystectomy. First, equipment is prepared in the operating room (P1). In this process, the medical support system 1 is started. Next, the patient is admitted to the operating room (P2). Next, the anesthesiologist administers general anesthesia to the patient (P3). Next, a time-out is carried out by the entire surgical team (P4). Specifically, the team confirms the patient, the surgical site, the surgical procedure, etc. Next, the surgeon sets up an access port to the abdominal cavity (P5). Specifically, a small hole is made in the abdomen and a trocar is inserted to set up the access port. A laparoscope, forceps, and energy device are inserted into the abdominal cavity via the tracker. Carbon dioxide is sent into the abdominal cavity from the insufflation device to fill it with gas, securing a surgical working space.

[0019] Next, the surgeon decompresses the gallbladder (P6) and pulls the gallbladder with a grasping forceps (P7). Next, the surgeon dissects the peritoneum covering the gallbladder with a dissector forceps (P8). Next, the surgeon exposes the cystic duct and cystic artery (P9) and exposes the gallbladder bed (P10). Next, the surgeon secures the cystic duct and cystic artery with clips (P11) and cuts the cystic duct and cystic artery with a scissor forceps (P12). Finally, the surgeon dissects the gallbladder from the gallbladder bed with a dissector forceps and retrieves the gallbladder by placing it in a retrieval bag.

[0020] The medical support device 10 according to the embodiment superimposes a guidance display on a laparoscopic image displayed in real time on the monitor 30 in steps P8 to P11, thereby supporting the surgeon in checking a safety zone during the procedure.

[0021] The medical support system 1 according to the embodiment uses a machine learning model that is trained to recognize important anatomical landmarks from live images during a procedure.

[0022] FIG. 3 is a diagram for explaining a method for generating a machine learning model according to an embodiment. In a simple case, an annotator with specialized knowledge, such as a doctor, annotates anatomical landmarks shown in a large number of laparoscopic images. The AI / machine learning system performs machine learning using a supervised data set of laparoscopic images with annotated anatomical landmarks as training data, and generates a machine learning model for landmark detection. For example, CNN, RNN, LSTM, etc., which are types of deep learning, can be used as the machine learning.

[0023] In the challenge case, a machine learning model is generated to detect safe zones in laparoscopic images of complex cases that do not show anatomical landmarks. Annotators with advanced expertise, such as experienced physicians, annotate safe zones in a large number of laparoscopic images that do not show anatomical landmarks. For example, safe zone lines are drawn in the laparoscopic images. In addition, annotations may be added to estimated positions of anatomical landmarks that are not shown in the laparoscopic images because they are covered by fatty tissue, etc. The AI / machine learning system performs machine learning on a supervised dataset of laparoscopic images with safe zone annotations as training data, and generates a machine learning model for safe zone detection.

[0024] The machine learning model for detecting at least one of the anatomical landmarks and the safe zone generated as described above is registered in the learning model holding unit 14 of the medical support device 10.

[0025] In laparoscopic images showing the gallbladder or the tissues surrounding the gallbladder taken during laparoscopic cholecystectomy, the B-SAFE landmarks can be used as anatomical landmarks. In the B-SAFE landmarks, the bile duct (B), Rouvierre's groove (S), the lower edge of the liver S4 (S), the hepatic artery (A), the umbilical fissure (F), and the intestinal structure (duodenum) (E) are used as landmarks. In addition, the Hilar Plate can also be used as an anatomical landmark. FIG. 4 is a diagram showing an example of a laparoscopic image showing the B-SAFE landmarks. FIG. 5 is a diagram showing an example of a laparoscopic image showing the Rouvierre's groove and the hilar plate. Note that a laparoscopic image showing the tissues surrounding the gallbladder refers to a laparoscopic image showing at least one of the above-mentioned B-SAFE landmarks, the cystic duct, and the cystic artery.

[0026] FIG. 6 is a flowchart showing the basic operation of the medical support device 10 according to the embodiment. The medical image acquisition unit 11 acquires a medical image (in this embodiment, a laparoscopic image showing the gallbladder or a laparoscopic image showing the tissues around the gallbladder) from the video processor 20 (S10). The segment information generation unit 12 generates segment information that defines the degree of safety of a medical procedure (in this embodiment, laparoscopic cholecystectomy) in terms of regions based on the medical image acquired by the medical image acquisition unit 11 (S20). The display generation unit 15 generates a guidance display to be superimposed on the medical image based on the segment information generated by the segment information generation unit 12 (S30). The display generation unit 15 superimposes the generated guidance display on the medical image input from the video processor 20 (S40). A specific description will be given below.

[0027] The segment information generating unit 12 estimates the positions of two or more anatomical landmarks based on the medical image input from the video processor 20. The segment information generating unit 12 detects the positions of two or more anatomical landmarks from the input medical image using a learning model read from the learning model holding unit 14. The segment information generating unit 12 generates segment information based on the positions of the two or more detected anatomical landmarks (specifically, the relative positions of the two or more anatomical landmarks). The segment information is information that defines the safety level of an area shown in the medical image when performing a medical procedure within the medical image.

[0028] The display generation unit 15 can generate a line graphic indicating boundaries between a plurality of segments defined by the segment information as a guidance display to be superimposed on the medical image. The display generation unit 15 superimposes the generated line graphic on the medical image as an OSD (On Screen Display).

[0029] Fig. 7 is a diagram showing an example of an image in which a line graphic L1 is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. In the example shown in Fig. 7, the segment information generating unit 12 detects the Rubiere groove RS and the lower edge S4b of the liver S4 as anatomical landmarks from the laparoscopic image showing the gallbladder GB. Note that instead of the lower edge S4b of the liver S4, the hilar plate or umbilical fissure located near the lower edge S4b of the liver S4 may be used.

[0030] In laparoscopic cholecystectomy, the cystic duct CD, which connects to the gallbladder GB, must be resected before it joins with the common hepatic duct (the hepatic duct after the right hepatic duct RHD and the left hepatic duct join) which connects to the liver. If the common bile duct CBD after the cystic duct CD and the common hepatic duct join is mistakenly resected, bile will not be able to flow from the liver to the duodenum DU.

[0031] The segment information generating unit 12 generates a line (hereinafter referred to as the R4U line) that passes through the detected Rubiere's groove RS and the lower edge S4b of the liver S4, and sets the segments above the R4U line as safe zone segments and the segments below the R4U line as unsafe zone segments. The display generating unit 15 generates a line graphic L1 that indicates the boundary (R4U line) between the safe zone segment and the unsafe zone segment, and superimposes it on the laparoscopic image.

[0032] The display generating unit 15 can generate a plate graphic indicating the area of ​​one segment defined by the segment information as a guidance display to be superimposed on the medical image. The display generating unit 15 superimposes the generated plate graphic on the medical image as an OSD.

[0033] Fig. 8 is a diagram showing an example of an image in which an unsafe zone graphic USZ is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. Fig. 9 is a diagram showing an example of an image in which a safe zone graphic SZ is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy.

[0034] In the example shown in FIG. 8, the display generating unit 15 generates an unsafe zone graphic USZ in the segment below the R4U line and superimposes it on the laparoscopic image. Both the unsafe zone graphic USZ and the line graphic L1 may be superimposed on the laparoscopic image. It is preferable that the unsafe zone graphic USZ is generated in a color (e.g., red) that attracts the doctor's attention. In FIG. 8, the unsafe zone graphic USZ is formed of a rectangular plate, but it is sufficient that the upper edge (upper side) of the plate is along the R4U line, and it may be formed of a parallelogram plate or a plate including a curve. The size of the unsafe zone graphic USZ is set to a size that covers at least the common bile duct CBD.

[0035] In the example shown in FIG. 9, the display generating unit 15 generates a safe zone graphic SZ in the segment above the R4U line and superimposes it on the laparoscopic image. Both the safe zone graphic SZ and the line graphic L1 may be superimposed on the laparoscopic image. It is desirable that the safe zone graphic SZ is generated in a color (e.g., green) that alerts the doctor that this is a recommended zone where treatment should be performed. In FIG. 9, the safe zone graphic SZ is formed of a rectangular plate, but as long as the lower edge (lower side) of the plate is aligned with the R4U line, it may be formed of a parallelogram plate or a plate including a curve.

[0036] When the unsafe zone graphic USZ and the safe zone graphic SZ are displayed in different colors, the display generating unit 15 may superimpose both the unsafe zone graphic USZ and the safe zone graphic SZ on the laparoscopic image.

[0037] 10 is a diagram showing another example of an image in which a guidance display is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. While viewing the laparoscopic image, the doctor performs the procedure for cholecystectomy in the area above the line graphic L1 indicating the R4U line.

[0038] The treatment tool recognition unit 13 shown in Fig. 1 recognizes the treatment tool 3 from a medical image. The treatment tool recognition unit 13 detects the treatment tool 3 from the laparoscopic image by comparing a laparoscopic image taken during laparoscopic surgery with a template image of the treatment tool 3. As template images, multiple images with different directions, protrusion lengths, and open / closed states are prepared for each treatment tool. In addition, for treatment tools with asymmetric shapes whose shape on the image changes when rotated, multiple images with different rotation angles are prepared.

[0039] The treatment tool recognition unit 13 generates an edge image in which the edges of the laparoscopic image are emphasized, and detects the shape of a line segment from within the edge image using template matching, Hough transformation, or the like. The treatment tool recognition unit 13 compares the detected line segment shape with a template image, and the treatment tool of the template image with the highest degree of match is determined as the detection result. Note that the treatment tool 3 may be recognized using a pattern detection algorithm using feature quantities such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed ​​Up Robust Features). Also, the treatment tool 3 may be recognized based on a learning model that is machine-learned by adding annotations to the position (edge, display area) of the treatment tool.

[0040] The display generation unit 15 generates different guidance displays according to the relative positional relationship between the treatment tool 3 recognized by the treatment tool recognition unit 13 and the segment defined by the segment information. The display generation unit 15 generates an alert-like guidance display according to the relative positional relationship between the recognized treatment tool 3 and the unsafe zone segment.

[0041] For example, if the recognized protruding portion of the tip of the treatment tool 3 is within the unsafe zone segment, the display generation unit 15 generates a more emphasized (e.g., darker red) unsafe zone graphic USZ as an alert-like guidance display.

[0042] 7 to 9, the scissors forceps 3a and the grasping forceps 3b are detected from the laparoscopic image. When the protruding portion of the tip of the scissors forceps 3a is within the unsafe zone segment, the display generating unit 15 generates a more emphasized unsafe zone graphic USZ.

[0043] In addition, when the protruding portion of the tip of the treatment tool 3 is located near the R4U line, the alert is frequently switched on and off. As a countermeasure, an R4U zone may be provided by widening the R4U line in the width direction, and the R4U zone may be set as a dead zone. The display generation unit 15 stops turning the alert on and off while the protruding portion of the tip of the treatment tool 3 is within the R4U zone. The display generation unit 15 turns off the alert when the protruding portion of the tip of the treatment tool 3 leaves the R4U zone and turns on the alert when the protruding portion leaves the R4U zone and enters the unsafe zone segment.

[0044] Note that the alert display based on the recognition of the treatment tool 3 is not a required function and can be omitted. In that case, the treatment tool recognition unit 13 of the medical support device 10 can be omitted. The display generation unit 15 may not superimpose a guidance display on the laparoscopic image in a normal state, but may superimpose an alert-like guidance display when an alert state occurs or is predicted to occur. In this case, the guidance display will not be displayed until the treatment tool 3 approaches the unsafe zone.

[0045] The segment information generating unit 12 can extract two or more feature points from the medical image in addition to the two or more anatomical landmarks. The segment information generating unit 12 detects feature quantities such as SIFT and SURF to extract feature points from the medical image. The feature points may be points that the segment information generating unit 12 can track, and may be scars unique to a patient or points with a special color.

[0046] The segment information generating unit 12 generates segment information based on the positions of two or more anatomical landmarks and the positions of two or more feature points. The segment information generating unit 12 defines a specific segment based on the relative positional relationship of these four or more detection targets.

[0047] 11 is a diagram showing an example of four detection targets in a laparoscopic image captured during laparoscopic cholecystectomy. The four detection targets are the Rubiere's groove RS, the lower edge S4b of the liver S4, the first feature point FA, and the second feature point FB.

[0048] When detection of at least one of the anatomical landmarks is interrupted, the segment information generating unit 12 generates segment information after detection of at least one of the anatomical landmarks is interrupted based on the positions of at least the two or more characteristic points. The display generating unit 15 changes the guidance display to be superimposed on the medical image based on the segment information after detection of at least one of the anatomical landmarks is interrupted. In other words, even if at least one of the anatomical landmarks falls outside the field of view of the laparoscope 2 due to zooming in or moving the laparoscope 2, the segment information generating unit 12 can estimate the relative positional relationship between the two or more anatomical landmarks based on the positions of at least the two or more characteristic points.

[0049] Fig. 12 is a diagram showing a laparoscopic image obtained by zooming in on the laparoscopic image shown in Fig. 11. In the laparoscopic image shown in Fig. 12, the Rubiere's groove RS, which is one of the anatomical landmarks, is outside the field of view of the laparoscope 2. Even in this case, the segment information generating unit 12 can create the R4U line from the relative positional relationship between the lower edge S4b of the liver S4, the first feature point FA, and the second feature point FB.

[0050] When the field of view of the laparoscope 2 changes, the segment information generating unit 12 can create an R4U line by tracking the movements of the two or more feature points, rather than the relative positional relationship between the two or more feature points. In this case, the segment information generating unit 12 tracks the positions of the two or more feature points extracted from the medical image using the Kanade-Lucas-Tomasi Feature Tracker (KLT) method, Mean-Shift search, or the like.

[0051] When detection of at least one of the anatomical landmarks is discontinued, the segment information generating unit 12 estimates a position change of the segment defined by the segment information after detection of at least one of the anatomical landmarks is discontinued based on the position change of at least the two or more feature points. The display generating unit 15 changes the position of the guidance display to be superimposed on the medical image based on the position change of the segment after detection of at least one of the anatomical landmarks is discontinued.

[0052] When a laparoscopic surgery support robot is used, the medical support device 10 can acquire position change information (hereinafter referred to as movement information) of the laparoscope 2 attached to the robot arm from the movable amount of each driving joint of the robot arm. In this case, the segment information generation unit 12 acquires the movement information of the laparoscope 2 from the robot system when the detection of at least one anatomical landmark is discontinued. Based on the acquired movement information of the laparoscope 2, the segment information generation unit 12 estimates the position change of the segment defined by the segment information after the detection of at least one anatomical landmark is discontinued. The display generation unit 15 changes the position of the guidance display to be superimposed on the medical image based on the position change of the segment after the detection of at least one anatomical landmark is discontinued.

[0053] FIG. 13 is a diagram showing an example of a change in a laparoscope image due to the displacement of the laparoscope 2. The field of view F1 of the laparoscope 2 changes to a field of view F1' due to the displacement of the laparoscope 2 by the robot arm. In the field of view F1', the Rubiere's groove RS, which is one of the anatomical landmarks, is no longer visible. The segment information generating unit 12 specifies a position change of the R4U line based on the movement information MV of the laparoscope 2. This enables the display generating unit 15 to cause the line graphic L1 based on the R4U line to follow the displacement of the laparoscope 2.

[0054] When a laparoscopic surgery robot is not used, the movement of the laparoscope 2 is detected by tracking specific markers in the laparoscopic image by image processing. The segment information generating unit 12 can also use the movement information of the laparoscope 2 detected by this image processing to estimate the displacement of the segment in the laparoscopic image. Note that when a laparoscopic surgery robot is used, more accurate movement and operation information of the laparoscope 2 can be obtained, so that the position of the anatomical landmark outside the field of view can be estimated more accurately.

[0055] When multiple feature points are detected from the laparoscopic image, the segment information generating unit 12 desirably selects feature points along the R4U line or close to the R4U line. If two feature points can be detected inside two anatomical landmarks along the R4U line, the R4U line can be easily created even if the anatomical landmarks are no longer visible from the field of view of the laparoscope 2 due to a zoom-in operation by the doctor. Furthermore, by tracking the movement of the laparoscope 2, the estimation accuracy of the R4U line can be improved. The positions of anatomical landmarks outside the field of view (for example, Rubiere's groove RS, the lower edge S4b of the liver S4) can also be easily estimated.

[0056] The segment information generating unit 12 can track the movement of tissue captured in a medical image. The tissue may be a tissue to be resected (e.g., a cystic duct). When the tissue to be resected is learned in a learning model, the segment information generating unit 12 uses the learning model to detect the tissue to be resected. When the tissue to be resected is registered as separate dictionary data, the segment information generating unit 12 uses the dictionary data to detect the tissue to be resected.

[0057] The segment information generating unit 12 can track the detected tissue to be resected by using a motion tracking function. The display generating unit 15 generates a guidance display that follows the movement of the tissue based on the tracking result of the movement of the tissue.

[0058] 14(A)-(B) are diagrams showing an example of an image in which a resection area Acd indicating the position of the cystic duct CD to be resected is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. The display generating unit 15 changes the position of the resection area Acd indicating the position of the cystic duct CD according to the movement of the cystic duct CD to be resected in the laparoscopic image. The resection area Acd is superimposed on the laparoscopic image, for example, with a green marker.

[0059] The segment information generating unit 12 can also track the movements of the two or more detected anatomical landmarks using a motion tracking function. Although not shown in Figures 14(A)-(B), the display generating unit 15 can change the guidance display (for example, the line graphic L1) based on the two or more anatomical landmarks based on the results of tracking the movements of the two or more anatomical landmarks.

[0060] The display generating unit 15 can generate a plurality of line graphics that indicate in stages the boundaries between a plurality of segments defined by the segment information as a guidance display to be superimposed on the medical image. The display generating unit 15 superimposes the generated plurality of line graphics on the medical image. The plurality of line graphics may be three or more line graphics that each pass through two anatomical landmarks and are located at different positions from each other. It is preferable that the three or more line graphics are generated in different colors from each other.

[0061] Fig. 15 is a diagram showing an example of an image in which three line graphics L1a-L1c are superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. In the example shown in Fig. 15, the segment information generating unit 12 detects the Rubiere's groove RS and the lower edge S4b of the liver S4 as anatomical landmarks. The display generating unit 15 superimposes color tiles of different colors on the Rubiere's groove RS and the lower edge S4b of the liver S4.

[0062] In the example shown in Figure 15, the display generation unit 15 superimposes a green first line graphic L1a passing through the upper end of Rubiere's groove RS and the upper end of the lower edge S4b of the liver S4, a yellow third line graphic L1c passing through the lower end of Rubiere's groove RS and the lower end of the lower edge S4b of the liver S4, and a yellow-green second line graphic L1b passing halfway between the first line graphic L1a and the third line graphic L1c on the laparoscopic image.

[0063] Fig. 16 is a diagram showing another example of an image in which three line graphics L1a-L1c are superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. Fig. 15 shows an example in which the three line graphics L1a-L1c are formed as straight lines, but as shown in Fig. 16, the three line graphics L1a-L1c may be formed as curved lines.

[0064] In the example shown in Figures 15 and 16, the area above the first line graphic L1a is the first segment (safe zone segment), and the area below the third line graphic L1c is the second segment (unsafe zone segment) that is less safe than the first segment. The three line graphics L1a-L1c divide the first and second segments into stages.

[0065] The display generating unit 15 can also generate an animation graphic in which a plurality of line graphics move from the second segment toward the first segment as a guidance display in the boundary area between the first segment and the second segment.

[0066] Fig. 17 is a diagram showing an example of an image in which an animation graphic is superimposed on a laparoscopic image taken during laparoscopic cholecystectomy. Although not shown in Fig. 17, the segment information generating unit 12 detects the Rubiere's groove RS and the umbilical fissure as anatomical landmarks.

[0067] In the example shown in FIG. 17, the display generation unit 15 superimposes three lines, a red line graphic Lr, a yellow line graphic Ly, and a green line graphic Lg, in that order from the bottom, on the laparoscopic image in the boundary area between the first segment and the second segment. The three line graphics move from the bottom to the top of the boundary area. Specifically, the red line graphic Lr springs out from the bottom of the boundary area, changes from red to yellow to green as it moves toward the top of the boundary area, and disappears when it reaches the top of the boundary area. Note that the number of line graphics displayed in the boundary area may be four or more.

[0068] By displaying an animated graphic in which multiple line graphics move like waves from the unsafe zone to the safe zone in the boundary area, the doctor can be visually guided to move away from the unsafe zone in the direction of the safe zone.

[0069] In the embodiment described above, if the segment information generator 12 cannot detect the R4U line, the display generator 15 may display an alert message on the monitor 30 to alert the doctor performing the procedure. This alert message can be used as an opportunity for an inexperienced doctor to consult with an experienced doctor or to have the experienced doctor take over the procedure, thereby contributing to minimizing the risk of damage to the common bile duct.

[0070] As described above, according to this embodiment, when performing laparoscopic surgery, the doctor can be fully aware of which part is safer and which part is less safe. The doctor can perform laparoscopic surgery safely by performing the procedure on the first segment. By superimposing the above-mentioned various guidance displays on the laparoscopic image, it is possible to prevent the common bile duct from being erroneously resected during laparoscopic cholecystectomy and to encourage the cystic duct and cystic artery to be resected at appropriate positions. For example, by displaying the R4U line as a guidance display, even an inexperienced doctor can always be aware of the R4U line and perform the procedure above the R4U line.

[0071] The present disclosure has been described above based on a number of embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing step, and that such modifications are also within the scope of the present disclosure.

[0072] In the above-described embodiment, anatomical landmarks are detected from the captured laparoscopic images using a machine learning model. In this regard, features such as SIFT, SURF, edges, and corners may be detected from the captured laparoscopic images, and anatomical landmarks may be detected based on the feature description of the laparoscopic images.

[0073] In addition, when a machine learning model for safe zone detection is prepared as in the above-mentioned challenge case, segment information can be generated directly without detecting anatomical landmarks. In this case, even if anatomical landmarks are not reflected in the laparoscopic image, safe zones and unsafe zones in the laparoscopic image can be identified. [Industrial Applicability]

[0074] INDUSTRIAL APPLICABILITY The present disclosure can be utilized in the technical field of displaying laparoscopic images. [Explanation of symbols]

[0075] 1 medical support system, 2 laparoscope, 3 treatment tool, 10 medical support device, 11 medical image acquisition unit, 12 segment information generation unit, 13 treatment tool recognition unit, 14 learning model storage unit, 15 display generation unit, 20 video processor, 30 monitor, GB gallbladder, DU duodenum, CD cystic duct, HD hepatic duct, HA hepatic artery, CBD common bile duct, RS Rubiere's groove, S4b inferior edge of liver S4.

Claims

1. A medical support device, comprising: having one or more processors; The processor, Acquire medical images; configured to generate, based on the medical image, a guidance display for superimposing on the medical image, the guidance display indicating a boundary between segments in the medical image having different degrees of medical treatment recommendation; The processor, generating a plurality of lines indicating the boundaries between the segments in stages as the guidance display; generating, as the guidance display, an animation in which the lines move from the second segment to the first segment in a boundary area between the first segment and a second segment having a lower degree of recommendation for medical treatment than the first segment; Medical support equipment.

2. 2. The medical support device according to claim 1, the processor estimates positions of two or more anatomical landmarks based on the medical image; generating the guidance display based on the estimated positions of the two or more anatomical landmarks. Medical support equipment.

3. 3. The medical support device according to claim 2, The processor acquires the medical image showing the gallbladder; The two or more anatomical landmarks include at least one of the following: Rouviere's groove, the inferior edge of the liver S4, the hepatic hilar plate, the hepatic artery, and the umbilical fissure; Medical support equipment.

4. 3. The medical support device according to claim 2, the processor generates the guidance display based on relative positions of the two or more anatomical landmarks. Medical support equipment.

5. 2. The medical support device according to claim 1, The processor recognizes a treatment tool from the medical image, changing the presence or absence of the guidance display or the manner of the display depending on the relative positional relationship between the treatment tool and the segment; Medical support equipment.

6. 6. The medical support device according to claim 5, The processor generates the guidance display, which is an alert, according to a relative positional relationship between the treatment tool and the segment. Medical support equipment.

7. 3. The medical support device according to claim 2, The processor further extracts two or more feature points from the medical image; generating the guidance display based on positions of the two or more anatomical landmarks and positions of the two or more feature points; Medical support equipment.

8. 8. The medical support device according to claim 7, When detection of at least one of the anatomical landmarks is lost, the processor: changing the guidance display based on positions of at least the two or more feature points after detection of at least one of the anatomical landmarks ceases; Medical support equipment.

9. 3. The medical support device according to claim 2, the processor extracts two or more feature points from the medical image, tracks positional changes of the two or more feature points, and, when detection of at least one of the anatomical landmarks is lost, estimates a positional change of the segment after detection of at least one of the anatomical landmarks is lost based on the positional changes of the two or more feature points; Varying a position of the guidance display based on a change in position of the segment after detection of at least one of the anatomical landmarks ceases. Medical support equipment.

10. 3. The medical support device according to claim 2, The processor acquires the medical image taken by an endoscope; acquiring motion information of the endoscope when detection of at least one of the anatomical landmarks is lost; changing a position of the guidance display based on movement information of the endoscope after detection of at least one of the anatomical landmarks ceases; Medical support equipment.

11. 2. The medical support device according to claim 1, The processor tracks tissue movement in the medical images; generating a guidance display that follows the movement of the tissue based on a result of tracking the movement of the tissue; Medical support equipment.

12. 2. The medical support device according to claim 1, The processor displays the medical image on which the guidance display is superimposed on a monitor connected to the medical support device. Medical support equipment.

13. 2. The medical support device according to claim 1, The processor acquires the medical image showing the gallbladder or tissue surrounding the gallbladder; The plurality of lines pass through the Rubierre's groove and the lower edge of the liver S4 shown in the medical image. Medical support equipment.

14. 2. The medical support device according to claim 1, The processor acquires the medical image showing the gallbladder or tissue surrounding the gallbladder; The plurality of lines pass through the Rubierre's groove and the umbilical fissure shown in the medical image. Medical support equipment.

15. 2. The medical support device according to claim 1, The plurality of lines are three or more lines, each of which passes through two anatomical landmarks and is at a different position from the others. Medical support equipment.

16. 16. The medical support device according to claim 15, The three or more lines are of different colors. Medical support equipment.

17. 2. The medical support device according to claim 1, The medical image shows the gallbladder or tissue surrounding the gallbladder. Medical support equipment.

18. A method for operating a medical support device, comprising: The medical support device includes: Obtaining a medical image; generating a guidance display based on the medical image, the guidance display indicating boundaries between segments in the medical image having different degrees of medical treatment recommendation, for overlaying on the medical image; Equipped with The medical support device includes: generating a plurality of lines indicating the boundaries between the segments in stages as the guidance display; generating, as the guidance display, an animation of a plurality of lines moving from a first segment to a second segment in a boundary area between the first segment and a second segment having a lower degree of recommendation for medical treatment than the first segment; A method for operating a medical support device.

19. 20. The method for operating a medical support device according to claim 18, further comprising: The medical support device includes: directing the procedure to be performed on the first segment; A method for operating a medical support device.

Images