Anatomical region identification method based on deep learning and 3D fluorescent laparoscope

Abstract

The invention discloses an anatomical region identification method based on deep learning and a 3D fluorescent laparoscope. The method comprises the following steps: acquiring an operation video of laparoscopic cholecystectomy through a 3D fluorescent laparoscope; extracting a short video of gall bladder triangular anatomy of the gall bladder neck from the operation video; an operation image set is extracted from the short video, and an anatomical area of the operation image set is provided with a fluorescent mark; marking an anatomical region for the extracted surgical image set according to a fluorescent identifier; training an anatomical region identification model; and marking an anatomical region for a real-time image acquired by the 3D fluorescent laparoscope in a real-time operation process through the anatomical region marking model. According to the method, the operation video is obtained through the 3D fluorescent laparoscope, the anatomical region in the operation video has the fluorescent mark, and the position of the anatomical region can be accurately reflected through the fluorescent mark, so that the anatomical region marked in the operation image set is more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and in particular relates to an anatomical region identification method based on deep learning and 3D fluorescence laparoscope. Background Art

[0002] Laparoscopic cholecystectomy (LC) is the mainstream surgical procedure and gold standard for benign gallbladder disease. With the rapid development of minimally invasive techniques, LC has been rapidly implemented in hospitals at all levels in my country. However, compared with open surgery, laparoscopic surgery faces challenges such as a two-dimensional field of view, limited tactile feedback, and instrument fulcrum effect, which increase the incidence of many serious complications. Iatrogenic bile duct injury is the most common and severe of these complications, severely impacting patients' survival prognosis and quality of life, and also having serious consequences for the surgeon's career and psychology.

[0003] Critical view of safety (CVS) technology, as a standard operating procedure for LC, is currently the primary method for preventing bile duct injury. However, it is constrained by multiple factors. The identification of important anatomical structures during surgery primarily relies on experienced senior physicians to provide surgical training, intraoperative guidance, or even take over the operation for younger surgeons. However, this model is plagued by a series of issues, including long learning cycles, high training costs, poor training quality, and difficulties in standardization. Exploring a more intelligent, efficient, and standardized surgical training and guidance system is of great significance for further reducing LC complications, improving the surgical skills of younger surgeons, freeing up training time for senior physicians, and optimizing medical resources.

[0004] With the advent of artificial intelligence (AI), computer vision has been widely used in medicine. It describes the ability of machines to understand images and videos, achieving near-human performance in areas such as object and scene recognition. Furthermore, algorithms such as deep neural networks can be trained without extensive data to learn to predict outcomes on new data. While deep learning has demonstrated promising results in fields such as diagnostic imaging and endoscopy, it has yet to be validated for real-time surgical guidance and decision support. Unlike images and videos from diagnostic imaging, ophthalmoscopy, or endoscopy, laparoscopic surgical videos exhibit greater variability in background noise, image quality, and in-situ objects. Furthermore, numerous surgical planes and anatomical structures are rarely clearly delineated and are often hidden or partially visible beneath fat and fibrous tissue. This presents a significant obstacle to the use of computer vision to provide clinically meaningful data during surgery. Therefore, the development and validation of a deep learning-based AI anatomical landmarking system for guiding and training laparoscopic surgery is a pressing and hot issue in the field of hepatobiliary surgery.

[0005] Fluorescence laparoscopy technology, full name indocyanine green (ICG) - labeled near - infrared (NIR) imaging fluorescence laparoscopy technology. Indocyanine green (ICG) is a near - infrared light contrast agent with good biocompatibility, which can be excited by external light with a wavelength of 750 - 800 nm and emit near - infrared light with a longer wavelength, presenting green under fluorescence laparoscopy, thus realizing the imaging of tissues and organs.

[0006] Fluorescence laparoscopy is the most advanced technology in current laparoscopy and has been widely used clinically. However, there is currently no research on the combination of fluorescence laparoscopy and artificial intelligence technology. Summary of the Invention

[0007] An object of the present invention is to provide an anatomical region identification method based on deep learning and 3D fluorescence laparoscopy.

[0008] According to the first aspect of the present invention, there is provided an anatomical region identification method based on deep learning and 3D fluorescence laparoscopy, including:

[0009] Obtain the surgical video of laparoscopic cholecystectomy through 3D fluorescence laparoscopy, wherein the surgical video meets the critical safety operative field technical standard;

[0010] Extract a short video of the anatomical dissection of Calot's triangle at the neck of the gallbladder from the surgical video of laparoscopic cholecystectomy;

[0011] Extract a set of surgical images from the short video of the anatomical dissection of Calot's triangle at the neck of the gallbladder, wherein the anatomical regions in the set of surgical images have fluorescence markings;

[0012] Label the anatomical regions for the extracted set of surgical images according to the fluorescence markings, wherein the anatomical regions include Rouviere sulcus, common bile duct, cystic duct, inferior margin of the medial segment of the left liver, and gallbladder;

[0013] Train a preset deep - learning - based image recognition model according to the set of surgical images with labeled anatomical regions to obtain an anatomical region identification model for laparoscopic cholecystectomy;

[0014] Label the anatomical regions for the real - time images collected by 3D fluorescence laparoscopy during the real - time surgery through the anatomical region identification model for laparoscopic cholecystectomy.

[0015] Optionally, the extracting a set of surgical images from the short video of the anatomical dissection of Calot's triangle at the neck of the gallbladder includes:

[0016] Extract a set of static images from the short video;

[0017] Compare the changes of each static image in the static image set to determine the optimal sampling frequency;

[0018] Extract the surgical image set from the short video of the cholecystic triangle anatomy at the neck of the gallbladder according to the optimal sampling frequency.

[0019] Optionally, the comparing the changes of each static image in the static image set to determine the optimal sampling frequency includes:

[0020] Determine the similarity between each static image according to the changes of each static image;

[0021] Traverse the static image set to determine a set of similar images from the static image set, wherein the similarity between adjacent images in the set of similar images exceeds a preset similarity threshold;

[0022] Determine the optimal sampling frequency according to the static image set.

[0023] Optionally, the labeling the anatomical regions for the extracted surgical image set includes:

[0024] Obtain a set of labeled images after multiple expert surgeons label the surgical image set respectively;

[0025] Determine a set of difference images according to the labeled image set, wherein the images in the set of difference images are the images in which the difference between the positions of the anatomical regions labeled by the multiple expert surgeons for the same surgical image exceeds a preset difference threshold;

[0026] Re-evaluate the set of difference images to determine the correct positions of the labeled anatomical regions of each image in the set of difference images.

[0027] Optionally, after training a preset deep learning-based image recognition model according to the surgical image set with labeled anatomical regions, the method further includes:

[0028] Label a test data set through the trained model, wherein the test data set includes multiple surgical images with labeled anatomical regions that have not been used for learning and training;

[0029] Calculate the local accuracy rate and the overall accuracy rate of the labeling according to the labeling results of the model for the test data set, wherein the local accuracy rate is the accuracy rate of the position of the anatomical region labeled by the model for each image in the test data set, and the overall accuracy rate is the proportion of the images accurately labeled by the model in the total number of images in the test data set;

[0030] If the local accuracy rate is lower than the local accuracy rate threshold or the overall accuracy rate is lower than the overall accuracy rate threshold, re-train the model.

[0031] Optionally, calculating the local accuracy rate and overall accuracy rate of the annotation based on the annotation result of the test data set by the model includes:

[0032] Obtain the position coordinates of the anatomical regions annotated in each image in the test data set by the model;

[0033] Calculate the coordinate difference between the standard position coordinates of the anatomical regions in each image of the test data set and the position coordinates of the anatomical regions annotated by the model;

[0034] Determine the local accuracy rate of the annotation of each image in the test data set by the model according to the coordinate difference;

[0035] Calculate the overall accuracy rate according to the local accuracy rate corresponding to each image in the test data set.

[0036] Optionally, the preset deep learning-based image recognition model is the YOL0v7 model.

[0037] According to the second aspect of the present invention, there is provided an anatomical region identification system based on deep learning and 3D fluorescence laparoscopy. The system applies the anatomical region identification method for laparoscopic cholecystectomy described in the first aspect of the present invention. The system includes:

[0038] An acquisition module, configured to acquire a surgical video of laparoscopic cholecystectomy through a 3D fluorescence laparoscope, wherein the surgical video meets the critical safety operative field technical standard;

[0039] A first extraction module, configured to extract a short video of the cystic triangle anatomy at the neck of the gallbladder from the surgical video of the laparoscopic cholecystectomy;

[0040] A second extraction module, configured to extract a set of surgical images from the short video of the cystic triangle anatomy at the neck of the gallbladder, wherein the anatomical regions in the set of surgical images have fluorescence markings;

[0041] A first annotation module, configured to annotate the anatomical regions on the extracted set of surgical images according to the fluorescence markings, wherein the anatomical regions include the Rouviere sulcus, common bile duct, cystic duct, inferior margin of the medial segment of the left liver, and gallbladder;

[0042] A training module, configured to train a preset deep learning-based image recognition model according to the set of surgical images with annotated anatomical regions to obtain an anatomical region identification model for laparoscopic cholecystectomy;

[0043] A second annotation module, configured to annotate the anatomical regions on the real-time images collected by the laparoscope during the real-time surgery through the anatomical region identification model for laparoscopic cholecystectomy.

[0044] According to a third aspect of the present invention, there is provided an electronic device, including a processor and a memory, where the memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, the steps of a method for identifying anatomical regions based on deep learning and 3D fluorescence laparoscopy as described in the first aspect of the present invention are implemented.

[0045] According to a fourth aspect of the present invention, there is provided a readable storage medium, where programs or instructions are stored on the readable storage medium, and when the programs or instructions are executed by a processor, the steps of a method for identifying anatomical regions based on deep learning and 3D fluorescence laparoscopy as described in the first aspect of the present invention are implemented.

[0046] The beneficial effects of the present invention are as follows: The present invention adopts artificial intelligence technology, based on image data such as laparoscopic surgery videos, relying on the image recognition technology of deep learning, completes the development of intraoperative marking software for laparoscopic surgery, and uses the software to guide junior young doctors to master and learn the CVS technology, improve the cognition of key anatomical points, perform laparoscopic cholecystectomy, shorten the learning curve of laparoscopic surgery, and reduce the probability of iatrogenic bile duct injury caused by lack of experience. On the other hand, in the present invention, the surgical video is obtained through a 3D fluorescence laparoscope, and the anatomical regions in the surgical video have fluorescence markings, and the positions of the anatomical regions can be accurately reflected through the fluorescence markings, making the anatomical regions marked in the surgical images more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a method for identifying anatomical regions based on deep learning and 3D fluorescence laparoscopy according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] Now, various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0049] The following description of at least one exemplary embodiment is actually merely illustrative and in no way restricts the present invention and its application or use.

[0050] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be regarded as part of the specification. In all the examples shown and discussed here, any specific value should be construed as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values.

[0051] It should be noted that like reference numerals and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

[0052] In the description and claims of the present invention, features related to the terms "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally means an "or" relationship between the associated objects before and after.

[0053] Laparoscopic cholecystectomy is the standard surgical procedure for treating benign gallbladder diseases and is also regarded as the starting procedure and "touchstone" for hepatobiliary surgeons to be competent in laparoscopic surgery. With the popularization of laparoscopic cholecystectomy, the rate of iatrogenic bile duct injury has increased significantly compared with the past, which has an adverse impact on the health of patients and the growth of young doctors. To improve the quality of surgery, traditional training mostly adopts modes such as intraoperative guidance by senior doctors or taking over the surgery, but there are problems such as a long learning cycle, high training costs, and difficulties in standardization.

[0054] Currently, the most commonly used solutions at home and abroad are: during the LC operation, experienced senior doctors train young doctors, give intraoperative guidance or take over the surgery, in order to meet the CVS standard. However, this mode has problems such as a long learning cycle, high training costs, poor training quality, and difficulties in standardization.

[0055] At present, artificial intelligence has gradually been used in the medical field, but it has not been developed and demonstrated in the surgical training of young doctors. Exploring the use of an artificial intelligence anatomical marking system as a training tool is very necessary to improve the surgical quality of junior doctors, shorten the learning curve, and reduce the medical risks during training.

[0056] Laparoscopic surgery videos can be stored in a server as a form of data for doctors to view and share. With the advent of the big data era, many data are used to discover, analyze, and solve problems, and the rich laparoscopic surgery data resources are no exception. And artificial intelligence is best at analyzing data and solving problems. Currently, artificial intelligence has demonstrated its feasibility and advantages in many fields of medical image recognition. Different from images and videos such as diagnostic imaging, ophthalmoscopes, or endoscopes, laparoscopic surgery videos have more variability in terms of background noise, image quality, and the scene. In addition, a large number of surgical planes and anatomical structures are almost never clearly depicted and are easily interfered by factors such as bleeding or smoke, which pose challenges to the anatomical identification of artificial intelligence in laparoscopic surgery. Currently, artificial intelligence image recognition systems have initially demonstrated their capabilities in the field of laparoscopic surgery, including surgical operation classification recognition, surgical stage recognition, surgical time prediction, identification of surgical instruments, and intraoperative anatomy. Some scholars have also tried to use the method of deep learning with convolutional neural networks to identify important anatomical structures in laparoscopic gastrectomy and laparoscopic hysterectomy. Although there are a series of problems such as the lack of standardized laparoscopic surgery datasets, insufficient training data volume, low recognition accuracy, and the inability to guide surgical operations in real time, they have revealed that the application of a deep learning-based artificial intelligence anatomical identification system in LC has great potential for improving surgical quality and surgical training efficiency.

[0057] As Figure 1 shown, this embodiment introduces an anatomical region identification method based on deep learning and 3D fluorescence laparoscopy, including step 11001600.

[0058] Step 1100: Obtain the surgical video of laparoscopic cholecystectomy through a 3D fluorescence laparoscope, where the surgical video meets the critical safety operative field technical standard.

[0059] First, collect multiple complete videos of LC surgeries through a 3D fluorescence laparoscope, requiring that the LC surgical operations strictly meet the CVS technical standard, the video acquisition resolution is 1920*1080p, the frame rate is 25 frames, and the saving format is MPEG4. If the clarity of the surgical video is affected by electrocautery and smoke gas in the abdominal cavity and results in low clarity, we will exclude it. Videos with a large amount of intraoperative bleeding, unclear vision, or inability to identify important anatomical sites due to severe inflammation also need to be excluded.

[0060] Step 1200: Extract a short video of the cystic triangle anatomy at the neck of the gallbladder from the surgical video of the laparoscopic cholecystectomy.

[0061] For each surgical video, extract the corresponding short video of the cystic triangle anatomy at the neck of the gallbladder.

[0062] Step 1300: Extract a set of surgical images from the short video of the dissection of the Calot triangle at the neck of the gallbladder, where the anatomical region of the set of surgical images has a fluorescence label.

[0063] Step 1400: Mark the anatomical regions on the extracted set of surgical images according to the fluorescence label, where the anatomical regions include the Rouviere sulcus, the common bile duct, the cystic duct, the lower edge of the medial segment of the left liver, and the gallbladder.

[0064] It is possible to mark all five regions of the Rouviere sulcus, the common bile duct, the cystic duct, the lower edge of the medial segment of the left liver, and the gallbladder in one surgical image. It is also possible to mark some of the five regions.

[0065] Step 1500: Train a preset deep learning-based image recognition model according to the set of surgical images with marked anatomical regions to obtain an anatomical region identification model for laparoscopic cholecystectomy.

[0066] The deep learning-based image recognition model can be the YOLOv7 model. Use the set of surgical images with marked anatomical regions as the training set to train the image recognition model.

[0067] Step 1600: Mark the anatomical regions on the real-time images collected by the 3D fluorescence laparoscope during the real-time surgical process through the anatomical region identification model for laparoscopic cholecystectomy.

[0068] During the real-time surgical process, collect the real-time images of the patient's body through the laparoscope. And input the real-time images collected by the laparoscope into the anatomical region identification model for laparoscopic cholecystectomy to identify the anatomical regions in the real-time images to help the doctor complete the surgery.

[0069] The present invention uses artificial intelligence technology, based on image data such as laparoscopic surgery videos, relying on deep learning image recognition technology, to complete the development of the intraoperative marking software for laparoscopic surgery, and uses the software to guide junior young doctors to master and learn the CVS technology, improve the understanding of key anatomical points, perform laparoscopic cholecystectomy, shorten the learning curve of laparoscopic surgery, and reduce the probability of iatrogenic biliary tract injury caused by lack of experience.

[0070] In this embodiment, step 1300 includes steps 1310-1330.

[0071] Step 1310: Extract a set of static images from the short video.

[0072] Step 1320: Compare the change situations of the static images in the set of static images to determine the optimal sampling frequency.

[0073] Step 1330: Extract the surgical image set from the short video of the dissection of the Calot triangle at the neck of the gallbladder according to the optimal sampling frequency.

[0074] During the operation, as the diseased tissue is continuously resected, the images captured by the laparoscope will change. In addition, as the operation progresses, it may be necessary to operate on different positions in the patient's body, and the images captured by the laparoscope will also change. The static image set contains all the image frames in the short video. For multiple static images with large changes, separate annotations are required. For multiple static images with small changes, only one annotation is needed to reduce redundant data and improve the model training efficiency.

[0075] By comparing the changes of each static image in the static image set, the optimal sampling frequency is determined. Sample the short video of the dissection of the Calot triangle at the neck of the gallbladder according to the optimal sampling frequency, and extract the surgical image set. On the one hand, it ensures the comprehensiveness of the surgical image set and will not miss some surgical images that need to be annotated. On the other hand, it can reduce redundant data.

[0076] Since the number of images in the static image set extracted from the short video is large, if all images are annotated, a large amount of redundant data will be generated, making the surgical image set used for training the model too large and reducing the efficiency. The present invention determines the optimal sampling frequency and extracts the surgical image set from the short video according to the optimal sampling frequency to eliminate redundant data.

[0077] In this embodiment, step 1320 includes steps 1321-1323.

[0078] Step 1321: Determine the similarity between each static image according to the change situation of each static image.

[0079] Step 1322: Traverse the static image set, and determine a similar image set from the static image set, where the similarity between adjacent images in the similar image set exceeds a preset similarity threshold.

[0080] Step 1323: Determine the optimal sampling frequency according to the static image set.

[0081] Start traversing from the first image in the static image set, and find multiple images whose similarity to the first image exceeds a preset similarity threshold. For example, the second to the tenth images in the static image set, the similarity between these images and the first image all exceeds the similarity threshold. Select the image with the latest time in chronological order of these images, that is, the tenth image, and put the tenth image into the similar image set. Then, find the image set whose similarity to the tenth image exceeds the similarity threshold according to the same method, and select the image with the latest time and put it into the similar image set. Repeat the above steps until the static image set is traversed completely to obtain the similar image set.

[0082] After obtaining the similar image set, determine the optimal sampling frequency according to the number of images in the similar image set and the number of image frames in the short video of the Calot triangle anatomy at the neck of the gallbladder.

[0083] By traversing the static image set, the static image set is divided into multiple groups of images. The similarity between each image in each group of images and the first image in the group of images exceeds a preset similarity threshold. For adjacent two groups of images, the similarity between all images in the latter group of images and the last image in the former group of images exceeds the similarity threshold. One image can be extracted from each group of images through the optimal sampling frequency as the surgical image for annotating the anatomical region.

[0084] In this embodiment, step 1400 includes steps 1410 - 1430.

[0085] Step 1410: Obtain the set of marked images after multiple expert surgeons mark the surgical image set respectively.

[0086] Step 1420: Determine the set of difference images according to the marked set of marked images. Among them, the images in the set of difference images are the images whose difference in the positions of the anatomical regions marked by the multiple expert surgeons for the same surgical image exceeds a preset difference threshold.

[0087] Step 1430: Re-evaluate the set of difference images and determine the correct positions of the anatomical regions marked by each image in the set of difference images.

[0088] During the process of marking the surgical image set, multiple expert surgeons mark the surgical image set respectively to obtain the set of marked images marked by each expert surgeon. The expert surgeon is an expert surgeon who has experienced more than 2000 LC surgeries. Since the consistency among the expert surgeons is poor, it is necessary to re-evaluate the set of marked images marked by each expert surgeon to eliminate the marked data with obvious errors.

[0089] For the same surgical image, if the content marked by each expert surgeon is different, it indicates that there are differences. For the marked images with large differences, they are put into the difference image set. Specifically, for a surgical image, each expert surgeon needs to mark the position of the anatomical region in the image. The difference between the specific coordinates of the marked positions of the anatomical regions by each expert surgeon in the marked image can be calculated. If this difference exceeds the preset difference degree threshold, then this marked image is considered a difference image.

[0090] After finding all the difference images, by re-evaluating the difference image set, eliminate the marked annotations with obvious errors, and determine the correct marked position of the anatomical region for each difference image.

[0091] In this embodiment, after training the preset deep learning-based image recognition model according to the surgical image set with marked anatomical regions, the method further includes: using the trained model to mark the test data set, where the test data set includes multiple surgical images with marked anatomical regions that have not been used for learning and training; calculating the local accuracy rate and the overall accuracy rate of the markings according to the marking results of the model on the test data set, where the local accuracy rate is the accuracy rate of the position of the anatomical region marked by the model for each image in the test data set, and the overall accuracy rate is the proportion of the images marked accurately by the model in the total number of images in the test data set; if the local accuracy rate is lower than the local accuracy rate threshold or the overall accuracy rate is lower than the overall accuracy rate threshold, retrain the model.

[0092] The test data set includes at least 2000 surgical images with marked anatomical regions that have not participated in the training.

[0093] The present invention tests the performance of the model through the test data set, calculates the overall accuracy rate and the local accuracy rate, and outputs the model after both the overall accuracy rate and the local accuracy rate meet the requirements. If they do not meet the requirements, then retrain until both the overall accuracy rate and the local accuracy rate meet the requirements.

[0094] In this embodiment, calculating the local accuracy rate and the overall accuracy rate of the markings according to the marking results of the model on the test data set includes: obtaining the position coordinates of the anatomical region marked by the model for each image in the test data set; calculating the coordinate difference between the standard position coordinates of the anatomical region in each image of the test data set and the position coordinates of the anatomical region marked by the model; determining the local accuracy rate of the model's marking for each image in the test data set according to the coordinate difference; calculating the overall accuracy rate according to the local accuracy rate corresponding to each image in the test data set.

[0095] The local accuracy is calculated based on the coordinate difference. The larger the coordinate difference, the lower the local accuracy. After calculating the local accuracy of each image in the test dataset, the average value of the local accuracies of all images is calculated, and the average value of the local accuracies of all images is used as the overall accuracy.

[0096] This embodiment introduces an anatomical region identification system based on deep learning and 3D fluorescence laparoscopy. The system applies an anatomical region identification method based on deep learning and 3D fluorescence laparoscopy according to any embodiment of the present invention. The system includes:

[0097] An acquisition module, configured to acquire a surgical video of laparoscopic cholecystectomy through a 3D fluorescence laparoscope, wherein the surgical video meets the critical safety surgical field technical standard;

[0098] A first extraction module, configured to extract a short video of the cystic triangle anatomy at the neck of the gallbladder from the surgical video of the laparoscopic cholecystectomy;

[0099] A second extraction module, configured to extract a set of surgical images from the short video of the cystic triangle anatomy at the neck of the gallbladder, wherein the anatomical region of the set of surgical images has a fluorescence label;

[0100] A first annotation module, configured to annotate the anatomical region of the extracted set of surgical images according to the fluorescence label, wherein the anatomical region includes the Rouviere sulcus, the common bile duct, the cystic duct, the lower edge of the medial segment of the left liver, and the gallbladder;

[0101] A training module, configured to train a preset deep learning-based image recognition model according to the set of surgical images with annotated anatomical regions, to obtain an anatomical region identification model for laparoscopic cholecystectomy;

[0102] A second annotation module, configured to annotate the anatomical region of the real-time image collected by the 3D fluorescence laparoscope during the real-time surgery through the anatomical region identification model for laparoscopic cholecystectomy.

[0103] This embodiment introduces an electronic device, including a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of an anatomical region identification method for laparoscopic cholecystectomy according to any embodiment of the present invention are implemented.

[0104] This embodiment introduces a readable storage medium. A program or instruction is stored on the readable storage medium. When the program or instruction is executed by a processor, the steps of an anatomical region identification method for laparoscopic cholecystectomy according to any embodiment of the present invention are implemented.

[0105] Although some specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and not for limiting the scope of the present invention. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

[0106] Those of ordinary skill in the art can realize that the modules and algorithm steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0107] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices and equipment described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0108] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of the devices or modules can be in an electrical, mechanical or other forms.

[0109] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present invention.

[0110] In addition, the various functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0111] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0112] The above description is only a preferred embodiment of the present application and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the technical solution formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the present application.

[0113] It should be understood that the magnitudes of the sequence numbers of the steps in the inventive content and embodiments of the present invention do not absolutely mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention. For the purposes of illustration and description, the foregoing description of the implementation of the present disclosure has been given. The foregoing description is not exhaustive nor is it intended to limit the present disclosure to the exact form disclosed. According to the above teachings, various modifications and variations are possible, or various modifications and variations may be obtained from the practice of the present disclosure. These embodiments are selected and described to illustrate the principles of the present disclosure and its practical applications, so that those skilled in the art can utilize the present disclosure in various embodiments and various modifications suitable for the specific purposes contemplated.

Claims

1. An anatomical region marking method based on deep learning and 3D fluorescence laparoscopy, characterized in that, Including: Obtaining a surgical video of laparoscopic cholecystectomy through a 3D fluorescence laparoscope, wherein the surgical video meets the critical safety operative field technical standard; Extracting a short video of the cystic triangle anatomy at the neck of the gallbladder from the surgical video of the laparoscopic cholecystectomy; Extracting a set of surgical images from the short video of the cystic triangle anatomy at the neck of the gallbladder, wherein the anatomical region of the set of surgical images has a fluorescence label; Marking the anatomical region for the extracted set of surgical images according to the fluorescence label, wherein the anatomical region includes the Rouviere sulcus, the common bile duct, the cystic duct, the lower edge of the medial segment of the left liver, and the gallbladder; Training a preset deep learning-based image recognition model according to the set of surgical images marked with the anatomical region to obtain an anatomical region identification model for laparoscopic cholecystectomy; Marking the anatomical region for the real-time images collected by the 3D fluorescence laparoscope during the real-time surgical process through the anatomical region identification model of the laparoscopic cholecystectomy.

2. The method according to claim 1, characterized in that The extracting a set of surgical images from the short video of the cystic triangle anatomy at the neck of the gallbladder includes: Extracting a set of static images from the short video; Comparing the change situations of the static images in the set of static images to determine the optimal sampling frequency; Extracting the set of surgical images from the short video of the cystic triangle anatomy at the neck of the gallbladder according to the optimal sampling frequency.

3. The method according to claim 2, wherein The comparing the change situations of the static images in the set of static images to determine the optimal sampling frequency includes: Determining the similarity between the static images according to the change situations of the static images; Traversing the set of static images to determine a set of similar images from the set of static images, wherein the similarity between adjacent images in the set of similar images exceeds a preset similarity threshold; Determining the optimal sampling frequency according to the set of static images.

4. The method according to claim 1, characterized in that, The marking the anatomical region for the extracted set of surgical images includes: Obtaining a set of marked images after multiple expert surgeons respectively mark the set of surgical images; Determining a set of difference images according to the set of marked images, wherein the images in the set of difference images are the images in which the difference in the positions of the anatomical regions marked by the multiple expert surgeons for the same surgical image exceeds a preset difference threshold; Re-evaluating the set of difference images to determine the correct positions of the anatomical regions marked for the images in the set of difference images.

5. The method according to claim 1, wherein After training a preset deep learning-based image recognition model according to the set of surgical images marked with the anatomical region, the method further includes: Marking a test data set through the trained model, wherein the test data set includes multiple surgical images marked with the anatomical region that have not been used for learning and training; Calculating the local accuracy rate and the overall accuracy rate of the marking according to the marking results of the model for the test data set, wherein the local accuracy rate is the accuracy rate of the position of the anatomical region marked by the model for each image in the test data set, and the overall accuracy rate is the proportion of the images accurately marked by the model in the test data set to the total number of images in the test data set; If the local accuracy rate is lower than the local accuracy rate threshold or the overall accuracy rate is lower than the overall accuracy rate threshold, re-training the model.

6. The method according to claim 1, characterized in that, Calculating the local accuracy and overall accuracy of the annotation based on the annotation results of the test data set by the model includes: Obtaining the position coordinates of the anatomical regions annotated in each image in the test data set by the model; Calculating the coordinate difference between the standard position coordinates of the anatomical regions in each image of the test data set and the position coordinates of the anatomical regions annotated by the model; Determining the local accuracy of the annotation of each image in the test data set by the model according to the coordinate difference; Calculating the overall accuracy according to the local accuracy corresponding to each image in the test data set.

7. The method according to claim 1, characterized in that, The preset deep learning-based image recognition model is the YOL0v7 model.

8. An anatomical region marking system based on deep learning and 3D fluorescence laparoscopy, characterized in that, The system applies the anatomical region identification method based on deep learning and 3D fluorescence laparoscope according to any one of claims 1-7. The system includes: An acquisition module for acquiring the surgical video of laparoscopic cholecystectomy through a 3D fluorescence laparoscope, wherein the surgical video meets the critical safety operative field technical standard; A first extraction module for extracting a short video of the cystic triangle anatomy at the neck of the gallbladder from the surgical video of laparoscopic cholecystectomy; A second extraction module for extracting a surgical image set from the short video of the cystic triangle anatomy at the neck of the gallbladder, wherein the anatomical regions in the surgical image set have fluorescence markings; A first annotation module for annotating the anatomical regions on the extracted surgical image set according to the fluorescence markings, wherein the anatomical regions include the Rouviere sulcus, common bile duct, cystic duct, inferior margin of the medial segment of the left liver, and gallbladder; A training module for training a preset deep learning-based image recognition model according to the surgical image set annotated with anatomical regions to obtain an anatomical region identification model for laparoscopic cholecystectomy; A second annotation module for annotating the anatomical regions on the real-time images collected by the 3D fluorescence laparoscope during the real-time surgery through the anatomical region identification model of laparoscopic cholecystectomy.

9. An electronic device, characterized in that, It includes a processor and a memory. The memory stores programs or instructions that can run on the processor. When the programs or instructions are executed by the processor, the steps of an anatomical region identification method based on deep learning and 3D fluorescence laparoscope according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that, Programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by the processor, the steps of an anatomical region identification method based on deep learning and 3D fluorescence laparoscope according to any one of claims 1 to 7 are implemented.

Images