Operation image processing method and device and medium

Abstract

The invention relates to the field of data processing, in particular to a surgical image processing method and device and a medium. The method comprises the steps that a to-be-processed operation image is collected, and a target operation scene corresponding to the operation image is recognized; extracting a first image frame from the surgical image; performing anatomical structure identification and preliminary labeling on the first image frame by adopting a surgical image labeling model to obtain a second image frame; pushing the second image frame to the user side for auditing, and adjusting the second image frame to obtain a third image frame; the third image frame is at least labeled with optimization labeling information of the key anatomical structure; and constructing a surgical image data set based on the third image frame and the corresponding optimized annotation information. According to the method, the surgical image can be automatically labeled through the surgical image labeling model, the automation degree of surgical image processing is improved, the surgical image is optimized based on the user auditing result, the image proofreading efficiency and reliability are further improved, and the method adapts to a surgical image processing scene.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a surgical image processing method, apparatus, and medium. Background Art

[0002] Minimally invasive surgery, especially laparoscopic and robot-assisted surgery, has gradually become the mainstream of surgical practice. Taking minimally invasive gastric cancer surgery as an example, during the operation, the lymph node dissection (LND) technology has high requirements, especially in the identification and dissection of anatomical structures around key blood vessels, facing great challenges. The main difficulties in lymph node dissection lie in correctly identifying the vascular sheath layer, dissecting lymphatic and adipose tissues, and exposing and ligating blood vessels, which pose extremely high requirements for surgeons' anatomical structure recognition ability. In addition, the vascular anatomical structure in gastric cancer surgery is complex and highly variable. A slight mistake may cause intraoperative vascular injury, leading to serious bleeding and other complications.

[0003] Currently, to reduce the surgical difficulty, improve intraoperative safety and surgical precision, surgical assistance technologies that use artificial intelligence (AI) technology to analyze surgical video images have been developed in the field of surgical data science (SDS). However, the medical image segmentation models in related technologies are usually used for the segmentation of articular cartilage, with a single application scenario, unable to handle the image segmentation and recognition of complex anatomical structures during surgery, and difficult to meet the image processing requirements in complex surgical scenarios. Therefore, there is an urgent need to propose a new technical solution to solve at least one of the above technical problems and provide a surgical image dataset with higher quality. Summary of the Invention

[0004] This application provides a surgical image processing method, apparatus, and medium to improve the automation degree of surgical image processing, enhance the image proofreading efficiency and reliability, and help adapt to more surgical image processing scenarios.

[0005] In a first aspect, an embodiment of this application provides a surgical image processing method, which includes:

[0006] Collect the surgical images to be processed, and identify the target surgical scenario corresponding to the surgical images; extract the first image frame from the surgical images; wherein, the first image frame at least includes: the key surgical operations in the target surgical scenario and the corresponding surgical space images; use the surgical image annotation model to perform anatomical structure recognition and preliminary annotation on the first image frame to obtain a second image frame; the initial annotation information of the key anatomical structures corresponding to the key surgical operations is marked in the second image frame; push the second image frame to the client for review, and receive the review result of the client to adjust the second image frame to obtain a third image frame; at least the optimized annotation information of the key anatomical structures is marked in the third image frame; construct a surgical image data set based on the third image frame and the corresponding optimized annotation information, and the surgical image data set is used to train and optimize the surgical image model.

[0007] In a second aspect, an embodiment of the present application provides a surgical image processing device, which at least includes the following units:

[0008] A collection unit configured to collect the surgical images to be processed and identify the target surgical scenario corresponding to the surgical images; an extraction unit configured to extract the first image frame from the surgical images; wherein, the first image frame at least includes: the key surgical operations in the target surgical scenario and the corresponding surgical space images; a labeling unit configured to perform anatomical structure recognition and preliminary labeling on the first image frame using a surgical image annotation model to obtain a second image frame; the initial annotation information of the key anatomical structures corresponding to the key surgical operations is marked in the second image frame; an optimization unit configured to push the second image frame to the client for review, and receive the review result of the client to adjust the second image frame to obtain a third image frame; at least the optimized annotation information of the key anatomical structures is marked in the third image frame; a construction unit configured to construct a surgical image data set based on the third image frame and the corresponding optimized annotation information, and the surgical image data set is used to train and optimize the surgical image model.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, and the electronic device includes:

[0010] At least one processor, a memory, and an input-output unit;

[0011] Wherein, the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the surgical image processing method in the first aspect.

[0012] In a fourth aspect, a computer-readable storage medium is provided, which includes instructions, and when the instructions are run on a computer, the computer is caused to execute the surgical image processing method in the first aspect.

[0013] The beneficial effects of this application are as follows: A surgical image processing method, device, and medium are provided. In this technical solution, the surgical image annotation model can be used to automatically annotate surgical images, improving the automation degree of surgical image processing. And based on the user review results, the surgical images are optimized to further improve the image proofreading efficiency and reliability, adapting to the surgical image processing scenario. Description of the Drawings

[0014] Figure 1 It is a schematic flowchart of a surgical image processing method according to an embodiment of this application;

[0015] Figures 2 to 5 It is a schematic diagram of the principle of a surgical image processing method according to an embodiment of this application. Detailed Embodiments

[0016] To solve at least one technical problem in the related art, embodiments of this application provide a surgical image processing method, device, and medium. In this technical solution, first, the surgical images to be processed are collected, and the target surgical scenario corresponding to the surgical images is identified. Furthermore, the first image frame is extracted from the surgical images; wherein, the first image frame at least includes: the key surgical operations in the target surgical scenario and the corresponding surgical space images. Then, the first image frame is subjected to anatomical structure recognition and preliminary annotation using the surgical image annotation model to obtain the second image frame. The initial annotation information of the key anatomical structures corresponding to the key surgical operations is annotated in the second image frame. Then, the second image frame is pushed to the user terminal for review, and the review result of the user terminal is received to adjust the second image frame to obtain the third image frame; at least the optimized annotation information of the key anatomical structures is annotated in the third image frame. Finally, a surgical image dataset is constructed based on the third image frame and the corresponding optimized annotation information, and the surgical image dataset is used to train and optimize the surgical image model.

[0017] In the embodiments of the present application, on the one hand, by combining the surgical image annotation model and the user-side review optimization, the semi-automatic annotation of surgical images is realized. The high efficiency in the initial annotation can quickly perform batch automated processing on a large number of images, identify key anatomical structures and give initial annotations, improving the efficiency of surgical image processing. Synchronously, manual correction of the initial annotation results is adopted in the review stage to make up for the possible errors of the surgical image annotation model, ensuring the annotation accuracy and correctness of the surgical image annotation results, and thus helping to balance the annotation efficiency and annotation accuracy. For example, when dealing with some complex anatomical structures or situations with image interference, the surgical image annotation model may make misjudgments, and manual assistance correction can be used to correct them based on professional knowledge and experience. On the other hand, the present application covers multiple links such as data collection, scene classification, annotation specification, semi-automatic annotation, and quality control, forming a systematic and comprehensive process. Through the surgical image annotation model, the present application can accurately process key surgical operations and anatomical structures in different scenarios, helping to formulate annotation specifications and ensuring the unified standard of surgical image annotation. The semi-automatic annotation combined with manual correction and multi-level review in quality control ensures the high quality and consistency of the dataset from all aspects. When improving the annotation efficiency, the present application also ensures the annotation accuracy through manual correction and multi-level review. The multi-level review of the present application checks the annotation results from different angles, further reducing the error rate. This multi-link guarantee mechanism effectively makes up for the errors that may be caused by reasons such as the limitations of training data or algorithm defects of the surgical image annotation model, making the accuracy of the final annotation result higher and being able to more accurately reflect the key information in the surgical image.

[0018] In practical applications, the present application can not only be used for the analysis of specific tissue structures, but also provide a solid data foundation for various surgical research and application scenarios. Taking minimally invasive gastric cancer surgery as an example, the present application can contribute to the research in this field, train a more accurate surgical image model through the accurately annotated surgical image dataset, and thus promote the safety and accuracy of surgical operations. The present application can be extended to the image processing of other various surgical operations to meet the needs of image analysis in different surgical scenarios and provide strong support for the development of the entire surgical field.

[0019] For the technical solution of the present application, the surgical image processing solution provided by the embodiments of the present application can also be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with a surgical image processing method, etc.). The above-mentioned chips introduced in the above embodiments can also be installed in these electronic devices. Or, these electronic devices can also install a service program for executing the surgical image processing solution.

[0020] Figure 1 A flowchart of a surgical image processing method provided by an embodiment of the present application is shown as follows Figure 1 As shown, the method includes the following steps:

[0021] 101. Collect the surgical image to be processed and identify the target surgical scenario corresponding to the surgical image;

[0022] 102. Extract the first image frame from the surgical image;

[0023] 103. Use the surgical image annotation model to perform anatomical structure recognition and preliminary annotation on the first image frame to obtain a second image frame;

[0024] 104. Push the second image frame to the client for review and receive the review result of the client to adjust the second image frame to obtain a third image frame;

[0025] 105. Construct a surgical image dataset based on the third image frame and the corresponding optimized annotation information.

[0026] In an embodiment of the present application, the first image frame at least includes: key surgical operations in the target surgical scenario and the corresponding surgical space image. Exemplarily, in a minimally invasive gastric cancer surgery, when in the target surgical scenario of dissecting the left gastroepiploic vascular region, the key surgical operations are to expose, dissect, and free the left gastroepiploic artery (LGeA) and the left gastroepiploic vein (LGeV). In the first image frame at this time, a clear picture of the surgical instrument performing delicate operations on the left gastroepiploic vascular region will be presented. The surgical space image shows the position of this region in the entire surgical field of view, and surrounding relevant tissue organs can be seen, such as parts of the greater curvature of the stomach and parts of the spleen, providing a spatial reference for the doctor to help understand the spatial relationship of the surgical operation.

[0027] In another example, in the target surgical scenario of separating the cystic triangle in a cholecystectomy, the key surgical operations are to separate the cystic artery and the cystic duct. The first image frame will present the surgical instrument precisely separating in the cystic triangle region, and the surgical space image shows the positional relationship between the cystic triangle and surrounding structures such as the liver and bile ducts. The doctor can judge the course of the cystic artery and the cystic duct and their adjacency to surrounding tissues from the image to ensure the safety and accuracy of the surgical operation.

[0028] In this way, the first image frame containing the key surgical operation and the corresponding surgical space image is extracted, so that the subsequent processing can accurately focus on the core links in the operation. This avoids the processing of a large amount of irrelevant image information, saves computing resources and time costs, and improves the pertinence of the analysis of key surgical steps. For example, in a complex surgical video, there may be many transitional images or non-critical operation images. Only extracting the image frames where the key information is located can make the subsequent annotation and analysis work more efficient. The surgical space image provides the spatial background in which the key surgical operation occurs, which helps doctors, medical researchers, etc. to better understand the actual situation of the surgical operation. By observing the positional relationship of each tissue and organ in the surgical space image, the sequence and potential risks of the surgical operation can be more clearly grasped. For example, in the above-mentioned gastric cancer surgery, the doctor can understand the positional relationship between the left gastroepiploic vessel and the surrounding tissues through the surgical space image to avoid accidentally injuring other important structures when dissecting the blood vessels.

[0029] When annotating the first image frame, the annotators can more accurately annotate the key anatomical structures because they have both an intuitive display of the key surgical operations and the surgical spatial images to assist in understanding the positional relationship of the anatomical structures. For example, when annotating the left gastroepiploic vessel, the surgical spatial images can help the annotators clearly define the starting and ending points of the vessel and the boundary with the surrounding tissues, thereby improving the accuracy of the annotation.

[0030] The surgical image dataset constructed based on the first image frame, when used to train the surgical image model, can enable the model to learn more representative and valuable features, learn important patterns and laws in the surgical process from key surgical operations and surgical spatial relationships, and improve the recognition and analysis capabilities of surgical images.

[0031] In the embodiment of the present application, the second image frame is annotated with initial annotation information of key anatomical structures corresponding to the key surgical operation.

[0032] For example, in the right gastric regional vascular clearance scenario of minimally invasive gastric cancer surgery, the key surgical operation is to process the right gastric artery (RGA), right gastric vein (RGV) and proper hepatic artery (PHA). After the second image frame is processed by the surgical image annotation model, these key anatomical structures will be initially annotated. For example, the right gastric artery may be outlined with a red line, and the word "RGA" is marked next to it, indicating that this is the right gastric artery; the right gastric vein is marked with a blue line and marked "RGV"; the proper hepatic artery is also outlined with a red line and marked "PHA". The annotated lines will follow the edges of the blood vessels as much as possible to roughly outline their position and shape in the image. Although it is an initial annotation, it can clearly show the position of key blood vessels in the surgical field of view.

[0033] During a partial liver resection surgery, when performing the key surgical operation of removing the diseased tissue, the second image frame will label the diseased tissue of the liver, the surrounding normal liver tissue, and important intrahepatic blood vessels, etc. For example, the diseased tissue may be filled with yellow to represent it, the normal liver tissue is filled with green, and the branches of the portal vein in the liver are labeled with blue lines and marked as "PV branch". Through such labeling, different anatomical structures are initially distinguished, providing a basis for subsequent review and precise labeling.

[0034] Thus, the initial labeling information of the second image frame is automatically generated by the surgical image labeling model, greatly saving the time of manual labeling. For a large number of surgical images, the model can quickly analyze and initially label the key anatomical structures. Compared with manual frame-by-frame labeling, the labeling speed is greatly improved, providing an efficient starting point for subsequent manual review and optimization.

[0035] It provides a unified labeling starting point and standard mode for all image frames. Since the labeling is based on the same model, the initial labels between different image frames are consistent. This helps with subsequent manual review and processing. The labeling personnel can optimize and adjust on a relatively unified basis, avoiding the differences and confusion caused by different manual labeling starting points.

[0036] In addition, during the training process, the labeling model has learned a large amount of surgical image data and the corresponding anatomical structure labeling information. Therefore, when generating the initial label, it can pre-consider features such as the shape, position, and relationship with surrounding tissues of the anatomical structure, reducing the labeling errors caused by subjective judgment differences.

[0037] In the embodiments of the present application, the third image frame is at least labeled with optimized labeling information for key anatomical structures. Exemplarily, in the scenario of vascular dissection in the upper pancreatic region of a minimally invasive gastric cancer surgery, key anatomical structures such as the left gastric artery (LGA), left gastric vein (LGV), common hepatic artery (CHA), and splenic artery (SA) are involved. In the initial label of the second image frame, the surgical image labeling model may not distinguish the branch parts of the left gastric artery and the common hepatic artery clearly enough, and the labeling lines do not fit well with the actual edges of the blood vessels. In the third image frame, after being reviewed and adjusted by professional doctors, the labeling line of the left gastric artery is drawn more precisely along the blood vessel wall, clearly showing its branch direction; the labeling of the common hepatic artery is also optimized, with a clearer distinction from the boundaries of other surrounding blood vessels, and at the same time, the labeling color is more prominent for accurate identification during subsequent viewing and analysis.

[0038] In liver tumor resection surgery, the annotation of the tumor tissue in the second image frame roughly outlines the contour of the tumor, but the annotation of the transition area between the tumor and the surrounding normal liver tissue is not detailed enough. In the third image frame, the annotator (doctor) uses professional knowledge and experience to more carefully depict the tumor boundary and also annotate and distinguish the gradual change area between the tumor and the normal tissue. For example, different shades of color are used to represent different infiltration degrees of the tumor, making the annotation information of the tumor tissue more complete and accurate.

[0039] In this way, the initial annotation of the second image frame is optimized through manual review, significantly improving the accuracy of the annotation of key anatomical structures. Professionals, relying on their rich medical knowledge and clinical experience, can discover and correct the biases and deficiencies in the initial annotation of the model. For example, when annotating some complex anatomical structures or parts with individual differences, the model may make misjudgments or inaccurate annotations, while manual review can ensure that the annotation results highly match the actual anatomical situation, providing more reliable data support for subsequent surgical image analysis, surgical simulation, medical research, etc.

[0040] The optimized annotation information makes the third image frame more reliable when used to construct a surgical image dataset. In the process of constructing the dataset, high-quality annotated data is the key to ensuring the training effect of the surgical image model. The optimized annotation can accurately reflect the anatomical structure and operation information during the surgery, helping to train a more accurate surgical image model. For example, when training a model for surgical navigation, accurate annotated data can enable the model to better understand the anatomical structure relationship during the surgery, providing more accurate navigation prompts for the surgical doctor and improving the safety and success rate of the surgery.

[0041] In the embodiments of this application, the surgical image dataset is used to train and optimize the surgical image model.

[0042] Taking minimally invasive gastric cancer surgery as an example, a large number of surgical images are collected. After the aforementioned steps of acquisition, scene recognition, image frame extraction, annotation, etc., a surgical image dataset is constructed. The dataset contains surgical images of different patients, covering various key surgical scenarios, such as dissection of the left gastroepiploic vascular region, dissection of the vascular region of the greater curvature of the stomach, etc. Each image in the scenario has corresponding annotation information, such as accurate annotations of key anatomical structures like the left gastroepiploic artery and the right gastroepiploic vein. These annotated images, that is, the third image frame and its optimized annotation information, together form the dataset used to train and optimize the surgical image model.

[0043] For liver surgery, images during the operation are also collected, and image frames are extracted from different surgical procedures such as partial liver resection and tumor ablation. In the annotation process, key anatomical structures such as different parts of the liver, tumor location, surrounding blood vessels and bile ducts are annotated. For example, the boundaries of liver segments, the demarcation line between tumors and normal liver tissue, and the courses of hepatic veins and portal veins are clearly marked. These images with detailed annotations constitute the liver surgery image dataset, which is used for the training and optimization of image models related to liver surgery.

[0044] Thus, a rich and accurately annotated surgical image dataset provides a large number of effective learning samples for the surgical image model. By learning these data, the model can more accurately identify key anatomical structures, surgical operation steps, etc. in surgical images. For example, when training an image model for minimally invasive gastric cancer surgery, accurate annotation of various blood vessels and tissue information in the dataset enables the model to learn the characteristics and differences of different anatomical structures, so that it can more accurately identify key structures such as the left gastroepiploic artery and the left gastric vein in practical applications, improving the recognition accuracy.

[0045] A diverse surgical image dataset, containing images of different patients and different surgical scenarios, helps the model learn more extensive features and patterns, enhancing its generalization ability. This means that the model can not only perform well on the training data, but also accurately analyze new and unseen surgical images. For example, in the training of a liver surgery image model, the dataset covers images of tumors of different sizes and locations and different liver morphologies. After learning these data, the model can better adapt to liver surgery images in various actual clinical scenarios, rather than being limited to specific types of images in the training set.

[0046] The surgical image dataset contains images and annotation information corresponding to key surgical operations. By learning these contents, the model can better understand the surgical process and operation logic. For example, in the constructed gallbladder resection surgical image dataset, the images annotate the anatomical structure changes involved in each operation step from separating the triangle of Calot to removing the gallbladder. The model can learn the sequence and spatial relationship of these operations during the training process, so that it can more accurately analyze and predict surgical operations in practical applications, providing intelligent assistance for the surgery.

[0047] The model trained and optimized with the surgical image dataset can provide strong support for the intelligent development of surgery. For example, in a surgical navigation system, the trained model can be used to analyze surgical images in real time, providing prompts of anatomical structures and surgical operations for doctors, helping doctors perform surgical operations more accurately, and improving the safety and precision of the surgery.

[0048] Further optionally, in the embodiments of the present application, the initial annotation information and / or the optimized annotation information at least include: mask range information for indicating the position range of each structure, color coding information for indicating the type of each structure, dark area annotation information, tiny area annotation information, and multi-region annotation information. The following examples introduce the specific content of the above embodiments:

[0049] Mask range information: It corresponds to the requirements in the annotation range definition and is used to accurately indicate the boundary position of each key anatomical structure in the image. For example, in an image of a minimally invasive gastric cancer surgery, for the left gastroepiploic artery, the mask range information will outline the specific distribution range of the blood vessel in the image with a specific graphic (such as a polygon), strictly following the principle of only annotating the exposed and recognizable boundaries of key anatomical structures, and clearly defining the boundary between the blood vessel and the surrounding tissues.

[0050] Color coding information: According to the annotation color coding rules, different types of anatomical structures are assigned specific colors. The color coding information reflects this corresponding relationship in the annotation information. For example, when annotating the liver, the color coding information indicates that it is green. In this way, when viewing the annotated image, different types of structures can be quickly distinguished by color, facilitating intuitive identification for annotators and subsequent users of the annotated data.

[0051] Dark area annotation information: For the special handling requirements of dark area annotation, when there are recognizable structures in a dark area (such as the edge of an image with low brightness) or an area slightly covered by smoke, blood, fat, or fascia, the dark area annotation information will record the annotation situation of these structures in the dark area. For example, if there is a small segment of recognizable blood vessel in the darker area at the edge of a surgical image, the dark area annotation information will describe the position, orientation, and other relevant annotation details of the blood vessel in the dark area.

[0052] Tiny area annotation information: When a structure is only partially visible in a very small area, the tiny area annotation information will record the annotation situation of these tiny areas. For example, in a surgical image, a certain key blood vessel may only have a very small part exposed due to being blocked by surrounding tissues. The tiny area annotation information will record in detail the position, shape, and other information of this tiny visible part to ensure accurate annotation even for small areas.

[0053] Multi-region annotation information: If the same key structure spans multiple small regions in a single frame, or multiple key structures appear in a single frame, the multi-region annotation information will record the annotation situation of these structures in different regions and their relationships. For example, in a complex surgical scene image, a part of the stomach may appear in the upper left and lower right corners of the image respectively. The multi-region annotation information will accurately record the annotation information of the stomach in these two regions and indicate that they belong to the same stomach structure.

[0054] The above-mentioned annotation information helps to accurately reflect various features of key anatomical structures in the image. The mask range information precisely defines the structural boundaries, the color-coding information intuitively differentiates the structural types, and the annotation information regarding the handling of special cases, such as dark areas, tiny areas, and multi-region annotations, covers the annotation requirements in complex situations, enabling the annotations to best conform to the actual anatomical structure conditions and avoiding omissions or incorrect annotations. In addition, the unified annotation specifications and rich annotation information content enable different annotators to follow the same standards when dealing with the same or similar surgical images. For example, no matter which annotator processes the minimally invasive gastric cancer surgery images, the right gastric artery will be color-coded in red and its boundary will be accurately annotated according to the requirements of the mask range information, thus ensuring the consistency of the annotation results among different images and different annotators.

[0055] For those who will subsequently use these annotated data to train and optimize the surgical image model and conduct surgical image analysis, the rich annotation information provides a comprehensive and accurate data foundation. The model can learn the positions, types, and features in special cases of different structures, thereby more accurately identifying and analyzing surgical images. For example, when training a model for identifying key surgical steps, the multi-region annotation information can help the model understand the relationship between key structures and surgical steps when they appear in different positions, improving the model's analysis ability.

[0056] As an optional embodiment, in 101, collect the surgical images to be processed and identify the target surgical scenario corresponding to the surgical images.

[0057] In an optional example, assume that the surgical images are extracted from: the complete video of minimally invasive distal gastrectomy with D2 lymph node dissection. These videos are recorded by a laparoscopic / robotic camera system, with a high resolution of 1920×1080 pixels and the MPEG-4 format, providing a high-quality data foundation for image collection. In 101, the images can be extracted frame by frame from these surgical videos to obtain the surgical images to be processed.

[0058] Specifically, according to the clinical guidelines of minimally invasive distal gastrectomy with D2 lymph node dissection, sort out each key operation step and the corresponding scenario of this surgery. For example, the surgical process can be systematically divided into eight key surgical scenarios, such as the dissection of the left gastroepiploic vascular area and the dissection of the vascular area of the greater curvature of the stomach. By analyzing the features such as surgical operations, anatomical structures, and instrument use shown in the surgical images, match each surgical image to be processed with the sorted surgical scenarios, thereby identifying the target surgical scenario corresponding to the surgical image. For example, if the image shows the dissection of the left gastroepiploic artery and the left gastroepiploic vein, it can be determined that the target surgical scenario corresponding to this image is the dissection of the left gastroepiploic vascular area.

[0059] Further optionally, the key anatomical structures involved in different surgical scenarios are different. In practical applications, the types of key anatomical structures include, but are not limited to, the following parts: stomach, peritoneum, omentum, diaphragm, spleen, liver, small intestine, colon, pancreas, blood vessels, lymph nodes, etc. Among these key anatomical structures, the identification and treatment of blood vessels are particularly important, especially the key blood vessels, which play a decisive role in the safety and effectiveness of the surgery.

[0060] The following will introduce in detail the classification of surgical scenarios for minimally invasive distal gastrectomy with D2 lymph node dissection.

[0061] Minimally invasive distal gastrectomy with D2 lymph node dissection is an important surgical procedure for treating gastric cancer, and its clinical guidelines specify in detail the standard procedures and operation specifications of the surgery. Through in-depth analysis of the surgical guidelines, a professional medical team can sort out the detailed process of the entire surgery, from the initial steps of the surgery to the operations at each stage, and then to the final finishing work, and clarify them one by one. On this basis, the key operation steps are further identified. These key operation steps are the core links for the success of the surgery and play a decisive role in the safety and effectiveness of the surgery. For example, during the process of lymph node dissection, how to accurately identify and separate the tissues around the lymph nodes to avoid damaging important blood vessels and nerves is a key operation step.

[0062] Based on the analysis results of the surgical guidelines, the entire surgical process is systematically divided into eight key surgical scenarios. Each scenario has a clear operation goal and a specific anatomical area, covering specific operations on key anatomical structures. This division method makes the surgical process clearer and helps to conduct targeted research, teaching, and subsequent image processing and analysis on each stage of the surgery.

[0063] Types of key anatomical structures: This surgery involves a variety of important anatomical structures, including the stomach, peritoneum, omentum, diaphragm, spleen, liver, small intestine, colon, pancreas, blood vessels, lymph nodes, etc. These structures are interrelated during the surgical process, and their accurate identification and treatment are crucial for the success of the surgery. For example, as the main object of the surgery, the resection range and method of the stomach need to be determined according to the lesion situation and the relationship with the surrounding anatomical structures; while tissues such as the peritoneum and omentum may need to be separated or protected during the surgery to ensure a clear surgical field and avoid postoperative adhesions and other complications.

[0064] Distribution of key blood vessels: Among the many key anatomical structures, the identification and treatment of blood vessels are particularly critical, especially 15 key blood vessels, which play a decisive role in the safety and effectiveness of the surgery. Exemplarily, the following is the specific distribution of these key blood vessels in the eight surgical scenarios:

[0065] Scenario 1 (Dissection of the left gastroepiploic regional lymph nodes): This scenario mainly involves the dissection of the left gastroepiploic artery (LGeA) and the left gastroepiploic vein (LGeV). The left gastroepiploic vessels are responsible for supplying blood to the left part of the greater curvature of the stomach. During the operation, these vessels need to be carefully dissected and ligated to avoid complications such as bleeding.

[0066] Scenario 2 (Dissection of the greater curvature regional lymph nodes): The operation mainly targets the gastric greater curvature vascular arcade (GCVs). The gastric greater curvature vascular arcade is a group of vascular structures on the greater curvature side of the stomach and plays an important role in the blood supply to the greater curvature of the stomach. During the operation, the vessels in this area need to be carefully dissected to remove the potentially metastatic lymph nodes.

[0067] Scenario 3 (Dissection of the subpyloric regional lymph nodes): The focus is on the treatment of the right gastroepiploic artery (RGeA) and the right gastroepiploic vein (RGeV). The vascular anatomy in the subpyloric area is relatively complex, and the right gastroepiploic vessels supply blood to the pylorus and the beginning of the duodenum. When dissecting this area, precise operation is required to avoid damaging the vessels and causing bleeding, while ensuring the complete removal of the lymph nodes in this area to reduce the risk of tumor metastasis and recurrence.

[0068] Scenario 4 (Exposing the gastroduodenal artery): The main goal is to expose the gastroduodenal artery (GDA) as an anatomical landmark for lymph node dissection. The gastroduodenal artery is an important vessel connecting the common hepatic artery and the superior pancreaticoduodenal artery. When performing operations in this area, special care is needed because once the gastroduodenal artery is damaged, it may cause severe bleeding, affecting the smooth progress of the operation and the prognosis of the patient.

[0069] Scenario 5 (Dissection of the right gastric regional lymph nodes): It involves the right gastric artery (RGA), the right gastric vein (RGV), and the proper hepatic artery (PHA). The right gastric artery supplies blood to the right part of the lesser curvature of the stomach, while the proper hepatic artery is one of the main blood supply vessels of the liver. In this scenario, these vessels need to be accurately identified and processed to ensure sufficient resection of the stomach while avoiding damage to important vessels such as the proper hepatic artery to maintain the normal blood supply of the liver.

[0070] Scenario 6 (Dissection of the superior pancreatic regional lymph nodes): It involves the left gastric artery (LGA), the left gastric vein (LGV), the common hepatic artery (CHA), and the splenic artery (SA). The vascular structure in the superior pancreatic area is complex, and these vessels are intertwined, supplying blood to multiple important organs such as the stomach, pancreas, and spleen. When dissecting this area, fine operation is required to accurately identify each vessel, avoid accidental injury, and ensure the safety and effectiveness of the operation.

[0071] Scenario 7 (Dissection of Group 12a Lymph Nodes): The main focus is on the portal vein (PV) and the inferior vena cava (IVC). The Group 12a lymph nodes are located around the portal vein. When dissecting the lymph nodes in this area, the operation needs to be carried out without damaging the portal vein and the inferior vena cava. The portal vein is responsible for collecting the blood reflux from most of the organs in the abdominal cavity back to the liver, while the inferior vena cava is the largest vein in the human body, responsible for bringing the venous blood from the lower limbs, pelvis, and abdomen back to the heart. Once these two major blood vessels are damaged, extremely serious consequences will occur, so great care is needed during the operation.

[0072] Scenario 8 (Dissection of Lymph Nodes in the Lesser Curvature Area of the Stomach): The operation is mainly carried out on the lesser curvature vascular arch (LcVs) of the stomach. The lesser curvature vascular arch provides blood supply to the lesser curvature of the stomach. In this scenario, the lesser curvature vascular arch needs to be carefully dissected and treated to ensure the complete removal of the metastatic lymph nodes that may exist in this area.

[0073] This detailed division and definition of the surgical scenarios and key anatomical structures provide a clear and accurate framework for the standardized operation of surgery, medical education, and related medical research, which helps to improve the surgical quality and the treatment effect of patients.

[0074] As an alternative embodiment, in 102, continuous image frames are extracted from the surgical images according to a preset frequency and image specifications; the extracted continuous image frames are frame-screened to screen out the continuous image frames in which the clarity, surgical operation process images, and anatomical structure images meet the preset screening conditions as the first image frames.

[0075] Specifically, the preset frequency and image specifications are to ensure the consistency and usability of the extracted image frames. Exemplarily, the preset frequency is 5 frames per second, and the image specifications are: PNG format images with a resolution of 1920×1080 pixels. Using a video processing tool (such as the FFmpeg framework), continuous image frames are extracted from the surgical images (i.e., the frames of the surgical video) at a preset frequency of 5 frames per second (which can be set according to actual needs). At the same time, ensure that the extracted image frames meet the resolution requirements of 1920×1080 pixels and the format is PNG (or other suitable formats for processing) to meet the needs of subsequent processing and analysis.

[0076] Due to certain differences in the recording environment and equipment of the surgical video, some of the extracted image frames may have clarity problems such as blurring and excessive noise. Therefore, it is necessary to set the preset screening conditions for clarity. For example, indicators such as the contrast and sharpness of the image need to reach a certain threshold. The clarity of the continuous image frames is evaluated through image processing algorithms, and the image frames that do not meet the clarity standards are removed.

[0077] Minimally invasive distal gastrectomy with D2 lymph node dissection has clear key operative steps, such as dissecting, exposing, clamping, and transecting key blood vessels, etc. The preset screening conditions require that the image frames must contain clear images of these key surgical procedures. For example, for the image frames of the gastric right regional blood vessel dissection scenario, the images showing the operations on blood vessels such as the right gastric artery and the right gastric vein need to be displayed, otherwise the image frame will be excluded.

[0078] Each surgical scenario involves specific key anatomical structures, such as the stomach, blood vessels, lymph nodes, etc. The key anatomical structures corresponding to the scenario must be clearly shown in the image frames. For example, in the image frames of the superior pancreatic regional blood vessel dissection scenario, key anatomical structures such as the left gastric artery, the left gastric vein, the common hepatic artery, and the splenic artery should be clearly visible, otherwise it does not meet the screening conditions.

[0079] After the above screening, the consecutive image frames that meet the preset screening conditions in terms of clarity, surgical procedure images, and anatomical structure images are used as the first image frames, and these first image frames will provide a high-quality data basis for subsequent annotation and dataset construction.

[0080] In the step of frame screening for the extracted consecutive image frames, further optionally, the extracted image frames are screened according to two core criteria. One is image clarity, ensuring that the image has no problems such as blurring, excessive noise, or poor lighting, so as to clearly present the surgical scenario. The other is content integrity, requiring that each frame of the image must cover the relevant operation processes of the key surgical scenario and its corresponding key anatomical structures. For example, in minimally invasive distal gastrectomy with D2 lymph node dissection, for the operations on key blood vessels, key steps such as dissection, exposure, clamping, and transection should be clearly shown in the image. For example, if a frame of the image is the left gastroepiploic blood vessel regional dissection scenario, the image should clearly show the surgical instrument performing dissection or clamping on the left gastroepiploic artery or the left gastroepiploic vein. This step can be assisted by a machine learning model for screening and then submitted to a junior surgeon for review, or it can be manually reviewed by a junior surgeon, and there is no limitation here.

[0081] Furthermore, in order to improve the accuracy of image classification, the selected images will be submitted to expert surgeons for a second review of the rationality of image selection. The experts will consider various aspects such as the overall surgical process and the typicality of anatomical structures to ensure that the selected images can accurately reflect the key surgical scenarios and operations, avoid misselection due to insufficient experience of junior doctors, and ensure the reliable quality of the images finally used for subsequent processing. This step can also be assisted by a model trained from an expert experience database for scoring, so as to further reduce the review workload of expert surgeons, improve the review efficiency, and reduce the burden of manual review.

[0082] In an optional embodiment of the present application, algorithms such as structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) can be used to quantify image clarity. SSIM can measure the similarity of two images in structure, brightness and contrast, while PSNR evaluates image quality by calculating the power ratio of signal to noise. By setting appropriate thresholds, such as SSIM needs to be higher than 0.8 and PSNR is greater than 30dB, the algorithm automatically screens out image frames with standard clarity. For example, the Canny edge detection algorithm is used to identify edge information in an image. If the edge is clear and rich, it indicates that the image is rich in details and has high clarity. At the same time, the image texture is analyzed using methods such as grayscale co-occurrence matrix. Images with clear and regular textures also mean higher clarity. Based on these analysis results, automatic screening of image clarity is achieved. For example, a convolutional neural network (CNN) model is constructed, such as using target detection frameworks such as Faster R-CNN and YOLO. Taking minimally invasive distal gastrectomy with D2 lymph node dissection as an example, a large amount of labeled image data containing different key surgical scenes and operations is used to train the model. During the training process, the model learns the features of key anatomical structures (such as various blood vessels) and surgical instruments in different scenarios, so that it can automatically detect whether there are key surgical scenes and corresponding operations in new image frames. For example, when it is detected that a surgical instrument is performing a clamping-like action on a target marked as the left gastroepiploic artery in the image, the image frame is determined to contain a key surgical operation. Then, the key points on the surgical instruments and key anatomical structures are first identified through a key point detection algorithm, such as OpenPose. Then, based on the motion trajectory and relative position changes of these key points, the action recognition algorithm is used to determine whether key operations such as dissection, exposure, clamping, and disconnection have occurred. For example, when it is detected that two key points representing vascular clamps are gradually approaching and closing, and are at the marked blood vessel position, it is determined to be a clamping operation. Therefore, multiple deep learning models with different structures or training data are used to analyze the image. For example, a model based on the ResNet architecture and a model based on the VGG architecture, if multiple models have consistent judgment results on the same image, it indicates that the screening result of the image is relatively reliable; if there is a disagreement, the image is marked for further manual review. For example, an image quality assessment database is established to store previously selected high-quality images and their feature information. For newly selected images, the similarity and feature matching between them and the images in the database are calculated. If the similarity and matching are within a reasonable range, the image selection is considered reasonable; otherwise, manual review is performed.

[0083] As an optional embodiment, it is assumed that the surgical image annotation model includes at least: an image encoder, a hint encoder, and a mask encoder. In combination with the model structure shown, the following is a detailed introduction to the surgical image annotation model in the picture in combination with the previous steps:

[0084] The image encoder is the starting part of the surgical image annotation model and is responsible for reducing the dimension of the input surgical image (i.e., the first image frame). It can extract key feature information from the original high-dimensional surgical image and transform it into image embedding features. These image embedding features are a compact and representative representation of the surgical image, containing important visual information of the image, but with relatively low dimensions, facilitating subsequent processing.

[0085] For example, for a complex minimally invasive gastric cancer surgery image, the image encoder will analyze various visual elements such as pixel information, texture, and color in the image, and then through a series of operations such as convolution and pooling, compress the originally large amount of image data into a set of image embedding features. These features may be a vector, where each element contains partial feature information of the image, such as the texture features and shape features of certain regions in the image.

[0086] The hint encoder is mainly used to process the contour information corresponding to the candidate anatomical structures in the identified first image frame. It converts this contour information, specifically point hint information and box hint information, into corresponding multi-dimensional hint embedding features. The point hint information and box hint information describe the outer contour of the candidate anatomical structure from different angles. The point hint information may be some key point coordinates, and the box hint information may be the bounding box information around the candidate anatomical structure. The hint encoder converts this information into a multi-dimensional feature representation that the model can understand and process, so as to fuse and further process with the image embedding features.

[0087] Suppose the left gastroepiploic artery is identified as a candidate anatomical structure in the surgical image, and its contour may be represented by some key points (such as the starting point and bending point of the artery), and the coordinates of these key points constitute the point hint information; at the same time, an arterial region can be roughly framed by a rectangular box, and the information of this box is the box hint information. The hint encoder will encode these point coordinates and box information to generate multi-dimensional hint embedding features, such as a vector containing multiple dimensions, and each dimension contains specific feature encodings of this hint information.

[0088] The mask encoder includes at least multiple layers of decoders, and each layer of decoder includes at least components such as self-attention layer, cross-attention layer, multi-layer perceptron, and cross-attention calculation layer. Its main function is to fuse and process the image embedding features and the corresponding multi-dimensional hint embedding features to generate the corresponding candidate mask image. The self-attention layer is used to capture the internal dependencies of the features, the cross-attention layer is used to establish the association between the image embedding features and the hint embedding features, the multi-layer perceptron further performs non-linear transformation and processing on the features, and the cross-attention calculation layer interacts and calculates between features at different levels, and finally generates a candidate mask image that can represent the position and shape of the candidate anatomical structure in the surgical image.

[0089] Taking the left gastroepiploic artery as an example, the mask encoder will synthesize the embedding features of the entire surgical image output by the image encoder and the multi-dimensional cue embedding features of the left gastroepiploic artery output by the cue encoder. Through the processing of the multi-layer decoder, the self-attention layer may focus on the internal relationships between the artery-related regions in the image, and the cross-attention layer will accurately match and associate the cue information of the artery with the corresponding regions in the image. The multi-layer perceptron performs complex feature transformation and combination, and finally generates a candidate mask image. This candidate mask image is a binary image, where the region where the artery is located is marked as the foreground (e.g., pixel value is 1), and other regions are the background (pixel value is 0), thus clearly outlining the approximate contour and position of the left gastroepiploic artery in the surgical image.

[0090] Based on the above model structure, first, the surgical image (the first image frame) enters the image encoder for dimensionality reduction processing to obtain image embedding features. Then, the model identifies the contour information corresponding to the candidate anatomical structure in the first image frame to obtain point cue information and box cue information. These information enter the cue encoder and are converted into multi-dimensional cue embedding features. Then, the image embedding features and the multi-dimensional cue embedding features are input into the mask encoder. After complex processing by the multi-layer decoder, a candidate mask image is generated and pushed to the user for confirmation. If the user feedbacks a confirmation instruction, the candidate mask image is used as the second image frame; if no confirmation instruction is received, the contour information of the candidate anatomical structure is returned for re-identification, and the subsequent process is continued until a candidate mask image recognized by the user is generated as the second image frame.

[0091] Through the collaborative work of the image encoder, cue encoder, and mask encoder, this surgical image annotation model can automatically identify and preliminarily annotate the anatomical structure of the surgical image, providing a basis for subsequent manual review and correction steps, helping to improve the efficiency and accuracy of surgical image annotation, and providing strong support for the analysis and processing of surgical images.

[0092] Furthermore, based on the above model structure, in 103, the first image frame is subjected to anatomical structure recognition and preliminary annotation using the surgical image annotation model to obtain the second image frame, including the following steps:

[0093] 201. Input the first image frame into the image encoder for dimensionality reduction processing to obtain image embedding features;

[0094] 202. Identify the contour information corresponding to the candidate anatomical structure in the first image frame to obtain point cue information and box cue information for the candidate anatomical structure; the point cue information and box cue information are respectively used to describe the external contour of the candidate anatomical structure;

[0095] 203. Use a prompt encoder to convert point prompt information and box prompt information into corresponding multi-dimensional prompt embedding features;

[0096] 204. Input the image embedding features and the corresponding multi-dimensional prompt embedding features into a mask encoder to generate corresponding candidate mask images, and push the candidate mask images to the user for confirmation; wherein, the mask encoder includes at least multiple decoder layers, and each decoder layer includes at least a self-attention layer, a cross-attention layer, a multi-layer perceptron, and a cross-attention calculation layer;

[0097] 205. If a confirmation instruction feedback by the user is received, use the candidate mask image as the second image frame;

[0098] 206. If a confirmation instruction feedback by the user is not received, jump to the step of identifying the contour information corresponding to the candidate anatomical structure in the first image frame, and continue to execute the processing flow corresponding to the prompt encoder and the mask encoder.

[0099] In step 201, the first image frame is input into an image encoder for dimensionality reduction processing to obtain image embedding features. In principle, the main function of the image encoder is to perform dimensionality reduction processing on the input first image frame. For example, assume that the first image frame is an image of a minimally invasive gastric cancer surgery, containing anatomical structures such as the stomach and blood vessels, with a resolution of 1920×1080. Input this image into the image encoder, and through a series of convolutional operations or self-attention mechanisms (depending on the specific architecture of the encoder), convert it into a low-dimensional vector representation, such as a 1×256 vector. This vector contains the key feature information of the image, which can be regarded as a compressed representation of the original image, retaining the important information about the key anatomical structures and surgical scenes in the image, but with greatly reduced dimensions, facilitating subsequent processing.

[0100] In step 202, identify the contour information corresponding to the candidate anatomical structure in the first image frame to obtain point prompt information and box prompt information for the candidate anatomical structure. The main focus of this step is to obtain the contour information of the possible candidate anatomical structures in the first image frame. These contour information are the basis for generating annotation prompts. The point prompt information can be to select some key points on the contour of the anatomical structure, such as the starting point, ending point, and branching points of blood vessels. Through these points, the model can better understand the shape and position of the structure; the box prompt information is to draw a bounding box around the structure to enclose the candidate anatomical structure. These two types of prompt information describe the outer contour of the candidate anatomical structure from different angles and provide the basis for subsequent annotation.

[0101] For example, for the left gastroepiploic artery in the above gastric cancer surgery image, the annotator or the automatic algorithm can mark a point at the starting position of the artery as point prompt information, and at the same time draw a rectangular box in the area where the artery is located as box prompt information to help the model determine the position and approximate range of the artery. These prompt information can be added manually or automatically generated by an algorithm. For example, an edge detection algorithm based on the image is used to find the contour of the blood vessel, and then these prompt information are determined.

[0102] In step 203, a prompt encoder is used to convert the point prompt information and the box prompt information into corresponding multi-dimensional prompt embedding features. In principle, the point prompt information and the box prompt information are simple representations on the two-dimensional image plane. To enable the model to better utilize this information, it is necessary to convert them into multi-dimensional prompt embedding features. The prompt encoder will map this information into a higher-dimensional space, converting the simple point and box information into embedding features that the model can understand and process. This process involves complex mapping functions and encoding techniques, encoding different types of prompt information into vector forms for subsequent combination with the image embedding features for processing.

[0103] For example, for the point prompt and box prompt of the left gastroepiploic artery obtained previously, the prompt encoder may convert them into a 256-dimensional vector representation. This vector contains the coordinate information of the point, the position and size information of the box, as well as their relative position relationships, etc., converting the original spatial position information into a feature representation suitable for model processing so that the model can better combine this information with the image embedding features.

[0104] In step 204, the image embedding features and the corresponding multi-dimensional prompt embedding features are input into the mask encoder to generate a corresponding candidate mask image, and the candidate mask image is pushed to the user for confirmation. In principle, the mask encoder is the core part of the entire annotation model, including multiple layers of decoders. Each layer of the decoder includes at least a self-attention layer, a cross-attention layer, a multi-layer perceptron, and a cross-attention calculation layer. These components work together, using the image embedding features and the multi-dimensional prompt embedding features to gradually generate the candidate mask image. The self-attention layer and the cross-attention layer can capture the relationships between different parts, and the multi-layer perceptron performs non-linear transformations. The finally generated candidate mask image is the preliminary annotation result of the candidate anatomical structure. The generated candidate mask image will show different colors or gray values to distinguish different anatomical structures or backgrounds.

[0105] For example, when processing gastric cancer surgery images, the image embedding features and the multi-dimensional hint embedding features of the left gastroepiploic artery are input into the mask encoder. After being processed by multiple decoders, a candidate mask image is generated. In this image, the area where the left gastroepiploic artery is located may be marked with a specific color (such as red), indicating that this area is the preliminary annotation result of the left gastroepiploic artery. Then the candidate mask image is pushed to the user (who can be an annotator or a doctor) for the user to confirm whether the annotation is correct.

[0106] In step 205, if a confirmation instruction feedback from the user is received, the candidate mask image is used as the second image frame. In principle, when the user believes that the generated candidate mask image meets the expectations, the user will issue a confirmation instruction. This instruction indicates that the annotation result generated by the model has met the requirements of the user. In this case, the candidate mask image can be used as the second image frame and enter the next processing flow.

[0107] In step 206, if a confirmation instruction feedback from the user is not received, it jumps to the step of identifying the contour information corresponding to the candidate anatomical structure in the first image frame, and continues to execute the processing flows corresponding to the hint encoder and the mask encoder. Specifically, when the user believes that the candidate mask image is inaccurate, no confirmation instruction will be sent, and it will automatically return to step 202 to re-find the contour information of the candidate anatomical structure, adjust the point hint information and the box hint information, and then be processed again by the hint encoder and the mask encoder. This loop process can be iterated continuously until the user is satisfied, ensuring that the finally generated annotation information is accurate and reliable. For example, if the annotator finds that the annotation of the candidate mask image of the left gastroepiploic artery is inaccurate, perhaps the position or the range is incorrect, the annotator will start to identify the contour information again, re-generate the point hint and the box hint, re-generate the candidate mask image, and keep adjusting until the annotator is satisfied and confirms.

[0108] Through the collaborative work of the image encoder, the hint encoder, and the mask encoder, combined with the feedback from the user, the above steps achieve the structure recognition and preliminary annotation of the first image frame, forming the second image frame. This not only utilizes the automation ability of the model but also combines the professional judgment of the user, ensuring the accuracy and reliability of the annotation.

[0109] As an example, the following is a detailed introduction to the preliminary annotation using the annotation software (X-Anylabeling) and the Segment Anything Model (SAM) model fine-tuned with surgical images. In step 201, the first image frame is input into an image encoder composed of a pre-trained Vision Transformer (ViT) architecture for visual image processing. ViT processes the input surgical image and converts it into low-dimensional image embedding features. This dimensionality reduction process is achieved by dividing the image into multiple small patches, encoding each patch as a vector, and then through a series of operations such as self-attention mechanisms and multi-layer perceptrons, compressing the original high-dimensional image information into a low-dimensional feature representation, namely the image embedding.

[0110] For example, for a surgical image of 1920×1080, ViT will divide it into multiple small image patches (such as 16×16 patches). Each patch undergoes linear mapping and position encoding, and then passes through a multi-head self-attention layer and a feed-forward neural network layer. Finally, the entire image is transformed into a low-dimensional vector, such as a 1×256 vector. This vector contains rich feature information in the image, reducing the subsequent computational complexity while improving the feature extraction efficiency.

[0111] The annotator uses the X-Anylabeling software to perform annotation operations based on the anatomical structures in the first image frame. "Point prompts" and "box prompts" are auxiliary information provided by the annotator to the model to help the model better locate and understand key anatomical structures. Positive prompts (+) indicate that the area specified by the annotator belongs to the target anatomical structure. For example, the annotator clicks on a point on the left gastroepiploic artery and marks it as "+", indicating that the area where the point is located is part of the left gastroepiploic artery; negative prompts (-) indicate that the area where the point is located does not belong to the target anatomical structure and are used to exclude some possible confusing areas. For example, the annotator clicks on some areas in the image that are similar to blood vessels but not the left gastroepiploic artery and marks them as "-". And the "box prompt" is a rectangle or polygon box drawn by the annotator around the anatomical structure, macroscopically enclosing the scope of the target structure. For example, a rectangle box is drawn around the area where the left gastroepiploic artery is located to enclose the entire blood vessel and its branches, to help the model more quickly determine the approximate scope.

[0112] In step 203, the mask encoder converts the "point cues" and "box cues" given by the annotator into multi-dimensional cue embedding features. First, for the "point cues", which contain the coordinates (x, y) of the points and the positive / negative attributes, and for the "box cues" which contain the coordinates (x1, y1) and (x2, y2) of the upper left and lower right corners. The mask encoder performs position encoding based on these coordinates and attribute information. For example, these spatial position information are converted into a 256-dimensional vector embedding (cue embedding), which is used as one of the inputs to the model.

[0113] In step 204, the image embedding and the generated cue embedding are input into the mask encoder together. Exemplarily, the mask encoder can be an optimized ViT architecture, which includes multiple decoder layers, and each decoder layer has components such as self-attention, cross-attention, multi-layer perceptron update, and re-cross-attention calculation.

[0114] The self-attention layer allows the model to focus on the relationships between information at different positions, and the cross-attention layer interacts the information between the cue embedding and the image embedding, enabling the model to focus on key regions according to the annotator's cues. The multi-layer perceptron update provides non-linear capabilities for information transmission and feature transformation, and the re-cross-attention calculation further strengthens the information interaction.

[0115] For example, when processing the annotation of the left gastroepiploic artery, the multiple decoder layers will, according to the annotator's cues, through the collaborative work of these layers, combine the image embedding and the cue embedding to accurately analyze the position, shape and other information of the left gastroepiploic artery in the image inside the model.

[0116] After being processed by the multiple decoder layers, the output of the decoder will undergo an upsampling operation to map the low-resolution feature back to the resolution size of the original image. Finally, a candidate mask image is generated through a dot product operation. This candidate mask image will display different colors or gray values to distinguish different anatomical structures or backgrounds, and is pushed to the user (annotator) for confirmation. If the user is satisfied with the annotation result (step 205), this candidate mask image serves as the second image frame, containing the preliminary annotation information; if not satisfied (step 206), it will return to step 202, re-identify the contour information of the candidate anatomical structure, adjust the "point cues" and "box cues", and continue the subsequent processing flow.

[0117] Since the original SAM model was not trained on surgical images, its generalization ability for key anatomical structures in surgical images is insufficient. To improve the effect of the initial segmentation, the SAM model is fine-tuned using an open-source surgical image dataset (such as CholecSeg8k). During the fine-tuning process, the surgical images and their corresponding annotation information in the CholecSeg8k dataset are input into the SAM model, and the parameters of the model are adjusted to make the model better adapt to the characteristics of surgical images and annotation tasks. For example, through the backpropagation algorithm, the weights of the ViT and mask encoder are updated according to the error between the prediction result and the ground truth annotation, enabling the model to more accurately identify key anatomical structures and generate masks when processing surgical images, so that in the new surgical image annotation task, more accurate initial annotation information can be generated according to the prompts of the annotator.

[0118] In this way, by combining the manual prompts of the annotator and the fine-tuned SAM model, the initial annotation of surgical images can be efficiently completed, which not only gives play to the automation ability of the model but also utilizes the professional knowledge of the annotator, improving the accuracy and efficiency of annotation.

[0119] As an optional embodiment, in 104, pushing the second image frame to the user side for review and receiving the review result of the user side to adjust the second image frame to obtain the third image frame includes:

[0120] Pushing the second image frame to multiple users for cross-review and correction; receiving the correction instructions of multiple users for the initial annotation information in the second image frame, and adjusting the second image frame based on the correction instructions to obtain the third image frame.

[0121] In the embodiment of the present application, at least the optimized annotation information of the corrected key anatomical structure is marked in the third image frame.

[0122] In this optional embodiment, step 104 mainly involves the review and correction of the second image frame to obtain a third image frame with higher quality. Refer to Figure 3 As shown, the second image frame can be Figure 3 the artificial intelligence-assisted annotation result in Figure 3 and the third image frame can be

[0123] The purpose of the cross-review here is to push the second image frame to multiple users for cross-review, aiming to utilize the perspectives and professional knowledge of different users to comprehensively discover possible problems in the initial annotation information as much as possible, which helps to examine the annotation results from multiple angles and improve the accuracy and comprehensiveness of the review.

[0124] Specifically, assume there are multiple users participating in the review, such as User A, User B, User C, etc. The second image frame is pushed to these users respectively, and each user reviews the initial annotation information in the image frame. For example, in the annotation review of minimally invasive gastric cancer surgery images, User A may be good at identifying vascular structures, User B has a deeper understanding of the gastric anatomical structure, and User C is experienced in surgical operation procedures. Each user carefully checks the annotation content in the image based on their own professional advantages and understanding of the annotation specifications.

[0125] Secondly, during the review process, if a user discovers a problem with the initial annotation information, a corresponding correction instruction will be generated. These problems may include inaccurate annotation ranges, annotation colors not conforming to the coding rules, improper handling of special cases (such as dark areas, small-area structures), etc. For example, User A discovers that the annotation range of the left gastroepiploic artery is too small and does not fully cover the visible part of the blood vessel, so a correction instruction is generated indicating that the annotation range of this artery should be expanded; User B notices that the annotation color of a certain area of lymph nodes does not match the specified color coding and issues a correction instruction to change the annotation color.

[0126] The correction instructions are diverse. For different types of annotation errors or non-standard situations, different instruction contents will be generated. At the same time, these instructions are highly targeted, clearly indicating the specific parts that need to be corrected and the correction methods, so as to accurately adjust the second image frame based on these instructions later.

[0127] Finally, after receiving the correction instructions from multiple users, the second image frame will be adjusted accordingly. For example, according to the instruction from User A to expand the annotation range of the left gastroepiploic artery, the annotation area of this artery will be correspondingly expanded in the image; according to the instruction from User B to change the annotation color of the lymph nodes, the annotation color will be adjusted to the color that conforms to the coding rules. By executing all the correction instructions one by one, the initial annotation information of the second image frame is comprehensively corrected.

[0128] For example, after processing all the correction instructions, the initial annotation information of the second image frame is optimized, thus forming the third image frame. The third image frame is at least marked with optimized annotation information of the corrected key anatomical structures, and these information are more accurate, standardized, and more in line with the actual anatomical structure and annotation requirements.

[0129] In actual operation, due to the complexity of the anatomical structure and the variability of the image itself, the annotation range often needs to be further optimized and adjusted.

[0130] For example, in a minimally invasive gastric cancer surgery image, the surgical image annotation model initially determines the annotation range of the right gastroepiploic artery. However, due to factors such as tissue occlusion and uneven lighting in the image, it may not fully cover all the pixels belonging to this blood vessel. Based on anatomical knowledge and the actual image situation, the annotation range can be appropriately expanded to ensure that all visible parts of the blood vessel are completely included. Conversely, if the annotation range generated by the surgical image annotation model is too large and includes some non-vascular tissues, the annotator needs to narrow the annotation range.

[0131] The edges of the annotation boxes generated by the surgical image annotation model may be too sharp or uneven, which does not conform to the natural shape of the actual anatomical structure. For example, when annotating the edge of the liver, the edge of the annotation box generated by the surgical image annotation model is serrated, while the actual edge of the liver is a relatively smooth curve. At this time, annotation points can be added or deleted, and by fine-tuning the positions of the annotation points, the edges of the annotation box can be made smoother and more natural, and better conform to the shape of the actual anatomical structure.

[0132] In surgical images, due to the possible similarity in the colors of different tissues and structures, the surgical image annotation model may misidentify some pixels with colors similar to the target blood vessel as the target blood vessel, thus generating incorrect annotation boxes. For example, in the image of the gastroduodenal artery region, the color of the surrounding adipose tissue is similar to that of the gastroduodenal artery under specific lighting conditions, and the surgical image annotation model may misjudge some adipose tissue as a blood vessel and generate an incorrect annotation box. When reviewing, based on an accurate understanding of the anatomical structure, the annotator can identify these misjudgments and instruct the model to automatically delete these incorrect annotation boxes.

[0133] The surgical process involves multiple different scenarios, and the key blood vessels in different scenarios are different. The surgical image annotation model may misidentify the key blood vessels belonging to other scenarios as the key blood vessels in this scenario. For example, in the scenario of dissecting the subpyloric vascular region, the surgical image annotation model incorrectly identifies and generates an annotation box for the vascular arcade on the lesser curvature side in the scenario of dissecting the vessels in the lesser curvature region of the stomach. Relying on familiarity with the surgical procedure and the key anatomical structures in each scenario, the annotator can discover the incorrect annotations caused by this scenario confusion and delete the corresponding annotation boxes.

[0134] In surgical images, some key blood vessels may be located in areas with lower brightness in the picture, which makes it difficult for the surgical image annotation model to accurately identify and generate annotation boxes. For example, in some laparoscopic surgery images, due to instrument occlusion or light source angle problems, some blood vessels are in shadow areas, and the surgical image annotation model may not be able to clearly identify their contours. The annotator can, based on their own experience, manually annotate these key blood vessels even in low-brightness areas, ensuring that they are accurately included in the annotation range by understanding the position and shape of the anatomical structure.

[0135] There may be a small amount of blood or adipose tissue covering the surface of the key blood vessels, which affects the accurate recognition of blood vessels by the surgical image annotation model. For example, during the surgical operation, blood may adhere to the surface of the blood vessels, making the boundaries of the blood vessels blurred, and it is difficult for the surgical image annotation model to generate a complete and accurate annotation box. With professional knowledge and rich experience, annotators can see through these coverings, judge the true contour of the blood vessels, and manually perform annotations to obtain the correct annotation range.

[0136] There are individual differences in human anatomical structures, and sometimes there are rare cases of anatomical variations of key blood vessels. The surgical image annotation model performs annotations based on existing training data and may not be able to recognize these rare anatomical variations, thus unable to generate an annotation box. When encountering such a situation, annotators can, based on their in-depth understanding of anatomy and clinical experience, manually annotate the key blood vessels with these anatomical variations to ensure the integrity and accuracy of the annotations.

[0137] In addition, the colors of different tissues in surgical images may be similar, and the surgical image annotation model is prone to generating incorrect annotations due to color misjudgment. For example, in the area around the pancreas, the color of adipose tissue is similar to that of the pancreas in some images, and the surgical image annotation model may misidentify some adipose tissue as the pancreas and perform annotations. Or, due to multiple surgical scenarios involved, the key anatomical structures in different scenarios are different, and the surgical image annotation model may make incorrect annotations due to scene confusion. Taking gastric resection surgery as an example, the surgical image annotation model may mislabel the blood vessels in the image of the vascular dissection scene in the lesser curvature area of the stomach as those in the image of the vascular dissection scene in the subpyloric area. Therefore, with professional knowledge, annotators can clearly distinguish these misjudgments, delete the incorrect annotation boxes, and ensure that the annotation results only target the true pancreatic tissue.

[0138] Regarding the annotation of low-brightness areas, some key anatomical structures may be located in areas with low brightness in the image, and it is difficult for the surgical image annotation model to accurately recognize them. For example, in some deep surgical areas, due to insufficient light, structures such as blood vessels are in the shadow. With rich experience and a clear understanding of the position of anatomical structures, annotators can accurately judge the contour of the structures even under low-brightness conditions and manually annotate these key structures, so that the key anatomical structures in low-brightness areas can also be accurately annotated.

[0139] Regarding the annotation of covered structures, key anatomical structures are sometimes covered by blood, adipose tissue, etc., which affects the recognition of the surgical image annotation model. For example, during the surgical operation, blood may adhere to the surface of the blood vessels, blurring the blood vessel boundaries. Annotators can see through these coverings, based on anatomical knowledge and experience, judge the true shape and position of the blood vessels, and manually complete the annotations to ensure that the covered key anatomical structures are not omitted and the annotation range accurately reflects their true form.

[0140] For anatomical variation annotation, there are individual differences in human anatomical structures, and rare anatomical variations may occur, which are difficult for the surgical image annotation model to recognize based on conventional training data. For example, the blood vessel courses or branching patterns of some patients are different from common anatomical structures. Annotators with profound anatomical knowledge and clinical experience can accurately manually annotate the key structures of these anatomical variations, enabling the annotation results to completely and accurately reflect individual differences.

[0141] The optimized annotation information of these corrected key anatomical structures makes the third image frame more accurate and reliable in surgical image processing. It provides a high-quality data foundation for subsequent construction of a high-quality surgical image dataset, contributing to improving the accuracy of the surgical image analysis model.

[0142] Through the above artificial correction process, annotators comprehensively and meticulously adjust and improve the preliminary annotation results generated by the surgical image annotation model, making the annotation results more accurately reflect the key anatomical structures in the surgical images, laying a solid foundation for subsequent construction of a high-quality surgical image dataset and related medical research and applications.

[0143] As an optional embodiment, assume that the optimized annotation information includes: multiple optimized annotation information obtained based on the correction instructions of multiple users. Based on this assumption, in 105, constructing a surgical image dataset based on the third image frame and the corresponding optimized annotation information includes:

[0144] Performing consistency calculation based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to each of the multiple optimized annotation information; generating annotation masks corresponding to the multiple optimized annotation information, and using the intersection part between the multiple annotation masks as the consensus mask of the third image frame; pushing the third image frame to the user for secondary review, and receiving the secondary review results of the user for the third image frame; performing data fusion based on the secondary review results, confidence levels, and the consensus mask to construct a surgical image dataset.

[0145] Specifically, assume that three annotators, A, B, and C, annotated the same third image frame (which shows the surgical scene in the region of the right gastroepiploic artery), generating three optimized annotation information. Annotator A believes that a certain branch of the right gastroepiploic artery should be included in the annotation scope, while annotator B believes that this branch part should not be included. The annotation of annotator C is similar to that of A but with a slightly different scope. Through consistency calculation, for example, using the Dice coefficient to measure the overlap degree of two annotation results (the value range is [0, 1], and the closer to 1, the higher the consistency). Calculate the Dice coefficient for the annotation results of A and B, and assume that the result is 0.7; calculate the Dice coefficient for the annotation results of A and C, and the result is 0.9. Through a series of similar calculations and analyses, a confidence level is assigned to each annotation information. For example, according to the calculation results, the confidence level of annotator A may be set to 0.85, that of B to 0.7, and that of C to 0.9.

[0146] Thus, by obtaining the confidence level through consistency calculation, the reliability of each annotator's annotation result can be quantified. This helps to more reasonably weigh the weights of different annotation information during the subsequent data fusion process, avoid having too much impact on the quality of the data set due to the misjudgment of individual annotators, and improve the accuracy and reliability of the data set.

[0147] For the optimized annotation information of the above three annotators, annotation masks are generated respectively. An annotation mask is a binary image, where the region where the target anatomical structure (such as the right gastroepiploic artery and its branches) is located is white (or other specified colors), and the background is black. The three generated annotation masks respectively show the annotation scopes of the three annotators for the right gastroepiploic artery on the image. Then, by calculating the intersection part of these three annotation masks, a consensus mask is obtained. In this example, the region covered by the consensus mask is the overlapping part of the annotation results of the three annotators, representing the commonly recognized region of the right gastroepiploic artery.

[0148] Thus, using the intersection of multiple annotation masks as the consensus mask can synthesize the opinions of multiple annotators, highlight the annotation parts that everyone recognizes, and reduce the errors caused by individual differences. The consensus mask generated in this way is more representative and accurate, providing a relatively objective and accurate annotation basis for constructing a surgical image data set, and enhancing the consistency and stability of the data set.

[0149] Furthermore, the third image frame with the consensus mask is pushed to a senior surgeon (as the user) for secondary review. After viewing the image, the doctor finds that at a small branch of the right gastroepiploic artery in the consensus mask, although it is the intersection of the three annotators, the scope is slightly smaller compared to the actual anatomical structure. The doctor gives the secondary review result, suggesting appropriately expanding the annotation scope at this branch.

[0150] Thus, the secondary review introduces in-depth knowledge and experience in the professional field, which can discover problems that may be overlooked in the previous steps, further optimize the annotation, ensure that the annotation information in the dataset highly conforms to the actual medical situation, and improve the quality and usability of the dataset.

[0151] Finally, combine the secondary review results of the doctor (suggesting to expand the annotation range of a certain branch of the right gastroepiploic artery), the confidence levels of each annotator (such as A: 0.85, B: 0.7, C: 0.9), and the consensus mask to adjust the annotation information. Since C has a relatively high confidence level and its annotation range at this branch is relatively more in line with the doctor's suggestion, when adjusting the annotation range, more reference is made to C's annotation method. Finally, integrate the adjusted annotation information with the third image frame and build it into the surgical image dataset.

[0152] Thus, this data fusion method fully utilizes the confidence levels obtained from the consistency calculation, the consensus mask, and the secondary review results of professional users, comprehensively considers multiple factors, makes the constructed surgical image dataset more accurate and reliable, and can better reflect the anatomical structure and operation conditions in actual surgeries. A high-quality dataset is of great significance for subsequent applications such as surgical image analysis, surgical simulation, and disease diagnosis, and can improve the training effect of relevant models and the accuracy of applications.

[0153] As an optional embodiment, in the above steps, consistency calculation is performed based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to each of the multiple optimized annotation information, including: calculating the consistency index parameters between the multiple optimized annotation information in the third image frame; determining the confidence levels corresponding to each of the multiple optimized annotation information based on the consistency index parameters and the weight coefficients of the multiple optimized annotation information.

[0154] In the embodiments of the present application, the consistency index parameters at least include: the overlap degree (Dice) coefficient, the intersection over union (IOU), and the normalized Hausdorff distance parameter between the multiple optimized annotation information.

[0155] Among them, the Dice coefficient is mainly used to measure the overlap degree of two annotation results. Essentially, it focuses on the proportion of the overlapping part between two annotation areas (completed by different annotators) in the overall annotation area. When the Dice coefficient is 1, it means that the two annotation results completely overlap, that is, the annotation ranges of the two annotators for the same anatomical structure are exactly the same; when the Dice coefficient is 0, it means that the two annotation results have no overlapping part, which may indicate a large disagreement between the annotators.

[0156] For example, in combination with Figure 4Given the calculation principle, assume that two annotators, A and B, annotate the left gastroepiploic artery in a minimally invasive gastric cancer surgery image. The area of the artery region marked by annotator A is 100 square pixels, and the area of the region marked by annotator B is 120 square pixels. The area of the overlapping region of the annotation results is 80 square pixels. According to the Dice coefficient formula, the calculation gives This value indicates that there is a certain degree of overlap between the annotation results of the two annotators, but there are still some differences that need further analysis and coordination.

[0157] IOU measures the ratio of the overlapping area of two annotation results to the total area (i.e., the union area of the two annotation results). Compared with the Dice coefficient, IOU focuses more on reflecting the proportion of the overlapping part in the merged overall region. When IOU is 1, it means that the two annotation results not only completely overlap, but also there is no additional non-overlapping part in the merged region; when IOU is 0, it means that the two annotation results have no overlap, and the union region is completely the sum of two independent parts.

[0158] Continuing with the above example of annotating the left gastroepiploic artery, combined with Figure 4 the given calculation principle, the overlapping area of the annotation results of annotators A and B is 80 square pixels, and the union area of the annotation results is 100 + 120 - 80 = 140 square pixels. According to the IOU formula, the calculation gives This indicates that in the merged overall annotation region, the overlapping part accounts for approximately 0.57, which also reflects that there are certain differences in the annotations of the two annotators.

[0159] The normalized Hausdorff distance (HD) is mainly used to measure the maximum distance between two sets (here the annotation results of two annotators), and it focuses more on evaluating the maximum deviation of the segmentation edges. Its calculation is based on the distances between all pairs of points in the two annotation results, and the maximum distance is found as the Hausdorff distance. For example, combined with Figure 4 the given calculation principle, the Hausdorff distance (HD) can be expressed as the following formula:

[0160]

[0161] where (X,Y) represent the annotation results of the two annotators respectively, and d(x,y) represents the distance between X and Y. To reduce the computational complexity and achieve normalization, the value of HD(X,Y) is divided by the diagonal length of the two-dimensional image to obtain the normalized HD (i.e., HDnorm), with a value range of [0,1]. The closer it is to 0, the higher the consistency.

[0162] Since the original Hausdorff distance is affected by factors such as image size, in order to reduce the computational complexity and achieve normalization, the HD value is divided by the diagonal length of the two-dimensional image to obtain HDnorm. When HDnorm is close to 0, it indicates that the deviation of the segmentation edges between the two annotation results is small, that is, the consistency of the annotators in annotating the anatomical structure boundaries is high. When HDnorm is close to 1, it indicates that the deviation of the segmentation edges is large, and there may be significant annotation differences.

[0163] Suppose in a surgical image, the annotation results of annotator A and B for the liver edge are obtained, and the Hausdorff distance is calculated to be 30 pixels. If the diagonal length of this image is 100 pixels, then HDnorm is This value indicates that there is a certain deviation in the annotation of the liver edge by the two annotators, but it is still within the acceptable range, and it is necessary to further check the specific annotation situation to determine whether adjustment is needed.

[0164] These consistency index parameters each have their own characteristics in evaluating the consistency between multiple optimized annotation information. By comprehensively using these indexes, it is possible to more comprehensively and accurately judge the annotation consistency between different annotators, thereby providing an important reference basis for subsequent work such as dataset construction and annotation quality control.

[0165] In the embodiments of this application, the weight coefficients of multiple optimized annotation information are associated with the user attributes corresponding to the multiple optimized annotation information; the user attributes corresponding to the multiple optimized annotation information at least include: medical level, number of surgical cases.

[0166] Further optionally, after performing consistency calculation based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to the multiple optimized annotation information respectively, the reliability levels of multiple users can be determined based on the user attributes; among them, the higher the medical level and the more the number of surgical cases corresponding to multiple users, the higher the reliability levels corresponding to multiple users. If multiple users belong to the same reliability level, then retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold, or re-extract the optimized annotation information with a confidence level lower than the set confidence level threshold for annotation. If multiple users do not belong to the same reliability level, then retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold. Finally, based on the confidence levels corresponding to the multiple optimized annotation information respectively, batch summarize the multiple optimized annotation information.

[0167] In the above steps, two user attributes, namely medical level and the number of surgical cases, are mainly referred to. Users with a higher medical level have richer medical knowledge, more accurate cognition of anatomical structures, and thus provide more accurate information during annotation. At the same time, the accumulation of a large number of surgical cases enables users to accumulate rich experience in actual clinical operations, have a deeper understanding of various situations during the surgical process, which also helps to improve the annotation accuracy.

[0168] Exemplarily, multiple users are divided into different reliability levels according to different combinations of medical level and the number of surgical cases. For example, they are divided into three levels: high, medium, and low. High-reliability-level users are usually experts with profound attainments in this field and who have participated in a large number of surgical cases; medium-reliability-level users may be attending physicians with certain clinical experience and professional knowledge; low-reliability-level users may be residents or interns with relatively less experience.

[0169] When multiple users are at the same reliability level, a confidence threshold is set, such as 0.7. For the optimized annotation information with a confidence not lower than this threshold, because it has shown a relatively high credibility in the previous consistency calculation, it is regarded as having a relatively high reliability and is retained. These information can provide valuable data support for constructing the dataset. For example, among a group of annotators, annotators A and B both belong to the medium-reliability level. The confidence of A's annotation of a certain key anatomical structure is 0.8, and the confidence of B is 0.75. Their annotation information will be retained.

[0170] For the optimized annotation information with a confidence lower than the set threshold, because its credibility is relatively low and there may be relatively large errors, it needs to be processed. There are two processing methods. One is to re-obtain the annotation information, that is, let the annotator re-annotate the image, expecting to obtain a more accurate result through re-annotation. The other is to re-extract the image corresponding to the annotation information and hand it over to other annotators for annotation to obtain a more reliable annotation result. For example, annotator C also belongs to the medium-reliability level, but the confidence of C's annotation of the same anatomical structure is only 0.6, which is lower than the threshold. At this time, C can be asked to re-annotate the image, or the image can be extracted and handed over to other annotators at the same level for annotation.

[0171] When multiple users do not belong to the same reliability level, the optimized annotation information with a confidence level not lower than the set confidence threshold is still retained. This is because regardless of the user reliability level, as long as the annotation confidence level reaches a certain standard, it indicates that the annotation has a certain reliability and can be incorporated into the dataset construction. For the optimized annotation information with a confidence level lower than the set confidence threshold, only the method of re-acquisition is adopted. For example, an expert with a high reliability level and an intern doctor with a low reliability level annotate the same image. The annotation confidence level of the expert is 0.8, and that of the intern doctor is 0.6. The annotation information of the expert is retained, and the annotation information of the intern doctor needs to be re-acquired, that is, let the intern doctor annotate the image again. This processing method takes into account the differences of users with different reliability levels. For the low-confidence annotations of users with a low reliability level, the accuracy is improved by re-acquiring the annotations, and at the same time, excessive reliance on the annotation data of users with a low reliability level is avoided.

[0172] Batch summary is carried out according to the confidence levels corresponding to the optimized annotation information respectively. For example, it can be sorted from high to low according to the confidence level. The annotation information with a confidence level of 0.9 - 1.0 is classified into the first batch, 0.8 - 0.89 into the second batch, and so on. In this way, when constructing the dataset, the annotation information of different confidence level batches can be flexibly selected according to different requirements and uses. For application scenarios with extremely high requirements for data accuracy, such as medical research or the training of surgical navigation systems, the annotation information of high-confidence batches can be preferentially adopted; while for some scenarios with relatively large requirements for data scale and relatively lower requirements for accuracy, such as preliminary model exploratory training, more batches of annotation information can be considered to be incorporated to increase data diversity.

[0173] It can be understood that in the multi-level review, junior doctors are responsible for ensuring the accuracy of each image. In this case, there will be omissions. The review by senior doctors will make up for this defect, mainly considering the following two points: (1) Senior doctors are usually doctors with rich surgical experience, such as having performed more than 300 gastric cancer surgeries and being familiar with anatomical structures; (2) The final review by senior doctors is a front-to-back comparison review, which takes into account the temporality of the images. Since all the images are collected from surgical videos, the anatomical structures in the images have continuity before and after. In this case, it is easy for senior doctors to find the defects in the annotations and complete the modification.

[0174] Finally, through batch summary, the annotation information can be managed and utilized more precisely, giving full play to the value of annotation information with different confidence levels, improving the quality and applicability of the surgical image dataset, and providing more reliable data support for various subsequent medical applications based on this dataset.

[0175] Through the above steps, after completing the consistency calculation for multiple optimized annotation information in the third image frame, the annotation information is further screened, processed, and summarized in combination with user attributes and confidence levels, so as to construct a more accurate, reliable, and wider-applicable surgical image dataset.

[0176] In practical applications, since the annotation work of data itself is huge in workload and very time-consuming, it is difficult to form an independent team to re-review all data when the number of team members is limited. To ensure that the annotation is as accurate as possible, an overall consistency check is set after all data annotation is completed. 5% of the subset data is randomly selected from the complete dataset to complete the above consistency check to ensure the consistency of the data quality, test threshold, and the aforementioned settings.

[0177] Further optionally, the confidence threshold is calculated based on the overlap coefficient, intersection over union, and normalized Hausdorff distance parameter among multiple optimized annotation information. For example, the first threshold corresponding to the overlap coefficient among multiple optimized annotation information is set to 0.85, the first threshold corresponding to the intersection over union among multiple optimized annotation information is set to 0.80, and the first threshold corresponding to the normalized Hausdorff distance parameter among multiple optimized annotation information is set to 0.05. It can be understood that annotators / reviewers are divided according to their professional levels. For example, they are divided into general annotators and professional doctors. Professional doctors are responsible for review. The annotation training and knowledge reserves received by general annotators are similar. Therefore, no weights are set for each annotator when calculating the intersection to increase the complexity of the annotation work. For example, to avoid too large a difference (i.e., low consistency) among three annotation masks, a threshold needs to be set to provide a reference for subsequent reviewers. The threshold is set to Dice < 0.85; IOU < 0.80; normalized Hdnorm > 0.05. If it is lower than this threshold, the reviewer will intervene to modify it, and after summarization, it will be taught and trained in batches.

[0178] As an optional embodiment, in 105, generating annotation masks corresponding to multiple optimized annotation information includes: generating annotation masks corresponding to multiple optimized annotation information based on the structural contour information indicated by the multiple optimized annotation information. Further, in 105, using the intersection part among multiple annotation masks as the consensus mask of the third image frame includes: calculating the position coordinates of the multiple annotation masks to obtain the position coordinate information of the intersection area among the multiple annotation masks; constructing a consensus mask based on the position coordinate information of the intersection area.

[0179] Specifically, in 105, each optimized annotation information contains the structural contour information of key anatomical structures, and these contour information describe the position and shape of the key anatomical structures in the image. For example, in an image of minimally invasive gastric cancer surgery, for the left gastroepiploic artery, its optimized annotation information may include contour information such as the starting point, ending point, branching point, and bending point of the artery.

[0180] To generate the annotation mask, in 105, these structural contour information are converted into actual mask images. Generally speaking, the annotation mask is a binary image (usually a grayscale image) with the same size as the original image. The pixel values inside the structural contour are set to 1 (or 255, indicating that this area belongs to the annotation area), while the pixel values outside the contour are set to 0 (indicating that this area is the background). For the above example of the left gastroepiploic artery, according to its structural contour information, the corresponding pixels in the mask image are set to 1 at the corresponding positions, and other areas are set to 0, thus generating an annotation mask representing the left gastroepiploic artery.

[0181] For multiple optimized annotation information, corresponding annotation masks will be generated respectively for different key anatomical structures (such as the left gastroepiploic artery, the right gastroepiploic vein, etc.) or different annotation versions of the same structure according to their respective structural contour information.

[0182] Furthermore, in 105, it is necessary to determine the position coordinate information of each annotation mask. Since the annotation mask is an image with the same size as the original image, each pixel has its corresponding coordinate. For each annotation mask, the area where the internal pixel value is 1 is the annotation area, and it is necessary to calculate the position coordinates of these annotation areas. For example, in a 1024×1024 image, for each annotation mask, it is necessary to find out which pixel positions (x, y) have a value of 1.

[0183] Taking the annotation masks of the left gastroepiploic artery and the right gastroepiploic vein as examples, the coordinate sets of their respective annotation areas are found respectively. The annotation area of the left gastroepiploic artery may contain multiple coordinate points, and the annotation area of the right gastroepiploic vein also has its corresponding coordinate set.

[0184] In 105, the intersection area between multiple annotation masks is obtained, that is, the pixel positions that are marked as 1 in all annotation masks are found. For two annotation masks, it can be achieved through a logical AND operation (AND operation). The pixel values at the same position in the two annotation masks are logically ANDed. If both pixel values are 1, then this position belongs to the intersection area. For example, for the annotation masks of the left gastroepiploic artery and the right gastroepiploic vein, only when the pixel values of the two masks are both 1 at the same position, this position belongs to their intersection area.

[0185] For multiple annotation masks, the pairwise intersection operation can be performed step by step to obtain the position coordinate information of the final intersection region. This process can use methods such as loops or matrix operations to check each position and find the pixel positions that belong to the annotated region in all annotation masks.

[0186] In 105, based on the obtained position coordinate information of the intersection region, a consensus mask is constructed. In the consensus mask, only the pixel values of the position coordinates belonging to the intersection region are set to 1 (or 255), and the pixel values of other positions are set to 0. Continuing with the example of the left gastroepiploic artery and the right gastroepiploic vein, if the coordinate set obtained through the intersection operation, in the consensus mask, the pixel values of these positions are 1, and the remaining positions are 0. In this way, the consensus mask reflects the common part of the annotations of different key anatomical structures by multiple annotators and represents the common recognition of the annotations. Such a consensus mask is of great significance in subsequent work such as dataset construction and annotation verification. It synthesizes the opinions of multiple annotators, reduces individual annotation errors, and provides a more reliable and objective annotation basis.

[0187] Through the above steps, generating annotation masks corresponding to multiple optimized annotation information and calculating their intersection to obtain a consensus mask helps to improve the consistency and accuracy of annotations, provides better data support for subsequent surgical image analysis, model training, etc., and makes the constructed surgical image dataset more reliable and stable.

[0188] Exemplarily, in combination with Figure 5 the shown annotation mask acquisition process, for a given surgical image, multiple annotators (such as Annotator-1, Annotator-2, Annotator-3 in the figure) respectively annotate the key structures or regions of interest in the image. Each annotator will determine the range of the region to be annotated according to their own understanding and judgment.

[0189] Taking Annotator-1 as an example, their annotation results will generate an annotation mask (Mask-1). This annotation mask is a binary image with the same size as the original surgical image. At the positions where the annotator believes are key structures or regions of interest, the pixel value is set to 1 (or other specific values indicating that the region belongs to the annotation area), while at other positions, the pixel value is set to 0 (indicating the background area). Similarly, Annotator-2 and Annotator-3 will also generate their respective annotation masks (Mask-2 and Mask-3), which represent their annotation scopes of the key structures in the surgical image. After obtaining the annotation masks of multiple annotators (such as Mask-1, Mask-2, Mask-3), the consensus mask is obtained by calculating the intersection of these annotation masks. Specifically, each pixel position is checked. If the pixel value at this position is 1 in all annotation masks (i.e., this position is considered a key structure area in all annotators' annotations), then the pixel value at this position in the consensus mask is set to 1; otherwise, it is set to 0.

[0190] The consensus mask obtained in this way represents the key structure areas commonly recognized by multiple annotators. It synthesizes the opinions of multiple annotators, reduces the possible errors and subjective biases of a single annotator, and provides a relatively more reliable and objective annotation result. For example, as Figure 5 can be seen, the overlapping area of the three annotation masks (i.e., the consensus mask) is the part of the key structure recognized by all annotators. This consensus mask can be used for subsequent data analysis, model training, etc. Because it is based on the consensus of multiple annotators, it can better reflect the real structure information and improve the quality and reliability of the data. In summary, by generating annotation masks by multiple annotators and then taking the intersection to obtain the consensus mask, the accuracy and consistency of annotation can be effectively improved, providing a better data foundation for research and applications related to surgical images.

[0191] In the above or following embodiments, after a series of operations such as collecting surgical images, identifying surgical scenarios, extracting image frames, performing preliminary annotations (such as obtaining the second image frame using a surgical image annotation model), and manual review and correction (obtaining the third image frame), the reviewed images and mask data generated based on optimized annotation information have been obtained. At this time, in 105, these images and mask data need to be integrated to lay a foundation for constructing a complete data set. For example, in the process of processing the surgical images of minimally invasive distal gastrectomy with D2 lymph node dissection, for each reviewed image, the corresponding annotation mask data is found and they are associated to form a complete data unit.

[0192] During the construction of the folder structure, the surgical folders can be created as follows: multiple surgical folders named after the video numbers are set under the root folder of the dataset, and each folder represents the surgical records of a patient. This step preliminarily classifies the data according to the surgical videos of different patients, facilitating the management of surgical-related data of different patients.

[0193] The scene sub-folders can be created by further dividing each surgical folder into 8 sub-folders numbered from S1 to S8, which respectively correspond to specific scenes during the surgery. For example, S1 corresponds to the scene of dissecting the left gastroepiploic vascular area, and S2 corresponds to the scene of dissecting the vascular area of the greater curvature of the stomach, etc. This subdivision according to the surgical scenes enables the data of each scene to be stored and managed independently, facilitating data analysis for different surgical scenes.

[0194] Multiple surgical images in PNG format and their corresponding segmentation mask files are stored in each scene folder. These files are named according to a specific format, namely {video number}{scene number}{image sequence number}.png and {video number}{scene number}{image sequence number}_mask.png respectively. For example, "8_S7_13.png" represents the 13th surgical image in the S7 scene of the surgical record with video number 8; "8_S7_13_mask.png" is the segmentation mask file corresponding to this image. This naming method clarifies the correspondence between the image and the mask file, and in subsequent data processing and analysis, the mask data corresponding to a certain image can be quickly and accurately found.

[0195] For tabular files, each surgical folder contains a tabular file (suffix.csv) for recording detailed information of images. This tabular file is an important index for dataset management and query. It integrates the key information of image and mask files, as well as information such as surgical scenarios and key blood vessels in the annotation. In the tabular file, the video number can be set as an integer type and used as the primary key to uniquely identify each surgical record. It is unique in the entire dataset and can accurately distinguish the surgical video data of different patients. For example, the surgical video numbers of different patients are different, and all surgical-related data of a specific patient can be quickly located through the video number. The scene number can be set as a string type to distinguish specific scenarios during the surgery, such as "S1" to "S8". Through the scene number, all images and related information under a specific surgical scenario can be quickly filtered out, facilitating special analysis for different scenarios. The image sequence number can be set as an integer type to identify the order of images in this scenario. Combining the video number and the scene number can uniquely determine each image in a surgical scenario, ensuring the orderliness and accuracy of the data. The image file name can be set as a string type to record the image file name in the format of {video number}{scene number}{image sequence number}.png. This field clarifies the specific name of the image file, facilitating the search for the corresponding image file in the folder. The mask file name can be set as a string type to store the name of the corresponding segmentation mask file in the format of {video number}{scene number}{image sequence number}_mask.png. Corresponding to the image file name, it clarifies the specific name of the mask file, facilitating the acquisition of mask data corresponding to the image. The key anatomical name can be set as a string type to identify the key blood vessels annotated in the image, such as "left gastric artery", "right gastric vein", etc. This field records the information of the key anatomical structures annotated in the image, providing important information for studying the situation of specific key blood vessels during the surgery. Thus, the video number, as the main key field, jointly with the scene number and the image sequence number, uniquely identifies each frame of image and its mask. This design ensures the accuracy and traceability of the data. During the data management and analysis process, each data unit can be accurately located without data confusion and misuse.

[0196] In the technical solution of this application, the surgical image annotation model can automatically annotate surgical images, improving the automation degree of surgical image processing. And based on the user review results, the surgical images are optimized to further improve the image proofreading efficiency and reliability, adapting to the surgical image processing scenario.

[0197] In another embodiment of the present application, a surgical image processing device is further provided. The device includes the following units: an acquisition unit configured to acquire a surgical image to be processed and identify a target surgical scenario corresponding to the surgical image; an extraction unit configured to extract a first image frame from the surgical image; wherein the first image frame at least includes: key surgical operations in the target surgical scenario and corresponding surgical space images; a labeling unit configured to perform anatomical structure recognition and preliminary labeling on the first image frame using a surgical image labeling model to obtain a second image frame; the second image frame is labeled with initial labeling information of key anatomical structures corresponding to the key surgical operations; an optimization unit configured to push the second image frame to the client for review and receive the review result of the client to adjust the second image frame to obtain a third image frame; at least the optimized labeling information of the key anatomical structures is labeled in the third image frame; a construction unit configured to construct a surgical image data set based on the third image frame and the corresponding optimized labeling information, and the surgical image data set is used to train and optimize the surgical image model.

[0198] Further optionally, the extraction unit, which extracts the first image frame from the surgical image, is configured to: extract consecutive image frames from the surgical image according to a preset frequency and image specification; perform frame screening on the extracted consecutive image frames to screen out consecutive image frames whose clarity, surgical operation process images, and anatomical structure images meet the preset screening conditions as the first image frame.

[0199] Further optionally, the preset frequency is 5 frames per second, and the image specification is: a PNG format image with a resolution of 1920×1080 pixels.

[0200] Further optionally, the surgical image annotation model at least includes: an image encoder, a prompt encoder, and a mask encoder; an annotation unit that uses the surgical image annotation model to perform anatomical structure recognition and preliminary annotation on the first image frame to obtain a second image frame, and is configured to: input the first image frame into the image encoder for dimensionality reduction processing to obtain image embedding features; identify the contour information corresponding to the candidate anatomical structures in the first image frame to obtain point prompt information and box prompt information for the candidate anatomical structures; the point prompt information and the box prompt information are respectively used to describe the external contours of the candidate anatomical structures; use the prompt encoder to convert the point prompt information and the box prompt information into corresponding multi-dimensional prompt embedding features; input the image embedding features and the corresponding multi-dimensional prompt embedding features into the mask encoder to generate corresponding candidate mask images, and push the candidate mask images to the user for confirmation; wherein, the mask encoder at least includes multiple decoder layers, and each decoder layer at least includes a self-attention layer, a cross-attention layer, a multi-layer perceptron, and a cross-attention calculation layer; if a confirmation instruction feedback by the user is received, then use the candidate mask image as the second image frame; or, if a confirmation instruction feedback by the user is not received, then jump to the step of identifying the contour information corresponding to the candidate anatomical structures in the first image frame, and continue to execute the processing flows corresponding to the prompt encoder and the mask encoder.

[0201] Further optionally, an optimization unit that pushes the second image frame to the user side for review and receives the review result of the user side to adjust the second image frame to obtain a third image frame, and is configured to: push the second image frame to multiple users for cross-review and correction; receive the correction instructions of multiple users for the initial annotation information in the second image frame, and adjust the second image frame based on the correction instructions to obtain a third image frame; wherein, the third image frame is at least labeled with: optimized annotation information of the corrected key anatomical structures.

[0202] Further optionally, the optimized annotation information includes: multiple optimized annotation information obtained based on the correction instructions of multiple users; an optimization unit that constructs a surgical image dataset based on the third image frame and the corresponding optimized annotation information, and is configured to: perform consistency calculation based on the multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to the multiple optimized annotation information respectively; generate annotation masks corresponding to the multiple optimized annotation information, and use the intersection part between the multiple annotation masks as the consensus mask of the third image frame; push the third image frame to the user for secondary review, and receive the secondary review result of the user for the third image frame; perform data fusion according to the secondary review result, the confidence level, and the consensus mask to construct a surgical image dataset.

[0203] Further optionally, the optimization unit performs a consistency calculation based on a plurality of optimized annotation information in the third image frame to obtain the confidence levels corresponding to the respective optimized annotation information, and is configured to: calculate the consistency index parameters between the plurality of optimized annotation information in the third image frame; wherein the consistency index parameters at least include: the overlap coefficient between the plurality of optimized annotation information, the intersection over union, and the normalized Hausdorff distance parameter; determine the confidence levels corresponding to the respective optimized annotation information based on the consistency index parameters and the weight coefficients of the plurality of optimized annotation information; wherein the weight coefficients of the plurality of optimized annotation information are associated with the user attributes corresponding to the plurality of optimized annotation information; the user attributes corresponding to the plurality of optimized annotation information at least include: medical level and the number of surgical cases.

[0204] Further optionally, after the optimization unit performs a consistency calculation based on a plurality of optimized annotation information in the third image frame to obtain the confidence levels corresponding to the respective optimized annotation information, it is further configured to: determine the reliability levels of the respective users based on the user attributes; wherein the higher the medical level and the more the number of surgical cases corresponding to the respective users, the higher the reliability levels of the respective users; if the plurality of users belong to the same reliability level, then retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold, or re-extract the optimized annotation information with a confidence level lower than the set confidence level threshold for annotation; if the plurality of users do not belong to the same reliability level, then retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold; batch-summarize the plurality of optimized annotation information based on the confidence levels corresponding to the respective optimized annotation information.

[0205] Further optionally, the set confidence level threshold is calculated based on the overlap coefficient between the plurality of optimized annotation information, the intersection over union, and the normalized Hausdorff distance parameter; wherein the first threshold corresponding to the overlap coefficient between the plurality of optimized annotation information is set to 0.85, the first threshold corresponding to the intersection over union between the plurality of optimized annotation information is set to 0.80, and the first threshold corresponding to the normalized Hausdorff distance parameter between the plurality of optimized annotation information is set to 0.05.

[0206] Further optionally, the optimization unit generates annotation masks corresponding to the plurality of optimized annotation information, and is configured to: generate annotation masks corresponding to the plurality of optimized annotation information based on the structural contour information indicated by the plurality of optimized annotation information; use the intersection part between the plurality of annotation masks as the consensus mask of the third image frame, including: calculating the position coordinates of the plurality of annotation masks to obtain the position coordinate information of the intersection area between the plurality of annotation masks; constructing the consensus mask based on the position coordinate information of the intersection area.

[0207] Further optionally, the initial annotation information and / or the optimized annotation information at least include: mask range information for indicating the position range of each structure, color coding information for indicating the type to which each structure belongs, dark area annotation information, small area annotation information, and multi-region annotation information.

[0208] The device can implement various steps in the above method embodiments, which will not be elaborated here for the time being.

[0209] The embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above surgical image processing method is implemented. This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the above surgical image processing method is implemented. Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various changes and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A surgical image processing method, characterized in that, The method includes: Collecting the surgical images to be processed and identifying the target surgical scenario corresponding to the surgical images; Extracting a first image frame from the surgical images; wherein, the first image frame at least includes: the key surgical operations in the target surgical scenario and the corresponding surgical space images; Performing anatomical structure recognition and preliminary annotation on the first image frame by using a surgical image annotation model to obtain a second image frame; the initial annotation information of the key anatomical structures corresponding to the key surgical operations is annotated in the second image frame; Pushing the second image frame to the client for review and receiving the review result of the client to adjust the second image frame to obtain a third image frame; at least the optimized annotation information of the key anatomical structures is annotated in the third image frame; Constructing a surgical image dataset based on the third image frame and the corresponding optimized annotation information, and the surgical image dataset is used for training and optimizing a surgical image model.

2. The surgical image processing method according to claim 1, wherein The extracting a first image frame from the surgical images includes: Extracting consecutive image frames from the surgical images according to a preset frequency and image specification; Performing frame screening on the extracted consecutive image frames to screen out the consecutive image frames whose clarity, surgical operation process images and anatomical structure images meet the preset screening conditions as the first image frame; The preset frequency is 5 frames per second, and the image specification is: a PNG format image with a resolution of 1920×1080 pixels.

3. The surgical image processing method according to claim 1, wherein The surgical image annotation model at least includes: an image encoder, a prompt encoder, and a mask encoder; The performing anatomical structure recognition and preliminary annotation on the first image frame by using a surgical image annotation model to obtain a second image frame includes: Inputting the first image frame into the image encoder for dimensionality reduction processing to obtain an image embedding feature; Identifying the contour information corresponding to the candidate anatomical structures in the first image frame to obtain point prompt information and box prompt information for the candidate anatomical structures; the point prompt information and the box prompt information are respectively used to describe the external contour of the candidate anatomical structures; Converting the point prompt information and the box prompt information into corresponding multi-dimensional prompt embedding features by using the prompt encoder; Inputting the image embedding feature and the corresponding multi-dimensional prompt embedding features into the mask encoder to generate corresponding candidate mask images, and pushing the candidate mask images to the user for confirmation; wherein, the mask encoder at least includes multiple decoder layers, and each decoder layer at least includes a self-attention layer, a cross-attention layer, a multi-layer perceptron, and a cross-attention calculation layer; If a confirmation instruction feedback by the user is received, using the candidate mask image as the second image frame; or, If a confirmation instruction feedback by the user is not received, jumping to the step of identifying the contour information corresponding to the candidate anatomical structures in the first image frame, and continuing to execute the processing flows corresponding to the prompt encoder and the mask encoder.

4. The surgical image processing method according to claim 1, wherein The pushing the second image frame to the client for review and receiving the review result of the client to adjust the second image frame to obtain a third image frame includes: Push the second image frame to multiple users for cross-review and correction; Receive correction instructions from multiple users for the initial annotation information in the second image frame, and adjust the second image frame based on the correction instructions to obtain a third image frame; Wherein, at least the optimized annotation information of the corrected key anatomical structures is marked in the third image frame.

5. The surgical image processing method according to claim 4, characterized in that The optimized annotation information includes: multiple optimized annotation information obtained based on the correction instructions of multiple users; Constructing a surgical image dataset based on the third image frame and the corresponding optimized annotation information includes: Performing consistency calculation based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to the multiple optimized annotation information respectively; Generate annotation masks corresponding to the multiple optimized annotation information, and use the intersection part between the multiple annotation masks as the consensus mask of the third image frame; Push the third image frame to the user for secondary review, and receive the secondary review result of the user for the third image frame; Perform data fusion according to the secondary review result, the confidence level, and the consensus mask to construct the surgical image dataset.

6. The surgical image processing method according to claim 5, wherein, Performing consistency calculation based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to the multiple optimized annotation information respectively, includes: Calculate the consistency index parameters between multiple optimized annotation information in the third image frame; wherein, the consistency index parameters at least include: the overlap coefficient between multiple optimized annotation information, the intersection over union, and the normalized Hausdorff distance parameter; Determine the confidence levels corresponding to the multiple optimized annotation information respectively based on the consistency index parameters and the weight coefficients of the multiple optimized annotation information; Wherein, the weight coefficients of the multiple optimized annotation information are associated with the user attributes corresponding to the multiple optimized annotation information; the user attributes corresponding to the multiple optimized annotation information at least include: medical level, number of surgical cases.

7. The surgical image processing method according to claim 6, wherein After performing consistency calculation based on multiple optimized annotation information in the third image frame to obtain the confidence levels corresponding to the multiple optimized annotation information respectively, further includes: Determine the reliability levels of multiple users respectively based on the user attributes; wherein, the higher the medical level and the more the number of surgical cases corresponding to multiple users respectively, the higher the reliability levels corresponding to multiple users respectively; If multiple users belong to the same reliability level, retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold, or re-extract the optimized annotation information with a confidence level lower than the set confidence level threshold for annotation; If multiple users do not belong to the same reliability level, retain the optimized annotation information with a confidence level not lower than the set confidence level threshold, and re-obtain the optimized annotation information with a confidence level lower than the set confidence level threshold; Batch summarize the multiple optimized annotation information based on the confidence levels corresponding to the multiple optimized annotation information respectively; Among them, the set confidence threshold is calculated based on the overlap coefficient, intersection over union, and normalized Hausdorff distance parameter among multiple optimized annotation information; Among them, the first threshold corresponding to the overlap coefficient among multiple optimized annotation information is set to 0.85, the first threshold corresponding to the intersection over union among multiple optimized annotation information is set to 0.80, and the first threshold corresponding to the normalized Hausdorff distance parameter among multiple optimized annotation information is set to 0.

05.

8. The surgical image processing method according to claim 5, wherein Generating the annotation mask corresponding to the multiple optimized annotation information includes: Generating the annotation mask corresponding to the multiple optimized annotation information based on the structural contour information indicated by the multiple optimized annotation information; Taking the intersection part among the multiple annotation masks as the consensus mask of the third image frame includes: Calculating the position coordinates of the multiple annotation masks to obtain the position coordinate information of the intersection area among the multiple annotation masks; Constructing the consensus mask based on the position coordinate information of the intersection area.

9. A surgical image processing device, characterized in that, The device includes the following units, where An acquisition unit configured to acquire the surgical image to be processed and identify the target surgical scenario corresponding to the surgical image; An extraction unit configured to extract a first image frame from the surgical image; where the first image frame at least includes: the key surgical operations in the target surgical scenario and the corresponding surgical space image; A labeling unit configured to perform anatomical structure recognition and preliminary labeling on the first image frame using a surgical image labeling model to obtain a second image frame; the initial labeling information of the key anatomical structures corresponding to the key surgical operations is labeled in the second image frame; An optimization unit configured to push the second image frame to the client for review and receive the review result of the client to adjust the second image frame to obtain a third image frame; at least the optimized annotation information of the key anatomical structures is labeled in the third image frame; A construction unit configured to construct a surgical image dataset based on the third image frame and the corresponding optimized annotation information, and the surgical image dataset is used to train and optimize the surgical image model.

10. A computer-readable storage medium, characterized in that, A computer software program is stored in the storage medium, and when the computer software program is executed by a processor, it implements the surgical image processing method according to any one of claims 1-8.

Images