Method for constructing expert operation experience data set and automatic semantic segmentation system

Abstract

The invention provides a method for constructing an expert operation experience data set and an automatic semantic segmentation system. The method comprises the following steps: selecting representative frames from an original video stream by an expert physician according to a preset labeling protocol, and carrying out manual labeling; a refined semantic segmentation model based on the YOLO-SAM is trained; using the trained semantic segmentation model to segment other frames in the original video stream; and the segmentation result is fused with the original video stream data and the manual annotation data, and a structured expert operation experience data set is constructed. The automatic semantic segmentation system comprises an input module, a YOLO module, an SAM module and an output module, the YOLO module performs target coarse positioning on an image to generate a bounding box with a category label, and provides a bounding box prompt for the SAM module; the SAM module generates a pixel-level high-precision segmentation mask corresponding to each bounding box; a representative frame manually labeled in the method is used for training. According to the method, an expert operation experience data set can be constructed, and automatic labeling, semantic segmentation and the like can be realized.

Description

Technical Field

[0001] This invention belongs to the field of medical data processing and computer-aided surgery technology, and specifically refers to a method for constructing an expert surgical experience dataset and an automatic semantic segmentation system. Background Technology

[0002] In the field of surgery, although robot-assisted surgical systems are widely used, the mainstream technology still adopts a master-slave teleoperation mode, and its operational performance is highly dependent on the surgeon's personal experience and operating skills. Partial nephrectomy, as a typical laparoscopic surgery, is characterized by limited operating space, complex vascular pathways, and varied surgical steps.

[0003] However, there is currently a lack of standardized, fine-grained annotation methods for such surgical scenarios, and a complete integrated dataset of "surgical images-segmentation results" has not yet been established. How to effectively utilize neural network algorithms to learn, transfer, and apply expert surgical experience data, and thus provide surgeons with intelligent guidance and strategy support at the level of expert experience, has become a core technical bottleneck that urgently needs to be solved in the process of promoting the intelligent and autonomous development of surgical robots. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a method for constructing an expert surgical experience dataset and an automatic semantic segmentation system, the technical solution of which is as follows: On the one hand, a method for constructing an expert surgical experience dataset is provided, the method comprising: S1. Acquire the raw video stream of a partial nephrectomy; S2. Representative frames are selected from the original video stream by expert physicians according to a preset annotation protocol, and the representative frames are manually annotated. S3. Using manually annotated representative frames, train a refined semantic segmentation model based on YOLO-SAM; S4. Using the trained semantic segmentation model, segment other frames in the original video stream; S5. The segmentation results are fused with the original video stream data and manually labeled data to construct a structured dataset of expert surgical experience.

[0005] Optionally, the preset annotation protocols in S2 include five annotation protocols: annotation frame selection, label category, annotation workflow, general annotation rules, and specific annotation rules.

[0006] Optionally, the selection of the annotation frame includes: For each suturing action segment in the surgical video, four representative frames are labeled: the starting frame of the suturing action segment and the frames at the lower quartile, median, and upper quartile. The label categories include: Robotic instruments, laparoscopic instruments, other surgical tools, and anatomical structures; The annotation workflow includes: The annotations were completed collaboratively by trained non-medical annotators and expert physicians, and all annotated images were reviewed and verified by multiple expert physicians with clinical experience. The general annotation rules include: Because the label category includes multi-joint instruments, in order to both represent the joints of the instrument and treat these parts as a whole, the label category adopts the following naming rules: For jointed instruments, the naming format is: <instrument name>-left / right / fourth arm-<instrument component>. Instrument components include the instrument end, instrument joints, and instrument rods. The left arm, right arm, and fourth arm refer to instruments located on the left and right sides, respectively, and instruments that appear as auxiliary fourth arm instruments, excluding the left and right side instruments within the frame. For articulated instruments, the naming format is <instrument name>-<instrument component>, where instrument components include the instrument end, instrument joint, and instrument rod; For general surgical instruments, their names correspond directly to their names; For anatomical structures, their naming corresponds directly to their names; The specific annotation rules include: (1) Each pixel is assigned a semantic category, and occluded objects are labeled according to their visible parts; (2) Pliers with holes are labeled as solid pliers; (3) Objects partially obscured by liquid are approximated based on their expected shape; (4) Completely submerged objects are not labeled; (5) Tool parts with weak lighting or blurred due to lighting will be excluded; (6) Blurred suture lines or instrument rods under low light conditions will extend to the edges of the image; (7) Motion blur on fast-moving components will be annotated with complete visible outlines; (8) The wound refers to the area remaining after the renal tumor is removed, and the edge of the wound is defined by the instrument cutting edge of the renal capsule; (9) Perirenal fat is considered part of the kidney markings, while pararenal fat is excluded; (10) Each label category has a color code.

[0007] Optionally, the semantic segmentation model includes a YOLO module and a SAM module. The YOLO module is used for coarse target localization, generating bounding boxes with category labels, and providing bounding box prompts to the SAM module. The SAM module is used to generate pixel-level high-precision segmentation masks corresponding to each bounding box.

[0008] Optionally, each record in the expert surgical experience dataset includes: original frame images aligned and associated by timestamps: original RGB images extracted from the surgical video stream, retaining the most original visual information; manually labeled data: pixel-level fine annotations performed by expert physicians on representative frames; and automatically labeled results: fine-grained segmentation masks generated by the semantic segmentation model for segmenting other frames.

[0009] On the other hand, a system for constructing an expert surgical experience dataset is provided, the system comprising: The acquisition module is used to acquire the raw video stream of a partial nephrectomy. The manual annotation module is used to select representative frames from the original video stream by expert physicians according to a preset annotation protocol, and to manually annotate the representative frames. The training module is used to train a refined semantic segmentation model based on YOLO-SAM using manually annotated representative frames. The segmentation module is used to segment other frames in the original video stream using the trained semantic segmentation model; The module is used to integrate the segmentation results with the original video stream data and manually labeled data to build a structured dataset of expert surgical experience.

[0010] On the other hand, an automatic semantic segmentation system is provided, which includes an input module, a YOLO module, a SAM module, and an output module; The input module is used to input images acquired during robot-assisted laparoscopic partial nephrectomy. The YOLO module is used to perform coarse target localization on the image, generate bounding boxes with category labels, and provide bounding box prompts to the SAM module. The SAM module is used to generate a pixel-level high-precision segmentation mask corresponding to each bounding box; The output module is used to output the high-precision segmentation mask as the automatic annotation result or semantic segmentation result of the image; The automatic semantic segmentation system is trained using representative frames manually annotated in the method.

[0011] Optionally, the automatic semantic segmentation system is further adapted to automatically annotate images of ordinary laparoscopic surgery and other medical surgeries. The ordinary laparoscopic surgery images and other medical surgeries are input into the automatic semantic segmentation system to obtain automatic annotation results.

[0012] Optionally, the automatic semantic segmentation system is further adapted to perform semantic segmentation on ordinary laparoscopic surgical images and other medical surgical images. The ordinary laparoscopic surgical images and other medical surgical images are input into the automatic semantic segmentation system to obtain semantic segmentation results.

[0013] Optionally, the automatic semantic segmentation system is further applicable to other downstream clinical automation tasks, including but not limited to: predicting needle entry and exit points and making decisions on key clinical surgical steps.

[0014] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method for constructing an expert surgical experience dataset.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above-described method for constructing an expert surgical experience dataset.

[0016] The beneficial effects of the technical solution provided by this invention include at least the following: (1) This invention achieves refined segmentation of various instruments and tissues in kidney surgery scenarios, with segmentation accuracy reaching the pixel level. It can identify and segment more than 12 types of surgical instruments and anatomical structures such as wounds, kidneys, and tumors, including robotic needle holders, laparoscopic scissors, suction devices, sutures, Hem-o-lock clips, etc. For surgical instruments, this invention can further segment the functional components of the instruments (such as ends, joints, and rods), which provides richer information for subsequent analysis of the instrument's operating posture and usage techniques. Experiments show that on a partial nephrectomy dataset, the mIoU (mean intersection-over-union ratio) of the instrument and tissue segmentation of this invention both reach over 80%.

[0017] (2) This invention significantly improves the efficiency and consistency of data annotation. Traditional pixel-level manual annotation of each image frame takes 30-60 minutes, while the automated process based on YOLO-SAM in this invention can complete the segmentation of a single image frame in seconds. Only minor corrections are required by experts later, improving the efficiency of traditional manual annotation by tens of times. At the same time, through standardized protocols, including instrument component definition standards, tissue boundary judgment rules, and occlusion handling specifications, the consistency of annotation quality is guaranteed, effectively solving the problem of subjective differences between different annotators. This efficient annotation capability makes it possible to construct large-scale, high-quality datasets, providing sufficient data support for deep learning model training.

[0018] (3) This invention enhances the robustness of the model in real surgical environments. Through the collaborative working mechanism of YOLO-SAM, it effectively overcomes surgical environment interferences such as tissue deformation, instrument occlusion, and lighting changes. The YOLO model first provides reliable target localization, accurately identifying the target even under undesirable lighting or partial occlusion. The SAM model performs segmentation under the guidance of the bounding box provided by YOLO, avoiding the blindness of searching the entire image and significantly improving the segmentation stability in complex backgrounds and dynamic environments. The results of this high-precision semantic segmentation can be used to identify and label safe operating areas and dangerous areas (such as key anatomical structures like the renal vein and renal artery), providing surgeons with intuitive safety boundary prompts and danger area warnings, effectively avoiding intraoperative damage and improving surgical safety. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method for constructing an expert surgical experience dataset provided by an embodiment of the present invention; Figure 2 This is a structural block diagram of a system for constructing an expert surgical experience dataset provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of an automatic semantic segmentation system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] This invention provides a method for constructing an expert surgical experience dataset, which can be implemented by an electronic device, such as a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps: S1. Acquire the raw video stream of a partial nephrectomy; This invention involves acquiring raw video streams during robot-assisted laparoscopic partial nephrectomy.

[0023] S2. Representative frames are selected from the original video stream by expert physicians according to a preset annotation protocol, and the representative frames are manually annotated. In this invention, expert physicians annotate key surgical instruments and human tissues such as "kidney," "tumor," and "wound" on selected representative frames. To improve the quality and consistency of manual image annotation and ensure that annotators and urologists have received appropriate training, thereby minimizing annotation discrepancies, five annotation protocols are pre-defined.

[0024] Optionally, the preset annotation protocols in S2 include five annotation protocols: annotation frame selection, label category, annotation workflow, general annotation rules, and specific annotation rules.

[0025] Optionally, the selection of the annotation frame includes: For each suturing action segment in the surgical video, four representative frames are labeled: the starting frame of the suturing action segment and the frames at the lower quartile, median, and upper quartile (for example, a video containing 100 suturing action segments will select 100 × 4 = 400 frames for semantic segmentation labeling). The label categories include: Robotic instruments (e.g., robotic needle holders, robotic grippers, robotic scissors), laparoscopic instruments (laparoscopic needle holders, laparoscopic scissors, Hem-o-lock releasers, Bulldog releasers), other surgical tools (e.g., suction devices, suture needles, sutures, Bulldog clips, Hem-o-locks, trocars), and anatomical structures (e.g., wounds, kidneys, ureters, resected tumors); The annotation workflow includes: The annotations were completed collaboratively by trained non-medical annotators and expert physicians. All annotated images were reviewed and verified by multiple expert physicians with clinical experience (such as four urologists with more than 10 years of clinical experience). The general annotation rules include: Because the label category includes multi-joint instruments, in order to both represent the joints of the instrument and treat these parts as a whole, the label category adopts the following naming rules: For articulated instruments (such as robotic needle holders and robotic grippers), the naming format is: <instrument name>-left / right / fourth arm-<instrument component>. Instrument components include instrument end effector, instrument joints, and instrument rods. Left arm, right arm, and fourth arm refer to instruments located on the left and right sides, respectively, and instruments that appear as auxiliary fourth arm instruments in addition to the left and right side instruments within the frame. For articulated instruments (such as robotic scissors, laparoscopic needle holders, laparoscopic scissors, and Hem-o-lock release devices), the naming format is <instrument name>-<instrument component>, where instrument components include the instrument end effector, instrument joints, and instrument rods; For general surgical instruments (such as suction devices, suture needles, sutures, Bulldog clips, Hem-o-locks, and puncture instruments), their names correspond directly to their names. For anatomical structures (such as wounds, kidneys, renal arteries, renal veins, ureters, and resected tumors), their names correspond directly to their names; The dataset constructed in this embodiment of the invention includes surgical scenarios (such as insufficient light at the frame edges) and anatomical structures that are more complex than other surgical datasets. In order to cope with various different operating conditions and extreme situations, and to preserve contextual information even when there is insufficient light or liquid partial occlusion causing objects to be unclear, a set of specific standard rules has been carefully designed.

[0026] The specific annotation rules include: (1) Each pixel is assigned a semantic category, and occluded objects are labeled according to their visible parts; (2) Pliers with holes (e.g., grippers) are labeled as solid pliers; (3) Objects partially obscured by liquid are approximated based on their expected shape; (4) Completely submerged objects are not labeled; (5) Tool parts with weak lighting or blurred due to lighting will be excluded; (6) Blurred suture lines or instrument rods under low light conditions will extend to the edges of the image; (7) Motion blur on fast-moving parts (e.g., sutures) will be annotated with the complete visible outline; (8) The wound refers to the area remaining after the renal tumor is removed, and the edge of the wound is defined by the instrument cutting edge of the renal capsule; (9) Perirenal fat is considered part of the kidney markings, while pararenal fat is excluded; (10) Each label category has a color code (e.g., instrument end: #33FFFF, instrument joint: #00FF00).

[0027] The above outlines five annotation protocols established in this invention: a multi-level, fine-grained annotation category system defined specifically for the characteristics of kidney surgery; a fine-grained segmentation category system comprising more than 12 categories; and, in particular, functional component decomposition of surgical instruments (ends, joints, and rods). Five annotation protocols are established to ensure annotation consistency. This forms the foundation for constructing a "fine-grained" dataset. Without such refined category definitions, the dataset would remain at a coarse-grained "instrument-tissue" level, and the constructed dataset would fail to accurately reflect the key details of expert surgical experience. Without standardized annotation protocols, the results from different annotators would vary significantly, compromising data quality.

[0028] The role of this protocol in this embodiment of the invention is to form an initial data pool, providing a data foundation for subsequent fine-grained analysis and correlation.

[0029] After constructing the original data pool, this embodiment of the invention uses a refined semantic segmentation model based on YOLO-SAM to solve the coarse-grained segmentation problem.

[0030] S3. Using manually annotated representative frames, train a refined semantic segmentation model based on YOLO-SAM; Optionally, the semantic segmentation model includes a YOLO (You Only Look Once, YOLO) module and a SAM (Segment Anything Model, SAM) module. The YOLO module is used for coarse target localization, generating bounding boxes with category labels, and providing bounding box prompts to the SAM module. The SAM module is used to generate pixel-level high-precision segmentation masks corresponding to each bounding box.

[0031] S4. Using the trained semantic segmentation model, segment other frames in the original video stream; Optionally, the YOLO module is used for coarse target localization and prompt generation: Input: Frames from the original video stream other than the representative frames; Processing: Other frames are input into the trained YOLO module, which quickly and accurately identifies all targets of interest in the image, such as surgical instruments, kidneys, tumors, wound areas, etc., and generates an accurate bounding box for each identified target.

[0032] Output: A series of bounding boxes with category labels.

[0033] In this embodiment of the invention, the key function is to enable rapid and automatic localization of critical targets. It narrows the "where to look" information from a full-image search to a specific bounding box region, significantly reducing the complexity of subsequent segmentation tasks and providing precise bounding box prompts for the SAM module. This solves the problems of inefficient manual annotation and the blind calculations performed on the entire image by general segmentation models.

[0034] Optionally, the SAM module is used to implement fine-grained pixel-level segmentation: Input: a) The original image from other frames; b) A bounding box prompt from the YOLO module.

[0035] Processing: The original image and each bounding box cue are input into the SAM model. SAM is a powerful and general-purpose segmentation model characterized by its ability to achieve high-quality, zero-sample, pixel-level segmentation guided by cue points. In this embodiment of the invention, the bounding boxes provided by YOLO serve as input cue points for SAM, guiding SAM to perform fine-grained segmentation within those boxes.

[0036] Output: Pixel-level high-precision segmentation masks corresponding to each bounding box. These masks clearly delineate the kidney parenchyma, tumor boundaries, irregular shapes of the wound, and precise outlines of surgical instruments.

[0037] In this embodiment of the invention, the SAM model is crucial for achieving fine-grained segmentation. Leveraging its powerful generalization ability and edge segmentation accuracy, and guided by YOLO, the SAM model can overcome interference from tissue deformation, occlusion, and lighting variations, outputting segmentation results far exceeding those of traditional methods. This directly addresses the core deficiency of existing technologies—the inability to distinguish fine structures.

[0038] This invention presents a kidney surgery data processing workflow that combines manual annotation with automated segmentation. It establishes a complete data processing flow from original surgical video acquisition and expert manual annotation to automated segmentation enhancement based on YOLO-SAM, forming a standardized dataset construction paradigm. This is the foundation for building high-quality datasets. Without systematic manual annotation guidance, automated segmentation alone cannot guarantee medical accuracy; without subsequent automated segmentation enhancement, relying solely on manual annotation makes it difficult to construct large-scale datasets. The YOLO model is used for rapid detection and coarse localization of multiple targets in the surgical scene, generating bounding boxes as spatial cues. These bounding box cues, along with the original image, are then input into the SAM model to achieve pixel-level fine segmentation. This is the core architecture for achieving high-efficiency, high-precision segmentation. Without YOLO's precise localization, the SAM model would blindly search the entire image, resulting in high computational cost and difficulty in guaranteeing accuracy; without SAM's fine segmentation capabilities, YOLO's bounding boxes alone cannot obtain pixel-level detail information. The collaborative work of both models resolves the efficiency and accuracy contradiction faced by a single model in complex surgical scenarios.

[0039] S5. The segmentation results are fused with the original video stream data and manually labeled data to construct a structured dataset of expert surgical experience.

[0040] In this embodiment of the invention, after processing a large number of surgical videos using the above-mentioned "annotation-training-segmentation" method, the segmentation results are fused with the original data and the annotated data to construct a structured dataset of expert surgical experience.

[0041] Optionally, each record in the expert surgical experience dataset includes: original frame images aligned and associated by timestamps: original RGB images extracted from the surgical video stream, retaining the most original visual information; manually labeled data: pixel-level fine annotations performed by expert physicians on representative frames; and automatically labeled results: fine-grained segmentation masks generated by the semantic segmentation model for segmenting other frames.

[0042] Optionally, the fine-grained segmentation mask includes pixel-level labels for multiple semantic categories, specifically including: Robotic instruments (including instrument end effector, instrument joints, and instrument rods, distinguishing between left and right arms): robotic needle holder, robotic gripper, robotic scissors b. Laparoscopic instruments (including instrument ends and levers): laparoscopic needle holder, laparoscopic scissors, Hem-o-lock release device, Bulldog release device c. Other surgical instruments: suction device, suture needles, sutures, Bulldog clips, Hem-o-lock, trocar. d. Anatomical structures: wound, kidney, removed tumor.

[0043] like Figure 2 As shown, this embodiment of the invention also provides a system for constructing an expert surgical experience dataset, the system comprising: Acquisition module 210 is used to acquire the raw video stream of partial nephrectomy. The manual annotation module 220 is used to select representative frames from the original video stream by expert physicians according to a preset annotation protocol, and to manually annotate the representative frames. Training module 230 is used to train a refined semantic segmentation model based on YOLO-SAM using manually annotated representative frames; The segmentation module 240 is used to segment other frames in the original video stream using the trained semantic segmentation model; Module 250 is used to fuse the segmentation results with the original video stream data and manually labeled data to build a structured dataset of expert surgical experience.

[0044] The system for constructing an expert surgical experience dataset provided in this embodiment of the invention has a functional structure that corresponds to the method for constructing an expert surgical experience dataset provided in this embodiment of the invention, and will not be described again here.

[0045] like Figure 3 As shown, this embodiment of the invention also provides an automatic semantic segmentation system, which includes an input module 310, a YOLO module 320, a SAM module 330, and an output module 340. The input module 310 is used to input images acquired during robot-assisted laparoscopic partial nephrectomy. The YOLO module 320 is used to perform coarse target localization on the image, generate bounding boxes with category labels, and provide bounding box prompts to the SAM module. The SAM module 330 is used to generate a pixel-level high-precision segmentation mask corresponding to each bounding box; The output module 340 is used to output the high-precision segmentation mask as the automatic annotation result or semantic segmentation result of the image; The automatic semantic segmentation system is trained using representative frames manually annotated in the method.

[0046] Although the automatic semantic segmentation system of this invention uses robotic laparoscopic surgery as its training data, the proposed annotation protocol and automatic annotation framework can be transferred to ordinary laparoscopic surgery scenarios and other medical surgical scenarios.

[0047] Optionally, the automatic semantic segmentation system is further adapted to automatically annotate images of ordinary laparoscopic surgery and other medical surgeries. The ordinary laparoscopic surgery images and other medical surgeries are input into the automatic semantic segmentation system to obtain automatic annotation results.

[0048] Optionally, the automatic semantic segmentation system is further adapted to perform semantic segmentation on ordinary laparoscopic surgical images and other medical surgical images. The ordinary laparoscopic surgical images and other medical surgical images are input into the automatic semantic segmentation system to obtain semantic segmentation results.

[0049] Optionally, the automatic semantic segmentation system is further applicable to other downstream clinical automation tasks, including but not limited to: predicting needle entry and exit points and making decisions on key clinical surgical steps.

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present invention. The electronic device 400 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 401 and one or more memories 402. The memory 402 stores at least one instruction, which is loaded and executed by the processor 401 to implement the steps of the method for constructing the expert surgical experience dataset described above.

[0051] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the method for constructing an expert surgical experience dataset. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0052] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing an expert surgical experience dataset, characterized in that, The method includes: S1. Acquire the raw video stream of a partial nephrectomy; S2. Representative frames are selected from the original video stream by expert physicians according to a preset annotation protocol, and the representative frames are manually annotated. S3. Using manually annotated representative frames, train a refined semantic segmentation model based on YOLO-SAM; S4. Using the trained semantic segmentation model, segment other frames in the original video stream; S5. The segmentation results are fused with the original video stream data and manually labeled data to construct a structured dataset of expert surgical experience.

2. The method according to claim 1, characterized in that, The preset annotation protocols in S2 include five annotation protocols: annotation frame selection, label category, annotation workflow, general annotation rules, and specific annotation rules.

3. The method according to claim 2, characterized in that, The selection of the labeled frame includes: For each suturing action segment in the surgical video, four representative frames are labeled: the starting frame of the suturing action segment and the frames at the lower quartile, median, and upper quartile. The label categories include: Robotic instruments, laparoscopic instruments, other surgical tools, and anatomical structures; The annotation workflow includes: The annotations were completed collaboratively by trained non-medical annotators and expert physicians, and all annotated images were reviewed and verified by multiple expert physicians with clinical experience. The general annotation rules include: Because the label category includes multi-joint instruments, in order to both represent the joints of the instrument and treat these parts as a whole, the label category adopts the following naming rules: For jointed instruments, the naming format is: <instrument name>-left / right / fourth arm-<instrument component>. Instrument components include the instrument end, instrument joints, and instrument rods. The left arm, right arm, and fourth arm refer to instruments located on the left and right sides, respectively, and instruments that appear as auxiliary fourth arm instruments, excluding the left and right side instruments within the frame. For articulated instruments, the naming format is <instrument name>-<instrument component>, where instrument components include the instrument end, instrument joint, and instrument rod; For general surgical instruments, their names correspond directly to their names; For anatomical structures, their naming corresponds directly to their names; The specific annotation rules include: (1) Each pixel is assigned a semantic category, and occluded objects are labeled according to their visible parts; (2) Pliers with holes are labeled as solid pliers; (3) Objects partially obscured by liquid are approximated based on their expected shape; (4) Completely submerged objects are not labeled; (5) Tool parts with weak lighting or blurred due to lighting will be excluded; (6) Blurred suture lines or instrument rods under low light conditions will extend to the edges of the image; (7) Motion blur on fast-moving components will be annotated with complete visible outlines; (8) The wound refers to the area remaining after the renal tumor is removed, and the edge of the wound is defined by the instrument cutting edge of the renal capsule; (9) Perirenal fat is considered part of the kidney markings, while pararenal fat is excluded; (10) Each label category has a color code.

4. The method according to claim 1, characterized in that, The semantic segmentation model includes a YOLO module and a SAM module. The YOLO module is used for coarse target localization, generating bounding boxes with category labels, and providing bounding box prompts to the SAM module. The SAM module is used to generate pixel-level high-precision segmentation masks corresponding to each bounding box.

5. The method according to claim 1, characterized in that, Each record in the expert surgical experience dataset includes original frame images aligned and associated by timestamps: original RGB images extracted from the surgical video stream, preserving the most original visual information; Manually labeled data: pixel-level fine-grained annotations of representative frames by expert physicians; Automatically labeled results: fine-grained segmentation masks generated by the semantic segmentation model for segmenting other frames.

6. A system for constructing an expert surgical experience dataset, characterized in that, The system includes: The acquisition module is used to acquire the raw video stream of a partial nephrectomy. The manual annotation module is used to select representative frames from the original video stream by expert physicians according to a preset annotation protocol, and to manually annotate the representative frames. The training module is used to train a refined semantic segmentation model based on YOLO-SAM using manually annotated representative frames. The segmentation module is used to segment other frames in the original video stream using the trained semantic segmentation model; The module is used to integrate the segmentation results with the original video stream data and manually labeled data to build a structured dataset of expert surgical experience.

7. An automatic semantic segmentation system, characterized in that, The automatic semantic segmentation system includes an input module, a YOLO module, a SAM module, and an output module; The input module is used to input images acquired during robot-assisted laparoscopic partial nephrectomy. The YOLO module is used to perform coarse target localization on the image, generate bounding boxes with category labels, and provide bounding box prompts to the SAM module. The SAM module is used to generate a pixel-level high-precision segmentation mask corresponding to each bounding box; The output module is used to output the high-precision segmentation mask as the automatic annotation result or semantic segmentation result of the image; The automatic semantic segmentation system is trained using representative frames manually annotated according to any one of the methods described in claims 1-5.

8. The system according to claim 7, characterized in that, The automatic semantic segmentation system is further applicable to the automatic annotation of ordinary laparoscopic surgical images and other medical surgical images. The ordinary laparoscopic surgical images and other medical surgical images are input into the automatic semantic segmentation system to obtain automatic annotation results.

9. The system according to claim 7, characterized in that, The automatic semantic segmentation system is further applicable to semantic segmentation of ordinary laparoscopic surgical images and other medical surgical images. The ordinary laparoscopic surgical images and other medical surgical images are input into the automatic semantic segmentation system to obtain semantic segmentation results.

10. The system according to claim 7, characterized in that, The automatic semantic segmentation system is further applicable to other downstream clinical automation tasks, including but not limited to: predicting needle entry and exit points and making decisions on key clinical surgical steps.

Images