Submaxillary gland operation process identification method based on deep learning

Abstract

The invention discloses a submaxillary gland operation process identification method based on deep learning, and belongs to the technical field of deep learning image processing, and the method comprises the steps: collecting and sorting the video data of a minimally-invasive submaxillary gland operation, and constructing a minimally-invasive submaxillary gland operation video database; according to the anatomical structure and the operation steps of the submaxillary gland operation, an operation process standardization labeling scheme is made; carrying out key frame extraction on the operation video data, and labeling each frame of video according to an operation process standardized labeling scheme; based on a residual neural network and a long-short-term memory network, constructing a deep learning model fusing spatial features and time sequence features; training a deep learning model fusing spatial features and time sequence features based on the labeled operation video data; and analyzing and evaluating the operation of the operator in the operation process by using the trained deep learning model. According to the method, an objective and efficient operation evaluation mechanism can be provided, and skill improvement of an operator is assisted.

Description

Technical Field

[0001] This application relates to the field of deep learning image processing technology, and in particular to a deep learning-based method for recognizing the surgical procedure of the submandibular gland. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, its application in the medical field is becoming increasingly widespread, particularly in preoperative diagnosis, treatment planning, and prognosis prediction, where significant progress has been made. Surgery, as a crucial component of modern medicine, is characterized by its complexity, large data volume, and dynamic changes. Currently, the application of AI in surgical procedures is still in its early stages. Achieving a deeper understanding of the surgical process through AI is a key step in promoting automated and even autonomous surgery.

[0003] The application of artificial intelligence in oral medicine is mainly focused on tasks based on static images, such as image analysis and pathological identification, including caries detection, periodontal disease diagnosis, and oral tumor identification. However, specific research methods are still lacking for the analysis of oral surgical procedures, especially surgical workflows. Surgical workflow analysis not only helps assess the surgeon's skills but also provides real-time operational suggestions at different surgical stages, assisting doctors in making more accurate judgments and reducing surgical risks. Furthermore, building a learning database of surgical video data to provide reference for junior clinicians can also help improve medical standards. Surgical procedures naturally possess both image and temporal characteristics; therefore, it is necessary to build models capable of simultaneously processing spatial and temporal information to achieve accurate identification and analysis of surgical workflows.

[0004] Currently, the field of oral surgery faces the following main challenges: A lack of high-quality surgical video data: Due to the complexity of surgical procedures and patient privacy protection, obtaining high-quality surgical video data is difficult; Unclear definitions of surgical steps and procedures: Some oral and maxillofacial surgical steps and procedures lack unified annotation and clear definitions, leading to potential subjectivity and inconsistencies in data annotation. These challenges limit the training and validation of artificial intelligence, increase the learning difficulty, and affect its application effectiveness in surgical procedure recognition and analysis. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this application provides a deep learning-based method for identifying the surgical procedure of the submandibular gland, which can identify and analyze the surgical process and thus assess the surgeon's skills.

[0006] To achieve the aforementioned objectives, the technical solution adopted in this application is as follows: This application provides a deep learning-based method for recognizing the surgical procedure of the submandibular gland, including: S1: Collect and organize video data of minimally invasive submandibular gland surgery, and build a video data library of minimally invasive submandibular gland surgery. S2: Based on the anatomical structure and operation steps of submandibular gland surgery, develop a standardized labeling scheme for the surgical procedure; S3: Extract keyframes from the surgical video data and annotate each video frame according to the standardized annotation scheme of the surgical procedure; S4: Construct a deep learning model that integrates spatial features and time series features based on residual neural networks and long short-term memory networks; S5: Training a deep learning model that integrates spatial and temporal features based on labeled surgical video data; S6: Utilize a trained deep learning model that integrates spatial and temporal features to analyze and evaluate the surgeon's actions during the surgical procedure.

[0007] Furthermore, the step of extracting keyframes from the surgical video data and annotating each video frame according to a standardized annotation scheme for the surgical procedure includes: S301: Extract keyframes from each video segment in the surgical video data by extracting one frame per second to obtain key image information during the surgical process. S302: According to the standardized annotation scheme of the surgical procedure, a portion of the surgical video was extracted, and the extracted keyframes were independently annotated frame by frame. S303: Check the consistency of the annotations. If the consistency coefficient is greater than 0.9, then the remaining videos should be annotated together. S304: Review the annotated surgical video data to ensure the accuracy of the annotated data.

[0008] Furthermore, in the deep learning model that integrates spatial features and temporal series features, the residual neural network extracts spatial image features from the surgical video frames; the long short-term memory network captures the temporal dynamic features of the surgical procedure in the surgical video; and Softmax integrates the spatial image features and temporal dynamic features.

[0009] Furthermore, the training of the deep learning model fusing spatial and temporal features based on the labeled surgical video data includes: A deep learning model that integrates spatial and temporal features was trained and evaluated using a 5-fold cross-validation method.

[0010] Furthermore, the analysis and evaluation of the surgeon's actions during the surgical procedure using a trained deep learning model that integrates spatial and temporal features includes: S601: Input the surgeon's minimally invasive submandibular gland surgery video into a trained deep learning model that integrates spatial and temporal features. The deep learning model automatically identifies and labels each stage of the surgical procedure. S602: Analyze the surgical phase sequence output by the deep learning model, and calculate the frequency of hesitation behavior and the number of transitions between hesitation behaviors during the operation. S603: Based on the frequency of hesitant behaviors and the number of transitions between hesitant behaviors, analyze and evaluate the surgeon's operational fluency and decision-making decisiveness, thereby completing the assessment of the surgeon's operational skills.

[0011] Furthermore, the deep learning-based method for recognizing the surgical procedure of the submandibular gland also includes: The annotation of new surgical videos was completed using a trained deep learning model that integrates spatial and temporal features.

[0012] Furthermore, the annotation of the new surgical video using a trained deep learning model that integrates spatial and temporal features includes: The new surgical video is input into a pre-trained deep learning model that integrates spatial and temporal features. The deep learning model automatically generates preliminary surgical procedure annotations. The annotators review and revise the initial surgical procedure annotations to complete the annotation work.

[0013] The beneficial effects of this application are: This application provides a deep learning-based method for recognizing the surgical procedure in submandibular gland surgery. It employs a standardized annotation system for minimally invasive submandibular gland surgery procedures, constructing a repeatable surgical data annotation framework, providing a solid foundation for subsequent video processing and analysis of this type of surgery. Furthermore, by introducing a deep learning model that fuses image and temporal features, the surgical procedure can be identified more accurately, improving the model's recognition precision. In addition, the trained model can be used to evaluate the surgeon's operational skills, providing an objective and efficient surgical evaluation mechanism to assist in skill improvement. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart illustrating a deep learning-based method for recognizing the surgical procedure of the submandibular gland, as provided in an embodiment of this application.

[0016] Figure 2 This is a schematic diagram illustrating a minimally invasive submandibular gland surgery procedure provided in an embodiment of this application.

[0017] Figure 3 This is a schematic diagram of a video extraction and annotation scheme provided in an embodiment of this application.

[0018] Figure 4 A deep learning model framework diagram provided for embodiments of this application.

[0019] Figure 5 This is a schematic diagram of the overall process of a deep learning-based submandibular gland surgery procedure recognition method provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0021] This application provides a standardized and reproducible annotation scheme for minimally invasive submandibular gland surgery, and develops a deep learning-based intelligent submandibular gland surgery procedure recognition method based on the characteristics of the surgery. This method can identify and analyze the surgical process, thereby assessing the surgeon's skills. (See also: [link to relevant documentation]). Figure 1 , Figure 1 The diagram shown is a flowchart illustrating a deep learning-based method for identifying the surgical procedure of the submandibular gland, as provided in an embodiment of this application. The method includes: S1: Construction of a video database for minimally invasive submandibular gland surgery: Collect and organize high-quality video data of minimally invasive submandibular gland surgery, and establish a surgical video database covering various surgical scenarios and surgeons.

[0022] S2: Development of a standardized surgical procedure annotation scheme: Based on the anatomical structure and operation steps of submandibular gland surgery, a standardized and repeatable surgical procedure annotation scheme is developed, which clarifies the definition and boundaries of each surgical stage to ensure the consistency and accuracy of data annotation.

[0023] like Figure 2 As shown, Figure 2This paper presents a labeling scheme for minimally invasive submandibular gland surgery, comprising six stages: cavity creation, localization, dissection, examination, and deliberation. In the cavity creation stage, the labeling scheme is as follows: dissect along the deep surface of the platysma muscle until reaching the anterior border of the submandibular gland, creating sufficient surgical space. The starting frame is when the scalpel first appears in the surgical field, and the ending frame is when the dissection along the deep surface of the platysma muscle reaches the anterior border of the submandibular gland, at which point the scalpel leaves the field of view. In the localization stage, the labeling scheme is as follows: the facial vein, located deep to the platysma muscle and passing through the surface of the submandibular gland, is an important anatomical landmark for locating the submandibular gland in endoscopic surgery. In the gland localization stage, the surgeon typically cuts the facial vein to precisely locate the gland. The starting frame is when the scalpel appears in the surgical field to locate the submandibular gland, and the ending frame is when the capsule of the submandibular gland is exposed or the facial vein is cut. During the dissection phase, after the submandibular gland is clearly exposed, the surgeon uses surgical instruments to completely separate the gland from the surrounding tissue and remove the specimen from the surgical field. Its starting frame is when the scalpel first appears in the surgical field after initial exposure of the submandibular gland, and its ending frame is when the submandibular gland is no longer visible in the field. During the examination phase, the procedure is as follows: after removal of the submandibular gland, the surgeon irrigates or examines the surgical area to assess the wound, ensure hemostasis, check for bleeding, and place a drainage tube. Its starting frame is when surgical instruments appear in the surgical field for wound examination, and its ending frame is when the surgical instruments are removed from the surgical field. During the hesitation phase, which includes changing surgical instruments, cleaning the surgical field, and considering subsequent surgical procedures, its starting frame is when the surgical instruments are not visible in the field or remain relatively still for more than three consecutive frames, and its ending frame is the starting frame of any other phase (cavitation, localization, dissection, examination). It is important to note that if an operation occurs outside the current stage, the start time is marked when the scalpel first contacts the non-target tissue, and the current stage record is restored when it re-contacts the target tissue; scalpels include monopolar energy devices, bipolar energy devices, ultrasonic devices, and steel scalpels.

[0024] S3: Data Processing: Keyframes were extracted from the acquired surgical videos, and each frame was annotated according to the established annotation scheme. 10% of the videos were selected for simultaneous annotation by two annotators, and consistency was checked. After passing the consistency check, the annotations were separated. The annotated results were reviewed by multiple senior physicians, and any uncertainties were discussed and analyzed before finalization.

[0025] like Figure 3 As shown, Figure 3 A video extraction and annotation scheme is provided, which preprocesses the original surgical video by downsampling, extracts key frames, and annotates the video frame by frame.

[0026] S4: Construction and Training of Deep Learning Models: Based on Residual Neural Network (ResNet50) and Long Short-Term Memory Network (LSTM), a deep learning model integrating spatial and temporal features is constructed. ResNet50 is used to extract spatial image features from surgical video frames, while LSTM is used to capture temporal dynamic features in the surgical procedure, achieving accurate identification of the surgical process.

[0027] like Figure 4 As shown, Figure 4 The deep learning model framework in this application is provided. ResNet50 extracts spatial image features from surgical video frames, and then LSTM captures the temporal dynamic features in the surgical procedure. The two are fused through Softmax to output annotation operations.

[0028] S5: Using a trained model, analyze the surgeon's actions during the surgical process, assess their skill level, provide objective feedback to the surgeon, and assist in improving their skills.

[0029] In one embodiment of this application, such as Figure 5 As shown, video data of minimally invasive submandibular gland surgery was first collected and preliminarily evaluated to exclude videos that did not meet quality requirements, such as those with incomplete surgical procedures, interrupted recordings, or blurry images. Videos meeting the requirements were then uniformly numbered and organized. Subsequently, keyframes were extracted from each video segment at a rate of one frame per second to capture key image information during the surgery. Following a pre-established standardized annotation scheme, each extracted keyframe was annotated frame by frame. To ensure consistency and accuracy of annotation, 10% of the videos were randomly selected and annotated independently by two or more annotators. After annotation, Cohen's Kappa coefficient was used for consistency testing. If the consistency coefficient was greater than 0.9, it was considered highly consistent, and the remaining videos were annotated collaboratively by multiple annotators. Finally, all annotation results were submitted to three senior physicians with extensive clinical experience for review. For any uncertainties or controversies during the annotation process, expert discussions were organized, and consensus was reached based on clinical practice and surgical guidelines to ensure the accuracy and authority of the annotated data.

[0030] After extracting and annotating keyframes from the surgical videos, a deep learning model fusing ResNet50 and Long Short-Term Memory (LSTM) networks was used to identify the minimally invasive submandibular gland surgery procedure. To ensure the model's generalization ability and stability, a single surgical video was used as the smallest unit of division, and a 5-fold cross-validation method was employed to evaluate model performance. All videos were randomly distributed as much as possible to ensure a roughly balanced distribution of cases in each fold. During cross-validation, one fold was selected as the validation set in each iteration, and the remaining four folds were used as the training set. In each round of training, the model's performance metrics on the validation set were recorded, including accuracy, recall, precision, F1 score, and area under the receiver operating system (AOS). The model was trained and evaluated on the validation set, and the evaluation metrics were recorded. The above steps were repeated until each subset served as a validation set, resulting in five evaluation metrics. The average of the five evaluation metrics was taken as the final model performance evaluation metric.

[0031] After training the deep learning model, it was applied to evaluate videos of minimally invasive submandibular gland surgery performed on new surgeons. Specifically, the surgical video of a new surgeon was input into the trained model, which automatically identified and labeled each stage of the surgical procedure. By analyzing the surgical stage sequence output by the model, the frequency of "hesitation" behavior during the surgery and the number of transitions between "hesitation" behaviors were calculated. These indicators can be used to quantify the surgeon's operational fluency and decision-making decisiveness, thereby providing an objective evaluation of their surgical skills. Furthermore, the trained model can also be used to assist in the annotation of new surgical videos. Specifically, the new surgical video is input into the model, which automatically generates preliminary surgical procedure annotations. Subsequently, the model's pre-annotated results are manually reviewed and revised to ensure accuracy and consistency. This method significantly reduces the workload of manual annotation and improves annotation efficiency.

[0032] This application provides a deep learning-based method for recognizing the surgical procedure in submandibular gland surgery. It employs a standardized annotation system for minimally invasive submandibular gland surgery procedures, constructing a repeatable surgical data annotation framework, providing a solid foundation for subsequent video processing and analysis of this type of surgery. Furthermore, by introducing a deep learning model that fuses image and temporal features, the surgical procedure can be identified more accurately, improving the model's recognition precision. In addition, the trained model can be used to evaluate the surgeon's operational skills, providing an objective and efficient surgical evaluation mechanism to assist in skill improvement.

[0033] It should be noted that those skilled in the art will recognize that the embodiments described herein are for the purpose of helping readers understand the principles of this application, and should be understood as not limiting the scope of protection of this application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this application without departing from the essence of this application, and these modifications and combinations are still within the scope of protection of this application.

Claims

1. A deep learning-based method for recognizing the surgical procedure of the submandibular gland, characterized in that, include: S1: Collect and organize video data of minimally invasive submandibular gland surgery, and build a video data library of minimally invasive submandibular gland surgery. S2: Based on the anatomical structure and operation steps of submandibular gland surgery, develop a standardized labeling scheme for the surgical procedure; S3: Extract keyframes from the surgical video data and annotate each video frame according to the standardized annotation scheme of the surgical procedure; S4: Construct a deep learning model that integrates spatial features and time series features based on residual neural networks and long short-term memory networks; S5: Training a deep learning model that integrates spatial and temporal features based on labeled surgical video data; S6: Utilize a trained deep learning model that integrates spatial and temporal features to analyze and evaluate the surgeon's actions during the surgical procedure.

2. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 1, characterized in that, The process of extracting keyframes from surgical video data and annotating each video frame according to a standardized annotation scheme for the surgical procedure includes: S301: Extract keyframes from each video segment in the surgical video data by extracting one frame per second to obtain key image information during the surgical process. S302: According to the standardized annotation scheme of the surgical procedure, a portion of the surgical video was extracted, and the extracted keyframes were independently annotated frame by frame. S303: Check the consistency of the annotations. If the consistency coefficient is greater than 0.9, then the remaining videos should be annotated together. S304: Review the annotated surgical video data to ensure the accuracy of the annotated data.

3. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 1, characterized in that, In the deep learning model that integrates spatial and temporal features, the residual neural network extracts spatial image features from surgical video frames; the long short-term memory network captures the temporal dynamic features of the surgical procedure in the surgical video. Softmax fuses spatial image features and temporal dynamic features.

4. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 1, characterized in that, The training of a deep learning model that integrates spatial and temporal features based on labeled surgical video data includes: A deep learning model that integrates spatial and temporal features was trained and evaluated using a 5-fold cross-validation method.

5. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 1, characterized in that, The method of using a trained deep learning model that integrates spatial and temporal features to analyze and evaluate the surgeon's actions during the surgical procedure includes: S601: Input the surgeon's minimally invasive submandibular gland surgery video into a trained deep learning model that integrates spatial and temporal features. The deep learning model automatically identifies and labels each stage of the surgical procedure. S602: Analyze the surgical phase sequence output by the deep learning model, and calculate the frequency of hesitation behavior and the number of transitions between hesitation behaviors during the operation. S603: Based on the frequency of hesitant behaviors and the number of transitions between hesitant behaviors, analyze and evaluate the surgeon's operational fluency and decision-making decisiveness, thereby completing the assessment of the surgeon's operational skills.

6. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 1, characterized in that, Also includes: The annotation of new surgical videos was completed using a trained deep learning model that integrates spatial and temporal features.

7. The deep learning-based method for recognizing the surgical procedure of the submandibular gland according to claim 6, characterized in that, The annotation of new surgical videos using a trained deep learning model that integrates spatial and temporal features includes: The new surgical video is input into a pre-trained deep learning model that integrates spatial and temporal features. The deep learning model automatically generates preliminary surgical procedure annotations. The annotators review and revise the initial surgical procedure annotations to complete the annotation work.

Images