Shoulder arthroscopy operation assisting method and system based on deep learning
Through the deep learning-based auxiliary method of shoulder arthroscopic surgery, the problems of low accuracy and long learning cycle of shoulder arthroscopic surgery are solved, and the surgical accuracy and efficiency are improved, as well as the optimization and utilization of medical resources are achieved.
Patent Information
- Application Number
- CN202411535418.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-25
AI Technical Summary
Shoulder arthroscopy relies on traditional surgical methods, resulting in low surgical accuracy and surgeons need long-term training to achieve steady-state surgical results.
Using a shoulder arthroscopic surgery assisted method based on deep learning, collect expert surgical videos, frame processing and label pictures, and use DconnNet and RetinaNet models to extract features, establish a shoulder arthroscopic surgery assisted model, and assist in real-time surgery.
It improves the accuracy and efficiency of shoulder arthroscopy, shortens the doctor's learning cycle, improves the utilization of medical resources, and reduces the incidence of surgical complications.
Smart Images
Figure CN119515796A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence medical technology, and in particular to a shoulder arthroscopic surgery auxiliary method and system based on deep learning. Background Art
[0002] Currently, arthroscopic surgery has become the main surgical method for many shoulder diseases. Compared with traditional open surgery, arthroscopic surgery has smaller incisions, lower risk of deltoid injury, fewer complications, lower probability of re-admission, better intra-articular visibility, less postoperative pain, and faster postoperative recovery. Therefore, both surgeons and patients show a preference for arthroscopy. However, shoulder arthroscopy is technically more challenging than knee and other arthroscopic procedures because of the shape of the joint and the potential danger of traction. The complication rate of shoulder arthroscopy is high: 4.6% to 10.6%, and common complications are diverse: shoulder arthritis, nerve injury, pneumothorax, and iatrogenic rotator cuff tears, etc., and the incidence of complications increases with the complexity of the operation. However, the surgeon's experience, knowledge of surgical anatomy, and the correct use of instruments can improve the technical success rate and reduce complications.
[0003] At the same time, shoulder specialists need 30-50 Latarjet surgeries (one of the shoulder arthroscopy procedures) to achieve a steady-state surgical effect, and the learning curve is steep and long. Relying on traditional shoulder arthroscopy methods and the doctor's personal experience and qualifications is not enough to perform high-precision shoulder arthroscopy.
[0004] Therefore, it is necessary to use emerging technologies to assist in the implementation of shoulder arthroscopic surgery in order to improve surgical efficiency and accuracy and integrate medical resources more efficiently. Summary of the invention
[0005] The present invention provides a deep learning-based shoulder arthroscopic surgery auxiliary method and system to solve the defects of the prior art that shoulder arthroscopic surgery relies on traditional surgical methods resulting in low surgical accuracy.
[0006] In a first aspect, the present invention provides a shoulder arthroscopy surgery assisting method based on deep learning, comprising: Collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; Extracting features of the labeled image based on the DconnNet model to obtain image features; The RetinaNet model is used to learn the image features to obtain the lesion area to be determined; Using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels, and establishing a shoulder arthroscopy surgery auxiliary model; The real-time surgical data is input into the shoulder arthroscopy surgery assistance model, and the surgery assistance results are output.
[0007] According to a deep learning-based shoulder arthroscopy surgery assisting method provided by the present invention, an expert-level shoulder arthroscopy surgery video is collected, the shoulder arthroscopy surgery video is framed to obtain a framed image, and the framed image is labeled to obtain a labeled image, including: Performing frame processing on the shoulder arthroscopy surgery video according to the time series, removing abnormal data, and forming the framed image; The framed images are annotated using an annotation tool to form the annotated images, wherein the labels include lesion location and name labels, arthroscopic anatomical region labels, surgical instrument labels, and danger warning region labels.
[0008] According to a shoulder arthroscopy surgery auxiliary method based on deep learning provided by the present invention, the features of the labeled image are extracted based on the DconnNet model to obtain the image features, including: The annotated image is input into the DconnNet model, and the encoder of the DconnNet model performs deep feature extraction on the annotated image through a convolution layer and a pooling layer to generate a multi-scale output feature map; The decoder of the DconnNet model gradually reshapes the multi-scale output feature map into an image having the same size as the annotated image through deconvolution; The contrast loss function is used for training and optimization to obtain the image features.
[0009] According to a deep learning-based shoulder arthroscopy surgery assisting method provided by the present invention, the RetinaNet model learns the image features to obtain the lesion area to be determined, including: Initializing with RetinaNet weights pre-trained on a large dataset, and inputting the image features into the RetinaNet model; The RetinaNet model uses a built-in feature pyramid network structure to extract features at different scales and learn the features of the lesion area frame by frame; The model accuracy is optimized by using the Focal Loss function to determine the lesion area to be determined.
[0010] According to a shoulder arthroscopy surgery assisting method based on deep learning provided by the present invention, a deep learning model is used to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery assisting model, including: The data with surgical instrument labels are trained, with lesion labels as surgical targets, surgical instrument labels as surgical methods, and surgical operations as surgical output results, and potential confounding factors are identified; A causal diagram was used to represent the causal relationship between lesions, surgical instruments and surgical procedures, and confounding factors were identified in the causal diagram. The shoulder arthroscopy surgery auxiliary model is obtained by training the lesion area to be determined with the surgical instrument label added.
[0011] According to a shoulder arthroscopy surgery assisting method based on deep learning provided by the present invention, after using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels, and establishing a shoulder arthroscopy surgery assisting model, the method further includes: Get missed and misdetected images and add label data; The variational autoencoder algorithm is used to modify the image and retrain the model.
[0012] According to a shoulder arthroscopy surgery assistance method based on deep learning provided by the present invention, after real-time surgical data is input into the shoulder arthroscopy surgery assistance model and the surgical assistance result is output, the method further includes: The lesion area and surgical planning path are displayed in real time, and warning information of dangerous areas is output.
[0013] In a second aspect, the present invention further provides a shoulder arthroscopy surgery auxiliary system based on deep learning, comprising: A preprocessing module, for collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; An extraction module, used to extract features of the annotated image based on a DconnNet model to obtain image features; A determination module, used to learn the image features through a RetinaNet model to obtain a lesion area to be determined; A training module, used to train the data consisting of the lesion area to be determined and the surgical instrument labels using a deep learning model to establish a shoulder arthroscopy surgery auxiliary model; The processing module is used to input real-time surgical data into the shoulder arthroscopy surgery assistance model and output the surgery assistance results.
[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the deep learning-based shoulder arthroscopic surgery assisted method as described above is implemented.
[0015] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the deep learning-based shoulder arthroscopic surgery-assisted methods described above.
[0016] The deep learning-based shoulder arthroscopic surgery assistance method and system provided by the present invention, by applying artificial intelligence technology to shoulder arthroscopic surgery assistance technology, can achieve the purpose of improving the medical technology level of junior physicians to a certain extent, shortening the learning cycle, improving the imbalance of medical resources in different hospitals, and promoting the development of new digital medical surgical instruments. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a flowchart of a shoulder arthroscopy surgery auxiliary method based on deep learning provided by the present invention; Figure 2 It is a principle framework diagram of the DconnNet model provided by the present invention; Figure 3 It is a principle framework diagram of the RetinaNet model provided by the present invention; Figure 4 It is a structural schematic diagram of a shoulder arthroscopy surgery auxiliary system based on deep learning provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] In view of the shortcomings of the prior art, the present invention proposes a shoulder arthroscopic surgery method based on deep learning. Figure 1 is a flowchart of a shoulder arthroscopy surgery auxiliary method based on deep learning provided by an embodiment of the present invention, such as Figure 1 As shown, including: Step 100: collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; Step 200: extracting features of the annotated image based on the DconnNet model to obtain image features; Step 300: learning the image features by the RetinaNet model to obtain the lesion area to be determined; Step 400: using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery auxiliary model; Step 500: input the real-time surgical data into the shoulder arthroscopy surgery assistance model, and output the surgery assistance results.
[0021] Specifically, the shoulder arthroscopy surgical auxiliary device disclosed in the embodiment of the present invention includes three components. The first component is a video acquisition component for collecting intraoperative shoulder joint surgery video data; the second component is an artificial intelligence analysis unit, which can learn the anatomy, lesions, surgical instruments, etc. during surgery and establish a model through artificial deep learning; the third component is an interactive terminal, which can display the lesion area in real time and plan the surgical path, and give early warning prompts for dangerous areas. Doctors can use voice commands to interact with the system and adjust the surgical path to reduce surgical complications.
[0022] The corresponding methods include: collecting a large number of expert-level shoulder arthroscopic surgery videos, framing the videos, segmenting and labeling the images after framing, labeling the arthroscopic anatomy, lesion areas, surgical instruments, etc.; training the data with lesion area labels, using the trained deep learning model to identify the lesion area in each image, and obtaining the lesion area to be determined based on the lesion area; training the images with anatomical and surgical instrument labels, using the deep learning model to detect key elements of the surgery in real time, and establishing a surgical system assistance and early warning model; after the model accuracy is improved to the ideal range, guide the surgery, store the data in real time into the device, and perform optimization iterations.
[0023] The present invention applies artificial intelligence technology to shoulder arthroscopic surgery auxiliary technology, which can improve the medical technology level of junior physicians to a certain extent, shorten the learning cycle, improve the imbalance of medical resources in different hospitals, and promote the development of new digital medical surgical instruments.
[0024] Based on the above embodiment, an expert-level shoulder arthroscopy surgery video is collected, the shoulder arthroscopy surgery video is framed to obtain a framed image, and the framed image is labeled to obtain a labeled image, including: Performing frame processing on the shoulder arthroscopy surgery video according to the time series, removing abnormal data, and forming the framed image; The framed images are annotated using an annotation tool to form the annotated images, wherein the labels include lesion location and name labels, arthroscopic anatomical region labels, surgical instrument labels, and danger warning region labels.
[0025] Specifically, an embodiment of the present invention obtains video data of expert-level shoulder arthroscopic surgery, frames the video data in time series, removes abnormal data therein, including data with full-image blur, large-scale phantoms, extreme lighting, and incomplete shooting, to form sequential frame image data.
[0026] The acquired image data is labeled, and the target objects on the image are calibrated using the labeling tool software. The labels include: lesions, labels of various anatomical regions under arthroscopy, labels of surgical instruments such as scalpels, electrocoagulation knives, electrocuting knives, bipolar, etc., labels of recommended surgical entry areas, and labels of dangerous warning areas.
[0027] Based on the above embodiment, the features of the annotated image are extracted based on the DconnNet model to obtain the image features, including: The annotated image is input into the DconnNet model, and the encoder of the DconnNet model performs deep feature extraction on the annotated image through a convolution layer and a pooling layer to generate a multi-scale output feature map; The decoder of the DconnNet model gradually reshapes the multi-scale output feature map into an image having the same size as the annotated image through deconvolution; The contrast loss function is used for training and optimization to obtain the image features.
[0028] Specifically, Figure 2 As shown, the embodiment of the present invention establishes a DconnNet neural network as an encoder. Specifically, an independent DconnNet network structure is first designed and used for feature extraction. The encoder generates multi-scale feature maps through convolutional layers, which are then input into the decoder module. The decoder gradually reshapes these feature maps of different scales into the same size as the input image. , and generates accurate segmentation results that match the original image. During the decoding process, the low-level features are fused with the high-level features through the jump connection mechanism to fully restore the image details and enhance the segmentation accuracy. To further optimize the segmentation effect, the model generates the encoder output of the image after different enhancement methods. and Select positive and negative examples and calculate the contrast loss function
[0029]
[0030] in, Represents the similarity measurement function between feature vectors. By minimizing this contrast loss function, the model can better distinguish the target area from the background area, thereby improving the segmentation accuracy.
[0031] The lesion location and name, arthroscopic anatomical region labels, surgical instrument labels, and danger warning region labels are added to the sequence frame image data. The target objects on the image are calibrated using annotation tool software.
[0032] Specifically, Figure 3 As shown, the embodiment of the present invention uses the RetinaNet weights pre-trained on a large data set for initialization, inputs the labeled lesion image into the model, and uses the RetinaNet model to build a lesion recognition model. The loss model uses the Focal Loss loss function. The Focal Loss function expression is:
[0033] in, and γ are manually set adjustment parameters to balance the impact of positive and negative samples. is the model's predicted probability for the correct category. For RetinaNet The weights of the layer neural network, For RetinaNet The input features of the layer neural network, is the number of layers of the neural network in the RetinaNet model. The model parameters are updated through forward propagation and back propagation. The trained deep learning model is used to identify the lesion area in each image, and the lesion area to be determined is obtained based on the lesion area.
[0034] It should be noted that after receiving the label file, the RetinaNet model uses its built-in FPN (feature pyramid network) structure to extract features at different scales and learn the features of the lesion area frame by frame. The FocalLoss loss function is calculated through forward propagation , and then optimize the model weights through back propagation to gradually improve the detection accuracy of the lesion area. The Focal Loss function reduces the influence of easy-to-classify samples during training, so that the model pays more attention to the lesion area that is difficult to detect, thereby further enhancing the detection performance. The data with lesion area labels are trained, and the backbone network is initialized using the RetinaNet weights pre-trained on large datasets such as ImageNet. The labeled lesion images are input into the RetinaNet model, and the loss function is calculated through forward propagation. Then, the model parameters are updated through back propagation. The trained deep learning model is used to identify the lesion area in each image, and the lesion area to be determined is obtained based on the lesion area.
[0035] Based on the above embodiment, a deep learning model is used to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery auxiliary model, including: The data with surgical instrument labels are trained, with lesion labels as surgical targets, surgical instrument labels as surgical methods, and surgical operations as surgical output results, and potential confounding factors are identified; A causal diagram was used to represent the causal relationship between lesions, surgical instruments and surgical procedures, and confounding factors were identified in the causal diagram. The shoulder arthroscopy surgery auxiliary model is obtained by training the lesion area to be determined with the surgical instrument label added.
[0036] Specifically, the embodiment of the present invention establishes a recommended surgical operation model by learning surgical instrument labels and lesion labels, and constructs a causal graph by learning surgical instrument labels and lesion labels as causal variables, and uses causal inference methods, such as multi-layer Bayesian networks or structural equation models (SEM), to clarify the causal relationship between surgical instruments and lesions. By collecting and annotating a large amount of surgical process data, a deep learning model is used to extract features of surgical instruments and lesions. Tools such as Do-Calculus are used to eliminate confounding bias and ensure the accuracy of causal relationships. Finally, based on the evaluation results of causal relationships, a recommended surgical operation model is designed to provide surgical operation suggestions based on causal logic and optimize surgical decisions and effects.
[0037] The data with surgical instrument labels are trained, with the lesion labels as the treatment target (T), the surgical instrument labels as the treatment means (M), and the surgical operation as the output result (Y), while identifying potential confounding factors (C). Use a causal graph to represent the causal relationship between lesions, surgical instruments, and surgical operations. At the same time, the confounding factors are clearly identified in the causal graph, and their potential impact on the causal relationship is considered. The impact of the confounding factors is controlled through statistical methods (such as regression analysis, propensity score matching, etc.). The data with surgical instrument labels are trained to establish a surgical operation model. In the establishment of this model, in addition to learning the relationship between surgical instruments and lesions, the treatment plans and operation procedures for complications such as bleeding are also learned.
[0038] According to the treatment of the lesion, it is divided into three key time points: before, during and after the lesion treatment. Key points are set according to the lesion treatment, and a key point detection model is established through RetinaNet. RetinaNet gradually learns the position and changes of key points through multi-scale feature extraction and prediction mechanism. The model uses forward propagation to calculate errors and optimizes weights through back propagation. After training, RetinaNet can detect and monitor the changes of key points in the lesion treatment process in real time.
[0039] Based on the above embodiment, after using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery auxiliary model, the method further includes: Get missed and misdetected images and add label data; The variational autoencoder algorithm is used to modify the image and retrain the model.
[0040] Specifically, the embodiment of the present invention adds label data again for missed detection and false detection parts, and repeatedly strengthens training to continuously improve the accuracy of the surgical operating system model.
[0041] In order to optimize the model, the embodiment of the present invention adopts the variational auto-encoder (VAE) algorithm to modify the misdiagnosed and missed images, and performs encoding and decoding operations of the latent space z on these images through VAE to generate representative corrected images that retain key features. , whose goal is to minimize the combined loss function of reconstruction error and KL divergence:
[0042] in, is the original image, To reconstruct the image, is the approximate posterior distribution, is the prior distribution, is the total loss function of VAE. Based on the generated corrected images, refined label data is added again to construct a new training set Subsequently, the reinforcement learning (RL) method is used to repeatedly strengthen the model training, by maximizing the long-term cumulative reward To optimize model performance and improve robustness against misdiagnosis and missed diagnosis.
[0043] Based on the above embodiment, after the real-time surgical data is input into the shoulder arthroscopy surgery assistance model and the surgery assistance result is output, the method further includes: The lesion area and surgical planning path are displayed in real time, and warning information of dangerous areas is output.
[0044] Specifically, the embodiment of the present invention can give early warning reminders based on the real-time surgical process. If no accidents occur during the operation and the system does not issue a danger warning, if an accident occurs during the operation, the system will give a red danger warning and give remedial measures suggestions according to the guidelines. If the scalpel approaches the dangerous area, the system will give a yellow danger warning to remind the surgical team to pay attention to potential operational risks. If an abnormal situation occurs during the operation, the system will immediately issue a red danger warning and generate corresponding remedial measures suggestions based on the preset medical guidelines to provide timely decision-making support for doctors. At the manual interaction terminal, the surgeon can ask questions, and the terminal will give answers to the corresponding questions by learning from previous expert surgery videos and guidelines.
[0045] After the model is established, it monitors the surgical situation in real time and gives surgical path suggestions to the surgeon. At the same time, the device has voice recognition and output functions, which can realize human-computer interaction between the doctor and the device, further improve the surgeon's operating skills and reduce surgical risks.
[0046] The shoulder arthroscopy surgery assistance system based on deep learning provided by the present invention is described below. The shoulder arthroscopy surgery assistance system based on deep learning described below and the shoulder arthroscopy surgery assistance method based on deep learning described above can be referenced to each other.
[0047] Figure 4 is a schematic diagram of the structure of a shoulder arthroscopy surgery auxiliary system based on deep learning provided by an embodiment of the present invention, such as Figure 4 As shown, it includes: a preprocessing module 41, an extraction module 42, a determination module 43, a training module 44 and a processing module 45, wherein: The preprocessing module 41 is used to collect expert-level shoulder arthroscopy surgery videos, perform frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and label the framed images to obtain labeled images; the extraction module 42 is used to extract the features of the labeled images based on the DconnNet model to obtain image features; the determination module 43 is used to learn the image features by the RetinaNet model to obtain the lesion area to be determined; the training module 44 is used to use a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery assistance model; the processing module 45 is used to input real-time surgical data into the shoulder arthroscopy surgery assistance model and output the surgery assistance results.
[0048] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a shoulder arthroscopy surgery assistance method based on deep learning, the method comprising: collecting an expert-level shoulder arthroscopy surgery video, performing frame processing on the shoulder arthroscopy surgery video to obtain a framed image, and labeling the framed image to obtain a labeled image; extracting features of the labeled image based on the DconnNet model to obtain image features; learning the image features by the RetinaNet model to obtain a lesion area to be determined; using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument label to establish a shoulder arthroscopy surgery assistance model; inputting real-time surgical data into the shoulder arthroscopy surgery assistance model, and outputting surgical assistance results.
[0049] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0050] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the deep learning-based shoulder arthroscopy surgery assistance method provided by the above-mentioned methods, the method comprising: collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; extracting features of the labeled images based on a DconnNet model to obtain image features; learning the image features by a RetinaNet model to obtain a lesion area to be determined; using a deep learning model to train data consisting of the lesion area to be determined and surgical instrument labels to establish a shoulder arthroscopy surgery assistance model; inputting real-time surgical data into the shoulder arthroscopy surgery assistance model to output surgical assistance results.
[0051] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A shoulder arthroscopy surgery assisted method based on deep learning, characterized in that: include: Collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; Extracting features of the labeled image based on the DconnNet model to obtain image features; The RetinaNet model is used to learn the image features to obtain the lesion area to be determined; Using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels, and establishing a shoulder arthroscopy surgery auxiliary model; The real-time surgical data is input into the shoulder arthroscopy surgery assistance model, and the surgery assistance results are output.
2. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: Collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images, including: Performing frame processing on the shoulder arthroscopy surgery video according to the time series, removing abnormal data, and forming the framed image; The framed images are annotated using an annotation tool to form the annotated images, wherein the labels include lesion location and name labels, arthroscopic anatomical region labels, surgical instrument labels, and danger warning region labels.
3. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: The features of the annotated image are extracted based on the DconnNet model to obtain the image features, including: The annotated image is input into the DconnNet model, and the encoder of the DconnNet model performs deep feature extraction on the annotated image through a convolution layer and a pooling layer to generate a multi-scale output feature map; The decoder of the DconnNet model gradually reshapes the multi-scale output feature map into an image having the same size as the annotated image through deconvolution; The contrast loss function is used for training and optimization to obtain the image features.
4. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: The RetinaNet model learns the image features to obtain the lesion area to be determined, including: Initializing with RetinaNet weights pre-trained on a large dataset, and inputting the image features into the RetinaNet model; The RetinaNet model uses a built-in feature pyramid network structure to extract features at different scales and learn the features of the lesion area frame by frame; The model accuracy is optimized by using the Focal Loss function to determine the lesion area to be determined.
5. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: The deep learning model is used to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery auxiliary model, including: The data with surgical instrument labels are trained, with lesion labels as surgical targets, surgical instrument labels as surgical methods, and surgical operations as surgical output results, and potential confounding factors are identified; A causal diagram was used to represent the causal relationship between lesions, surgical instruments and surgical procedures, and confounding factors were identified in the causal diagram. The shoulder arthroscopy surgery auxiliary model is obtained by training the lesion area to be determined with the surgical instrument label added.
6. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: After using a deep learning model to train the data consisting of the lesion area to be determined and the surgical instrument labels to establish a shoulder arthroscopy surgery auxiliary model, the method further includes: Get missed and misdetected images and add label data; The variational autoencoder algorithm is used to modify the image and retrain the model.
7. The deep learning-based shoulder arthroscopy surgery assisting method according to claim 1, characterized in that: After inputting the real-time surgical data into the shoulder arthroscopy surgery assistance model and outputting the surgery assistance result, the method further includes: The lesion area and surgical planning path are displayed in real time, and warning information of dangerous areas is output.
8. A shoulder arthroscopy surgery auxiliary system based on deep learning, characterized in that: include: A preprocessing module, for collecting expert-level shoulder arthroscopy surgery videos, performing frame processing on the shoulder arthroscopy surgery videos to obtain framed images, and labeling the framed images to obtain labeled images; An extraction module, used to extract features of the annotated image based on a DconnNet model to obtain image features; A determination module, used to learn the image features through a RetinaNet model to obtain a lesion area to be determined; A training module, used to train the data consisting of the lesion area to be determined and the surgical instrument labels using a deep learning model to establish a shoulder arthroscopy surgery auxiliary model; The processing module is used to input real-time surgical data into the shoulder arthroscopy surgery assistance model and output the surgery assistance results.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the deep learning-based shoulder arthroscopic surgery assisted method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep learning-based shoulder arthroscopic surgery assisted method as described in any one of claims 1 to 7 is implemented.