A Deep Learning-Based Method and System for Assisting Knee Arthroscopic Surgery

Abstract

This invention provides a deep learning-based method and system for assisting knee arthroscopy, comprising the following steps: obtaining preoperative CT / MRI images of the knee joint containing the target lesion; acquiring information about the target lesion from all CT / MRI images to be analyzed; extracting lesion features of the target lesion and related lesions respectively; acquiring a large number of knee arthroscopy videos, performing frame-by-frame processing and labeling on the videos; extracting knee arthroscopy images during surgery and performing three-dimensional reconstruction of the surgical area to obtain a real-time intraoperative knee joint model; identifying and highlighting the lesion site under real-time knee arthroscopy based on the patient's preoperative image data; training the labeled data to obtain a deep learning algorithm model; improving the model accuracy to an ideal range, guiding the surgery, and performing optimization iterations; this invention performs image fusion and localization, performs accurate image segmentation, constructs a model, and continuously optimizes and upgrades the model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for assisting knee arthroscopic surgery based on deep learning. Background Technology

[0002] Knee arthroscopy, as a minimally invasive surgery, aims to perform the procedure without damaging the normal structures within the knee joint. It is characterized by minimal trauma and rapid recovery, making it a fundamental surgical method in orthopedics. However, due to its complexity, it is prone to various complications, including intra-articular injury, nerve injury, vascular injury, compartment syndrome, instrument breakage, postoperative pain, thrombosis, synovitis, thromboembolism, infection, synovial fistula, complex regional pain syndrome, and infrapatellar contracture syndrome. The complication rate is as high as 2%, and it increases with the complexity of the surgery. Conversely, the incidence decreases with improved surgical techniques. Therefore, strengthening arthroscopic surgical skills training is crucial to avoiding complications during arthroscopic surgery.

[0003] However, knee arthroscopy is complex and requires considerable time and patience to learn. Artificial intelligence, as an emerging discipline, integrates the research, development, and application of theories, methods, and applications for simulating, extending, and enhancing human intelligence. It can autonomously learn from combinations of structured and unstructured data to generate corresponding decision support. Therefore, knee arthroscopy systems assisted by artificial intelligence can, to some extent, improve the medical skills of junior physicians and alleviate the imbalance of medical resources among different hospitals. Summary of the Invention

[0004] This invention provides a deep learning-based method and system for assisting knee arthroscopy, which addresses the shortcomings of existing technologies that lack effective auxiliary means for knee arthroscopy.

[0005] In a first aspect, the present invention provides a deep learning-based method for assisting knee arthroscopic surgery, comprising:

[0006] Acquire knee arthroscopic surgical images and videos, preprocess and fuse the knee arthroscopic surgical images and videos to obtain a preprocessed fused knee arthroscopic surgical image;

[0007] A knee arthroscopic surgery lesion feature extraction and fusion network is constructed based on a convolutional neural network model. The preprocessed knee arthroscopic surgery fusion image is input into the knee arthroscopic surgery lesion feature extraction and fusion network to obtain knee arthroscopic image features.

[0008] Generative adversarial networks are used to identify features extracted from the knee arthroscopy images to obtain the lesion region to be determined.

[0009] The lesion area to be determined is reconstructed in three dimensions to generate a knee arthroscopic surgical model;

[0010] Knee arthroscopic surgery is performed based on the aforementioned knee arthroscopic surgical model, and surgical warning information is generated.

[0011] According to a deep learning-based method for assisting knee arthroscopy surgery provided by the present invention, preprocessing of the knee arthroscopy surgical images includes:

[0012] Annotate CT / MRI images of the knee joint containing lesions to obtain the surgical site;

[0013] The surgical site is learned by using a convolutional neural network, and the learned convolutional neural network identifies unlearned CT / MRI images of the knee joint containing lesions to obtain preprocessed knee arthroscopic surgical images.

[0014] According to a deep learning-based method for assisting knee arthroscopy surgery provided by the present invention, preprocessing of the knee arthroscopy surgery video includes:

[0015] The knee arthroscopy surgery video is processed into frames according to the time sequence to form sequence frame image data;

[0016] The sequence frame image data is labeled to obtain labeled sequence frame image data, wherein the labels include labels for each anatomical region below the knee joint, surgical instrument labels, recommended incision area labels, and danger warning area labels.

[0017] According to a deep learning-based method for assisting knee arthroscopy provided by the present invention, the sequence frame image data is labeled to obtain labeled sequence frame image data, including:

[0018] Several surgical videos were taken at three stages: before lesion resection, during lesion resection, and after lesion excision. The number of surgical videos before lesion resection and after lesion excision was equal and less than the number of surgical videos during lesion resection. Several images were extracted from each surgical video.

[0019] Add labels to the captured images to obtain the initially labeled sequence of frame image data;

[0020] The initially labeled sequence frame image data is divided into a training set and a test set according to a preset ratio;

[0021] The training set is trained using the cross-entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into the convolutional neural network, the trained convolutional neural network model is tested using the test set, and the training parameters are adjusted according to the performance on the test set to obtain the labeled sequence frame image data.

[0022] According to a deep learning-based method for assisting knee arthroscopy surgery provided by the present invention, the method fuses the knee arthroscopy surgical image and the knee arthroscopy surgical video to obtain a preprocessed fused knee arthroscopy surgical image, including:

[0023] The preprocessed knee arthroscopic surgical images and the labeled sequence frame image data are input into a multi-feature fusion module for feature extraction processing. The multi-feature fusion module consists of a graph convolutional network module and a neural convolutional network module.

[0024] The graph convolutional network module obtains multi-scale extracted graph feature data for multiple consecutive frames, and the neural convolutional network module obtains multi-scale extracted feature image data for multiple consecutive frames.

[0025] The multi-scale extracted graph feature data and the multi-scale extracted feature image data of the consecutive multi-frames are fused together and categorized according to the same subject to obtain the preprocessed knee arthroscopic surgery fused image.

[0026] According to the present invention, a deep learning-based method for assisting knee arthroscopy surgery is provided, which uses generative adversarial networks to identify features extracted from the knee arthroscopy images to obtain the lesion region to be determined, including:

[0027] Construct a generative adversarial network model, set up a generator and a discriminator, and determine the generator loss function and the discriminator loss function;

[0028] The generator and the discriminator adversarially generate the extracted features from the knee arthroscopy image;

[0029] Image fusion was performed using the patellar crest as a reference point, and voxel-based reconstruction was performed on the surgical area to obtain the lesion area to be determined.

[0030] According to the present invention, a deep learning-based method for assisting knee arthroscopy surgery is provided, which performs three-dimensional reconstruction of the lesion region to be determined to generate a knee arthroscopy surgical model, including:

[0031] A three-dimensional model of an organ is constructed using three-dimensional reconstruction technology, and the three-dimensional model of the organ is visualized using a visualization toolkit.

[0032] The knee arthroscopy surgical model is obtained by performing collision detection during the surgical process based on the bounding box algorithm.

[0033] According to the present invention, a deep learning-based method for assisting knee arthroscopy is provided, which performs knee arthroscopy based on the knee arthroscopy model and generates surgical warning information, including:

[0034] The knee arthroscopy surgical model outputs a knee arthroscopy surgical video stream;

[0035] The YOLOv7 model is used to perform real-time analysis on the knee arthroscopy surgery video stream and output the surgical warning information, which includes different warning levels.

[0036] Secondly, the present invention also provides a deep learning-based knee arthroscopic surgery assistance system, comprising:

[0037] The preprocessing and fusion module is used to acquire knee arthroscopic surgical images and knee arthroscopic surgical videos, and to preprocess and fuse the knee arthroscopic surgical images and knee arthroscopic surgical videos to obtain preprocessed fused knee arthroscopic surgical images.

[0038] The image segmentation module is used to construct a knee arthroscopic surgery lesion feature extraction and fusion network based on a convolutional neural network model. The preprocessed knee arthroscopic surgery fusion image is input into the knee arthroscopic surgery lesion feature extraction and fusion network to obtain knee arthroscopic image extracted features.

[0039] The lesion identification module is used to identify the features extracted from the knee arthroscopy image using a generative adversarial network to obtain the lesion area to be determined.

[0040] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the lesion area to be determined and generate a knee arthroscopic surgery model.

[0041] The operation warning module is used to generate surgical warning information when performing knee arthroscopy based on the knee arthroscopy surgical model.

[0042] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep learning-based knee arthroscopic surgery assistance method as described above.

[0043] The present invention provides a deep learning-based method and system for assisting knee arthroscopy, which analyzes, fuses, and reconstructs three-dimensional images of the knee joint and videos of knee arthroscopy through deep learning technology, identifies information such as lesion areas under the endoscope and in the imaging, surgical instruments, and anatomical structures, and provides real-time surgical operation suggestions and warnings, effectively improving the accuracy and operability of the surgery. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the deep learning-based knee arthroscopic surgery assistance method provided by the present invention.

[0046] Figure 2 This is a schematic diagram of the image fusion principle framework provided by the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the adversarial values ​​between the GAN model generator and discriminator provided by this invention;

[0048] Figure 4 This is a schematic diagram of the feature extraction network provided by the present invention;

[0049] Figure 5 This is a framework diagram of the real-time surgical monitoring and guidance module provided by the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of the deep learning-based knee arthroscopic surgery assistance system provided by the present invention;

[0051] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] To address the shortcomings of existing technologies, this invention proposes a deep learning-based method for assisting knee arthroscopy, aiming to promote the development of new digital medical surgical instruments based on artificial intelligence technology.

[0054] Figure 1 This is a flowchart illustrating the deep learning-based knee arthroscopic surgery assistance method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes:

[0055] Step 100: Acquire knee arthroscopic surgical images and knee arthroscopic surgical videos, preprocess and fuse the knee arthroscopic surgical images and knee arthroscopic surgical videos to obtain preprocessed fused knee arthroscopic surgical images;

[0056] Step 200: Construct a knee arthroscopy lesion feature extraction and fusion network based on a convolutional neural network model, and input the preprocessed knee arthroscopy fusion image into the knee arthroscopy lesion feature extraction and fusion network to obtain knee arthroscopy image extraction features;

[0057] Step 300: Generative adversarial network is used to identify the features extracted from the knee arthroscopy image to obtain the lesion area to be determined;

[0058] Step 400: Perform three-dimensional reconstruction of the lesion area to be determined to generate a knee arthroscopy surgical model;

[0059] Step 500: Perform knee arthroscopy based on the knee arthroscopy model and generate surgical warning information.

[0060] The deep learning-based knee arthroscopy surgical assistance system proposed in this invention includes an image segmentation module, a lesion identification module, a three-dimensional reconstruction module, a surgical operation module, and an early warning module.

[0061] In general, the process of this invention is as follows: first, acquire knee arthroscopic surgical images, preprocess the images, then construct a fusion network based on knee arthroscopic surgical lesion feature extraction, identify the lesion area to be determined and highlight the area, and finally establish a surgical assistance system and an early warning system based on the trained knee arthroscopic surgical lesion feature extraction network, and perform iterative optimization.

[0062] Based on the above embodiments, the preprocessing of the knee arthroscopic surgical images includes:

[0063] Annotate CT / MRI images of the knee joint containing lesions to obtain the surgical site;

[0064] The surgical site is learned by using a convolutional neural network, and the learned convolutional neural network identifies unlearned CT / MRI images of the knee joint containing lesions to obtain preprocessed knee arthroscopic surgical images.

[0065] The preprocessing of the knee arthroscopic surgery video includes:

[0066] The knee arthroscopy surgery video is processed into frames according to the time sequence to form sequence frame image data;

[0067] The sequence frame image data is labeled to obtain labeled sequence frame image data, wherein the labels include labels for each anatomical region below the knee joint, surgical instrument labels, recommended incision area labels, and danger warning area labels.

[0068] The process involves labeling the sequence frame image data to obtain labeled sequence frame image data, including:

[0069] Several surgical videos were taken at three stages: before lesion resection, during lesion resection, and after lesion excision. The number of surgical videos before lesion resection and after lesion excision was equal and less than the number of surgical videos during lesion resection. Several images were extracted from each surgical video.

[0070] Add labels to the captured images to obtain the initially labeled sequence of frame image data;

[0071] The initially labeled sequence frame image data is divided into a training set and a test set according to a preset ratio;

[0072] The training set is trained using the cross-entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into the convolutional neural network, the trained convolutional neural network model is tested using the test set, and the training parameters are adjusted according to the performance on the test set to obtain the labeled sequence frame image data.

[0073] The process involves fusing the knee arthroscopic surgical images and the knee arthroscopic surgical video to obtain a preprocessed fused knee arthroscopic surgical image, including:

[0074] The preprocessed knee arthroscopic surgical images and the labeled sequence frame image data are input into a multi-feature fusion module for feature extraction processing. The multi-feature fusion module consists of a graph convolutional network module and a neural convolutional network module.

[0075] The graph convolutional network module obtains multi-scale extracted graph feature data for multiple consecutive frames, and the neural convolutional network module obtains multi-scale extracted feature image data for multiple consecutive frames.

[0076] The multi-scale extracted graph feature data and the multi-scale extracted feature image data of the consecutive multi-frames are fused together and categorized according to the same subject to obtain the preprocessed knee arthroscopic surgery fused image.

[0077] Specifically, the procedure involves obtaining preoperative CT / MRI images of the knee joint containing the target lesion, acquiring information about the target lesion from all CT / MRI images to be analyzed, acquiring at least one CT or MRI image of the target site, and acquiring information about the location and size of the lesion.

[0078] Simultaneously, video data of expert-level knee arthroscopic surgery is obtained. This video data is then processed into frame-by-frame sequence image data according to a time sequence. The acquired image data is labeled with tags including: labels for various anatomical regions below the knee joint, labels for surgical instruments such as scalpels, electrocautery knives, electrosurgical cutters, and bipolar instruments, labels for recommended incision areas, and labels for danger warning areas.

[0079] The labeling process includes: dividing the lesion into three stages: before, during, and after resection; selecting 1000 surgical videos for the first and third stages, and 5000 surgical videos for the second stage, with 20 images extracted from each video; adding labels for each organ region, recommended incision area, and danger warning area to the selected images. The labels include: labels for each anatomical region below the knee joint, recommended incision area, and danger warning area. The labeled images are divided into three datasets (pre-resection dataset, during resection dataset, and post-resection dataset), with each dataset split in a 7:3 ratio, i.e., 70% of the images are used for the training set and 30% for the test set. A loss function (cross-entropy) is designed. Let p(x) be the target distribution and q(x) be the prediction distribution. Import the training set, initialize the network weights, and set the training parameters (learning rate, batch size, number of training epochs, etc.). Import the training set into the CNN model. Validate the trained CNN model using a validation set and adjust the training parameters, such as the learning rate and batch size, based on the performance on the validation set. To prevent overfitting, training can be stopped when the performance on the validation set no longer improves.

[0080] Labeled continuous image data is input into a multi-feature fusion module for feature extraction, resulting in multi-frame extracted and fused feature image data. This multi-feature fusion module consists of a graph convolutional network module and a neural convolutional network module. After feature extraction by the graph convolutional network module, multi-scale extracted graph feature data and multi-scale extracted feature image data are obtained from the neural convolutional network module. Relevant data, including input features and corresponding output labels, are collected. Data is named and categorized: knee arthroscopic surgery images of the same subject are stored in the same folder and named with a selection number. Other patient information (such as age, gender, height, and weight) is stored in a database, with the folder name corresponding to the image number. Data cleaning is then performed, including image naming, case sensitivity and space correction, and removal of incomplete samples (missing images, missing ratings, etc.). Images from multiple batches of databases are merged to form a dataset for the knee arthroscopic surgery recognition task. Training and Test Set Data Splitting and Validation Fairness: In this embodiment, the training and test sets are split in a 7:3 ratio. To address the issues of data balance between the training and test sets and the fairness of test set validation, feature extraction and classifier training are performed on the patient training set. The classifier is then tested using a completely different patient set (i.e., the test set). The division of the original data into training and test sets is random.

[0081] like Figure 2 As shown, the method for fusing lesions under knee arthroscopy with those observed during arthroscopy includes: acquiring CT / MRI of the knee joint containing the lesion preoperatively to obtain target lesion attribute information; comprehensively scanning and identifying the knee joint's anatomical features, including the suprapatellar bursa, patellofemoral joint, medial femoral space, intercondylar fossa, and lateral knee space, using knee arthroscopy; then, using the patellar crest as a reference point, fusing the lesions based on a generative adversarial network (GAN); and marking and highlighting the lesion site using the previously acquired CT / MRI information.

[0082] Based on the above embodiments, a generative adversarial network is used to identify the features extracted from the knee arthroscopy image to obtain the lesion region to be determined, including:

[0083] Construct a generative adversarial network model, set up a generator and a discriminator, and determine the generator loss function and the discriminator loss function;

[0084] The generator and the discriminator adversarially generate the extracted features from the knee arthroscopy image;

[0085] Image fusion was performed using the patellar crest as a reference point, and voxel-based reconstruction was performed on the surgical area to obtain the lesion area to be determined.

[0086] Specifically, such as Figure 4 As shown, features extracted from intraoperative knee arthroscopy images are used to generate images based on the adversarial interaction between a GAN model generator and a discriminator. Image fusion is performed with the patellar crest as the base point, and voxel-based reconstruction of the surgical area is performed to obtain an intraoperative real-time knee joint model. Based on the patient's preoperative imaging data, the lesion site under real-time knee arthroscopy is identified and highlighted to guide the surgical operation.

[0087] It's important to note that building a GAN model involves setting up a generator and a discriminator, and defining loss functions for both. The generator fuses images with similar features, and its loss function measures the difference between the generated and target images. The discriminator is a binary classification model that accepts real images and images generated by the generator and attempts to distinguish between them. By training the generator and discriminator alternately, they learn in competition with each other, improving the generator's ability to generate realistic images and enhancing the discriminator's classification accuracy. Figure 3 This is a schematic diagram illustrating the adversarial values ​​between the generator and discriminator in a GAN model.

[0088] Based on the above embodiments, a three-dimensional reconstruction of the lesion area to be determined is performed to generate a knee arthroscopic surgical model, including:

[0089] A three-dimensional model of an organ is constructed using three-dimensional reconstruction technology, and the three-dimensional model of the organ is visualized using a visualization toolkit.

[0090] The knee arthroscopy surgical model is obtained by performing collision detection during the surgical process based on the bounding box algorithm.

[0091] Specifically, in this embodiment of the invention, the voxel-based reconstruction is a method for converting medical images into three-dimensional models, which utilizes data from all two-dimensional slices to form a three-dimensional model through the stacking of voxels.

[0092] Based on the lesion treatment, three key time points were identified: before lesion treatment, during lesion treatment, and after lesion treatment. Key points were set according to the lesion treatment, and a key point detection model was established using a convolutional neural network (CNN). At the end of the network, one or more convolutional layers were used to extract image features, such as lines and textures. The kernels of these layers had specific sizes and strides to match the fineness of the edges. Post-processing, such as non-maximum suppression (NMS) or edge tracking algorithms, was performed on the edge map generated by the CNN to remove false positives and connect broken edges.

[0093] like Figure 5As shown, based on learning from labeled data, a diagnosis is established, a surgical plan is set, the specific surgical procedures are understood, and a surgical operating system model is built. For missed and false detections, a variational autoencoder (VAE) algorithm is used to add more labeled data, and training is repeatedly strengthened to continuously improve the accuracy of the surgical operating system model. Based on CT or MRI scan data, a 3D model of the organ is constructed using 3D reconstruction techniques such as the Marching Cubes algorithm, and visualized using tools such as VTK or OpenGL. To achieve realistic surgical simulation, accurate collision detection is required to ensure that the interaction between surgical instruments and virtual tissues conforms to physical laws. Collision detection can be performed using techniques such as bounding box algorithms. The force feedback module in the virtual surgical system needs to ensure system stability and transparency. Specific control algorithms, such as the maximum output force control algorithm (MOFC), can be designed to balance stability and transparency.

[0094] In addition, to optimize the model, the variational autoencoder (VAE) algorithm was used to modify the images of misdiagnosed and missed diagnoses, and reinforcement learning (RL) was used to add labeled data again and repeatedly strengthen the training.

[0095] Based on the above embodiments, knee arthroscopy is performed using the knee arthroscopy model, and surgical warning information is generated, including:

[0096] The knee arthroscopy surgical model outputs a knee arthroscopy surgical video stream;

[0097] The YOLOv7 model is used to perform real-time analysis on the knee arthroscopy surgery video stream and output the surgical warning information, which includes different warning levels.

[0098] Specifically, the system uses high-quality sensors and imaging equipment to collect data in real time during surgery, including physiological parameters, the position and movement of surgical instruments, etc. A convolutional neural network (CNN) is applied to detect key surgical elements in real time, performing image segmentation, feature extraction, and enhancement. The data is stored in the device in real time to improve the accuracy of surgical planning and navigation. The CNN parameters are optimized based on parameters during the surgery to reduce computational resource consumption and improve processing speed. Post-operative feedback from doctors and the surgical team is collected to understand their actual user experience and suggestions for improvement. Based on the real-time surgical process, medical instruments monitor the patient's physiological parameters (such as blood pressure and pulse) and YOLOv7 analyzes the surgical video stream in real time to identify and track the position of the scalpel and its distance relative to the lesion and important tissues. When the scalpel approaches a danger zone, the system triggers a yellow alert based on the YOLOv7 detection results, alerting the surgical team to potential operational risks. If any abnormalities occur during the surgery, the system immediately issues a red danger alert and generates corresponding remedial measures based on preset medical guidelines, providing timely decision support for doctors.

[0099] A key innovation of YOLOv7 in object detection is its advanced label assignment mechanism, known as coarse-to-fine guided label assignment. Compared to traditional label assignment methods, this new approach uses a progressive process, first providing the model with general label information and then gradually refining this information, thereby improving the accuracy and efficiency of object detection. In the initial stage, the model receives a relatively broad label, which helps it quickly learn and capture the general features of the target. Subsequently, as the model's understanding of the data deepens, the label assignment becomes more precise, guiding the model to focus on more subtle features, thus achieving accurate target localization and classification. Furthermore, YOLOv7 introduces expansion and compound scaling methods, which play a crucial role in model design. The expansion method allows the model to flexibly adjust the network's depth and width according to different application requirements and computational resources. This flexibility enables YOLOv7 to adapt to various computing environments, from edge devices to cloud servers. The compound scaling method further optimizes the model's parameter usage and computational efficiency by applying different scaling strategies to different parts of the model, achieving a reduction in computational load and model size while maintaining high accuracy. The combination of these technologies enables YOLOv7 to excel in real-time object detection tasks, showing significant improvements in both speed and accuracy. YOLOv7 plays a crucial role in scenarios such as medical image analysis. The training steps include: data preprocessing; installing YOLOv7 algorithm dependencies; preparing pre-training weights for the YOLOv7 algorithm; training the YOLOv7 model using the preprocessed data; and improving the model's convergence speed and generalization ability by adjusting the learning rate and applying a learning rate decay strategy. Monitoring metrics such as precision and recall during training, as well as evaluating the model's performance on the validation set, helps to promptly identify potential overfitting or underfitting issues. Data augmentation techniques are used to expand the training dataset, increasing data diversity and thus improving the model's robustness and generalization ability. After determining the optimal number of training epochs, the order of the training data is shuffled to avoid the influence of data order on the model training results. This optimal number of epochs is then used for final training of the model to obtain a YOLOv7 model with optimal performance and stability.

[0100] Different warning systems are set based on the level of risk: a yellow warning indicates a potential risk, reminding the surgical team to pay attention to the procedure; a red warning indicates a serious risk, in which case the system will pause the surgery and provide suggestions for remedial measures. The warnings are triggered as follows: a yellow warning is triggered when surgical instruments approach important tissues such as nerves or blood vessels; a red warning is triggered when surgical instruments enter a danger zone. When a yellow warning occurs, the surgical team is advised to pay attention to the procedure and relevant operational suggestions are provided, such as adjusting the position, direction, and speed of surgical instruments. When a red warning occurs, the surgery is stopped, and suggestions for remedial measures are provided, such as using hemostatic forceps to stop bleeding or using sutures to close the wound. Voice prompts are issued based on the warning situation, and the time and content of the warning information are recorded for postoperative analysis. Setting up warning information can effectively prevent accidents during surgery, improve surgical safety; detect potential risks in advance and take measures to reduce surgical risks; reduce downtime during surgery, improve surgical efficiency; and help junior physicians better master surgical techniques and improve their surgical skills.

[0101] The following describes the deep learning-based knee arthroscopy surgery assistance system provided by the present invention. The deep learning-based knee arthroscopy surgery assistance system described below can be referred to in correspondence with the deep learning-based knee arthroscopy surgery assistance system method described above.

[0102] Figure 6 This is a schematic diagram of the structure of the deep learning-based knee arthroscopic surgery assistance system provided in an embodiment of the present invention, as shown below. Figure 6 As shown, it includes: a preprocessing fusion module 61, an image segmentation module 62, a lesion identification module 63, a 3D reconstruction module 64, and an operation early warning module 65, wherein:

[0103] The preprocessing fusion module 61 is used to acquire knee arthroscopic surgical images and videos, preprocess and fuse the knee arthroscopic surgical images and videos to obtain a preprocessed fused knee arthroscopic surgical image; the image segmentation module 62 is used to construct a knee arthroscopic surgical lesion feature extraction fusion network based on a convolutional neural network model, input the preprocessed fused knee arthroscopic surgical image to the knee arthroscopic surgical lesion feature extraction fusion network to obtain knee arthroscopic image extraction features; the lesion identification module 63 is used to identify the knee arthroscopic image extraction features using a generative adversarial network to obtain the lesion region to be determined; the three-dimensional reconstruction module 64 is used to perform three-dimensional reconstruction of the lesion region to be determined to generate a knee arthroscopic surgical model; the operation warning module 65 is used to perform knee arthroscopic surgery based on the knee arthroscopic surgical model and generate surgical warning information.

[0104] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a deep learning-based knee arthroscopy surgery assistance method. This method includes: acquiring knee arthroscopy images and videos; preprocessing and fusing the knee arthroscopy images and videos to obtain a preprocessed fused knee arthroscopy image; constructing a knee arthroscopy lesion feature extraction and fusion network based on a convolutional neural network model; inputting the preprocessed fused knee arthroscopy image into the knee arthroscopy lesion feature extraction and fusion network to obtain knee arthroscopy image extraction features; using a generative adversarial network to identify the knee arthroscopy image extraction features to obtain a lesion region to be determined; performing three-dimensional reconstruction of the lesion region to be determined to generate a knee arthroscopy model; performing knee arthroscopy based on the knee arthroscopy model and generating surgical warning information.

[0105] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based method for assisting knee arthroscopic surgery, characterized in that, include: Acquire knee arthroscopic surgical images and videos, preprocess and fuse the knee arthroscopic surgical images and videos to obtain a preprocessed fused knee arthroscopic surgical image; A knee arthroscopic surgery lesion feature extraction and fusion network is constructed based on a convolutional neural network model. The preprocessed knee arthroscopic surgery fusion image is input into the knee arthroscopic surgery lesion feature extraction and fusion network to obtain knee arthroscopic image features. Generative adversarial networks are used to identify features extracted from the knee arthroscopy images to obtain the lesion region to be determined. The lesion area to be determined is reconstructed in three dimensions to generate a knee arthroscopic surgical model; A three-dimensional model of an organ is constructed using three-dimensional reconstruction technology, and the three-dimensional model of the organ is visualized using a visualization toolkit. Collision detection during the operation is performed based on the bounding box algorithm to obtain the knee arthroscopy surgery model, and the knee arthroscopy surgery video stream is output from the knee arthroscopy surgery model; A real-time target detection model is used to analyze the knee arthroscopy surgery video stream in real time and output the surgical warning information, which includes different warning levels.

2. The deep learning-based knee arthroscopic surgery assistance method according to claim 1, characterized in that, Preprocessing of the knee arthroscopic surgical images includes: Annotate CT / MRI images of the knee joint containing lesions to obtain the surgical site; The surgical site is learned by using a convolutional neural network, and the learned convolutional neural network identifies unlearned CT / MRI images of the knee joint containing lesions to obtain preprocessed knee arthroscopic surgical images.

3. The deep learning-based knee arthroscopic surgery assistance method according to claim 2, characterized in that, Preprocessing of the knee arthroscopic surgery video includes: The knee arthroscopy surgery video is processed into frames according to the time sequence to form sequence frame image data; The sequence frame image data is labeled to obtain labeled sequence frame image data, wherein the labels include labels for each anatomical region below the knee joint, surgical instrument labels, recommended incision area labels, and danger warning area labels.

4. The deep learning-based knee arthroscopic surgery assistance method according to claim 3, characterized in that, The sequence frame image data is labeled to obtain labeled sequence frame image data, including: Several surgical videos were taken at three stages: before lesion resection, during lesion resection, and after lesion excision. The number of surgical videos before lesion resection and after lesion excision was equal and less than the number of surgical videos during lesion resection. Several images were extracted from each surgical video. Add labels to the captured images to obtain the initially labeled sequence of frame image data; The initially labeled sequence frame image data is divided into a training set and a test set according to a preset ratio; The training set is trained using the cross-entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into the convolutional neural network, the trained convolutional neural network model is tested using the test set, and the training parameters are adjusted according to the performance on the test set to obtain the labeled sequence frame image data.

5. The deep learning-based knee arthroscopic surgery assistance method according to claim 4, characterized in that, The arthroscopic surgical images and the arthroscopic surgical video are fused to obtain a preprocessed fused arthroscopic surgical image, including: The preprocessed knee arthroscopic surgical images and the labeled sequence frame image data are input into a multi-feature fusion module for feature extraction processing. The multi-feature fusion module consists of a graph convolutional network module and a neural convolutional network module. The graph convolutional network module obtains multi-scale extracted graph feature data for multiple consecutive frames, and the neural convolutional network module obtains multi-scale extracted feature image data for multiple consecutive frames. The multi-scale extracted graph feature data and the multi-scale extracted feature image data of the consecutive multi-frames are fused together and categorized according to the same subject to obtain the preprocessed knee arthroscopic surgery fused image.

6. The deep learning-based knee arthroscopic surgery assistance method according to claim 1, characterized in that, Generative adversarial networks are used to identify features extracted from the knee arthroscopy images to obtain the lesion region to be determined, including: Construct a generative adversarial network model, set up a generator and a discriminator, and determine the generator loss function and the discriminator loss function; The generator and the discriminator adversarially generate the extracted features from the knee arthroscopy image; Image fusion was performed using the patellar crest as a reference point, and voxel-based reconstruction was performed on the surgical area to obtain the lesion area to be determined.

7. A deep learning-based knee arthroscopic surgery assistance system, based on the deep learning-based knee arthroscopic surgery assistance method according to any one of claims 1 to 6, characterized in that, include: The preprocessing and fusion module is used to acquire knee arthroscopic surgical images and knee arthroscopic surgical videos, and to preprocess and fuse the knee arthroscopic surgical images and knee arthroscopic surgical videos to obtain preprocessed fused knee arthroscopic surgical images. The image segmentation module is used to construct a knee arthroscopic surgery lesion feature extraction and fusion network based on a convolutional neural network model. The preprocessed knee arthroscopic surgery fusion image is input into the knee arthroscopic surgery lesion feature extraction and fusion network to obtain knee arthroscopic image extracted features. The lesion identification module is used to identify the features extracted from the knee arthroscopy image using a generative adversarial network to obtain the lesion area to be determined. The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the lesion area to be determined and generate a knee arthroscopic surgery model. The operation warning module is used to generate surgical warning information when performing knee arthroscopy based on the knee arthroscopy surgical model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deep learning-based knee arthroscopic surgery assistance method as described in any one of claims 1 to 6.

Images