Knee arthroscopic surgery assisting method and system based on deep learning

Abstract

The invention provides a knee arthroscopic surgery auxiliary method and system based on deep learning, and the method comprises the following steps: obtaining a knee joint preoperative CT / MRI image containing a target focus, obtaining the information about the target focus in all to-be-analyzed CT / MRI images, and extracting the focus features of the target focus and the related focus; a large number of knee arthroscopic surgery videos are collected, and framing processing and label labeling are carried out on the videos; the method comprises the following steps: extracting an intra-operative knee arthroscopic image, performing three-dimensional reconstruction on an operative area to obtain an intra-operative real-time knee joint model, and identifying and highlighting a focus part under a real-time knee arthroscope according to preoperative image data of a patient; training the data added with the labels to obtain a deep learning algorithm model; after the precision of the model is improved to an ideal range, an operation is guided, and optimization iteration is carried out; according to the invention, fusion positioning is carried out on the image, accurate segmentation processing is carried out on the image, the model is constructed, and continuous optimization and upgrading of the model are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a knee arthroscopic surgery auxiliary method and system based on deep learning. Background Art

[0002] As a minimally invasive surgery, knee arthroscopy aims to perform surgery without damaging the normal structure of the knee joint. It has the characteristics of small trauma and fast recovery, and has become a basic surgical method in orthopedics. Due to its complexity, its common complications are diverse: intra-articular injury, nerve injury, vascular injury, compartment syndrome, instrument breakage, postoperative pain, thrombosis, synovitis, thromboembolism, infection, synovial fistula, complex regional pain syndrome, infrapatellar contracture syndrome, and the complication rate is as high as 2%. As the complexity of the operation increases, the incidence of complications also increases. As the operation technology increases, its incidence will also decrease. Therefore, in order to avoid complications of arthroscopic surgery, it is urgent to strengthen the skill training of arthroscopic surgery.

[0003] However, knee arthroscopy is complex and requires considerable time and patience to learn. As an emerging discipline, artificial intelligence is a technology that integrates the research and development of theories, methods, and applications for simulating, extending, and enhancing human intelligence. It can autonomously learn structured and unstructured data combinations to generate corresponding decision support. Therefore, relying on the artificial intelligence-assisted knee arthroscopy surgery system, the medical technology level of junior physicians can be improved to a certain extent, and the imbalance of medical resources in different hospitals can also be improved. Summary of the invention

[0004] The present invention provides a deep learning-based knee arthroscopic surgery assisting method and system to solve the defect that there is no better assisting means for knee arthroscopic surgery in the prior art.

[0005] In a first aspect, the present invention provides a method for assisting knee arthroscopy surgery based on deep learning, comprising: Acquire a knee arthroscopy surgery image and a knee arthroscopy surgery video, preprocess and fuse the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a preprocessed knee arthroscopy surgery fusion image; Constructing a fusion network for extracting lesion features from knee arthroscopy surgery based on a convolutional neural network model, inputting the preprocessed knee arthroscopy surgery fusion image into the fusion network for extracting lesion features from knee arthroscopy surgery, and obtaining knee arthroscopy image extraction features; A generative adversarial network is used to identify the knee arthroscopy image extraction features to obtain the lesion area to be determined; Performing three-dimensional reconstruction on the lesion area to be determined to generate a knee arthroscopy surgery model; Knee arthroscopy surgery is performed based on the knee arthroscopy surgery model to generate surgery warning information.

[0006] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, the knee arthroscopy surgery image is preprocessed, including: Annotate the knee joint CT / MRI images containing the lesions to obtain the site to be operated on; The surgical site is identified and learned through a convolutional neural network, and the unlearned knee joint CT / MRI image containing the lesion is identified by the learned convolutional neural network to obtain a preprocessed knee arthroscopy surgical image.

[0007] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, the knee arthroscopy surgery video is preprocessed, including: Performing frame processing on the knee arthroscopy surgery video according to a time sequence to form sequence frame image data; The sequence frame image data are labeled to obtain labeled sequence frame image data, wherein the labels include labels of anatomical regions under the knee joint, surgical instrument labels, recommended surgical entry region labels, and danger warning region labels.

[0008] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, labeling is performed on the sequence frame image data to obtain labeled sequence frame image data, including: A plurality of surgical videos at three stages, namely, before lesion resection, during lesion resection, and after lesion resection, wherein the number of surgical videos before lesion resection and after lesion resection is equal and less than the number of surgical videos during lesion resection, and a plurality of pictures are captured from each surgical video; Add label annotations to the captured images to obtain the initial annotated sequence frame image data; Dividing the initially annotated sequence frame image data into a training set and a test set according to a preset ratio; The training set is trained using a cross entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into a convolutional neural network, the trained convolutional neural network model is tested using the test set and the training parameters are adjusted according to the performance on the test set to obtain the annotated sequence frame image data.

[0009] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, the knee arthroscopy surgery image and the knee arthroscopy surgery video are fused to obtain a pre-processed knee arthroscopy surgery fusion image, including: Inputting the pre-processed knee arthroscopy surgery image and the annotated sequence frame image data into a multi-feature fusion module for feature extraction processing, wherein the multi-feature fusion module is composed of a graph convolutional network module and a neural convolutional network module; The graph convolutional network module obtains graph feature data of multiple frames extracted at multiple scales, and the neural convolutional network module obtains feature image data of multiple frames extracted at multiple scales; The graph feature data extracted from the continuous multiple frames at multiple scales and the feature image data extracted from the continuous multiple frames at multiple scales are fused, and are classified and sorted according to the same subject to obtain the preprocessed knee arthroscopy surgery fusion image.

[0010] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, a generative adversarial network is used to identify the knee arthroscopy image extraction features to obtain the lesion area to be determined, including: Build a generative adversarial network model, set the generator and discriminator, and determine the generator loss function and the discriminator loss function; The generator and the discriminator generate the knee arthroscopy image extraction features in a confrontation manner; The image fusion is performed with the patellar ridge as the base point, and the surgical area is reconstructed based on voxels to obtain the lesion area to be determined.

[0011] According to a deep learning-based knee arthroscopy surgery auxiliary method provided by the present invention, three-dimensional reconstruction of the lesion area to be determined is performed to generate a knee arthroscopy surgery model, including: constructing a three-dimensional model of the organ by using a three-dimensional reconstruction technique, and visualizing the three-dimensional model of the organ using a visualization toolkit; Collision detection during surgery is performed based on a bounding box algorithm to obtain the knee arthroscopy surgery model.

[0012] According to a deep learning-based knee arthroscopy surgery assisting method provided by the present invention, knee arthroscopy surgery is performed based on the knee arthroscopy surgery model to generate surgery warning information, including: The knee arthroscopy surgery model outputs a knee arthroscopy surgery video stream; The YOLOv7 model is used to perform real-time analysis on the knee arthroscopy surgery video stream, and the surgery warning information is output, wherein the surgery warning information includes prompt information for distinguishing different warning levels.

[0013] In a second aspect, the present invention further provides a knee arthroscopy surgery auxiliary system based on deep learning, comprising: A preprocessing and fusion module is used to obtain a knee arthroscopy surgery image and a knee arthroscopy surgery video, and preprocess and fuse the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a preprocessed knee arthroscopy surgery fusion image; An image segmentation module is used to construct a knee arthroscopy surgery lesion feature extraction fusion network based on a convolutional neural network model, input the preprocessed knee arthroscopy surgery fusion image into the knee arthroscopy surgery lesion feature extraction fusion network, and obtain knee arthroscopy image extraction features; A lesion recognition module, used to use a generative adversarial network to identify the knee arthroscopic image extraction features and obtain a lesion area to be determined; A three-dimensional reconstruction module, used for performing three-dimensional reconstruction of the lesion area to be determined and generating a knee arthroscopy surgery model; An operation warning module is used to perform knee arthroscopic surgery based on the knee arthroscopic surgery model and generate surgery warning information.

[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the deep learning-based knee arthroscopic surgery assisted method as described above is implemented.

[0015] The deep learning-based knee arthroscopy surgery auxiliary method and system provided by the present invention analyzes, fuses, and performs three-dimensional reconstruction of knee joint imaging data and knee arthroscopy surgery videos through deep learning technology, identifies microscopic and imaging lesion areas, surgical instruments, anatomical structures and other information, and provides real-time surgical operation suggestions and warnings, effectively improving surgical accuracy and operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a deep learning-based knee arthroscopy surgery auxiliary method provided by the present invention; Figure 2 It is a framework diagram of the image fusion principle provided by the present invention; Figure 3 It is a schematic diagram of the mutual confrontation value between the GAN model generator and the discriminator provided by the present invention; Figure 4 It is a schematic diagram of a feature extraction network provided by the present invention; Figure 5 This is a framework diagram of a real-time surgery monitoring and guidance module provided by the present invention; Figure 6 It is a structural schematic diagram of a knee arthroscopy surgery auxiliary system based on deep learning provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In view of the shortcomings of the existing technology, the present invention proposes a knee arthroscopy surgery auxiliary method based on deep learning, so as to achieve the purpose of promoting the development of new digital medical surgical instruments based on artificial intelligence technology.

[0020] Figure 1 is a flowchart of a deep learning-based knee arthroscopy surgery assisting method provided by an embodiment of the present invention, such as Figure 1 As shown, including: Step 100: Acquire a knee arthroscopy surgery image and a knee arthroscopy surgery video, preprocess and fuse the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a preprocessed knee arthroscopy surgery fusion image; Step 200: constructing a knee arthroscopy surgery lesion feature extraction fusion network based on a convolutional neural network model, inputting the preprocessed knee arthroscopy surgery fusion image into the knee arthroscopy surgery lesion feature extraction fusion network, and obtaining knee arthroscopy image extraction features; Step 300: using a generative adversarial network to identify the knee arthroscopy image extraction features and obtain a lesion area to be determined; Step 400: performing three-dimensional reconstruction on the lesion area to be determined to generate a knee arthroscopy surgery model; Step 500: Perform knee arthroscopy surgery based on the knee arthroscopy surgery model and generate surgery warning information.

[0021] The deep learning-based knee arthroscopy surgery auxiliary system proposed in the embodiment of the present invention includes an image segmentation module, a lesion recognition module, a three-dimensional reconstruction module, a surgical operation module and an early warning module.

[0022] In general, the process of the embodiment of the present invention is to first acquire knee arthroscopy surgery images, preprocess the images, then construct a fusion network based on knee arthroscopy surgery lesion feature extraction, identify the lesion area to be determined and highlight the area, and finally establish a surgical assistance system and early warning system based on the trained knee arthroscopy surgery lesion feature extraction network, and perform iterative optimization.

[0023] Based on the above embodiment, the knee arthroscopy surgery image is preprocessed, including: Annotate the knee joint CT / MRI images containing the lesions to obtain the site to be operated on; The surgical site is identified and learned through a convolutional neural network, and the unlearned knee joint CT / MRI image containing the lesion is identified by the learned convolutional neural network to obtain a preprocessed knee arthroscopy surgical image.

[0024] The knee arthroscopy surgery video is preprocessed, including: Performing frame processing on the knee arthroscopy surgery video according to a time sequence to form sequence frame image data; The sequence frame image data are labeled to obtain labeled sequence frame image data, wherein the labels include labels of anatomical regions under the knee joint, surgical instrument labels, recommended surgical entry region labels, and danger warning region labels.

[0025] The step of labeling the sequence frame image data to obtain the labeled sequence frame image data includes: A plurality of surgical videos at three stages, namely, before lesion resection, during lesion resection, and after lesion resection, wherein the number of surgical videos before lesion resection and after lesion resection is equal and less than the number of surgical videos during lesion resection, and a plurality of pictures are captured from each surgical video; Add label annotations to the captured images to obtain the initial annotated sequence frame image data; Dividing the initially annotated sequence frame image data into a training set and a test set according to a preset ratio; The training set is trained using a cross entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into a convolutional neural network, the trained convolutional neural network model is tested using the test set and the training parameters are adjusted according to the performance on the test set to obtain the annotated sequence frame image data.

[0026] The step of fusing the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a pre-processed knee arthroscopy surgery fusion image includes: Inputting the pre-processed knee arthroscopy surgery image and the annotated sequence frame image data into a multi-feature fusion module for feature extraction processing, wherein the multi-feature fusion module is composed of a graph convolutional network module and a neural convolutional network module; The graph convolutional network module obtains graph feature data of multiple frames extracted at multiple scales, and the neural convolutional network module obtains feature image data of multiple frames extracted at multiple scales; The graph feature data extracted from the continuous multiple frames at multiple scales and the feature image data extracted from the continuous multiple frames at multiple scales are fused, and are classified and sorted according to the same subject to obtain the preprocessed knee arthroscopy surgery fusion image.

[0027] Specifically, obtain a preoperative CT / MRI image of the knee joint containing the target lesion, obtain information about the target lesion in all CT / MRI images to be analyzed, obtain at least one CT or MRI image to be analyzed including the target site, and obtain the site and size information of the lesion.

[0028] At the same time, video data of expert-level knee arthroscopy surgery is obtained, and the video data is framed according to the time series to form sequence frame image data. The acquired image data is labeled, and the labels include: labels of various anatomical areas under the knee joint, labels of surgical instruments such as scalpels, electrocoagulation knives, electrocuting knives, bipolar, etc., labels of recommended knife entry areas, and labels of dangerous warning areas.

[0029] The labeling step includes: according to the three stages before lesion resection, during lesion resection and after lesion resection, 1,000 surgical videos are selected in the first and third stages, and 5,000 surgical videos are selected in the second stage, and 20 pictures are captured for each video; the selected pictures are respectively added with labels of various organ regions, labels of recommended incision areas, and labels of dangerous warning areas. The labels include: labels of various anatomical regions under the knee joint, labels of recommended incision areas, and labels of dangerous warning areas. The labeled pictures are divided into three data sets (a data set before lesion resection, a data set during lesion resection, and a data set after lesion resection), and each data set is divided into a ratio of 7:3, that is, 70% of the pictures are used for training sets and 30% of the pictures are used for test sets. The loss function (cross entropy , p(x) is the target distribution, q(x) is the predicted distribution), import the training set, initialize the network weights, set the training parameters (learning rate, batch size, number of training rounds, etc.), import the training set into the CNN model, validate the trained CNN model using the validation set and adjust the training parameters such as learning rate, batch size, etc. according to the performance on the validation set. To prevent overfitting, stop training when the performance on the validation set no longer improves.

[0030] The continuous image data with labels is input into the multi-feature fusion module for feature extraction processing to obtain continuous multi-frame extracted fusion feature image data, wherein the multi-feature fusion module is composed of a graph convolutional network module and a neural convolutional network module. After the graph convolutional network module performs feature extraction processing, the graph feature data of the continuous multi-frame multi-scale extraction under the graph convolutional network module is obtained, and the feature image data of the continuous multi-frame multi-scale extraction under the neural convolutional network module is obtained. Collect relevant data, which include input features and corresponding output labels. Data naming and classification, the knee arthroscopy surgery pictures of the same subject are stored in the same folder and the folder is named with the screening number. Other information of the patient such as age, gender, height, weight) and other information are stored in the database, and the sequence number corresponds to the name of the picture folder. Then perform data cleaning, name the pictures, unify the upper and lower case and space processing, eliminate samples with incomplete information (missing pictures, missing ratings, etc.), merge multiple batches of databases and pictures, and organize the pictures into a data set for knee arthroscopy recognition tasks. Training set and test set data split and verification fairness: In this embodiment, the training set and test set sample size are split according to 7:3. In order to solve the problem of training set and test set data split balance and test set verification fairness, our feature extraction and classifier training are all performed in the patient training set. Then, the classifier is tested using a test set of patients that are completely different from the training set. The original data is randomly divided into training set and test set.

[0031] like Figure 2 As shown in the figure, the method of achieving fusion between knee arthroscopy lesions and arthroscopy includes obtaining CT / MRI of the knee joint containing the lesion before surgery to obtain the target lesion attribute information corresponding to the target lesion. The knee arthroscopy is used to perform a comprehensive scan and identification of the suprapatellar bursa, patellar joint, medial femoral space, intercondylar fossa, and lateral knee space of the knee joint under knee arthroscopy. Then, the patellar ridge is used as a reference point, and fusion is performed based on the generative adversarial network (GAN). The lesion site is marked and highlighted through the previously obtained CT / MRI information.

[0032] Based on the above embodiment, a generative adversarial network is used to identify the knee arthroscopy image extraction features to obtain the lesion area to be determined, including: Build a generative adversarial network model, set the generator and discriminator, and determine the generator loss function and the discriminator loss function; The generator and the discriminator generate the knee arthroscopy image extraction features in a confrontation manner; The image fusion is performed with the patellar ridge as the base point, and the surgical area is reconstructed based on voxels to obtain the lesion area to be determined.

[0033] Specifically, Figure 4As shown in the figure, the features extracted from the intraoperative knee arthroscopy images are used to generate images based on the mutual confrontation between the GAN model generator and the discriminator, and the images are fused with the mid-patellar ridge as the base point. The surgical area is reconstructed based on voxels to obtain a real-time knee joint model during surgery. The lesion site under the real-time knee arthroscopy is identified and highlighted according to the patient's preoperative imaging data to guide the surgical operation.

[0034] It should be noted that the GAN model is constructed, the generator and discriminator are set, and the generator loss function and the discriminator loss function are defined. The generator fuses images with the same features and uses the generator loss function to measure the difference between the generated image and the target image. The discriminator is a binary classification model that accepts real images and images generated by the generator and tries to distinguish them. By alternately training the generator and the discriminator, they compete with each other to learn, improve the generator's ability to generate realistic images, and improve the classification accuracy of the discriminator. Figure 3 Schematic diagram of the mutual adversarial values ​​between the generator and discriminator of the GAN model.

[0035] Based on the above embodiment, three-dimensional reconstruction is performed on the lesion area to be determined to generate a knee arthroscopy surgery model, including: constructing a three-dimensional model of the organ by using a three-dimensional reconstruction technique, and visualizing the three-dimensional model of the organ using a visualization toolkit; Collision detection during surgery is performed based on a bounding box algorithm to obtain the knee arthroscopy surgery model.

[0036] Specifically, in the embodiment of the present invention, the voxel-based reconstruction used is a method for converting a medical image into a three-dimensional model, using data of all two-dimensional slices to form a three-dimensional model by stacking voxels.

[0037] According to the treatment of the lesion, it is divided into three key time points: before lesion treatment, during lesion treatment, and after lesion treatment. Key points are set according to the lesion treatment, and a key point detection model is established through a convolutional neural network (CNN). One or more convolutional layers are used at the end of the network to extract image features such as lines and textures. The convolution kernels of these layers have specific sizes and steps to match the fineness of the edges. The edge map generated by CNN is post-processed, such as non-maximum suppression (NMS) or edge tracking algorithms, to remove false positives and connect broken edges.

[0038] like Figure 5As shown in the figure, based on learning the labeled data, the diagnosis is established, the surgical plan is set, the specific operation steps of the surgery are known, and the surgical operation system model is established. At the same time, for the missed and false detection parts, the variational autoencoder algorithm (VAE) is used to add label data again, and the training is repeatedly strengthened to continuously improve the accuracy of the surgical operation system model. Based on CT or MRI scan data, a three-dimensional model of the organ is constructed through three-dimensional reconstruction technology such as the Marching Cubes algorithm, and visualized using tools such as VTK or OpenGL. In order to achieve realistic surgical simulation, accurate collision detection is required to ensure that the interaction between surgical instruments and virtual tissues conforms to physical laws. Collision detection can be performed using technologies such as the bounding box algorithm. The force feedback module in the virtual surgical system needs to ensure the stability and transparency of the system. Specific control algorithms, such as the maximum output force control algorithm (MOFC), can be designed to balance stability and transparency.

[0039] In addition, in order to achieve the optimization of the model, the variational autoencoder algorithm (VAE) was used to modify the misdiagnosed and missed images, and the reinforcement learning (RL) method was used to add labeled data again and repeatedly strengthen the training.

[0040] Based on the above embodiment, performing knee arthroscopy surgery based on the knee arthroscopy surgery model and generating surgery warning information include: The knee arthroscopy surgery model outputs a knee arthroscopy surgery video stream; The YOLOv7 model is used to perform real-time analysis on the knee arthroscopy surgery video stream, and the surgery warning information is output, wherein the surgery warning information includes prompt information for distinguishing different warning levels.

[0041] Specifically, based on high-quality sensors and imaging devices, real-time data collection during the operation, including physiological parameters, the position and movement of surgical instruments, etc., convolutional neural networks (CNN) are applied to detect key elements of the operation in real time, perform image segmentation, feature extraction and enhancement, and store the data in real time in the device to improve the accuracy of surgical planning and navigation, and optimize CNN parameters according to the parameters during the operation, reduce computing resource consumption, and improve processing speed. Feedback from doctors and surgical teams is collected after surgery to understand their actual experience and suggestions for improvement of the system. According to the real-time surgical process, medical instruments are used to monitor the patient's physiological parameters (such as blood pressure, pulse, etc.) and YOLOv7 is used to analyze the surgical video stream in real time to identify and track the position of the scalpel and the distance relative to the lesion and important tissues. When the scalpel approaches the dangerous area, the system triggers a yellow warning based on the detection results of YOLOv7 to remind the surgical team of potential operational risks. If an abnormal situation occurs during the operation, the system will immediately issue a red danger warning, and generate corresponding remedial measures based on the preset medical guidelines to provide timely decision support for doctors.

[0042] An important innovation of YOLOv7 in the field of object detection is its advanced label assignment mechanism, which is called coarse-to-fine guided label assignment. Compared with traditional label assignment methods, this new method improves the accuracy and efficiency of object detection by providing the model with rough label information first and then gradually refining this information through a gradual process. In the initial stage, the model will obtain a relatively broad label, which helps the model to quickly learn and capture the general characteristics of the target. Later, as the model's understanding of the data deepens, the label assignment will become more precise, guiding the model to focus on more subtle features, thereby achieving accurate positioning and classification of the target. In addition, YOLOv7 also introduces extension and compound scaling methods, which play a key role in model design. Extension methods allow the model to flexibly adjust the depth and width of the network according to different application requirements and computing resources. This flexibility enables YOLOv7 to adapt to various computing environments from edge devices to cloud servers. The compound scaling method further optimizes the parameter usage and computational efficiency of the model. By applying different scaling strategies to different parts of the model, it can reduce the amount of computation and model size while maintaining high accuracy. The combination of these technologies enables YOLOv7 to perform well in real-time target detection tasks, with significant improvements in both speed and accuracy. YOLOv7 plays an important role in scenarios such as medical image analysis. The training steps include: data preprocessing; installing YOLOv7 algorithm dependencies; preparing pre-trained weights for the YOLOv7 algorithm; training the YOLOv7 model using preprocessed data; improving the convergence speed and generalization ability of the model by adjusting the learning rate and applying the learning rate decay strategy. Monitoring indicators such as precision and recall during training, as well as evaluating the performance of the model on the validation set, helps to promptly discover potential overfitting or underfitting problems, using data augmentation techniques to expand the training data set and increase data diversity, thereby improving the robustness and generalization ability of the model; after determining the optimal number of training rounds, re-shuffle the order of the training data to avoid the influence of the data order on the model training results, and then use this optimal number of rounds to perform final training on the model to obtain the YOLOv7 model with the best performance and stability.

[0043] Different warnings are set according to the degree of risk. Yellow warning: indicates potential risk, reminding the surgical team to pay attention to the operation; red warning: indicates serious risk, the system will suspend the operation and provide remedial measures. The warning trigger conditions are: when the surgical instrument approaches important tissues such as nerves and blood vessels, the yellow warning is triggered; when the surgical instrument enters the dangerous area, the red warning is triggered. When the yellow warning occurs, the surgical team is prompted to pay attention to the operation and provide relevant operation suggestions, such as adjusting the position, direction, speed, etc. of the surgical instrument; when the red warning occurs, the prompt stops the operation and provides remedial measures, such as using hemostatic forceps to stop bleeding, using sutures to suture the wound, etc. According to the warning situation, a voice prompt is issued, and the time and content of the warning information are recorded for postoperative analysis. The setting of warning information can effectively avoid accidents during surgery and improve surgical safety; discover potential risks in advance and take measures to reduce surgical risks; reduce the pause time during surgery and improve surgical efficiency; help junior doctors better master surgical skills and improve surgical level.

[0044] The deep learning-based knee arthroscopy surgery assistance system provided by the present invention is described below. The deep learning-based knee arthroscopy surgery assistance system described below and the deep learning-based knee arthroscopy surgery assistance system method described above can be referenced to each other.

[0045] Figure 6 is a schematic diagram of the structure of a knee arthroscopy surgery auxiliary system based on deep learning provided by an embodiment of the present invention, such as Figure 6 As shown, it includes: a pre-processing fusion module 61, an image segmentation module 62, a lesion recognition module 63, a three-dimensional reconstruction module 64 and an operation warning module 65, wherein: The preprocessing fusion module 61 is used to obtain knee arthroscopic surgery images and knee arthroscopic surgery videos, preprocess and fuse the knee arthroscopic surgery images and the knee arthroscopic surgery videos, and obtain the preprocessed knee arthroscopic surgery fusion images; the image segmentation module 62 is used to construct a knee arthroscopic surgery lesion feature extraction fusion network based on a convolutional neural network model, input the preprocessed knee arthroscopic surgery fusion image into the knee arthroscopic surgery lesion feature extraction fusion network, and obtain knee arthroscopic image extraction features; the lesion recognition module 63 is used to use a generative adversarial network to identify the knee arthroscopic image extraction features and obtain the lesion area to be determined; the three-dimensional reconstruction module 64 is used to perform three-dimensional reconstruction of the lesion area to be determined and generate a knee arthroscopic surgery model; the operation warning module 65 is used to perform knee arthroscopic surgery based on the knee arthroscopic surgery model and generate surgery warning information.

[0046] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute a knee arthroscopy surgery auxiliary method based on deep learning, which includes: obtaining knee arthroscopy surgery images and knee arthroscopy surgery videos, preprocessing and fusing the knee arthroscopy surgery images and the knee arthroscopy surgery videos to obtain preprocessed knee arthroscopy surgery fusion images; constructing a knee arthroscopy surgery lesion feature extraction fusion network based on a convolutional neural network model, inputting the preprocessed knee arthroscopy surgery fusion image into the knee arthroscopy surgery lesion feature extraction fusion network to obtain knee arthroscopy image extraction features; using a generative adversarial network to identify the knee arthroscopy image extraction features to obtain a lesion area to be determined; performing three-dimensional reconstruction on the lesion area to be determined to generate a knee arthroscopy surgery model; performing knee arthroscopy surgery based on the knee arthroscopy surgery model to generate surgery warning information.

[0047] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0048] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0049] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based knee arthroscopy surgery auxiliary method, characterized in that: include: Acquire a knee arthroscopy surgery image and a knee arthroscopy surgery video, preprocess and fuse the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a preprocessed knee arthroscopy surgery fusion image; Constructing a fusion network for extracting lesion features from knee arthroscopy surgery based on a convolutional neural network model, inputting the preprocessed knee arthroscopy surgery fusion image into the fusion network for extracting lesion features from knee arthroscopy surgery, and obtaining knee arthroscopy image extraction features; A generative adversarial network is used to identify the knee arthroscopy image extraction features to obtain the lesion area to be determined; Performing three-dimensional reconstruction on the lesion area to be determined to generate a knee arthroscopy surgery model; Knee arthroscopy surgery is performed based on the knee arthroscopy surgery model to generate surgery warning information.

2. The deep learning-based knee arthroscopy surgery assisting method according to claim 1, characterized in that: Preprocessing the knee arthroscopy surgery image includes: Annotate the knee joint CT / MRI images containing the lesions to obtain the site to be operated on; The surgical site is identified and learned through a convolutional neural network, and the unlearned knee joint CT / MRI image containing the lesion is identified by the learned convolutional neural network to obtain a preprocessed knee arthroscopy surgical image.

3. The deep learning-based knee arthroscopy surgery assisting method according to claim 2, characterized in that: The knee arthroscopy surgery video is preprocessed, including: Performing frame processing on the knee arthroscopy surgery video according to a time sequence to form sequence frame image data; The sequence frame image data are labeled to obtain labeled sequence frame image data, wherein the labels include labels of anatomical regions under the knee joint, surgical instrument labels, recommended surgical entry region labels, and danger warning region labels.

4. The deep learning-based knee arthroscopy surgery assisting method according to claim 3, characterized in that: Labeling the sequence frame image data to obtain labeled sequence frame image data includes: A plurality of surgical videos at three stages, namely, before lesion resection, during lesion resection, and after lesion resection, wherein the number of surgical videos before lesion resection and after lesion resection is equal and less than the number of surgical videos during lesion resection, and a plurality of pictures are captured from each surgical video; Add label annotations to the captured images to obtain the initial annotated sequence frame image data; Dividing the initially annotated sequence frame image data into a training set and a test set according to a preset ratio; The training set is trained using a cross entropy loss function, the network weights are initialized, the training parameters are set, the training set is imported into a convolutional neural network, the trained convolutional neural network model is tested using the test set and the training parameters are adjusted according to the performance on the test set to obtain the annotated sequence frame image data.

5. The deep learning-based knee arthroscopy surgery assisting method according to claim 4, characterized in that: The knee arthroscopy surgery image and the knee arthroscopy surgery video are fused to obtain a pre-processed knee arthroscopy surgery fusion image, including: Inputting the pre-processed knee arthroscopy surgery image and the annotated sequence frame image data into a multi-feature fusion module for feature extraction processing, wherein the multi-feature fusion module is composed of a graph convolutional network module and a neural convolutional network module; The graph convolutional network module obtains graph feature data of multiple frames extracted at multiple scales, and the neural convolutional network module obtains feature image data of multiple frames extracted at multiple scales; The graph feature data extracted from the continuous multiple frames at multiple scales and the feature image data extracted from the continuous multiple frames at multiple scales are fused, and are classified and sorted according to the same subject to obtain the preprocessed knee arthroscopy surgery fusion image.

6. The deep learning-based knee arthroscopy surgery assisting method according to claim 1, characterized in that: Generative adversarial networks are used to identify the knee arthroscopy image extraction features to obtain the lesion area to be determined, including: Build a generative adversarial network model, set the generator and discriminator, and determine the generator loss function and the discriminator loss function; The generator and the discriminator generate the knee arthroscopy image extraction features in a confrontation manner; The image fusion is performed with the patellar ridge as the base point, and the surgical area is reconstructed based on voxels to obtain the lesion area to be determined.

7. The deep learning-based knee arthroscopy surgery assisting method according to claim 1, characterized in that: Performing three-dimensional reconstruction on the lesion area to be determined to generate a knee arthroscopy surgery model, including: constructing a three-dimensional model of the organ by using a three-dimensional reconstruction technique, and visualizing the three-dimensional model of the organ using a visualization toolkit; Collision detection during surgery is performed based on a bounding box algorithm to obtain the knee arthroscopy surgery model.

8. The deep learning-based knee arthroscopy surgery assisting method according to claim 1, characterized in that: Performing knee arthroscopy surgery based on the knee arthroscopy surgery model and generating surgery warning information include: The knee arthroscopy surgery model outputs a knee arthroscopy surgery video stream; The YOLOv7 model is used to perform real-time analysis on the knee arthroscopy surgery video stream, and the surgery warning information is output, wherein the surgery warning information includes prompt information for distinguishing different warning levels.

9. A knee arthroscopy surgery auxiliary system based on deep learning, characterized in that: include: A preprocessing and fusion module is used to obtain a knee arthroscopy surgery image and a knee arthroscopy surgery video, and preprocess and fuse the knee arthroscopy surgery image and the knee arthroscopy surgery video to obtain a preprocessed knee arthroscopy surgery fusion image; An image segmentation module is used to construct a knee arthroscopy surgery lesion feature extraction fusion network based on a convolutional neural network model, input the preprocessed knee arthroscopy surgery fusion image into the knee arthroscopy surgery lesion feature extraction fusion network, and obtain knee arthroscopy image extraction features; A lesion recognition module, used to use a generative adversarial network to identify the knee arthroscopic image extraction features and obtain a lesion area to be determined; A three-dimensional reconstruction module, used for performing three-dimensional reconstruction of the lesion area to be determined and generating a knee arthroscopy surgery model; An operation warning module is used to perform knee arthroscopic surgery based on the knee arthroscopic surgery model and generate surgery warning information.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the deep learning-based knee arthroscopic surgery assisted method as described in any one of claims 1 to 8 is implemented.

Images