Intelligent recognition system for gallbladder operation organs

Abstract

The invention discloses an intelligent recognition system for gallbladder operation organs, which relates to the technical field of medical instruments and comprises a neural network recognition end, an image processing end and a user interface end. The neural network recognition end analyzes the real-time image of the operation by using a segmentation model trained by an improved deep learning yov8 algorithm, detects and recognizes the gallbladder, other key parts and surgical instruments, and marks the gallbladder, other key parts and surgical instruments in a video in real time; the neural network recognition end comprises a model training module and a real-time recognition module, the model training module carries out training by constructing and optimizing a deep neural network model and utilizing a large number of high-quality labeled medical image data sets, and Mosaic and MixUp data enhancement technologies are adopted in the training process. According to the method, a series of denoising, contrast enhancement and artifact correction technologies are adopted in image preprocessing, so that the image quality is ensured, the recognition accuracy is improved, and the error rate is reduced.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to an intelligent organ recognition system for gallbladder surgery. Background Technology

[0002] In laparoscopic surgery, accurately locating organs and monitoring the position of surgical instruments in real time are crucial for surgical precision and efficiency. However, in traditional laparoscopic surgery without the intervention of AI technology, medical personnel need to face extremely urgent situations on-site and make surgical decisions quickly, which often requires a high level of professional knowledge and rich experience, posing a challenge for general surgeons, especially young doctors.

[0003] With the development of artificial intelligence technology, computer vision and deep learning have been widely applied in medical image analysis. Currently, many studies have begun to explore the use of neural network algorithms for automated analysis of medical images to improve the accuracy and speed of diagnosis. For example, convolutional neural networks (CNNs) have been widely used in the classification, segmentation, and detection of medical images, achieving significant results. Simultaneously, the integration of artificial intelligence algorithms with laparoscopic surgery is propelling minimally invasive surgery into an era of precision, with applications spanning the entire process from preoperative planning and intraoperative navigation to postoperative management.

[0004] However, the application of most existing artificial intelligence algorithms in laparoscopic surgery is not yet mature, especially in intraoperative assistance. On the one hand, relevant surgical data is scarce, and it is difficult to obtain real surgical data from external sources; on the other hand, many studies in this area directly train surgical data using open-source YOLOv8 algorithm models without corresponding adaptation and optimization. Due to the complexity of laparoscopic surgical data and the need for real-time detection, the actual application effect of the trained models is not particularly ideal, thus leaving room for improvement. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an intelligent organ recognition system for gallbladder surgery. Its advantages lie in the use of a series of noise reduction, contrast enhancement, and artifact correction techniques in image preprocessing to ensure image quality, improve recognition accuracy, and reduce the error rate.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An intelligent organ recognition system for gallbladder surgery includes a neural network recognition terminal, an image processing terminal, and a user interface terminal;

[0008] The neural network recognition end uses a segmentation model trained with an improved deep learning YOLOv8 algorithm to analyze real-time surgical images, detect and identify the gallbladder and other key parts and surgical instruments, and annotate them in the video in real time.

[0009] The neural network recognition module includes a model training module and a real-time recognition module. The model training module constructs and optimizes a deep neural network model, trains it using a large number of high-quality labeled medical image datasets, and employs Mosaic and MixUp data augmentation techniques during training. It replaces the ordinary Conv module with a full-dimensional dynamic convolution ODConv module, integrates a CBAM attention mechanism module to focus on key anatomical structures, and uses the PyTorch framework for model training.

[0010] The real-time recognition module is used to efficiently analyze real-time acquired images during surgery using a trained YOLO model, and to identify and label the gallbladder and other key parts, as well as surgical instruments, in the surgical video in real time. By introducing innovative real-time image segmentation and instance separation technology, it can accurately distinguish overlapping and similar organs and instruments. The recognition results are displayed on the image through intelligent color coding and dynamic bounding box selection, and combined with augmented reality (AR) technology, the real-time recognition information is presented in a three-dimensional overlay form.

[0011] The present invention is further configured such that the image processing end includes an image processing module and a classification algorithm module. The image processing module is responsible for performing comprehensive preprocessing on the real-time video stream to ensure that high-quality training data is provided for the algorithm model. First, based on advanced noise removal algorithms, including Gaussian filtering and mean filtering, various noise interferences in the image are accurately removed. At the same time, combined with adaptive filtering technology, the filtering parameters are dynamically adjusted to adapt to different noise characteristics and image conditions, thereby achieving adaptive noise reduction.

[0012] The present invention is further configured such that the classification algorithm module is used to filter the processed image using an organ classification model to remove a large amount of image data that is irrelevant to the surgical operation and obtain effective data that can be used for algorithm training.

[0013] The present invention is further configured such that, in terms of contrast enhancement, the classification algorithm module adopts adaptive histogram equalization (AHE) technology to dynamically adjust the grayscale distribution of the image; combined with advanced image correction algorithms, the system can effectively correct artifacts, distortions and other visual interferences in the image.

[0014] The present invention is further configured such that the user interface is used to display real-time surgical images and recognition results, and the interface supports touch operation; the recognition results of the neural network are displayed in the form of image overlay to help doctors understand the progress of the surgery in real time and make decisions.

[0015] The present invention is further configured such that the neural network algorithm needs to collect data, collecting image data of 36,000 cases of hepatobiliary laparoscopic surgery, including various types of normal and lesion data, and marking key anatomical structures (gallbladder, extrahepatic bile duct, cystic duct, cystic artery) and commonly used surgical instruments (grasping forceps and electrocautery).

[0016] The present invention is further configured such that the neural network algorithm requires data preprocessing. By implementing advanced image normalization techniques, the input image is comprehensively preprocessed, including high-precision spatial transformation (adaptive size adjustment) and advanced grayscale mapping. This process utilizes adaptive interpolation algorithms and multi-scale data recalibration techniques to dynamically adjust the image size and grayscale value distribution, so that the image data has optimal visual characteristics and information density when entering the subsequent analysis stage. In addition, an image feature enhancement mechanism is introduced, which adaptively adjusts the image contrast and brightness through intelligent algorithms.

[0017] The present invention is further configured such that the neural network algorithm requires image annotation and model training, and the labelme software is used to segment and annotate 36,000 cases of general surgical hepatobiliary surgery image data; the annotated image data is divided into training set, validation set and test set in a ratio of 8:1:1.

[0018] The present invention is further configured such that the neural network algorithm model selects the small-scale YOLOv8N benchmark model, and makes a series of improvements based on the YOLOv8 algorithm network, replacing the original convolution module with full-dimensional dynamic convolution, and adding a CBAM attention module to the backbone network, and using cross-validation method during training.

[0019] The present invention is further configured such that the neural network algorithm model adjusts the model's hyperparameters (learning rate, batch size, and weight decay) through random search or Bayesian optimization methods to obtain the optimal model performance; the model is trained on a high-performance GPU cluster using the PyTorch framework, and the training time is 72 hours; K-fold cross-validation is used to evaluate the model performance; in each validation, different folds are used as the validation set, and the rest are used as the training set, and the process is repeated K times, and the average performance is finally taken as the evaluation criterion for the model.

[0020] The beneficial effects of this invention are as follows:

[0021] 1. Real-time performance and efficiency: This invention utilizes deep learning technology and efficient image processing algorithms to perform real-time processing and analysis of images during surgery. Compared to the latency and processing bottlenecks of existing technologies, this system significantly improves the speed of information processing and response time during surgery.

[0022] 2. High Accuracy and Robustness: By using an advanced neural network model, this invention achieves higher accuracy in detecting and identifying the gallbladder and surgical instruments. The system has been trained and validated using a large amount of medical image data, enabling it to maintain high-precision recognition performance in complex laparoscopic surgical environments. This invention employs a series of denoising, contrast enhancement, and artifact correction techniques in image preprocessing to ensure image quality, improve recognition accuracy, and reduce the error rate.

[0023] 3. Scene adaptability: Existing technologies usually rely on complex equipment and environmental conditions, while this invention designs an intelligent organ recognition system for gallbladder surgery that is adapted to laparoscopic surgery, achieving plug-and-play functionality through simple hardware deployment.

[0024] 4. Simplified user interface and interactive operation: The present invention features a simple and intuitive user interface that displays comprehensive information, including real-time images and recognition results, which are displayed on the same screen as the surgical footage, helping doctors to fully control the situation during the operation.

[0025] In summary, this invention significantly outperforms existing technologies in terms of real-time performance, accuracy, intelligence, and ease of use, greatly improving the efficiency and effectiveness of identifying key parts such as the gallbladder and surgical instruments during hepatobiliary surgery. It has significant practical value and broad application prospects. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0027] Figure 2 This is a schematic diagram of the image processing module structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention;

[0028] Figure 3 This is a schematic diagram of the algorithm classification module structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention;

[0029] Figure 4 This is a schematic diagram of the neural network algorithm structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0030] Figure 5 This is a schematic diagram of the ODConv module structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0031] Figure 6 This is a schematic diagram of the ODConv module structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0032] Figure 7This is a schematic diagram of the training process structure of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0033] Figure 8 This is a schematic diagram illustrating the model loading and usage process of an intelligent organ recognition system for gallbladder surgery proposed in this invention.

[0034] Figure 9 This is a schematic diagram illustrating the cross-validation recall rate of an intelligent organ recognition system for gallbladder surgery proposed in this invention. Detailed Implementation

[0035] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0036] The embodiments of this patent are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.

[0037] Reference Figure 1-9 A smart organ recognition system for gallbladder surgery consists of three parts: an image processing terminal, a neural network recognition terminal, and a user interface terminal.

[0038] The image processing module comprises an image processing module and a classification algorithm module. The image processing module is responsible for comprehensive preprocessing of the real-time video stream to ensure high-quality training data for the algorithm model. First, based on advanced noise removal algorithms—Gaussian filtering and mean filtering—various noise interferences in the image are accurately removed, significantly improving image clarity and detail. Simultaneously, adaptive filtering technology is combined to dynamically adjust filtering parameters to adapt to different noise characteristics and image conditions, achieving adaptive noise reduction and further optimizing image quality. The classification algorithm module filters the processed image using an organ classification model, removing a large amount of image data irrelevant to the surgical procedure, obtaining effective data suitable for algorithm training.

[0039] In terms of contrast enhancement, the module employs Adaptive Histogram Equalization (AHE) technology, which dynamically adjusts the image's grayscale distribution to improve both global and local contrast, making details more prominent and information richer. Combined with advanced image correction algorithms, the system can effectively correct artifacts, distortions, and other visual interference in images, ensuring the authenticity and accuracy of image data.

[0040] The neural network recognition module utilizes a segmentation model trained with an improved deep learning YOLOv8 algorithm to analyze real-time surgical images, detect and identify the gallbladder and other key areas, as well as surgical instruments, and annotate them in real-time within the video. It mainly includes the following modules:

[0041] Model Training: This module constructs and optimizes a deep neural network model, training it using a large dataset of high-quality labeled medical images. During training, data augmentation techniques such as Mosaic and MixUp (including rotation, flipping, random cropping and scaling, and color gamut changes) are employed to improve the model's generalization ability in complex scenarios. The ordinary Conv module is replaced with a full-dimensional dynamic convolutional ODConv module, which enhances feature discrimination in complex backgrounds through dynamic weight allocation. A CBAM attention mechanism module is integrated to focus on key anatomical structures, further enhancing the model's ability to extract features from fine structures. The PyTorch framework is used for model training. Furthermore, a combination of transfer learning and self-supervised learning is employed to effectively shorten training time and improve the model's recognition accuracy.

[0042] Real-time recognition: During surgery, a trained YOLO model is used to efficiently analyze real-time acquired images, identifying and labeling the gallbladder, other key areas, and surgical instruments in the surgical video in real time. By introducing innovative real-time image segmentation and instance separation technologies, the system can accurately distinguish overlapping and similar organs and instruments. The recognition results are displayed on the image through intelligent color coding and dynamic bounding box selection, and combined with augmented reality (AR) technology, the real-time recognition information is presented in a three-dimensional overlay, facilitating precise operation and rapid positioning by the doctor.

[0043] The user interface displays real-time surgical images and recognition results. The interface supports touch operation for easy use by doctors during surgery. The neural network's recognition results are displayed as an overlay on the user interface, helping doctors understand the surgical progress and make decisions in real time.

[0044] Neural network algorithms:

[0045] 1. Data Acquisition:

[0046] Data source: Image data from 36,000 cases of hepatobiliary laparoscopic surgery were collected. These data include various types of normal and diseased cases, and key anatomical structures (gallbladder, extrahepatic bile duct, cystic duct, cystic artery) and commonly used surgical instruments (grasping forceps, electrocautery).

[0047] 2. Data preprocessing:

[0048] By implementing advanced image normalization techniques, comprehensive preprocessing of the input images is performed, including high-precision spatial transformation (adaptive resizing) and advanced grayscale mapping, to ensure the consistency and accuracy of the image data. This process utilizes adaptive interpolation algorithms and multi-scale data recalibration techniques to dynamically adjust the image size and grayscale value distribution, ensuring that the image data possesses optimal visual characteristics and information density before entering the subsequent analysis stage. Furthermore, an image feature enhancement mechanism is introduced, using intelligent algorithms to adaptively adjust image contrast and brightness, optimizing data quality and thereby improving the accuracy of subsequent processing and analysis.

[0049] 3. Image annotation and model training:

[0050] Labeling tool: Labelme software was used to segment and label images from 36,000 general surgery hepatobiliary procedures.

[0051] Dataset partitioning: The labeled image data is divided into training, validation and test sets in an 8:1:1 ratio to ensure that the model’s performance is representative on different datasets.

[0052] Neural network algorithm model: First, considering the real-time performance and accuracy of the algorithm, the small-scale YOLOv8N benchmark model was selected. A series of improvements were made to the YOLOv8 algorithm network, replacing the original convolution module with full-dimensional dynamic convolution, and adding a CBAM attention module to the backbone network. During training, cross-validation was used to improve the generalization ability of the model.

[0053] Hyperparameter optimization: Adjust the model's hyperparameters (such as learning rate, batch size, weight decay, etc.) through random search or Bayesian optimization methods to obtain the best model performance.

[0054] 4. Training platform:

[0055] Training the model on a high-performance GPU cluster using the PyTorch framework takes approximately 72 hours.

[0056] 5. Validate the model:

[0057] K-fold cross-validation is used to evaluate model performance. In each validation, a different fold is used as the validation set, and the rest are used as the training set. This process is repeated K times, and the average performance is taken as the evaluation metric for the model.

[0058] The performance of the neural network model of this invention on the test set is as follows:

[0059] Gallbladder and surgical instrument testing:

[0060] Gallbladder detection and identification:

[0061] Accuracy: 92.4%;

[0062] Recall rate: 81.6%;

[0063] Precision: 90.7%;

[0064] F1 Score: 90.3%;

[0065] IoU (Intersection over Union): 63.1%;

[0066] Dice (dice similarity coefficient): 71.3%;

[0067] Surgical instrument detection and identification:

[0068] ele_knife (electrosurgical knife):

[0069] Accuracy: 97.7%;

[0070] Recall rate: 98.3%;

[0071] Accuracy: 96.1%;

[0072] F1 score: 95.6%;

[0073] IoU: 67.8%;

[0074] Dice: 77.4%;

[0075] grasper:

[0076] Accuracy: 92.3%;

[0077] Recall rate: 87.5%;

[0078] Accuracy: 90.9%;

[0079] F1 score: 90.2%;

[0080] IoU: 53.6%;

[0081] Dice: 63.1%;

[0082] Detection and identification of other parts:

[0083] extrahepatic bile duct:

[0084] Accuracy: 65.6%;

[0085] Recall rate: 47.8%;

[0086] Accuracy: 64.9%;

[0087] F1 score: 64.2%;

[0088] IoU: 22.9%;

[0089] Dice: 27.7%;

[0090] cystic duct:

[0091] Accuracy: 67.6%;

[0092] Recall rate: 48.5%;

[0093] Accuracy: 67.9%;

[0094] F1 score: 66.1%;

[0095] IoU: 23.5%;

[0096] Dice: 28.5%;

[0097] cystic artery:

[0098] Accuracy: 57.2%;

[0099] Recall rate: 41.3%;

[0100] Accuracy: 57.5%;

[0101] F1 score: 55.8%;

[0102] IoU: 15.4%;

[0103] Dice: 19.5%.

[0104] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An intelligent organ recognition system for gallbladder surgery, characterized in that, This includes the neural network recognition end, the image processing end, and the user interface end; The neural network recognition end uses a segmentation model trained with an improved deep learning YOLOv8 algorithm to analyze real-time surgical images, detect and identify the gallbladder and other key parts and surgical instruments, and annotate them in the video in real time. The neural network recognition module includes a model training module and a real-time recognition module. The model training module constructs and optimizes a deep neural network model, trains it using a large number of high-quality labeled medical image datasets, and employs Mosaic and MixUp data augmentation techniques during training. It replaces the ordinary Conv module with a full-dimensional dynamic convolution ODConv module, integrates a CBAM attention mechanism module to focus on key anatomical structures, and uses the PyTorch framework for model training. The real-time recognition module is used to efficiently analyze real-time acquired images during surgery using a trained YOLO model, and to identify and label the gallbladder and other key parts, as well as surgical instruments, in the surgical video in real time. By introducing innovative real-time image segmentation and instance separation technology, it can accurately distinguish overlapping and similar organs and instruments. The recognition results are displayed on the image through intelligent color coding and dynamic bounding box selection, and combined with augmented reality (AR) technology, the real-time recognition information is presented in a three-dimensional overlay form.

2. The intelligent organ recognition system for gallbladder surgery according to claim 1, characterized in that, The image processing unit includes an image processing module and a classification algorithm module. The image processing module is responsible for comprehensive preprocessing of the real-time video stream to ensure high-quality training data for the algorithm model. First, based on advanced noise removal algorithms, including Gaussian filtering and mean filtering, various noise interferences in the image are accurately removed. At the same time, combined with adaptive filtering technology, the filtering parameters are dynamically adjusted to adapt to different noise characteristics and image conditions, thereby achieving adaptive noise reduction.

3. The intelligent organ recognition system for gallbladder surgery according to claim 2, characterized in that, The classification algorithm module is used to filter the processed images using an organ classification model, removing a large amount of image data that is irrelevant to the surgical procedure, and obtaining effective data that can be used for algorithm training.

4. The intelligent organ recognition system for gallbladder surgery according to claim 3, characterized in that, In terms of contrast enhancement, the classification algorithm module employs adaptive histogram equalization (AHE) technology to dynamically adjust the grayscale distribution of the image. Combined with advanced image correction algorithms, the system can effectively correct artifacts, distortions, and other visual interferences in the image.

5. The intelligent organ recognition system for gallbladder surgery according to claim 1, characterized in that, The user interface is used to display real-time surgical images and recognition results, and the interface supports touch operation; the recognition results of the neural network are displayed in the form of image overlay to help doctors understand the progress of the surgery in real time and make decisions.

6. The intelligent organ recognition system for gallbladder surgery according to claim 1, characterized in that, The neural network algorithm requires data acquisition, collecting image data from 36,000 cases of hepatobiliary laparoscopic surgery. This data includes various types of normal and lesion cases, and marks key anatomical structures (gallbladder, extrahepatic bile duct, cystic duct, cystic artery) and commonly used surgical instruments (grasping forceps and electrocautery).

7. The intelligent organ recognition system for gallbladder surgery according to claim 6, characterized in that, The neural network algorithm requires data preprocessing. By implementing advanced image normalization techniques, the input image undergoes comprehensive preprocessing, including high-precision spatial transformation (adaptive size adjustment) and advanced grayscale mapping. This process utilizes adaptive interpolation algorithms and multi-scale data recalibration techniques to dynamically adjust the image size and grayscale value distribution, ensuring that the image data possesses optimal visual characteristics and information density when entering the subsequent analysis stage. Furthermore, an image feature enhancement mechanism is introduced, using intelligent algorithms to adaptively adjust image contrast and brightness.

8. The intelligent organ recognition system for gallbladder surgery according to claim 7, characterized in that, The neural network algorithm requires image annotation and model training. The labelme software was used to segment and annotate 36,000 general surgery hepatobiliary surgery image data. The annotated image data was divided into training set, validation set and test set in a ratio of 8:1:

1.

9. The intelligent organ recognition system for gallbladder surgery according to claim 8, characterized in that, The neural network algorithm model selected the small-scale YOLOv8N benchmark model and made a series of improvements based on the YOLOv8 algorithm network. The original convolution module was replaced with full-dimensional dynamic convolution, and a CBAM attention module was added to the backbone network. Cross-validation was used during training.

10. The intelligent organ recognition system for gallbladder surgery according to claim 9, characterized in that, The neural network algorithm model adjusts the model's hyperparameters (learning rate, batch size, and weight decay) through random search or Bayesian optimization methods to obtain optimal model performance. The model is trained on a high-performance GPU cluster using the PyTorch framework, and the training time is 72 hours. K-fold cross-validation is used to evaluate the model performance. In each validation, different folds are used as the validation set, and the rest are used as the training set. This process is repeated K times, and the average performance is taken as the model's evaluation criterion.

Images