Digestive endoscopy image intelligent labeling system and method based on deep learning

Abstract

The invention relates to the technical field of medical images, in particular to a digestive endoscopy image intelligent labeling system based on deep learning and a method thereof. The deep learning algorithm module is in communication connection with the preprocessing module and is used for receiving the standardized digestive tract endoscope image data sent by the preprocessing module; based on the standardized digestive tract endoscopic image data, executing a deep learning algorithm to carry out image annotation; generating labeled digestive tract endoscopic image data and unlabeled digestive tract endoscopic image data; the label storage module is in communication connection with the deep learning algorithm module; the micro-service module is in communication connection with the deep learning algorithm module and the label storage module; the universal interface module is in communication connection with the label storage module and the micro-service module; carrying out standardized storage on the marked digestive tract endoscopic image data; and an API interface is provided for an external system to access labeled digestive tract endoscopic image data, so that the generalization ability and labeling accuracy of the model are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a deep learning-based intelligent annotation system and method for digestive endoscopy images. Background Art

[0002] With the continuous advancement of medical technology, digestive endoscopy has become an important means of diagnosing and treating digestive tract diseases. However, the analysis and annotation of digestive endoscopic images has long been a time-consuming and labor-intensive task. Traditional manual annotation methods are not only inefficient, but also easily affected by subjective factors, making it difficult to ensure the consistency and accuracy of the annotation results. In recent years, with the rapid development of artificial intelligence technology, especially the breakthrough progress of deep learning algorithms in the field of computer vision, new opportunities have been brought about for the intelligent annotation of digestive endoscopic images.

[0003] At present, some research institutions and medical institutions have begun to try to apply deep learning technology to the analysis and annotation of digestive endoscopy images. These methods usually use a single deep learning model, such as convolutional neural network (CNN) or regional convolutional neural network (R-CNN), to classify digestive endoscopy images or detect targets. Although these methods have achieved certain results in certain specific tasks, there are still some obvious limitations and problems.

[0004] First, existing methods can only handle lesions of a single scale or a single type, and have difficulty dealing with multi-scale and multi-type lesions commonly found in digestive endoscopy images. For example, some methods may perform well in detecting small polyps, but are powerless against large ulcers or tiny early cancers. Second, existing methods have poor generalization capabilities, and their performance often drops significantly when faced with images collected by different devices and hospitals. This is mainly due to the limitations of the training data and the deficiencies of the model itself.

[0005] In addition, most existing methods lack effective data management and annotation quality control mechanisms. They usually focus on the algorithm itself, while ignoring key links such as data preprocessing, storage, and version control. This leads to uneven data quality and unstable model training effects, which are difficult to meet the needs of actual clinical applications. At the same time, the lack of effective annotation quality control and human-computer collaboration mechanisms also makes it difficult to ensure the reliability and consistency of annotation results.

[0006] In addition, existing methods generally have problems of low computational efficiency and difficulty in scalability. With the increase in data volume and model complexity, traditional single-machine processing methods can no longer meet the needs of real-time processing. Although distributed processing can improve processing capabilities, how to effectively manage and coordinate distributed resources and ensure the consistency of processing results is still an urgent problem to be solved.

[0007] In view of the above problems, there is an urgent need for a system and method that can comprehensively, efficiently and accurately perform intelligent annotation of digestive endoscopy images. The system should be able to adapt to multi-scale and multi-type lesions, have good generalization capabilities, be able to effectively manage and control data quality, support efficient distributed processing, and achieve human-computer collaborative annotation quality control. Summary of the invention

[0008] The present invention is proposed to solve the above technical problems. The present invention provides a deep learning-based digestive endoscopy image intelligent annotation system and method thereof. The system adopts a modular design, including a preprocessing module, a deep learning algorithm module, an annotation storage module, a microservice module and a general interface module. This architectural design not only solves many problems in the prior art, but also brings a series of significant technical effects.

[0009] The present invention proposes a digestive endoscopy image intelligent annotation system based on deep learning, comprising:

[0010] Preprocessing module for:

[0011] Receiving collected digestive tract endoscopy image data;

[0012] Cleaning and standardizing the digestive tract endoscopy image data;

[0013] A deep learning algorithm module is connected to the preprocessing module for:

[0014] Receiving standardized digestive tract endoscopy image data sent by the preprocessing module;

[0015] Based on the standardized digestive tract endoscopy image data, executing a deep learning algorithm to perform image annotation;

[0016] Generate labeled digestive tract endoscopy image data and unlabeled digestive tract endoscopy image data;

[0017] The annotation saving module is connected to the deep learning algorithm module for:

[0018] Receiving the labeled digestive tract endoscopy image data sent by the deep learning algorithm module;

[0019] Storing the labeled digestive tract endoscopy image data;

[0020] A microservice module is connected to the deep learning algorithm module and the annotation storage module for:

[0021] Receiving unlabeled digestive tract endoscopy image data sent by the deep learning algorithm module;

[0022] Based on the unlabeled digestive tract endoscopy image data, performing parallel labeling processing;

[0023] Generate new annotated digestive tract endoscopy image data;

[0024] The universal interface module is connected to the annotation storage module and the microservice module for:

[0025] Receiving the annotated digestive tract endoscopy image data sent by the annotation storage module and the microservice module;

[0026] Standardizing and storing the labeled digestive tract endoscopy image data;

[0027] An API interface is provided for external systems to access the annotated gastrointestinal endoscopic image data.

[0028] Preferably, the preprocessing module comprises:

[0029] A data receiving unit, used for receiving the collected digestive tract endoscopy image data;

[0030] A data cleaning unit, connected to the data receiving unit, is used to:

[0031] Data that did not conform to the characteristics of gastrointestinal endoscopy images were eliminated;

[0032] Remove duplicate and outlier data;

[0033] A data standardization unit, connected to the data cleaning unit, is used to:

[0034] resizing the cleaned digestive tract endoscopy image data;

[0035] Perform image enhancement operations;

[0036] Generate standardized gastrointestinal endoscopy image data.

[0037] Preferably, the deep learning algorithm module includes:

[0038] An image processing unit, used for preprocessing standardized digestive tract endoscopy image data;

[0039] A labeling unit, connected to the image processing unit, is used to:

[0040] Annotate the preprocessed image data based on the improved mask rcnn network model;

[0041] The improved mask rcnn network model includes:

[0042] Fusion attention mechanism;

[0043] Use ResNeXt152 as feature extractor;

[0044] Add path aggregation network and channel attention mechanism;

[0045] Use FPN network to generate feature pyramid;

[0046] Add CSP2 layer;

[0047] A model optimization unit, connected to the labeling unit, is used to:

[0048] Optimizing parameters of the improved mask rcnn network model;

[0049] Perform model compression operations to improve model accuracy.

[0050] Preferably, the annotation saving module includes:

[0051] A labeling receiving unit, used for receiving labeled digestive tract endoscopy image data;

[0052] A data processing unit is connected to the annotation receiving unit and is used to:

[0053] Performing format conversion on the labeled digestive tract endoscopy image data;

[0054] Generate data structures suitable for storage;

[0055] A storage unit connected to the data processing unit, and configured to:

[0056] The processed and annotated digestive tract endoscopy image data is stored in the MongoDB database;

[0057] Use Numpy arrays for data storage and indexing.

[0058] Preferably, the microservice module includes:

[0059] Task allocation unit, used to:

[0060] Receiving unlabeled digestive tract endoscopy image data;

[0061] Allocating the unlabeled digestive tract endoscopy image data to a plurality of parallel processing units;

[0062] A parallel processing unit is connected to the task allocation unit and is used to:

[0063] Simultaneously process multiple unlabeled digestive tract endoscopy image data;

[0064] Perform image annotation operations;

[0065] A result aggregation unit is connected to the parallel processing unit and is used for:

[0066] Collect annotation results from multiple parallel processing units;

[0067] Integrate and generate new annotated gastrointestinal endoscopy image data.

[0068] Preferably, the universal interface module comprises:

[0069] Data normalization unit for:

[0070] Receiving annotated digestive tract endoscopy image data from the annotation preservation module and the microservice module;

[0071] Unifying the format and optimizing the structure of the labeled digestive tract endoscopy image data;

[0072] A storage management unit, connected to the data standardization unit, is used to:

[0073] storing the standardized and annotated digestive tract endoscopy image data in a designated storage location;

[0074] Manage versions and access permissions of data;

[0075] An API interface unit is connected to the storage management unit and is used to:

[0076] Provides a RESTful API interface to support external systems to query and obtain annotated gastrointestinal endoscopy image data;

[0077] Implement security authentication and authorization mechanism for data access.

[0078] As a preference, it also includes:

[0079] The annotation visualization module is in communication with the deep learning algorithm module and the annotation preservation module, and is used to:

[0080] Receiving annotated digestive tract endoscopy image data;

[0081] Generate a visual display of annotation results;

[0082] Supports dynamic adjustment of the color, transparency and size of the annotation box of the annotation results;

[0083] The annotation review module is in communication with the annotation visualization module and is used to:

[0084] Manually review the visual annotation results;

[0085] Generate audit feedback information;

[0086] The audit results are fed back to the deep learning algorithm module for model optimization.

[0087] As a preference, it also includes:

[0088] Database modules for:

[0089] Various data generated during the operation of the storage system, including original image data, annotation data, model parameters, etc.

[0090] Provide efficient data retrieval and update mechanism;

[0091] Hardware support modules for:

[0092] Provide computing resources and storage resources for each functional module of the system;

[0093] Realize load balancing and resource scheduling to ensure efficient operation of the system.

[0094] Preferably, the deep learning algorithm module further includes:

[0095] Multi-scale object box regression unit, used for:

[0096] Perform regression calculations on targets of different scales based on the FPN network;

[0097] For each scale, regress the category and position of the ROI object box in its respective area;

[0098] The target box fusion method is adopted, and the ROI pooling layer and 1×1 convolution layer in the mask rcnn network model are used to realize multi-scale target box regression.

[0099] The intelligent annotation method of digestive endoscopy images based on deep learning includes the following steps:

[0100] S1: receiving collected digestive tract endoscopy image data through a preprocessing module, and cleaning and standardizing the digestive tract endoscopy image data;

[0101] S2: The deep learning algorithm module receives the standardized digestive tract endoscopy image data sent by the preprocessing module, executes the deep learning algorithm to perform image annotation, and generates annotated digestive tract endoscopy image data and unannotated digestive tract endoscopy image data;

[0102] S3: The annotation storage module receives the annotated digestive tract endoscopy image data sent by the deep learning algorithm module, and stores the annotated digestive tract endoscopy image data;

[0103] S4: The microservice module receives the unlabeled digestive tract endoscopy image data sent by the deep learning algorithm module, performs parallel labeling processing, and generates new labeled digestive tract endoscopy image data;

[0104] S5: The general interface module receives the annotated digestive tract endoscopy image data sent by the annotation storage module and the microservice module, stores the annotated digestive tract endoscopy image data in a standardized manner, and provides an API interface for external system access;

[0105] The deep learning algorithm in step S2 adopts an improved mask rcnn network model, including a fusion attention mechanism, using ResNeXt152 as a feature extractor, adding a path aggregation network and a channel attention mechanism, using an FPN network to generate a feature pyramid, and adding a CSP2 layer;

[0106] The parallel annotation processing in step S4 is performed simultaneously by multiple parallel processing units to improve the annotation efficiency;

[0107] The method also includes a labeling visualization step and a labeling review step, which are used to generate a visual display of the labeling results and perform manual review, and the review results are used to optimize the deep learning algorithm model.

[0108] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0109] First, the system of the present invention cleans and standardizes the input digestive endoscopy images through the preprocessing module, which greatly improves the accuracy and efficiency of subsequent processing. This step effectively solves the problem of uneven data quality in the prior art and provides high-quality input data for the deep learning algorithm.

[0110] Secondly, the deep learning algorithm module of the present invention adopts an improved mask rcnn network model, integrating a number of innovative technologies, such as attention mechanism, ResNeXt152 feature extractor, path aggregation network, etc. These improvements enable the model to effectively handle multi-scale and multi-type lesions, significantly improving the generalization ability and annotation accuracy of the model. In particular, the introduction of the multi-scale target box regression unit solves the problem that it is difficult to simultaneously handle lesions of different sizes in the prior art.

[0111] Thirdly, the microservice module of the present invention adopts an advanced microservice architecture to achieve efficient parallel processing. This not only greatly improves the processing efficiency of the system, but also solves the problem of difficult expansion in the prior art. The system can dynamically adjust the number of processing units according to demand and easily cope with data processing requirements of different scales.

[0112] In addition, the annotation preservation module and the universal interface module of the present invention jointly construct a complete data management system. The annotation preservation module uses the MongoDB database for data storage, supporting complex annotation information storage and efficient data retrieval. The universal interface module provides a standardized data access interface, greatly improving the interoperability of the system. These designs effectively solve the problems of chaotic data management and difficulty in sharing in the prior art.

[0113] Finally, the present invention introduces a labeling visualization module and a labeling review module to establish a human-machine collaborative labeling quality control mechanism. This not only improves the reliability and consistency of the labeling results, but also provides a feedback channel for the continuous optimization of the model, forming a continuously evolving closed-loop system.

[0114] In summary, the system of the present invention comprehensively solves the problems existing in the prior art through the integration of modular design and advanced technology, and realizes efficient, accurate and reliable intelligent annotation of digestive endoscopy images. This system not only greatly improves the efficiency and accuracy of digestive endoscopy image analysis, but also provides strong support for the early diagnosis and precise treatment of digestive tract diseases, and has important clinical application value and broad market prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 It is a logic block diagram of the entire system of the present invention.

[0116] Figure 2 It is a logic block diagram of the preprocessing module of the present invention.

[0117] Figure 3 This is a logical block diagram of the deep learning algorithm module of the present invention.

[0118] Figure 4 It is a logic block diagram of the annotation preservation module of the present invention.

[0119] Figure 5 This is a logic block diagram of the microservice module of the present invention.

[0120] Figure 6 It is a logic block diagram of the universal interface module of the present invention.

[0121] Figure 7 It is a logic block diagram of the annotation visualization module and the annotation review module of the present invention. DETAILED DESCRIPTION

[0122] See also Figure 1-7 The present invention provides a digestive endoscopy image intelligent annotation system and method based on deep learning. The system aims to improve the annotation efficiency and accuracy of digestive endoscopy images and provide strong support for medical research and clinical diagnosis.

[0123] like Figure 1 As shown, the digestive endoscopy image intelligent annotation system based on deep learning of the present invention includes a preprocessing module 1, a deep learning algorithm module 2, an annotation storage module 3, a microservice module 4 and a general interface module 5. These modules are connected through the Internet to form a complete annotation system.

[0124] The preprocessing module 1 is used to receive the collected digestive tract endoscopy image data, and to clean and standardize the data.

[0125] The deep learning algorithm module 2 is connected to the preprocessing module 1 for receiving the preprocessed standardized digestive tract endoscopy image data. This module is the core of the system and is responsible for executing the deep learning algorithm for image annotation and generating annotated and unannotated digestive tract endoscopy image data. The present invention adopts an improved mask rcnn network model and integrates a number of innovative technologies to improve the accuracy and efficiency of annotation.

[0126] The annotation storage module 3 is connected to the deep learning algorithm module 2 for receiving and storing the annotated digestive tract endoscopy image data. In one embodiment of the present invention, the annotation storage module 3 uses a MongoDB database for data storage. As a non-relational database, MongoDB has the characteristics of high performance, high availability and easy scalability, and is very suitable for storing a large amount of image data and corresponding annotation information.

[0127] The microservice module 4 is connected to the deep learning algorithm module 2 and the annotation storage module 3. The module receives unlabeled digestive tract endoscopy image data, performs parallel annotation processing, and generates new labeled digestive tract endoscopy image data. The adoption of the microservice architecture enables the system to have better scalability and concurrent processing capabilities. For example, when processing a large amount of image data, the number of processing units can be dynamically increased, significantly improving the processing efficiency of the system.

[0128] The general interface module 5 is connected to the annotation storage module 3 and the microservice module 4. It is responsible for receiving the annotated digestive tract endoscopy image data, storing it in a standardized manner, and providing an API interface for external systems to access. The design of the API interface follows the RESTful principle, so that external systems can easily query and obtain annotated image data.

[0129] The system of the present invention realizes efficient and intelligent annotation of digestive endoscopy images through the collaborative work of the above modules. From the preprocessing of raw image data, to the application of deep learning algorithms, to the storage and access of annotation results, a complete workflow is formed. This system design not only improves the efficiency and accuracy of annotation, but also has good scalability and adaptability, and can meet the use of medical institutions of different sizes and needs.

[0130] In a preferred embodiment of the present invention, the preprocessing module 1 includes a data receiving unit 11, a data cleaning unit 12 and a data standardization unit 13. This modular design makes the preprocessing process more efficient and controllable.

[0131] The data receiving unit 11 is mainly responsible for receiving the collected digestive tract endoscopy image data. In practical applications, these data may come from endoscopy devices of different brands and models, so the data format and quality may be different. The data receiving unit 11 is compatible with multiple data formats, such as DICOM, JPEG, PNG, etc., to ensure that the system can process image data from different sources.

[0132] The main task of the data cleaning unit 12 is to remove data that does not conform to the characteristics of the digestive tract endoscopy image. This step is crucial to improving the efficiency and accuracy of subsequent processing. Specifically, the data cleaning unit 12 can perform the following operations:

[0133] 1. Remove duplicate data: Identify and delete duplicate images by comparing the hash values ​​or other features of the images.

[0134] 2. Eliminate low-quality images: For example, a clarity threshold can be set to eliminate blurry images. Usually, the clarity of an image can be calculated using the Laplacian operator. If the clarity is lower than the set threshold (e.g., 50, in the range of 0-100), the image quality is considered to be insufficient.

[0135] 3. Remove non-gastrointestinal endoscopic images: Use simple machine learning models (such as SVM or decision tree) to quickly identify non-gastrointestinal endoscopic images and remove them.

[0136] The data standardization unit 13 is responsible for resizing and image enhancement operations on the cleaned digestive tract endoscopy image data. Standardization processing can significantly improve the effect of subsequent deep learning algorithms. In one embodiment of the present invention, the data standardization unit 13 can perform the following operations:

[0137] 1. Resizing: Resize all images to a uniform size, such as 512x512 pixels. This helps maintain computational efficiency and ensures consistency of model input.

[0138] 2. Brightness and contrast adjustment: Use histogram equalization technology to increase the contrast of the image. This is very important for enhancing the visibility of the internal structures of the digestive tract.

[0139] 3. Noise removal: Apply techniques such as Gaussian filtering or median filtering to reduce the noise in the image. For digestive endoscopy images, a 5x5 Gaussian kernel with a σ value of 1.5 can be used, which can usually effectively remove noise while retaining details.

[0140] 4. Data augmentation: Increase the diversity of training data through operations such as rotation, flipping, scaling, etc. For example, images can be randomly rotated ±15 degrees, flipped horizontally and vertically, and randomly scaled ±10%.

[0141] Through these processes, the preprocessing module 1 can provide high-quality, standardized input data for the deep learning algorithm module 2, thereby improving the performance and reliability of the entire system.

[0142] The deep learning algorithm module 2 is the core component of the system of the present invention, which includes an image processing unit 21, an annotation unit 22, and a model optimization unit 23. These three units work together to achieve efficient and accurate digestive endoscopy image annotation.

[0143] After receiving the standardized digestive tract endoscopy image data from the preprocessing module 1, the image processing unit 21 will perform further processing. The main purpose of this unit is to provide optimal input for the deep learning model. In one embodiment of the present invention, the image processing unit 21 may perform the following operations:

[0144] 1. Image segmentation: Segmenting the image into multiple regions of interest (ROIs) helps the model better focus on important anatomical structures or potential lesion areas.

[0145] 2. Feature enhancement: Use specific image processing techniques to enhance certain features, such as edge sharpening or texture enhancement. This can be achieved by applying techniques such as the Sobel operator or Gabor filter.

[0146] 3. Color space conversion: Convert the image from RGB color space to a color space more suitable for medical image analysis, such as Lab or YCbCr.

[0147] The annotation unit 22 is the core of the deep learning algorithm module 2 and is responsible for performing the actual image annotation task. The present invention adopts an improved mask rcnn network model and integrates a number of innovative technologies. Mask rcnn is a powerful instance segmentation algorithm that can perform target detection and semantic segmentation at the same time. The present invention has made a number of improvements to the original mask rcnn to better adapt to the characteristics of digestive endoscopy images.

[0148] Specifically, the improved mask rcnn network model includes the following innovations:

[0149] Attention mechanism: Introducing spatial attention and channel attention mechanisms helps the model better focus on important image areas and features. Spatial attention can be expressed as follows:

[0150] 1. Attention mechanism: Introducing spatial attention and channel attention mechanisms helps the model better focus on important image areas and features. Spatial attention can be expressed as follows:

[0151] A s =σ(f 7×7 ([AvgPool(F);MaxPool(F)])),

[0152] Among them, F is the input feature map, f 7×7 represents a 7×7 convolution operation, and σ is the sigmoid activation function.

[0153] 2. ResNeXt152 as a feature extractor: Compared with the commonly used ResNet backbone network, ResNeXt152 has stronger feature extraction capabilities. Its basic building blocks can be expressed as:

[0154]

[0155] Where X is the input, T i is a series of transformation functions, where C is the cardinality.

[0156] 3. Path Aggregation Network (PANet): PANet improves feature fusion by enhancing the feature hierarchy through bottom-up and top-down paths. Its core idea can be expressed by the following formula:

[0157] P′ l =Conv(P l +Upsample(P′ l+1 )),

[0158] Among them, P l is the feature map of layer l, P′ l It is the enhanced feature map.

[0159] 4.FPN network: Feature pyramid network (FPN) can generate multi-scale feature representations, which is conducive to detecting objects of different sizes. The core operation of FPN can be expressed as:

[0160] 5.P l =Conv(F l +Upsample(P l+1 )),

[0161] Among them, F l is the feature map obtained from the backbone network, P l It is the feature map generated by FPN.

[0162] 6.CSP2 layer: The CSP (Cross Stage Partial Network) structure reduces the amount of computation while maintaining accuracy by dividing the feature map into two parts, one part passes through the dense block and the other part passes directly. Its basic structure can be expressed as:

[0163] Y=[T(X part1 ),X part2 ],

[0164] Among them, X part1 and X part2 are the two parts of the input feature map, and T is a series of transformation functions.

[0165] These improvements enable the model to better handle the characteristics of digestive endoscopy images, such as lesions of different scales and complex texture information.

[0166] The model optimization unit 23 is responsible for further optimizing the trained model to improve the performance and efficiency of the model. In one embodiment of the present invention, the model optimization unit 23 may perform the following operations:

[0167] 1. Parameter optimization: Use advanced optimization algorithms such as Adam or RAdam and adopt a learning rate scheduling strategy. For example, you can use cosine annealing learning rate scheduling:

[0168]

[0169] Among them, η t is the learning rate of the tth step, T is the total number of steps, η m in and η m ax are the minimum and maximum learning rates respectively.

[0170] 2. Model compression: Use techniques such as knowledge distillation, pruning, or quantization to reduce model size. For example, with quantization, 32-bit floating point numbers can be converted to 8-bit integers, greatly reducing model size and inference time.

[0171] 3. Model integration: Use model integration techniques, such as bagging or boosting, to further improve the generalization and robustness of the model.

[0172] Through these optimizations, the deep learning algorithm module 2 of the present invention can achieve fast and efficient digestive endoscopy image annotation while ensuring high accuracy.

[0173] The annotation storage module 3 is an important component responsible for data persistence in the system of the present invention. It is closely connected with the deep learning algorithm module 2, receives the annotated digestive tract endoscopy image data, and stores the data safely and efficiently. In a preferred embodiment of the present invention, the annotation storage module 3 includes an annotation receiving unit 31, a data processing unit 32 and a storage unit 33.

[0174] The annotation receiving unit 31 is mainly responsible for receiving the annotated digestive tract endoscopy image data from the deep learning algorithm module 2. This unit is designed with a buffer mechanism that can handle a large amount of sudden data input to ensure that the data will not be lost due to insufficient processing speed. Specifically, the annotation receiving unit 31 uses a Redis-based message queue system that can temporarily store a large amount of annotated data and process the data one by one according to the first-in-first-out (FIFO) principle.

[0175] The data processing unit 32 receives the data from the annotation receiving unit 31 and performs necessary processing and conversion on the data. In one embodiment of the present invention, the data processing unit 32 performs the following operations:

[0176] 1. Data validation: Check if the received data is complete and in the correct format. This can be achieved using a predefined JSON schema.

[0177] 2. Data conversion: Convert the labeled data into a format suitable for storage. For example, the pixel-level segmentation mask can be converted into a more compact polygonal representation:

[0178] Polygon={(x1,y1),(x2,y2),...,(x n ,y n )},

[0179] Among them, (x i ,y i ) are the vertex coordinates of the polygon.

[0180] 3. Metadata generation: Generate corresponding metadata for each annotated data, such as annotation time, annotator ID, model version used, etc. These metadata are very important for subsequent data management and analysis.

[0181] 4. Data compression: Losslessly compress large image data to save storage space. Advanced image compression algorithms such as JPEG2000 or WebP can be used.

[0182] The storage unit 33 is responsible for persistently storing the processed annotated data. The present invention uses a MongoDB database to store these data. MongoDB is a document-based database that is very suitable for storing data with complex structures and variable sizes, such as medical images and their annotated information. In an embodiment of the present invention, the design of the storage unit 33 takes the following aspects into consideration:

[0183] 1. Data model design: A suitable data model is designed for gastrointestinal endoscopic images and their annotation information.

[0184] 2. Index optimization: Create appropriate indexes for the database based on common query patterns. For example, you can create a composite index for the 'patient_id' and 'study_date' fields to speed up the query of patient history data.

[0185] 3. Sharding strategy: For large-scale data, use MongoDB's sharding function to distribute the data on multiple servers. Sharding can be performed based on 'patient_id' or 'study_date' to ensure that related data is stored on the same shard, improving query efficiency.

[0186] 4. Data backup: Implement regular data backup strategies to ensure data security. You can use MongoDB's replication set function to achieve real-time data backup and failover.

[0187] Through such a design, the annotation storage module 3 can efficiently and reliably store and manage a large amount of digestive tract endoscopy image annotation data, providing a solid data foundation for subsequent data analysis and model optimization. In a preferred embodiment of the present invention, the microservice module 4 is a key component, which adopts an advanced microservice architecture, greatly improving the scalability and concurrent processing capabilities of the system. The microservice module 4 includes a task allocation unit 41, a parallel processing unit 42 and a result summary unit 43. These three units work together to achieve efficient parallel annotation processing.

[0188] The task allocation unit 41 is responsible for receiving unlabeled digestive tract endoscopy image data and allocating the data to multiple parallel processing units. In one embodiment of the present invention, the task allocation unit 41 adopts a dynamic load balancing algorithm. The algorithm not only considers the current load status of each parallel processing unit, but also considers factors such as historical processing speed and image complexity. For example, the following formula can be used to calculate the weight of each processing unit:

[0189]

[0190] Among them, W i is the weight of the ith processing unit, L i is the current load, Ti is the historical average processing time, C is the estimated value of image complexity, and α and β are adjustment parameters. The task allocation unit 41 dynamically allocates tasks according to these weights to ensure that system resources are optimally utilized.

[0191] The parallel processing unit 42 is the core of the microservice module 4 and is responsible for processing multiple unlabeled gastrointestinal endoscopic image data simultaneously. Each parallel processing unit is an independent microservice instance and can be dynamically expanded according to demand. In a preferred embodiment of the present invention, the parallel processing unit 42 uses a lightweight deep learning model, such as MobileNetV3 or EfficientNet-B0, which greatly reduces the demand for computing resources while maintaining high accuracy, so that a single server can run multiple parallel processing units at the same time.

[0192] Preferably, the parallel processing unit 42 also uses a model quantization technology to convert the 32-bit floating point model into an 8-bit integer model. This not only reduces the model size, but also significantly improves the inference speed. For example, for the MobileNetV3 model, the quantized model size can be reduced by 75%, while the inference speed can be increased by 2-3 times. The specific quantization process can be expressed as:

[0193] Q(r)=round(r / scale+zero p oint),

[0194] Among them, r is the original floating point value, scale and zero p oint is the quantization parameter, and Q(r) is the integer value after quantization.

[0195] The result summary unit 43 is responsible for collecting the annotation results of multiple parallel processing units and integrating them to generate new annotated digestive tract endoscopy image data. In one embodiment of the present invention, the result summary unit 43 adopts the idea of ​​ensemble learning and comprehensively considers the output results of multiple parallel processing units. For example, for classification tasks, a weighted voting method can be used:

[0196]

[0197] Among them, C is the final classification result, N is the number of parallel processing units, and w j is the weight of the jth processing unit, c j is the classification result of the jth processing unit, and I is the indicator function.

[0198] Through this design, microservice module 4 can efficiently process a large amount of unlabeled gastrointestinal endoscopic image data, significantly improving the processing power and scalability of the system.

[0199] In the system of the present invention, the universal interface module 5 plays a vital role. It is not only responsible for the standardized storage of data, but also provides a flexible API interface, so that the external system can easily access the annotated digestive tract endoscopy image data. The universal interface module 5 includes a data standardization unit 51, a storage management unit 52 and an API interface unit 53.

[0200] The data standardization unit 51 is responsible for receiving the annotated gastrointestinal endoscopic image data from the annotation storage module 3 and the microservice module 4, and unifying the format and optimizing the structure of these data. In a preferred embodiment of the present invention, the data standardization unit 51 adopts the FHIR (Fast Healthcare Interoperability Resources) standard to organize and represent medical image data. FHIR is a standard widely used in medical information exchange. It provides a flexible data model and rich resource types, which is very suitable for representing complex medical images and their annotation information.

[0201] The storage management unit 52 is responsible for storing the standardized and labeled digestive tract endoscopy image data in a designated storage location and managing the data version and access rights. In one embodiment of the present invention, the storage management unit 52 uses a distributed file system HDFS (Hadoop Distributed File System) to store a large amount of image data. HDFS has the characteristics of high fault tolerance and high throughput, and is very suitable for storing and processing large-scale medical image data.

[0202] Preferably, the storage management unit 52 also implements a data version control function. Each time the data is updated, a new version is created instead of directly overwriting the old data. This not only allows the data change history to be tracked, but also allows easy rollback to the previous version when needed. The implementation of version control can be based on the idea of ​​Git, where each version is a complete data snapshot, and storage space is saved by calculating differences.

[0203] The API interface unit 53 provides a RESTful API interface to support external systems to query and obtain annotated digestive tract endoscopy image data. In a preferred embodiment of the present invention, the API interface unit 53 uses GraphQL technology to design the API. Compared with the traditional REST API, GraphQL provides more flexible query capabilities, and the client can accurately specify the required data fields, avoiding the problem of over-acquisition or under-acquisition.

[0204] In addition, the API interface unit 53 also implements a strict security authentication and authorization mechanism. The present invention uses the OAuth2.0 protocol for identity authentication and uses role-based access control (RBAC) to manage data access rights. This ensures that sensitive medical data can only be accessed by authorized users or systems.

[0205] Through such a design, the universal interface module 5 not only realizes the standardized storage and efficient management of data, but also provides a flexible and secure data access method, greatly improving the interoperability and availability of the system.

[0206] In the system of the present invention, in order to further improve the quality and reliability of annotation, an annotation visualization module 6 and an annotation review module 7 are introduced. These two modules work together to ensure the accuracy and consistency of the annotation results.

[0207] The annotation visualization module 6 is in communication with the deep learning algorithm module 2 and the annotation storage module 3. Its main function is to present the annotated digestive tract endoscopy image data in an intuitive and easy-to-understand manner. In a preferred embodiment of the present invention, the annotation visualization module 6 uses advanced Web GL technology to achieve high-performance 2D and 3D visualization effects.

[0208] Specifically, the annotation visualization module 6 supports the following functions:

[0209] 1. Multi-layer visualization: Multiple layers of information such as the original image, segmentation mask, and bounding box can be displayed simultaneously. Users can flexibly view different annotation information by adjusting the transparency of each layer.

[0210] 2. Interactive operations: Supports operations such as zooming, panning, and rotating, allowing users to observe annotation results from different angles and scales.

[0211] 3. Color mapping: Use scientific color mapping schemes (such as Viridis or Magma) to represent different categories of annotations to improve visual recognition.

[0212] 4. Dynamic adjustment: Supports real-time adjustment of annotation color, transparency, and annotation box size. For example, the following formula can be used to dynamically adjust the color saturation:

[0213] S′=S·(1+α·(V-0.5)),

[0214] Where S and V are the original saturation and brightness values, respectively, S′ is the adjusted saturation, and α is a user-adjustable parameter.

[0215] The annotation review module 7 is in communication with the annotation visualization module 6. Its main function is to manually review the visualized annotation results, generate review feedback information, and feed back the review results to the deep learning algorithm module 2 for model optimization. In one embodiment of the present invention, the annotation review module 7 includes an annotation review unit 71, a result output unit 72, and an review result visualization unit 73.

[0216] The annotation review unit 71 provides a user-friendly interface, allowing medical experts to review and modify the annotation results. It supports the following functions:

[0217] 1. Batch review: You can load multiple annotation results at one time to improve review efficiency.

[0218] 2. Quick correction: Provide shortcut keys and mouse gestures to enable experts to quickly modify incorrect annotations.

[0219] 3. Uncertain marking: For cases that are difficult to determine, they can be marked as uncertain for further discussion.

[0220] The result output unit 72 is responsible for organizing the audit results into structured data and feeding it back to the annotation storage module 3. It will generate a detailed audit report, including information such as the modified content and the reason for the modification. This information is very important for subsequent model optimization and quality control.

[0221] The audit result visualization unit 73 is responsible for displaying the statistical data in the audit process in the form of charts. For example, it can generate a trend chart of the change in annotation accuracy over time, or a distribution pie chart of different types of errors, etc. These visualization results help the team understand the overall situation of annotation quality and discover and solve problems in a timely manner.

[0222] Preferably, the system of the present invention also implements a feedback loop mechanism. The audit results are not only used to correct the current annotation errors, but also fed back to the deep learning algorithm module 2 for continuous optimization of the model. For example, the following strategies can be used to update the model:

[0223] 1. Sample importance weighting: For the error cases found in the audit, increase their sampling weight in the next round of training.

[0224] 2. Adversarial sample generation: Based on the audit results, generate adversarial samples that are difficult to classify to improve the robustness of the model.

[0225] 3. Active learning: Prioritize cases marked as uncertain in the audit for model retraining.

[0226] By introducing the annotation visualization module 6 and the annotation review module 7, the system of the present invention not only improves the interpretability of the annotation results, but also establishes a continuously optimized closed-loop system to continuously improve the quality and accuracy of the annotation. This is particularly important for medical imaging data such as digestive endoscopy images, because high-quality annotations are directly related to the accuracy of subsequent medical research and clinical diagnosis. In a preferred embodiment of the present invention, the deep learning algorithm module 2 also includes a multi-scale target box regression unit 24. The introduction of this unit significantly improves the system's ability to detect and segment lesions of different sizes, which is crucial for the intelligent annotation of digestive endoscopy images.

[0227] The multi-scale target frame regression unit 24 performs regression calculations on targets of different scales based on the FPN (Feature Pyramid Network) network. The FPN network constructs a multi-scale feature pyramid through top-down paths and lateral connections, so that the model can detect targets at different resolutions. In the present invention, the implementation of the FPN network can be expressed as:

[0228] P l =Conv 1×1 (C l )+Upsample(P l+1 ),

[0229] Among them, P l is the feature map of layer l, C l is the feature map of the corresponding layer extracted from the backbone network, Conv 1×1 It represents a 1×1 convolution operation, and Upsample represents an upsampling operation.

[0230] For each scale, the multi-scale object frame regression unit 24 will regress the category and position of the ROI (Region of Interest) object frame of its respective region. This process can be expressed as:

[0231] t k =w k p+b k ,

[0232]

[0233] Among them, t k is the original prediction for the kth class, w k and b k are learnable weights and biases, p is the input feature vector, φ is the activation function (usually the Softmax function is used), is the final prediction result.

[0234] Preferably, the multi-scale target frame regression unit 24 adopts a target frame fusion method, and uses the ROI pooling layer and 1×1 convolution layer in the mask rcnn network model to realize multi-scale target frame regression. The ROI pooling layer can pool candidate regions of different sizes into a feature map of a fixed size, while the 1×1 convolution layer is used for dimensionality reduction and feature fusion. This process can be expressed as:

[0235] f=Conv 1×1 (ROIPool(F,R)),

[0236] Among them, F is the input feature map, R is the candidate region, ROIPool represents the ROl pooling operation, Conv 1×1 represents a 1×1 convolution operation, and f is the final feature.

[0237] Through this design, the multi-scale target frame regression unit 24 can effectively handle the scale change problem commonly seen in digestive endoscopy images. For example, the system can accurately detect and segment smaller polyps or larger ulcers. This greatly improves the applicability and accuracy of the system of the present invention.

[0238] Finally, the present invention also provides a method for intelligent annotation of digestive endoscopy images based on deep learning. This method is closely integrated with the above system, fully utilizing the advantages of each module of the system to achieve efficient and accurate annotation of digestive endoscopy images.

[0239] The method comprises the following steps:

[0240] S1: The collected digestive tract endoscopic image data is received through the preprocessing module 1, and the digestive tract endoscopic image data is cleaned and standardized. In this step, the system will first remove images of poor quality, such as blurred, too dark or too bright images. Then, the retained images are standardized, including resizing, contrast enhancement and other operations. This step provides high-quality input data for the subsequent deep learning algorithm.

[0241] S2: The deep learning algorithm module 2 receives the standardized gastrointestinal endoscopy image data sent by the preprocessing module 1, executes the deep learning algorithm for image annotation, and generates annotated gastrointestinal endoscopy image data and unannotated gastrointestinal endoscopy image data. In this step, the system uses an improved mask rcnn network model, which integrates the attention mechanism, uses ResNeXt152 as a feature extractor, adds a path aggregation network and a channel attention mechanism, uses an FPN network to generate a feature pyramid, and adds a CSP2 layer. These improvements enable the model to better handle the characteristics of gastrointestinal endoscopy images, such as lesions of different scales, complex texture information, etc.

[0242] S3: The annotation storage module 3 receives the annotated digestive tract endoscopy image data sent by the deep learning algorithm module 2, and stores the annotated digestive tract endoscopy image data. In this step, the system uses the MongoDB database to store the annotated data, and uses the characteristics of its document-based database to flexibly store complex annotation information.

[0243] S4: Microservice module 4 receives the unlabeled digestive tract endoscopy image data sent by deep learning algorithm module 2, performs parallel labeling processing, and generates new labeled digestive tract endoscopy image data. This step fully utilizes the advantages of the microservice architecture and greatly improves the processing efficiency by labeling simultaneously through multiple parallel processing units.

[0244] S5: The general interface module 5 receives the annotated digestive tract endoscopy image data sent by the annotation storage module 3 and the microservice module 4, stores the annotated digestive tract endoscopy image data in a standardized manner, and provides an API interface for external system access. This step ensures the standardization and accessibility of the annotated data, which is convenient for subsequent research and application.

[0245] Preferably, in step S2, the multi-scale target frame regression unit 24 also performs regression calculations on targets of different scales, and regresses the category and position of the ROI object frame of each scale in its respective area. This process effectively improves the system's ability to detect and segment lesions of different sizes.

[0246] In addition, the method also includes a labeling visualization step and a labeling review step. The labeling visualization step generates a visual labeling result display through the labeling visualization module 6, and supports dynamic adjustment of the color, transparency and labeling box size of the labeling result. The labeling review step is manually reviewed by the labeling review module 7, and the review feedback information is generated, and the review results are fed back to the deep learning algorithm module 2 for model optimization. The introduction of these two steps establishes a continuously optimized closed-loop system to continuously improve the quality and accuracy of labeling.

[0247] Through the above steps, the method of the present invention realizes efficient and intelligent annotation of digestive endoscopy images. The method makes full use of advanced technologies such as deep learning technology, microservice architecture, and data visualization, which not only improves the efficiency and accuracy of annotation, but also establishes a sustainable and optimized annotation system. This is of great significance for improving the diagnostic accuracy of digestive diseases and assisting doctors in making clinical decisions.

[0248] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep learning-based intelligent annotation system for digestive endoscopy images, characterized by: include: Preprocessing module for: Receiving collected digestive tract endoscopy image data; Cleaning and standardizing the digestive tract endoscopy image data; A deep learning algorithm module is connected to the preprocessing module for: Receiving standardized digestive tract endoscopy image data sent by the preprocessing module; Based on the standardized digestive tract endoscopy image data, executing a deep learning algorithm to perform image annotation; Generate labeled digestive tract endoscopy image data and unlabeled digestive tract endoscopy image data; The annotation saving module is connected to the deep learning algorithm module for: Receiving the labeled digestive tract endoscopy image data sent by the deep learning algorithm module; Storing the labeled digestive tract endoscopy image data; A microservice module is connected to the deep learning algorithm module and the annotation storage module for: Receiving unlabeled digestive tract endoscopy image data sent by the deep learning algorithm module; Based on the unlabeled digestive tract endoscopy image data, performing parallel labeling processing; Generate new annotated digestive tract endoscopy image data; The universal interface module is connected to the annotation storage module and the microservice module for: Receiving the annotated digestive tract endoscopy image data sent by the annotation storage module and the microservice module; Standardizing and storing the labeled digestive tract endoscopy image data; An API interface is provided for external systems to access the annotated gastrointestinal endoscopic image data.

2. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The preprocessing module comprises: A data receiving unit, used for receiving the collected digestive tract endoscopy image data; A data cleaning unit, connected to the data receiving unit, is used to: Data that did not conform to the characteristics of gastrointestinal endoscopy images were eliminated; Remove duplicate and outlier data; A data standardization unit, connected to the data cleaning unit, is used to: resizing the cleaned digestive tract endoscopy image data; Perform image enhancement operations; Generate standardized gastrointestinal endoscopy image data.

3. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The deep learning algorithm module includes: An image processing unit, used for preprocessing standardized digestive tract endoscopy image data; A labeling unit, connected to the image processing unit, is used to: Annotate the preprocessed image data based on the improved mask rcnn network model; The improved mask rcnn network model includes: Fusion attention mechanism; Use ResNeXt152 as feature extractor; Add path aggregation network and channel attention mechanism; Use FPN network to generate feature pyramid; Add CSP2 layer; A model optimization unit, connected to the labeling unit, is used to: Optimizing parameters of the improved mask rcnn network model; Perform model compression operations to improve model accuracy.

4. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The annotation saving module comprises: A labeling receiving unit, used for receiving labeled digestive tract endoscopy image data; A data processing unit is connected to the annotation receiving unit and is used to: Performing format conversion on the labeled digestive tract endoscopy image data; Generate data structures suitable for storage; A storage unit connected to the data processing unit, and configured to: The processed and annotated digestive tract endoscopy image data is stored in the MongoDB database; Use Numpy arrays for data storage and indexing.

5. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The microservice module includes: Task allocation unit, used to: Receiving unlabeled digestive tract endoscopy image data; Allocating the unlabeled digestive tract endoscopy image data to a plurality of parallel processing units; A parallel processing unit is connected to the task allocation unit and is used to: Simultaneously process multiple unlabeled digestive tract endoscopy image data; Perform image annotation operations; A result aggregation unit is connected to the parallel processing unit and is used to: Collect annotation results from multiple parallel processing units; Integrate and generate new annotated gastrointestinal endoscopy image data.

6. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The universal interface module comprises: Data normalization unit for: Receiving annotated digestive tract endoscopy image data from the annotation preservation module and the microservice module; Unifying the format and optimizing the structure of the labeled digestive tract endoscopy image data; A storage management unit, connected to the data standardization unit, is used to: storing the standardized and annotated digestive tract endoscopy image data in a designated storage location; Manage versions and access permissions of data; An API interface unit is connected to the storage management unit and is used to: Provides a RESTful API interface to support external systems to query and obtain annotated gastrointestinal endoscopy image data; Implement security authentication and authorization mechanism for data access.

7. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: Also includes: The annotation visualization module is in communication with the deep learning algorithm module and the annotation preservation module, and is used to: Receiving annotated digestive tract endoscopy image data; Generate a visual display of annotation results; Supports dynamic adjustment of the color, transparency and size of the annotation box of the annotation results; The annotation review module is in communication with the annotation visualization module and is used to: Manually review the visual annotation results; Generate audit feedback information; The audit results are fed back to the deep learning algorithm module for model optimization.

8. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: Also includes: Database modules for: All kinds of data generated during the operation of the storage system, including original image data, annotation data, and model parameters; Provide efficient data retrieval and update mechanism; Hardware support modules for: Provide computing resources and storage resources for each functional module of the system; Realize load balancing and resource scheduling to ensure efficient operation of the system.

9. The deep learning-based digestive endoscopy image intelligent annotation system according to claim 1 is characterized in that: The deep learning algorithm module also includes: Multi-scale object box regression unit, used for: Perform regression calculations on targets of different scales based on the FPN network; For each scale, regress the category and position of the ROI object box in its respective area; The target box fusion method is adopted, and the ROI pooling layer and 1×1 convolution layer in the mask rcnn network model are used to realize multi-scale target box regression.

10. A method for intelligent annotation of digestive endoscopy images based on deep learning, characterized in that: The following steps are involved: S1: receiving collected digestive tract endoscopy image data through a preprocessing module, and cleaning and standardizing the digestive tract endoscopy image data; S2: The deep learning algorithm module receives the standardized digestive tract endoscopy image data sent by the preprocessing module, executes the deep learning algorithm to perform image annotation, and generates annotated digestive tract endoscopy image data and unannotated digestive tract endoscopy image data; S3: The annotation storage module receives the annotated digestive tract endoscopy image data sent by the deep learning algorithm module, and stores the annotated digestive tract endoscopy image data; S4: The microservice module receives the unlabeled digestive tract endoscopy image data sent by the deep learning algorithm module, performs parallel labeling processing, and generates new labeled digestive tract endoscopy image data; S5: The general interface module receives the annotated digestive tract endoscopy image data sent by the annotation storage module and the microservice module, stores the annotated digestive tract endoscopy image data in a standardized manner, and provides an API interface for external system access; The deep learning algorithm in step S2 adopts an improved mask rcnn network model, including a fusion attention mechanism, using ResNeXt152 as a feature extractor, adding a path aggregation network and a channel attention mechanism, using an FPN network to generate a feature pyramid, and adding a CSP2 layer; The parallel annotation processing in step S4 is performed simultaneously by multiple parallel processing units to improve the annotation efficiency; The method also includes a labeling visualization step and a labeling review step, which are used to generate a visual display of the labeling results and perform manual review, and the review results are used to optimize the deep learning algorithm model.

Images