Gastrointestinal tract lesion image processing method and system based on convolutional neural network

Abstract

The invention provides a gastrointestinal tract lesion image processing method based on a convolutional neural network. The gastrointestinal tract lesion image processing method comprises the steps of S1, collecting gastrointestinal tract multi-type lesion image data; s2, preprocessing the multi-type lesion image data, and performing feature extraction; s3, dividing the preprocessed data into a training set and a test set, and performing data enhancement processing on the training set; s4, constructing a convolutional neural network model structure framework; S5, training the convolutional neural network model by using the enhanced training set, and evaluating by using a test set; and S6, performing lesion type prediction on a gastrointestinal tract lesion image to be processed by using the convolutional neural network model. According to the method, the image is denoised and enhanced, the convolutional neural network model based on the attention mechanism is constructed, local and global features are mined, the position of the lesion is accurately positioned, and the accuracy of lesion recognition is improved.

Description

Technical Field

[0001] This application relates to the technical field of medical image pattern recognition and analysis, and particularly to a gastrointestinal lesion image processing method based on a convolutional neural network. Background Art

[0002] Artificial intelligence (AI) is a broad and multidisciplinary field that encompasses concepts from computer science, engineering, philosophy, and linguistics, aiming to understand and design systems that exhibit or simulate human intelligence. In recent years, the application of AI in inflammatory bowel disease (IBD) has increased significantly. Various AI classifiers can be applied to analyze and integrate large datasets, ranging from clinical data in electronic health records, molecular data including gene expression and protein-based research, to a wide range of datasets composed of endoscopic and histological images. The application of AI has the potential to improve the accuracy and precision of predicting IBD risk, diagnosing, assessing disease severity, and predicting the outcomes of various IBD therapies. Currently, the application of AI methods in IBD is limited to the research environment and has not been adopted in real-life clinical practice. In addition, research on the application of AI in the context of IBD colorectal cancer surveillance or prediction is urgently needed.

[0003] Machine learning (ML) is a sub-discipline of AI in which computer algorithms apply statistical models to learn predictive associations from examples in a provided dataset. Deep learning algorithms, especially convolutional neural networks, have achieved good results in image classification problems. With the development of artificial intelligence and image processing technologies, it has become a new development direction to identify different types of images through machine learning methods.

[0004] In the identification of gastrointestinal diseases, existing technologies for human gastrointestinal endoscopy detection focus on the capsule endoscope itself, and the processing and pattern recognition research of lesion images lags behind, which has become a bottleneck restricting the capsule endoscope detection system. Moreover, image lesion recognition technologies focus on the pattern recognition of specific lesions. However, due to the diversity of lesions, even for the same type of lesion, its features are variable, which is likely to result in different subjective judgments. In addition, conventional digital image processing and pattern recognition algorithms are also difficult to handle the complex endoscopic images, resulting in low specificity and sensitivity of the recognition methods. Summary of the Invention

[0005] In view of this, it is necessary to provide a gastrointestinal lesion image processing method based on a convolutional neural network, which utilizes a variety of image processing technologies to denoise and enhance the image, constructs a convolutional neural network model based on an attention mechanism, mines local and global features, accurately locates the position of the lesion, improves the accuracy of lesion recognition, and provides a basis for the location and judgment of the lesion.

[0006] In a first aspect, an embodiment of the present application provides a method for processing gastrointestinal lesion images based on a convolutional neural network, characterized in that the method includes the following operating steps:

[0007] S1: Collect multi-type gastrointestinal lesion image data;

[0008] S2: Preprocess the multi-type lesion image data and perform feature extraction;

[0009] S3: Divide the preprocessed data into a training set and a test set, and perform data augmentation processing on the training set;

[0010] S4: Construct the framework of a convolutional neural network model;

[0011] S5: Use the augmented training set to train the convolutional neural network model, use the test set to evaluate the generalization ability of the convolutional neural network model, and adjust the hyperparameters of the convolutional neural network model;

[0012] S6: Use the convolutional neural network model to predict the lesion type of the gastrointestinal lesion image to be processed.

[0013] In one embodiment, in the step S1, collecting multi-type gastrointestinal lesion image data includes:

[0014] Design a multi-threaded real-time image processing framework using the Python language;

[0015] Create a queue for inter-thread communication;

[0016] Use multiple image acquisition threads in the multi-threaded real-time image processing framework to read video images from the electronic medical record archives to obtain an image frame sequence;

[0017] Put the image frame sequence into the queue;

[0018] Transmit image data through the inter-thread communication queue.

[0019] In one embodiment, the step S2: preprocess the multi-type lesion image data and perform feature extraction, including:

[0020] Use the image processing thread in the multi-threaded real-time image processing framework to execute image preprocessing tasks, including:

[0021] Step 1, obtain an image from the queue;

[0022] Step 2, use asynchronous programming technology to process the image, and process and algorithmically analyze the collected image data, specifically:

[0023] According to the importance index of gastrointestinal lesion images, the three RGB components of the images are weighted and averaged to obtain a grayscale image. The calculation formula for the grayscale value Gray(x,y) is as follows:

[0024]

[0025] Among them, α and β are the weight values of the three RGB components respectively, and R(x,y), G(x,y), and B(x,y) are the three components of the pixel (x,y) in RGB;

[0026] The grayscale image is segmented, and different weight values are used to perform filtering and denoising processing on the segmented images respectively. The formula for the filtering algorithm is:

[0027]

[0028] Among them, A and B are the weight values on the components of the pixel (x,y) at the center of the block respectively, u is the adjustment coefficient, N is the total number of pixels, and f(x,y) is the image to be processed;

[0029] Feature extraction is performed using an asynchronous image processing thread, including:

[0030] Select the lesion part in the image as the point of interest POI;

[0031] Select the point of interest POI as the candidate area by sliding the window;

[0032] Feature extraction of color, texture, and morphology is performed on the candidate areas respectively.

[0033] In an embodiment, in step S3, the preprocessed data is divided into a training set and a test set, and data augmentation processing is performed on the training set, including:

[0034] The image data is divided into a training set and a test set according to the preset ratio of 8:2;

[0035] Calculate the grayscale value histogram of the original images in the training set;

[0036] Stretch the grayscale values of the original images and then calculate the cumulative distribution function;

[0037] Perform mapping according to the cumulative distribution function.

[0038] In an embodiment, in step S4, a convolutional neural network model structure framework is constructed, which is in turn: an input layer, an embedding layer, a first enhanced convolutional layer, a first fusion layer, a second enhanced convolutional layer, a feature screening and suppression layer, a second fusion layer, a third enhanced convolutional layer, a third fusion layer, a fourth enhanced convolutional layer, a pooling layer, a conventional convolutional layer, a fully connected layer, and an output layer.

[0039] In one embodiment, the enhanced convolutional layer is as follows:

[0040] Perform a normalization operation on the feature map;

[0041] Divide the feature map after the normalization operation into a conventional convolutional layer channel and a conventional attention layer channel according to a preset channel ratio;

[0042] Perform a convolution operation on the feature map through the conventional convolutional layer channel;

[0043] Perform an attention operation on the feature map through the conventional attention layer channel;

[0044] Fuse the results of the convolution operation and the attention operation, and sequentially perform a conventional convolution operation, a normalization operation, and a conventional convolution operation.

[0045] In one embodiment, the feature screening and suppression layer includes:

[0046] Extract features from the image to obtain a feature set including color, texture, and morphology, and form a fused multi-feature matrix A;

[0047] Set different weights for the extracted image feature indicators;

[0048] Screen the extracted image feature indicators through a weight threshold, and the screening formula is as follows:

[0049]

[0050] where G i represents the multi-feature fusion weight of the i-th index, t i represents the variation weight of the i-th index, δ represents the weight of the i-th index, j represents the j-th sample, k represents the entropy coefficient, n represents the number of samples, and r ij represents the row vector of the fused multi-feature matrix A.

[0051] In one embodiment, in step S5, training the convolutional neural network model using the enhanced training set includes:

[0052] Initialize the parameters in the convolutional neural network model;

[0053] Train the parameters in the convolutional neural network model according to the preset label data in the training set;

[0054] Adopt the cross-entropy function as the loss function to evaluate the difference between the model prediction value and the actual value, and iteratively train the convolutional neural network model in batches;

[0055] Use the gradient descent method to adjust the model parameters.

[0056] In one embodiment, in the step S5, a test set is used to evaluate the generalization ability of the convolutional neural network model, and hyperparameter adjustment is performed on the convolutional neural network model, including:

[0057] According to the evaluation results, the model is adjusted and optimized, adjusting the network structure, optimization algorithm or hyperparameters; evaluating E valuate The formula is:

[0058]

[0059] where h and m represent the number of data sets, and TP h represents the h-th positive sample, and FN h represents the h-th negative sample;

[0060] According to the comparison between E valuate and the threshold, training and evaluation are repeatedly performed until the model performance reaches the expected standard.

[0061] In one embodiment, in the S6, the convolutional neural network model is used to predict the lesion type of the gastrointestinal lesion image to be processed, including:

[0062] Use the trained convolutional neural network model to perform regression prediction on the gastrointestinal lesion image to be processed, and output the predicted value type and the location of the lesion through the output layer.

[0063] In a second aspect, an embodiment of the present application provides a gastrointestinal lesion image processing device based on a convolutional neural network, which is applied to the gastrointestinal lesion image processing method based on a convolutional neural network as described in the first aspect.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0065] A processor;

[0066] A memory for storing processor-executable instructions;

[0067] wherein, when the processor is configured to execute the instructions, it implements the gastrointestinal lesion image processing method based on a convolutional neural network as described in the first aspect.

[0068] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, where the instructions direct a device to execute the gastrointestinal lesion image processing method based on a convolutional neural network as described in the first aspect.

[0069] A gastrointestinal lesion image processing method based on a convolutional neural network provided by an embodiment of the present application can denoise and enhance images, construct a convolutional neural network model based on an attention mechanism, mine local and global features, accurately locate the position of the lesion, improve the accuracy of lesion recognition, and provide a basis for the location and judgment of the lesion.

[0070] The beneficial effects of the system specifically include: providing a gastrointestinal lesion image processing method based on a convolutional neural network, using a variety of image processing techniques to denoise and enhance images, constructing a convolutional neural network model based on an attention mechanism, mining local and global features, accurately locating the position of the lesion, improving the accuracy of lesion recognition, and providing a basis for the location and judgment of the lesion.

[0071] (1) Using a variety of image processing techniques to denoise and enhance images, improving the accuracy of the samples, and thus improving the accuracy of model classification and recognition.

[0072] (2) Designing a multi-threaded real-time image processing framework using the Python language, dividing the threads of image processing respectively, and using asynchronous programming techniques to process images to improve the efficiency of image processing.

[0073] (3) Setting different weights for the extracted image feature indicators according to the characteristics of the field to improve the accuracy of image feature screening.

[0074] (4) Constructing a convolutional neural network model based on an attention mechanism, mining local and global features, accurately locating the position of the lesion, and improving the accuracy of lesion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic flowchart of a gastrointestinal lesion image processing method based on a convolutional neural network provided by an embodiment of the present application.

[0076] Figure 2 It is a schematic structural framework diagram of a convolutional neural network model provided by an embodiment of the present application.

[0077] Figure 3 It is a schematic framework diagram of an enhanced convolutional layer provided by an embodiment of the present application.

[0078] Figure 4 It is a schematic diagram of an electronic terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0080] It should be noted that in the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the description of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0081] It should be noted that in the embodiments of the present application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. The features defined with "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0082] Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present application.

[0083] The present application provides a gastrointestinal lesion image processing method based on a convolutional neural network, which can improve the automation and intelligent management level of a substation to achieve remote monitoring, intelligent control and security protection, and ensure the safe and efficient operation of the substation even in the case of unattended operation.

[0084] Figure 1 It is a schematic flow chart of a gastrointestinal lesion image processing method based on a convolutional neural network provided by an embodiment of the present application. As Figure 1 shown, a gastrointestinal lesion image processing method based on a convolutional neural network includes at least the following operating steps:

[0085] S1: Collect gastrointestinal multi-type lesion image data.

[0086] Specifically, in the step S1, collecting gastrointestinal multi-type lesion image data includes:

[0087] Design a multi-thread real-time image processing framework using the Python language;

[0088] Create a queue for inter-thread communication;

[0089] Use multiple image acquisition threads in the multi-threaded real-time image processing framework to read the imaging video from the electronic medical record, and obtain a sequence of image frames;

[0090] Put the sequence of image frames into a queue;

[0091] Transfer image data through an inter-thread communication queue.

[0092] It can be understood that the thread division of labor: in a multi-threaded environment, the tasks of image acquisition, processing, and display can be assigned to different threads. For example, one thread is responsible for obtaining image data from the electronic medical record, another thread is responsible for processing the acquired image data and performing algorithm analysis, and the last thread is responsible for displaying the processed result on the screen.

[0093] In the embodiment of the present application, since there may be data dependency relationships between different threads, measures need to be taken to ensure data consistency and integrity. This usually involves using mutexes, condition variables, or other synchronization primitives to coordinate access between threads. At the same time, in order to avoid blocking the UI thread, asynchronous programming techniques can be used to process images, and asynchronous processing is also required. For example, in Qt, the signal and slot mechanism can be used to implement communication between the background thread and the main thread.

[0094] S2: Preprocess the multi-type lesion image data and perform feature extraction.

[0095] Specifically, the step S2: Preprocess the multi-type lesion image data and perform feature extraction, includes:

[0096] Use the image processing thread in the multi-threaded real-time image processing framework to execute image preprocessing tasks, including:

[0097] Step 1, obtain an image from the queue.

[0098] Step 2, use asynchronous programming techniques to process the image, and process and perform algorithm analysis on the acquired image data.

[0099] It can be understood that the image preprocessing algorithm is suitable for being completed on an FPGA, with a simple principle and requiring fast processing. Usually, there are grayscale conversion, median, mean filtering, etc., as well as color space conversion algorithms. A grayscale image is a special color image (a color image with R = G = B) with only one color component, a single channel of 0 - 255.

[0100] It can be understood that the process of converting an RGB image into a grayscale image is called image grayscale processing. For general workpieces, the workpieces obtained by a CCD are usually RGB images. For each image, each pixel usually consists of three channels, namely the RGB three channels. For a color image, when transmitted to a computer, too much information is transmitted, which makes it very difficult for the computer to directly process the color image. When R = G = B, the image can be represented by only one component, which reduces a lot of the workload for the computer.

[0101] It can be understood that conventional grayscale processing, such as mean grayscale processing, will blur the feature information and edge information of the image. Especially for some detailed parts, it will cause the image to lose many key features and result in image distortion. Therefore, in the embodiments of the present application, a method of grayscale screening processing according to index weights is set.

[0102] In the embodiments of the present application, according to the importance index of the gastrointestinal lesion image, the three RGB components of the image are weighted and averaged to obtain a grayscale image. The calculation formula for the grayscale value Gray(x, y) is:

[0103]

[0104] Among them, α and β are the weights of the three RGB components respectively, and R(x, y), G(x, y), and B(x, y) are the three components of the pixel (x, y) on RGB respectively.

[0105] In the embodiments of the present application, the three components are weighted and averaged with different weights. Compared with the conventional conversion of RGB components to grayscale images, the weighted processed image, such as the grayscale image of the Y component, has more layering, making the diseased part more prominent.

[0106] In the embodiments of the present application, the grayscale image is segmented, and different weights are used to perform filtering and denoising processing on the segmented images respectively. The formula of the filtering algorithm is:

[0107]

[0108] Among them, A and B are the weights of the pixel (x, y) component at the center point of the block respectively, u is an adjustment coefficient, N is the total number of pixels, and f(x, y) is the image to be processed.

[0109] In the embodiments of the present application, by performing block filtering operations on the grayscale image, the step size of the sliding window processing can be increased. At the same time, the key diseased part can be quickly located, and feature extraction is mainly performed on this part, providing an accurate data source for subsequent data processing.

[0110] In the embodiments of the present application, an asynchronous image processing thread is used for feature extraction, including:

[0111] Select the diseased part in the image as the point of interest (POI);

[0112] Use a sliding window to frame the point of interest (POI) as a candidate region;

[0113] Extract the features of color, texture, and morphology of the candidate region respectively.

[0114] Specifically, in modern software development, efficiently processing concurrent tasks has become a key challenge. As a versatile programming language, Python provides various methods for processing asynchronous tasks. Multithreading is another method for processing concurrent tasks. Python's threading module allows the creation and management of threads, and each thread can execute tasks independently. Multithreading is suitable for I / O-intensive tasks.

[0115] S3: Divide the preprocessed data into a training set and a test set, and perform data augmentation on the training set.

[0116] Specifically, in step S3, dividing the preprocessed data into a training set and a test set, and performing data augmentation on the training set includes:

[0117] Divide the image data into a training set and a test set according to the preset ratio of 8:2;

[0118] Calculate the grayscale value histogram of the original images in the training set;

[0119] Stretch the grayscale values of the original images and then calculate the cumulative distribution function;

[0120] Perform mapping according to the cumulative distribution function.

[0121] It can be understood that the data set is divided into a training set, a validation set, and a test set. The training set is used for the learning and parameter optimization of the model, the validation set is used for adjusting the model hyperparameters and model selection, and the test set is used for the final evaluation of the model performance. The training set is used to learn the parameters and weights of the model to enable it to have prediction ability. The validation set is used to evaluate the generalization ability of the model on new data, optimize the model parameters, and select the best hyperparameters.

[0122] In the embodiments of the present application, the data division ratio of the training set and the test set is set to 8:2, and most of the data is used as the training set because deep learning algorithms require a very large amount of data. For problems with massive datasets, a higher proportion of data is allocated to the training set. The training set can help train the model, and the parameters of the fitting curve are determined through the data in the training set. The validation set is used for model selection, that is, for the final optimization and determination of the model, and is used to assist in the construction of the model, that is, to train hyperparameters.

[0123] S4: Construct the structural framework of the convolutional neural network model.

[0124] Specifically, as Figure 2 shown, it is a schematic diagram of the structural framework of the convolutional neural network model provided by an embodiment of the present application. In step S4, the structural framework of the convolutional neural network model is constructed in sequence as: input layer, embedding layer, first enhanced convolutional layer, first fusion layer, second enhanced convolutional layer, feature screening and suppression layer, second fusion layer, third enhanced convolutional layer, third fusion layer, fourth enhanced convolutional layer, pooling layer, conventional convolutional layer, fully connected layer, and output layer.

[0125] It can be understood that this structure is divided into four stages, and spatial downsampling is performed using the embedding layer or the fusion layer before each stage. At the same time, the number of spatial channels can also be expanded.

[0126] It can be understood that, as Figure 3 shown, it is a schematic diagram of the enhanced convolutional layer framework provided by an embodiment of the present application. The enhanced convolutional layer is:

[0127] Perform a normalization operation on the feature map;

[0128] Divide the feature map after the normalization operation into a conventional convolutional layer channel and a conventional attention layer channel according to a preset channel ratio;

[0129] Perform a convolution operation on the feature map through the conventional convolutional layer channel;

[0130] Perform an attention operation on the feature map through the conventional attention layer channel;

[0131] Fuse the results of the convolution operation and the attention operation, and perform a conventional convolution operation, a normalization operation, and a conventional convolution operation in sequence.

[0132] In order to fully extract the global feature information and local feature information of gastrointestinal disease images, in the embodiments of the present application, as Figure 3As shown in the figure, in the structure of the enhanced convolutional layer, an attention mechanism is added. The feature map can be separated into two parts through channels. One part performs a convolution operation on the feature map through the channels of the conventional convolutional layer, and the other part performs an attention operation on the feature map through the channels of the conventional attention layer. In this way, both global feature information and local feature information are extracted, making the extracted feature information richer and more comprehensive.

[0133] At the same time, the structure of the enhanced convolutional layer introducing the attention mechanism avoids using all the data sets for convolution operations, reduces the redundant computational amount and the number of memory accesses, and can extract spatial features more effectively and quickly.

[0134] In the embodiment of the present application, the feature screening and suppression layer includes:

[0135] Extract features from the image to obtain a feature set including color, texture, and morphology, and form a fused multi-feature matrix A;

[0136] Set different weights for the extracted image feature indicators;

[0137] Screen the extracted image feature indicators through a weight threshold, and the screening formula is as follows:

[0138]

[0139] Where G i represents the multi-feature fusion weight of the i-th index, t i represents the variation weight of the i-th index, δ represents the weight of the i-th index, j represents the j-th sample, k represents the entropy coefficient, n represents the number of samples, and r ij represents the row vector of the fused multi-feature matrix A.

[0140] Specifically, the principle of the screening formula is that when a certain feature weight is less than the threshold δ, it is considered that this type of feature index makes little contribution to the identification of the diseased plant and can be directly set to 0 and discarded; on the contrary, when a certain feature weight is greater than or equal to the threshold δ, it is considered that this type of feature index makes a great contribution to the identification of the diseased plant and can be enhanced through the weight. The feature screening and suppression layer can be used to selectively enhance the expression of lesion features and suppress less useful features.

[0141] S5: Use the enhanced training set to train the convolutional neural network model, use the test set to evaluate the generalization ability of the convolutional neural network model, and perform hyperparameter adjustment on the convolutional neural network model.

[0142] Specifically, in the step S5, using the enhanced training set to train the convolutional neural network model includes:

[0143] Initialize the parameters in the convolutional neural network model;

[0144] Train the parameters in the convolutional neural network model according to the preset label data in the training set;

[0145] Use the cross-entropy function as the loss function to evaluate the difference between the model prediction value and the actual value, and iteratively train the convolutional neural network model in batches;

[0146] Use the gradient descent method to adjust the model parameters.

[0147] It can be understood that in step S5, the test set is used to evaluate the generalization ability of the convolutional neural network model, and hyperparameter adjustment of the convolutional neural network model includes:

[0148] According to the evaluation results, adjust and optimize the model, adjust the network structure, optimization algorithm or hyperparameters; evaluate E valuate The formula is:

[0149]

[0150] where h and m represent the number of data sets, and TP h represents the h-th positive sample, and FN h represents the h-th negative sample;

[0151] According to the comparison between E valuate and the threshold, repeatedly perform training and evaluation until the model performance reaches the expected standard.

[0152] In the embodiments of the present application, by comparing E valuate with the threshold, repeatedly perform training and evaluation until the model performance reaches the expected standard.

[0153] S6: Use the convolutional neural network model to predict the lesion type of the gastrointestinal lesion image to be processed.

[0154] Specifically, in S6, using the convolutional neural network model to predict the lesion type of the gastrointestinal lesion image to be processed includes:

[0155] Use the trained convolutional neural network model to perform regression prediction on the gastrointestinal lesion image to be processed, and output the prediction value type and the location of the lesion through the output layer.

[0156] It can be understood that the prediction value type includes various types of gastrointestinal diseases, which can be determined according to the sampled sample data and are not limited herein.

[0157] It can be understood that the gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application utilizes a variety of image processing techniques to denoise and enhance the images, constructs a convolutional neural network model based on an attention mechanism to mine local and global features, accurately locate the positions of the lesions, and improve the accuracy of lesion recognition, providing a basis for the positioning and judgment of the lesions.

[0158] The gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application utilizes a variety of image processing techniques to denoise and enhance the images, improve the accuracy of the samples, and further improve the accuracy of model classification and recognition.

[0159] The gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application designs a multi-threaded real-time image processing framework using the Python language, divides the threads for image processing respectively, and uses asynchronous programming techniques to process the images, improving the efficiency of image processing.

[0160] The gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application sets different weights for the extracted image feature indicators according to the domain characteristics, improving the accuracy of image feature screening.

[0161] The gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application constructs a convolutional neural network model based on an attention mechanism to mine local and global features, accurately locate the positions of the lesions, and improve the accuracy of lesion recognition.

[0162] It can be understood that the gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiments of the present application can be implemented by a computer device with the same functional modules. Its functions are the same as or similar to those Figure 1 shown in part, and will not be elaborated here.

[0163] In the embodiments of the present application, a data acquisition module is used to acquire gastrointestinal multi-type lesion image data. For the specific method, please refer to Figure 1 its corresponding description together, and the present application will not elaborate here.

[0164] In the embodiments of the present application, a data processing module is used to preprocess the multi-type lesion image data and extract features. For the specific method, please refer to Figure 1 its corresponding description together, and the present application will not elaborate here.

[0165] In the embodiments of the present application, a data division module is used to divide the preprocessed data into a training set and a test set, and perform data augmentation processing on the training set. For the specific method, please refer to Figure 1 its corresponding description together, and the present application will not elaborate here.

[0166] In an embodiment of the present application, a model construction module is configured to construct a convolutional neural network model structure framework. For the specific method, please refer to Figure 1 its corresponding description, which will not be elaborated in this application.

[0167] In an embodiment of the present application, a model training module is configured to train the convolutional neural network model by using the enhanced training set, use the test set to evaluate the generalization ability of the convolutional neural network model, and perform hyperparameter adjustment on the convolutional neural network model. For the specific method, please refer to Figure 1 its corresponding description, which will not be elaborated in this application.

[0168] In an embodiment of the present application, a prediction output module is configured to use the convolutional neural network model to predict the type of lesion for the gastrointestinal lesion image to be processed. For the specific method, please refer to Figure 1 its corresponding description, which will not be elaborated in this application.

[0169] It can be understood that the gastrointestinal lesion image processing method based on a convolutional neural network provided by the embodiment of the present application utilizes a variety of image processing technologies to perform denoising and enhancement processing on the image, constructs a convolutional neural network model based on an attention mechanism, mines local and global features, accurately locates the position of the lesion, improves the accuracy of lesion recognition, and provides a basis for the location and judgment of the lesion.

[0170] See Figure 4 , Figure 4 is an electronic terminal device provided by an embodiment of the present application. As Figure 4 shown, the electronic terminal device at least includes the following parts: one or more processors, one or more input devices, one or more output devices, and one or more memories. The above-mentioned processors, input devices, output devices, and memories communicate with each other through a communication bus. The memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to execute the program instructions stored in the memory. Among them, the processor is configured to call the program instructions to perform the functions of each module / unit in the above-mentioned device embodiments.

[0171] In an embodiment of the present application, a computer-readable storage medium includes instructions that direct a device to execute the method according to the first aspect. For example, the instructions direct the device to execute the gastrointestinal lesion image processing method based on a convolutional neural network shown in the steps of Figure 1 .

[0172] It should be understood that in the embodiments of the present invention, the so-called processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. It should be noted that a part of the electronic device in the above embodiments may also be implemented by a computer. In this case, the program for implementing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer and executed to implement it.

[0173] The input device may include a touchpad, a fingerprint sensor (for collecting the fingerprint information and the direction information of the fingerprint) of the user, a microphone, etc., and the output device may include a display (such as an LCD), a speaker, etc.

[0174] It should be noted that the "computer" mentioned here refers to the computer built in the electronic device, which is a computer adopting hardware including an OS, peripheral devices, etc. In addition, the "computer-readable recording medium" refers to removable media such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and storage devices such as a hard disk built in the computer.

[0175] Moreover, the "computer-readable recording medium" may include: a medium that dynamically stores a program in a short time like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line; a medium that stores a program within a fixed time like a volatile memory inside a computer of a server or a client in this case. In addition, the above program may be a part of the program for implementing the above functions, and may also be a program that can implement the above functions by combining with the program recorded in the computer.

[0176] In addition, the electronic device in the above embodiments can also be implemented as an aggregate (device group) composed of multiple devices. Each device constituting the device group may have some or all of the functions or function blocks of the electronic device in the above embodiments. As the device group, it is sufficient to have all the functions or function blocks of the electronic device.

[0177] It can be understood that the gastrointestinal lesion image processing method, electronic device, and storage medium provided by the embodiments of the present application utilize a variety of image processing technologies to denoise and enhance the image, construct a convolutional neural network model based on the attention mechanism, mine local and global features, accurately locate the lesion position, improve the accuracy of lesion recognition, and provide a basis for the localization and judgment of lesions. By using a variety of image processing technologies to denoise and enhance the image, the accuracy of the samples is improved, and thus the accuracy of model classification and recognition is improved. A multi-threaded real-time image processing framework is designed using the Python language, with thread division of labor for image processing, and asynchronous programming technology is used to process images to improve the efficiency of image processing. According to the characteristics of the field, different weights are set for the extracted image feature indicators to improve the accuracy of image feature screening. A convolutional neural network model based on the attention mechanism is constructed to mine local and global features, accurately locate the lesion position, and improve the accuracy of lesion recognition.

[0178] Those of ordinary skill in the art of this technology should recognize that the above embodiments are only used to illustrate the present application and are not intended to limit the present application. As long as appropriate changes and variations are made to the above embodiments within the scope of the spirit of the present application, they fall within the scope of protection required by the present application.

Claims

1. A gastrointestinal lesion image processing method based on convolutional neural network, characterized in that: The method comprises the following steps: S1: Collect image data of multiple types of gastrointestinal lesions; S2: preprocessing the multi-type lesion image data and performing feature extraction; S3: dividing the preprocessed data into a training set and a test set, and performing data enhancement processing on the training set; S4: Construct the convolutional neural network model structure framework; S5: using the enhanced training set to train the convolutional neural network model, using the test set to evaluate the generalization ability of the convolutional neural network model, and adjusting the hyperparameters of the convolutional neural network model; S6: Use the convolutional neural network model to predict the lesion type of the gastrointestinal lesion image to be processed.

2. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 1, characterized in that: In step S1, collecting image data of multiple types of gastrointestinal tract lesions includes: Design a multi-threaded real-time image processing framework using Python language; Create a queue for inter-thread communication; Using multiple image acquisition threads in the multi-threaded real-time image processing framework to read image videos from electronic medical records to obtain image frame sequences; Put the image frame sequence into a queue; Image data is passed through an inter-thread communication queue.

3. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 2, characterized in that: The step S2: preprocessing the multi-type lesion image data and performing feature extraction, including: Using the image processing thread in the multi-threaded real-time image processing framework, image preprocessing tasks are performed, including: Step 1, get the image from the queue; Step 2: Use asynchronous programming technology to process the image and perform algorithm analysis on the collected image data, specifically: According to the importance index of the gastrointestinal lesion image, the three components of RGB of the image are weighted averaged to obtain a grayscale image. The grayscale value Gray (x, y) is calculated as follows: in, α and β are the weights of the three components of RGB, respectively. R(x,y), G(x,y), and B(x,y) are the three components of the pixel (x,y) on RGB. The grayscale image is divided into blocks, and different weights are used to filter and denoise the block images respectively. The formula of the filtering algorithm is: Where A and B are the weights of the pixel (x, y) component at the center of the block, u is the adjustment coefficient, N is the total number of pixels, and f(x, y) is the image being processed; Use an asynchronous image processing thread for feature extraction, including: Select the lesion site in the image as the point of interest POI; Use the sliding window to select the point of interest (POI) as the candidate area; The color, texture and shape features of the candidate regions are extracted respectively.

4. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 1, characterized in that: In step S3, the preprocessed data is divided into a training set and a test set, and data enhancement processing is performed on the training set, including: The image data is divided into training set and test set according to the preset ratio of 8:2; Calculate the gray value histogram of the original image of the training set; The grayscale value of the original image is stretched and then a cumulative distribution function is calculated; Mapping is done according to the cumulative distribution function.

5. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 4, characterized in that: In the step S4, a convolutional neural network model structure framework is constructed, which is sequentially: input layer, embedding layer, first enhanced convolution layer, first fusion layer, second enhanced convolution layer, feature screening suppression layer, second fusion layer, third enhanced convolution layer, third fusion layer, fourth enhanced convolution layer, pooling layer, conventional convolution layer, fully connected layer and output layer.

6. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 5, characterized in that: The enhanced convolutional layer is: Normalize the feature map; Dividing the normalized feature map into conventional convolutional layer channels and conventional attention layer channels according to a preset channel ratio; Performing a convolution operation on the feature map through the conventional convolution layer channel; Performing an attention operation on the feature map through the conventional attention layer channel; The convolution operation results and the attention operation results are fused, and regular convolution operation, normalization operation, and regular convolution operation are performed in sequence.

7. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 5, characterized in that: The feature screening suppression layer comprises: Extract features from the image to obtain a feature set including color, texture, and shape to form a fused multi-feature matrix A; Setting different weights for the extracted image feature indicators; The extracted image feature indicators are screened by weight threshold, and the screening formula is as follows: Among them, G i represents the multi-feature fusion weight of the i-th indicator, t i represents the variation weight of the i-th indicator, δ represents the weight of the i-th indicator, j represents the j-th sample, k represents the entropy coefficient, n represents the number of samples, r ij Represents the row vector of the fused multi-feature matrix A.

8. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 6, characterized in that: In step S5, the convolutional neural network model is trained using the enhanced training set, including: Initialize the parameters in the convolutional neural network model; Train the parameters in the convolutional neural network model according to the preset label data in the training set; The cross entropy function is used as the loss function to evaluate the difference between the model prediction value and the actual value, and the convolutional neural network model is trained iteratively in batches; The model parameters are tuned using gradient descent.

9. The gastrointestinal lesion image processing method based on convolutional neural network according to claim 8, characterized in that: In step S5, the test set is used to evaluate the generalization ability of the convolutional neural network model, and the hyperparameters of the convolutional neural network model are adjusted, including: According to the evaluation results, adjust and optimize the model, adjust the network structure, optimize the algorithm or hyperparameters; evaluate E valuate The formula is: Where h and m represent the number of data sets, TP h represents the hth positive sample, FN h represents the hth negative sample; According to E valuate The training and evaluation are repeated by comparing with the threshold until the model performance reaches the expected standard.

10. The gastrointestinal tract lesion image processing method based on convolutional neural network according to claim 5, characterized in that: In S6, using the convolutional neural network model to predict the lesion type of the gastrointestinal lesion image to be processed includes: The trained convolutional neural network model is used to perform regression prediction on the gastrointestinal lesion images to be processed, and the predicted value type and lesion location are output through the output layer.

Images