Digestive tract focus identification method and system based on endoscope picture
Through the combination of multimodal medical image fusion network and SVM model, the original color and infrared image characteristics of the endoscopic image lesion recognition method in noise and diversified morphology recognition is solved, and the limitations of the digestive tract endoscopic image lesion recognition method are improved. The identification accuracy and robustness are achieved, which significantly improves the recognition effect of digestive lesions.
Patent Information
- Application Number
- CN202411053283.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing gastrointestinal endoscopic image lesion recognition method has limitations in dealing with image noise, light changes and recognition of different lesion morphology. The recognition diversity is low, resulting in a decrease in the practicality and effectiveness of the identification system.
Using a multimodal medical image fusion network, shape, texture and color features are extracted by combining endoscopic original color images and infrared images, and feature fusion is performed, and the image is reconstructed and identified using generators and discriminators, combined with SVM The model makes exception type decisions to improve identification accuracy.
It significantly improves the accuracy and robustness of gastrointestinal lesions, reduces the burden on doctors, and solves the problems of traditional methods in noise processing and diversified lesion morphology recognition.
Smart Images

Figure CN119027726A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent processing of medical images, and in particular to a method and system for identifying digestive tract lesions based on endoscopic images. Background Art
[0002] In recent years, with the continuous improvement of medical technology, medical imaging technology has made great breakthroughs and played an extremely important role in clinical medicine. However, due to different imaging principles, the contents contained in the images generated by different imaging devices are also different.
[0003] Endoscopic diagnosis of gastrointestinal lesions has always been a difficult problem in clinical practice. The manual diagnosis process is susceptible to a variety of negative factors, such as the doctor's lack of experience, fatigue, limited diagnostic time, a huge number of images, and complex and changeable lesion appearances. Therefore, with the help of computer technology, the development of auxiliary diagnosis methods for gastrointestinal lesions based on the image information provided by the endoscope will provide a powerful supplement to the existing endoscopic diagnosis, which will help reduce missed diagnosis and misdiagnosis of gastrointestinal lesions and improve diagnostic efficiency, and has very important scientific significance.
[0004] Medical image fusion technology can extract the unique and significant features of each modality of medical images based on the spatiotemporal correlation and information complementarity of multiple images, and fuse these features into one image. The application of this technology can greatly reduce the number of films that doctors need to read and effectively improve the efficiency of disease diagnosis and treatment. At the same time, with the rapid development of deep learning, research in this field has also made breakthrough progress. Therefore, it is necessary to apply medical image fusion technology to lesion information recognition to reduce the occurrence of misdiagnosis and missed diagnosis.
[0005] In the prior art, the publication number CN113034500A discloses a gastrointestinal endoscopy image lesion based on a multi-channel structure, which includes: a training data acquisition and annotation module for acquiring multiple sets of training data; a preprocessing module for preprocessing the gastrointestinal endoscopy image to obtain a processed gastrointestinal endoscopy image; a model construction module for constructing a preset network model; a model training module for inputting the processed gastrointestinal endoscopy image into a preset network model, and using a preset loss function to train the network model to obtain a recognition model; a recognition module for using the recognition model combined with a non-maximum suppression algorithm to recognize the input gastrointestinal endoscopy image. This method trains a large amount of labeled data, and the model can recognize common lesion types. However, these methods still have limitations in processing image noise, illumination changes, and recognition of different lesion morphologies. For example, the recognition model can only recognize a single type of image, resulting in low recognition diversity, which not only wastes resources, but also reduces the practicality and effectiveness of the recognition system.
[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention
[0007] The purpose of the present invention is to provide a method and system for identifying gastrointestinal lesions based on endoscopic images to solve the problems raised in the above-mentioned background technology.
[0008] To achieve the above object, the present invention provides the following technical solutions:
[0009] The digestive tract lesion recognition method based on endoscopic images includes the following specific steps:
[0010] Collecting an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers, and bleeding;
[0011] Preprocessing the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing;
[0012] Extracting discriminative features of the original color image and the infrared image according to the endoscope original color image and the endoscope infrared image obtained after preprocessing, wherein the discriminative features include shape features, texture features and color features, and performing feature fusion on the features of the original color image and the corresponding infrared image features according to the extracted discriminative features to obtain fused image features;
[0013] The obtained fused image features are used as input of a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input into the generator, the generator performs image reconstruction to generate a fused image, and the discriminator is used to identify the difference between the fused image and the input image features;
[0014] An abnormality recognition is performed on the fused image using an image lesion recognition model, wherein the abnormality includes polyps, ulcers, and bleeding, different types of fused images are sorted, polyps, ulcers, and bleeding areas in the fused image are manually marked, and the image lesion recognition model is trained using the manually marked polyps, ulcers, and bleeding areas in the fused image as labels, to obtain an image lesion recognition model with a fused image as input and a decision value of polyps, ulcers, and bleeding abnormalities as output;
[0015] The endoscope original color images and the endoscope infrared images to be identified in the same time series are collected, and after feature extraction and fusion, a fused image to be identified is generated through a multimodal medical image fusion network, and the fused image to be identified is input into the image lesion recognition model obtained by the method of establishing the image lesion recognition model to obtain the abnormal decision value of the fused image to be identified;
[0016] The abnormal decision values of the fused image to be identified are analyzed to generate weighted decision values, which are compared with pre-set decision boundaries for different lesion types to determine the type of abnormality that exists.
[0017] Furthermore, endoscopic original color images and endoscopic infrared images of the same target individual are collected, wherein the method for collecting endoscopic original color images and endoscopic infrared images of the same target individual is: use a thermal infrared camera and an endoscopic camera to perform video monitoring of the digestive tract of the same target individual, respectively, and extract frame images from the collected videos as sample images.
[0018] Furthermore, the collected endoscope original color image and endoscope infrared image are preprocessed respectively, and the preprocessing includes image denoising and enhancement processing, and the logic of the preprocessing is as follows:
[0019] The denoising method of wavelet transform is used to denoise the original color image of the endoscope and the infrared image of the endoscope. The specific steps of the denoising method of wavelet transform include: decomposing the original color image of the endoscope and the infrared image of the endoscope through wavelet transform to obtain the wavelet coefficients of the image at different scales and directions; thresholding the wavelet coefficients to set the low-amplitude wavelet coefficients to zero and retain the high-amplitude wavelet coefficients; inversely transforming the wavelet coefficients after thresholding, reconstructing the processed coefficients into an image, and completing the denoising of the original color image of the endoscope and the infrared image of the endoscope;
[0020] Bilateral filtering is used to enhance the details of the endoscope original color image and the endoscope infrared image. The formula for the specific filtering transformation is:
[0021]
[0022] In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is:
[0023]
[0024]
[0025] In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value at and Gaussian function The standard deviation of .
[0026] Furthermore, according to the original color image and the infrared image of the endoscope obtained after preprocessing, the discriminative features of the original color image and the infrared image are extracted, and the discriminative features include shape features, texture features and color features. The method for extracting the discriminative features of the image is based on a convolutional neural network, and the logic of feature extraction is as follows:
[0027] Collect a dataset of endoscope original color images and endoscope infrared images containing various shapes, textures, and color features, and use the shape, texture, and color features of polyps, ulcers, and hemorrhages as labels for training;
[0028] The input layer of the convolutional neural network receives the original color image and infrared image of the endoscope obtained after preprocessing. The convolution kernel is used in the convolution layer to perform a convolution operation on the input image to extract the discriminative features. The formula based on the convolution operation is:
[0029]
[0030] In the formula, The convolution result feature map is at position The value of is the convolution kernel at position The weight of is the size of the convolution kernel, Represents the input image at position The pixel value of It is a bias term, which is added to the result of each convolution operation to adjust the value of the output feature map; the function activation layer uses the ReLU activation function.
[0031] Furthermore, according to the extracted discriminative features, the features of the original color image are fused with the features of the corresponding infrared image to obtain fused image features, wherein the logic for fusion of the features of the original color image with the features of the corresponding infrared image is: the energy attribute fusion strategy is used to fuse the extracted features, and the formula for the energy attribute fusion strategy is:
[0032]
[0033]
[0034] In the formula, and They represent the features of the original color image and the corresponding infrared image after being processed by the energy attribute fusion strategy. represents the modulation parameters, where , and Respectively represent the original color image and the corresponding infrared image in The pixel value at the point, and Respectively represent the characteristic values of the original color image and the corresponding infrared image;
[0035] The features of the original color image processed by the energy attribute fusion strategy and the corresponding infrared image are and The fusion is performed to obtain the fused image features, and the formula based on it is:
[0036]
[0037] In the formula, Represents the fused image features obtained by fusion.
[0038] Furthermore, the obtained fused image features are used as input of a multimodal medical image fusion network, which includes a generator and two discriminators. The generator is DenseNet, and the discriminator compares the generated fused image with the input source image to determine whether the fused image is true or false. If it is determined to be false, it means that the difference in feature distribution between the fused image and the input source image does not meet the requirements, and the generator parameters need to be adjusted and iterated multiple times until the optimal generator is obtained.
[0039] Furthermore, an image lesion recognition model is used to perform abnormality recognition on the fused image, wherein the image lesion recognition model is an SVM model, and the fused image to be recognized is input into the image lesion recognition model obtained by the method of establishing the image lesion recognition model, and the output of the SVM model is an abnormality decision value that the abnormality existing in the fused image to be recognized belongs to the polyp, ulcer and bleeding area; wherein the abnormality decision value of the fused image to be recognized is , For the The abnormal decision value vector of the fused image samples, Represent the decision values of the fused image from the decision boundaries of polyp, ulcer and bleeding, respectively.
[0040] Furthermore, the formula for generating the weighted decision value based on the abnormal decision value analysis of the fused image to be identified is:
[0041]
[0042] In the formula, For the The weighted decision value of the fused image samples, is the weight coefficient matrix, Represents the weight coefficient matrix The transpose of , where the weight coefficient matrix The expression is:
[0043]
[0044] In the formula, and are the weight coefficients of polyp, ulcer and bleeding decisions respectively, among which, and are greater than 0 and ;
[0045] The weighted decision value is compared with the pre-set decision limits for different lesion types to determine the type of abnormality present. The logic is as follows: the decision limits for polyps, ulcers, and hemorrhages are and ,in ;
[0046] when When the abnormality is detected, the abnormality type is judged to be a polyp;
[0047] when When the abnormality is detected, it is judged that the abnormality type is ulcer;
[0048] when When the abnormality is detected, it is judged to be bleeding;
[0049] when , it is judged that there is no abnormality.
[0050] The present invention also provides a digestive tract lesion recognition system based on endoscopic images, and the digestive tract lesion recognition system based on endoscopic images is used to execute the above-mentioned digestive tract lesion recognition method based on endoscopic images, comprising:
[0051] A sample image acquisition module, used to acquire an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers and bleeding;
[0052] An image preprocessing module, used to preprocess the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing;
[0053] A feature extraction and fusion module is used to extract the original color image and the infrared image discrimination features according to the endoscope original color image and the endoscope infrared image obtained after preprocessing, wherein the discrimination features include shape features, texture features and color features, and according to the extracted discrimination features, the features of the original color image are feature-fused with the corresponding infrared image features to obtain fused image features;
[0054] An image reconstruction module, used to use the obtained fused image features as input to a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input to the generator, and the generator performs image reconstruction to generate a fused image. The discriminator is used to identify the difference between the fused image and the input image features;
[0055] A lesion recognition model training module is used to construct an image lesion recognition model to perform abnormality recognition on a fused image, wherein the abnormalities include polyps, ulcers, and bleeding, sort out different types of fused images, manually mark the polyps, ulcers, and bleeding areas in the fused images, and train the image lesion recognition model using the manually marked polyps, ulcers, and bleeding areas in the fused images as labels, to obtain an image lesion recognition model with a fused image as input and a polyp, ulcer, and bleeding abnormality decision value as output;
[0056] The lesion abnormality analysis module is used to collect the endoscope original color images and the endoscope infrared images to be identified in the same time series, extract and fuse the features, generate the fused image to be identified through the multimodal medical image fusion network, input the fused image to be identified into the image lesion recognition model obtained by the method of establishing the image lesion recognition model, and obtain the abnormal decision value of the fused image to be identified;
[0057] The lesion abnormality type determination module is used to analyze the abnormal decision value of the fused image to be identified to generate a weighted decision value, compare the weighted decision value with the pre-set decision boundaries of different lesion types, and determine the type of abnormality that exists.
[0058] Compared with the prior art, the present invention has the following beneficial effects:
[0059] The method comprises collecting an endoscopic original color image and an endoscopic infrared image of the same target individual as sample data, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, including polyps, ulcers and bleeding; preprocessing the sample images to improve the sample image quality, extracting features of the original color image and the infrared image using a convolutional neural network, and fusing the extracted features; reconstructing the image using a generator according to the fused features to generate a fused image, manually marking the abnormalities in the fused image, using the marked images as training samples of an image lesion recognition model, training the model with the marked abnormalities in the image as labels, inputting the fused image to be identified into the trained model, and obtaining an abnormal decision value of the fused image to be identified; generating a weighted decision value based on the obtained abnormal decision value of the fused image to be identified, and comparing the weighted decision value with pre-set decision boundaries of different lesion types to determine the type of abnormality that exists. Through the combination of multimodal image fusion and integrated learning models, more comprehensive information content is provided, which can significantly improve the recognition accuracy and robustness of gastrointestinal lesions, reduce the burden on doctors, and solve the difficulties of traditional methods in noise processing, illumination changes and diverse lesion morphology recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is a schematic diagram of the overall method flow of the present invention;
[0061] Figure 2 It is a schematic diagram of the overall system structure of the present invention. DETAILED DESCRIPTION
[0062] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.
[0063] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0064] Example:
[0065] See also Figure 1 , the present invention provides a technical solution:
[0066] The digestive tract lesion recognition method based on endoscopic images includes the following specific steps:
[0067] Step 1: collecting an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers, and bleeding;
[0068] The endoscopic original color image and the endoscopic infrared image of the same target individual are collected, wherein the method for collecting the endoscopic original color image and the endoscopic infrared image of the same target individual is: use a thermal infrared camera and an endoscopic camera to respectively perform video monitoring of the digestive tract of the same target individual, and extract frame images from the collected videos as sample images.
[0069] Step 2: preprocessing the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing;
[0070] The logic of its preprocessing is: using the wavelet transform denoising method to denoise the endoscope infrared image;
[0071] Among them, the denoising method of wavelet transform is used for denoising. The specific steps of the wavelet transform denoising method include: converting the original color image and infrared image into grayscale image, normalizing the image so that the grayscale value is between 0 and 1; selecting a suitable wavelet basis function and determining the wavelet function (such as Daubechies wavelet db4); selecting the number of layers of wavelet decomposition, generally 3 to 5 layers; thresholding the wavelet coefficients, selecting the threshold method, the threshold method includes hard threshold or soft threshold, and calculating the threshold; performing inverse wavelet transform on the wavelet coefficients after thresholding, and reconstructing the denoised image; wherein the threshold is calculated using the VisuShrink method, and the specific calculation formula is:
[0072]
[0073] In the formula, is the threshold value, is the standard deviation of the noise, is the total number of pixels in the image;
[0074] Bilateral filtering is used to enhance the details of the original color image and infrared image, and further reduce the noise. The formula based on the specific filtering transformation is:
[0075]
[0076] In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is:
[0077]
[0078]
[0079] In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value at and Gaussian function The standard deviation of .
[0080] It measures the spatial relationship between x and y, so that pixels closer to each other have a greater impact on the grayscale value at x; It measures the relationship between the value ranges of x and y, so that points with similar grayscale values in the neighborhood have greater weights in the calculation; and are all Gaussian functions, with standard deviations of and In flat areas, the grayscale value changes little in all directions, and the range weight In the neighborhood, it is close to 1. At this time, only the position relationship affects the filtering result. It is equivalent to Gaussian filtering and has a good noise suppression effect.
[0081] Step 3: Extract the discriminative features of the original color image and the infrared image according to the endoscopic original color image and the endoscopic infrared image obtained after preprocessing, wherein the discriminative features include shape features, texture features and color features, and perform feature fusion on the features of the original color image and the corresponding infrared image features according to the extracted discriminative features to obtain fused image features;
[0082] The method for extracting image discriminative features is based on convolutional neural network, and the logic of feature extraction is as follows:
[0083] Collect a dataset of endoscope original color images and endoscope infrared images containing various shapes, textures, and color features, and use the shape, texture, and color features of polyps, ulcers, and hemorrhages as labels for training;
[0084] The input layer of the convolutional neural network receives the original color image and infrared image of the endoscope obtained after preprocessing. The convolution kernel is used in the convolution layer to perform a convolution operation on the input image to extract the discriminative features. The formula based on the convolution operation is:
[0085]
[0086] In the formula, The convolution result feature map is at position The value of is the convolution kernel at position The weights of the convolution kernel are at the position The weight of Usually initialized randomly, if you need to change the weight , the loss function can be designed to update the weights according to the actual usage scenario; is the size of the convolution kernel, Represents the input image at position The pixel value of It is a bias term, which is added to the result of each convolution operation to adjust the value of the output feature map; the function activation layer uses the ReLU activation function.
[0087] The logic for fusing the features of the original color image with the corresponding infrared image features is: using the energy attribute fusion strategy to fuse the extracted features. The formula for the energy attribute fusion strategy is:
[0088]
[0089]
[0090] In the formula, and They represent the features of the original color image and the corresponding infrared image after being processed by the energy attribute fusion strategy. represents the modulation parameters, where , and Respectively represent the original color image and the corresponding infrared image in The pixel value at the point, and Respectively represent the characteristic values of the original color image and the corresponding infrared image;
[0091] The features of the original color image processed by the energy attribute fusion strategy and the corresponding infrared image are and The fusion is performed to obtain the fused image features, and the formula based on it is:
[0092]
[0093] In the formula, Represents the fused image features obtained by fusion.
[0094] Step 4: Using the obtained fused image features as input to a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input to the generator, and the generator performs image reconstruction to generate a fused image. The discriminator is used to identify the difference between the fused image and the input image features.
[0095] The multimodal medical image fusion network includes a generator and two discriminators. The generator is DenseNet. DenseNet directly connects the feature maps of all previous layers at each layer, which can better capture the feature associations between multimodal images. The discriminator compares the generated fusion image with the input source image to determine whether the fusion image is true or false. If it is determined to be false, it means that the feature distribution difference between the fusion image and the input source image does not meet the requirements. It is necessary to adjust the generator parameters and perform multiple iterations until the optimal generator is obtained.
[0096] Two symmetric discriminators and , which can balance the extraction and fusion of different modal features to a large extent. Specifically, The goal is to accurately classify the original color image and the fused images at each stage through continuous training and optimization. same, The task is to distinguish the infrared image and the fused images at different stages as much as possible.
[0097] The adversarial training process between the generator and the discriminator includes, first, training the discriminator based on the original color image and the current fusion image. Train so that It has a certain classification ability, that is, it can judge the original color image as true and the current fused image as false. The fused image is judged as false, which means that the feature distribution of the fused image and the original color image is quite different. It also means that the generator parameters need to be further optimized and updated to fit the pixel distribution law of the image, so that the fused image can retain more features from the original color image and the infrared image. Training methods and discriminators Similarly, the infrared image can be judged as true and the current fused image can be judged as false. and When both are output as true, the currently generated fused image is used as output.
[0098] Step 5: using the image lesion recognition model to identify abnormalities in the fused image, the abnormalities include polyps, ulcers and bleeding, sorting out different types of fused images, manually marking the polyps, ulcers and bleeding areas in the fused images, and using the manually marked polyps, ulcers and bleeding areas in the fused images as labels to train the image lesion recognition model, thereby obtaining an image lesion recognition model whose input is the fused image and whose output is the abnormal decision value of polyps, ulcers and bleeding;
[0099] The image lesion recognition model is an SVM model. Before labeling, the target and category of labeling need to be clearly defined: polyps are usually protrusions in the intestine; ulcers may appear as surface damage or sunken areas; bleeding may appear as abnormal red areas or blood accumulation; select appropriate image labeling tools for manual labeling. These tools usually have a graphical user interface and support image loading and labeling. Commonly used labeling tools include: LabelMe and VGG Image Annotator (VIA), use the labeling tool to load the endoscopic image to be labeled; select the appropriate labeling tool (such as rectangular box, polygon, free drawing, etc.) according to the category; polyps: you can use the polygon tool to accurately depict its outline, ulcers: use the free drawing tool to mark its shape or use the polygon tool, bleeding: you can use the rectangular box or polygon tool to mark the bleeding area.
[0100] For example, the abnormal decision value of a polyp is output, and its value indicates the degree to which the abnormal situation in the fused image belongs to a polyp situation. The larger the corresponding abnormal decision value, the closer the abnormal decision value is to the decision boundary, that is, the greater the probability that the abnormal situation is a polyp. The abnormal decision values of ulcers and bleeding are the same as those of polyps, and will not be elaborated here.
[0101] Step 6: Collect the endoscope original color image and the endoscope infrared image to be identified in the same time series, extract and fuse the features, generate the fused image to be identified through the multimodal medical image fusion network, input the fused image to be identified into the image lesion recognition model obtained by the method of establishing the image lesion recognition model, and obtain the abnormal decision value of the fused image to be identified;
[0102] The fused image is identified for abnormality using an image lesion recognition model, wherein the image lesion recognition model is an SVM model. The fused image to be identified is input into the image lesion recognition model obtained by the method for establishing the image lesion recognition model. The output of the SVM model is an abnormality decision value of the abnormality existing in the fused image to be identified, which belongs to the polyp, ulcer and bleeding area; wherein the abnormality decision value of the fused image to be identified is , For the The abnormal decision value vector of the fused image samples, They represent the decision values of the fused image from the decision boundaries of polyps, ulcers, and bleeding, respectively, where the abnormal decision value represents the distance from the decision boundary.
[0103] Step 7: Analyze the abnormal decision values of the fused image to be identified to generate a weighted decision value, compare the weighted decision value with the pre-set decision boundaries of different lesion types, and determine the type of abnormality that exists.
[0104] The formula for generating the weighted decision value based on the abnormal decision value analysis of the fused image to be identified is:
[0105]
[0106] In the formula, For the The weighted decision value of the fused image samples, is the weight coefficient matrix, Represents the weight coefficient matrix The transpose of , where the weight coefficient matrix The expression is:
[0107]
[0108] In the formula, and are the weight coefficients of polyp, ulcer and bleeding decisions respectively, among which, and are greater than 0 and ;
[0109] The weighted decision value is compared with the pre-set decision limits for different lesion types to determine the type of abnormality present. The logic is as follows: the decision limits for polyps, ulcers, and hemorrhages are and ,in ;
[0110] when When the abnormality is detected, the abnormality type is judged to be a polyp;
[0111] when When the abnormality is detected, it is judged that the abnormality type is ulcer;
[0112] when When the abnormality is detected, it is judged to be bleeding;
[0113] when , it is judged that there is no abnormality.
[0114] Among them, polyps and ulcers may be accompanied by bleeding. In some medical scenarios, polyp detection may be more important than ulcer detection, so ,In actual situations, the decision boundaries can be dynamically changed according to the specific ,medical scenarios.
[0115] See also Figure 2 The present invention also provides a digestive tract lesion recognition system based on endoscopic images, wherein the digestive tract lesion recognition system based on endoscopic images is used to execute the above-mentioned digestive tract lesion recognition method based on endoscopic images, comprising:
[0116] A sample image acquisition module, used to acquire an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers and bleeding;
[0117] An image preprocessing module, used to preprocess the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing;
[0118] A feature extraction and fusion module is used to extract the original color image and the infrared image discrimination features according to the endoscope original color image and the endoscope infrared image obtained after preprocessing, wherein the discrimination features include shape features, texture features and color features, and according to the extracted discrimination features, the features of the original color image are feature-fused with the corresponding infrared image features to obtain fused image features;
[0119] An image reconstruction module, used to use the obtained fused image features as input to a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input to the generator, and the generator performs image reconstruction to generate a fused image. The discriminator is used to identify the difference between the fused image and the input image features;
[0120] A lesion recognition model training module is used to construct an image lesion recognition model to perform abnormality recognition on a fused image, wherein the abnormalities include polyps, ulcers, and bleeding, sort out different types of fused images, manually mark the polyps, ulcers, and bleeding areas in the fused images, and train the image lesion recognition model using the manually marked polyps, ulcers, and bleeding areas in the fused images as labels, to obtain an image lesion recognition model with a fused image as input and a polyp, ulcer, and bleeding abnormality decision value as output;
[0121] The lesion abnormality analysis module is used to collect the endoscope original color images and the endoscope infrared images to be identified in the same time series, extract and fuse the features, generate the fused image to be identified through the multimodal medical image fusion network, input the fused image to be identified into the image lesion recognition model obtained by the method of establishing the image lesion recognition model, and obtain the abnormal decision value of the fused image to be identified;
[0122] The lesion abnormality type determination module is used to analyze the abnormal decision value of the fused image to be identified to generate a weighted decision value, compare the weighted decision value with the pre-set decision boundaries of different lesion types, and determine the type of abnormality that exists.
[0123] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.
[0124] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.
[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0126] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A method for identifying digestive tract lesions based on endoscopic images, characterized in that: The specific steps include: Collecting an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers, and bleeding; Preprocessing the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing; Extracting discriminative features of the original color image and the infrared image according to the endoscope original color image and the endoscope infrared image obtained after preprocessing, wherein the discriminative features include shape features, texture features and color features, and performing feature fusion on the features of the original color image and the corresponding infrared image features according to the extracted discriminative features to obtain fused image features; The obtained fused image features are used as input of a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input into the generator, the generator performs image reconstruction to generate a fused image, and the discriminator is used to identify the difference between the fused image and the input image features; An abnormality recognition is performed on the fused image using an image lesion recognition model, wherein the abnormality includes polyps, ulcers, and bleeding, different types of fused images are sorted, polyps, ulcers, and bleeding areas in the fused image are manually marked, and the image lesion recognition model is trained using the manually marked polyps, ulcers, and bleeding areas in the fused image as labels, to obtain an image lesion recognition model with a fused image as input and a decision value of polyps, ulcers, and bleeding abnormalities as output; The endoscope original color images and the endoscope infrared images to be identified in the same time series are collected, and after feature extraction and fusion, a fused image to be identified is generated through a multimodal medical image fusion network, and the fused image to be identified is input into the image lesion recognition model obtained by the method of establishing the image lesion recognition model to obtain the abnormal decision value of the fused image to be identified; The abnormal decision values of the fused image to be identified are analyzed to generate weighted decision values, which are compared with pre-set decision boundaries for different lesion types to determine the type of abnormality that exists.
2. The method for identifying digestive tract lesions based on endoscopic images according to claim 1, characterized in that: The endoscopic original color image and the endoscopic infrared image of the same target individual are collected, wherein the method for collecting the endoscopic original color image and the endoscopic infrared image of the same target individual is: use a thermal infrared camera and an endoscopic camera to respectively perform video monitoring of the digestive tract of the same target individual, and extract frame images from the collected videos as sample images.
3. The method for identifying digestive tract lesions based on endoscopic images according to claim 2, characterized in that: The collected endoscope original color image and endoscope infrared image are preprocessed respectively, and the preprocessing includes image denoising and enhancement processing. The logic of the preprocessing is as follows: The denoising method of wavelet transform is used to denoise the original color image of the endoscope and the infrared image of the endoscope. The specific steps of the denoising method of wavelet transform include: decomposing the original color image of the endoscope and the infrared image of the endoscope through wavelet transform to obtain the wavelet coefficients of the image at different scales and directions; thresholding the wavelet coefficients to set the low-amplitude wavelet coefficients to zero and retain the high-amplitude wavelet coefficients; inversely transforming the wavelet coefficients after thresholding, reconstructing the processed coefficients into an image, and completing the denoising of the original color image of the endoscope and the infrared image of the endoscope; Bilateral filtering is used to enhance the details and further reduce the noise of the endoscope original color image and the endoscope infrared image. The formula based on the specific filtering transformation is: In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is: In the formula, is the coordinate vector in the image coordinate space, is the coordinate vector The gray value of and They are The standard deviation of .
4. The method for identifying digestive tract lesions based on endoscopic images according to claim 3, characterized in that: According to the original color image and the infrared image of the endoscope obtained after preprocessing, the discriminative features of the original color image and the infrared image are extracted, and the discriminative features include shape features, texture features and color features, wherein the method for extracting the discriminative features of the image is based on a convolutional neural network. The logic behind its feature extraction is: Collect a dataset of endoscope original color images and endoscope infrared images containing various shapes, textures, and color features, and use the shape, texture, and color features of polyps, ulcers, and hemorrhages as labels for training; The input layer of the convolutional neural network receives the original color image and infrared image of the endoscope obtained after preprocessing. The convolution kernel is used in the convolution layer to perform a convolution operation on the input image to extract the discriminative features. The formula based on the convolution operation is: In the formula, The convolution result feature map is at position The value of is the convolution kernel at position The weight of is the size of the convolution kernel, Represents the input image at position The pixel value of It is a bias term, which is added to the result of each convolution operation to adjust the value of the output feature map; the function activation layer uses the ReLU activation function.
5. The method for identifying digestive tract lesions based on endoscopic images according to claim 4, characterized in that: According to the extracted discriminative features, the features of the original color image are fused with the features of the corresponding infrared image to obtain the fused image features. The logic for fusion of the features of the original color image with the features of the corresponding infrared image is: the energy attribute fusion strategy is used to fuse the extracted features. The formula for the energy attribute fusion strategy is: In the formula, and They represent the features of the original color image and the corresponding infrared image after being processed by the energy attribute fusion strategy. represents the modulation parameters, where , and Respectively represent the original color image and the corresponding infrared image in The pixel value at the point, and Respectively represent the characteristic values of the original color image and the corresponding infrared image; The original color image processed by the energy attribute fusion strategy is combined with the corresponding infrared image features. and The fusion image features are obtained by fusion, and the formula is as follows: In the formula, Represents the fused image features obtained by fusion.
6. The method for identifying digestive tract lesions based on endoscopic images according to claim 5, characterized in that: The obtained fused image features are used as the input of a multimodal medical image fusion network, which includes a generator and two discriminators. The generator is DenseNet, and the discriminator compares the generated fused image with the input source image to determine whether the fused image is true or false. If it is determined to be false, it means that the difference in feature distribution between the fused image and the input source image does not meet the requirements, and the generator parameters need to be adjusted and iterated multiple times until the optimal generator is obtained.
7. The method for identifying digestive tract lesions based on endoscopic images according to claim 1, characterized in that: The fused image is identified for abnormality using an image lesion recognition model, wherein the image lesion recognition model is an SVM model. The fused image to be identified is input into the image lesion recognition model obtained by the method for establishing the image lesion recognition model. The output of the SVM model is an abnormality decision value of the abnormality existing in the fused image to be identified, which belongs to the polyp, ulcer and bleeding area; wherein the abnormality decision value of the fused image to be identified is , For the The abnormal decision value vector of the fused image samples, Represent the decision values of the fused image from the decision boundaries of polyp, ulcer and bleeding, respectively.
8. The method for identifying digestive tract lesions based on endoscopic images according to claim 7, characterized in that: The formula for generating the weighted decision value based on the abnormal decision value analysis of the fused image to be identified is: In the formula, For the The weighted decision value of the fused image samples, is the weight coefficient matrix, Represents the weight coefficient matrix The transpose of , where the weight coefficient matrix The expression is: In the formula, and are the weight coefficients of polyp, ulcer and bleeding decisions respectively, among which, and are greater than 0 and ; The weighted decision value is compared with the pre-set decision limits for different lesion types to determine the type of abnormality present. The logic is as follows: the decision limits for polyps, ulcers, and hemorrhages are and ,in ; when When the abnormality is detected, the abnormality type is judged to be a polyp; when When the abnormality is detected, it is judged to be ulcer; when When the abnormality is detected, it is judged to be bleeding; when , it is judged that there is no abnormality.
9. A digestive tract lesion recognition system based on endoscopic images, characterized in that: The digestive tract lesion recognition system based on endoscopic images is used to execute the digestive tract lesion recognition system method based on endoscopic images according to any one of claims 1 to 8, comprising: A sample image acquisition module, used to acquire an endoscopic original color image and an endoscopic infrared image of the same target individual, wherein the endoscopic original color image and the endoscopic infrared image are abnormal digestive tract images, and the abnormal digestive tract images include polyps, ulcers and bleeding; An image preprocessing module, used to preprocess the collected endoscope original color image and endoscope infrared image respectively, wherein the preprocessing includes image denoising and enhancement processing; A feature extraction and fusion module is used to extract the original color image and the infrared image discrimination features according to the endoscope original color image and the endoscope infrared image obtained after preprocessing, wherein the discrimination features include shape features, texture features and color features, and according to the extracted discrimination features, the features of the original color image are feature-fused with the corresponding infrared image features to obtain fused image features; An image reconstruction module, used to use the obtained fused image features as input to a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a generator and two discriminators. The fused image features are input to the generator, and the generator performs image reconstruction to generate a fused image. The discriminator is used to identify the difference between the fused image and the input image features; A lesion recognition model training module is used to construct an image lesion recognition model to perform abnormality recognition on a fused image, wherein the abnormalities include polyps, ulcers, and bleeding, sort out different types of fused images, manually mark the polyps, ulcers, and bleeding areas in the fused images, and train the image lesion recognition model using the manually marked polyps, ulcers, and bleeding areas in the fused images as labels, to obtain an image lesion recognition model with a fused image as input and a polyp, ulcer, and bleeding abnormality decision value as output; The lesion abnormality analysis module is used to collect the endoscope original color images and the endoscope infrared images to be identified in the same time series, extract and fuse the features, generate the fused image to be identified through the multimodal medical image fusion network, input the fused image to be identified into the image lesion recognition model obtained by the method of establishing the image lesion recognition model, and obtain the abnormal decision value of the fused image to be identified; The lesion abnormality type determination module is used to analyze the abnormal decision value of the fused image to be identified to generate a weighted decision value, compare the weighted decision value with the pre-set decision boundaries of different lesion types, and determine the type of abnormality that exists.
Citation Information
Patent Citations
Medical image fusion method based on gradient filtering
CN115100093A
Computer-aided reference system and method for fusing multi-modal mammary gland image
CN116958059A
Facial pain expression category generation method based on emotion migration
CN118072371A
