Digestive endoscopy auxiliary system based on artificial intelligence

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an artificial intelligence-based digestive endoscopy auxiliary system, which comprises a data acquisition and preprocessing module, a feature extraction module, a feature fusion module, a disease analysis module and a report generation module, the data acquisition and preprocessing module is used for acquiring a digestive endoscopy image and preprocessing the digestive endoscopy image; the digestive endoscopy image comprises a white light image and a Raman spectrum image; the method comprises the following steps: performing feature fusion by combining a Raman spectrum image to obtain a fusion feature vector, realizing multi-scale and multi-dimensional analysis on a digestive endoscopy image, and outputting an analysis result, namely, on the basis of the prior art, performing fusion by utilizing a feature vector extracted from a white light image and a feature vector extracted from the Raman spectrum image to obtain a fusion feature vector; according to the method, multi-scale and multi-dimensional analysis of the digestive endoscopy image is realized, and the recognition accuracy of each stage of lesion is improved, so that more accurate pathological analysis and auxiliary diagnosis are carried out, and missed diagnosis and misdiagnosis are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular is a digestive endoscopy auxiliary system based on artificial intelligence. Background Art

[0002] Digestive endoscopy has important clinical significance in diagnosis, evaluation of lesions, treatment and follow-up observation, and is an indispensable tool in the management of gastrointestinal diseases. In clinical diagnosis, doctors first need to make a judgment on the presence and type of lesions based on a large number of images produced by endoscopy within a limited time; however, due to the large differences in the diagnostic level of endoscopists, there is a problem of incomplete observation of the anatomical parts of the digestive tract, which leads to missed diagnosis of lesions. It is time-consuming and economically expensive. The differences between different types of lesions are subtle, difficult to distinguish, and easy to misdiagnose. There are also large differences between the same lesions. The larger the lesion area, the coexistence of multiple scales, and the unclear characteristics of small lesions, which are easy to miss.

[0003] Prior art, such as CN117854705A discloses an upper gastrointestinal multi-lesion multi-task intelligent diagnosis method, device, equipment and medium, the method comprising: inputting an upper gastrointestinal endoscopic examination image into an upper gastrointestinal multi-task classification model to generate fusion features corresponding to the upper gastrointestinal endoscopic examination image; processing the fusion features through the upper gastrointestinal multi-task classification model to obtain the image anatomical part and lesion image corresponding to the upper gastrointestinal endoscopic examination image; inputting the lesion image into an upper gastrointestinal multi-lesion segmentation model to generate a lesion area target segmentation result; and performing diagnosis based on the image anatomical part and lesion area target segmentation result corresponding to the upper gastrointestinal endoscopic examination image.

[0004] In the above-mentioned prior art, considering the existence of multi-scale characteristics of diseases, multi-scale feature verification methods are integrated into the process of processing digestive endoscopy images to perform symptom examination. However, based on practical applications, since the above-mentioned digestive endoscopy images are white light images, they are mainly used to provide morphological information of the disease, including color, texture and surface structure. However, some gastrointestinal diseases do not show obvious shape, color and texture in the early stage, but more changes in the internal molecular structure. Therefore, even if multi-scale feature verification is performed to verify the existence of the disease, the white light images of early symptoms cannot reflect obvious changes in shape, color and texture, which may cause the early symptoms to be ignored and lead to misdiagnosis.

[0005] To this end, the present invention provides a digestive endoscopy auxiliary system based on artificial intelligence. Summary of the invention

[0006] In order to make up for the deficiencies of the prior art, at least one technical problem raised in the background technology is solved.

[0007] The technical solution adopted by the present invention to solve the technical problem is: the digestive endoscopy auxiliary system based on artificial intelligence described in the present invention comprises:

[0008] A data acquisition and preprocessing module, used for acquiring and preprocessing digestive endoscopic images; the digestive endoscopic images include white light images and Raman spectroscopy images;

[0009] A feature extraction module is used to extract features from the preprocessed digestive endoscopy image, and obtain a first feature vector corresponding to the white light image and a second feature vector corresponding to the Raman spectrum image;

[0010] A feature fusion module is used to fuse the extracted first feature vector and the second feature vector to obtain a multi-dimensional fused feature vector;

[0011] A symptom analysis module, based on a symptom classifier, inputs the fused feature vector into the symptom classifier, determines the probability distribution and severity level of the symptom based on the symptom classifier, and outputs the probability distribution and severity level of the symptom as an analysis result;

[0012] The report generation module generates auxiliary diagnosis reports based on the analysis results.

[0013] Preferably, the preprocessing method of the digestive endoscopy image is:

[0014] De-noising the white light image, based on any one of the mean filter, median filter, and bilateral filter;

[0015] Perform illumination correction on white light images, using either histogram equalization or Retinex algorithm to correct image illumination;

[0016] Perform image segmentation on the white light image based on the image segmentation algorithm to obtain the local feature area and background area;

[0017] The Raman spectrum image is denoised and the data is normalized, and noise is removed based on either baseline correction or wavelet transform.

[0018] Preferably, the method of removing noise from the Raman spectrum image using wavelet transform is:

[0019] Based on the wavelet basis function, the Raman spectrum image is decomposed according to the preset decomposition layer number, and the decomposed image is classified into low-frequency components and high-frequency components;

[0020] Remove noise from the high-frequency components of the image based on the threshold method;

[0021] The denoised high-frequency components are combined with the low-frequency components to obtain a combined image;

[0022] The combined image is reconstructed using the selected wavelet basis function and decomposition layer number to obtain the denoised Raman spectrum image.

[0023] Preferably, the feature extraction method of the white light image is:

[0024] The white light image is processed through the pre-trained CNN model to output multiple feature maps;

[0025] Select the maximum pixel value in each feature map;

[0026] Global max pooling is applied after the last convolutional layer to form a global feature vector;

[0027] Based on the combination of convolutional layers and pooling layers, local feature vectors are extracted;

[0028] Concatenate the local eigenvector and the global eigenvector in the same dimension to obtain the first eigenvector;

[0029] The feature extraction method of the Raman spectrum image is:

[0030] Convert the denoised Raman spectrum image into a data set;

[0031] Process the data set based on the principal component analysis algorithm to obtain eigenvalues ​​and eigenvectors;

[0032] Sort by eigenvalue size and obtain the principal component whose sorting is greater than the median;

[0033] The selected principal components are combined to obtain the second eigenvector.

[0034] Preferably, the method for obtaining the multi-dimensional combined fusion feature vector is:

[0035] Obtain a first eigenvector f1 and a second eigenvector f2;

[0036] Evaluate the importance of the first eigenvector f1 and the second eigenvector f2 to determine weight factors w1 and w2;

[0037] According to the formula:

[0038] F=w1×f1+w2×f2

[0039] Among them, F is the fused feature vector, w1 is the weight factor of the first feature vector, and w2 is the weight factor of the second feature vector.

[0040] Preferably, the method for obtaining the multi-dimensional combined fusion feature vector also includes:

[0041] Obtain the dimension d1 of the first eigenvector and the dimension d2 of the second eigenvector;

[0042] Determine whether d1 is equal to d2;

[0043] If d1 = d2, the weighted fusion of the first eigenvector f1 and the second eigenvector f2 can be performed;

[0044] If d1 ≠ d2, the dimensions are matched based on the principal component analysis algorithm, including:

[0045] If d1 < d2, pad the dimension d1 of the first eigenvector until d1 = d2;

[0046] If d1 > d2, pad the dimension d2 of the second eigenvector until d1 = d2.

[0047] Preferably, the method for determining the probability distribution and severity level of a disease based on a disease classifier is as follows:

[0048] Obtain the fused eigenvector;

[0049] Input the fused eigenvector into the trained disease classifier;

[0050] Based on the fully connected neural network, perform forward propagation on the fused eigenvector;

[0051] Based on the softmax activation function, output the probability distribution map of each class for the fused eigenvector, and at the same time, based on the linear activation function, output a continuous value;

[0052] Select the class with the highest probability in the probability distribution map as the most likely disease;

[0053] Compare the probability of the most likely disease with a threshold. When the probability is greater than the threshold, it is determined that the disease exists, otherwise it is determined that the disease does not exist;

[0054] Output the severity level of the disease based on the continuous value and the preset continuous value - degree comparison table;

[0055] Combine and output the disease and the corresponding severity level as the analysis result.

[0056] Preferably, the method for determining the probability distribution and severity level of a disease based on a disease classifier further includes:

[0057] Determine the corresponding white light image according to the fused eigenvector of the determined disease;

[0058] Process the white light image based on the edge detection algorithm to segment the disease area contour and the background contour;

[0059] Overlay a semi - transparent color layer on the segmented disease area contour to obtain a disease - prominent image.

[0060] Preferably, it further comprises a biopsy area determination module, wherein the biopsy area determination module is used to determine the biopsy position according to the analysis result;

[0061] The method for determining the biopsy location is:

[0062] Obtain the probability distribution map of each category output by the lesion classifier;

[0063] Select the region corresponding to the maximum probability of any lesion;

[0064] The center point of the area was used as the intended biopsy location;

[0065] The planned biopsy location is displayed and reviewed to determine whether the planned biopsy location meets clinical requirements, and the determined biopsy location is output.

[0066] Preferably, the training method of the disease classifier is:

[0067] Collect digestive endoscopy images, including white light images and corresponding Raman spectroscopy images;

[0068] Preprocess the digestive endoscopy images and complete feature extraction;

[0069] The features are fused, and the fused feature vector is labeled according to the disease corresponding to the digestive endoscopy image to form a labeled fused feature vector;

[0070] Based on a part of the labeled fused feature vectors, the original classifier built based on the SVM algorithm is trained;

[0071] Based on another part of the labeled fusion feature vectors as a validation set, the classifier is cross-validated, the parameters of the model are optimized, and the disease classifier is obtained.

[0072] The beneficial effects of the present invention are as follows:

[0073] The artificial intelligence-based digestive endoscopy auxiliary system described in the present invention combines Raman spectral images for feature fusion, obtains fused feature vectors, realizes multi-scale and multi-dimensional analysis of digestive endoscopy images, and outputs analysis results. That is to say, on the basis of the existing technology, the feature vectors extracted from white light images and the feature vectors extracted from Raman spectral images are fused to realize multi-scale and multi-dimensional analysis of digestive endoscopy images, improve the accuracy of recognition of various stages of lesions, thereby performing more accurate pathological analysis and auxiliary diagnosis, and avoiding missed diagnosis and misdiagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present invention will be further described below in conjunction with the accompanying drawings.

[0075] Figure 1 It is a perspective view of the present invention. DETAILED DESCRIPTION

[0076] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0077] like Figure 1 As shown, an artificial intelligence-based digestive endoscopy auxiliary system described in an embodiment of the present invention includes a data acquisition and preprocessing module, a feature extraction module, a feature fusion module, a symptom analysis module and a report generation module; the data acquisition and preprocessing module is used to acquire and preprocess digestive endoscopy images; the digestive endoscopy images include white light images and Raman spectrum images; the feature extraction module is used to extract features from the preprocessed digestive endoscopy images, and obtain a first feature vector corresponding to the white light image and a second feature vector corresponding to the Raman spectrum image respectively; the feature fusion module is used to fuse the extracted first feature vector and the second feature vector to obtain a multi-dimensional fused feature vector; the symptom analysis module inputs the fused feature vector into the symptom classifier based on the symptom classifier, determines the probability distribution and severity level of the symptom based on the symptom classifier, and outputs the probability distribution and severity level of the symptom as the analysis result; the report generation module generates an auxiliary diagnosis report based on the analysis result.

[0078] In the above-mentioned prior art, considering the existence of multi-scale characteristics of diseases, multi-scale feature verification methods are integrated into the process of processing digestive endoscopy images to perform symptom examination. However, based on practical applications, since the above-mentioned digestive endoscopy images are white light images, they are mainly used to provide morphological information of the disease, including color, texture and surface structure. However, some upper gastrointestinal diseases do not show obvious shape, color and texture in the early stage, but more changes in the internal molecular structure. Therefore, even if multi-scale feature verification is performed to verify the existence of the disease, the white light images of early symptoms cannot reflect obvious changes in shape, color and texture, which may cause the early symptoms to be ignored and lead to misdiagnosis.

[0079] Based on the above, during the digestive endoscopy, in addition to the white light image, the Raman spectrum image can also be obtained. Raman spectrum is an inelastic scattering spectrum, which can obtain fingerprint information such as the molecular structure, vibration mode, functional group, etc. of the substance. It does not require a complicated sample preparation process, has little water interference in biological tissues, and is very sensitive to changes in the biochemical components of proteins, nucleic acids, phospholipids and sugars. It can be widely used in the analysis of biological molecular structures. It is a non-destructive, fast and highly sensitive optical detection technology. In one embodiment of the present invention, by simultaneously acquiring the white light image and the Raman spectrum image, the white light image and the Raman spectrum image are preprocessed respectively to obtain the first eigenvector and the second eigenvector, and then the first eigenvector and the second eigenvector are fused to obtain the fused eigenvector. It is worth noting that in the prior art, based on the means of multi-scale feature verification, the lesion characteristics can be more comprehensively captured to improve the accuracy and reliability of diagnosis. However, since the white light image of the early stage of the disease cannot reflect the obvious scale characteristics, it may be ignored and misdiagnosed. Here, the Raman spectrum image is combined for feature fusion to obtain the fused feature vector, realize the multi-scale and multi-dimensional analysis of the digestive endoscopy image, and output the analysis result. That is to say, on the basis of the prior art, the feature vector extracted by the white light image and the feature vector extracted by the Raman spectrum image are fused to realize the multi-scale and multi-dimensional analysis of the digestive endoscopy image, improve the recognition accuracy of each stage of the lesion, so as to perform more accurate pathological analysis and auxiliary diagnosis, and avoid missed diagnosis and misdiagnosis.

[0080] Demonstratively, digestive endoscopy images collected in real time from patients are obtained, including white light images and Raman spectrum images. Based on the preprocessing of the white light images and the Raman spectrum images, a first eigenvector and a second eigenvector are obtained. Subsequently, the first eigenvector and the second eigenvector are fused based on a feature fusion module. Based on constraints under the same dimension, a fused feature vector is obtained. Finally, the fused feature vector is input into a lesion classifier. Based on the expression of the fused feature vector, the probability distribution and severity level of the lesion can be determined. Based on the threshold judgment, it is determined whether the lesion actually exists, thereby judging whether the patient has symptoms and the corresponding severity level. From the perspective of practical application, in one embodiment, based on the collaborative processing of white light images and Raman spectrum images, multi-scale and multi-dimensional analysis of digestive endoscopy images is realized to avoid the problem of missed diagnosis caused by the unclear morphological structure of the lesion in the early stage. Based on multi-scale and multi-dimensional image processing, doctors can be assisted in accurately analyzing and diagnosing possible symptoms of patients.

[0081] In one embodiment, the preprocessing method of the digestive endoscopy image is:

[0082] De-noising the white light image, based on any one of the mean filter, median filter, and bilateral filter;

[0083] Perform illumination correction on white light images, using either histogram equalization or Retinex algorithm to correct image illumination;

[0084] Perform image segmentation on the white light image based on the image segmentation algorithm to obtain the local feature area and background area;

[0085] The Raman spectrum image is denoised and the data is normalized, and noise is removed based on either baseline correction or wavelet transform.

[0086] In one embodiment, the method of removing noise from the Raman spectrum image using wavelet transform is:

[0087] Based on the wavelet basis function, the Raman spectrum image is decomposed according to the preset decomposition layer number, and the decomposed image is classified into low-frequency components and high-frequency components;

[0088] Remove noise from the high-frequency components of the image based on the threshold method;

[0089] The denoised high-frequency components are combined with the low-frequency components to obtain a combined image;

[0090] The combined image is reconstructed using the selected wavelet basis function and decomposition layer number to obtain the denoised Raman spectrum image.

[0091] Raman spectral image preprocessing is a prerequisite for image analysis and diagnosis. Based on the preprocessed Raman spectral image, support can be provided for subsequent feature extraction. In one embodiment of the present invention, based on the selected wavelet basis function, such as Daubechies wavelet, Morlet wavelet, etc., the wavelet basis function determines the characteristics of wavelet decomposition. The appropriate wavelet basis function can effectively improve the accuracy and computational efficiency of decomposition. In addition, the number of decomposition layers needs to be set. In theory, the more decomposition layers there are, the finer the decomposed frequency components are, but the amount of calculation also increases. Therefore, the appropriate wavelet basis function and the number of decomposition layers can decompose the Raman spectral image into components of different frequencies, including high-frequency components and low-frequency components; by setting a threshold, the pixels with intensity values ​​lower than the threshold in the high-frequency component are set to 0, that is, noise removal is completed; then the denoised low-frequency component is combined with the high-frequency component to obtain a combined image, and then based on the selected wavelet basis function and the number of decomposition layers, the combined image is reconstructed to obtain a denoised Raman spectral image. Based on the above preprocessing, support can be provided for subsequent feature extraction of the Raman spectral image.

[0092] In one embodiment, the feature extraction method of the white light image is:

[0093] The white light image is processed through the pre-trained CNN model to output multiple feature maps;

[0094] Select the maximum pixel value in each feature map;

[0095] Global max pooling is applied after the last convolutional layer to form a global feature vector;

[0096] Based on the combination of convolutional layers and pooling layers, local feature vectors are extracted;

[0097] Concatenate the local eigenvector and the global eigenvector in the same dimension to obtain the first eigenvector;

[0098] The feature extraction method of the Raman spectrum image is:

[0099] Convert the denoised Raman spectrum image into a data set;

[0100] Process the data set based on the principal component analysis algorithm to obtain eigenvalues ​​and eigenvectors;

[0101] Sort by eigenvalue size and obtain the principal component whose sorting is greater than the median;

[0102] The selected principal components are combined to obtain the second eigenvector.

[0103] For feature extraction of white light images, reference can be made to the prior art, that is, extraction of global feature vectors and local feature vectors are performed respectively, wherein the extraction of global feature vectors is used to summarize the overall information of white light images, including for image classification. As is known to all, a variety of diseases may be reflected in digestive endoscopy images, including gastric cancer, gastric mucosal dysplasia, gastrointestinal metaplasia, gastritis, gastric polyps, gastric ulcers, other gastric lesions, etc., and the information expressed by the white light images corresponding to each disease is not exactly the same. The extraction of global feature vectors is conducive to the classification and comparison of diseases. In addition, the extraction of local feature vectors allows users to identify specific targets or areas in the image and provide image detail information, such as texture features, shape features, color features, edge features, etc., wherein texture features are based on local binary patterns to describe texture information, and shape features are based on local binary patterns to describe texture information. The feature is based on the contour detection algorithm to identify the target contour, including calculating the geometric features of the contour, such as area, perimeter, rectangularity, circularity and invariant moment, while the color feature is based on the color histogram to show the distribution of different colors in the image. The edge feature is based on the edge monitoring algorithm to identify the edge of the target, which is similar to the shape feature. Based on the above, by extracting features from the white light image, and then splicing the local feature vector with the global feature vector, that is, combining the local feature vector with the global feature vector in dimension to obtain a new, longer feature vector, the first feature vector representing the white light image can be obtained. It is worth noting that the spliced ​​first feature vector contains all the information of the original local features and global features. In the splicing process, all features are assumed to be equally important. In addition, splicing will increase the dimension of the first feature vector, for example:

[0104] If the dimension of the local feature vector is di and the dimension of the global feature vector is dj, then the dimension of the first feature vector is di+dj;

[0105] Among them, based on the trained CNN model, after the preprocessed white light image is input, since the CNN model includes multiple convolution layers, and each convolution layer contains multiple convolution kernels, when the convolution kernel slides on the white light image, a dot product operation can be performed on each position to generate a feature map, and each convolution kernel generates a feature map, and multiple feature maps are obtained. The feature map is a two-dimensional or three-dimensional matrix, which records the feature information extracted when the convolution kernel slides on the white light image.

[0106] Based on the above, after the feature extraction of the white light image, the feature extraction of the Raman spectrum image is also required. In one embodiment of the present invention, the Raman spectrum image is preferentially preprocessed, including denoising and normalization, to ensure that the spectral data of each pixel in the image can be on the same scale, and then the spectrum image is converted into a high-dimensional vector, in which each element represents the spectral information of a pixel. If the image size is M×N, each pixel has P spectral bands, and the image will be reorganized into a three-dimensional matrix of M×N×P, which can be regarded as a P-dimensional data set with M×N samples. Based on the principal component analysis calculation, the P-dimensional data set is standardized and then calculated. The covariance matrix of all samples is decomposed to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​are sorted and the principal components corresponding to the eigenvectors are selected according to the eigenvectors whose sorting is greater than the median. These principal components can explain most of the variance in the Raman spectrum image. The second eigenvector is obtained by combining the selected principal components. The eigenvalue indicates the variance of the data along the direction of the corresponding eigenvector. A large eigenvalue means that the data has a large variance in the direction of the eigenvector, that is, the data changes significantly in this direction. The eigenvector defines the new direction of data change, that is, the principal component. Each eigenvector corresponds to a possible direction of data change.

[0107] In one embodiment, the method for obtaining the multi-dimensional combined fusion feature vector is:

[0108] Obtain a first eigenvector f1 and a second eigenvector f2;

[0109] Evaluate the importance of the first eigenvector f1 and the second eigenvector f2 to determine weight factors w1 and w2;

[0110] According to the formula:

[0111] F=w1×f1+w2×f2

[0112] Among them, F is the fused feature vector, w1 is the weight factor of the first feature vector, and w2 is the weight factor of the second feature vector.

[0113] After obtaining the first feature vector corresponding to the white light image and the second feature vector corresponding to the Raman spectrum image, it is necessary to fuse these two feature vectors to achieve the purpose of multi-dimension. Based on this, in an embodiment of the present invention, the first feature vector f1 and the second feature vector f2 are obtained, and the weight factor is determined according to the importance evaluation. The fused feature vector is calculated according to the formula and expressed by F; among them, it should be noted that the weight factor is allocated according to the importance of the white light image and the Raman spectrum image for the diagnosis task. It can be understood that an initial weight is set, for example, w1 = 0.5, w2 = 0.5; which means that the initial weights of the first feature vector and the second feature vector corresponding to the white light image and the Raman spectrum image are equal. A weight grid is defined and expressed as:

[0114] weights_grid = [(w1, 1 - w1) for w1 in np.linspace(0, 1, 21)]

[0115] Among them, np.linspace(0, 1, 21) generates 21 equally spaced values from 0 to 1, representing possible weight assignments;

[0116] Perform cross-validation on each weight combination, and select the weight combination with the highest average accuracy as the weight factors w1 and w2 according to the cross-validation results;

[0117] In addition, when calculating the fused feature vector, it is necessary to ensure that the dimensions of the first feature vector f1 and the second feature vector f2 are equal.

[0118] In one embodiment, the method for obtaining the fused feature vector of the multi-dimension combination further includes:

[0119] Obtain the dimension d1 of the first feature vector and the dimension d2 of the second feature vector;

[0120] Judge whether d1 and d2 are equal;

[0121] If d1 = d2, then the weighted fusion of the first feature vector f1 and the second feature vector f2 can be performed;

[0122] If d1 ≠ d2, then the dimensions are matched based on the principal component analysis algorithm, including:

[0123] If d1 < d2, then fill the dimension d1 of the first feature vector until d1 = d2;

[0124] If d1 > d2, then fill the dimension d2 of the second feature vector until d1 = d2.

[0125] When the dimensions of the first eigenvector f1 and the second eigenvector f2 are not equal, i.e., d1≠d2, since weighted calculation requires that each eigenvector be operated in the same dimensional space, therefore, it is necessary to analyze and process the dimension d1 of the first eigenvector and the dimension d2 of the second eigenvector. In an embodiment of the present invention, if d1<d2, then the dimension d1 of the first eigenvector is filled until d1 = d2; if d1>d2, then the dimension d2 of the second eigenvector is filled until d1 = d2. Through dimension filling, it is possible to effectively avoid the influence of different data sources on the feature fusion of the first eigenvector f1 and the second eigenvector f2 corresponding to the Raman spectrum image and the white light image due to differences in expression methods; Exemplary:

[0126] Since the first eigenvector is obtained by splicing the local eigenvector and the global eigenvector, the dimension of the first eigenvector may not be equal to the dimension of the second eigenvector, i.e., d1≠d2. In this case, it is impossible to directly obtain the fused eigenvector by weighted fusion. It is also necessary to detect the dimension distribution of the eigenvectors and fill the dimension of the first eigenvector or the second eigenvector with a relatively lower dimension, so that the dimension of the first eigenvector is equal to the dimension of the second eigenvector, and then weighted fusion is performed to obtain the fused eigenvector.

[0127] In one embodiment, the method for determining the probability distribution and severity level of a disease based on a disease classifier is:

[0128] Obtain the fused eigenvector;

[0129] Input the fused eigenvector into the trained disease classifier;

[0130] Based on the fully connected neural network, perform forward propagation on the fused eigenvector;

[0131] Based on the softmax activation function, output the probability distribution diagram of each category for the fused eigenvector, and at the same time, based on the linear activation function, output a continuous value;

[0132] Select the category with the highest probability in the probability distribution diagram as the most likely disease;

[0133] Compare the probability of the most likely disease with a threshold. When the probability is greater than the threshold, it is determined that the disease exists, otherwise it is determined that the disease does not exist;

[0134] Output the severity level of the disease based on the continuous value and the preset continuous value - degree comparison table;

[0135] Combine and output the disease and the corresponding severity level as the analysis result.

[0136] After obtaining the fused feature vector, it is also necessary to analyze the content expressed by the fused feature vector to obtain the analysis result. In one embodiment of the present invention, a disease classifier is pre-trained, and based on inputting the fused feature vector into the trained disease classifier, a prediction or analysis result can be output. Based on the prediction or analysis result, it can be used to characterize whether a disease exists and the severity of the disease. Specifically, after the fused feature vector is input, the disease classifier can compare the similarity between the fused feature vector and the feature vector corresponding to the known disease, and output the probability distribution of multiple disease categories based on the similarity calculation result. It can be understood that, assuming that the fused feature vector is input into the disease classifier, the classifier outputs the following probability distribution:

[0137] Normal: 10%

[0138] Gastritis: 20%

[0139] Ulcers: 50%

[0140] Cancer: 20%

[0141] Based on the above, the probability distribution output by the classifier has the maximum probability corresponding to ulcer. The probability corresponding to ulcer is compared with the threshold. Here, the threshold is set to 40%. Based on this, it can be judged that the disease exists and the disease is the most likely disease, that is, ulcer. In addition, the classifier not only outputs the probability distribution corresponding to the disease category, but also outputs the continuous value corresponding to the most likely disease based on the linear activation function. The continuous value can be understood as a value between 0 and 100, where 0 represents no disease and 100 represents a very serious disease. The severity level of the disease is output according to the preset continuous value-degree comparison table. Assuming that the continuous value is 50, according to the continuous value-degree comparison table, it is as follows:

[0142] Continuous Values degree grade 0-20 micro 1 20-60 light 2 60-85 middle 3 85-100 Heavy 4

[0143] Based on the above table, it can be obtained that when the continuous value is 50, the severity of the corresponding disease is mild and the severity level is level 2;

[0144] Based on the above, by inputting the fusion features into the disease classifier, the fusion features can be analyzed and predicted, and then the analysis results can be output, including whether the disease exists and the severity of the disease. For possible early symptoms, even if they are not easy to detect on the white light image, they can be reflected in the spectral features, that is, by inputting the fusion features into the disease classifier, the existence of the disease can be detected more accurately and earlier, thereby avoiding missed diagnosis and misdiagnosis caused by single imaging and single analysis; it is worth noting that after the analysis results are output, the doctor needs to analyze the analysis results and the corresponding digestive endoscopy images to determine whether the analysis results are true; when the analysis results output by the lesion classifier are inconsistent with the doctor's analysis results, it means that the lesion classifier needs to be optimized. In one embodiment of the present invention, it is possible to consider periodically optimizing the lesion classifier. Specifically, based on the accuracy rate, for example, if the analysis results output by the lesion classifier are consistent with the doctor's analysis results, it is recorded as accurate analysis, otherwise it is recorded as inaccurate analysis, and then the analysis accuracy of the lesion classifier is calculated, with a fixed number of uses of 100 times as a cycle, according to the formula:

[0145]

[0146] Among them, S C represents the accuracy of the analysis, and C represents the number of times the analysis is accurate;

[0147] Then set a threshold, for example, when the analysis accuracy of the lesion classifier is lower than 80%, it is considered that the lesion classifier needs to be optimized.

[0148] In one embodiment, the method for determining the probability distribution and severity level of a symptom based on a symptom classifier further includes:

[0149] Determine a corresponding white light image according to the fused feature vector of the determined disease;

[0150] Process white light images based on edge detection algorithms to segment the contours of the diseased area and the background;

[0151] A semi-transparent color layer is superimposed on the outline of the diseased area obtained by segmentation to obtain a disease-highlighted image.

[0152] After the analysis results are output, an analysis report is output based on the analysis results. Based on the application of the template, a corresponding analysis report can be made based on the corresponding patient information, digestive endoscopy images and analysis results, and displayed on the computer of the department doctor. The department doctor can assist in diagnosing the patient's digestive endoscopy images based on the analysis report, so as to avoid missed diagnosis and misdiagnosis. In addition, in order to further improve the doctor's analysis and diagnosis of the patient's digestive endoscopy images, in one embodiment of the present invention, based on the determined symptoms, the corresponding white light image and Raman spectrum image can be obtained, and the white light image can be processed based on the edge detection algorithm to segment the symptom area contour and the background contour. That is, it can be understood that based on the fused feature vector extracted from the multiple digestive endoscopy images taken by the patient, if the symptom is determined to exist after the symptom classifier, the corresponding white light image is extracted, and Based on the edge detection algorithm, the outline of the diseased area is segmented, and then the outline of the diseased area is processed to obtain a disease-highlighted image, which is used to further assist doctors in diagnosing the patient's disease. Exemplarily, based on the above, in the probability distribution map output by the lesion classifier, the probability of ulcer is the largest. Based on the probability distribution area of ​​ulcer, based on the edge monitoring algorithm, the probability distribution position and background outline of the ulcer are segmented, and after superimposing a semi-transparent color layer, a disease-highlighted image can be obtained based on the ulcer area. In addition, in addition to processing one disease to obtain a disease-highlighted image, multiple diseases can be superimposed with semi-transparent color layers according to possible diseases, so as to obtain multiple disease-highlighted images and display them in the same white light image. It should be noted that the semi-transparent color layer corresponding to each disease needs to be labeled with possible diseases.

[0153] In one embodiment, it further comprises a biopsy region determination module, wherein the biopsy region determination module is used to determine a biopsy position according to the analysis result;

[0154] The method for determining the biopsy location is:

[0155] Obtain the probability distribution map of each category output by the lesion classifier;

[0156] Select the region corresponding to the maximum probability of any lesion;

[0157] The center point of the area was used as the intended biopsy location;

[0158] The planned biopsy location is displayed and reviewed to determine whether the planned biopsy location meets clinical requirements, and the determined biopsy location is output.

[0159] After the fusion feature vector formed based on the white light image and the Raman spectrum image is input into the lesion classifier, the probability distribution diagram of each category can be output accordingly. For example, the classifier outputs the following probability distribution:

[0160] Normal: 10%

[0161] Gastritis: 20%

[0162] Ulcers: 50%

[0163] Cancer: 20%

[0164] Based on the above probability distribution graph, although there is only a 20% probability of cancer, and 20% is lower than the threshold, it does not mean that the lesion is not cancer. It only means that the most likely lesion is an ulcer. Based on the above, it may be the early stage of cancer. Therefore, it is necessary to combine biopsy for verification. In one embodiment of the present invention, after the lesion classifier outputs the probability distribution graph of each category, based on the probability distribution graph, the probability distribution area corresponding to the cancer is selected. The probability distribution area refers to the pixels in the image that are classified as cancer. Generally, the higher the probability value, the more likely it is that the area is cancer. Therefore, in order to avoid missed diagnosis or misdiagnosis, based on the obtained probability distribution graph, The probability distribution map is obtained, the probability distribution area corresponding to the cancer is selected, and then the center point of the area is selected as the predetermined biopsy position. Usually, after obtaining the predetermined biopsy position, it is still necessary for the doctor to review it and determine that the predetermined biopsy position meets the clinical requirements before the determined biopsy position can be output. Based on the above, after the probability distribution map output by the lesion classifier, based on the possible types of symptoms, even if the probability is lower than the threshold, selective biopsy is required according to the possible symptoms. In one embodiment of the present invention, based on the probability distribution map, the doctor is helped to accurately output the determined biopsy position, thereby avoiding missed diagnosis and misdiagnosis.

[0165] In one embodiment, the training method of the disease classifier is:

[0166] Collect digestive endoscopy images, including white light images and corresponding Raman spectroscopy images;

[0167] Preprocess the digestive endoscopy images and complete feature extraction;

[0168] The features are fused, and the fused feature vector is labeled according to the disease corresponding to the digestive endoscopy image to form a labeled fused feature vector;

[0169] Based on a part of the labeled fused feature vectors, the original classifier built based on the SVM algorithm is trained;

[0170] Based on another part of the labeled fusion feature vectors as a validation set, the classifier is cross-validated, the parameters of the model are optimized, and the disease classifier is obtained.

[0171] In order to be able to identify the multi-scale features of the lesion in the subsequent image recognition process, in the process of constructing the lesion classifier, it is also necessary to construct the multi-scale features corresponding to the first eigenvector, including training the original data based on image pyramid, multi-scale convolution and multi-scale feature fusion. After the fusion feature vector of the first eigenvector and the second eigenvector is input into the lesion classifier, the multi-scale features based on the white light image can be identified, thereby verifying the features of the lesions at various stages. The original data refers to the collected digestive endoscopy images, including white light images of any lesion at various stages and possible Raman spectrum images. Based on the white light images of any lesion at various stages, based on image pyramid, multi-scale convolution and multi-scale feature fusion, a lesion classifier that can verify the probability distribution and severity level of the lesion corresponding to the fused feature vector can be trained, thereby realizing multi-scale and multi-dimensional lesion auxiliary diagnosis analysis.

[0172] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A digestive endoscopy auxiliary system based on artificial intelligence, characterized in that: include: A data acquisition and preprocessing module, used for acquiring and preprocessing digestive endoscopic images; the digestive endoscopic images include white light images and Raman spectroscopy images; A feature extraction module is used to extract features from the preprocessed digestive endoscopy image, and obtain a first feature vector corresponding to the white light image and a second feature vector corresponding to the Raman spectrum image; A feature fusion module is used to fuse the extracted first feature vector and the second feature vector to obtain a multi-dimensional fused feature vector; A symptom analysis module, based on a symptom classifier, inputs the fused feature vector into the symptom classifier, determines the probability distribution and severity level of the symptom based on the symptom classifier, and outputs the probability distribution and severity level of the symptom as an analysis result; The report generation module generates auxiliary diagnosis reports based on the analysis results.

2. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 1, characterized in that: The preprocessing method of the digestive endoscopy image is: De-noising the white light image, based on any one of the mean filter, median filter, and bilateral filter; Perform illumination correction on white light images, using either histogram equalization or Retinex algorithm to correct image illumination; Perform image segmentation on the white light image based on the image segmentation algorithm to obtain the local feature area and background area; The Raman spectrum image is denoised and the data is normalized, and noise is removed based on either baseline correction or wavelet transform.

3. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 2 is characterized in that: The method for removing noise from the Raman spectrum image using wavelet transform is as follows: Based on the wavelet basis function, the Raman spectrum image is decomposed according to the preset decomposition layer number, and the decomposed image is classified into low-frequency components and high-frequency components; Remove noise from the high-frequency components of the image based on the threshold method; The denoised high-frequency components are combined with the low-frequency components to obtain a combined image; The combined image is reconstructed using the selected wavelet basis function and decomposition layer number to obtain the denoised Raman spectrum image.

4. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 3 is characterized in that: The feature extraction method of the white light image is: The white light image is processed through the pre-trained CNN model to output multiple feature maps; Select the maximum pixel value in each feature map; Global max pooling is applied after the last convolutional layer to form a global feature vector; Based on the combination of convolutional layers and pooling layers, local feature vectors are extracted; Concatenate the local eigenvector and the global eigenvector in the same dimension to obtain the first eigenvector; The feature extraction method of the Raman spectrum image is: Convert the denoised Raman spectrum image into a data set; Process the data set based on the principal component analysis algorithm to obtain eigenvalues ​​and eigenvectors; Sort by eigenvalue size and obtain the principal component whose sorting is greater than the median; The selected principal components are combined to obtain the second eigenvector.

5. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 1 is characterized in that: The method for obtaining the multi-dimensional combined fusion feature vector is: Obtain a first eigenvector f1 and a second eigenvector f2; Evaluate the importance of the first eigenvector f1 and the second eigenvector f2 to determine weight factors w1 and w2; According to the formula: F=w1×f1+w2×f2 Among them, F is the fused feature vector, w1 is the weight factor of the first feature vector, and w2 is the weight factor of the second feature vector.

6. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 5, characterized in that: The method for obtaining the multi-dimensional combined fusion feature vector further includes: Obtain the dimension d1 of the first feature vector and the dimension d2 of the second feature vector; Determine whether d1 is equal to d2; If d1 = d2, then weighted fusion of the first feature vector f1 and the second feature vector f2 can be performed; If d1 ≠ d2, then match the dimensions based on the principal component analysis algorithm, including: If d1 < d2, then fill the dimension d1 of the first feature vector until d1 = d2; If d1 > d2, then fill the dimension d2 of the second feature vector until d1 = d2.

7. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 1 is characterized in that: The method for determining the probability distribution and severity level of a disease based on a disease classifier is as follows: Obtain the fusion feature vector; Input the fusion feature vector into the trained disease classifier; Based on the fully connected neural network, perform forward propagation on the fusion feature vector; Based on the softmax activation function, output the probability distribution map of each category for the fusion feature vector, and at the same time, based on the linear activation function, output a continuous value; Select the category with the highest probability in the probability distribution map as the most likely disease; Compare the probability of the most likely disease with a threshold. When the probability is greater than the threshold, it is determined that the disease exists, otherwise it is determined that the disease does not exist; Output the severity level of the disease based on the continuous value and the preset continuous value - degree comparison table; Combine and output the disease and the corresponding severity level as the analysis result.

8. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 7, characterized in that: The method for determining the probability distribution and severity level of a disease based on a disease classifier further includes: According to the fusion feature vector of the determined disease, determine the corresponding white light image; Process the white light image based on the edge detection algorithm to segment the disease area contour and the background contour; Overlay a semi-transparent color layer on the segmented disease area contour to obtain a disease prominent image.

9. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 1, characterized in that: It further includes a biopsy area determination module, and the biopsy area determination module is used to determine the biopsy location according to the analysis result; The method for determining the biopsy location is as follows: Obtain the probability distribution map of each category output by the lesion classifier; Select the region corresponding to the maximum probability value of any lesion; Take the center point of the region as the predetermined biopsy location; Display the predetermined biopsy location and conduct a review. If it is determined that the predetermined biopsy location meets the clinical requirements, output the determined biopsy location.

10. The artificial intelligence-based digestive endoscopy auxiliary system according to claim 1, characterized in that: The training method of the disease classifier is as follows: Collect endoscopic images, including white light images and corresponding Raman spectra images; Preprocess the endoscopic images and complete feature extraction; Fuse the features and label the fusion feature vector according to the disease corresponding to the endoscopic image to form a labeled fusion feature vector; Based on a part of the labeled fusion feature vectors, train the original classifier constructed based on the SVM algorithm; Based on another part of the labeled fusion feature vectors as the validation set, perform cross-validation on the classifier to optimize the parameters of the model and obtain the disease classifier.

Images