Colitis diagnosis system, device and storage medium

Abstract

The present application discloses a colitis diagnosis system, device and storage medium. The system includes: an endoscope, a differential diagnosis device and an output device, wherein the endoscope is used to generate an intestinal internal image corresponding to the patient and send the intestinal internal image to the differential diagnosis device; and the output device is used to receive diagnostic information sent by the differential diagnosis device and output the diagnostic information, the diagnostic information is used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, and the differential diagnosis device is used to generate diagnostic information based on the intestinal internal image and the test index information corresponding to the patient's blood test.

Description

Technical Field

[0001] The present application relates to the field of medical instrument technology, and in particular to a colitis diagnosis system, device and storage medium. Background Art

[0002] Ulcerative colitis (UC) is a chronic intestinal inflammatory immune disease with a protracted course that seriously affects the patient's quality of life. UC patients are at increased risk of cytomegalovirus (CMV) infection after receiving hormones, immunosuppressants, and biologics. Studies have shown that UC patients with CMV infection have higher disease activity and the risk of hormone resistance, which is worse after recovery. However, UC and UC combined with CMV colitis are extremely similar in clinical manifestations, and endoscopic features have certain differential diagnostic value, but clinicians often lack awareness and the detection rate of biopsy is low. Therefore, a differential diagnosis model is needed to help identify UC and UC combined with CMV colitis, and taking tissue biopsies for pathological verification has certain clinical significance.

[0003] Currently available artificial intelligence technologies can assist in the identification of ulcerative colitis. The mainstream methods are: a) using machine learning or deep learning models for classification based on clinical test index information; b) using deep convolutional neural networks to complete classification tasks based on endoscopic intestinal images.

[0004] Most of the existing artificial intelligence technologies used to identify ulcerative colitis only consider one type of information: internal intestinal images or test indicator information, and the extraction operation of image features is also relatively rough, resulting in low accuracy of classification results.

[0005] The artificial intelligence technologies used in the above-mentioned existing technologies for identifying ulcerative colitis mostly only consider one type of information among the internal intestinal images or test index information, and the extraction operation of image features is also relatively rough, which leads to the technical problem of low accuracy of classification results. No effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a colitis diagnosis system, device and storage medium to at least solve the technical problem that the artificial intelligence technology used to identify ulcerative colitis in the prior art mostly only considers one kind of information among the internal intestinal images or test index information, and the extraction operation of image features is also relatively rough, resulting in low accuracy of classification results.

[0007] According to one aspect of an embodiment of the present application, a colitis diagnosis system is provided, comprising: an endoscope, a differential diagnosis device, and an output device, wherein the endoscope is used to generate an internal intestinal image corresponding to the patient, and send the internal intestinal image to the differential diagnosis device; and the output device is used to receive diagnostic information sent by the differential diagnosis device and output the diagnostic information, the diagnostic information being used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, and the differential diagnosis device is used to generate diagnostic information based on the internal intestinal image and test index information corresponding to the patient's blood test.

[0008] According to another aspect of an embodiment of the present application, a storage medium is also provided, the storage medium includes a stored program, wherein when the program is running, the processor performs the following steps: generating diagnostic information based on an internal intestinal image and test index information corresponding to a blood test of the patient.

[0009] According to another aspect of an embodiment of the present application, a colitis diagnostic device is also provided, including: an information generation module for generating diagnostic information based on an internal intestinal image and test index information corresponding to a blood test of a patient.

[0010] According to another aspect of an embodiment of the present application, a colitis diagnostic device is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: generating diagnostic information based on an internal intestinal image and test index information corresponding to a blood test of a patient.

[0011] In an embodiment of the present application, the differential diagnosis device fuses the test index information and the internal intestinal image to generate fused information, and extracts features from the fused information, and then classifies it to determine the diagnostic information. Compared with the prior art that only considers the internal intestinal image or only considers the test index information, the present technical solution classifies according to both the test index information and the internal intestinal image, so that the consideration of the input data is more comprehensive, and the accuracy of the classification results is improved. In addition, the present technical solution extracts features from the internal intestinal image and the test index information through multiple feature extraction modules, so that the extraction operation is more refined, and the accuracy of the classification results is improved. This solves the technical problem that most of the artificial intelligence technologies used to identify ulcerative colitis in the prior art only consider one type of information in the internal intestinal image or the test index information, and the extraction operation of the image features is also relatively rough, resulting in a low accuracy rate of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 is a schematic diagram of a colitis diagnostic system according to the first aspect of Example 1 of the present application;

[0014] Figure 2 is a flowchart of operations performed by the differential diagnosis device of the colitis diagnosis system according to the first aspect of Example 1 of the present application;

[0015] Figure 3 is a schematic diagram of a differential diagnosis model deployed in the differential diagnosis device according to the first aspect of Example 1 of the present application;

[0016] Figure 4 is a schematic diagram of multiple feature extraction modules of the differential diagnosis model according to the first aspect of Example 1 of the present application;

[0017] Figure 5 is a schematic diagram of a feature extraction module of a differential diagnosis model according to the first aspect of Example 1 of the present application;

[0018] Figure 6 is a schematic diagram of a feature fusion module of a differential diagnosis model according to the first aspect of Example 1 of the present application;

[0019] Figure 7 is a schematic diagram of a fully connected layer of a feature fusion module of a differential diagnosis model according to the first aspect of Example 1 of the present application;

[0020] Figure 8 is another flowchart diagram of the operations performed by the differential diagnosis device of the colitis diagnosis system according to the first aspect of Example 1 of the present application;

[0021] Fig. 9A The intestinal internal image which has not been denoised according to the first aspect of Embodiment 1 of the present application;

[0022] Fig. 9B The intestinal internal image that has been subjected to denoising processing according to the first aspect of Embodiment 1 of the present application;

[0023] Fig.10 is a schematic diagram of a colitis diagnostic device according to Example 2 of the present application; and

[0024] Fig.11 It is a schematic diagram of the colitis diagnostic device described in Example 3 of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] Figure 1 Schematic diagram of the colitis diagnosis system according to Example 1 of the present application is shown. Figure 1 As shown, a colitis diagnostic system includes: an endoscope 110, a differential diagnosis device 120 and an output device 130, wherein the endoscope 110 is used to generate an internal intestinal image corresponding to the patient and send the internal intestinal image to the differential diagnosis device 120; and the output device 130 is used to receive the diagnostic information sent by the differential diagnosis device 120 and output the diagnostic information.

[0029] also, Figure 2 A flowchart of the operations performed by the differential diagnosis device 120 is further shown. Figure 2 As shown, the differential diagnosis device 120 is configured to perform the following operations:

[0030] S202: Generate diagnostic information based on the intestinal internal image and the test index information corresponding to the patient's blood test.

[0031] Specifically, refer to Figure 1As shown, the colitis diagnosis system 100 described in this embodiment includes an endoscope 110, a differential diagnosis device 120, and an output device 130. The endoscope 110 and the differential diagnosis device 120 are connected in communication to generate an intestinal internal image corresponding to the patient. The endoscope 110 then sends the generated intestinal internal image to the differential diagnosis device 120.

[0032] The differential diagnosis device 120 is provided with a differential diagnosis model for determining diagnostic information based on the intestinal internal image. The differential diagnosis device 120 receives the intestinal internal image sent by the endoscope 110, and then inputs the intestinal internal image into the differential diagnosis model. At the same time, the differential diagnosis device 120 inputs the test index information into the differential diagnosis model. The test index information is used to indicate the blood test information corresponding to the patient, for example, it may include: white blood cell count, neutrophil count, hemoglobin, eosinophil count, lymphocyte count, platelets, immunoglobulin A and erythrocyte sedimentation rate.

[0033] Furthermore, the differential diagnosis model extracts features from the received intestinal internal images through multiple feature extraction modules and outputs corresponding feature maps. The differential diagnosis model then splices and fuses the test index information and the feature map to generate fusion information. The differential diagnosis model then extracts features from the fusion information to generate feature information.

[0034] Furthermore, the differential diagnosis device 120 generates a classification result as the diagnosis information according to the feature information through a softmax classifier, wherein the classification result includes ulcerative colitis, ulcerative colitis combined with cytomegalovirus colitis and normal.

[0035] Furthermore, the differential diagnosis device 120 sends the diagnosis information to the output device 130 , and the output device 130 outputs the diagnosis information.

[0036] As described in the background technology, currently available artificial intelligence technologies can assist in the identification of ulcerative colitis. The mainstream methods are: a) Classification based on clinical test index information using machine learning or deep learning models. b) Classification tasks are completed using deep convolutional neural networks based on endoscopic intestinal images. Most of the existing artificial intelligence technologies used to identify ulcerative colitis only consider one type of information, either the intestinal image or the test index information, and the image feature extraction operation is relatively rough, resulting in a low accuracy rate in the classification results.

[0037] In view of the technical problems described above, through the technical solutions of the embodiments of the present application, the differential diagnosis device fuses the test index information and the internal intestinal image to generate fused information, and extracts features from the fused information, and then classifies it to determine the diagnostic information. Compared with the prior art that only considers the internal intestinal image or only considers the test index information, the present technical solution classifies according to both the test index information and the internal intestinal image, so that the consideration of the input data is more comprehensive, and the accuracy of the classification results is improved. In addition, the present technical solution extracts features from the internal intestinal image and the test index information through multiple feature extraction modules, so that the extraction operation is more refined, and the accuracy of the classification results is improved. This solves the technical problem that most of the artificial intelligence technologies used to identify ulcerative colitis in the prior art only consider one type of information in the internal intestinal image or the test index information, and the extraction operation of the image features is also relatively rough, resulting in a low accuracy rate of the classification results.

[0038] Optionally, the operation of generating diagnostic information based on the internal intestinal image and the test index information corresponding to the patient's blood test includes: performing feature extraction on the internal intestinal image through a first feature extraction module to generate first feature information; fusing the first feature information and the test index information through a feature fusion module to generate first fusion information; performing feature extraction on the first fusion information through a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information using a softmax classifier to determine the diagnostic information corresponding to the patient.

[0039] Specifically, refer to Figure 3 As shown, the differential diagnosis device 120 receives the intestinal internal image sent by the endoscope 110, and then inputs the intestinal internal image into the differential diagnosis model.

[0040] The convolution layer 201 in the differential diagnosis model extracts features from the input intestinal internal image to generate a feature map 301. The convolution kernel of the convolution layer 201 is a 7×7 convolution kernel. The differential diagnosis device 120 then inputs the feature map 301 into the batch normalization layer 202 in the differential diagnosis model, performs mean and variance normalization on the feature map 301 through the batch normalization layer 202, and processes the feature map 301 after batch normalization using the ReLU activation function to generate a feature map 302. The differential diagnosis device 120 then inputs the feature map 302 into the maximum pooling layer 203 of the differential diagnosis model, performs a pooling operation on the feature map 302 through the 3×3 maximum pooling layer 203, reduces the size of the feature map 302, and generates a feature map 303. Thus, the differential diagnosis device 120 extracts shallow features of the intestinal internal image through the convolution layer 201, the batch normalization layer 202+ReLU, and the maximum pooling layer 203.

[0041] Further, the differential diagnosis device 120 inputs the feature graph 303 into the feature extraction module 204 (i.e., the first feature extraction module) of the differential diagnosis model, and extracts features from the feature graph 303 through the feature extraction module 204 to generate the feature graph 304. Then, the differential diagnosis device 120 inputs the feature graph 304 into the feature extraction module 205 (i.e., the first feature extraction module) of the differential diagnosis model, and extracts features from the feature graph 304 through the feature extraction module 205 to generate the feature graph 305. Then, the differential diagnosis device 120 inputs the feature graph 305 into the feature extraction module 206 (i.e., the first feature extraction module) of the differential diagnosis model, and extracts features from the feature graph 305 through the feature extraction module 206 to generate the feature graph 306 (i.e., the first feature information). Thus, the differential diagnosis device 120 extracts deep features of the internal intestinal image through the feature extraction modules 204, 205, and 206.

[0042] Further, the differential diagnosis device 120 inputs the characteristic graph 306 (i.e., the first characteristic information) into the characteristic fusion module 207, and simultaneously inputs the test index information into the characteristic fusion module 207. The test index information is used to indicate the blood test information corresponding to the patient, and may include, for example, white blood cell count, neutrophil count, hemoglobin, eosinophil count, lymphocyte count, platelet, immunoglobulin A and erythrocyte sedimentation rate.

[0043] Furthermore, the differential diagnosis device 120 concatenates and fuses the test index information and the feature graph 306 through the feature fusion module 207 , thereby generating a feature graph 307 (ie, first fusion information).

[0044] Furthermore, the differential diagnosis device 120 inputs the feature graph 307 (ie, the first fusion information) into the feature extraction module 208 (ie, the second feature extraction module) of the differential diagnosis model, and the feature extraction module 208 extracts features from the feature graph 307 to generate the feature graph 308 (ie, the second feature information).

[0045] Furthermore, the differential diagnosis device 120 inputs the feature map 308 to the convolution layer 209 of the differential diagnosis model, and extracts features from the feature map 308 through the convolution layer 209 to generate the feature map 309. The convolution kernel of the convolution layer 209 is a 1×1 convolution kernel.

[0046] Furthermore, the differential diagnosis device 120 inputs the feature map 309 to the batch normalization layer 210 of the differential diagnosis model, and performs mean and variance normalization processing on the feature map 309 through the batch normalization layer 210 to generate the feature map 310 .

[0047] Further, the differential diagnosis device 120 fuses the feature map 306 generated by the feature extraction module 206 and the feature map 310 generated by the batch normalization layer 210 through concat(), thereby generating a feature map 311 (ie, second fusion information).

[0048] Furthermore, the differential diagnosis device 120 generates a classification result as diagnosis information according to the feature graph 311 through the softmax classifier 213. The classification result includes ulcerative colitis, ulcerative colitis combined with cytomegalovirus colitis, and normal.

[0049] Therefore, the technical solution performs feature extraction on the internal intestinal images and test index information through multiple feature extraction modules, making the extraction operation more refined and improving the accuracy of the classification results.

[0050] Optionally, the operation of performing feature extraction on the internal image of the intestine through the first feature extraction module to generate the first feature information includes: performing feature extraction on the internal image of the intestine through the first feature extraction unit of the first feature extraction module to generate a first feature map; and performing feature extraction on the first feature map through the second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information.

[0051] Specifically, refer to Figure 4 As shown, the feature extraction module 204 includes two feature extraction units, namely, feature extraction unit 2041 and feature extraction unit 2042. And the feature extraction modules 205, 206 and 207 all include the same two feature extraction units as the feature extraction module 204.

[0052] More specifically, after the maximum pooling layer 203 outputs the feature map 303 , the differential diagnosis device 120 inputs the feature map 303 into the feature extraction module 204 , and performs feature extraction on the feature map 303 through the feature extraction unit 2041 of the feature extraction module 204 to generate a corresponding feature map 3031 (ie, a first feature map).

[0053] Furthermore, the differential diagnosis device 120 inputs the feature graph 3031 into the feature extraction unit 2042 , and the feature extraction unit 2042 performs feature extraction on the feature graph 3031 to generate a corresponding feature graph 304 (ie, a second feature graph).

[0054] Furthermore, the differential diagnosis device 120 inputs the feature graph 304 into the feature extraction module 205 , and the feature extraction unit 2051 of the feature extraction module 205 extracts features from the feature graph 304 to generate a corresponding feature graph 3041 (ie, a first feature graph).

[0055] Furthermore, the differential diagnosis device 120 inputs the feature graph 3041 into the feature extraction unit 2052 , and the feature extraction unit 2052 extracts features from the feature graph 3041 to generate a corresponding feature graph 305 (ie, a second feature graph).

[0056] Furthermore, the differential diagnosis device 120 inputs the feature graph 305 into the feature extraction module 206 , and the feature extraction unit 2061 of the feature extraction module 206 performs feature extraction on the feature graph 305 to generate a corresponding feature graph 3051 (ie, a first feature graph).

[0057] Furthermore, the differential diagnosis device 120 inputs the feature graph 3051 into the feature extraction unit 2062 , and the feature extraction unit 2062 extracts features from the feature graph 3051 to generate a corresponding feature graph 306 (ie, a second feature graph).

[0058] Therefore, the present technical solution performs feature extraction on the internal intestinal image through multiple feature extraction modules, making the extraction operation more refined and improving the accuracy of the classification results.

[0059] Optionally, the operation of performing feature extraction on the internal image of the intestine through the first feature extraction unit of the first feature extraction module to generate a first feature map includes: performing feature extraction on the internal image of the intestine through the first convolutional layer of the first feature extraction unit to generate a third feature map; processing the third feature map through the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; performing feature extraction on the fourth feature map through the second convolutional layer of the first feature extraction unit to generate a fifth feature map; and batch normalization processing on the fifth feature map through the second batch normalization layer of the first feature extraction unit to generate the first feature map.

[0060] Specifically, refer to Figure 5 As shown, the differential diagnosis device 120 inputs the feature map 303 corresponding to the internal image of the intestine into the feature extraction unit 2041 (i.e., the first feature extraction unit), and extracts features from the feature map 303 through the convolution layer 20411 (i.e., the first convolution layer) in the feature extraction unit 2041, generating a feature map 30311 (i.e., the third feature map). The convolution kernel of the convolution layer 20411 is a 3×3 convolution kernel. After that, the differential diagnosis device 120 inputs the feature map 30311 into the batch normalization layer 20412 (i.e., the first batch normalization layer), performs mean and variance normalization on the feature map 30311 through the batch normalization layer 20412, and uses the ReLU activation function to process the feature map 30311 after batch normalization, generating a feature map 30312 (i.e., the fourth feature map).

[0061] Furthermore, the differential diagnosis device 120 inputs the feature map 30312 to the convolution layer 20413 (i.e., the second convolution layer), extracts features from the feature map 30312 through the convolution layer 20413, and generates a feature map 30313 (i.e., the fifth feature map). The convolution kernel of the convolution layer 20413 is a 3×3 convolution kernel.

[0062] Further, the differential diagnosis device 120 inputs the feature map 30313 to the batch normalization layer 20414 (ie, the second batch normalization layer), performs mean and variance normalization on the feature map 30313 through the batch normalization layer 20414, and generates a feature map 3031 (ie, the first feature map).

[0063] Further, the differential diagnosis device 120 inputs the feature map 3031 to the feature extraction unit 2042. According to the convolution layer 20411, batch normalization layer 20412+ReLU, convolution layer 20413 and batch normalization layer 20414 in the feature extraction unit 2041, the feature map 303 is processed to generate the feature map 3031, and the feature map 304 is processed by the convolution layer 20421, batch normalization layer 20422+ReLU, convolution layer 20423 and batch normalization layer 20424 in the feature extraction unit 2042, and the feature map 304 is generated, which will not be repeated here. The convolution kernels of the convolution layer 20421 and the convolution layer 20423 are 3×3 convolution kernels.

[0064] In addition, the feature extraction units in the feature extraction modules 205, 206 and 208 have the same structure as the feature extraction unit in the feature extraction module 204, and also include a convolution layer, a batch normalization layer+ReLU, a convolution layer and a batch normalization layer, so the processing method for the feature map is the same as the processing method for the feature map in the feature extraction module 204. No further details will be given here.

[0065] Therefore, the technical solution performs feature extraction on the internal intestinal image through multiple feature extraction units, making the extraction operation more refined and improving the accuracy of the classification results.

[0066] Optionally, the operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector.

[0067] Specifically, refer to Figure 6 As shown, the differential diagnosis device 120 inputs the feature map 306 (i.e., the first feature information) into the feature fusion module 207, and performs a global average pooling operation on the feature map 306 through the global average pooling layer 2071 (i.e., the first global average pooling layer) in the feature fusion module 207 to generate a feature vector D. The feature vector D is specifically D(d1, d2, ..., d c ), c is the number of channels.

[0068] Furthermore, the differential diagnosis device 120 inputs the feature vector D into the clustering layer 2072. The differential diagnosis device 120 processes the test index information to generate a test index feature vector A (a1, a2, ..., a n ), and inputs the test index feature vector to the clustering layer 2072. Where n represents the number of test index information. Then, the differential diagnosis device 120 fuses the feature vector D and the test index feature vector A through the clustering layer 2072 according to a pre-set fusion formula to obtain fused feature information Z The fusion formula is as follows:

[0069]

[0070] in i =1~ n , j =1~c, that is, the test index characteristic vector A is a 1~a n , the eigenvector D is d 1~ d c , K is the convolution kernel in the aggregation layer, and the fusion feature information Z is specifically Z(z1,z2,...,z c ).

[0071] Further, refer to Figure 7 As shown, the differential diagnosis device 120 fuses the feature information Z (z1, z2, ..., z c ) is input to the fully connected layer 2073 (i.e., the first fully connected layer), and the bias vector E(e1, e2, ..., e c ) and the channel weight vector F(f1,f2,...,f c For example, the feature graph 306 is a feature graph B (b1, b2, ..., b c ). Where b j represents the pixel value corresponding to the jth channel of the feature map 306. j represents the weight corresponding to the jth channel, e j Bias value corresponding to the j-th channel.

[0072] Furthermore, the differential diagnosis device 120 converts the characteristic graph B (b1, b2, ..., b c ) is multiplied by the channel weight vector F(f1,f2,...,f c ) and the weight corresponding to the channel, plus the bias vector E(e1,e2,...,e c ) in the channel, thereby generating a feature map G(g1,g2,...,g c ) (i.e., the first fusion information). The calculation formula is:

[0073]

[0074] Therefore, the technical solution completes the classification task by fusing the test index information and the intestinal internal image and using the generated fusion information as the weight of the image channel attention. Therefore, the technical solution performs classification based on the test index information and the intestinal internal image, making the consideration of the input data more comprehensive and improving the accuracy of the classification results.

[0075] Optionally, the operation of classifying the second fusion information using a softmax classifier to determine the diagnosis information corresponding to the patient includes: processing the second fusion information using a ReLU activation function to generate feature data; performing global average pooling on the feature data through a second global average pooling layer to generate a second feature vector; processing the second feature vector through a second fully connected layer to generate a third feature vector; and classifying the third feature vector using a softmax classifier to determine the diagnosis information corresponding to the patient.

[0076] Specifically, refer to Figure 8 As shown, the differential diagnosis device 120 processes the feature map 311 (ie, the second fusion information) through the ReLU activation function to generate corresponding feature data.

[0077] Furthermore, the differential diagnosis device 120 inputs the feature data into the global average pooling layer 211 (ie, the second global average pooling layer), performs global average pooling processing on the feature data through the global average pooling layer 211, and generates a corresponding feature vector (ie, the second feature vector).

[0078] Furthermore, the differential diagnosis device 120 inputs the feature vector (i.e., the second feature vector) to the fully connected layer 212 (i.e., the second fully connected layer), processes the feature vector (i.e., the second feature vector) through the fully connected layer 212, and generates a corresponding feature vector (i.e., the third feature vector).

[0079] Further, the differential diagnosis device 120 inputs the feature vector (i.e., the third feature vector) into the softmax classifier 213, and classifies the third feature vector through the softmax classifier 213, and determines that the classification result is ulcerative colitis, ulcerative colitis combined with cytomegalovirus colitis, or normal. Then, the differential diagnosis device 120 uses the classification result as the diagnosis information corresponding to the patient.

[0080] Therefore, this technical solution can reduce the number of model parameters and the amount of calculation through global average pooling, thereby improving the efficiency of determining classification results.

[0081] Optionally, the differential diagnosis device 120 is further configured to perform the following operations: denoising the intestinal tract internal image.

[0082] Specifically, before the differential diagnosis device 120 inputs the intestinal tract image into the differential diagnosis model for processing, the differential diagnosis device 120 performs denoising processing on the intestinal tract image.

[0083] For example, Fig. 9AThe image of the intestines without denoising in this embodiment is shown, and the left side of the image of the intestines shows information such as age, gender and time. Fig. 9B An image of the inside of the intestine that has been denoised in this embodiment is shown.

[0084] refer to Fig. 9A as well as Fig. 9B As shown, the differential diagnosis device 120 uses the contour detection algorithm to detect the intestinal internal image that has not been denoised (reference Fig. 9A The right intestinal contour is identified by a set of several points, and then the maximum and minimum values ​​of the coordinates (x, y) in the set are calculated to obtain the coordinates of the four points of the rectangular frame surrounding the intestinal contour. Finally, the area corresponding to the rectangle with the four points (x_min, y_min), (x_min, y_max), (x_max, y_min), and (x_max, y_max) as vertices is taken as the denoised intestinal internal image (reference Fig. 9B as shown).

[0085] Therefore, the present technical solution performs denoising on the internal intestinal images to reduce the situation where noise affects the final diagnosis result, making the final diagnostic information more accurate.

[0086] In addition, this technical solution uses a GeForce3060 graphics card with a video memory capacity of 12GB to train the differential diagnosis model, and selects the cross entropy cost function as the loss function to supervise the optimization direction of the model. Adam is a variant of stochastic gradient descent, which has an excellent ability to escape the saddle point. Therefore, this technical solution uses it as an optimizer for model training, and fixes the learning rate to 1e-4. At the same time, gradient clipping is used to prevent jitter caused by excessive gradients during training. This technical solution collected a total of 2340 samples as training data, and performed 200 iterations with a batch size of 10. At the same time, in order to avoid the occurrence of overfitting, the images are randomly flipped and rotated horizontally and vertically during the training process to increase the diversity of the data set. Finally, after the network converges, the model parameters with the best performance are taken as the final result.

[0087] In addition, according to a second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is executed, the processor performs the following steps: generating diagnosis information based on the intestinal internal image and the test index information corresponding to the patient's blood test.

[0088] Therefore, according to this embodiment, the differential diagnosis device fuses the test index information and the internal intestinal image to generate fused information, and extracts features from the fused information, and then classifies it to determine the diagnostic information. Compared with the prior art that only considers the internal intestinal image or only considers the test index information, the present technical solution classifies according to both the test index information and the internal intestinal image, so that the consideration of the input data is more comprehensive, and the accuracy of the classification results is improved. In addition, the present technical solution extracts features from the internal intestinal image and the test index information through multiple feature extraction modules, so that the extraction operation is more refined, and the accuracy of the classification results is improved. This solves the technical problem that most of the artificial intelligence technologies used to identify ulcerative colitis in the prior art only consider one type of information in the internal intestinal image or the test index information, and the extraction operation of the image features is also relatively rough, resulting in a low accuracy rate of the classification results.

[0089] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0091] Example 2

[0092] Fig.10 The colitis diagnosis device 1000 according to the first aspect of this embodiment is shown, and the device 1000 corresponds to the method according to the first aspect of embodiment 1. Fig.10 As shown, the device 1000 includes: an information generating module 1010, which is used to generate diagnostic information according to the internal intestinal image and the test index information corresponding to the patient's blood test.

[0093] Optionally, the information generation module 1010 includes: a first generation submodule, used to perform feature extraction on the internal intestinal image through a first feature extraction module to generate first feature information; a second generation submodule, used to fuse the first feature information and the test index information through a feature fusion module to generate first fusion information; a third generation submodule, used to perform feature extraction on the first fusion information through a second feature extraction module to generate second feature information; a fourth generation submodule, used to fuse the first feature information and the second feature information to generate second fusion information; and an information determination submodule, used to classify the second fusion information using a softmax classifier to determine the diagnostic information corresponding to the patient.

[0094] Optionally, the first generating submodule includes: a first generating unit, used to perform feature extraction on the internal image of the intestine through the first feature extraction unit of the first feature extraction module, and generate a first feature map; and a second generating unit, used to perform feature extraction on the first feature map through the second feature extraction unit of the first feature extraction module, and generate a second feature map as the first feature information.

[0095] Optionally, the first generation unit includes: performing feature extraction on the internal intestinal image through the first convolution layer of the first feature extraction unit to generate a third feature map; processing the third feature map through the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; performing feature extraction on the fourth feature map through the second convolution layer of the first feature extraction unit to generate a fifth feature map; and batch normalization processing on the fifth feature map through the second batch normalization layer of the first feature extraction unit to generate the first feature map.

[0096] Optionally, the second generation submodule includes: a third generation unit, used to perform global average pooling on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; a fourth generation unit, used to fuse the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and a fifth generation unit, used to generate the first fusion information based on the first feature information, the channel weight vector and the bias vector.

[0097] Optionally, the information determination submodule includes: a sixth generation unit, used to process the second fusion information using a ReLU activation function to generate feature data; a seventh generation unit, used to perform global average pooling on the feature data through a second global average pooling layer to generate a second feature vector; an eighth generation unit, used to process the second feature vector through a second fully connected layer to generate a third feature vector; and an information determination unit, used to classify the third feature vector using a softmax classifier to determine the diagnostic information corresponding to the patient.

[0098] Optionally, the device 1000 further includes: a denoising module for performing denoising processing on the internal intestinal image.

[0099] Therefore, according to this embodiment, the differential diagnosis device fuses the test index information and the internal intestinal image to generate fused information, and extracts features from the fused information, and then classifies it to determine the diagnostic information. Compared with the prior art that only considers the internal intestinal image or only considers the test index information, the present technical solution classifies according to both the test index information and the internal intestinal image, so that the consideration of the input data is more comprehensive, and the accuracy of the classification results is improved. In addition, the present technical solution extracts features from the internal intestinal image and the test index information through multiple feature extraction modules, so that the extraction operation is more refined, and the accuracy of the classification results is improved. This solves the technical problem that most of the artificial intelligence technologies used to identify ulcerative colitis in the prior art only consider one type of information in the internal intestinal image or the test index information, and the extraction operation of the image features is also relatively rough, resulting in a low accuracy rate of the classification results.

[0100] Example 3

[0101] Fig.11 The colitis diagnosis device 1100 according to the first aspect of this embodiment is shown, and the device 1100 corresponds to the method according to the first aspect of embodiment 1. Fig.11 As shown, the device 1100 includes: a processor 1110; and a memory 1120, which is connected to the processor 1110 and is used to provide the processor 1110 with instructions for processing the following processing steps: generating diagnostic information based on the internal intestinal image and the test index information corresponding to the patient's blood test.

[0102] Optionally, the operation of generating diagnostic information based on the internal intestinal image and the test index information corresponding to the patient's blood test includes: performing feature extraction on the internal intestinal image through a first feature extraction module to generate first feature information; fusing the first feature information and the test index information through a feature fusion module to generate first fusion information; performing feature extraction on the first fusion information through a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information using a softmax classifier to determine the diagnostic information corresponding to the patient.

[0103] Optionally, the operation of performing feature extraction on the internal image of the intestine through the first feature extraction module to generate the first feature information includes: performing feature extraction on the internal image of the intestine through the first feature extraction unit of the first feature extraction module to generate a first feature map; and performing feature extraction on the first feature map through the second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information.

[0104] Optionally, the operation of performing feature extraction on the internal image of the intestine through the first feature extraction unit of the first feature extraction module to generate a first feature map includes: performing feature extraction on the internal image of the intestine through the first convolutional layer of the first feature extraction unit to generate a third feature map; processing the third feature map through the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; performing feature extraction on the fourth feature map through the second convolutional layer of the first feature extraction unit to generate a fifth feature map; and batch normalization processing on the fifth feature map through the second batch normalization layer of the first feature extraction unit to generate the first feature map.

[0105] Optionally, the operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector.

[0106] Optionally, the operation of classifying the second fusion information using a softmax classifier to determine the diagnosis information corresponding to the patient includes: processing the second fusion information using a ReLU activation function to generate feature data; performing global average pooling on the feature data through a second global average pooling layer to generate a second feature vector; processing the second feature vector through a second fully connected layer to generate a third feature vector; and classifying the third feature vector using a softmax classifier to determine the diagnosis information corresponding to the patient.

[0107] Optionally, the memory 1120 is further used to provide the processor 1110 with instructions for processing the following processing steps: denoising the internal intestinal image.

[0108] Therefore, according to this embodiment, the differential diagnosis device fuses the test index information and the internal intestinal image to generate fused information, and extracts features from the fused information, and then classifies it to determine the diagnostic information. Compared with the prior art that only considers the internal intestinal image or only considers the test index information, the present technical solution classifies according to both the test index information and the internal intestinal image, so that the consideration of the input data is more comprehensive, and the accuracy of the classification results is improved. In addition, the present technical solution extracts features from the internal intestinal image and the test index information through multiple feature extraction modules, so that the extraction operation is more refined, and the accuracy of the classification results is improved. This solves the technical problem that most of the artificial intelligence technologies used to identify ulcerative colitis in the prior art only consider one type of information in the internal intestinal image or the test index information, and the extraction operation of the image features is also relatively rough, resulting in a low accuracy rate of the classification results.

[0109] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0110] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0115] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A colitis diagnosis system, characterized in that: include: An endoscope (110), a differential diagnosis device (120), and an output device (130), wherein: The endoscope (110) is used to generate an intestinal internal image corresponding to the patient, and send the intestinal internal image to the differential diagnosis device (120); and The output device (130) is used to receive the diagnostic information sent by the differential diagnosis device (120) and output the diagnostic information, wherein the diagnostic information is used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, and The differential diagnosis device (120) is used to generate the diagnosis information according to the intestinal internal image and the test index information corresponding to the blood test of the patient, wherein The operation of generating the diagnosis information according to the intestinal internal image and the test index information corresponding to the blood test of the patient includes: extracting features of the intestinal internal image by a first feature extraction module to generate first feature information; fusing the first feature information and the test index information by a feature fusion module to generate first fusion information; extracting features of the first fusion information by a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information by a softmax classifier to determine the diagnosis information corresponding to the patient, wherein The operation of extracting features from the intestinal internal image through a first feature extraction module to generate first feature information includes: extracting features from the intestinal internal image through a first feature extraction unit of the first feature extraction module to generate a first feature map; and extracting features from the first feature map through a second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information, wherein The operation of extracting features of the internal intestinal image by the first feature extraction unit of the first feature extraction module to generate a first feature map includes: extracting features of the internal intestinal image by the first convolution layer of the first feature extraction unit to generate a third feature map; processing the third feature map by the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; extracting features of the fourth feature map by the second convolution layer of the first feature extraction unit to generate a fifth feature map; and performing batch normalization processing on the fifth feature map by the second batch normalization layer of the first feature extraction unit to generate the first feature map, and wherein The operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through a first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the inspection index feature vector corresponding to the inspection index information with the first feature vector through a clustering layer in the feature fusion module to generate fused feature information; processing the fused feature information through a first fully connected layer to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector through a preset calculation formula, wherein the calculation formula is: g j =b j *f j +e j , where g j represents the eigenvalue corresponding to the jth channel of the first fusion information, b j represents the feature value corresponding to the jth channel of the first feature information, f j represents the weight corresponding to the jth channel of the first feature information, e j The offset value corresponding to the j-th channel of the first feature information.

2. The system according to claim 1, characterized in that The operation of classifying the second fusion information by using a softmax classifier to determine the diagnosis information corresponding to the patient includes: Processing the second fusion information using a ReLU activation function to generate feature data; Performing global average pooling processing on the feature data through a second global average pooling layer to generate a second feature vector; Processing the second feature vector through a second fully connected layer to generate a third feature vector; and The third feature vector is classified using a softmax classifier to determine the diagnosis information corresponding to the patient.

3. The system according to claim 1, characterized in that The differential diagnosis device (120) is also configured to perform the following operations: denoising the intestinal internal image.

4. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the processor performs the following steps: Generate diagnostic information based on the intestinal internal image and the test index information corresponding to the patient's blood test, wherein the diagnostic information is used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, The operation of generating the diagnosis information according to the intestinal internal image and the test index information corresponding to the blood test of the patient includes: extracting features of the intestinal internal image by a first feature extraction module to generate first feature information; fusing the first feature information and the test index information by a feature fusion module to generate first fusion information; extracting features of the first fusion information by a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information by a softmax classifier to determine the diagnosis information corresponding to the patient, wherein The operation of extracting features from the intestinal internal image through a first feature extraction module to generate first feature information includes: extracting features from the intestinal internal image through a first feature extraction unit of the first feature extraction module to generate a first feature map; and extracting features from the first feature map through a second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information, wherein The operation of extracting features of the internal intestinal image by the first feature extraction unit of the first feature extraction module to generate a first feature map includes: extracting features of the internal intestinal image by the first convolution layer of the first feature extraction unit to generate a third feature map; processing the third feature map by the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; extracting features of the fourth feature map by the second convolution layer of the first feature extraction unit to generate a fifth feature map; and performing batch normalization processing on the fifth feature map by the second batch normalization layer of the first feature extraction unit to generate the first feature map, and wherein The operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector.

5. A colitis diagnostic device, characterized in that: include: An information generation module is used to generate diagnostic information based on the intestinal internal image and the test index information corresponding to the patient's blood test, wherein the diagnostic information is used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, wherein The operation of generating the diagnosis information according to the intestinal internal image and the test index information corresponding to the blood test of the patient includes: extracting features of the intestinal internal image by a first feature extraction module to generate first feature information; fusing the first feature information and the test index information by a feature fusion module to generate first fusion information; extracting features of the first fusion information by a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information by a softmax classifier to determine the diagnosis information corresponding to the patient, wherein The operation of extracting features from the intestinal internal image through a first feature extraction module to generate first feature information includes: extracting features from the intestinal internal image through a first feature extraction unit of the first feature extraction module to generate a first feature map; and extracting features from the first feature map through a second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information, wherein The operation of extracting features of the internal intestinal image by the first feature extraction unit of the first feature extraction module to generate a first feature map includes: extracting features of the internal intestinal image by the first convolution layer of the first feature extraction unit to generate a third feature map; processing the third feature map by the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; extracting features of the fourth feature map by the second convolution layer of the first feature extraction unit to generate a fifth feature map; and performing batch normalization processing on the fifth feature map by the second batch normalization layer of the first feature extraction unit to generate the first feature map, and wherein The operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector.

6. A colitis diagnostic device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Generate diagnostic information based on the intestinal internal image and the test index information corresponding to the patient's blood test, wherein the diagnostic information is used to indicate whether the patient suffers from ulcerative colitis or ulcerative colitis combined with cytomegalovirus colitis, The operation of generating the diagnosis information according to the intestinal internal image and the test index information corresponding to the blood test of the patient includes: extracting features of the intestinal internal image by a first feature extraction module to generate first feature information; fusing the first feature information and the test index information by a feature fusion module to generate first fusion information; extracting features of the first fusion information by a second feature extraction module to generate second feature information; fusing the first feature information and the second feature information to generate second fusion information; and classifying the second fusion information by a softmax classifier to determine the diagnosis information corresponding to the patient, wherein The operation of extracting features from the intestinal internal image through a first feature extraction module to generate first feature information includes: extracting features from the intestinal internal image through a first feature extraction unit of the first feature extraction module to generate a first feature map; and extracting features from the first feature map through a second feature extraction unit of the first feature extraction module to generate a second feature map as the first feature information, wherein The operation of extracting features of the internal intestinal image by the first feature extraction unit of the first feature extraction module to generate a first feature map includes: extracting features of the internal intestinal image by the first convolution layer of the first feature extraction unit to generate a third feature map; processing the third feature map by the first batch normalization layer and ReLU activation function of the first feature extraction unit to generate a fourth feature map; extracting features of the fourth feature map by the second convolution layer of the first feature extraction unit to generate a fifth feature map; and performing batch normalization processing on the fifth feature map by the second batch normalization layer of the first feature extraction unit to generate the first feature map, and wherein The operation of fusing the first feature information and the inspection index information through a feature fusion module to generate the first fusion information includes: performing global average pooling processing on the first feature information through the first global average pooling layer in the feature fusion module to generate a first feature vector; fusing the first feature vector and the inspection index information through the aggregation layer and the first fully connected layer in the feature fusion module to generate a bias vector and a channel weight vector; and generating the first fusion information according to the first feature information, the channel weight vector and the bias vector.

Images