System for evaluating severity of ulcerative colitis endoscope

Abstract

The invention relates to a system for evaluating the severity of an ulcerative colitis endoscope, and belongs to the technical field of medical systems. The system comprises a data acquisition module, a label calibration module, an ulcerative colitis endoscope severity evaluation model construction module, an ulcerative colitis endoscope severity evaluation model training module, an ulcerative colitis endoscope severity evaluation module and a second data acquisition module. A histological biopsy scoring and processing module; and a data processing and control module. The system provided by the invention can objectively evaluate the endoscopic disease activity of ulcerative colitis in real time and predict histological alleviation, can more accurately guide the formulation of a clinical treatment scheme, and is easy to popularize and apply.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical systems, and particularly relates to a system for evaluating the endoscopic severity of ulcerative colitis. Background Art

[0002] Ulcerative colitis (UC) is a chronic non-specific intestinal inflammatory disease, which has developed into a global disease in recent years, and its incidence and prevalence are increasing year by year. There is no gold standard for the diagnosis of UC. It mainly combines clinical, laboratory, imaging, endoscopic and histopathological manifestations for comprehensive analysis. Among them, colonoscopy and mucosal biopsy are the main basis for the diagnosis of UC and the key indicators for evaluating the treatment effect. However, the endoscopic score of ulcerative colitis is easily affected by the subjective factors of observers. There are mainly two scoring criteria for the endoscopic score of ulcerative colitis in clinical work, the Mayo endoscopic score (MES) and the ulcerative colitis endoscopic index of severity (UCEIS).

[0003] Machine learning is a branch discipline of artificial intelligence, which refers to the ability of a computer to apply algorithms to objectively imitate the pattern recognition ability of humans. CNN is a special neural network in machine learning, which overcomes the problem of inability to recognize images in traditional algorithms by maintaining the spatial relationship between pixels in an image.

[0004] Currently, there are some developments of artificial intelligence systems for evaluating the endoscopic severity of ulcerative colitis at home and abroad. Ozawa et al. developed a CNN system for differentiating Mayo endoscopic score remission (MES = 0, 1) and activity (MES = 2, 3). Stidham et al. also developed a CNN in a study for differentiating Mayo endoscopic score remission (MES = 0, 1) and activity (MES = 2, 3). However, these algorithms cannot give the specific score of MES, but can only distinguish remission or not. Subsequently, Bhambhvani et al. developed a CNN that can give the specific MES score. All of the above systems are based on a single MES scoring system, with relatively single functions. Most of them can only distinguish endoscopic remission or not, or predict histological remission, and cannot give specific endoscopic scores. At the same time, indicators such as accuracy, sensitivity, and specificity need to be further improved, and the efficiency and robustness of the system need to be further strengthened. At the same time, deep learning methods also show the potential to predict histological remission only using endoscopic images without mucosal biopsy samples. For example, Maeda et al. developed a real-time artificial intelligence system that can automatically predict histological active inflammation, but this system is also based on a single MES scoring system and does not evaluate the consistency between the UCEIS scoring system and histological results. Generally speaking, the transformation of artificial intelligence from a theoretical model to clinical practice still faces challenges, and its practicality and functional richness need to be further improved.

[0005] Therefore, how to overcome the deficiencies of the existing technology is an urgent problem to be solved in the current technical field. Summary of the Invention

[0006] The purpose of the present invention is to solve the deficiencies of the existing technology and provide a system for evaluating the endoscopic severity of ulcerative colitis.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A system for evaluating the endoscopic severity of ulcerative colitis, comprising:

[0009] A first data acquisition module, configured to acquire endoscopic images of ulcerative colitis patients and remove unclear images;

[0010] A label calibration module, connected to the first data acquisition module, configured to score the endoscopic images of ulcerative colitis patients and use the scores corresponding to each endoscopic image as its label;

[0011] The ulcerative colitis endoscopic severity assessment model construction module is used to construct an ulcerative colitis endoscopic severity assessment model; the ulcerative colitis endoscopic severity assessment model includes a Stem layer, a first Inception-ResNet layer, a first Reduction layer, a second Inception-ResNet layer, a second Reduction layer, a third Inception-ResNet layer, and a SoftMax layer connected in sequence;

[0012] The ulcerative colitis endoscopic severity assessment model training module is respectively connected to the first label calibration module, the label calibration module, and the ulcerative colitis endoscopic severity assessment model construction module, and is used to train and verify the ulcerative colitis endoscopic severity assessment model with the endoscopic images of ulcerative colitis patients as inputs and the labels corresponding to the endoscopic images as outputs;

[0013] The ulcerative colitis endoscopic severity assessment module is respectively connected to the first data acquisition module and the ulcerative colitis endoscopic severity assessment model training module, and is used to take the endoscopic images of the ulcerative colitis patients to be evaluated collected by the first data acquisition module as inputs and input them into the trained ulcerative colitis endoscopic severity assessment model, so as to obtain the ulcerative colitis endoscopic severity assessment result;

[0014] The second data acquisition module is connected to the first data acquisition module and is used to collect histological biopsy images of the corresponding ulcerative colitis patients;

[0015] The histological biopsy scoring and processing module is connected to the second data acquisition module and is used to score the biopsy images and obtain the histological judgment result corresponding to the score;

[0016] The data processing and control module is respectively connected to the first data acquisition module, the label calibration module, the ulcerative colitis endoscopic severity assessment model training module, the ulcerative colitis endoscopic severity assessment module, and the histological biopsy scoring and processing module. It is used to compare the endoscopic examination results corresponding to the evaluation results of the ulcerative colitis endoscopic severity assessment module with the histological judgment results of the histological biopsy scoring and processing module. If the accuracy rate of the ulcerative colitis endoscopic severity assessment exceeds the preset threshold, it controls the ulcerative colitis endoscopic severity assessment module to use the current ulcerative colitis endoscopic severity assessment model to evaluate the ulcerative colitis endoscopic severity. If the accuracy rate of the ulcerative colitis endoscopic severity assessment does not exceed the preset threshold, it controls the first data acquisition module to re-acquire the endoscopic images of new ulcerative colitis patients and remove the unclear images. And it controls the label calibration module to score the re-acquired endoscopic images of ulcerative colitis patients, and uses the scores corresponding to each endoscopic image as its label. Then it controls the ulcerative colitis endoscopic severity assessment model training module to re-train and validate the ulcerative colitis endoscopic severity assessment model by using the re-acquired endoscopic images, the previously acquired endoscopic images, and the labels corresponding to these images, so as to obtain a new ulcerative colitis endoscopic severity assessment model. Finally, it controls the ulcerative colitis endoscopic severity assessment module to use the new ulcerative colitis endoscopic severity assessment model to evaluate the ulcerative colitis endoscopic severity.

[0017] Further, preferably, in the first data acquisition module, the unclear images caused by feces, blur or halos are removed.

[0018] Further, preferably, in the label calibration module, the MES or UCEIS score is used to score the endoscopic images of ulcerative colitis patients.

[0019] Further, preferably, if MES = 0 points or UCEIS = 0 points, the endoscopic examination result is remission; if MES = 1 - 3 points or UCEIS = 1 - 8 points, the endoscopic examination result is active.

[0020] Further, preferably, in the ulcerative colitis endoscopic severity assessment model training module, the sample ratio of the training set used for training to the validation set used for validation is 3:1.

[0021] Further, preferably, in the ulcerative colitis endoscopic severity assessment model training module, it is also used to perform data augmentation on the endoscopic images, use the original images, the images after data augmentation, and their corresponding labels together as the sample set, and divide the sample set into a training set and a validation set.

[0022] Further, preferably, the data augmentation method includes horizontal flipping, vertical flipping, random scaling, and random rotation; the random rotation angle is 0-20°.

[0023] Further, preferably, in the histological biopsy scoring and processing module, when scoring the biopsy image, Geboes scoring is used. If Geboes ≤ 3.0, the histological judgment result is remission; if Geboes > 3.0, the histological judgment result is active inflammation.

[0024] Further, preferably, in the ulcerative colitis endoscopic severity assessment model construction module;

[0025] The Stem layer includes four sequentially connected 3×3 Convs, and a MaxPool is connected in parallel to the fourth 3×3 Conv. Then, the outputs of the fourth 3×3 Conv and a MaxPool are connected to a Filter. One output branch of this Filter is connected to another Filter through a 1×1 Conv and a 3×3 Conv; the other output branch of this Filter is connected to another Filter through a 1×1 Conv, a 7×1 Conv, a 1×7 Conv, and a 3×3 Conv; the output of the other Filter is respectively connected to a third Filter through a 3×3 Conv and a MaxPool;

[0026] The first Inception-ResNet layer includes two Relu activations, three 3×3 Convs, and four 1×1 Convs; the output of the first Relu activation is respectively connected to the inputs of the first to third 1×1 Convs; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the first 3×3 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the second 3×3 Conv and the third 3×3 Conv; the output of the first Relu activation and the output of the fourth 1×1 Conv are also connected to the input of the second Relu activation;

[0027] The first Reduction layer includes a MaxPool, three 3×3 Convs, two Filters, and a 1×1 Conv; the output of the first Filter is connected to the inputs of the MaxPool, the first 3×3 Conv, and the 1×1 Conv respectively; the output of the 1×1 Conv is connected to the input of the second Filter through the second 3×3 Conv and the third 3×3 Conv; the outputs of the MaxPool and the first 3×3 Conv are connected to the input of the second Filter respectively;

[0028] The second Inception-ResNet layer includes two Relu activations, three 1×1 Convs, a 7×1 Conv, and a 1×7 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×7 Conv, and 7×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the outputs of the first Relu activation and the third 1×1 Conv are also connected to the input of the second Relu activation;

[0029] The second Reduction layer includes a MaxPool, four 3×3 Convs, a Filter, and three 1×1 Convs; the output of the second Inception-ResNet layer is connected to the first to third 1×1 Convs and the MaxPool respectively; the output of the MaxPool is connected to the input of the Filter; the first 1×1 Conv is connected to the input of the Filter through the first 3×3 Conv and the second 3×3 Conv; the second 1×1 Conv is connected to the input of the Filter through the third 3×3 Conv; the third 1×1 Conv is connected to the input of the Filter through the fourth 3×3 Conv;

[0030] The third Inception-ResNet layer includes two Relu activations, three 1×1 Convs, one 1×3 Conv and one 3×1 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×3 Conv, and 3×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the output of the first Relu activation and the output of the third 1×1 Conv are also connected to the input of the second Relu activation;

[0031] The third Inception-ResNet layer is connected to the SoftMax layer through dropout;

[0032] The described SoftMax layer is a SoftMax classifier.

[0033] Furthermore, preferably, the preset threshold is 90%.

[0034] The present invention may further include a display module for displaying the evaluation result and the endoscopic examination result corresponding to the evaluation result (the endoscopic examination result is remission or activity). In predicting histological remission, the present invention compares the predicted histological remission with the predicted histological remission of the system of the present invention based on the pathologist's judgment of the activity and remission of histological images.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The present invention establishes an endoscopic severity assessment model for ulcerative colitis, which can objectively evaluate the endoscopic disease activity of ulcerative colitis in real time and predict histological remission, more accurately guide the formulation of clinical treatment plans, and enhance the practicality and richness of functions.

[0037] The accuracy of the CNN system developed by Bhambhvani et al. mentioned in the background technology, which can give specific MES scores, is 85.7%, the sensitivity is 72.4%, and the positive predictive value is 77.7%; the accuracy of the system of the present invention in predicting MES scores is 97.04, the sensitivity is 98.43, and the positive predictive value is 96.36. The accuracy of the real-time artificial intelligence system developed by Maeda et al. mentioned in the background technology in predicting histological active inflammation is 81.5%, and the evaluation accuracy of the present system is as high as 91.28%. Brief Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic structural diagram of the system for evaluating the endoscopic severity of ulcerative colitis according to the present invention;

[0040] Figure 2 It is a schematic structural diagram of the evaluation model for the endoscopic severity of ulcerative colitis;

[0041] Figure 3 It is a schematic flow diagram of the system for evaluating the endoscopic severity of ulcerative colitis according to the present invention;

[0042] Figure 4 It is a confusion matrix diagram of the training results of the application example of the present invention. Detailed implementation manners

[0043] The following further describes the present invention in detail in combination with embodiments.

[0044] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the description of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" to another element, it can be directly connected to other elements, or there may also be intermediate elements. The phrase "and / or" used herein includes any unit and all combinations of one or more related listed items.

[0045] In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more. The orientation or state relationships indicated by terms such as "inside", "above", "below", etc. are based on the orientation or state relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "installation", "connection", and "provided with" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as the general understanding of those of ordinary skill in the field to which the present invention belongs. It should also be understood that terms defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as here.

[0048] As Figure 1 shown, a system for evaluating the endoscopic severity of ulcerative colitis includes:

[0049] A first data acquisition module 101, configured to acquire endoscopic images of ulcerative colitis patients and remove unclear images;

[0050] A label calibration module 102, connected to the first data acquisition module 101, configured to score the endoscopic images of ulcerative colitis patients and use the scores corresponding to each endoscopic image as its label;

[0051] A module 103 for constructing an evaluation model of the endoscopic severity of ulcerative colitis, configured to construct an evaluation model of the endoscopic severity of ulcerative colitis; the evaluation model of the endoscopic severity of ulcerative colitis includes a Stem layer, a first Inception-ResNet layer, a first Reduction layer, a second Inception-ResNet layer, a second Reduction layer, a third Inception-ResNet layer, and a SoftMax layer connected in sequence;

[0052] A training module 104 for the evaluation model of the endoscopic severity of ulcerative colitis, connected to the first label calibration module 102, the label calibration module 103, and the module 103 for constructing the evaluation model of the endoscopic severity of ulcerative colitis respectively, configured to use the endoscopic images of ulcerative colitis patients as input and the labels corresponding to the endoscopic images as output to train and verify the evaluation model of the endoscopic severity of ulcerative colitis;

[0053] The ulcerative colitis endoscopic severity assessment module 105 is respectively connected to the first data acquisition module 101 and the ulcerative colitis endoscopic severity assessment model training module 104, and is used to take the endoscopic images of the ulcerative colitis patients to be evaluated collected by the first data acquisition module 101 as input and input them into the trained ulcerative colitis endoscopic severity assessment model, so as to obtain the ulcerative colitis endoscopic severity assessment result;

[0054] The second data acquisition module 106 is connected to the first data acquisition module 101 and is used to collect the histological biopsy images of the corresponding ulcerative colitis patients;

[0055] The histological biopsy scoring and processing module 107 is connected to the second data acquisition module 106 and is used to score the biopsy images and obtain the histological judgment result corresponding to the score;

[0056] The data processing and control module 108 is respectively connected to the first data acquisition module 101, the label calibration module 102, the ulcerative colitis endoscopic severity assessment model training module 104, the ulcerative colitis endoscopic severity assessment module 105, and the histological biopsy scoring and processing module 107, and is used to compare the endoscopic examination result corresponding to the assessment result of the ulcerative colitis endoscopic severity assessment module 105 with the histological judgment result of the histological biopsy scoring and processing module 107. If the accuracy rate of the ulcerative colitis endoscopic severity assessment exceeds the preset threshold, the ulcerative colitis endoscopic severity assessment module 105 is controlled to use the current ulcerative colitis endoscopic severity assessment model to perform the ulcerative colitis endoscopic severity assessment; if the accuracy rate of the ulcerative colitis endoscopic severity assessment does not exceed the preset threshold, the first data acquisition module 101 is controlled to re-collect the endoscopic images of new ulcerative colitis patients and remove the unclear images; and the label calibration module 102 is controlled to score the re-collected endoscopic images of the ulcerative colitis patients, and use the scores corresponding to each endoscopic image as its label; then the ulcerative colitis endoscopic severity assessment model training module 104 is controlled to use the re-collected endoscopic images, the previously collected endoscopic images and the labels corresponding to these images to re-train and verify the ulcerative colitis endoscopic severity assessment model to obtain a new ulcerative colitis endoscopic severity assessment model, and finally the ulcerative colitis endoscopic severity assessment module 105 is controlled to use the new ulcerative colitis endoscopic severity assessment model to perform the ulcerative colitis endoscopic severity assessment.

[0057] In the first data acquisition module 101, unclear images caused by feces, blur or halos are removed.

[0058] In the label calibration module 102, the endoscopic images of patients with ulcerative colitis are scored using the MES or UCEIS score.

[0059] If MES = 0 or UCEIS = 0, the endoscopic examination result is remission; if MES = 1 - 3 or UCEIS = 1 - 8, the endoscopic examination result is active.

[0060] In the ulcerative colitis endoscopic severity assessment model training module 105, the sample ratio of the training set used for training and the validation set used for validation is 3:1.

[0061] In the ulcerative colitis endoscopic severity assessment model training module 105, it is also used to perform data augmentation on the endoscopic images. The original image, the data-augmented image, and their corresponding labels are used together as a sample set, and the sample set is divided into a training set and a validation set.

[0062] The data augmentation methods include horizontal flipping, vertical flipping, random scaling, and random rotation; the random rotation angle is 0 - 20°.

[0063] In the histological biopsy scoring and processing module 107, when scoring the biopsy images, the Geboes score is used. If Geboes ≤ 3.0, the histological judgment result is remission; if Geboes > 3.0, the histological judgment result is active inflammation.

[0064] In the ulcerative colitis endoscopic severity assessment model construction module 103;

[0065] The Stem layer includes four sequentially connected 3×3 Convs, and a MaxPool is connected in parallel to the fourth 3×3 Conv. Then, the outputs of the fourth 3×3 Conv and a MaxPool are connected to a Filter. One output branch of this Filter is connected to another Filter through a 1×1 Conv and a 3×3 Conv; the other output branch of this Filter is connected to another Filter through a 1×1 Conv, a 7×1 Conv, a 1×7 Conv, and a 3×3 Conv; the output of the other Filter is respectively connected to a third Filter through a 3×3 Conv and a MaxPool;

[0066] The first Inception-ResNet layer includes two Relu activations, three 3×3 Convs, and four 1×1 Convs; the output of the first Relu activation is connected to the inputs of the first to third 1×1 Convs respectively; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the first 3×3 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the second 3×3 Conv and the third 3×3 Conv; the output of the first Relu activation and the output of the fourth 1×1 Conv are also connected to the input of the second Relu activation;

[0067] The first Reduction layer includes a MaxPool, three 3×3 Convs, two Filters, and a 1×1 Conv; the output of the first Filter is connected to the inputs of the MaxPool, the first 3×3 Conv, and the 1×1 Conv respectively; the output of the 1×1 Conv is connected to the input of the second Filter through the second 3×3 Conv and the third 3×3 Conv; the outputs of the MaxPool and the first 3×3 Conv are connected to the input of the second Filter respectively;

[0068] The second Inception-ResNet layer includes two Relu activations, three 1×1 Convs, a 7×1 Conv, and a 1×7 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×7 Conv, and 7×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the output of the first Relu activation and the output of the third 1×1 Conv are also connected to the input of the second Relu activation;

[0069] The second Reduction layer includes a MaxPool, four 3×3 Convs, a Filter, and three 1×1 Convs; the outputs of the second Inception-ResNet layer are respectively connected to the first to third 1×1 Convs and the MaxPool; the output of the MaxPool is connected to the input of the Filter; the output of the first 1×1 Conv is connected to the input of the Filter through the first 3×3 Conv and the second 3×3 Conv; the output of the second 1×1 Conv is connected to the input of the Filter through the third 3×3 Conv; the output of the third 1×1 Conv is connected to the input of the Filter through the fourth 3×3 Conv;

[0070] The third Inception-ResNet layer includes two Relu activations, three 1×1 Convs, a 1×3 Conv, and a 3×1 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×3 Conv, and 3×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the output of the first Relu activation and the output of the third 1×1 Conv are also connected to the input of the second Relu activation;

[0071] The third Inception-ResNet layer is connected to the SoftMax layer through dropout;

[0072] The said SoftMax layer is a SoftMax classifier.

[0073] The preset threshold is 90%.

[0074] Application example

[0075] The system for evaluating the endoscopic severity of ulcerative colitis provided in this example has two ulcerative colitis endoscopic severity evaluation models based on the MES score and the UCEIS score;

[0076] We collected the clinical data of UC patients who underwent endoscopic examinations in the Department of Gastroenterology, the First Affiliated Hospital of Kunming Medical University from January 2018 to December 2022. According to the Lennard-Jones criteria, based on typical clinical manifestations, endoscopic examinations, and histological examinations, a total of 1124 UC patients were diagnosed with ulcerative colitis and included in the study. The Truelove and Witts criteria were used to classify the disease activity and severity. After excluding unclear images due to feces, blurring, or halos, a total of 12,257 endoscopic images were collected. We used 9807 images of 872 patients from January 2018 to June 2021 as the training set. 2450 endoscopic images of 252 UC patients from July 2021 to December 2022 were used as the validation set. Among the 252 UC patients in the validation set, 218 patients underwent biopsies, and their clinical data, 2127 endoscopic images, and 763 biopsy images were used to evaluate the performance of predicting histological remission. Before training, data augmentation was first performed. The data augmentation methods included horizontal flipping, vertical flipping, random scaling, and random rotation; the random rotation angle was 0-20°.

[0077] Several endoscopy experts with more than ten years of practice scored the endoscopic images according to the MES and UCEIS scoring systems. If there were different scores, several independent endoscopy experts with more than 15 years of practice made the final decision after discussion. MES = 0 or UCEIS = 0 was defined as endoscopic remission. At the same time, we used the Geboes score to evaluate the histological severity of inflammation. Remission was defined as Geboes ≤ 3.0, and active inflammation was defined as Geboes > 3.0. All histological images were evaluated by several pathologists with more than 10 years of practice. If the opinions of several pathologists were inconsistent, the final result was determined through discussion. The pathologists and endoscopists made the evaluations without knowing any clinical information.

[0078] The ulcerative colitis endoscopic severity assessment model includes a sequentially connected Stem layer, a first Inception-ResNet layer, a first Reduction layer, a second Inception-ResNet layer, a second Reduction layer, a third Inception-ResNet layer, and a SoftMax layer; by integrating the "residual" structure proposed in ResNet into the Inception module, we accelerated the training and improved the performance. The Inception module enabled us to capture sparse and non-sparse features of the same layer and use 1×1 convolutions to reduce the number of parameters, improve the recognition speed, and promote faster network convergence. Dropout was used to reduce the weights and improve the network robustness. Finally, the SoftMax classifier was applied to calculate the corresponding probabilities.

[0079] We compared the MES and UCEIS scores of each endoscopic image one - to - one with the histological results to predict the histological remission of ulcerative colitis. The experiments were conducted using Windows 10, Spyder editor, and SPSS (v26.0) software. The computer settings included: a CPU of the AMD Ryzen 7 model and a GPU of the NVIDIA GeForce RTX 2080Ti model. All programs were implemented using the open - source framework Keras30, with a TensorFlow backend and a Python port.

[0080] 1. The ability of the system of the present invention to predict disease activity and remission

[0081] When the system of the present invention based on the MES score predicts the remission or activity of endoscopic pictures, that is, predicts MES = 0, the accuracy is 97.04, the 95% confidence interval is [96.26:97.62], the sensitivity is 98.43, the 95% confidence interval is [97.84:99.01], and the positive predictive value is 96.36, the 95% confidence interval is [95.02:97.69]. When the system of the present invention based on the MES score predicts disease activity, that is, predicts MES 1 - 3, the accuracy is 90.15, the 95% confidence interval is [89.49:90.82], the sensitivity is 83.66, the 95% confidence interval is [82.94:84.38], and the positive predictive value is 85.60, the 95% confidence interval is [83.39:87.81]. When the system of the present invention based on the UCEIS score predicts disease activity, that is, predicts UCEIS 1 - 8, the accuracy is 85.29 [84.23:86.35], the 95% confidence interval is [84.23:86.35], the sensitivity is 77.75, the 95% confidence interval is [76.51:78.98], and the positive predictive value is 67.07, the 95% confidence interval is [65.58:68.56].

[0082] Table 2 The system of the present invention predicts disease activity and remission

[0083]

[0084] 2) Confusion matrix

[0085] The confusion matrix of the training results of the examples of the present invention is as Figure 4 shown, where Figure A is the system of the present invention based on the MES score predicting endoscopic pictures as endoscopic remission or activity, Figure B is the system of the present invention based on the MES score predicting disease activity, and Figure C is the system of the present invention based on the UCEIS score predicting disease activity.

[0086] 3) The system of the present invention predicts the accuracy of each label

[0087] When the system of the present invention based on the MES score predicts that the endoscopic image is in endoscopic remission or active, there are a total of 1450 endoscopic images with MES = 0, among which 1424 are correctly identified by the CNN model, and the accuracy is 98.21%. There are a total of 1000 endoscopic images with MES = 1 - 3, among which 954 are correctly identified by the system of the present invention, and the accuracy is 95.40%. When the system of the present invention based on the MES score predicts the disease activity, similarly, the accuracy of predicting MES = 0 is 98.41%, the accuracy of MES = 1 is 71.80%, the accuracy of MES = 2 is 85.85%, and the accuracy of MES = 3 is 77.57%. When the system of the present invention based on the UCEIS score predicts the disease activity, the accuracy of predicting UCEIS = 0 is 98.41%, the accuracy of UCEIS = 1 is 78.41%, the accuracy of UC EIS = 2 is 64.02%, the accuracy of UCEIS = 3 is 62.25%, the accuracy of UCEIS = 4 is 72.67%, the accuracy of UCEIS = 5 is 59.78%, the accuracy of UCEIS = 6 is 65.61%, the accuracy of UCEIS = 7 is 66.66%, and the accuracy of UCEIS = 7 is 77.61%.

[0088] Table 2 Accuracy of predicting disease activity based on MES and UCEIS scores

[0089]

[0090]

[0091] 2. Ability to predict histological remission

[0092] Taking the results of histological biopsy images as the gold standard, according to the biopsy results, a total of 113 patients achieved histological remission and 107 patients were in histological activity. The system of the present invention correctly judged that 102 patients achieved histological remission, misjudged 11 patients who had achieved histological remission as being in histological activity, correctly judged that 97 patients were in histological activity, and misjudged 8 patients who were still in histological activity as being in histological remission. After calculation, the accuracy of the CNN in judging histology is 91.28%, and the Kappa coefficient is 0.826.

[0093] Table 3 Predictive value for histological remission

[0094]

[0095] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An endoscopic severity assessment system for ulcerative colitis, characterized in that, Including: A first data acquisition module (101) for acquiring endoscopic images of ulcerative colitis patients and removing unclear images; A label calibration module (102), connected to the first data acquisition module (101), for scoring the endoscopic images of ulcerative colitis patients and using the scores corresponding to each endoscopic image as its label; A module (103) for constructing an endoscopic severity assessment model of ulcerative colitis, used to construct an endoscopic severity assessment model of ulcerative colitis; the endoscopic severity assessment model of ulcerative colitis includes a sequentially connected Stem layer, a first Inception-ResNet layer, a first Reduction layer, a second Inception-ResNet layer, a second Reduction layer, a third Inception-ResNet layer, and a SoftMax layer; A training module (104) for the endoscopic severity assessment model of ulcerative colitis, connected to the first label calibration module (102), the label calibration module (103), and the module (103) for constructing the endoscopic severity assessment model of ulcerative colitis respectively, for training and validating the endoscopic severity assessment model of ulcerative colitis with the endoscopic images of ulcerative colitis patients as the input and the labels corresponding to the endoscopic images as the output; An endoscopic severity assessment module (105) of ulcerative colitis, connected to the first data acquisition module (101) and the training module (104) for the endoscopic severity assessment model of ulcerative colitis respectively, for using the endoscopic images of the ulcerative colitis patients to be evaluated collected by the first data acquisition module (101) as the input and inputting them into the trained endoscopic severity assessment model of ulcerative colitis, so as to obtain the endoscopic severity assessment result of ulcerative colitis; A second data acquisition module (106), connected to the first data acquisition module (101), for acquiring histological biopsy images of the corresponding ulcerative colitis patients; A histological biopsy scoring and processing module (107), connected to the second data acquisition module (106), for scoring the biopsy images and obtaining the histological judgment result corresponding to the score; The data processing and control module (108) is respectively connected to the first data acquisition module (101), the label calibration module (102), the ulcerative colitis endoscopic severity assessment model training module (104), the ulcerative colitis endoscopic severity assessment module (105), and the histological biopsy scoring and processing module (107). It is used to compare the endoscopic examination results corresponding to the evaluation results of the ulcerative colitis endoscopic severity assessment module (105) with the histological judgment results of the histological biopsy scoring and processing module (107). If the accuracy rate of the ulcerative colitis endoscopic severity assessment exceeds the preset threshold, it controls the ulcerative colitis endoscopic severity assessment module (105) to use the current ulcerative colitis endoscopic severity assessment model to evaluate the ulcerative colitis endoscopic severity. If the accuracy rate of the ulcerative colitis endoscopic severity assessment does not exceed the preset threshold, it controls the first data acquisition module (101) to re-acquire the endoscopic images of new ulcerative colitis patients and remove the unclear images. And it controls the label calibration module (102) to score the re-acquired endoscopic images of ulcerative colitis patients, and takes the scores corresponding to each endoscopic image as its label. Then it controls the ulcerative colitis endoscopic severity assessment model training module (104) to use the re-acquired endoscopic images, the previously acquired endoscopic images and the labels corresponding to these images to re-train and validate the ulcerative colitis endoscopic severity assessment model to obtain a new ulcerative colitis endoscopic severity assessment model. Finally, it controls the ulcerative colitis endoscopic severity assessment module (105) to use the new ulcerative colitis endoscopic severity assessment model to evaluate the ulcerative colitis endoscopic severity.

2. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, characterized in that, In the first data acquisition module (101), the unclear images caused by feces, blurring or halos are removed.

3. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein, In the label calibration module (102), the MES or UCEIS score is used to score the endoscopic images of ulcerative colitis patients.

4. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 2, wherein If MES = 0 points or UCEIS = 0 points, the endoscopic examination result is remission. If MES = 1 - 3 points or UCEIS = 1 - 8 points, the endoscopic examination result is active.

5. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein, In the ulcerative colitis endoscopic severity assessment model training module (105), the sample ratio of the training set used for training to the validation set used for validation is 3:

1.

6. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein In the ulcerative colitis endoscopic severity assessment model training module (105), it is also used to perform data augmentation on the endoscopic images, and uses the original images, the data-augmented images and their corresponding labels together as a sample set, and divides the sample set into a training set and a validation set.

7. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 6, wherein The data augmentation methods include horizontal flipping, vertical flipping, random scaling and random rotation; the random rotation angle is 0 - 20°.

8. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein In the histological biopsy scoring and processing module (107), when scoring the biopsy images, the Geboes score is used. If Geboes ≤ 3.0, the histological judgment result is remission. If Geboes > 3.0, the histological judgment result is active inflammation.

9. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein In the ulcerative colitis endoscopic severity assessment model construction module (103); The Stem layer includes four sequentially connected 3×3 Convs, and a MaxPool is connected in parallel to the fourth 3×3 Conv. Then, the outputs of the fourth 3×3 Conv and a MaxPool are connected to a Filter. One output branch of this Filter is connected to another Filter through a 1×1 Conv and a 3×3 Conv; the other output branch of this Filter is connected to another Filter through a 1×1 Conv, a 7×1 Conv, a 1×7 Conv, and a 3×3 Conv; the output of another Filter is connected to a third Filter through a 3×3 Conv and a MaxPool respectively; The first Inception-ResNet layer includes two Relu activations, three 3×3 Convs, and four 1×1 Convs; the output of the first Relu activation is connected to the inputs of the first to third 1×1 Convs respectively; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the first 3×3 Conv; the output of the first 1×1 Conv is connected to the input of the fourth 1×1 Conv through the second 3×3 Conv and the third 3×3 Conv; the output of the first Relu activation and the output of the fourth 1×1 Conv are also connected to the input of the second Relu activation; The first Reduction layer includes a MaxPool, three 3×3 Convs, two Filters, and a 1×1 Conv; the output of the first Filter is connected to the inputs of the MaxPool, the first 3×3 Conv, and the 1×1 Conv respectively; the output of the 1×1 Conv is connected to the input of the second Filter through the second 3×3 Conv and the third 3×3 Conv; the outputs of the MaxPool and the first 3×3 Conv are connected to the input of the second Filter respectively; The second Inception-ResNet layer includes two Relu activations, three 1×1 Convs, one 7×1 Conv, and one 1×7 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×7 Conv, and 7×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the output of the first Relu activation and the output of the third 1×1 Conv are also connected to the input of the second Relu activation; The second Reduction layer includes one MaxPool, four 3×3 Convs, one Filter, and three 1×1 Convs; the output of the second Inception-ResNet layer is connected to the first to third 1×1 Convs and MaxPool respectively; the output of MaxPool is connected to the input of Filter; the first 1×1 Conv is connected to the input of Filter through the first 3×3 Conv and the second 3×3 Conv; the second 1×1 Conv is connected to the input of Filter through the third 3×3 Conv; the third 1×1 Conv is connected to the input of Filter through the fourth 3×3 Conv; The third Inception-ResNet layer includes two Relu activations, three 1×1 Convs, one 1×3 Conv, and one 3×1 Conv; one output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the first 1×1 Conv, 1×3 Conv, and 3×1 Conv; the other output branch of the first Relu activation is connected to the input of the third 1×1 Conv through the second 1×1 Conv; the output of the first Relu activation and the output of the third 1×1 Conv are also connected to the input of the second Relu activation; The third Inception-ResNet layer is connected to the SoftMax layer through dropout; The SoftMax layer mentioned above is a SoftMax classifier.

10. The system for evaluating the endoscopic severity of ulcerative colitis according to claim 1, wherein The preset threshold is 90%.

Images