Crohn disease mucosa healing prediction method and system

Abstract

The invention relates to the field of artificial intelligence, in particular to a Crohn disease mucosa healing prediction method and system, and aims to improve the prediction accuracy. The prediction system comprises a preprocessing module and a neural network model. The preprocessing module is used for preprocessing the pre-treatment original ultrasonic image of the to-be-predicted lesion intestinal segment and the corresponding original clinical data; the neural network model comprises a feature extraction module, a first preset number of residual modules, a fusion module and a judgment module. The feature extraction module is used for acquiring a first feature map from the preprocessed ultrasonic image; the first preset number of residual modules are connected in series and are used for extracting a second feature map with deep features from the first feature map; the fusion module fuses the second feature map and the preprocessed clinical data; and the judgment module judges whether the to-be-predicted diseased intestinal segment can achieve mucosa healing according to the fusion features. The attention information is added for the image features by using the clinical data, so that the prediction accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and system for predicting mucosal healing of Crohn's disease. Background Art

[0002] Crohn's disease (CD) is a chronic inflammatory bowel disease of the digestive tract with unknown etiology, often involving the small intestine. It is difficult to differentiate and has many refractory patients. The disease is protracted and recurrent, often leading to loss of labor, and seriously affecting the patient's physical and mental health. In recent years, the global incidence of Crohn's disease has been on the rise, and the incidence in my country has increased dramatically in the past 20 years.

[0003] Intestinal mucosal healing (MH) is the treatment endpoint of Crohn's disease. However, the heterogeneity of Crohn's disease patients' responses to drugs is strong and lacks regularity, which poses a great challenge to the clinical selection of appropriate drugs. The repeated dressing changes increase the economic burden of patients, increase the surgical rate, and increase the difficulty of achieving intestinal mucosal healing. Therefore, if the possibility of mucosal healing can be accurately predicted, it can help doctors preliminarily judge the effectiveness of the current treatment plan and adjust the treatment strategy in a timely manner.

[0004] In the prior art, a prediction model is established based on manually summarized ultrasound quantitative indicators (such as intestinal wall thickness, blood flow typing, the presence or absence of complications, etc.) and clinical indicators. However, this prediction model extracts less effective information, so it has low accuracy and limited clinical application. Summary of the invention

[0005] In order to solve the above problems in the prior art, the present invention proposes a method and system for predicting mucosal healing of Crohn's disease, which improves the accuracy of prediction.

[0006] In a first aspect of the present invention, a system for predicting mucosal healing of Crohn's disease is provided, the system comprising:

[0007] A preprocessing module, used for preprocessing the original ultrasound image of the intestinal segment to be predicted to have a lesion and the corresponding original clinical data before treatment;

[0008] A neural network model is used to calculate whether the intestinal segment to be predicted to have a lesion can achieve mucosal healing based on the preprocessed clinical data and the preprocessed ultrasound image;

[0009] Wherein, the neural network model has been trained and includes:

[0010] A feature extraction module, used for acquiring a first feature map from the preprocessed ultrasound image;

[0011] A first preset number of residual modules are connected in series and used to extract deep feature representations from the first feature map to obtain a second feature map;

[0012] A fusion module, used for fusing the second feature map with the preprocessed clinical data to obtain a fusion feature;

[0013] A judgment module is used to judge whether the intestinal segment to be predicted with lesions can achieve mucosal healing according to the fusion characteristics.

[0014] Preferably, the original clinical data includes: categorical data and numerical data;

[0015] The preprocessing module comprises:

[0016] A clinical data conversion unit, used for converting the categorical data in the original clinical data into hot-single-hot encoding and keeping the numerical data unchanged, thereby obtaining the preprocessed clinical data;

[0017] A rectangular frame cutting unit, used for cutting out a rectangular frame image containing a lesion from the original ultrasound image;

[0018] An image scaling unit, used for scaling the rectangular frame image according to a preset size;

[0019] The pixel value normalization unit is used to normalize each pixel value in the scaled rectangular frame image to a range of [-1, 1], so as to obtain the preprocessed ultrasound image.

[0020] Preferably, the feature extraction module comprises:

[0021] A ResNet50 network is used to preliminarily extract a feature map from the preprocessed ultrasound image;

[0022] 7x7 convolutional layer, used to reduce the size of the initially extracted feature map;

[0023] The maximum delay layer is used to perform a dimensionality reduction operation on the feature map after the size reduction, so as to obtain the first feature map.

[0024] Preferably, each of the residual modules comprises:

[0025] The first 3x3 convolutional layer and the second 3x3 convolutional layer are connected in series to perform deep extraction of input features of the residual module;

[0026] A Squeeze-and-excitation (SE) unit, used to introduce an attention mechanism into the output features of the second 3x3 convolutional layer;

[0027] A first 1x1 convolutional layer, configured to receive the input features of the residual module and adjust the dimensions to match the output feature dimensions of the compression and excitation unit;

[0028] A first adding unit is used to add the output of the first 1x1 convolutional layer to the output of the compression and excitation unit.

[0029] Preferably, the compression and excitation unit comprises:

[0030] A first global pooling layer, used to compress the information of each channel in the output features of the second 3x3 convolutional layer into a single value;

[0031] A first fully connected layer, used to reduce the number of channels of the output features of the first global pooling layer to reduce the amount of calculation and remove redundant information;

[0032] A ReLU activation function is used to add nonlinear features to the output of the first fully connected layer;

[0033] The second fully connected layer is used to restore the number of channels of the output feature of the ReLU activation function and output a vector with the same number of channels as the output feature of the second 3x3 convolutional layer; each element in the vector corresponds to the weight of a channel, indicating the importance of the channel;

[0034] A sigmoid activation function is used to map the output of the second fully connected layer to between 0 and 1 to obtain the attention weight of each channel;

[0035] The first attention unit is used to receive the output features of the second 3x3 convolutional layer and add the attention weight.

[0036] Preferably, the fusion module includes:

[0037] A second global pooling layer, used to compress the information of each channel in the second feature map into a single value;

[0038] A splicing unit, used for splicing the output features of the second global pooling layer with the preprocessed clinical data to obtain a spliced ​​feature;

[0039] A third fully connected layer and a fourth fully connected layer connected in series, used to extract higher-level abstract features from the concatenated features;

[0040] A second attention unit, configured to add attention information to the second feature map using the abstract feature;

[0041] The third 3x3 convolutional layer and the fourth 3x3 convolutional layer are connected in series to extract features from the feature map with attention information added;

[0042] A second 1x1 convolutional layer is used to receive the second feature map and adjust the dimension to match the output feature dimension of the fourth 3x3 convolutional layer;

[0043] The second adding unit is used to add the output of the second 1x1 convolutional layer and the output of the fourth 3x3 convolutional layer to obtain the fusion feature.

[0044] Preferably, the judging module comprises:

[0045] A third global pooling layer is used to compress the information of each channel in the fused feature into a single value;

[0046] The fifth fully connected layer is used to perform binary classification according to the output of the third global pooling layer, so as to determine whether the diseased intestinal segment to be predicted can achieve mucosal healing.

[0047] Preferably, the system further comprises: a training module;

[0048] The training module includes:

[0049] A raw data collection unit, used to collect a second preset number of ultrasound image samples of the diseased intestinal segments before treatment, clinical data corresponding to each diseased intestinal segment, and a label value indicating whether the diseased intestinal segment has achieved mucosal healing, so as to construct a raw data set; the label value is obtained by follow-up colonoscopy and / or cross-sectional imaging assessment;

[0050] A data processing unit, configured to call the preprocessing module to preprocess each ultrasound image sample and corresponding clinical data in the original data set;

[0051] An image amplification unit, used for horizontally, vertically and randomly flipping each pre-processed ultrasound image to perform image amplification;

[0052] A data division unit, used to divide the amplified image according to a preset ratio, and then construct a training set and a validation set using the amplified image and the preprocessed clinical data based on the correspondence between each image and the clinical data;

[0053] A training unit, used for selecting one image and corresponding clinical data from the training set each time and inputting them into the neural network model to obtain a prediction value, calculating a loss function, adjusting the parameters of the neural network model, and repeating the process until the loss function no longer decreases, thereby obtaining a trained neural network model;

[0054] The verification unit is used to perform a 5-fold cross-validation on the trained neural network model using the verification set, and to perform external verification and reliability verification on the trained neural network model.

[0055] Preferably, the first preset number is 3;

[0056] The categorical data include: patient gender, lesion location, smoking history, previous intestinal surgery history, medication, and medication type;

[0057] The numerical data include: patient age, disease duration, simplified Crohn's disease activity index, and high-sensitivity C-reactive protein level;

[0058] The preset size is: 512x1024.

[0059] A second aspect of the present invention provides a method for predicting mucosal healing of Crohn's disease, the method comprising:

[0060] Preprocessing the original ultrasound images and corresponding original clinical data of the intestinal segment to be predicted to have lesions before treatment;

[0061] Inputting the preprocessed clinical data and the preprocessed ultrasound images into the neural network model to determine whether the intestinal segment to be predicted can achieve mucosal healing;

[0062] in,

[0063] The neural network model has been trained and includes:

[0064] A feature extraction module, used for acquiring a first feature map from the preprocessed ultrasound image;

[0065] A first preset number of residual modules are connected in series and used to extract deep feature representations from the first feature map to obtain a second feature map;

[0066] A fusion module, used for fusing the second feature map with the preprocessed clinical data to obtain a fusion feature;

[0067] A judgment module is used to judge whether the intestinal segment to be predicted with lesions can achieve mucosal healing according to the fusion characteristics.

[0068] The present invention has the following beneficial effects:

[0069] The Crohn's disease mucosal healing prediction system proposed in the present invention first uses a feature extraction module to preliminarily extract a first feature map from ultrasound image data, then uses three series-connected residual blocks to extract deep-level feature representations to obtain a second feature map, and then fuses the second feature map with clinical data, and finally determines whether the diseased intestinal segment to be predicted can achieve mucosal healing based on the fused features. The neural network model can extract information in ultrasound images that the human eye cannot distinguish, and then use clinical data to add attention information to image features. Therefore, the neural network model trained with a large amount of data can improve the accuracy of prediction. The residual module can alleviate the gradient vanishing problem of the deep neural network model during back propagation. By connecting multiple residual modules in series, the present invention can gradually extract more complex and abstract features, thereby improving the accuracy and robustness of feature extraction.

[0070] In the preprocessing module, the rectangular frame image containing the lesion site is cropped so that only the rectangular frame image can be input into the neural network model for judgment later, which can prevent confusing information from being introduced into the model.

[0071] In the feature extraction module, using 7x7 convolutional layers and maximum pooling layers on the feature maps extracted by ResNet50 can further reduce the size of the feature maps, increase the receptive field, and reduce the amount of computation.

[0072] Adding compression and excitation units to the residual module can guide the model to focus on important channel features.

[0073] In summary, the present invention pre-processes the original ultrasound images and original clinical data and inputs them into the neural network model to predict whether the diseased intestinal segment can achieve mucosal healing, which effectively improves the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the main structure of an embodiment of a Crohn's disease mucosal healing prediction system in the present invention;

[0075] Figure 2 It is a main structural schematic diagram of the preprocessing module in an embodiment of the present invention;

[0076] Figure 3 It is a schematic diagram of the main structure of the feature extraction module in the neural network model of an embodiment of the present invention;

[0077] Figure 4 Schematic diagram of the main structure of the residual module in the neural network model of the embodiment of the present invention;

[0078] Figure 5 It is a schematic diagram of the main structure of the fusion module in the neural network model of an embodiment of the present invention;

[0079] Figure 6It is a schematic diagram of the main structure of the judgment module in the neural network model of an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] It should be noted that, in the description of the present invention, the terms "first" and "second" are only for the convenience of description, and do not indicate or imply the relative importance of the devices, elements or parameters, and therefore cannot be understood as limiting the present invention. In addition, the term "and / or" in the present invention is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0083] Ultrasound examination has the advantages of being easy to operate, fast, radiation-free and repeatable. Ultrasound images can be used to comprehensively evaluate intestinal lesions, and the activity of inflammation can be evaluated through the full layer of the small intestinal wall, intestinal wall blood flow and intestinal periphery performance. Artificial intelligence can process a large amount of data, extract information that cannot be distinguished by the human eye from ultrasound images, and combine it with clinical data such as medication information to achieve the function of predicting intestinal mucosal healing. Therefore, in the present invention, ultrasound images and clinical data are input into the neural network model together for prediction.

[0084] Figure 1 Schematic diagram of the main structure of the embodiment of the Crohn's disease mucosal healing prediction system of the present invention. Figure 1 As shown, the prediction system of this embodiment includes: a preprocessing module 110 and a neural network model 120.

[0085] In this embodiment, the preprocessing module 110 is used to preprocess the original ultrasound image and corresponding original clinical data of the predicted diseased intestinal segment before treatment; the neural network model 120 is used to calculate whether the predicted diseased intestinal segment can achieve mucosal healing based on the preprocessed clinical data and the preprocessed ultrasound image.

[0086] Among them, the neural network model 120 has been trained and includes: a feature extraction module 121, a first preset number of residual modules (3 residual modules 122, 123 and 124), a fusion module 125 and a judgment module 126.

[0087] The biggest role of the residual module is to alleviate the gradient vanishing problem. Due to the stacked multi-layer network structure, the deep neural network model is prone to the gradient vanishing problem during back propagation, which makes training difficult. The jump connection in the residual module allows the gradient to bypass the middle layer directly during back propagation and pass to the earlier layer, alleviating the risk of gradient vanishing. The number of residual modules is adjusted according to the experimental results. The goal is to allow the network to have enough depth to learn complex patterns without making the model difficult to train. One residual module may not be enough to capture complex features, and too many residual modules may cause the neural network model to overfit due to insufficient data and excessive parameters. In this embodiment, the first preset number is 3, that is, 3 residual modules connected in series are used.

[0088] The feature extraction module 121 is used to obtain a first feature map from the preprocessed ultrasound image; the three residual modules 122, 123 and 124 are connected in series and are used to extract deep feature representations from the first feature map to obtain a second feature map; the fusion module 125 is used to fuse the second feature map with the preprocessed clinical data to obtain a fusion feature; the judgment module 126 is used to judge whether the diseased intestinal segment to be predicted can achieve mucosal healing based on the fusion feature.

[0089] Figure 2 Schematic diagram of the main structure of the preprocessing module in the embodiment of the present invention. Figure 2 As shown, the preprocessing module 110 in this embodiment may include: a clinical data conversion unit 111 , a rectangular frame cropping unit 112 , an image scaling unit 113 and a pixel value normalization unit 114 .

[0090] In this embodiment, the original clinical data includes: categorical data and numerical data; the clinical data conversion unit 111 is used to convert the categorical data in the original clinical data into hot single encoding and keep the numerical data unchanged, thereby obtaining the preprocessed clinical data; the rectangular frame cropping unit 112 is used to crop a rectangular frame image containing the lesion site from the original ultrasound image; the image scaling unit 113 is used to scale the rectangular frame image according to a preset size; the pixel value normalization unit 114 is used to normalize each pixel value in the scaled rectangular frame image to the range of [-1, 1], thereby obtaining the preprocessed ultrasound image.

[0091] In this embodiment, the categorical data include: patient gender, lesion site, smoking history, previous intestinal surgery history, drug treatment and drug type, etc.; the numerical data include: patient age, disease duration, simplified Crohn's disease activity index (CDAI), and high-sensitivity C-reactive protein level (hsCRP), etc.; the preset size is: 512x1024.

[0092] In an embodiment of the present invention, a radiologist uses Colabeler (QuickEasy Technology) software to outline lesions such as the intestinal wall, fistula, and abscess in the original ultrasound image, and a rectangular frame cropping unit 112 calculates a minimum rectangular frame that can cover the outlined area and crops it. Subsequently, only the rectangular frame image can be input into the model for judgment to prevent confusing information from being introduced into the model.

[0093] Figure 3 FIG. 1 is a schematic diagram of the main structure of the feature extraction module in the neural network model of the embodiment of the present invention. Figure 3 As shown, the feature extraction module 121 of this embodiment includes: a ResNet50 network 1211, a 7x7 convolutional layer 1212 and a maximum delay layer 1213.

[0094] Among them, the ResNet50 network 1211 is used to preliminarily extract feature maps from the preprocessed ultrasound image; the 7x7 convolutional layer 1212 is used to reduce the size of the preliminarily extracted feature maps; the maximum delay layer 1213 is used to perform dimensionality reduction operations on the feature maps after the size reduction, so as to obtain the first feature map.

[0095] Figure 4 1 is a schematic diagram of the main structure of the residual module in the neural network model of the embodiment of the present invention. The structures of the three residual modules 122, 123 and 124 are exactly the same, and only the residual module 122 is introduced here. Figure 4 As shown, the residual module 122 in this embodiment includes: a first 3x3 convolution layer 1221, a second 3x3 convolution layer 1222, a compression and excitation unit 1223, a first 1x1 convolution layer 1224 and a first addition unit 1225.

[0096] Among them, the first 3x3 convolution layer 1221 and the second 3x3 convolution layer 1222 connected in series are used to perform deep extraction of the input features of the residual module; the compression and excitation unit 1223 is used to introduce an attention mechanism into the output features of the second 3x3 convolution layer; the first 1x1 convolution layer 1224 is used to receive the input features of the residual module and adjust the dimensions to match the output feature dimensions of the compression and excitation unit; the first addition unit 1225 is used to add the output of the first 1x1 convolution layer to the output of the compression and excitation unit.

[0097] The compression and excitation unit 1223 in this embodiment includes: a first global pooling layer 12231, a first fully connected layer 12232, a ReLU activation function 12233, a second fully connected layer 12234, a sigmoid activation function 12235 and a first attention unit 12236.

[0098] Among them, the first global pooling layer 12231 is used to compress the information of each channel in the output features of the second 3x3 convolutional layer 1222 into a single value; the first fully connected layer 12232 is used to reduce the number of channels of the output features of the first global pooling layer 12231 to reduce the amount of calculation and remove redundant information; the ReLU activation function 12233 is used to add nonlinear features to the output of the first fully connected layer 12232; the second fully connected layer 12234 is used to restore the number of channels of the output features of the ReLU activation function 12233, and output a vector with the same number of channels as the output features of the second 3x3 convolutional layer 1222; each element in the vector corresponds to the weight of a channel, indicating the importance of the channel; the sigmoid activation function 12235 is used to map the output of the second fully connected layer to between 0 and 1, and obtain the attention weights of each channel; the scale operation is performed in the first attention unit 12236, the output features of the second 3x3 convolutional layer 1222 are received and the attention weights are added.

[0099] Figure 5 FIG. 1 is a schematic diagram of the main structure of the fusion module in the neural network model of the embodiment of the present invention. Figure 5 As shown, the fusion module 125 of this embodiment includes: a second global pooling layer 1251, a splicing unit 1252, a third fully connected layer 1253, a fourth fully connected layer 1254, a second attention unit 1255, a third 3x3 convolutional layer 1256, a fourth 3x3 convolutional layer 1257, a second 1x1 convolutional layer 1258, and a second addition unit 1259.

[0100] Among them, the second global pooling layer 1251 is used to compress the information of each channel in the second feature map into a single value; the splicing unit 1252 is used to splice the output features of the second global pooling layer with the preprocessed clinical data to obtain spliced ​​features; the third fully connected layer 1253 and the fourth fully connected layer 1254 connected in series are used to extract higher-level abstract features from the spliced ​​features; the second attention unit 1255 is used to add attention information to the second feature map using abstract features; the third 3x3 convolutional layer 1256 and the fourth 3x3 convolutional layer 1257 connected in series are used to extract features from the feature map with added attention information; the second 1x1 convolutional layer 1258 is used to receive the second feature map and adjust the dimension to match the output feature dimension of the fourth 3x3 convolutional layer; the second addition unit 1259 is used to add the output of the second 1x1 convolutional layer to the output of the fourth 3x3 convolutional layer to obtain a fused feature.

[0101] In this embodiment, the concat operation is used in the concatenation unit 1252 to fuse the image features and clinical data together. Assuming that the image feature dimension is I and the clinical data dimension is C, the length of the fused feature is I+C, and then the fused feature dimension is converted to 2I through the third fully connected layer 1253 and the fourth fully connected layer 1254. The scale operation is performed in the second attention unit 1255, taking the first half of the fused feature as C1 (scaling coefficient) and the second half as C2 (bias value), and dynamically scaling and offsetting the value of the image feature (represented by F), that is, C1*F+C2.

[0102] Figure 6 FIG. 1 is a schematic diagram of the main structure of the judgment module in the neural network model of the embodiment of the present invention. Figure 6 As shown, in this embodiment, the judgment module 126 includes: a third global pooling layer 1261 and a fifth fully connected layer 1262.

[0103] Among them, the third global pooling layer 1261 is used to compress the information of each channel in the fusion feature into a single value; the fifth fully connected layer 1262 is used to perform binary classification according to the output of the third global pooling layer, so as to determine whether the diseased intestinal segment to be predicted can achieve mucosal healing.

[0104] In an optional embodiment, the prediction system may further include: a training module ( Figure 1 The training module includes: an original data collection unit, a data processing unit, an image amplification unit, a data division unit, a training unit, and a verification unit.

[0105] Among them, the original data collection unit is used to collect a second preset number of ultrasound image samples of the diseased intestinal segment before treatment, the clinical data corresponding to each diseased intestinal segment, and the label value of whether the diseased intestinal segment has achieved mucosal healing, so as to construct an original data set; the label value is obtained by follow-up colonoscopy and / or cross-sectional imaging evaluation; the data processing unit is used to call the preprocessing module 110 to preprocess each ultrasound image sample and the corresponding clinical data in the original data set; the image amplification unit is used to horizontally, vertically and randomly flip each preprocessed ultrasound image for image amplification; the data division unit is used to divide the amplified image according to a preset ratio, and then based on the correspondence between each image and the clinical data, use the amplified image and the preprocessed clinical data to construct a training set and a validation set; the training unit is used to select one image and the corresponding clinical data from the training set each time, input them into the neural network model to obtain a predicted value, calculate the loss function, adjust the parameters of the neural network model, and repeat the execution until the loss function no longer decreases, so as to obtain a trained neural network model; the verification unit is used to use the verification set to perform a 5-fold cross-validation on the trained neural network model, and perform external verification and reliability verification on the trained neural network model.

[0106] In this embodiment, the verification unit first performs a 5-fold cross-validation on the neural network model, and the obtained verification results are shown in Table 1 below:

[0107] Table 1 5-fold cross validation results of the model

[0108]

[0109] As shown in Figure 1, the average AUROC (Area Under the Receiver Operating Characteristic Curve) was verified to be 0.77. The sensitivity of predicting MH (mucosal healing) of the diseased intestinal segment was 74.6%, and the specificity was 71.6%. The sensitivity of the model in predicting that Crohn's disease patients did not achieve MH was 82.5%, and the specificity was 42.7%.

[0110] Then the neural network model was externally validated, and the results are shown in Table 2 below:

[0111] Table 2 External validation results of the model

[0112]

[0113] As shown in Table 2, the average AUROC obtained by validation was 0.73. The average accuracy, sensitivity, specificity, PPV (Positive Predictive Value) and NPV (Negative Predictive Value) for predicting failure to achieve MH were 68.6%, 69.5%, 68.1%, 54.8% and 80.0%, respectively.

[0114] In order to confirm the reliability of the neural network model, the present embodiment also uses a heat map (Grad-CAM) technology to display its decision-making process. The heat map reveals that the neural network model of the present embodiment mainly captures the information of the thickened intestinal wall, serosal surface and surrounding mesentery in the ultrasound image, which is consistent with the conventional ideas of ultrasound diagnosis, thus confirming the reliability of the model.

[0115] Based on the same technical concept as the above-mentioned Crohn's disease mucosal healing prediction system embodiment, the present invention also provides a Crohn's disease mucosal healing prediction method embodiment. The prediction method of this embodiment includes steps S10-S20:

[0116] Step S10: pre-processing the original ultrasound image of the intestinal segment to be predicted to have lesions before treatment and the corresponding original clinical data.

[0117] Step S20: input the preprocessed clinical data and the preprocessed ultrasound image into the neural network model to determine whether the diseased intestinal segment to be predicted can achieve mucosal healing.

[0118] Among them, the neural network model has been trained and includes: a feature extraction module, a first preset number of residual modules, a fusion module, and a judgment module.

[0119] The feature extraction module is used to obtain a first feature map from a preprocessed ultrasound image; a first preset number of residual modules are connected in series and used to extract deep feature representations from the first feature map to obtain a second feature map; the fusion module is used to fuse the second feature map with the preprocessed clinical data to obtain a fusion feature; and the judgment module is used to judge whether the diseased intestinal segment to be predicted can achieve mucosal healing based on the fusion feature.

[0120] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0121] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A Crohn's disease mucosal healing prediction system, characterized in that: The system comprises: A preprocessing module, used for preprocessing the original ultrasound image of the intestinal segment to be predicted to have a lesion and the corresponding original clinical data before treatment; A neural network model is used to calculate whether the intestinal segment to be predicted to have a lesion can achieve mucosal healing based on the preprocessed clinical data and the preprocessed ultrasound image; in, The neural network model has been trained and includes: A feature extraction module, used for acquiring a first feature map from the preprocessed ultrasound image; A first preset number of residual modules are connected in series and used to extract deep feature representations from the first feature map to obtain a second feature map; A fusion module, used for fusing the second feature map with the preprocessed clinical data to obtain a fusion feature; A judgment module is used to judge whether the intestinal segment to be predicted with lesions can achieve mucosal healing according to the fusion characteristics.

2. The Crohn's disease mucosal healing prediction system according to claim 1, characterized in that: The original clinical data includes: categorical data and numerical data; The preprocessing module comprises: A clinical data conversion unit, used for converting the categorical data in the original clinical data into hot-single-hot encoding and keeping the numerical data unchanged, thereby obtaining the preprocessed clinical data; A rectangular frame cutting unit, used for cutting out a rectangular frame image containing a lesion from the original ultrasound image; An image scaling unit, used for scaling the rectangular frame image according to a preset size; The pixel value normalization unit is used to normalize each pixel value in the scaled rectangular frame image to a range of [-1, 1], so as to obtain the preprocessed ultrasound image.

3. The Crohn's disease mucosal healing prediction system according to claim 1, characterized in that: The feature extraction module comprises: A ResNet50 network is used to preliminarily extract a feature map from the preprocessed ultrasound image; 7x7 convolutional layer, used to reduce the size of the initially extracted feature map; The maximum delay layer is used to perform a dimensionality reduction operation on the feature map after the size reduction, so as to obtain the first feature map.

4. The Crohn's disease mucosal healing prediction system according to claim 1, characterized in that: Each of the residual modules comprises: The first 3x3 convolutional layer and the second 3x3 convolutional layer are connected in series to perform deep extraction of input features of the residual module; A compression and excitation unit, used to introduce an attention mechanism into the output features of the second 3x3 convolutional layer; A first 1x1 convolutional layer, configured to receive the input features of the residual module and adjust the dimensions to match the output feature dimensions of the compression and excitation unit; A first adding unit is used to add the output of the first 1x1 convolutional layer to the output of the compression and excitation unit.

5. The Crohn's disease mucosal healing prediction system according to claim 4, characterized in that: The compression and excitation unit comprises: A first global pooling layer, used to compress the information of each channel in the output features of the second 3x3 convolutional layer into a single value; A first fully connected layer, used to reduce the number of channels of the output features of the first global pooling layer to reduce the amount of calculation and remove redundant information; A ReLU activation function is used to add nonlinear features to the output of the first fully connected layer; The second fully connected layer is used to restore the number of channels of the output feature of the ReLU activation function and output a vector with the same number of channels as the output feature of the second 3x3 convolutional layer; each element in the vector corresponds to the weight of a channel, indicating the importance of the channel; A sigmoid activation function is used to map the output of the second fully connected layer to between 0 and 1 to obtain the attention weight of each channel; The first attention unit is used to receive the output features of the second 3x3 convolutional layer and add the attention weight.

6. The Crohn's disease mucosal healing prediction system according to claim 1, characterized in that: The fusion module includes: A second global pooling layer, used to compress the information of each channel in the second feature map into a single value; A splicing unit, used for splicing the output features of the second global pooling layer with the preprocessed clinical data to obtain a spliced ​​feature; A third fully connected layer and a fourth fully connected layer connected in series, used to extract higher-level abstract features from the concatenated features; A second attention unit, configured to add attention information to the second feature map using the abstract feature; The third 3x3 convolutional layer and the fourth 3x3 convolutional layer are connected in series to extract features from the feature map with attention information added; A second 1x1 convolutional layer is used to receive the second feature map and adjust the dimension to match the output feature dimension of the fourth 3x3 convolutional layer; The second adding unit is used to add the output of the second 1x1 convolutional layer and the output of the fourth 3x3 convolutional layer to obtain the fusion feature.

7. The Crohn's disease mucosal healing prediction system according to claim 1, characterized in that: The judging module comprises: A third global pooling layer is used to compress the information of each channel in the fused feature into a single value; The fifth fully connected layer is used to perform binary classification according to the output of the third global pooling layer, so as to determine whether the diseased intestinal segment to be predicted can achieve mucosal healing.

8. The Crohn's disease mucosal healing prediction system according to claim 2, characterized in that: The system also includes: a training module; The training module includes: A raw data collection unit, used to collect a second preset number of ultrasound image samples of the diseased intestinal segments before treatment, clinical data corresponding to each diseased intestinal segment, and a label value indicating whether the diseased intestinal segment has achieved mucosal healing, so as to construct a raw data set; the label value is obtained by follow-up colonoscopy and / or cross-sectional imaging assessment; A data processing unit, configured to call the preprocessing module to preprocess each ultrasound image sample and corresponding clinical data in the original data set; An image amplification unit, used for horizontally, vertically and randomly flipping each pre-processed ultrasound image to perform image amplification; A data division unit, used to divide the amplified image according to a preset ratio, and then construct a training set and a validation set using the amplified image and the preprocessed clinical data based on the correspondence between each image and the clinical data; A training unit, used for selecting one image and corresponding clinical data from the training set each time and inputting them into the neural network model to obtain a prediction value, calculating a loss function, adjusting the parameters of the neural network model, and repeating the process until the loss function no longer decreases, thereby obtaining a trained neural network model; The verification unit is used to perform a 5-fold cross-validation on the trained neural network model using the verification set, and to perform external verification and reliability verification on the trained neural network model.

9. The Crohn's disease mucosal healing prediction system according to claim 2, characterized in that: The first preset number is 3; The categorical data include: patient gender, lesion location, smoking history, previous intestinal surgery history, medication, and medication type; The numerical data include: patient age, disease duration, simplified Crohn's disease activity index, and high-sensitivity C-reactive protein level; The preset size is: 512x1024.

10. A method for predicting mucosal healing of Crohn's disease, characterized in that: The method comprises: Preprocessing the original ultrasound images and corresponding original clinical data of the intestinal segment to be predicted to have lesions before treatment; Inputting the preprocessed clinical data and the preprocessed ultrasound images into the neural network model to determine whether the intestinal segment to be predicted can achieve mucosal healing; in, The neural network model has been trained and includes: A feature extraction module, used for acquiring a first feature map from the preprocessed ultrasound image; A first preset number of residual modules are connected in series and used to extract deep feature representations from the first feature map to obtain a second feature map; A fusion module, used for fusing the second feature map with the preprocessed clinical data to obtain a fusion feature; A judgment module is used to judge whether the intestinal segment to be predicted with lesions can achieve mucosal healing according to the fusion characteristics.

Images