Acidiphilic granulocytic gastroenteritis typing system and method based on multi-modal data

Abstract

The invention relates to the technical field of artificial intelligence, and provides an eosinophilic gastroenteritis typing system and method based on multi-modal data, and the method comprises the steps: obtaining a digestive endoscopy image, an ultrasonic endoscopy image, a CT examination image and clinical manifestation data of a to-be-detected person collected in the same time period; the gastrointestinal mucosa lesion state is recognized through a digestive endoscopy image and is represented by numerical features; identifying the abnormal thickening condition of each film layer structure of the gastrointestinal wall through an ultrasonic endoscope image, and representing the abnormal thickening condition by numerical features; lesion hierarchical distribution is judged through an ultrasonic endoscope image, and numerical feature representation is carried out; determining seroperitoneum and intestinal wall thickening conditions through a CT examination image, and performing numerical characteristic representation on the seroperitoneum and intestinal wall thickening conditions; clinical symptom features are obtained through the clinical manifestation data and are represented by numerical features; generating a feature vector based on multi-modal data by using each numerical value feature representation; and inputting the feature vectors into a dynamic typing prediction model to realize the typing prediction of eosinophilic gastroenteritis, thereby improving the accuracy of auxiliary diagnosis and evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a granulocytic gastritis typing system and method based on multi-modal data. BACKGROUND

[0002] Granulocytic gastritis (EGE) is a rare disease that can involve multiple layers of tissue in the digestive tract, and is a kind of eosinophilic gastrointestinal disease (EGIDS). Due to the low incidence of the disease, no specific clinical manifestations, and no clear diagnostic criteria for EGE, it is easy to cause clinical misdiagnosis and missed diagnosis.

[0003] Currently, the Talley standard is usually used to assist in the diagnosis of granulocytic gastritis. The Talley standard includes: ① There are gastrointestinal symptoms. ② Biopsy pathology shows that there is eosinophilic infiltration in one or more parts of the gastrointestinal tract from the esophagus to the colon, or there is radiological colon abnormality with surrounding eosinophilic proliferation. ③ Exclude parasitic infections and diseases other than the gastrointestinal tract with eosinophilic proliferation, such as connective tissue disease, eosinophilia, Crohn's disease, lymphoma, etc. The Talley standard alone cannot comprehensively evaluate the patient's condition.

[0004] It can be seen that how to improve the accuracy of the auxiliary diagnosis and evaluation of granulocytic gastritis lesions has important clinical application value. SUMMARY

[0005] In view of the above problems, the present application is proposed in order to provide a granulocytic gastritis typing system and method based on multi-modal data to overcome the above problems.

[0006] In one aspect of the present application, a granulocytic gastritis typing method based on multi-modal data is provided, and the method comprises: acquiring digestive endoscopy images, ultrasonic endoscopy images, CT examination images and clinical performance data collected by a subject in the same period; The gastrointestinal mucosa abnormal region detection is performed on the digestive endoscope image to identify the gastrointestinal mucosa lesion state, and a first numerical feature representation of the gastrointestinal mucosa lesion state is obtained by using numerical coding; the contour boundary detection is performed on each membrane layer structure of the gastrointestinal wall in the ultrasonic endoscope image, and the abnormal thickening of each membrane layer structure of the gastrointestinal wall is identified, and a second numerical feature representation of the abnormal thickening of the membrane layer structure of the gastrointestinal wall is obtained by using numerical coding; the lesion area of the eosinophilic gastroenteritis is detected according to the ultrasonic endoscope image, and the lesion level distribution is determined according to the positional relationship between the lesion area and the contour boundary of each membrane layer structure of the gastrointestinal wall, and a third numerical feature representation of the lesion level distribution is obtained by using numerical coding; the effusion region boundary and the intestinal wall thickening region detection are performed on the CT examination image to determine whether there is abdominal effusion and intestinal wall thickening, and a fourth numerical feature representation of the abdominal effusion and intestinal wall thickening is obtained by using numerical coding; the clinical symptom features of the to-be-tested person are extracted according to the clinical manifestation data, and a fifth numerical feature representation of the clinical symptom features is obtained by using numerical coding; The obtained various numerical feature representations are combined to generate a feature vector based on multi-modal data; The feature vector based on multi-modal data is input into a pre-trained dynamic typing prediction model to obtain a prediction result of the eosinophilic gastroenteritis typing category of the to-be-tested person.

[0007] Optionally, the dynamic typing prediction model is obtained by training an extreme gradient boosting tree model based on a pre-constructed data set, the data set includes feature vector samples based on multi-modal data generated according to historical digestive endoscope images, ultrasonic endoscope images, CT examination images, biopsy pathology reports and clinical manifestation data of different patients, and eosinophilic gastroenteritis typing categories of the corresponding patients; the training process of the dynamic typing prediction model includes: optimizing the extreme gradient boosting tree model according to a grid search or random search algorithm, training the optimized extreme gradient boosting tree model, and obtaining the dynamic typing prediction model.

[0008] Optionally, the method further includes: Obtaining biopsy pathology reports, blood test reports and fecal examination reports of the to-be-tested person within the same period; Before the obtained various numerical feature representations are combined to generate a feature vector based on multi-modal data, the method further includes: Extracting an eosinophil index category of the to-be-tested person according to the biopsy pathology report, and obtaining a sixth numerical feature representation of the eosinophil index category by using numerical coding; Extracting a blood eosinophil index category of the to-be-tested person according to the blood test report, and obtaining a seventh numerical feature representation of the blood eosinophil index category by using numerical coding; According to the feces examination report, the parasitic infection of the to-be-tested person is extracted, and a numerical coding method is used to obtain an eighth numerical feature representation of the parasitic infection.

[0009] Optionally, the contour boundary of each membrane layer structure of the gastrointestinal wall in the endoscopic ultrasound image is detected, and abnormal thickening of each membrane layer structure of the gastrointestinal wall is identified, including: Identify the contour boundaries of the mucosa layer, submucosa layer, muscularis propria layer and serosa layer in the endoscopic ultrasound image. Detect the thickness of the mucosa layer, muscularis propria layer and serosa layer using the built-in ruler of the image, and compare the measured value with the preset standard value to determine the abnormal thickening of each membrane layer structure of the gastrointestinal wall.

[0010] Optionally, detecting the thickness of the mucosa layer using the built-in ruler of the image includes: Reconstruct a blank image of the same size according to the size resolution of the endoscopic ultrasound image, and draw the contour boundaries of the mucosa layer and the submucosa layer onto the blank image using a color different from the background color of the blank image. Obtain the contour point set of the mucosa layer and the submucosa layer, calculate the distance from each position point in the contour point set of the mucosa layer to each point set in the submucosa layer, and take the minimum distance as the vertical distance from the current point of the mucosa layer to the submucosa layer. Calculate the vertical distance from all points of the mucosa layer to the submucosa layer in turn, take the farthest distance of the calculated vertical distance as the pixel distance of the mucosa layer, and convert the pixel distance to the physical size of the mucosa layer thickness through the built-in ruler of the image.

[0011] Optionally, according to the endoscopic ultrasound image, the lesion area of eosinophilic gastroenteritis is detected, and the lesion level distribution is determined according to the positional relationship between the lesion area and the contour boundaries of each membrane layer structure of the gastrointestinal wall, including: Identify the lesion area in the endoscopic ultrasound image and mark the lesion rectangular frame using a marker frame. Reconstruct a blank image of the same size according to the size resolution of the endoscopic ultrasound image. Draw the contour boundaries of each membrane layer structure of the gastrointestinal wall and the lesion rectangular frame onto the blank image using a color different from the background color of the blank image. Judge the overlapping state of the lesion rectangular frame and the contour boundaries of each membrane layer structure, if the lesion rectangular frame overlaps with the contour boundaries of at least one membrane layer structure, it indicates that the current lesion belongs to a cross-layer lesion, and if the lesion rectangular frame does not overlap with the contour boundaries of the membrane layer structure, it indicates that the current lesion belongs to a single-layer lesion.

[0012] Optionally, judging the overlapping state of the lesion rectangular frame and the contour boundaries of each membrane layer structure includes: A rectangular frame binary image mask image is drawn for each lesion rectangular frame, and a contour binary image mask image is drawn for the contour boundary of each film layer structure; The intersection area of the rectangular frame binary image mask image and the contour binary image mask image of each film layer structure is sequentially calculated, and if the area of the intersection area is greater than zero, it is determined that the lesion rectangular frame overlaps with the contour boundary of the current film layer structure.

[0013] Optionally, effusion area boundary and intestinal wall thickening area detection are performed on the CT examination image to determine whether there is peritoneal effusion and intestinal wall thickening, including: The CT examination image of each layer is input into a pre-trained volume data segmentation network model to obtain a 3D mask image of the current CT layer to identify the effusion area boundary and the intestinal wall thickening area; wherein the volume data segmentation network model is obtained by training a 3D-Unet convolutional neural network model based on a pre-constructed data set, and the data set includes a plurality of training sample pairs composed of an original CT image in nii file format and a 3D mask image in nii file format in which the effusion area and the intestinal wall thickening area are outlined with boundaries, the pixel value 1 in the 3D mask data represents the effusion area, the pixel value 2 represents the intestinal wall thickening area, and the pixel 0 is the background area; The intestinal wall thickening size and the peritoneal effusion volume are calculated according to the size of the effusion area boundary and the intestinal wall thickening area of the CT examination image of each layer; The peritoneal effusion and intestinal wall thickening are determined according to the intestinal wall thickening size and the peritoneal effusion volume.

[0014] Another aspect of the present application also provides a multi-modal data-based eosinophilic gastroenteritis typing system, which comprises functional modules for implementing the multi-modal data-based eosinophilic gastroenteritis typing method as described above, specifically comprising: The acquisition module is configured to acquire digestive endoscopy images, ultrasonic endoscopy images, CT examination images and clinical manifestation data collected from the subject within the same period; The disease identification and feature representation module is configured to perform gastrointestinal mucosa abnormal region detection on the digestive endoscope image to identify a gastrointestinal mucosa lesion state, acquire a first numerical feature representation of the gastrointestinal mucosa lesion state in a numerical coding manner, perform contour boundary detection on each membrane layer structure of the gastrointestinal wall in the ultrasonic endoscope image, identify abnormal thickening of each membrane layer structure of the gastrointestinal wall, acquire a second numerical feature representation of the abnormal thickening of the membrane layer structure of the gastrointestinal wall in a numerical coding manner, detect a lesion area of the eosinophilic gastroenteritis according to the ultrasonic endoscope image, determine lesion hierarchical distribution according to a positional relationship between the lesion area and the contour boundary of each membrane layer structure of the gastrointestinal wall, acquire a third numerical feature representation of the lesion hierarchical distribution in a numerical coding manner, perform effusion region boundary and intestinal wall thickening region detection on the CT examination image to determine whether there is abdominal effusion and intestinal wall thickening, acquire a fourth numerical feature representation of the abdominal effusion and intestinal wall thickening in a numerical coding manner, and extract clinical symptom features of the to-be-tested person according to clinical manifestation data, acquire a fifth numerical feature representation of the clinical symptom features in a numerical coding manner. The multi-modal feature generation module is configured to combine the obtained numerical feature representations to generate a feature vector based on multi-modal data. The typing prediction module is configured to input the feature vector based on multi-modal data into a pre-trained dynamic typing prediction model to obtain a prediction result of the eosinophilic gastroenteritis typing category of the to-be-tested person.

[0015] Another aspect of the present application also provides an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor; the computer program is executed by the processor to implement the steps of the above-mentioned eosinophilic gastroenteritis typing method based on multi-modal data.

[0016] The eosinophilic gastroenteritis typing system and method based on multi-modal data provided by the embodiments of the present application can more comprehensively evaluate the patient's condition, effectively improve the accuracy of auxiliary diagnosis and evaluation of eosinophilic gastroenteritis lesions, and has important clinical application value by combining artificial intelligence technology with the Talley diagnostic standard to construct an artificial intelligence-based eosinophilic gastroenteritis dynamic typing system.

[0017] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings provided herein are for illustrative purposes only and, therefore, should not be considered to be limiting in any way. In the drawings: Figure 1 A flow chart of a method for classifying eosinophilic gastroenteritis based on multi-modal data according to an embodiment of the present application; Figure 2 A schematic diagram of obtaining mucosa layer thickness according to an embodiment of the present application; Figure 3 A schematic diagram of labeling lesion area of eosinophilic gastroenteritis using rectangular box according to an embodiment of the present application; Figure 4 A schematic diagram of determining lesion level distribution according to an embodiment of the present application; Figure 5 A structural schematic diagram of a system for classifying eosinophilic gastroenteritis based on multi-modal data according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure can be embodied in various forms without being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0020] Those of ordinary skill in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field of the present disclosure. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0021] Embodiment One The present embodiment provides a method for classifying eosinophilic gastroenteritis based on multi-modal data, as shown in Figure 1 The method for classifying eosinophilic gastroenteritis based on multi-modal data according to the present application includes the following steps: S11, acquire the digestive endoscopy image, the ultrasonic endoscopy image, the CT examination image and the clinical performance data collected in the same period of the to-be-tested person. The ultrasonic endoscopy image includes the gastroscope ultrasonic image, the duodenum and colon ultrasonic image. The present application fuses the digestive endoscopy image, the ultrasonic endoscopy image, the CT examination image and the clinical performance data of the patient, and constructs an eosinophilic gastroenteritis typing system and method based on multi-modal data through artificial intelligence AI recognition technology, so as to more comprehensively evaluate the patient's condition.

[0022] S12, gastrointestinal mucosa abnormal area detection is performed on the digestive endoscopy image to identify the gastrointestinal mucosa lesion state, wherein the gastrointestinal mucosa lesion state includes a normal state, a mucosa abnormal non-early cancer state and an early cancer state, a first numerical feature representation of the gastrointestinal mucosa lesion state is obtained by using a numerical coding method; contour boundary detection is performed on each membrane layer structure of the gastrointestinal wall in the ultrasonic endoscopy image, and abnormal thickening of each membrane layer structure of the gastrointestinal wall is identified, a second numerical feature representation of the abnormal thickening of the gastrointestinal wall membrane layer structure is obtained by using a numerical coding method; the lesion area of eosinophilic gastroenteritis is detected according to the ultrasonic endoscopy image, and the lesion level distribution is determined according to the positional relationship between the lesion area and the contour boundary of each membrane layer structure of the gastrointestinal wall, a third numerical feature representation of the lesion level distribution is obtained by using a numerical coding method; the effusion area boundary and the intestinal wall thickening area of the CT examination image are detected to determine whether there is peritoneal effusion and intestinal wall thickening, a fourth numerical feature representation of the peritoneal effusion and intestinal wall thickening is obtained by using a numerical coding method; the clinical symptom features of the to-be-tested person are extracted according to the clinical performance data, and a fifth numerical feature representation of the clinical symptom features is obtained by using a numerical coding method. Specifically, the numerical feature representation of each modality data can be obtained by One-hot coding.

[0023] S13, the obtained numerical feature representations are combined to generate a feature vector based on multi-modal data; S14, input the feature vector based on multi-modal data into a pre-trained dynamic typing prediction model to obtain the eosinophilic gastroenteritis typing category prediction result of the to-be-tested person.

[0024] In one embodiment, the dynamic typing prediction model is obtained by training an extreme gradient boosting tree model based on a pre-constructed data set, the data set including feature vector samples based on multi-modal data generated according to historical digestive endoscopy images, endoscopic ultrasound images, CT examination images, biopsy pathology reports and clinical manifestation data of different patients and EGE typing categories of the corresponding patients; wherein the EGE typing categories are: not EGE, mucosal EGE, muscular layer EGE and serosal EGE. The training process of the dynamic typing prediction model includes: optimizing the extreme gradient boosting tree model according to a grid search or random search algorithm, training the optimized extreme gradient boosting tree model, and obtaining the dynamic typing prediction model.

[0025] In another embodiment, the dynamic typing prediction model is obtained by training a KNN (K-Nearest Neighbor) model, and the training data set includes feature vector samples based on multi-modal data generated according to historical digestive endoscopy images, endoscopic ultrasound images, CT examination images, biopsy pathology reports and clinical manifestation data of different patients and EGE typing categories of the corresponding patients.

[0026] The EGE typing method based on multi-modal data provided in the embodiments of the present application combines artificial intelligence technology with the Talley diagnostic standard, constructs an artificial intelligence-based EGE dynamic typing system, identifies the gastrointestinal mucosa lesion state through digestive endoscopy images, identifies the abnormal thickening of each membrane layer structure of the gastrointestinal wall and the lesion level distribution through endoscopic ultrasound images, identifies the peritoneal effusion and intestinal wall thickening through CT examination images, and extracts clinical symptom features through clinical manifestation data, thereby more comprehensively evaluating the patient's condition from the gastrointestinal mucosa lesion state, the abnormal thickening of each membrane layer structure of the gastrointestinal wall, the lesion level distribution, the peritoneal effusion and intestinal wall thickening, and the clinical symptom features, effectively improving the accuracy of auxiliary diagnosis and evaluation of EGE lesions, and having important clinical application value.

[0027] In the present embodiment, the gastrointestinal mucosa abnormal region detection of the digestive endoscopy image in step S12 is performed to identify the gastrointestinal mucosa lesion state, and the specific implementation process is as follows: The pre-trained abnormal region segmentation network model is used to realize the gastrointestinal mucosa abnormal region detection of the digestive endoscopy image. The training process of the abnormal region segmentation network model is as follows: collecting digestive endoscopy image data, contour labeling mucosal rugae thickening, hyperemia, ulcer and early cancer regions, training the segmentation network model to realize inputting a digestive endoscopy image and outputting the contour region of mucosal rugae thickening, hyperemia, ulcer and early cancer in the image, and realizing the gastrointestinal mucosa abnormal region detection.

[0028] In this embodiment, the step S12 is to detect the contour boundary of each membrane layer structure in the ultrasonic endoscopic image, and identify the abnormal thickening of each membrane layer structure of the gastrointestinal wall. The specific implementation process is as follows: The contour boundaries of the mucosa layer, submucosa layer, muscularis propria layer and serosa layer in the ultrasonic endoscopic image are identified. Specifically, a pre-trained membrane layer structure segmentation network model can be used to detect the contour boundaries of the mucosa layer, submucosa layer, muscularis propria layer and serosa layer in the ultrasonic endoscopic image. Then, the thickness of the mucosa layer, muscularis propria layer and serosa layer is detected by using the built-in ruler of the image, and the measured value is compared with the preset standard value to determine the abnormal thickening of each membrane layer structure of the gastrointestinal wall. Specifically, the detection of the mucosa layer thickness by using the built-in ruler of the image includes: reconstructing a blank image of the same size according to the size resolution of the ultrasonic endoscopic image, and drawing the contour boundaries of the mucosa layer and the submucosa layer onto the blank image by using a color different from the background color of the blank image; obtaining the contour point set of the mucosa layer and the submucosa layer, calculating the distance from each position point in the contour point set of the mucosa layer to each point set in the submucosa layer, and taking the minimum distance as the vertical distance from the current point of the mucosa layer to the submucosa layer; sequentially calculating the vertical distance from all points of the mucosa layer to the submucosa layer, taking the farthest distance of the calculated vertical distance as the pixel distance of the mucosa layer, and converting the pixel distance into the physical size of the mucosa layer thickness by using the built-in ruler of the image.

[0029] The ultrasonic endoscopic EUS image can show the thickening level and range of the intestinal wall, and can accurately diagnose in layers. The EUS image can clearly show the 5-layer structure of the gastrointestinal wall from inside to outside: mucosa layer, submucosa layer, muscularis propria layer, and serosa layer. The training process of the membrane layer structure segmentation network model is as follows: collect ultrasonic endoscopic image data of gastroscopy, mark the contour boundaries of the 5-layer structure in the image, train a membrane layer structure segmentation network model, which can extract the contour boundaries of the mucosa layer, submucosa layer, muscularis propria layer, and serosa layer from the EUS image, so as to further judge whether the mucosa layer, muscularis propria layer, and serosa layer are thickened.

[0030] The judgment of whether the mucosa layer is thickened is as follows: using the built-in ruler of the EUS image, measuring the thickness of the mucosa layer perpendicular to the tube wall, and judging that the mucosa layer is thickened when the mucosa layer thickness of the gastroscope ultrasonic image is ≥1.5mm, and judging that the mucosa layer is thickened when the mucosa layer thickness of the duodenum and colon ultrasonic image is ≥2.0mm.

[0031] The judgment of whether the muscularis propria layer is thickened is as follows: using the built-in ruler of the EUS image, measuring the thickness of the muscularis propria layer perpendicular to the tube wall, and judging that the muscularis propria layer is thickened when the muscularis propria layer thickness of the gastroscope ultrasonic image is ≥2.5mm, and judging that the muscularis propria layer is thickened when the muscularis propria layer thickness of the duodenum and colon ultrasonic image is ≥2.0mm.

[0032] Whether the serosal layer is thickened is determined as follows: use the built-in ruler of the EUS image to measure the thickness of the serosal layer perpendicular to the tube wall. If the serosal layer thickness on the ultrasound image is ≥1.5 mm, it is judged to be thickened.

[0033] Furthermore, in a specific embodiment, the process of obtaining the thickness of the mucosal layer is described by taking the mucosal layer as an example. The calculation methods of the muscular layer and the serous layer thickness are basically similar and will not be described in detail. The layered contours in the ultrasound image are identified by the membrane layer structure segmentation model and redrawn onto a blank image with a black background of the EUS size resolution, such as Figure 2 As shown in the figure, the outlines of the mucosal layer and the submucosa are drawn in white on a blank image, and then the thickness of the region from the mucosal layer to the submucosa is detected. The specific implementation method for detecting the thickness of the region from the mucosal layer to the submucosa is as follows: the outline point set of the mucosal layer is obtained, which is the point set of the large outline in the image. The distance from one point to the set of all points of the small outline (the submucosa) is calculated. The minimum distance is the vertical distance M1 from this point on the mucosal layer to the submucosa. The vertical distances M2, M3, ..., Mn from all points on the large outline to the submucosa are calculated in sequence. The longest of these vertical distances is calculated as the pixel distance of the mucosal layer. The pixel distance is converted into the physical size of the mucosal layer thickness using the built-in ruler of the image stored in the DICOM file.

[0034] In this embodiment, in step S12, the lesion area of ​​eosinophilic gastroenteritis is detected based on the ultrasound endoscopy image, and the hierarchical distribution of the lesion is determined based on the positional relationship between the lesion area and the contour boundaries of each membrane layer structure of the gastrointestinal wall. The specific implementation process is as follows: The lesion area in the endoscopic ultrasound image is identified and a lesion rectangular frame is marked by using a marker frame; a blank image of the same size is reconstructed according to the size resolution of the endoscopic ultrasound image; the outline boundaries of each membrane layer structure of the gastrointestinal wall and the lesion rectangular frame are drawn on the blank image by using a color different from the background color of the blank image; the overlapping state of the lesion rectangular frame and the outline boundaries of each membrane layer structure is judged, if the lesion rectangular frame overlaps with the outline boundaries of at least one membrane layer structure, it indicates that the current lesion belongs to a cross-layer lesion, if the lesion rectangular frame does not overlap with the outline boundaries of the membrane layer structure, it indicates that the current lesion belongs to a single-layer lesion. The judgment of the overlapping state of the lesion rectangular frame and the outline boundaries of each membrane layer structure specifically includes: a rectangular frame binary image mask image is drawn for each lesion rectangular frame, and a contour binary image mask image is drawn for the outline boundaries of each membrane layer structure; the intersection areas of the rectangular frame binary image mask image and the contour binary image mask image of each membrane layer structure are calculated in turn, that is, the intersection areas of the lesion area and the thickening area of the membrane layer structure are calculated, and if the intersection area is greater than zero, it is determined that the lesion rectangular frame overlaps with the outline boundaries of the current membrane layer structure. The identification of the lesion area in the endoscopic ultrasound image and the marking of the lesion rectangular frame by using the marker frame specifically include: a pre-trained target detection model is used to identify the lesion area in the endoscopic ultrasound image and mark the lesion rectangular frame.

[0035] The training process of the target detection model is as follows: collecting endoscopic ultrasound image data of a gastroenteroscopy, marking the lesion area of eosinophilic gastroenteritis in the image with a rectangular frame, as shown in Figure 3 , training a YOLOV8 target detection model of the lesion area can realize the input of an EUS image, and can identify the lesion rectangular frame area coordinates.

[0036] Further, the lesion rectangular frame area is verified with the gastrointestinal wall abnormal thickening area obtained by the foregoing operation, to further judge whether the lesion area and the thickening area overlap and the type of overlap, and to determine whether the disease is a cross-layer lesion or a single-layer lesion through the overlap type of the lesion and the thickening area.

[0037] In one specific embodiment, the layered outline boundaries of each membrane layer structure and the lesion rectangular frame area in the ultrasound image are identified and redrawn on a blank image of the size resolution of the EUS on a black background, the layered outline boundaries of each membrane layer structure are drawn on the blank image, and the outline of the mucosal layer, the outline of the submucosal layer and the lesion rectangular frame are drawn on the blank image in white, as shown in Figure 4 , which is an example of the mucosal layer, and the muscular layer and the serosal layer are the same.

[0038] Specific operation: Figure 4The large profile boundary is the profile of the submucosal layer, and the small profile boundary is the profile of the mucosal layer. First, a binary image mask image is drawn for each lesion rectangular frame, the inside of the rectangular frame is white, and the outside is black, which is called rectangleImage; a binary image mask image is drawn for the profile of the submucosal layer, the inside of the profile is white, and the outside is black, which is called BigImage; a binary image mask image is drawn for the profile of the mucosal layer, the inside of the profile is white, and the outside is black, which is called SmallImage; rectangleImage is sequentially subjected to logical operation (such as bitwise AND) with BigImage and SmallImage to calculate the intersection area thereof. If the intersection area is greater than zero, it indicates that the lesion rectangular frame overlaps with the profile. If the rectangular frame overlaps with both profiles, it indicates that the rectangular frame spans the two membrane layer structures, and the lesion belongs to a cross-layer lesion. If the rectangular frame does not overlap with both profiles, it is further determined whether the rectangular frame is between the two profiles. The condition is that the center point coordinates of the rectangular frame are within the profile range of the submucosal layer, and the center point coordinates are outside the profile range of the mucosal layer, and the lesion completely belongs to the mucosal layer, which is a single-layer lesion. The comprehensive judgment result of the lesion is obtained by combining the above lesion level distribution judgment and the abnormal thickening condition of each membrane layer structure.

[0039] In this embodiment, the fluid accumulation region boundary and the intestinal wall thickening region are detected in step S12 to determine whether there is abdominal fluid accumulation and intestinal wall thickening. The specific implementation process is as follows: The CT examination images of each layer are input into the pre-trained body data segmentation network model to obtain 3D mask images of the current CT layer to identify the fluid accumulation region boundary and the intestinal wall thickening region. The body data segmentation network model is obtained by training a 3D-Unet convolutional neural network model based on a pre-constructed data set. The data set includes a plurality of training sample pairs composed of an original CT image in nii file format and a 3D mask image in nii file format in which the fluid accumulation region and the intestinal wall thickening region are outlined. In the 3D mask data, the pixel value 1 represents the fluid accumulation region, the pixel value 2 represents the intestinal wall thickening region, and the pixel 0 is the background region. The intestinal wall thickening size and the abdominal fluid accumulation volume are calculated according to the size of the fluid accumulation region boundary and the intestinal wall thickening region of each layer of the CT examination image. Whether there is abdominal fluid accumulation and intestinal wall thickening is determined according to the intestinal wall thickening size and the abdominal fluid accumulation volume.

[0040] Specifically, the fluid accumulation volume calculation formula is as follows: Fluid volume (ml) = Σ (single-layer cross-sectional area of each CT layer x thickness of the corresponding CT layer) Wherein, the single-layer cross-sectional area = pixel number x single-pixel area, and the single-pixel area can be obtained based on the CT resolution.

[0041] Patients with eosinophilic gastroenteritis have characteristics of intestinal wall thickening and ascites in CT images. The application uses a pre-trained body data segmentation network model to realize the identification of the boundary of the effusion area and the intestinal wall thickening area. The training process of the body data segmentation network model is as follows: collect CT images of 10,000 patients with eosinophilic gastroenteritis and 20,000 normal / other gastrointestinal disease control patients, first preprocess the data: call the open source BOACT part segmentation model (body and organ analysis by radiologists at the point of care based on CT) to preprocess the abdominal CT data, and extract the abdominal cavity area to exclude the interference of extracorporeal artifacts and non-abdominal organ images. Label the image: use the boundary to outline the effusion area and the intestinal wall thickening area, and construct the 3D mask data of the CT body data in the format of nii file. The pixel value 1 in the 3D mask data represents the effusion area, the pixel value 2 represents the intestinal wall thickening area, and the pixel 0 is the background area. Convert the original CT image into nii file format, and the mask body data of the nii file as the input of the 3D-Unet segmentation network, train the 3D-Unet segmentation network of the CT body data, obtain the body data segmentation network model, realize the conversion of an input CT image into nii file format, and output the mask body data of the nii file through neural network identification. The pixel value 1 in the mask body data represents the effusion area, the pixel value 2 represents the intestinal wall thickening area, and the pixel 0 is the background area. Identify the boundary of the effusion area and the intestinal wall thickening area, output the binary mask of each CT layer (0 for background, 1 for effusion area, and 2 for intestinal wall thickening area), and then calculate the volume of the intestinal wall thickening and ascites in turn. If it exceeds the set threshold, it is judged that the CT image is intestinal wall thickening and ascites.

[0042] In an optional embodiment of the application, the multi-modal data is not limited to digestive endoscopy images, endoscopic ultrasound images, CT examination images and clinical manifestation data, but can also include biopsy pathology reports, blood test reports and stool test reports. Further, the application further includes obtaining the biopsy pathology report, blood test report and stool test report of the subject within the same period when obtaining the collected data of the subject within the same period. Furthermore, before combining the obtained numerical feature representations to generate a feature vector based on multimodal data, the method further includes: extracting the eosinophil index category of the subject according to the biopsy pathology report, the eosinophil index categories including normal, slightly high, and very high, and using a numerical coding method to obtain a sixth numerical feature representation of the eosinophil index category; extracting the blood eosinophil index category of the subject according to the blood test report, and using a numerical coding method to obtain a seventh numerical feature representation of the blood eosinophil index category; extracting the parasitic infection status of the subject according to the stool test report, and using a numerical coding method to obtain an eighth numerical feature representation of the parasitic infection status. When combining the obtained numerical feature representations to generate a feature vector based on multimodal data, all numerical feature representations are combined to generate a feature vector based on multimodal data.

[0043] Specifically, the eosinophilic gastroenteritis classification method based on multimodal data is obtained by pre-collecting historical data of different patients, including clinical manifestation data of patients' complaints, blood test reports, stool test reports, CT examination images, endoscopic examination images, biopsy pathology reports, ultrasound endoscopy images, and EGE classification (mucosal type (type I), muscular type (type II), serosal type (type III)).

[0044] The clinical manifestation data of the patient's chief complaint is processed by NLP to analyze whether there are symptoms such as nausea, vomiting, abdominal pain, diarrhea, etc., and then expressed in decimal and binary, as shown in Table 1: Table 1 Clinical manifestation data corresponding to symptoms Chief complaint Decimal and binary representation No symptoms 0 0000 Nausea 1 0001 Vomiting 2 0010 Abdominal pain 3 0011 Diarrhea 4 0100 We extract data from patients' blood test reports, such as routine blood tests, to determine whether eosinophils are elevated. We also extract data from stool tests to determine whether parasitic infections are present. We also extract data from CT scans to determine whether there is intestinal wall thickening or ascites. We also extract data from endoscopic lesions and pathological biopsy reports to determine whether eosinophils are ≥20 / HP. We also numerically represent abnormal thickening of the gastrointestinal wall, the distribution of lesions, and the EGE classification in EUS images. This allows us to construct a multimodal database specifically for eosinophilic gastroenteritis. Based on the structured attribute values ​​of the database, we generate custom feature vectors.

[0045] Furthermore, the clinical manifestation data of the patient's chief complaint was integrated and simplified, the classification categories were reduced, and converted into one-hot coding features to obtain the corresponding numerical feature representation, as shown in Table 2: Table 2 Numerical characteristics of clinical manifestation data Pre-integration Post-integration No symptoms No symptoms0000 Nausea Upper gastrointestinal symptoms0001 Vomiting Upper gastrointestinal symptoms0001 Abdominal pain Lower gastrointestinal symptoms0010 Diarrhea Lower gastrointestinal symptoms0010 If the patient has multiple symptoms, they are added together, for example, if the patient has both upper and lower gastrointestinal symptoms, 0001+0010=0011.

[0046] The eosinophil index of the patient's blood routine is divided into normal, high, and very high categories, and converted into One-Hot encoding features.

[0047] The patient's fecal examination indicators are extracted, and whether there is a parasitic infection is indicated by the numbers 0 and 1, and converted into One-Hot encoding features.

[0048] The CT examination report is extracted, and whether the intestinal wall is thickened is indicated by the numbers 0 and 1, and converted into One-Hot encoding features.

[0049] Whether there is ascites, indicated by the numbers 0 and 1, and converted into One-Hot encoding features.

[0050] The lesion state of the endoscope is extracted and divided into normal, abnormal mucosa non-early cancer, and early cancer, indicated by the numbers 0, 1, and 2, and converted into One-Hot encoding features.

[0051] The eosinophil index of the endoscopic biopsy report is extracted and divided into normal, high, and very high categories, and converted into One-Hot encoding features.

[0052] The gastrointestinal wall abnormal thickening of the EUS image is divided into normal, high, and very high categories, and converted into One-Hot encoding features.

[0053] The classification of EGE is: not EGE, mucosal type, muscular layer type, and serosal type.

[0054] The above structured data is constructed into a feature vector, generating a feature vector based on multi-modal data, for training a dynamic classification prediction model.

[0055] When implementing the prediction of the classification of eosinophilic gastroenteritis for a new patient, the lesion state of the endoscope, the EUS gastrointestinal wall abnormal thickening, the CT examination intestinal wall thickening, and the ascites are extracted, and the patient's blood routine, fecal examination, and pathological report content are extracted, converted into One-Hot encoding and formed into a feature vector, and input into the trained dynamic classification prediction model to implement the classification prediction of EGE.

[0056] For the method embodiments, the acts performed as part of the methods are illustrated in a manner generally consistent with a list of functions to be performed in the processes. The acts are not necessarily performed in the order described and could be performed in other orders or simultaneously. Furthermore, not all of the acts described are necessary to implement and practice the methods.

[0057] Embodiment Two The embodiment of the present application provides a multi-modal data-based eosinophilic gastroenteritis typing system, the system comprising functional modules for implementing the multi-modal data-based eosinophilic gastroenteritis typing method according to any one of the preceding embodiments. Figure 5 The embodiment of the present application provides a multi-modal data-based eosinophilic gastroenteritis typing system, the system comprising functional modules for implementing the multi-modal data-based eosinophilic gastroenteritis typing method according to any one of the preceding embodiments. Figure 5 The system comprises: The acquisition module 501 is configured to acquire digestive endoscopy images, ultrasonic endoscopy images, CT examination images and clinical manifestation data collected in the same period by a subject to be tested. The condition recognition and feature representation module 502 is configured to perform gastrointestinal mucosa abnormal region detection on the digestive endoscopy images to recognize gastrointestinal mucosa lesion states, and obtain a first numerical feature representation of the gastrointestinal mucosa lesion states by using a numerical coding method; perform contour boundary detection on each membrane layer structure of the gastrointestinal wall in the ultrasonic endoscopy images, and recognize abnormal thickening of each membrane layer structure of the gastrointestinal wall, and obtain a second numerical feature representation of the abnormal thickening of the membrane layer structure of the gastrointestinal wall by using a numerical coding method; detect a lesion area of eosinophilic gastroenteritis according to the ultrasonic endoscopy images, and determine lesion hierarchical distribution according to a positional relationship between the lesion area and the contour boundaries of each membrane layer structure of the gastrointestinal wall, and obtain a third numerical feature representation of the lesion hierarchical distribution by using a numerical coding method; perform effusion region boundary and intestinal wall thickening region detection on the CT examination images to determine whether there is peritoneal effusion and intestinal wall thickening, and obtain a fourth numerical feature representation of the peritoneal effusion and intestinal wall thickening by using a numerical coding method; extract clinical symptom features of the subject to be tested according to the clinical manifestation data, and obtain a fifth numerical feature representation of the clinical symptom features by using a numerical coding method. The multi-modal feature generation module 503 is configured to combine the obtained numerical feature representations to generate a multi-modal data-based feature vector. The typing prediction module 504 is configured to input the multi-modal data-based feature vector into a pre-trained dynamic typing prediction model to obtain a prediction result of an eosinophilic gastroenteritis typing category of the subject to be tested.

[0058] The obtaining module 501 is further configured to obtain a biopsy pathology report, a blood test report, and a stool test report of the to-be-tested person in the same period. Before the multi-modal feature generation module 503 combines the obtained various numerical feature representations to generate a feature vector based on multi-modal data, the condition identification and feature representation module 502 is further configured to extract an eosinophil index category of the to-be-tested person according to the biopsy pathology report, and obtain a sixth numerical feature representation of the eosinophil index category by using a numerical coding method; extract a blood eosinophil index category of the to-be-tested person according to the blood test report, and obtain a seventh numerical feature representation of the blood eosinophil index category by using a numerical coding method; and extract a parasitic infection condition of the to-be-tested person according to the stool test report, and obtain an eighth numerical feature representation of the parasitic infection condition by using a numerical coding method.

[0059] In the embodiment of the present application, the condition identification and feature representation module 502 comprises a gastrointestinal wall abnormal thickening detection unit, configured to perform contour boundary detection on each membrane layer structure of the gastrointestinal wall in the endoscopic ultrasound image, and identify the abnormal thickening condition of each membrane layer structure of the gastrointestinal wall. Specifically, the contour boundaries of the mucosal layer, the submucosal layer, the muscularis propria layer, and the serosal layer in the endoscopic ultrasound image are identified; the thicknesses of the mucosal layer, the muscularis propria layer, and the serosal layer are detected by using an image built-in ruler, and the measured values are compared with preset standard values to determine the abnormal thickening condition of each membrane layer structure of the gastrointestinal wall. The detection of the thickness of the mucosal layer by using the image built-in ruler comprises: reconstructing a blank image of the same size according to the size resolution of the endoscopic ultrasound image, and drawing the contour boundary of the mucosal layer and the contour boundary of the submucosal layer onto the blank image by using a color different from the background color of the blank image; obtaining a contour point set of the mucosal layer and the submucosal layer, calculating the distance from each position point in the contour point set of the mucosal layer to each point set in the submucosal layer, and taking the minimum distance as the vertical distance from the current point of the mucosal layer to the submucosal layer; sequentially calculating the vertical distances from all points of the mucosal layer to the submucosal layer, taking the farthest distance of the calculated vertical distances as the pixel distance of the mucosal layer, and converting the pixel distance into the physical size of the thickness of the mucosal layer by using the image built-in ruler.

[0060] In the embodiment of the present application, the disease identification and feature representation module 502 comprises a lesion level detection unit, which is configured to detect a lesion area of the eosinophilic gastroenteritis according to the endoscopic ultrasound image, and determine the lesion level distribution according to the positional relationship between the lesion area and the contour boundary of each membrane layer structure of the gastrointestinal wall. Specifically, the lesion area in the endoscopic ultrasound image is identified and marked with a marker box to obtain a lesion rectangular frame; a blank image of the same size is reconstructed according to the size resolution of the endoscopic ultrasound image; the contour boundary of each membrane layer structure of the gastrointestinal wall and the lesion rectangular frame are drawn onto the blank image using a color different from the background color of the blank image; the overlapping state of the lesion rectangular frame and the contour boundary of each membrane layer structure is determined, and if the lesion rectangular frame overlaps with at least one contour boundary of the membrane layer structure, it indicates that the current lesion belongs to a cross-layer lesion, and if the lesion rectangular frame does not overlap with the contour boundary of the membrane layer structure, it indicates that the current lesion belongs to a single-layer lesion. The determination of the overlapping state of the lesion rectangular frame and the contour boundary of each membrane layer structure comprises: drawing a rectangular frame binary image mask image for each lesion rectangular frame, and drawing a contour binary image mask image for the contour boundary of each membrane layer structure; the intersection area of the rectangular frame binary image mask image and the contour binary image mask image of each membrane layer structure is calculated in sequence, and if the area of the intersection area is greater than zero, it is determined that the lesion rectangular frame overlaps with the contour boundary of the current membrane layer structure.

[0061] In the embodiment of the present application, the disease identification and feature representation module 502 comprises an intestinal wall thickening and abdominal effusion detection unit, which is configured to detect the effusion area boundary and the intestinal wall thickening area of the CT examination image to determine whether there is abdominal effusion and intestinal wall thickening. Specifically, the CT examination image of each layer is input into a pre-trained volume data segmentation network model to obtain a 3D mask image of the current CT layer to identify the effusion area boundary and the intestinal wall thickening area; wherein the volume data segmentation network model is obtained by training a 3D-Unet convolutional neural network model based on a pre-constructed data set, the data set comprises a plurality of training sample pairs composed of an original CT image in nii file format and a 3D mask image in nii file format in which the effusion area and the intestinal wall thickening area are outlined with boundaries, the pixel value 1 in the 3D mask data represents the effusion area, the pixel value 2 represents the intestinal wall thickening area, and the pixel 0 is the background area; the intestinal wall thickening size and the abdominal effusion volume are calculated according to the size of the effusion area boundary and the intestinal wall thickening area of the CT examination image of each layer; whether there is abdominal effusion and intestinal wall thickening is determined according to the intestinal wall thickening size and the abdominal effusion volume.

[0062] In the implementation process of the second embodiment, reference can be made to the first embodiment, and the corresponding technical effects are achieved.

[0063] Embodiment three An embodiment of the present invention provides a computer program product having a computer program stored thereon. When the computer program is executed by a processor, the steps in the above embodiment of the method for eosinophilic gastroenteritis typing based on multimodal data are implemented, for example Figure 1 Steps S11-S14 are shown.

[0064] In the specific implementation process of Example 3, reference may be made to Example 1, and the corresponding technical effects are achieved.

[0065] Example 4 An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the embodiment of the method for eosinophilic gastroenteritis typing based on multimodal data are implemented, for example Figure 1 Steps S11-S14 are shown.

[0066] In the specific implementation process of the fourth embodiment, you can refer to the first embodiment, which has the corresponding technical effects. Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, any of the claimed embodiments may be used in any combination.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for eosinophilic gastroenteritis typing based on multimodal data, characterized in that: The method comprises: Obtain digestive endoscopy images, ultrasound endoscopy images, CT examination images and clinical manifestation data collected from the subjects during the same period; The digestive endoscopy image is subjected to gastrointestinal mucosal abnormal area detection to identify the gastrointestinal mucosal lesion state, and a first numerical feature representation of the gastrointestinal mucosal lesion state is obtained by using a numerical coding method; the contour boundary detection is performed on each membrane layer structure of the gastrointestinal wall in the ultrasound endoscopy image, and the abnormal thickening of each membrane layer structure of the gastrointestinal wall is identified, and a second numerical feature representation of the abnormal thickening of the membrane layer structure of the gastrointestinal wall is obtained by using a numerical coding method; the lesion area of ​​eosinophilic gastroenteritis is detected based on the ultrasound endoscopy image, and the lesion layer distribution is determined based on the positional relationship between the lesion area and the contour boundary of each membrane layer structure of the gastrointestinal wall, and a third numerical feature representation of the lesion layer distribution is obtained by using a numerical coding method; the effusion area boundary and the intestinal wall thickening area are detected on the CT examination image to determine whether there is ascites and intestinal wall thickening, and a fourth numerical feature representation of ascites and intestinal wall thickening is obtained by using a numerical coding method; the clinical symptom characteristics of the subject are extracted based on the clinical manifestation data, and a fifth numerical feature representation of the clinical symptom characteristics is obtained by using a numerical coding method; The obtained numerical feature representations are combined to generate a feature vector based on multimodal data; The feature vector based on the multimodal data is input into a pre-trained dynamic typing prediction model to obtain a prediction result of the eosinophilic gastroenteritis typing category of the subject.

2. The method according to claim 1, characterized in that The dynamic typing prediction model is obtained by training an extreme gradient boosting tree model based on a pre-constructed dataset. The dataset includes multimodal data-based feature vector samples and corresponding eosinophilic gastroenteritis typing categories generated based on historical digestive endoscopic images, ultrasound endoscopic images, CT examination images, biopsy pathology reports, and clinical manifestation data of different patients; The training process of the dynamic classification prediction model includes: optimizing the extreme gradient boosting tree model according to a grid search or random search algorithm, training the optimized extreme gradient boosting tree model, and obtaining the dynamic classification prediction model.

3. The method according to claim 1, characterized in that The method further comprises: Obtain biopsy pathology reports, blood test reports, and stool test reports from the subjects within the same period; Before combining the obtained numerical feature representations to generate a feature vector based on multimodal data, the following steps are also included: Extract the eosinophil index category of the subject according to the biopsy pathology report, and use numerical coding to obtain the sixth numerical feature representation of the eosinophil index category; Extracting the blood eosinophil index category of the subject according to the blood test report, and obtaining the seventh numerical feature representation of the blood eosinophil index category using a numerical coding method; The parasitic infection status of the subject is extracted according to the stool examination report, and the eighth numerical feature representation of the parasitic infection status is obtained by numerical coding.

4. The method according to claim 1, wherein Detect the contour boundaries of each membrane layer of the gastrointestinal wall in ultrasound endoscopic images and identify abnormal thickening of each membrane layer of the gastrointestinal wall, including: Identify the contour boundaries of the mucosa, submucosa, muscularis propria, and serosa in endoscopic ultrasound images; The thickness of the mucosal layer, muscularis propria and serosa is detected using the built-in ruler of the image, and the measured values ​​are compared with the preset standard values ​​to determine the abnormal thickening of the various membrane layers of the gastrointestinal wall.

5. The method according to claim 4, characterized in that The use of built-in rulers to detect the thickness of the mucosal layer includes: A blank image of the same size is reconstructed based on the size and resolution of the ultrasound endoscopic image, and the outline boundaries of the mucosal layer and the submucosal layer are drawn on the blank image using a color different from the background color of the blank image; Obtain the contour point sets of the mucosal layer and the submucosa, calculate the distance between each position point in the mucosal layer contour point set and each point set in the submucosa, and use the minimum distance as the vertical distance from the current point in the mucosal layer to the submucosa; The vertical distances from all points of the mucosal layer to the submucosal layer were calculated in sequence, and the longest vertical distance was used as the pixel distance of the mucosal layer. The pixel distance was converted into the physical size of the mucosal layer thickness using the built-in ruler of the image.

6. The method according to claim 1, characterized in that Detecting the lesion area of ​​eosinophilic gastroenteritis based on the ultrasound endoscopic image and determining the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of each membrane layer structure of the gastrointestinal wall, including: Identify the lesion area in the ultrasound endoscopy image and mark it with a marker frame to obtain a lesion rectangular frame; A blank image of the same size is reconstructed based on the size resolution of the endoscopic ultrasound image; The outline boundary of each membrane layer structure of the gastrointestinal wall and the lesion rectangular frame are drawn on the blank image using a color different from the background color of the blank image; Determine the overlapping status of the lesion rectangular frame and the contour boundaries of each membrane layer structure. If the lesion rectangular frame overlaps with the contour boundary of at least one membrane layer structure, it means that the current lesion is a cross-layer lesion. If the lesion rectangular frame does not overlap with the contour boundary of the membrane layer structure, it means that the current lesion is a single-layer lesion.

7. The method according to claim 6, characterized in that The overlapping status of the lesion rectangular frame and the contour boundaries of each membrane layer structure is determined as follows: Draw a rectangular frame binary image mask image for each lesion rectangular frame, and draw a contour binary image mask image for the contour boundary of each membrane layer structure; The intersection areas of the rectangular frame binary image mask image and the contour binary image mask image of each membrane layer structure are calculated in sequence. If the area of ​​the intersection area is greater than zero, it is determined that the lesion rectangular frame overlaps with the contour boundary of the current membrane layer structure.

8. The method according to claim 1, characterized in that CT images are used to detect the boundaries of the effusion area and the intestinal wall thickening area to determine whether there is ascites and intestinal wall thickening, including: Inputting the CT examination images of each layer into a pre-trained volume data segmentation network model to obtain a 3D mask image of the current CT layer to identify the boundaries of the effusion area and the intestinal wall thickening area; wherein the volume data segmentation network model is obtained by training a 3D-Unet convolutional neural network model based on a pre-constructed data set, and the data set includes a number of training sample pairs consisting of original CT images in nii file format and corresponding 3D mask images in nii file format with boundaries outlining the effusion area and the intestinal wall thickening area, wherein the pixel value 1 in the 3D mask data represents the effusion area, the pixel value 2 represents the intestinal wall thickening area, and the pixel 0 represents the background area; The size of intestinal wall thickening and the volume of ascites were calculated based on the boundaries of the effusion area and the size of the intestinal wall thickening area in the CT examination images of each layer; The presence of ascites and intestinal wall thickening was determined based on the size of the intestinal wall thickening and the volume of ascites.

9. A multimodal data-based eosinophilic gastroenteritis typing system, characterized in that: The system includes a functional module for implementing the eosinophilic gastroenteritis typing method based on multimodal data according to any one of claims 1 to 8, specifically including: An acquisition module is used to obtain digestive endoscopy images, ultrasound endoscopy images, CT examination images and clinical manifestation data collected from the subject during the same period; a symptom recognition and feature representation module for detecting abnormal gastrointestinal mucosal areas on the digestive endoscopy image to identify the state of gastrointestinal mucosal lesions, and obtaining a first numerical feature representation of the state of gastrointestinal mucosal lesions using a numerical coding method; detecting contour boundaries of various membrane layer structures of the gastrointestinal wall in the ultrasound endoscopy image, and identifying abnormal thickening of various membrane layer structures of the gastrointestinal wall, and obtaining a second numerical feature representation of the abnormal thickening of the membrane layer structures of the gastrointestinal wall using a numerical coding method; detecting lesion areas of eosinophilic gastroenteritis based on the ultrasound endoscopy image, and determining the hierarchical distribution of the lesions based on the positional relationship between the lesion areas and the contour boundaries of various membrane layer structures of the gastrointestinal wall, and obtaining a third numerical feature representation of the hierarchical distribution of the lesions using a numerical coding method; detecting effusion area boundaries and intestinal wall thickening areas on the CT examination image to determine whether there is ascites and intestinal wall thickening, and obtaining a fourth numerical feature representation of ascites and intestinal wall thickening using a numerical coding method; extracting clinical symptom characteristics of the subject based on clinical manifestation data, and obtaining a fifth numerical feature representation of the clinical symptom characteristics using a numerical coding method; A multimodal feature generation module is used to combine the obtained numerical feature representations to generate a feature vector based on multimodal data; The typing prediction module is used to input the feature vector based on the multimodal data into a pre-trained dynamic typing prediction model to obtain a prediction result of the eosinophilic gastroenteritis typing category of the subject.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images