System and method for classifying eosinophilic gastroenteritis based on multi-modal data

Abstract

The present application relates to the technical field of artificial intelligence, and provides a system and method for classifying eosinophilic gastroenteritis based on multi-modal data, which comprises: obtaining digestive endoscopy images, ultrasonic endoscopy images, CT examination images and clinical manifestation data collected by a subject in the same period; identifying the state of gastrointestinal mucosa lesions and representing the numerical characteristics through the digestive endoscopy images; identifying the abnormal thickening of each membrane layer structure of the gastrointestinal wall and representing the numerical characteristics through the ultrasonic endoscopy images; determining the lesion level distribution and representing the numerical characteristics through the ultrasonic endoscopy images; determining the peritoneal effusion and intestinal wall thickening and representing the numerical characteristics through the CT examination images; obtaining the clinical symptom characteristics and representing the numerical characteristics through the clinical manifestation data; generating a feature vector based on multi-modal data by representing each numerical characteristic; and inputting the feature vector into a dynamic classification prediction model to realize the classification prediction of eosinophilic gastroenteritis, thereby improving the accuracy of auxiliary diagnosis and evaluation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a classification system and method for eosinophilic gastroenteritis based on multimodal data. Background Technology

[0002] Eosinophilic gastroenteritis (EGE) is a rare gastrointestinal disorder characterized by an increase in eosinophils in multiple layers of the digestive tract. It is a type of eosinophilic gastrointestinal disorder (EGIDS). Due to its low incidence, nonspecific clinical manifestations, and the lack of clear diagnostic criteria, EGE is easily misdiagnosed or missed in clinical practice.

[0003] Currently, the Talley criteria are commonly used for the auxiliary diagnosis of eosinophilic gastroenteritis. The Talley criteria include: ① Presence of gastrointestinal symptoms; ② Biopsy pathology showing eosinophilic infiltration in one or more sites of the gastrointestinal tract from the esophagus to the colon, or radiographic colonic abnormalities with peripheral eosinophilia; ③ Exclusion of parasitic infections and non-gastrointestinal diseases causing eosinophilia, such as connective tissue diseases, eosinophilia, Crohn's disease, and lymphoma. However, relying solely on the Talley criteria cannot comprehensively assess the patient's condition.

[0004] Therefore, improving the accuracy of auxiliary diagnosis and assessment of eosinophilic gastroenteritis lesions has important clinical application value. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide an eosinophilic gastroenteritis typing system and method based on multimodal data to overcome the above problems.

[0006] One aspect of the present invention provides a method for typing eosinophilic gastroenteritis based on multimodal data, the method comprising:

[0007] Acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical data of the subjects within the same time period;

[0008] The gastrointestinal mucosa abnormality region was detected in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions. A first numerical feature representation of the gastrointestinal mucosal lesion state was obtained using a numerical coding method. The contour boundaries of each membrane layer structure of the gastrointestinal wall in the endoscopic ultrasound images were detected, and abnormal thickening of each membrane layer structure was identified. A second numerical feature representation of the abnormal thickening of the gastrointestinal wall membrane layer structure was obtained using a numerical coding method. The lesion area of ​​eosinophilic gastroenteritis was detected based on the endoscopic ultrasound images, and the lesion layer distribution was determined based on the positional relationship between the lesion area and the contour boundaries of each membrane layer structure of the gastrointestinal wall. A third numerical feature representation of the lesion layer distribution was obtained using a numerical coding method. The boundaries of the effusion area and the thickened intestinal wall area were detected in the CT scan images to determine whether ascites and intestinal wall thickening exist. A fourth numerical feature representation of ascites and intestinal wall thickening was obtained using a numerical coding method. The clinical symptom characteristics of the subjects were extracted based on clinical manifestation data, and a fifth numerical feature representation of the clinical symptom characteristics was obtained using a numerical coding method.

[0009] The obtained numerical feature representations are combined to generate feature vectors based on multimodal data;

[0010] The feature vector based on the multimodal data is input into a pre-trained dynamic typing prediction model to obtain the predicted typing category of eosinophilic gastroenteritis for the subject.

[0011] Optionally, the dynamic subtyping prediction model is obtained by training an extreme gradient boosting tree model based on a pre-built dataset. The dataset includes feature vector samples based on multimodal data generated from historical digestive endoscopy images, endoscopic ultrasound images, CT scan images, biopsy pathology reports, and clinical manifestation data of different patients, as well as the corresponding eosinophilic gastroenteritis subtyping categories for the patients. The training process of the dynamic subtyping prediction model includes: optimizing the extreme gradient boosting tree model according to a grid search or random search algorithm, training the optimized extreme gradient boosting tree model, and obtaining the dynamic subtyping prediction model.

[0012] Optionally, the method further includes:

[0013] Obtain biopsy pathology reports, blood test reports, and stool test reports from the subjects within the same time period;

[0014] Before combining the obtained numerical feature representations to generate feature vectors based on multimodal data, the following steps are also included:

[0015] Based on the biopsy pathology report, the eosinophil marker category of the subject was extracted, and the sixth numerical feature representation of the eosinophil marker category was obtained by numerical coding method;

[0016] Based on the blood test report, the blood eosinophil index category of the subject is extracted, and the seventh numerical feature representation of the blood eosinophil index category is obtained by numerical coding method;

[0017] Based on the fecal examination report, the parasite infection status of the subjects was extracted, and the eighth numerical feature representation of the parasite infection status was obtained by numerical coding.

[0018] Optionally, contour boundary detection is performed on the various membrane structures of the gastrointestinal wall in the endoscopic ultrasound images, and abnormal thickening of each membrane structure of the gastrointestinal wall is identified, including:

[0019] Identify the contours of the mucosa, submucosa, muscularis propria, and serosa in endoscopic ultrasound images;

[0020] The thickness of the mucosa, muscularis propria, and serosa was detected using an image-embedded scale. The measured values ​​were compared with preset standard values ​​to determine the abnormal thickening of each membrane structure of the gastrointestinal wall.

[0021] Optionally, using an image-embedded ruler to detect mucosal layer thickness includes:

[0022] A blank image of the same size is reconstructed based on the size resolution of the endoscopic ultrasound image. The outlines of the mucosal layer and the submucosal layer are drawn onto the blank image using a color different from the background color of the blank image.

[0023] Obtain the contour point set of the mucosa and submucosa, calculate the distance from each point in the mucosa contour point set to each point in the submucosa, and take the minimum distance as the vertical distance from the current point in the mucosa to the submucosa.

[0024] The vertical distance from all points in the mucosa to the submucosa is calculated sequentially. The distance with the longest calculated vertical distance is taken as the pixel distance of the mucosa. The pixel distance is then converted into the physical size of the mucosa thickness using the built-in image ruler.

[0025] Optionally, the lesion area of ​​eosinophilic gastroenteritis is detected based on endoscopic ultrasound images, and the lesion layer distribution is determined based on the positional relationship between the lesion area and the contour boundaries of various membrane structures of the gastrointestinal wall, including:

[0026] Identify lesion regions in endoscopic ultrasound images and mark them with a bounding box to obtain lesion rectangles;

[0027] A blank image of the same size is reconstructed based on the size resolution of the endoscopic ultrasound image;

[0028] The outlines of the various membrane structures of the gastrointestinal wall and the rectangular frames of lesions are drawn onto the blank image using a color different from the background color of the blank image.

[0029] Determine the overlap status between the lesion rectangle and the outline boundaries of each membrane structure. If the lesion rectangle overlaps with the outline boundary of at least one membrane structure, it indicates that the current lesion is a multi-layer lesion. If the lesion rectangle does not overlap with the outline boundary of any membrane structure, it indicates that the current lesion is a single-layer lesion.

[0030] Optionally, determining the overlap between the lesion rectangle and the contour boundaries of each membrane structure includes:

[0031] For each lesion rectangle, draw a binary image mask of the rectangle; for the contour boundaries of each membrane structure, draw a binary image mask of the contour.

[0032] The intersection regions of the rectangular bounding box binary image mask and the contour binary image mask of each membrane structure are calculated sequentially. If the area of ​​the intersection region is greater than zero, it is determined that the lesion bounding box overlaps with the contour boundary of the current membrane structure.

[0033] Optionally, the boundaries of the fluid accumulation area and the area of ​​intestinal wall thickening are detected on the CT scan images to determine whether ascites and intestinal wall thickening exist, including:

[0034] CT images from each layer are input into a pre-trained volume data segmentation network model to obtain a 3D mask image of the current CT layer, which is used to identify the boundaries of the fluid accumulation area and the intestinal wall thickening area. The volume data segmentation network model is trained on a 3D-Unet convolutional neural network model based on a pre-built dataset. The dataset includes several training sample pairs consisting of original CT images in nii file format and corresponding 3D mask images in nii file format with boundaries delineating the fluid accumulation area and the intestinal wall thickening area. In the 3D mask data, pixel value 1 represents the fluid accumulation area, pixel value 2 represents the intestinal wall thickening area, and pixel 0 is the background area.

[0035] The size of the intestinal wall thickening and the volume of the abdominal fluid are calculated based on the boundaries of the fluid accumulation area and the size of the thickened area in the intestinal wall in the CT images of each layer.

[0036] The presence of ascites and intestinal wall thickening can be determined based on the size of the intestinal wall thickening and the volume of ascites.

[0037] Another aspect of the present invention provides a eosinophilic gastroenteritis typing system based on multimodal data, the system comprising functional modules for implementing the eosinophilic gastroenteritis typing method based on multimodal data as described above, specifically including:

[0038] The acquisition module is used to acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical data of the subject within the same time period.

[0039] The symptom identification and feature representation module is used to detect abnormal areas of the gastrointestinal mucosa in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions, and to obtain a first numerical feature representation of the state of gastrointestinal mucosal lesions using a numerical encoding method; to detect the contour boundaries of each membrane structure of the gastrointestinal wall in the endoscopic ultrasound images and identify abnormal thickening of each membrane structure of the gastrointestinal wall, and to obtain a second numerical feature representation of the abnormal thickening of the membrane structure of the gastrointestinal wall using a numerical encoding method; to detect the lesion area of ​​eosinophilic gastroenteritis in the endoscopic ultrasound images and to determine the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of each membrane structure of the gastrointestinal wall, and to obtain a third numerical feature representation of the lesion layer distribution using a numerical encoding method; to detect the boundaries of the effusion area and the thickened area of ​​the intestinal wall in the CT examination images to determine whether there is ascites and intestinal wall thickening, and to obtain a fourth numerical feature representation of ascites and intestinal wall thickening using a numerical encoding method; and to extract the clinical symptom features of the subject based on the clinical manifestation data, and to obtain a fifth numerical feature representation of the clinical symptom features using a numerical encoding method.

[0040] The multimodal feature generation module is used to combine the obtained numerical feature representations to generate feature vectors based on multimodal data.

[0041] The typing prediction module is used to input the feature vector based on multimodal data into a pre-trained dynamic typing prediction model to obtain the typing category prediction result of the eosinophilic gastroenteritis of the test subject.

[0042] In another aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor; the computer program, when executed by the processor, implements the steps of the above-described method for classifying eosinophilic gastroenteritis based on multimodal data.

[0043] The eosinophilic gastroenteritis classification system and method based on multimodal data provided in this invention combines artificial intelligence technology with the Talley diagnostic criteria to construct an AI-based dynamic classification system for eosinophilic gastroenteritis. This system can more comprehensively assess the patient's condition and effectively improve the accuracy of auxiliary diagnosis and assessment of eosinophilic gastroenteritis lesions, thus having significant clinical application value.

[0044] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0045] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a method for classifying eosinophilic gastroenteritis based on multimodal data, according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the acquisition of mucosal layer thickness in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram showing the lesion area of ​​eosinophilic gastroenteritis marked with a rectangle in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram illustrating the determination of lesion layer distribution in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of an eosinophilic gastroenteritis classification system based on multimodal data according to an embodiment of the present invention. Detailed Implementation

[0051] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0053] Example 1

[0054] This invention provides a method for classifying eosinophilic gastroenteritis based on multimodal data, such as... Figure 1 As shown, the eosinophilic gastroenteritis classification method based on multimodal data proposed in this invention includes the following steps:

[0055] S11. Acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical data of the subject collected within the same time period. The endoscopic ultrasound images include gastroscopy, duodenal, and colonic ultrasound images. This invention integrates digestive endoscopy images, endoscopic ultrasound images, CT scan images, and patient clinical data, and constructs a multimodal data-based eosinophilic gastroenteritis classification system and method using artificial intelligence (AI) recognition technology to more comprehensively assess the patient's condition.

[0056] S12. The gastrointestinal mucosa abnormality region is detected in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions. The state of gastrointestinal mucosal lesions includes a normal state, a non-early cancer state with mucosal abnormalities, and an early cancer state. A first numerical feature representation of the gastrointestinal mucosal lesion state is obtained using a numerical encoding method. The contour boundaries of each membrane layer structure of the gastrointestinal wall in the endoscopic ultrasound images are detected, and abnormal thickening of each membrane layer structure is identified. A second numerical feature representation of the abnormal thickening of the gastrointestinal wall membrane layer structure is obtained using a numerical encoding method. Eosinophilic gastric lesions are detected based on the endoscopic ultrasound images. The lesion area of ​​enteritis is identified, and the lesion layer distribution is determined based on the positional relationship between the lesion area and the contour boundaries of various membranous structures of the gastrointestinal wall. A third numerical feature representation of the lesion layer distribution is obtained using numerical coding. The boundaries of the effusion area and the thickened intestinal wall area are detected on CT images to determine the presence of ascites and intestinal wall thickening. A fourth numerical feature representation of ascites and intestinal wall thickening is obtained using numerical coding. The clinical symptom characteristics of the subjects are extracted based on clinical manifestation data, and a fifth numerical feature representation of the clinical symptom characteristics is obtained using numerical coding. Specifically, one-hot encoding can be used to obtain the numerical feature representations of each modality of data.

[0057] S13. Combine the obtained numerical feature representations to generate a feature vector based on multimodal data;

[0058] S14. Input the feature vector based on the multimodal data into the pre-trained dynamic classification prediction model to obtain the eosinophilic gastroenteritis classification prediction result of the subject.

[0059] In one embodiment, the dynamic subtyping prediction model is trained on an extreme gradient boosting tree model based on a pre-built dataset. The dataset includes feature vector samples based on multimodal data generated from historical digestive endoscopy images, endoscopic ultrasound images, CT scan images, biopsy pathology reports, and clinical manifestation data of different patients, along with the corresponding eosinophilic gastroenteritis (EGE) subtyping categories. The EGE subtyping categories are: no EGE, mucosal EGE, muscularis propria EGE, and serosal EGE. The training process of the dynamic subtyping prediction model includes optimizing the extreme gradient boosting tree model using a grid search or random search algorithm, training the optimized extreme gradient boosting tree model, and obtaining the dynamic subtyping prediction model.

[0060] In another embodiment, the dynamic subtyping prediction model is trained based on the KNN (K-nearest neighbor) model. The training dataset includes feature vector samples based on multimodal data generated from different patients' historical digestive endoscopy images, endoscopic ultrasound images, CT scan images, biopsy pathology reports, and clinical manifestation data, as well as the corresponding eosinophilic gastroenteritis subtyping categories.

[0061] The eosinophilic gastroenteritis (EPO) classification method based on multimodal data provided in this invention combines artificial intelligence technology with the Talley diagnostic criteria to construct an AI-based dynamic classification system for EPO. This system identifies gastrointestinal mucosal lesions through digestive endoscopy images, abnormal thickening of various membrane layers and lesion distribution through endoscopic ultrasound images, ascites and intestinal wall thickening through CT scans, and extracts clinical symptom features from clinical data. This allows for a more comprehensive assessment of the patient's condition from multiple dimensions, including the state of gastrointestinal mucosal lesions, abnormal thickening of various membrane layers, lesion distribution, ascites and intestinal wall thickening, and clinical symptom features. This effectively improves the accuracy of auxiliary diagnosis and assessment of EPO and has significant clinical application value.

[0062] In this embodiment, step S12 involves detecting abnormal areas of the gastrointestinal mucosa in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions. The specific implementation process is as follows:

[0063] A pre-trained abnormal region segmentation network model is used to detect abnormal regions of the gastrointestinal mucosa in the digestive endoscopy images. The training process of the abnormal region segmentation network model is as follows: Endoscopic image data of the digestive tract is collected, and contour annotations are performed on areas with enlarged mucosal folds, congestion, ulceration, and early cancer. The segmentation network model is trained to take a digestive endoscopy image as input and output the contour regions of enlarged mucosal folds, congestion, ulceration, and early cancer in the image, thus realizing the detection of abnormal regions of the gastrointestinal mucosa.

[0064] In this embodiment, step S12 involves detecting the contour boundaries of each membrane layer structure of the gastrointestinal wall in the endoscopic ultrasound image and identifying abnormal thickening of each membrane layer structure. The specific implementation process is as follows:

[0065] This study identifies the contour boundaries of the mucosa, submucosa, muscularis propria, and serosa in endoscopic ultrasound images. Specifically, a pre-trained membrane structure segmentation network model can be used to detect the contour boundaries of these layers in endoscopic ultrasound images. Then, the thickness of the mucosa, muscularis propria, and serosa is measured using an image-embedded scale. The measured values ​​are compared with preset standard values ​​to determine abnormal thickening of each membrane structure in the gastrointestinal wall. Specifically, the method of detecting mucosal layer thickness using an image-built-in scale includes: reconstructing a blank image of the same size based on the size resolution of the endoscopic ultrasound image; drawing the contour boundaries of the mucosal layer and the submucosal layer onto the blank image using a color different from the background color of the blank image; acquiring the contour point sets of the mucosal layer and the submucosal layer; calculating the distance from each point in the mucosal layer contour point set to each point in the submucosal layer, and taking the minimum distance as the vertical distance from the current point in the mucosal layer to the submucosal layer; sequentially calculating the vertical distances from all points in the mucosal layer to the submucosal layer, taking the farthest calculated vertical distance as the pixel distance of the mucosal layer, and converting the pixel distance into the physical size of the mucosal layer thickness using the image-built-in scale.

[0066] Endoscopic ultrasound (EUS) images can accurately assess the thickness and extent of intestinal wall thickening in eosinophilic gastroenteritis, enabling layered diagnosis. EUS images clearly show the five layers of the gastrointestinal wall from the inside out: mucosa, submucosa, muscularis propria, and serosa. The training process for the membrane layer segmentation network model is as follows: Endoscopic ultrasound image data from gastrointestinal endoscopy examinations are collected, and the contour boundaries of the five layers are marked on the images. A membrane layer segmentation network model is then trained to extract the contour boundaries of the mucosa, submucosa, muscularis propria, and serosa from the EUS images, allowing for further assessment of whether the mucosa, muscularis propria, and serosa are thickened.

[0067] The following methods are used to determine whether the mucosal layer is thickened: Using the built-in scale of the EUS image, the mucosal layer thickness is measured perpendicular to the tube wall. If the mucosal layer thickness in the gastroscopy ultrasound image is ≥1.5mm, it is considered as mucosal layer thickening. If the mucosal layer thickness in the duodenal and colon ultrasound images is ≥2.0mm, it is considered as mucosal layer thickening.

[0068] The following methods are used to determine whether the muscular layer is thickened: Using the built-in scale of the EUS image, measure the thickness of the intrinsic muscular layer perpendicular to the tube wall. If the muscular layer thickness in the gastroscopic ultrasound image is ≥2.5mm, it is considered as muscular layer thickening. If the muscular layer thickness in the duodenal and colonic ultrasound images is ≥2.0mm, it is considered as muscular layer thickening.

[0069] The following method is used to determine whether the serous film layer has thickened: Using the built-in scale of the EUS image, the thickness of the serous film layer is measured perpendicular to the tube wall. If the thickness of the serous film layer in the ultrasound image is ≥1.5mm, it is considered that the serous film layer has thickened.

[0070] Furthermore, in a specific embodiment, the process of obtaining the mucosal layer thickness is explained using the mucosal layer as an example. The calculation methods for the muscle layer and serosa layer thickness are basically similar and will not be described in detail here. The layered contours in the ultrasound image are identified through a membrane structure segmentation model and redrawn onto a blank image with a black background at EUS size resolution, such as... Figure 2 As shown, the outlines of the mucosa and submucosa are drawn in white on a blank image, and then the thickness of the region from the mucosa to the submucosa is detected. The specific implementation of detecting the thickness of the region from the mucosa to the submucosa is as follows: Obtain the outline point set of the mucosa, the point set of the large outline in the image, calculate the distance from one point to all points of the small outline (submucosa), and the minimum distance is the vertical distance M1 from this point in the mucosa to the submucosa. Calculate the vertical distances M2, M3...Mn from all points on the large outline to the submucosa in sequence. The farthest of these vertical distances is the pixel distance of the mucosa. Using the built-in scale of the image stored in the DICOM file, the pixel distance is converted into the physical size of the mucosa thickness.

[0071] In this embodiment, step S12 involves detecting the lesion area of ​​eosinophilic gastroenteritis based on endoscopic ultrasound images, and determining the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of various membrane structures of the gastrointestinal wall. The specific implementation process is as follows:

[0072] The lesion region in the endoscopic ultrasound image is identified and marked with a bounding box to obtain the lesion rectangle. A blank image of the same size is reconstructed according to the size resolution of the endoscopic ultrasound image. The outlines of the various membrane structures of the gastrointestinal wall and the lesion rectangles are drawn on the blank image using a color different from the background color of the blank image. The overlap status of the lesion rectangle with the outlines of each membrane structure is determined. If the lesion rectangle overlaps with the outline of at least one membrane structure, it indicates that the current lesion is a multi-layer lesion. If the lesion rectangle does not overlap with the outline of any membrane structure, it indicates that the current lesion is a single-layer lesion. Specifically, determining the overlap between the lesion rectangle and the contour boundaries of each membrane structure includes: drawing a binary image mask for each lesion rectangle and a binary image mask for the contour boundaries of each membrane structure; sequentially calculating the intersection region between the binary image mask of the lesion rectangle and the binary image mask of the contour boundaries of each membrane structure, i.e., calculating the intersection region between the lesion region and the thickened region of the membrane structure; if the area of ​​the intersection region is greater than zero, it is determined that the lesion rectangle overlaps with the contour boundary of the current membrane structure. Specifically, identifying the lesion region in the endoscopic ultrasound image and marking it with a bounding box to obtain the lesion rectangle includes: using a pre-trained target detection model to identify the lesion region in the endoscopic ultrasound image and marking it with a bounding box.

[0073] The training process of the target detection model is as follows: Collect endoscopic ultrasound image data from gastrointestinal endoscopy examinations, and mark the lesion areas of eosinophilic gastroenteritis in the images with rectangular boxes, such as... Figure 3 As shown, training a YOLOV8 target detection model for a lesion region can identify the coordinates of the lesion's rectangular region from an input EUS image.

[0074] Furthermore, the rectangular area of ​​the lesion is then compared with the abnormally thickened area of ​​the gastrointestinal wall obtained by the aforementioned operation to verify the coordinate region, and further determine whether there is an overlap between the lesion area and the thickened area and the type of overlap. By the type of overlap between the lesion and the thickened area, it can be determined whether the disease is a multi-layer lesion or a single-layer lesion.

[0075] In one specific embodiment, the layered contour boundaries of each membrane structure and the rectangular bounding box region of the lesion in the ultrasound image are identified and redrawn onto a blank image with a black background at EUS size resolution. Taking the mucosa layer as an example, the contour lines of the mucosa layer, the submucosa layer, and the lesion rectangular bounding box are drawn in white on the blank image. Figure 4 As shown, this example uses the mucosa layer; the same applies to the muscular layer and serosa layer.

[0076] Specific operations: Figure 4The larger outline represents the submucosal layer, while the smaller outline represents the mucosal layer. First, a binary image mask is created for each lesion rectangle, with the inside of the rectangle being white and the outside black; this is called `rectangleImage`. A binary image mask is then created for the submucosal layer outline, with the inside of the outline being white and the outside black; this is called `BigImage`. A binary image mask is also created for the mucosal layer outline, with the inside of the outline being white and the outside black; this is called `SmallImage`. The `rectangleImage` is then intersected with the `BigImage` and `SmallImage` images sequentially using logical operations (such as bitwise AND). If the area of ​​the intersection region is greater than zero, it indicates that the lesion rectangle overlaps with the outline. If the rectangle overlaps with both contours, it indicates that the rectangle spans two membrane layers, and the lesion is a translaminar lesion. If the rectangle does not overlap with either contour, it is further determined whether the rectangle lies between the two contours. The condition for this is determined by calculating whether the coordinates of the rectangle's center point are within the contour range of the submucosa, and simultaneously outside the contour range of the mucosa. In this case, the lesion belongs entirely to the mucosa and is a single-layer lesion. A comprehensive assessment of the lesion is obtained by combining the above judgments on lesion layer distribution with the aforementioned judgments on abnormal thickening of various membrane structures.

[0077] In this embodiment, step S12 involves detecting the boundaries of the fluid accumulation area and the thickened intestinal wall area on the CT scan image to determine whether there is ascites and intestinal wall thickening. The specific implementation process is as follows:

[0078] CT images from each layer are input into a pre-trained volumetric data segmentation network model to obtain a 3D mask image of the current CT layer, which is used to identify the boundaries of the effusion area and the thickened intestinal wall area. The volumetric data segmentation network model is trained on a 3D-Unet convolutional neural network model based on a pre-built dataset. The dataset includes several training sample pairs consisting of original CT images in .nii file format and corresponding 3D mask images in .nii file format with boundaries delineating the effusion area and the thickened intestinal wall area. In the 3D mask data, pixel value 1 represents the effusion area, pixel value 2 represents the thickened intestinal wall area, and pixel 0 is the background area. The size of the thickened intestinal wall and the volume of the abdominal effusion are calculated based on the boundaries of the effusion area and the size of the thickened intestinal wall area in the CT images of each layer. The presence of abdominal effusion and intestinal wall thickening is determined based on the size of the thickened intestinal wall and the volume of the abdominal effusion.

[0079] Specifically, the formula for calculating the volume of accumulated fluid is as follows:

[0080] Fluid volume (ml) = Σ (single-layer cross-sectional area of ​​each CT slice × slice thickness of the corresponding CT slice)

[0081] The area of ​​a single-layer cross section is calculated as: number of pixels × area of ​​a single pixel. The area of ​​a single pixel can be obtained based on the CT resolution.

[0082] Patients with eosinophilic gastroenteritis exhibit intestinal wall thickening and ascites on CT images. This invention employs a pre-trained volume data segmentation network model to identify the boundaries of the ascites region and the intestinal wall thickening region. The training process of the volume data segmentation network model is as follows: CT images of 10,000 patients with eosinophilic gastroenteritis and 20,000 control patients with normal or other gastrointestinal diseases were collected. Data preprocessing was performed: the open-source BOACT site segmentation model (a CT-based body and organ analysis tool for radiologists at the nursing point) was used to preprocess the abdominal CT data, extracting the abdominal region to eliminate interference from external artifacts and images of non-abdominal organs. Image annotation: the ascites region and the intestinal wall thickening region were delineated with boundaries, constructing 3D mask data of the CT volume data in .nii file format. In the 3D mask data, pixel value 1 represents the ascites region, pixel value 2 represents the intestinal wall thickening region, and pixel 0 is the background region. The original CT image is converted into a .nii file format, and the mask volume data of the .nii file is used as input to the 3D-Unet segmentation network. The 3D-Unet segmentation network is trained on the CT volume data to obtain a volume data segmentation network model. This model converts a CT image into a .nii file format, and the neural network identifies and outputs the mask volume data of the .nii file. In the mask volume data, pixel value 1 represents the fluid accumulation area, pixel value 2 represents the intestinal wall thickening area, and pixel 0 is the background area. The boundaries of the fluid accumulation area and the intestinal wall thickening area are identified, and a binary mask is output for each CT slice (0 for background, 1 for fluid accumulation area, and 2 for intestinal wall thickening area). Then, the volume of intestinal wall thickening and ascites is calculated sequentially. If the volume exceeds a set threshold, the CT image is determined to be intestinal wall thickening and ascites.

[0083] In an optional embodiment of the present invention, the multimodal data is not limited to digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical manifestation data, but may also include biopsy pathology reports, blood test reports, and stool test reports. Furthermore, when acquiring data collected from the subject within the same time period, the present invention also includes acquiring the subject's biopsy pathology reports, blood test reports, and stool test reports within the same time period.

[0084] Furthermore, before combining the obtained numerical feature representations to generate a feature vector based on multimodal data, the process includes: extracting the eosinophil index category of the subject based on the biopsy pathology report (eosinophil index categories include normal, elevated, and very elevated), and obtaining a sixth numerical feature representation of the eosinophil index category using numerical encoding; extracting the blood eosinophil index category of the subject based on the blood test report, and obtaining a seventh numerical feature representation of the blood eosinophil index category using numerical encoding; and extracting the parasite infection status of the subject based on the stool test report, and obtaining an eighth numerical feature representation of the parasite infection status using numerical encoding. When combining the obtained numerical feature representations to generate a feature vector based on multimodal data, all numerical feature representations are combined to generate the feature vector based on multimodal data.

[0085] Specifically, the eosinophilic gastroenteritis (EGE) classification method based on multimodal data collects historical data from different patients in advance, including patient-reported clinical manifestations, blood test reports, stool test reports, CT images, endoscopic images, biopsy pathology reports, endoscopic ultrasound images, and EGE classification (mucosal type (Type I), muscular type (Type II), and serosal type (Type III)).

[0086] The clinical data reported by patients were processed using NLP to determine whether symptoms such as nausea, vomiting, abdominal pain, and diarrhea were present, and the data were represented in decimal and binary formats, as shown in Table 1.

[0087] Table 1. Representation of symptoms corresponding to clinical manifestation data

[0088] Chief complaint symptoms Decimal and binary representations Asymptomatic 0 0000 nausea 1 0001 Vomit 2 0010 stomach ache 3 0011 diarrhea 4 0100

[0089] This study extracts data from patients' blood test reports, such as eosinophil counts in routine blood tests, to determine if there is an increase in eosinophils. It also extracts data from patients' stool tests to detect parasitic infections. CT scan reports are used to identify intestinal wall thickening and ascites. Endoscopic lesions and pathological biopsy reports are analyzed to determine if eosinophil counts are ≥20 / HP. EUS images are used to depict abnormal thickening of the gastrointestinal wall, lesion layer distribution, and EGE classification, all represented numerically to construct a multimodal database of eosinophilic gastroenteritis. Custom feature vectors are generated based on the structured attribute values ​​of this database.

[0090] Furthermore, the clinical manifestation data reported by patients were integrated and simplified to reduce the number of categories, and then converted into One-Hot encoded features to obtain the corresponding numerical feature representations, as shown in Table 2:

[0091] Table 2 Numerical characteristics of clinical manifestation data

[0092] Before integration After integration Asymptomatic Asymptomatic 0000 nausea Upper gastrointestinal symptoms 0001 Vomit Upper gastrointestinal symptoms 0001 stomach ache Lower gastrointestinal symptoms 0010 diarrhea Lower gastrointestinal symptoms 0010

[0093] If a patient has multiple symptoms, they are added together. For example, if a patient has both upper and lower gastrointestinal symptoms, then 0001 + 0010 = 0011.

[0094] The eosinophil count in the patient's blood routine was classified into three categories: normal, elevated, and very elevated, and then converted into One-Hot coding features.

[0095] The fecal examination indicators of the patients were extracted to determine whether there was a parasitic infection, using the numbers 0 and 1, and converted into One-Hot encoded features.

[0096] Extracting information from CT scan reports, identifying intestinal wall thickening using numbers 0 and 1, and converting it into One-Hot encoded features.

[0097] Whether there is ascites is determined by using the numbers 0 and 1, and then converted into a One-Hot encoded feature.

[0098] The lesion status of the endoscope is extracted and divided into normal, mucosal abnormality non-early cancer, and early cancer, using the numbers 0, 1, and 2, and converted into One-Hot coding features.

[0099] Eosinophil counts extracted from endoscopic pathological biopsy reports were categorized into three classes: normal, elevated, and very elevated, and then converted into One-Hot coding features.

[0100] The abnormal thickening of the gastrointestinal wall in EUS images was classified into three categories: normal, high, and very high, and then converted into One-Hot encoded features.

[0101] EGE is classified into four categories: non-EGE, mucosal type, muscular type, and serosa type.

[0102] The structured data described above is used to construct a feature vector, generating a feature vector based on multimodal data, which can then be used to train a dynamic genotyping prediction model.

[0103] When predicting the eosinophilic gastroenteritis (EGE) subtype of new patients, the following data are obtained sequentially: endoscopic lesion status, EUS gastrointestinal wall thickening, CT scan intestinal wall thickening, ascites, and patient blood routine, stool examination, and pathology report content. These data are converted into One-Hot codes and feature vectors, which are then input into the trained dynamic subtype prediction model to achieve EGE subtype prediction.

[0104] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0105] Example 2

[0106] This invention provides a multimodal data-based eosinophilic gastroenteritis typing system, the system comprising a functional module for implementing the multimodal data-based eosinophilic gastroenteritis typing method as described in any of the preceding embodiments. Figure 5 The schematic diagram illustrates the structure of an eosinophilic gastroenteritis classification system based on multimodal data provided in an embodiment of the present invention. (Refer to...) Figure 5 The system described in this embodiment of the invention includes:

[0107] The acquisition module 501 is used to acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical manifestation data of the subject within the same time period.

[0108] The symptom recognition and feature representation module 502 is used to detect abnormal areas of the gastrointestinal mucosa in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions, and to obtain a first numerical feature representation of the state of gastrointestinal mucosal lesions using a numerical encoding method; to detect the contour boundaries of each membrane structure of the gastrointestinal wall in the endoscopic ultrasound images and to identify abnormal thickening of each membrane structure of the gastrointestinal wall, and to obtain a second numerical feature representation of the abnormal thickening of the membrane structure of the gastrointestinal wall using a numerical encoding method; to detect the lesion area of ​​eosinophilic gastroenteritis in the endoscopic ultrasound images and to determine the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of each membrane structure of the gastrointestinal wall, and to obtain a third numerical feature representation of the lesion layer distribution using a numerical encoding method; to detect the boundaries of the effusion area and the thickened area of ​​the intestinal wall in the CT examination images to determine whether there is ascites and intestinal wall thickening, and to obtain a fourth numerical feature representation of ascites and intestinal wall thickening using a numerical encoding method; and to extract the clinical symptom features of the subject based on the clinical manifestation data, and to obtain a fifth numerical feature representation of the clinical symptom features using a numerical encoding method.

[0109] The multimodal feature generation module 503 is used to combine the obtained numerical feature representations to generate feature vectors based on multimodal data;

[0110] The typing prediction module 504 is used to input the feature vector based on multimodal data into a pre-trained dynamic typing prediction model to obtain the typing category prediction result of the eosinophilic gastroenteritis of the test subject.

[0111] In this embodiment of the invention, the acquisition module 501 is also used to acquire the biopsy pathology report, blood test report and stool test report of the subject within the same time period;

[0112] Before the multimodal feature generation module 503 combines the obtained numerical feature representations to generate a feature vector based on multimodal data, the symptom recognition and feature representation module 502 is also used to extract the eosinophil index category of the subject based on the biopsy pathology report and obtain the sixth numerical feature representation of the eosinophil index category using numerical encoding; extract the blood eosinophil index category of the subject based on the blood test report and obtain the seventh numerical feature representation of the blood eosinophil index category using numerical encoding; and extract the parasite infection status of the subject based on the stool test report and obtain the eighth numerical feature representation of the parasite infection status using numerical encoding.

[0113] In this embodiment of the invention, the symptom recognition and feature representation module 502 includes a gastrointestinal wall abnormal thickening detection unit, used to detect the contour boundaries of various membrane structures of the gastrointestinal wall in endoscopic ultrasound images and identify the abnormal thickening of each membrane structure. Specifically, it is used to identify the contour boundaries of the mucosa, submucosa, muscularis propria, and serosa in endoscopic ultrasound images; to detect the thickness of the mucosa, muscularis propria, and serosa using an image-built-in scale; and to compare the measured values ​​with preset standard values ​​to determine the abnormal thickening of each membrane structure of the gastrointestinal wall. The method of detecting mucosal layer thickness using an image-embedded ruler includes: reconstructing a blank image of the same size based on the size resolution of the endoscopic ultrasound image; drawing the contour boundaries of the mucosal layer and the submucosal layer onto the blank image using a color different from the background color of the blank image; acquiring the contour point sets of the mucosal layer and the submucosal layer; calculating the distance from each point in the mucosal layer contour point set to each point in the submucosal layer, and taking the minimum distance as the vertical distance from the current point in the mucosal layer to the submucosal layer; sequentially calculating the vertical distance from all points in the mucosal layer to the submucosal layer, taking the farthest calculated vertical distance as the pixel distance of the mucosal layer, and converting the pixel distance into the physical size of the mucosal layer thickness using the image-embedded ruler.

[0114] In this embodiment of the invention, the symptom identification and feature representation module 502 includes a lesion layer detection unit, used to detect the lesion area of ​​eosinophilic gastroenteritis based on endoscopic ultrasound images, and to determine the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of various membrane structures of the gastrointestinal wall. Specifically, it is used to: identify the lesion area in the endoscopic ultrasound image and mark it with a marker box to obtain a lesion rectangle; reconstruct a blank image of the same size according to the size resolution of the endoscopic ultrasound image; draw the contour boundaries of various membrane structures of the gastrointestinal wall and the lesion rectangle on the blank image using a color different from the background color of the blank image; determine the overlap state between the lesion rectangle and the contour boundaries of each membrane structure; if the lesion rectangle overlaps with the contour boundary of at least one membrane structure, it indicates that the current lesion is a multi-layer lesion; if the lesion rectangle does not overlap with the contour boundaries of any membrane structure, it indicates that the current lesion is a single-layer lesion. The process of determining the overlap between the lesion rectangle and the contour boundaries of each membrane structure includes: drawing a binary image mask for each lesion rectangle and drawing a binary image mask for the contour boundaries of each membrane structure; sequentially calculating the intersection region between the binary image mask of the rectangle and the binary image mask of the contour of each membrane structure; if the area of ​​the intersection region is greater than zero, it is determined that the lesion rectangle overlaps with the contour boundary of the current membrane structure.

[0115] In this embodiment of the invention, the symptom recognition and feature representation module 502 includes an intestinal wall thickening and ascites detection unit, which is used to detect the boundary of the ascites area and the intestinal wall thickening area on CT scan images to determine whether there is ascites and intestinal wall thickening. Specifically, this includes: inputting CT images from each layer into a pre-trained volume data segmentation network model to obtain a 3D mask image of the current CT layer, in order to identify the boundaries of the effusion area and the intestinal wall thickening area; wherein, the volume data segmentation network model is obtained by training a 3D-Unet convolutional neural network model based on a pre-constructed dataset, the dataset including several training sample pairs consisting of original CT images in nii file format and corresponding 3D mask images in nii file format with boundaries delineated for the effusion area and the intestinal wall thickening area, wherein in the 3D mask data, pixel value 1 represents the effusion area, pixel value 2 represents the intestinal wall thickening area, and pixel 0 is the background area; calculating the intestinal wall thickening size and the ascites volume based on the effusion area boundaries and the size of the intestinal wall thickening area in the CT images of each layer; and determining whether ascites and intestinal wall thickening exist based on the intestinal wall thickening size and the ascites volume.

[0116] In the specific implementation process of Embodiment 2, you can refer to Embodiment 1, and it has the corresponding technical effects.

[0117] Example 3

[0118] This invention provides a computer program product storing a computer program. When executed by a processor, the computer program implements the steps described in the above embodiment of the eosinophilic gastroenteritis classification method based on multimodal data, for example... Figure 1 Steps S11-S14 are shown.

[0119] In the specific implementation process of Example 3, reference can be made to Example 1, and it has the corresponding technical effects.

[0120] Example 4

[0121] This invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program performs the steps described in the above embodiment of the eosinophilic gastroenteritis classification method based on multimodal data. Figure 1 Steps S11-S14 are shown.

[0122] In its specific implementation, Example 4 can be referred to Example 1, and has the corresponding technical effects.

[0123] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A classification system for eosinophilic gastroenteritis based on multimodal data, characterized in that, The system includes: The acquisition module is used to acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical data of the subject within the same time period. The symptom identification and feature representation module is used to detect abnormal areas of the gastrointestinal mucosa in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions, and to obtain a first numerical feature representation of the state of gastrointestinal mucosal lesions using a numerical encoding method; to detect the contour boundaries of each membrane structure of the gastrointestinal wall in the endoscopic ultrasound images and identify abnormal thickening of each membrane structure of the gastrointestinal wall, and to obtain a second numerical feature representation of the abnormal thickening of the membrane structure of the gastrointestinal wall using a numerical encoding method; to detect the lesion area of ​​eosinophilic gastroenteritis in the endoscopic ultrasound images and to determine the lesion layer distribution based on the positional relationship between the lesion area and the contour boundaries of each membrane structure of the gastrointestinal wall, and to obtain a third numerical feature representation of the lesion layer distribution using a numerical encoding method; to detect the boundaries of the effusion area and the thickened area of ​​the intestinal wall in the CT examination images to determine whether there is ascites and intestinal wall thickening, and to obtain a fourth numerical feature representation of ascites and intestinal wall thickening using a numerical encoding method; and to extract the clinical symptom features of the subject based on the clinical manifestation data, and to obtain a fifth numerical feature representation of the clinical symptom features using a numerical encoding method. The multimodal feature generation module is used to combine the obtained numerical feature representations to generate feature vectors based on multimodal data. The typing prediction module is used to input the feature vector based on multimodal data into a pre-trained dynamic typing prediction model to obtain the typing category prediction result of the eosinophilic gastroenteritis of the test subject.

2. The system according to claim 1, characterized in that, The acquisition module is also used to acquire biopsy pathology reports, blood test reports and stool test reports of the subject within the same time period; Before the multimodal feature generation module combines the obtained numerical feature representations to generate a feature vector based on multimodal data, the symptom recognition and feature representation module is also used to extract the eosinophil index category of the subject based on the biopsy pathology report, and to obtain the sixth numerical feature representation of the eosinophil index category using a numerical encoding method. Based on the blood test report, the blood eosinophil index category of the subject is extracted, and the seventh numerical feature representation of the blood eosinophil index category is obtained by numerical coding method; Based on the fecal examination report, the parasite infection status of the subjects was extracted, and the eighth numerical feature representation of the parasite infection status was obtained by numerical coding.

3. The system according to claim 1, characterized in that, The symptom identification and feature representation module includes a gastrointestinal wall abnormal thickening detection unit, which is used to identify the contour boundaries of the mucosa, submucosa, muscularis propria and serosa in endoscopic ultrasound images; the thickness of the mucosa, muscularis propria and serosa is detected by using the built-in scale of the image, and the measured values ​​are compared with preset standard values ​​to determine the abnormal thickening of each membrane structure of the gastrointestinal wall.

4. The system according to claim 3, characterized in that, The method of detecting mucosal layer thickness using an image-embedded ruler includes: reconstructing a blank image of the same size and resolution as the ultrasound endoscopic image; drawing the contour boundaries of the mucosal layer and the submucosal layer onto the blank image using a color different from the background color of the blank image; acquiring the contour point sets of the mucosal layer and the submucosal layer; calculating the distance from each point in the mucosal layer contour point set to each point in the submucosal layer, and taking the minimum distance as the vertical distance from the current point in the mucosal layer to the submucosal layer; sequentially calculating the vertical distances from all points in the mucosal layer to the submucosal layer, taking the farthest calculated vertical distance as the pixel distance of the mucosal layer, and converting the pixel distance into the physical size of the mucosal layer thickness using the image-embedded ruler.

5. The system according to claim 1, characterized in that, The symptom recognition and feature representation module includes a lesion layer detection unit, which is used to identify lesion areas in endoscopic ultrasound images and mark them with a bounding box to obtain lesion rectangles; reconstruct a blank image of the same size according to the size resolution of the endoscopic ultrasound image; draw the outlines of each membrane structure of the gastrointestinal wall and the lesion rectangles onto the blank image using a color different from the background color of the blank image; determine the overlap status of the lesion rectangles with the outlines of each membrane structure. If the lesion rectangle overlaps with the outline of at least one membrane structure, it indicates that the current lesion is a multi-layer lesion; if the lesion rectangle does not overlap with the outline of any membrane structure, it indicates that the current lesion is a single-layer lesion.

6. The system according to claim 5, characterized in that, The determination of the overlap between the lesion rectangle and the contour boundaries of each membrane structure includes: drawing a rectangular frame binary image mask for each lesion rectangle, and drawing a contour binary image mask for the contour boundaries of each membrane structure; sequentially calculating the intersection region between the rectangular frame binary image mask and the contour binary image mask of each membrane structure; if the area of ​​the intersection region is greater than zero, it is determined that the lesion rectangle overlaps with the contour boundary of the current membrane structure.

7. The system according to claim 1, characterized in that, The symptom recognition and feature representation module includes an intestinal wall thickening and ascites detection unit. This unit inputs CT images from various layers into a pre-trained volumetric data segmentation network model to obtain a 3D mask image of the current CT layer, thereby identifying the boundaries of the effusion area and the intestinal wall thickening area. The volumetric data segmentation network model is trained on a pre-constructed dataset using a 3D-Unet convolutional neural network model. The dataset includes several training sample pairs consisting of original CT images in .nii file format and corresponding 3D mask images in .nii file format with boundaries delineating the effusion area and intestinal wall thickening area. In the 3D mask data, pixel value 1 represents the effusion area, pixel value 2 represents the intestinal wall thickening area, and pixel 0 is the background area. The module calculates the intestinal wall thickening size and ascites volume based on the boundaries of the effusion area and the size of the intestinal wall thickening area in the CT images of each layer. The presence of ascites and intestinal wall thickening is determined based on the intestinal wall thickening size and ascites volume.

8. An electronic device, characterized in that, Includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when executed by the processor, the computer program implements the following method: Acquire digestive endoscopy images, endoscopic ultrasound images, CT scan images, and clinical data of the subjects within the same time period; The gastrointestinal mucosa abnormality areas are detected in the digestive endoscopy images to identify the state of gastrointestinal mucosal lesions, and the first numerical feature representation of the state of gastrointestinal mucosal lesions is obtained by numerical coding. The contour boundaries of each membrane layer in the gastrointestinal wall in endoscopic ultrasound images are detected, and the abnormal thickening of each membrane layer is identified. The second numerical feature representation of the abnormal thickening of the gastrointestinal wall membrane layer is obtained by numerical coding. The lesion area of ​​eosinophilic gastroenteritis was detected by endoscopic ultrasound images, and the lesion layer distribution was determined based on the positional relationship between the lesion area and the contour boundaries of each membrane layer of the gastrointestinal wall. The third numerical feature representation of the lesion layer distribution was obtained by numerical coding. The boundaries of the effusion area and the thickened area of ​​the intestinal wall were detected on CT scan images to determine whether there was ascites and intestinal wall thickening. The fourth numerical feature representation of ascites and intestinal wall thickening was obtained by numerical coding. The clinical symptom features of the subjects were extracted based on clinical manifestation data, and the fifth numerical feature representation of the clinical symptom features was obtained by numerical coding. The obtained numerical feature representations are combined to generate feature vectors based on multimodal data; The feature vector based on the multimodal data is input into a pre-trained dynamic typing prediction model to obtain the predicted typing category of eosinophilic gastroenteritis for the subject.

Images