Ucerative colitis virtual biopsy method and system based on multi-modal image

Abstract

The invention belongs to the technical field of medical auxiliary diagnosis, and discloses an ulcerative colitis virtual biopsy method and system based on a multi-modal image, and the method comprises the steps: building an ulcerative colitis medical image database; constructing a prediction data set according to the extracted quantitative features and the pathological grades of the ulcerative colitis patients; based on the prediction data set, constructing and training an ulcerative colitis virtual biopsy model; and acquiring new clinical data of a patient with ulcerative colitis in real time, inputting the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and outputting a pathological grading result of ulcerative colitis through extraction and prediction of pathological features. According to the method, automatic analysis of image and endoscopic image features of the lesion area can be realized, the pathological activity state of the ulcerative colitis patient can be accurately judged, and association with long-term prognosis of the patient is established, so that help is provided for clinicians to formulate an individualized UC treatment scheme.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary diagnosis, and in particular to a virtual biopsy method and system for ulcerative colitis based on multimodal images. Background Art

[0002] Ulcerative colitis (UC) is a subtype of inflammatory bowel disease (IBD), characterized by chronic nonspecific intestinal inflammation. In the past, clinical remission based on subjective symptom improvement was considered the main treatment goal of UC. In recent years, the treatment goals of UC have gradually evolved. In 2015, the STRIDE-I consensus released by the International Inflammatory Bowel Disease Research Organization introduced treat to target for the first time in the field of IBD, and clearly proposed that in addition to clinical remission, endoscopic remission of UC patients should be the treatment goal. However, more and more evidence shows that clinical remission or endoscopic remission cannot accurately predict the development of the natural course of UC, and only treating symptoms or maintaining endoscopic remission is not enough to prevent long-term complications or reduce the risk of recurrence. Many studies have shown that nearly half of patients still have persistent histological inflammation in the intestine even when they are in clinical and endoscopic remission, which is strongly correlated with the risk of clinical recurrence, colectomy and cancer in UC. Therefore, with the advancement of UC treatment methods and the improvement of patients' treatment expectations, the ultimate treatment goal of UC is expected to change from "Treat to target" to "Treat to clear", and histological remission will become the ideal outcome pursued by UC patients and clinicians.

[0003] Although UC histological scoring can more sensitively predict the progression and prognosis of the disease and has the feasibility of becoming a treatment target for UC, histopathological evaluation faces many difficulties in clinical practice. First, endoscopic biopsy is an invasive procedure, and its safety issues cannot be ignored. According to the 2020 ECCO consensus and the consensus on histopathological diagnosis of IBD, mucosal tissues from at least 5 sites (including the rectum and the terminal ileum) should be clamped for UC biopsy, and no less than 2 grains of tissue should be taken from each site. Biopsies should be performed in the most severely inflamed areas and ulcer edges, and mucosa without abnormalities under endoscopy should also be biopsied. If white light endoscopy is used, it is recommended to take 4 grains of tissue every 10 cm of colon, as well as any visible lesions in the entire colon. It can be seen that the UC biopsy process is quite complicated, with many biopsies, and its operability and safety in clinical work still need to be further verified. Secondly, colonoscopic biopsy depends on the operator's experience, and different endoscopists or pathologists may have different judgments on the characteristics of mucosal inflammation. The biopsy results are highly subjective and relatively poor in repeatability. It is worth noting that for patients with severe UC, especially those with acute severe ulcerative colitis (ASUC), although endoscopic examination and pathological evaluation have certain value in predicting and evaluating the response to drug rescue therapy, the risk of biopsy of intestinal mucosal tissue in ASUC patients is significantly increased. Figure 4 The figure below (preoperative endoscopic, imaging and postoperative pathological data of the patient, arrows indicate diseased intestinal tubes) is a clinical case of ASUC, which fully illustrates the risks of ASUC mucosal biopsy. The patient's preoperative colonoscopy showed extensive colonic lesions, and CT showed thick and stiff colonic walls with enlarged mesenteric lymph nodes. After salvage treatment, colonoscopy showed lesions in the entire colon, extensive congestion and edema of the mucosa, and partial mucosal loss. Sigmoid colon biopsy was attempted to assess the level of tissue inflammation, but unfortunately, intestinal perforation occurred, leading to emergency colectomy. Postoperative pathological results showed that the intestinal mucosa (entire colon) showed chronic active inflammation with superficial erosion and ulceration, the number of glands was significantly reduced, the ulcers were deep to the superficial muscle layer, and the whole layer of the tube wall was focally penetrated; the interstitial granulation tissue proliferated, a large number of lymphocytes, plasma cells and neutrophils infiltrated, blood vessels proliferated, dilated and congested, the submucosal layer was focally loose and edematous, the serosal surface was chronically active, and the lymph nodes around the intestine showed reactive hyperplasia. It can be seen that mucosal biopsy of ASUC patients is a very difficult choice for endoscopists, and how to safely, accurately and continuously monitor tissue inflammation levels is a clinical problem that needs to be solved urgently.

[0004] Therefore, how to provide a virtual biopsy method and system for ulcerative colitis based on multimodal images is a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a multimodal image-based ulcerative colitis virtual biopsy method and system to solve the problems in the prior art.

[0006] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0007] According to a first aspect of an embodiment of the present invention, a multimodal image-based virtual biopsy method for ulcerative colitis is provided.

[0008] In one embodiment, a multimodal image-based virtual biopsy method for ulcerative colitis includes:

[0009] Collect clinical data of patients with ulcerative colitis and perform image data quality processing on the clinical data; determine the pathological grade of patients with ulcerative colitis based on the processing results and establish a medical image database of ulcerative colitis, wherein the clinical data includes basic information of patients with ulcerative colitis, colonoscopy images and computer tomography images;

[0010] Perform image processing on colonoscopy images and computed tomography images in the ulcerative colitis medical image database, extract pathological quantitative features from the processed colonoscopy images and computed tomography images, and construct a prediction data set based on the extracted quantitative features and the pathological grading of ulcerative colitis patients;

[0011] Based on the prediction dataset, a deep convolutional neural network was used in combination with the Transformer network structure to build and train a virtual biopsy model for ulcerative colitis.

[0012] Acquire new clinical data of patients with ulcerative colitis in real time, input the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and output the pathological grading results of ulcerative colitis through the extraction and prediction of pathological features.

[0013] In one embodiment, the steps of collecting clinical data of patients with ulcerative colitis, performing image data quality processing on the clinical data, determining the pathological grade of the patients with ulcerative colitis according to the processing results, and establishing a medical image database of ulcerative colitis include the following steps:

[0014] Collect basic information, colonoscopy images, computed tomography images, and pathological section scans of patients with ulcerative colitis;

[0015] The clinical data of patients with ulcerative colitis were screened and collated according to pre-set screening criteria;

[0016] Perform pathological section analysis on the screened and sorted pathological section scan images, and determine the pathological grade of ulcerative colitis patients using pre-set pathological scoring standards;

[0017] Based on the basic information of patients with ulcerative colitis, colonoscopy images, computed tomography images and pathological grading are stored to obtain a medical image database of ulcerative colitis.

[0018] In one embodiment, the image processing of the colonoscopy images and the computed tomography images in the ulcerative colitis medical image database, and the extraction of pathological quantitative features from the colonoscopy images and the computed tomography images after the image processing, and the construction of the prediction data set according to the extracted quantitative features and the pathological grade of the ulcerative colitis patients include the following steps:

[0019] The semantic segmentation method is used to segment the colonoscopy images, and the Transformer network structure is used to extract the features of the segmented colonoscopy images to obtain colonoscopy features.

[0020] The filter is used to perform filter transformation on the computed tomography image to obtain a filter transformation image, and the radiomics features are extracted from the computed tomography image and the filter transformation image by using open source software to obtain radiomics features;

[0021] Colonoscopy features and radiomics features were used as pathological quantitative features of patients with ulcerative colitis, and the quantitative features were combined with the pathological grades of patients with ulcerative colitis to obtain a prediction data set.

[0022] In one embodiment, the method of segmenting the colonoscopy image using a semantic segmentation method and extracting features from the segmented colonoscopy image using a Transformer network structure to obtain colonoscopy features includes the following steps:

[0023] Preprocess the colonoscopy images, segment the preprocessed colonoscopy images into multiple image blocks using semantic segmentation method, and map each image block into a linear embedding sequence;

[0024] The encoder of the Transformer network is used to encode the linear embedding sequence and extract the features of the image block; the decoder of the Transformer network is used to decode the encoded image block features and generate the segmented pixel classification map through upsampling and softmax function;

[0025] According to the segmented pixel classification map, the features of the lesion area are extracted, and the extracted features are used as colonoscopy features.

[0026] In one embodiment, the preprocessing of the colonoscopy image comprises the following steps:

[0027] The ulcerative colitis lesion area in the colonoscopic image is set and annotated, and the invalid background of each colonoscopic image is removed according to the annotation results to obtain a valid intestinal image;

[0028] The effective intestinal image is adjusted to a uniform size, and the pixel value of the resized image is normalized to convert the pixel value of the intestinal image into a preset range;

[0029] The normalized intestinal image is subjected to data enhancement processing by using the data enhancement method to obtain the preprocessed intestinal image.

[0030] In one embodiment, the filtering and transforming of the computed tomography image using a filter to obtain a filtered transformed image, and extracting radiomics features from the computed tomography image and the filtered transformed image using open source software to obtain radiomics features include the following steps:

[0031] Transforming an image sequence of a computed tomography image using a filter to generate a filtered transformed image;

[0032] The diseased tissue areas of the computed tomography images and the filtered transformed images were annotated in three dimensions using open source software and annotation masks were generated;

[0033] Based on the annotated masks, the initial radiomic features of the lesion area were extracted from the computed tomography images and the filtered transformed images using radiomics analysis tools;

[0034] The extracted radiomics features were analyzed for consistency and correlation using correlation analysis, and redundant features were screened and eliminated to obtain the final radiomics features.

[0035] In one embodiment, the imaging omics features include: shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level run-length matrix features, gray-level region size matrix features, and gray-level dependence matrix features.

[0036] In one embodiment, the method of constructing and training a virtual biopsy model for ulcerative colitis based on a prediction data set using a deep convolutional neural network in combination with a Transformer network structure comprises the following steps:

[0037] The prediction data set is divided into a training set and a validation set, and the pathological quantitative features and pathological grades in the prediction data set are used as training labels;

[0038] Construct a network structure with dual inputs and combine it with a deep convolutional neural network and a Transformer network structure to establish a virtual biopsy model for ulcerative colitis;

[0039] The elastic network algorithm was used to perform 10-fold cross validation in the training set. Based on the validation results, the pathological quantitative features most relevant to the pathological grading were screened out. The ulcerative colitis virtual biopsy model was trained based on the screened pathological quantitative features and pathological grading in combination with the loss function.

[0040] The performance of the ulcerative colitis virtual biopsy model was evaluated using the validation set, and the parameters of the ulcerative colitis virtual biopsy model were optimized based on the evaluation results to obtain the final ulcerative colitis virtual biopsy model.

[0041] In one embodiment, the loss function is expressed as:

[0042] J(θ)=MSE(θ)+θλ(0.3Σ∣θ i ∣+0.7Σθ i 2 )

[0043] Where J(θ) represents the loss function, MSE represents the mean standard error, θ represents the model parameter set, λ represents the regularization coefficient, and Σ|θ i ∣ represents the L1 regularization term, Σθ i 2 represents the L2 regularization term.

[0044] According to a second aspect of an embodiment of the present invention, a multimodal image-based ulcerative colitis virtual biopsy system is provided.

[0045] In one embodiment, a multimodal image-based ulcerative colitis virtual biopsy system comprises:

[0046] A database construction module is used to collect clinical data of patients with ulcerative colitis and perform image data quality processing on the clinical data; determine the pathological grade of patients with ulcerative colitis according to the processing results and establish a medical image database of ulcerative colitis, wherein the clinical data includes basic information of patients with ulcerative colitis, colonoscopy images and computer tomography images;

[0047] A data set construction module, for performing image processing on colonoscopy images and computer tomography images in the ulcerative colitis medical image database, extracting pathological quantitative features from the colonoscopy images and computer tomography images after image processing, and constructing a prediction data set based on the extracted quantitative features and the pathological grading of ulcerative colitis patients;

[0048] A model building module is used to build and train a virtual biopsy model for ulcerative colitis based on a prediction dataset using a deep convolutional neural network combined with a Transformer network structure;

[0049] The pathology prediction module is used to obtain new clinical data of patients with ulcerative colitis in real time, input the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and output the pathological grading results of ulcerative colitis through the extraction and prediction of pathological features.

[0050] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0051] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0052] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0053] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0054] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0055] 1. The present invention uses virtual biopsy technology to analyze the CT images and matching endoscopic images of patients with ulcerative colitis. By integrating colonoscopy images and CT imaging features through a multimodal artificial intelligence method, a dual-channel data model is constructed to realize automatic analysis of the image and endoscopic image features of the lesion area, accurately determine the pathological activity status of patients with ulcerative colitis, and establish a correlation with the long-term prognosis of patients, thereby providing assistance to clinical physicians in formulating individualized UC treatment plans.

[0056] 2. The present invention applies endoscopic images and CT imaging features to analyze the pathological grading of UC tissues. Endoscopic images can be retrieved from the medical record system, and quantitative imaging features can be extracted from CT images routinely used in UC clinical practice. It has low economic cost and strong repeatability, thus making up for the problems of traditional invasive colonoscopic pathology evaluation that is painful, time-consuming, and difficult to repeat to a certain extent.

[0057] 3. The present invention introduces deep learning algorithms represented by Transformer and DCNN to systematically analyze medical high-dimensional data. Deep learning algorithms can automatically integrate input data based on their inherent mathematical characteristics in the absence of prior knowledge, and explore the potential connection between various factors and clinical outcomes, thereby achieving accurate prediction of clinical outcomes. Therefore, using deep learning technology for fusion image feature analysis can more accurately achieve non-invasive assessment of UC pathological grading.

[0058] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0060] Figure 1 is a flow chart showing a virtual biopsy method for ulcerative colitis based on multimodal images according to an exemplary embodiment;

[0061] Figure 2 is a principle block diagram of a multimodal image-based ulcerative colitis virtual biopsy according to an exemplary embodiment;

[0062] Figure 3 is a structural schematic diagram of a computer device according to an exemplary embodiment;

[0063] Figure 4 is a schematic diagram of the states of preoperative endoscopy, imaging and postoperative pathology of a patient in a virtual biopsy method for ulcerative colitis based on multimodal images according to an exemplary embodiment;

[0064] Figure 5 is a schematic diagram of three-dimensional annotation of lesion areas of UC patients in a virtual biopsy method for ulcerative colitis based on multimodal images according to an exemplary embodiment;

[0065] Figure 6 A dual-input network structure in a multimodal image-based virtual biopsy method for ulcerative colitis is shown according to an exemplary embodiment;

[0066] Figure 7 It is a technical roadmap for establishing a UC pathology virtual biopsy system in a multimodal image-based ulcerative colitis virtual biopsy method according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0068] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0069] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0070] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0071] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0072] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0073] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0074] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0075] Figure 1 An embodiment of the multimodal image-based ulcerative colitis virtual biopsy method of the present invention is shown.

[0076] In this optional embodiment, the multimodal image-based ulcerative colitis virtual biopsy method comprises:

[0077] Step S101, collecting clinical data of patients with ulcerative colitis, performing image data quality processing on the clinical data; determining the pathological grade of the patients with ulcerative colitis according to the processing results and establishing a medical image database of ulcerative colitis, wherein the clinical data includes basic information of the patients with ulcerative colitis, colonoscopy images and computer tomography images;

[0078] In this optional embodiment, when collecting clinical data of patients with ulcerative colitis, image data quality processing is performed on the clinical data; when determining the pathological grade of patients with ulcerative colitis based on the processing results and establishing a medical image database for ulcerative colitis, basic information, colonoscopy images, computer tomography images and pathological section scan images of patients with ulcerative colitis can be collected; the clinical data of patients with ulcerative colitis are screened and sorted according to pre-set screening criteria; pathological section analysis is performed on the screened and sorted pathological section scan images, and the pathological grade of patients with ulcerative colitis is determined according to pre-set pathological scoring criteria; based on the basic information of patients with ulcerative colitis, colonoscopy images, computer tomography images and pathological grades are stored to obtain a medical image database for ulcerative colitis.

[0079] It should be noted that clinical data (including general information of patients, laboratory test results, pathological sections, pathological molecular experimental results), abdominal / pelvic CT, colonoscopy images and pathological sections or their scanned images of UC patients who received standardized medical or surgical treatment in the hospital were collected retrospectively. Patient inclusion criteria were: ① Patients diagnosed with UC according to the WHO 2017 diagnostic criteria, combined with clinical, endoscopic and pathological results; ② MDCT examination, the scanning range must include the entire colorectum; ③ Undergoing colonoscopy, and the interval between CT and colonoscopy should not exceed 2 weeks; ④ Standardized multi-site pathological biopsy. Exclusion criteria: ① Patients who underwent colectomy for UC; ② Patients with poor intestinal preparation (Boston score less than 6 points); ③ Patients with massive intestinal bleeding that made it difficult to observe intestinal mucosal lesions; ④ Patients with poor colonoscopy imaging quality and unreasonable magnification that resulted in unclear surface texture; ⑤ Patients with insufficient pathological specimens or poor section quality that made it difficult to perform pathological scoring. Through telephone follow-up, the prognostic information of the included patients was investigated, including recurrence, surgical conditions, surgical time, upgraded medical treatment and UC-related tumor occurrence, etc. Ultimately, a comprehensive UC patient database was established, including at least 500 UC patients (approximately 2,500 intestinal segments), integrating imaging, pathology and prognostic information, thereby establishing a data foundation for subsequent research.

[0080] Three experienced endoscopists defined the lesion area, marked the sampling site according to the endoscopic report, and removed the invalid background of each image. Each patient had 25-35 colonoscopic images, and all endoscopic images belonging to the same intestinal segment of the same patient were marked with the same label. Three gastrointestinal imaging experts with more than 10 years of work experience were invited to use the open source annotation tool 3D Slicer to perform 3D stereoscopic annotation of the UC pathological biopsy intestinal area in MD CT. The labeling files of the same intestinal segment of the same patient were named with the same label, and experimental features were extracted for consistency testing. Three pathologists analyzed the pathological sections or pathological section scans of each patient, performed Geboes pathological scoring and consistency testing, and recorded them separately according to the patient ID number and lesion site. The three pathologists were blinded to the imaging and endoscopic data. The endoscopic images, image marking files, and pathological scoring results were named according to the patient ID number and the lesion sampling site to ensure one-to-one correspondence between the three.

[0081] Specifically, colonoscopy, CT images, and UC pathological grading include:

[0082] Colonoscopic images: Colonoscopic images and videos of UC patients undergoing colonoscopy were selected from the hospital's digestive endoscopy center database. All images and videos were taken under white light and non-magnification mode. Colonoscopic images were excluded due to insufficient intestinal preparation before the examination, blurred pixels, or unrecognizable due to severe lesions; colonoscopic videos were all video data of colonoscopy examinations in which the endoscope was inserted into the ileocecal region without biopsy or treatment.

[0083] CT images: All patients received MDCT (multi-slice spiral CT) through a 64-row multi-slice spiral CT scanner, and the scanning range covered the entire upper abdomen. The specific parameters were: tube voltage 120 kVp, tube current 220-400 mAs, detector collimation 64×0.625 mm, matrix size 512×512, pitch 1.375 mm, rotation time 0.5 s, and slice thickness 5.0 mm. All patients fasted for 8 hours to empty their stomach contents. Before the examination, they drank more than 800 ml of pure water to fill the stomach cavity. During the examination, the patients took the supine position and were asked to hold their breath. All CT images were reconstructed with an axial thickness of 5 mm. Afterwards, DICOM format image files were retrieved and exported from the hospital image archiving and communication system and medical imaging workstation, and stored for further image segmentation and analysis.

[0084] Pathological grading: Each center retrospectively analyzed the pathological sections, and at least three experienced pathologists analyzed the pathological sections or pathological section scans of each UC patient and performed Geboes pathological scoring and consistency test. The Geboes pathological scoring criteria are shown in Table 1.

[0085] Table 1 Geboes pathological scoring criteria

[0086]

[0087]

[0088] Step S103, performing image processing on the colonoscopy images and computer tomography images in the ulcerative colitis medical image database, extracting pathological quantitative features from the colonoscopy images and computer tomography images after image processing, and constructing a prediction data set according to the extracted quantitative features and the pathological grade of the ulcerative colitis patients;

[0089] In this optional embodiment, when performing image processing on colonoscopic images and computed tomography images in a medical image database of ulcerative colitis, and extracting pathological quantitative features from the image-processed colonoscopic images and computed tomography images, and constructing a prediction data set based on the extracted quantitative features and the pathological grading of patients with ulcerative colitis, the colonoscopic images can be segmented using a semantic segmentation method, and feature extraction can be performed on the segmented colonoscopic images using a Transformer network structure to obtain colonoscopic features; the computed tomography images are filtered and transformed using a filter to obtain a filtered transformed image, and radiomics features are extracted from the computed tomography images and the filtered transformed images using open source software to obtain radiomics features; the colonoscopic features and radiomics features are used as pathological quantitative features of patients with ulcerative colitis, and the quantitative features are combined with the pathological grading of patients with ulcerative colitis to obtain a prediction data set.

[0090] Specifically, the upper abdominal enhanced CT image sequences of UC patients were transformed using Wavelet filter, Laplacian of Gaussian filter (a filter combining Gaussian smoothing and Laplacian operator), and gradient filter. The open source radiomics analysis tool PyRadiomics was used to extract the following six types of radiomics features in the original image and the three types of filtered images, in the target area (i.e., the diseased intestinal segment) annotated by the imaging experts in three dimensions. 1) Shape-Based, 2) First-order statistics, 3) Gray level co-occurrence matrix, 4) Gray level runlength matrix, 5) Gray level size zone matrix, 6) Gray level dependence matrix.

[0091] Among them, Wavelet filter, Laplacian of Gaussian (LoG) filter and gradient filter are commonly used filters in image processing, which can be used to enhance image features, edge detection, noise reduction, etc. Wavelet transform can be used for multi-scale decomposition of images. LoG filter can be used for edge detection;

[0092] Gradient filters can be used to detect intensity changes in an image. To use them, simply call the Python function.

[0093] In this optional embodiment, when the colonoscopy image is segmented by using the semantic segmentation method and the feature extraction of the segmented colonoscopy image is performed using the Transformer network structure to obtain the colonoscopy feature, the colonoscopy image can be preprocessed, and the preprocessed colonoscopy image can be segmented into multiple image blocks by using the semantic segmentation method, and each image block is mapped to a linear embedding sequence; the linear embedding sequence is encoded by using the encoder of the Transformer network, and the features of the image blocks are extracted; the encoded image block features are decoded by using the decoder of the Transformer network, and a segmented pixel classification map is generated through upsampling and softmax function; according to the segmented pixel classification map, the features of the lesion area are extracted, and the extracted features are used as colonoscopy features.

[0094] In this optional embodiment, when preprocessing the colonoscopic image, the ulcerative colitis lesion area in the colonoscopic image can be set and labeled, and the invalid background of each colonoscopic image can be removed according to the labeling result to obtain a valid intestinal image; the valid intestinal image is adjusted to a uniform size, and the pixel value of the resized image is normalized to convert the pixel value of the intestinal image to a preset range; using the data enhancement method, the normalized intestinal image is data enhanced to obtain a preprocessed intestinal image.

[0095] It should be noted that at least two experienced endoscopists define the UC lesion area, remove the invalid background of each image, and convert the image into a standard image of the same size. Data virtual amplification is performed by rotation, translation, etc., and the labeled image is converted into a unified pixel format and normalized. This study uses the visual Transformer (ViT) method, which is a commonly used visual task backbone in academia and industry. It converts an entire image into image patches (patches) and converts them into a sequence and sends them to the Transformer encoding and decoding structure for feature modeling.

[0096] Specifically, an experienced endoscopist defines the UC lesion area, adjusts the image size to be consistent, adjusts the colonoscopy image to 224*224*3 according to the model input image size, and normalizes the pixel value to between 0 and 1. Based on the latest ViT research results, the present invention divides the image into patches and maps them into a linear embedding sequence (for a 224*224 image, the size of each patch image block is set to 16, so the original image can be segmented from the two dimensions of H and W, 224 / 16=14, 14×14=196, that is to say, the original image is divided into 196 16*16 patches, so 196 patches need to be embedded), and encodes them with an encoder. MaskTransformer then decodes the output of the encoder and class embedding (image features encoded by the linear embedding sequence), applies the softmax function after upsampling to achieve pixel classification, and outputs the final pixel segmentation map. The decoding stage uses a simple method of jointly processing image blocks and class embeddings. The decoder Mask Transformer can directly perform panoramic segmentation by replacing class embeddings with object embeddings. Upsampling means gradually increasing the receptive field. Softmax normalizes all outputs to between (0-1), and the largest value represents the final classification result.

[0097] In this optional embodiment, when a filter is used to filter and transform a computed tomography image to obtain a filtered transformed image, and radiomics features are extracted from the computed tomography image and the filtered transformed image through open source software to obtain radiomics features, the image sequence of the computed tomography image can be transformed by the filter to generate a filtered transformed image; the diseased tissue area of ​​the computed tomography image and the filtered transformed image is three-dimensionally annotated through open source software, and an annotation mask is generated; based on the annotation mask, the initial radiomics features of the diseased area are extracted from the computed tomography image and the filtered transformed image through an radiomics analysis tool; the extracted radiomics features are analyzed for consistency and correlation using a correlation analysis method, and redundant features are screened and eliminated to obtain the final radiomics features.

[0098] It should be noted that for computed tomography images (CT images), the open source software 3DSlicer was used to perform three-dimensional annotation of the lesion tissue area of ​​UC patients, paying attention to avoiding intestinal contents and periintestinal fat tissue, and being consistent with the endoscopic biopsy site (e.g. Figure 5 Then, the PyRadiomics package based on Python environment was used to extract more than 2000 quantitative image features. The ICC value (intraclass correlation coefficient) was used to evaluate the inter-observer consistency and intra-observer consistency of each image feature. The calculation formula is as follows:

[0099]

[0100] Where ICC is the intraclass correlation coefficient, which is used to evaluate the consistency between different observers (or multiple measurements by the same observer); MSR is the mean square of the row variable; MSC is the mean square of the column variable; MSE is the mean square of the error; k and n are the number of assessors and samples, respectively.

[0101] Step S105, based on the prediction data set, using a deep convolutional neural network and combining it with a Transformer network structure to construct and train a virtual biopsy model for ulcerative colitis;

[0102] In this optional embodiment, the method of constructing and training a virtual biopsy model for ulcerative colitis based on a prediction data set using a deep convolutional neural network in combination with a Transformer network structure includes the following steps:

[0103] The prediction data set is divided into a training set and a validation set, and the pathological quantitative features and pathological grades in the prediction data set are used as training labels;

[0104] Construct a network structure with dual inputs and combine it with a deep convolutional neural network and a Transformer network structure to establish a virtual biopsy model for ulcerative colitis;

[0105] The elastic network algorithm was used to perform 10-fold cross validation in the training set. Based on the validation results, the pathological quantitative features most relevant to the pathological grading were screened out. The ulcerative colitis virtual biopsy model was trained based on the screened pathological quantitative features and pathological grading in combination with the loss function.

[0106] The performance of the ulcerative colitis virtual biopsy model was evaluated using the validation set, and the parameters of the ulcerative colitis virtual biopsy model were optimized based on the evaluation results to obtain the final ulcerative colitis virtual biopsy model.

[0107] In this optional embodiment, the loss function is expressed as:

[0108] J(θ)=MSE(θ)+θλ(0.3Σ∣θ i ∣+0.7Σθ i 2 )

[0109] Where J(θ) represents the loss function, MSE represents the mean standard error, θ represents the model parameter set, λ represents the regularization coefficient, and Σ|θ i ∣ represents the L1 regularization term, Σθ i 2 represents the L2 regularization term.

[0110] It should be noted that the stability of quantitative imaging features was detected by inter-observer and intra-observer correlation tests. The inter-class correlation coefficients (ICC) were used to evaluate the inter-class consistency of each quantitative imaging feature. Quantitative imaging features with insufficient intra-class and inter-class consistency were removed. The Pearson correlation test was used to analyze the correlation between the features to remove redundant features.

[0111] In quantitative image feature analysis, the interclass correlation coefficient (ICC) is a statistic that measures the consistency between different observers. The following are the steps to use ICC to evaluate the consistency between groups of quantitative image features:

[0112] First, the data needs to be organized into a format suitable for ICC analysis. The results for each observer will be placed in a separate column, and each row represents a separate measured object.

[0113] Then, ICC was calculated using statistical software (such as SPSS, R, Stata, etc.). The value of ICC ranges from 0 to 1, where: ICC close to 0 indicates almost no consistency; ICC close to 1 indicates almost perfect consistency; it is generally believed that ICC ≥ 0.7 indicates good inter-group consistency.

[0114] In addition, different tissue pathology grades were used as labels for machine learning model training. In the training set cases, Lasso regression (Least Absolute Selection and Shrinkage Operator, LASSO) was used to screen the quantitative image feature set with the strongest correlation with the corresponding label through multi-fold cross-validation as the input parameter of the model. Attempts were made to construct a non-invasive assessment model for tissue pathology grading through various machine learning modeling methods such as logistic regression (LogisticRegression), support vector machine (SVM), random forest, artificial neural network, etc. The prediction performance of each model was evaluated through the area under the ROC curve (Aera Under ROC Curve, AUC), sensitivity, specificity, and accuracy in the training set and internal validation set, and the model prediction performance was improved through hyperparameter debugging to achieve the best prediction effect.

[0115] Lasso Regression is a penalized linear regression method that implements feature selection by adding L1 penalty (absolute value penalty) to the loss function. With the help of k-fold cross-validation, the quantitative image feature set with the strongest correlation with the target variable (label) can be screened. The following are the steps to use lasso regression to screen the feature set through multi-fold cross-validation:

[0116] First, make sure your dataset has been cleaned and preprocessed, including scaling features (standardization or normalization).

[0117] Select an appropriate k value (number of folds) and set up cross validation.

[0118] Use a statistical or machine learning library such as scikit-learn to perform lasso regression with cross validation.

[0119] Use the test set to evaluate the performance of the model and ensure that the feature selection is effective.

[0120] Specifically, if Figure 7 As shown in the figure, the present invention mainly constructs a dual-input network structure, where the two inputs correspond to the CT image data and colonoscopy images of the dataset respectively. After the input image size passes through the dataset layer, the unified input size is 224 × 224. The deep convolutional neural networks (DCNN) and Transformer are combined to comprehensively utilize the characteristics of the two algorithms to develop a UC pathology virtual biopsy system.

[0121] The overall structure of the model is shown in the figure below. It is divided into four modules: a dual-branch encoder for feature extraction and decomposition, a decoder for image reconstruction in training phase I or image fusion in training phase II, and a base / detail fusion layer for fusing features of different frequencies. Figure 6As shown in the figure, an improved U-Net structure is adopted. The convolution branch is used to extract the local features of the image in the encoding stage, and the TransformerEncoder module branch is used to extract the global features of the image. For the input noisy image, two convolution layers and one ReLU activation function are first used to extract the shallow features of the image. The shallow features are sent to the two branches to extract features respectively. In the TransformerEncoder branch, the number of multi-head self-attention heads is set to 2, 4, 6, and 1 in each layer. The number of channels of the two branches is 24, 48, 96, and 192. After the convolution branch and the TransformerEncoder branch extract features, the features extracted by the two branches are fused using the channel attention module and sent to the decoding end and the lower-layer TransformerEncoder module respectively. This process is repeated after downsampling the feature image.

[0122] also, Figure 6 In the figure, Feature Extraction means feature extraction, Feature Selection means feature selection, MLP means multi-layer perceptron, Norm means layer normalization, Multi-Head Attention means multi-head attention, Endoscopic lmage Feature means endoscopic image feature, Concatenate means feature concatenation, Outcome means output, Transformer Encoder: Transformer means encoder, Patch Embedding means image block encoding, Position Embedding means position encoding, Extra Learnable [class] Embedding means additional learnable [class] embedding, Linear Projection of Flattened Patches means linear transformation of flattened image patches, Radiomics Feature means radiomics feature, and Embedded Patches means image block encoding.

[0123] Validation model: The internal validation used the 10-fold cross-validation method to validate the model. The original data was divided into 10 subsets, each subset was used as a validation set, and the remaining 9 subsets were used as training sets. This was repeated 10 times in total, and the accuracy of the model was the average result of the 10 times. The external validation used data from the Eastern Theater General Hospital.

[0124] It should be noted that Transformer is a deep neural network mainly based on the self-attention mechanism, and was first applied in the field of natural language processing. Recently, large-scale Transformer models using self-supervised pre-training have demonstrated improved efficiency and scalability, such as BERT and GPT in natural language processing (NLP). The present invention innovatively integrates UC-related CT images and colonoscopy images, and the Transformer architecture is one of the most suitable methods. Multimodal image fusion modeling method based on deep convolutional neural networks (DCNN) and Transformer. The present invention extracts colonoscopy and CT image features respectively, and combines the Transformer-DCNN method with the most advanced ViT image classification model to design a deep learning model DCNN-ViT that fuses multimodal images, giving full play to the different advantages of multi-head attention mechanism and convolution.

[0125] In addition, the model in this paper is divided into two stages. In the first stage, CDDFuse first uses the Restormer block to extract cross-modal shallow features, and then introduces a dual-branch Transformer-CNN feature extractor, in which the Lite Transformer (LT) block uses long-range attention to process low-frequency global features, and the Invertible Neural Networks (INN) block is used to extract high-frequency local features. In the second stage, the aforementioned LT and INN modules will output the classification results, namely the UC pathological grade. The CT and endoscopic images are transmitted to the VIT network respectively, and the features are extracted separately, and the weights are shared. The CT and endoscopic information are fused through feature splicing to provide more comprehensive and accurate information feature information. The network is evaluated by the training group samples and tested in the validation group. The area under the characteristic curve (AUC), accuracy, sensitivity, specificity and other indicators are calculated to evaluate the performance of the model.

[0126] VIT network refers to Vision Transformer. Feature extraction process: first, the input image is processed into multiple patches, and then each patch is input into Vision Transformer in turn for feature extraction.

[0127] In addition, graphic annotation tools mainly include 3D-Slicer, ITK-SNAP and Photoshop. The integrated experimental platform for format conversion and training models (lesion identification, lesion location, lesion segmentation) is mainly based on Python and Metlab. Conventional statistical analysis (chi-square test, t-test, correlation analysis and survival analysis, etc.) is performed using SPSS software or R language.

[0128] Step S107, acquiring new clinical data of patients with ulcerative colitis in real time, inputting the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and outputting the pathological grading results of ulcerative colitis through the extraction and prediction of pathological features.

[0129] Figure 2 An embodiment of the multimodal image-based ulcerative colitis virtual biopsy system of the present invention is shown.

[0130] In this optional embodiment, the multimodal image-based ulcerative colitis virtual biopsy system includes:

[0131] The database construction module 201 is used to collect clinical data of patients with ulcerative colitis and perform image data quality processing on the clinical data; determine the pathological grade of the patients with ulcerative colitis according to the processing results and establish a medical image database of ulcerative colitis, wherein the clinical data includes basic information of patients with ulcerative colitis, colonoscopy images and computer tomography images;

[0132] The data set construction module 203 is used to perform image processing on the colonoscopy images and computer tomography images in the ulcerative colitis medical image database, extract pathological quantitative features from the colonoscopy images and computer tomography images after image processing, and construct a prediction data set according to the extracted quantitative features and the pathological grade of the ulcerative colitis patients;

[0133] A model building module 205 is used to build and train a virtual biopsy model for ulcerative colitis based on a prediction data set using a deep convolutional neural network in combination with a Transformer network structure;

[0134] The pathology prediction module 207 is used to obtain new clinical data of ulcerative colitis patients in real time, input the obtained new clinical data into the trained ulcerative colitis virtual biopsy model, and output the pathology grading results of ulcerative colitis through the extraction and prediction of pathological features.

[0135] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0136] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0138] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0140] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A virtual biopsy method for ulcerative colitis based on multimodal images, characterized in that: The method comprises the following steps: Collect clinical data of patients with ulcerative colitis and perform image data quality processing on the clinical data; determine the pathological grade of patients with ulcerative colitis based on the processing results and establish a medical image database of ulcerative colitis, wherein the clinical data includes basic information of patients with ulcerative colitis, colonoscopy images and computer tomography images; Perform image processing on colonoscopy images and computed tomography images in the ulcerative colitis medical image database, extract pathological quantitative features from the processed colonoscopy images and computed tomography images, and construct a prediction data set based on the extracted quantitative features and the pathological grading of ulcerative colitis patients; Based on the prediction dataset, a deep convolutional neural network was used in combination with the Transformer network structure to build and train a virtual biopsy model for ulcerative colitis. Acquire new clinical data of patients with ulcerative colitis in real time, input the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and output the pathological grading results of ulcerative colitis through the extraction and prediction of pathological features.

2. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 1, characterized in that: The clinical data of patients with ulcerative colitis are collected and image data quality processing is performed on the clinical data; Determining the pathological grade of ulcerative colitis patients according to the processing results and establishing a medical image database of ulcerative colitis includes the following steps: Collect basic information, colonoscopy images, computed tomography images, and pathological section scans of patients with ulcerative colitis; The clinical data of patients with ulcerative colitis were screened and collated according to pre-set screening criteria; Perform pathological section analysis on the screened and sorted pathological section scan images, and determine the pathological grade of ulcerative colitis patients using pre-set pathological scoring standards; Based on the basic information of patients with ulcerative colitis, colonoscopy images, computed tomography images and pathological grading are stored to obtain a medical image database of ulcerative colitis.

3. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 1, characterized in that: The method of performing image processing on the colonoscopy images and computer tomography images in the ulcerative colitis medical image database, extracting pathological quantitative features from the colonoscopy images and computer tomography images after image processing, and constructing a prediction data set according to the extracted quantitative features and the pathological grade of ulcerative colitis patients comprises the following steps: The semantic segmentation method is used to segment the colonoscopy images, and the Transformer network structure is used to extract the features of the segmented colonoscopy images to obtain colonoscopy features. The filter is used to perform filter transformation on the computed tomography image to obtain a filter transformation image, and the radiomics features are extracted from the computed tomography image and the filter transformation image by using open source software to obtain radiomics features; Colonoscopy features and radiomics features were used as pathological quantitative features of patients with ulcerative colitis, and the quantitative features were combined with the pathological grades of patients with ulcerative colitis to obtain a prediction data set.

4. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 3, characterized in that: The method of segmenting the colonoscopy image using the semantic segmentation method and extracting features from the segmented colonoscopy image using the Transformer network structure to obtain colonoscopy features includes the following steps: Preprocess the colonoscopy images, segment the preprocessed colonoscopy images into multiple image blocks using semantic segmentation method, and map each image block into a linear embedding sequence; The encoder of the Transformer network is used to encode the linear embedding sequence and extract the features of the image block; the decoder of the Transformer network is used to decode the encoded image block features and generate the segmented pixel classification map through upsampling and softmax function; According to the segmented pixel classification map, the features of the lesion area are extracted, and the extracted features are used as colonoscopy features.

5. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 4, characterized in that: The preprocessing of the colonoscopy image comprises the following steps: The ulcerative colitis lesion area in the colonoscopic image is set and annotated, and the invalid background of each colonoscopic image is removed according to the annotation results to obtain a valid intestinal image; The effective intestinal image is adjusted to a uniform size, and the pixel value of the resized image is normalized to convert the pixel value of the intestinal image into a preset range; The normalized intestinal image is subjected to data enhancement processing by using the data enhancement method to obtain the preprocessed intestinal image.

6. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 3, characterized in that: The method of using a filter to filter transform the computed tomography image to obtain a filter transform image, and extracting radiomics features from the computed tomography image and the filter transform image by open source software to obtain radiomics features comprises the following steps: Transforming an image sequence of a computed tomography image using a filter to generate a filtered transformed image; The diseased tissue areas of the computed tomography images and the filtered transformed images were annotated in three dimensions using open source software and annotation masks were generated; Based on the annotated masks, the initial radiomic features of the lesion area were extracted from the computed tomography images and the filtered transformed images using radiomics analysis tools; The extracted radiomics features were analyzed for consistency and correlation using correlation analysis, and redundant features were screened and eliminated to obtain the final radiomics features.

7. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 6, characterized in that: The imaging omics features include: shape features, first-order statistical features, grayscale co-occurrence matrix features, grayscale run matrix features, grayscale region size matrix features and grayscale dependence matrix features.

8. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 1, characterized in that: The method of constructing and training a virtual biopsy model for ulcerative colitis based on a prediction data set using a deep convolutional neural network and in combination with a Transformer network structure includes the following steps: The prediction data set is divided into a training set and a validation set, and the pathological quantitative features and pathological grades in the prediction data set are used as training labels; Construct a network structure with dual inputs and combine it with a deep convolutional neural network and a Transformer network structure to establish a virtual biopsy model for ulcerative colitis; The elastic network algorithm was used to perform 10-fold cross validation in the training set. Based on the validation results, the pathological quantitative features most relevant to the pathological grading were screened out. The ulcerative colitis virtual biopsy model was trained based on the screened pathological quantitative features and pathological grading in combination with the loss function. The performance of the ulcerative colitis virtual biopsy model was evaluated using the validation set, and the parameters of the ulcerative colitis virtual biopsy model were optimized based on the evaluation results to obtain the final ulcerative colitis virtual biopsy model.

9. The multimodal image-based virtual biopsy method for ulcerative colitis according to claim 8, characterized in that: The expression of the loss function is: J(θ)=MSE(θ)+θλ(0.3Σ∣θ i ∣+0.7θ i 2 ) Where J(θ) represents the loss function, MSE represents the mean standard error, θ represents the model parameter set, λ represents the regularization coefficient, and Σ|θ i ∣ represents the L1 regularization term, Σθ i 2 represents the L2 regularization term.

10. A virtual biopsy system for ulcerative colitis based on multimodal images, characterized in that: The system includes: A database construction module is used to collect clinical data of patients with ulcerative colitis and perform image data quality processing on the clinical data; determine the pathological grade of patients with ulcerative colitis according to the processing results and establish a medical image database of ulcerative colitis, wherein the clinical data includes basic information of patients with ulcerative colitis, colonoscopy images and computer tomography images; A data set construction module, for performing image processing on colonoscopy images and computer tomography images in the ulcerative colitis medical image database, extracting pathological quantitative features from the colonoscopy images and computer tomography images after image processing, and constructing a prediction data set based on the extracted quantitative features and the pathological grading of ulcerative colitis patients; A model building module is used to build and train a virtual biopsy model for ulcerative colitis based on a prediction dataset using a deep convolutional neural network combined with a Transformer network structure; The pathology prediction module is used to obtain new clinical data of patients with ulcerative colitis in real time, input the acquired new clinical data into the trained ulcerative colitis virtual biopsy model, and output the pathological grading results of ulcerative colitis through the extraction and prediction of pathological features.

Images