Disease prognosis prediction method and system based on image data, and storage medium

Abstract

The embodiment of the application provides a disease prognosis prediction method and system based on image data and a storage medium. The embodiment of the application is based on the data characteristics of WSI and MRI, and converts the image blocks in the WSI and the complementary effects of different sequence images in the MRI into WSI graph structure data and 3D MRI image matrices respectively according to the coordinates of the image blocks in the WSI. Taking the WSI graph structure data and the 3D MRI image matrices as the double-flow input of the neural network can realize the direct learning of the patient-level features, and can directly update and optimize the patient-level WSI and MRI features through the end-to-end training mode, thereby avoiding the information loss caused in the feature aggregation process, and capturing the context relationship inside the images, so that the accuracy of the prognosis prediction is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and image processing technology, specifically to a method, system, and storage medium for predicting disease prognosis based on image data. Background Technology

[0002] Prognostic prediction for various diseases (such as cancer) aims to use medical data to predict a patient's risk of disease progression, recurrence, or death over a future period. Predicting a patient's future prognostic risk can assist clinicians in developing more precise treatment plans. In the field of AI-assisted medical diagnosis, medical imaging, due to its ability to visually display specific changes in tissues and organs, is one of the earliest areas of focus and has been extensively researched. Magnetic Resonance Imaging (MRI) and Whole Slide Image (WSI) are two important tools in clinical diagnosis. MRI provides information on macroscopic disease progression and specific changes in tissue structure, while WSI, by observing tissue sections, assesses changes at the cellular level from a more microscopic perspective. By combining the macroscopic and microscopic perspectives of MRI and WSI, a more comprehensive assessment of lesions can be achieved, providing a more comprehensive and detailed basis for disease diagnosis.

[0003] With the continuous development of AI-assisted diagnosis and treatment, some methods integrating MRI and WSI feature analysis have emerged. These methods typically use machine learning techniques or convolutional neural networks to extract statistical and textural features of lesion regions from MRI and WSI respectively. Then, they simply concatenate and filter these features to construct a predictive model. However, the independent calculation methods for MRI and WSI features result in a lack of correlation between the features, leading to lower accuracy. Summary of the Invention

[0004] At least one embodiment of this application provides a disease prognosis prediction method, system, and storage medium based on image data, which addresses the problem of low accuracy in existing disease prognosis prediction methods.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a disease prognosis prediction method based on image data, including:

[0007] Acquire a first image and a second image of the lesion to be predicted, wherein the first image is at least one magnetic resonance imaging (MRI) image and the second image is at least one full-view digital pathology image (WSI);

[0008] The first image is preprocessed to obtain the image matrix of the lesion to be predicted; the second image is preprocessed to obtain the foreground image of the lesion to be predicted; the foreground image is divided into multiple image blocks by a sliding window of fixed size; and the multiple image blocks of the foreground image are represented as graph structure data according to the connectivity between the image blocks.

[0009] The image matrix and the graph structure data are input into a pre-trained multimodal contrastive learning framework to obtain a first MRI feature corresponding to the image matrix and a first WSI feature corresponding to the graph structure data. The multimodal contrastive learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network, and a second linear projection layer. The 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, while the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data.

[0010] Based on the first MRI feature and the first WSI feature, the prognosis of the lesion to be predicted is predicted.

[0011] Optionally, the first image is preprocessed to obtain an image matrix of the lesion to be predicted, including:

[0012] The first image is subjected to N4 bias field correction, resampling and intensity normalization to obtain the third image; the third images of different sequences are stitched together according to channels to obtain the image matrix of the lesion to be predicted.

[0013] Optionally, the second image is preprocessed to obtain the foreground image of the lesion to be predicted, including:

[0014] The second image is color-corrected to obtain a fourth image; the foreground image of the lesion to be predicted is separated from the fourth image using the Otsu method.

[0015] Optionally, based on the connectivity relationships between image patches, the multiple image patches of the foreground image are represented as graph structure data, including:

[0016] Multiple image patches belonging to the same foreground image are input into a pre-trained CTransPath network to obtain the image features of each image patch output by the CTransPath network.

[0017] Graph structure data is generated based on the image features of each image patch and the connectivity of each image patch in the foreground image.

[0018] Optionally, based on the first MRI feature and the first WSI feature, a prognostic prediction is made for the lesion site to be predicted, including:

[0019] The first MRI feature and the first WSI feature are respectively subjected to noise reduction processing to obtain the second MRI feature and the second WSI feature;

[0020] The outer product of the matrix of the second MRI feature and the second WSI feature is calculated to obtain the first fusion feature;

[0021] The first fused feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result generated by the multilayer perceptron.

[0022] Optionally, the method further includes:

[0023] Acquire sample data, which includes first and second sample images of multiple lesion samples and prognostic results of the lesion samples, wherein the first sample image is an MRI image and the second sample image is a WSI image;

[0024] The first sample image is preprocessed to obtain the sample image matrix of the lesion sample; the second sample image is preprocessed to obtain the foreground sample image of the lesion sample; the foreground sample image is divided into multiple image blocks by a sliding window of fixed size; and the multiple image blocks of the foreground sample image are represented as sample image structure data according to the connectivity between the image blocks.

[0025] Through multiple iterations, the similarity between the sample image matrix and sample image structure data of the lesion samples is learned. In each iteration, a sample image matrix and a sample image structure data are randomly selected and input into the multimodal contrast learning framework to obtain the third MRI feature corresponding to the sample image matrix and the third WSI feature corresponding to the sample image structure data. The similarity between the third MRI feature and the third WSI feature is calculated to obtain the similarity between the lesion sample corresponding to the third MRI feature and the lesion sample corresponding to the third WSI feature, and the similarity matrix between each lesion sample is updated. Based on the similarity matrix and the label matrix corresponding to the similarity matrix, the loss function of the multimodal contrast learning framework is constructed and the loss value is calculated. Based on the loss value, the multimodal contrast learning framework is optimized until the preset training termination condition is reached to obtain the trained multimodal contrast learning framework.

[0026] Optionally, the method further includes:

[0027] The sample image matrix and sample image structure data of the same lesion sample are input into the trained multimodal contrast learning framework to obtain the fourth MRI feature corresponding to the sample image matrix and the fourth WSI feature corresponding to the sample image structure data.

[0028] The fourth MRI feature and the fourth WSI feature are respectively subjected to noise reduction processing to obtain the fifth MRI feature and the fifth WSI feature;

[0029] The outer product of the matrices of the fifth MRI feature and the fifth WSI feature is calculated to obtain the second fusion feature;

[0030] The second fusion feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result of the lesion sample generated by the multilayer perceptron;

[0031] Based on the prognostic prediction results and prognostic results of the lesion samples, the multilayer perceptron is trained to obtain a trained multilayer perceptron.

[0032] Secondly, embodiments of this application provide a disease prognosis prediction system based on image data, including:

[0033] The acquisition module is used to acquire a first image and a second image of the lesion to be predicted, wherein the first image is at least one magnetic resonance imaging (MRI) image and the second image is at least one full-view digital pathology image (WSI).

[0034] The preprocessing module is used to preprocess the first image to obtain the image matrix of the lesion to be predicted; preprocess the second image to obtain the foreground image of the lesion to be predicted; divide the foreground image into multiple image blocks through a sliding window of fixed size; and represent the multiple image blocks of the foreground image as graph structure data according to the connectivity between the image blocks.

[0035] A feature conversion module is used to input the image matrix and the graph structure data into a pre-trained multimodal contrastive learning framework to obtain a first MRI feature corresponding to the image matrix and a first WSI feature corresponding to the graph structure data. The multimodal contrastive learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network, and a second linear projection layer. The 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, and the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data.

[0036] The prediction module is used to predict the prognosis of the lesion to be predicted based on the first MRI feature and the first WSI feature.

[0037] Optionally, the preprocessing module is further configured to:

[0038] Multiple image patches belonging to the same foreground image are input into a pre-trained CTransPath network to obtain the image features of each image patch output by the CTransPath network.

[0039] Graph structure data is generated based on the image features of each image patch and the connectivity of each image patch in the foreground image.

[0040] Thirdly, embodiments of this application provide a disease prognosis prediction system based on image data, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the method described in either the first or second aspect.

[0042] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in either the first or second aspect.

[0043] Compared with existing technologies, the disease prognosis prediction method, system, and storage medium based on image data provided in this application, taking into account the data characteristics of WSI and MRI, converts them into WSI graph structure data and 3D MRI image matrices respectively, based on the coordinates of image patches in WSI and the complementary effects of different sequence images in MRI. Using the WSI graph structure data and 3D MRI image matrices as dual-stream inputs to a neural network enables direct learning of patient-level features. Furthermore, end-to-end training allows for direct updating and optimization of patient-level WSI and MRI features, avoiding information loss that may occur during feature aggregation and capturing the internal contextual relationships of the images, thereby improving the accuracy of prognosis prediction. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1This is a flowchart of a disease prognosis prediction method based on image data according to an embodiment of this application;

[0046] Figure 2 This is an example diagram of a disease prognosis prediction method based on image data according to an embodiment of this application;

[0047] Figure 3 This is an example diagram of the multimodal contrastive learning framework according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a disease prognosis prediction system based on image data according to an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of a disease prognosis prediction system based on image data, according to another embodiment of this application. Detailed Implementation

[0050] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, the term "and / or" in this application describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0051] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc., in the instruction sent. An indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0052] As described in the background section, existing image-based disease prognosis prediction methods suffer from low accuracy. To address at least one of these problems, this application provides an image-based disease prognosis prediction method that can reduce or avoid these issues, improve communication efficiency, and enhance user experience.

[0053] Please refer to Figure 1This application provides a disease prognosis prediction method based on image data, comprising the following steps:

[0054] Step 11: Obtain a first image and a second image of the lesion to be predicted. The first image is at least one MRI image, and the second image is at least one WSI image.

[0055] Step 12: Perform image preprocessing on the first image to obtain the image matrix of the lesion to be predicted; perform image preprocessing on the second image to obtain the foreground image of the lesion to be predicted; divide the foreground image into multiple image blocks through a sliding window of fixed size; and represent the multiple image blocks of the foreground image as graph structure data according to the connectivity between the image blocks.

[0056] Here, image preprocessing of the first image typically includes: performing N4 bias field correction, resampling, and intensity normalization on the first image to obtain a third image; and stitching together the third images of different sequences according to channels to obtain the image matrix of the lesion to be predicted.

[0057] The second image is preprocessed, typically including: color correction to obtain a fourth image; and using Otsu's method to separate the foreground image of the lesion to be predicted from the fourth image. Additionally, in this embodiment, multiple image blocks belonging to the same foreground image are input into a pre-trained CTransPath network to obtain the image features of each image block output by the CTransPath network. Then, based on the image features of each image block and the connectivity of each image block in the foreground image, graph structure data corresponding to the foreground image is generated.

[0058] Step 13: Input the image matrix and the graph structure data into a pre-trained multimodal contrastive learning framework to obtain the first MRI feature corresponding to the image matrix and the first WSI feature corresponding to the graph structure data; wherein, the multimodal contrastive learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network and a second linear projection layer, the 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, and the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data.

[0059] Here, this embodiment of the application pre-trains a multimodal contrastive learning framework, and the specific training method will be described below. The multimodal contrastive learning framework includes a 3D medical image pre-training model (3D MedicalNet), a first linear projection layer, a graph neural network, and a second linear projection layer. The 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, and the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data. When the image matrix is ​​input into the multimodal contrastive learning framework, the MRI features corresponding to the image matrix can be generated through the 3D medical image pre-training model and the first linear projection layer; when the graph structure data is input into the multimodal contrastive learning framework, the WSI features corresponding to the graph structure data can be generated through the graph neural network and the second linear projection layer. Through step 13, multimodal data features, including MRI features and WSI features, can be obtained.

[0060] Step 14: Based on the first MRI feature and the first WSI feature, predict the prognosis of the lesion to be predicted.

[0061] Here, in step 14, the embodiments of this application perform noise reduction processing on the first MRI feature and the first WSI feature respectively to obtain the second MRI feature and the second WSI feature; then, the outer product of the matrix of the second MRI feature and the second WSI feature is calculated to obtain the first fused feature. Then, the first fused feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result generated by the multilayer perceptron.

[0062] Through the above steps, the embodiments of this application can better integrate MRI features and WSI features, thereby improving the accuracy of disease prognosis prediction.

[0063] The training of the multimodal contrastive learning framework is explained below.

[0064] Prior to step 11, the embodiments of this application can train a multimodal contrastive learning framework through the following steps:

[0065] Step 21: Obtain sample data, which includes first and second sample images of multiple lesion samples and prognostic results of the lesion samples, wherein the first sample image is an MRI image and the second sample image is a WSI image.

[0066] Step 22: Perform image preprocessing on the first sample image to obtain the sample image matrix of the lesion sample; perform image preprocessing on the second sample image to obtain the foreground sample image of the lesion sample; divide the foreground sample image into multiple image blocks through a sliding window of fixed size; and represent the multiple image blocks of the foreground sample image as sample image structure data according to the connectivity between the image blocks.

[0067] Step 23: Through multiple iterations, the similarity between the sample image matrix and sample image structure data of the lesion samples is learned. In each iteration, a sample image matrix and a sample image structure data are randomly selected and input into the multimodal contrast learning framework to obtain the third MRI feature corresponding to the sample image matrix and the third WSI feature corresponding to the sample image structure data. The similarity between the third MRI feature and the third WSI feature is calculated to obtain the similarity between the lesion sample corresponding to the third MRI feature and the lesion sample corresponding to the third WSI feature, and the similarity matrix between each lesion sample is updated. Based on the similarity matrix and the label matrix corresponding to the similarity matrix, the loss value of the loss function of the multimodal contrast learning framework is calculated. Based on the loss value, the multimodal contrast learning framework is optimized until the preset training termination condition is reached to obtain the trained multimodal contrast learning framework.

[0068] After step 23, the embodiments of this application can train a multilayer perceptron (MLP) for disease prognosis prediction through the following steps:

[0069] Step 31: Input the sample image matrix and sample image structure data of the same lesion sample into the trained multimodal contrast learning framework to obtain the fourth MRI feature corresponding to the sample image matrix and the fourth WSI feature corresponding to the sample image structure data.

[0070] Step 32: Denoise reduction processing is performed on the fourth MRI feature and the fourth WSI feature respectively to obtain the fifth MRI feature and the fifth WSI feature.

[0071] Step 33: Calculate the outer product of the matrix of the fifth MRI feature and the fifth WSI feature to obtain the second fusion feature.

[0072] Step 34: Convert the second fusion feature into a one-dimensional matrix and input it into a pre-trained multilayer perceptron to obtain the prognostic prediction result of the lesion sample generated by the multilayer perceptron.

[0073] Step 35: Based on the prognostic prediction results and prognostic results of the lesion samples, train the multilayer perceptron to obtain a trained multilayer perceptron.

[0074] The following is combined with Figure 2 The above-described method of the embodiments of this application will be further explained through a more specific example.

[0075] like Figure 2 As shown, this example mainly consists of seven parts: input image, image preprocessing of WSI and MRI, data structure transformation of WSI and MRI, pathology-image multimodal contrast learning, feature denoising and pooling using a gated attention mechanism, feature fusion through matrix outer product for multimodal information interaction, and inputting the fused features into a multilayer perceptron for prognostic prediction. These will be explained in detail below.

[0076] (1) Input Images. Obtain the magnetic resonance image to be predicted and the digital pathology image to obtain the MRI-WSI data pair. In this example, the magnetic resonance image may contain T1-weighted and T2-weighted sequences, and the digital pathology image is a pathology image based on hematoxylin-eosin staining (HE).

[0077] (2) Image preprocessing. Different preprocessing methods are adopted according to the characteristics of different modalities of images.

[0078] For magnetic resonance imaging (MRI) images: The brightness, spatial resolution, and intensity values ​​of MRI images may vary significantly between individuals and with different scanning devices. Standardization can be performed using the following procedure, detailed below:

[0079] Step a: N4 Bias Field Correction. Brightness differences in MRI images are often influenced by various factors, including patient position, scanner performance, and other unknown issues. These differences primarily stem from the inhomogeneity of the magnetic field in the MRI scanning equipment, known as the bias field. The bias field manifests as a low-frequency smooth signal, adversely affecting MRI image quality and potentially interfering with subsequent image analysis. In this example, the N4 bias field correction method is used as the first step in preprocessing the MRI. This method first estimates the bias field at different scales using a multi-scale approach. Subsequently, the N4 algorithm calculates the correction field, i.e., the corrected image, using the reciprocal of the estimated bias field and a parameterized function. This corrected image contains the bias field information that needs to be added to the original image. Next, the influence of the bias field is eliminated by multiplying or dividing the original MRI image with the correction field, making the corrected image more accurately reflect the true intensity of the tissue. To ensure the accuracy of the bias field calibration, the N4 algorithm performs multiple iterations, estimating a new bias field and applying the correction in each iteration.

[0080] Step b: Resampling. To unify the spatial resolution between different MRI images, this example uses the cubic spline interpolation method to resample the MRI images, with the target sampling spatial resolution set to [1,1,1]. Resampling helps unify images from different data sources, ensuring consistent pixel spacing in each dimension, improving image consistency and comparability, while reducing computational complexity.

[0081] Step c: Intensity normalization. Pixel values ​​in the MRI images are normalized using z-score standardization, ensuring that the pixel values ​​follow a distribution with a mean of 0 and a standard deviation of 1.

[0082] For digital pathology images (WSI images): WSI images can exhibit significant color variations due to differences in the source of the staining agent and lighting conditions. To enable computer analysis of large-sized WSI images, the following preprocessing steps are required:

[0083] Step A: Correct the color space of WSI using the Vahadane color correction method. Specifically, by decomposing the staining matrix and staining density map of the target image, the staining matrix of the image to be normalized is replaced with the staining matrix of the target image, and then combined with the corresponding staining density map, the color of the pathological image can be normalized.

[0084] Step B: Tissue Region Extraction. Besides containing tumor tissue, digital pathology images still contain a large number of blank areas. In this example, the WSI is first downsampled to obtain a thumbnail at 5x magnification, and the image is converted from the RGB color space to the HSV color space. The Otsu method is used to separate the foreground and background regions of the WSI, and thresholds (<15) for image saturation and entropy are set simultaneously to remove the image background and empty areas.

[0085] Step C: Small Image Patch Cropping. For digital pathology image data with hundreds of millions of pixels, their enormous size makes direct neural network processing and analysis difficult, unlike natural images. Therefore, the WSI needs to be segmented into multiple fixed-size image patches using a sliding window approach without overlap. To ensure consistent field of view across patches, this example sets the cropping of all patches to 20x, with a patch size of 256×256. For WSIs not included at 20x, the field of view is fixed, and they are cropped to larger or smaller sizes based on their inherent magnification, then the size is sampled back to 256×256. For example, at 40x, the cropped size is 512×512, and at 10x, it is 128×128.

[0086] Step D: Feature extraction of small image patches. Input small image patches belonging to the same WSI into the pre-trained CTransPath network to extract the basic features of the image patches.

[0087] (3) Data Structure Conversion. To obtain patient-level MRI and WSI data pairs, the MRI and WSI data need to be converted into a data format that can be used as input to a neural network. For MRI, images from different sequences are stitched together. Since a single-sequence MRI has only 1 channel, the matrix size obtained after superimposing the lesion regions from T1 and T2 sequences is (2, W, H, Z). Here, W, H, and Z represent the height, width, and depth of the image, respectively.

[0088] For WSI data, the edges of histopathological images composed of tens of thousands of image patches are irregular, and the image sizes vary significantly between different patients, making them unsuitable for processing using conventional methods for analyzing regular images. Graph structures, however, are not limited by the shape and size of the data and can generate graphs of different sizes for different datasets. This example uses the spatial location of image patches in the original WSI as a basis, and uses the 8-connected regions of the image patches as the criterion for adjacency, to represent tens of thousands of image patches as a graph structure data that can be directly input into a neural network.

[0089] G=(V,E)(1)

[0090] Here, node V represents an entity, in this example representing the 768-dimensional feature extracted from CTransPath for the corresponding image patch. Edge E represents the relationship between nodes, which can be represented by an adjacency matrix A. For graph G, the elements A of A... ij This indicates whether node i and node j are adjacent, i.e., whether they are connected by an edge. When they are adjacent, A... ij =1, otherwise A ij =0.

[0091] (4) Pathology-Imaging Multimodal Contrast Learning Method for Extracting Multimodal Features. This example proposes, for instance... Figure 3 The image-pathology-based multimodal contrastive learning framework shown learns the similarity between pathology and image data pairs of the same patient in a self-supervised manner, and extracts a set of MRI and WSI features corresponding to the predicted magnetic resonance image and digital pathology image, respectively.

[0092] The multimodal contrastive learning framework is pre-trained. In each iteration of the training process, a set of data is randomly extracted from MRI and WSI data of multiple lesion samples generated based on sample data. These data are then encoded using 3D MedicalNet and a graph neural network, respectively, and feature-mapped using linear projection. Subsequently, feature similarity is calculated to determine whether the two datasets originate from the same patient (i.e., from the same lesion sample). This example uses a cosine similarity function for similarity evaluation.

[0093]

[0094] P i and M i These represent feature vectors from pathological images and MRI images, respectively. The loss function of the network is constructed based on the calculated similarity matrix:

[0095]

[0096] This is the calculated similarity matrix, where y is the corresponding label matrix, and L... CE λ represents the cross-entropy function. λ0 and λ1 represent the weight parameters. The similarity matrix is ​​formed by the MRI features M of lesion sample i. i WSI features P of lesion sample j j The similarity matrix is ​​formed by the similarity between lesion samples i and j, where lesion sample i and lesion sample j are any two lesion samples in the sample data, which can be the same lesion sample or different lesion samples. For example, assuming there are N lesion samples, an N*N dimensional similarity matrix can be obtained. Among the elements This represents the similarity between lesion sample i and lesion sample j. The label matrix corresponding to the similarity matrix is ​​also an N*N dimensional matrix, where the elements y i,j This indicates whether lesion sample i and lesion sample j are the same lesion. For example, when they are the same lesion, y i,j The value is 1, otherwise y i,j The value is set to 0. The loss value is calculated based on the loss function, and the multimodal contrastive learning framework is optimized by minimizing the loss value.

[0097] (5) Feature Denoising. After obtaining a set of MRI and WSI features corresponding to the magnetic resonance image and digital pathology image to be predicted, an element with a value of 1 is first extended to the end of each modality feature to preserve the original feature representation of the fused MRI and WSI, as shown in formula (4):

[0098]

[0099] h u The dimension then becomes d h +1, where p and m represent the WSI pathology modality and MRI imaging modality, respectively. To avoid noise interference in the pathology or imaging feature vectors, this example can also introduce a gated attention mechanism module to suppress undesirable noise in the features. For each modality's feature h u Learn a linear transformation W p,m→u To evaluate the importance score z of each feature element u :

[0100] z u =sigmoid(W p,m→u h u (5)

[0101] In formula (5), z u W represents the attention score of the feature vector. p,m→i The weight matrix parameters are learned through feature gating, and the feature h is taken. i and z i If the element-wise product is used to generate the gated representation, then the feature h u Converted to h u,gated The definition is as follows:

[0102] h u,gated =z u *ReLU(MLP(h u (6)

[0103] ReLU(x) = max(0,x) (7)

[0104] z u The value of takes the range (0,1) and has a smooth gradient, which allows the attention score of the gating mechanism to be regarded as a probability value, used to control the preservation or suppression of features. MLP is a linear mapping, and ReLU is an activation function, whose main function is to introduce nonlinearity.

[0105] (6) Multimodal Feature Interaction and Fusion via Matrix Outer Product. After post-processing the features of MRI images and WSI pathology, this example uses matrix outer product to fuse the features of pathology and images. Matrix outer product is an operation between two matrices of arbitrary size; the specific operation involves multiplying each element of the first matrix by the second complete matrix. It can be represented as:

[0106]

[0107] Among them, h p,gated and h m,gated This indicates that it is the denoised feature vector corresponding to WSI pathology or MRI images, and the dimension obtained after feature fusion calculation is... By using matrix outer product operations, each value in the pathological feature vector can be interactively processed with the value in the image feature vector.

[0108] (7) Multilayer perceptron for predicting patient prognosis. After the feature fusion operation of MRI and WSI features, the feature dimension was expanded from one dimension to two dimensions, and the features between the two modalities were interacted and fused one by one. In order to obtain the final multimodal feature vector, the fused feature matrix was first flattened row by row, and then multilayer perceptron was used for prognosis prediction.

[0109] From the above, we can see that:

[0110] 1. Before extracting features from multimodal data, this embodiment first converts the slice-level WSI images into graph-structured data. Specifically, after cropping the WSI into small image patches, each small image patch is treated as a graph node based on its coordinates in the original image. All small image patches are input into a CTransPath network pre-trained on a public dataset to obtain image patch features, for example, image patch features with a length of 768 dimensions. Subsequently, the eight-neighbor spatial relationships between image patches are used to determine whether the coordinates of the image patches are adjacent, and the edge relationships between nodes are constructed, thereby obtaining a graph-structured representation of the patient's slice-level WSI data.

[0111] Through the above methods, this embodiment of the application, taking into account the data characteristics of WSI and MRI, converts them into WSI graph structure data and 3D MRI image matrices respectively, based on the coordinates of image patches in WSI and the complementary effects of different sequence images in MRI. Using the WSI graph structure data and 3D MRI image matrices as dual-stream inputs to a neural network enables direct learning of patient-level features. Furthermore, through end-to-end training, patient-level WSI and MRI features can be directly updated and optimized, thus avoiding information loss that may occur during feature aggregation and capturing the internal contextual relationships of the images.

[0112] 2. In the feature extraction process of multimodal data, this application proposes a pathology-image multimodal contrastive learning framework, which learns the feature association between WSI and MRI in a self-supervised manner. A pathology-image similarity loss function is designed to maximize the feature similarity between pathology-image pairs belonging to the same patient and minimize the feature similarity between pathology-image pairs from different patients. To achieve data alignment between pathology-image data pairs from the same patient, for MRI data, images from different sequences can be channel-stitched to obtain a 3D image matrix containing multiple MRI sequences. A 3D ResNet-10 pre-trained on other medical image datasets is used as the image feature encoder. For WSI data, the WSI image is used as input, and a graph neural network is used as the pathology image encoder. Compared to existing technologies, this application proposes a pathology-image multimodal contrastive learning framework for feature learning, constructing the association between pathology and image data features belonging to the same patient through a similarity loss function, and using this association to improve the extracted image features. Secondly, when performing fusion operations on features of different modalities, the embodiments of this application provide a more interactive fusion strategy, and perform noise filtering and dimensionality reduction before and after feature fusion, which is more conducive to analyzing the relationship between multimodal data.

[0113] 3. In the process of multimodal feature fusion for prognosis prediction, this application embodiment first uses linear projection to map the features of both modalities to the same representation space. Specifically, due to the differences in MRI and WSI data formats, the extracted feature vectors are located in different feature spaces. In order to effectively fuse the two, this application proposes to use linear projection to transform the features through a fully connected layer while unifying the feature dimensions. Subsequently, a gated attention mechanism is used to filter noise from the features of each modality, and the features are fused through matrix outer product and predicted using a multilayer perceptron to predict the patient's prognostic risk.

[0114] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0115] Please refer to Figure 4 This application also provides a disease prognosis prediction system based on image data, including:

[0116] The acquisition module 401 is used to acquire a first image and a second image of the lesion to be predicted, wherein the first image is at least one magnetic resonance imaging (MRI) image and the second image is at least one full-view digital pathology image (WSI).

[0117] The preprocessing module 402 is used to perform image preprocessing on the first image to obtain the image matrix of the lesion to be predicted; to perform image preprocessing on the second image to obtain the foreground image of the lesion to be predicted; to divide the foreground image into multiple image blocks through a sliding window of fixed size; and to represent the multiple image blocks of the foreground image as graph structure data according to the connectivity between the image blocks.

[0118] The feature conversion module 403 is used to input the image matrix and the graph structure data into a pre-trained multimodal contrast learning framework to obtain a first MRI feature corresponding to the image matrix and a first WSI feature corresponding to the graph structure data; wherein, the multimodal contrast learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network and a second linear projection layer, the 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, and the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data;

[0119] The prediction module 404 is used to predict the prognosis of the lesion to be predicted based on the first MRI feature and the first WSI feature.

[0120] Optionally, the preprocessing module 402 is further configured to perform N4 bias field correction, resampling and intensity normalization on the first image to obtain a third image; and to stitch together the third images of different sequences according to channels to obtain the image matrix of the lesion to be predicted.

[0121] Optionally, the preprocessing module 402 is further configured to perform color correction processing on the second image to obtain a fourth image; and to use the Otsu method to separate the foreground image of the lesion to be predicted from the fourth image.

[0122] Optionally, the preprocessing module 402 is further configured to input multiple image blocks belonging to the same foreground image into a pre-trained CTransPath network to obtain the image features of each image block output by the CTransPath network; and generate graph structure data based on the image features of each image block and the connectivity of each image block in the foreground image.

[0123] Optionally, the prediction module is further configured to:

[0124] The first MRI feature and the first WSI feature are respectively subjected to noise reduction processing to obtain the second MRI feature and the second WSI feature;

[0125] The outer product of the matrix of the second MRI feature and the second WSI feature is calculated to obtain the first fusion feature;

[0126] The first fused feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result generated by the multilayer perceptron.

[0127] Optionally, the above system may also include:

[0128] The first training module is used for:

[0129] Acquire sample data, which includes first and second sample images of multiple lesion samples and prognostic results of the lesion samples, wherein the first sample image is an MRI image and the second sample image is a WSI image;

[0130] The first sample image is preprocessed to obtain the sample image matrix of the lesion sample; the second sample image is preprocessed to obtain the foreground sample image of the lesion sample; the foreground sample image is divided into multiple image blocks by a sliding window of fixed size; and the multiple image blocks of the foreground sample image are represented as sample image structure data according to the connectivity between the image blocks.

[0131] Through multiple iterations, the similarity between the sample image matrix and sample image structure data of the lesion samples is learned. In each iteration, a sample image matrix and a sample image structure data are randomly selected and input into the multimodal contrast learning framework to obtain the third MRI feature corresponding to the sample image matrix and the third WSI feature corresponding to the sample image structure data. The similarity between the third MRI feature and the third WSI feature is calculated to obtain the similarity between the lesion sample corresponding to the third MRI feature and the lesion sample corresponding to the third WSI feature, and the similarity matrix between each lesion sample is updated. Based on the similarity matrix and the label matrix corresponding to the similarity matrix, the loss function of the multimodal contrast learning framework is constructed and the loss value is calculated. Based on the loss value, the multimodal contrast learning framework is optimized until the preset training termination condition is reached to obtain the trained multimodal contrast learning framework.

[0132] Optionally, the above system may also include:

[0133] The second training module is used for:

[0134] The sample image matrix and sample image structure data of the same lesion sample are input into the trained multimodal contrast learning framework to obtain the fourth MRI feature corresponding to the sample image matrix and the fourth WSI feature corresponding to the sample image structure data.

[0135] The fourth MRI feature and the fourth WSI feature are respectively subjected to noise reduction processing to obtain the fifth MRI feature and the fifth WSI feature;

[0136] The outer product of the matrices of the fifth MRI feature and the fifth WSI feature is calculated to obtain the second fusion feature;

[0137] The second fusion feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result of the lesion sample generated by the multilayer perceptron;

[0138] Based on the prognostic prediction results and prognostic results of the lesion samples, the multilayer perceptron is trained to obtain a trained multilayer perceptron.

[0139] It should be noted that the device in this embodiment corresponds to the method described above, and the implementation methods in each of the above embodiments are applicable to the device embodiment and can achieve the same technical effect. The device provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0140] Another embodiment of this application provides a disease prognosis prediction system based on image data, such as... Figure 5 As shown, it includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the various processes of the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0141] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0142] Among them, Figure 5 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 500 and memory represented by memory 520 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 510 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, user interface 530 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0143] The processor 500 is responsible for managing the bus architecture and general processing, while the memory 520 can store the data used by the processor 500 when performing operations.

[0144] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described embodiments of the disease prognosis prediction method based on image data, and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0145] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described disease prognosis prediction method embodiment based on image data and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0148] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A disease prognosis prediction method based on image data, characterized in that, include: Acquire a first image and a second image of the lesion to be predicted, wherein the first image is at least one magnetic resonance imaging (MRI) image and the second image is at least one full-view digital pathology image (WSI); The first image is preprocessed to obtain the image matrix of the lesion to be predicted; The second image is preprocessed to obtain the foreground image of the lesion to be predicted. The foreground image is divided into multiple image blocks by a sliding window of fixed size. Based on the connectivity between the image blocks, the multiple image blocks of the foreground image are represented as graph structure data. The image matrix and the graph structure data are input into a pre-trained multimodal contrastive learning framework to obtain a first MRI feature corresponding to the image matrix and a first WSI feature corresponding to the graph structure data. The multimodal contrastive learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network, and a second linear projection layer. The 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, while the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data. Based on the first MRI feature and the first WSI feature, the prognosis of the lesion to be predicted is predicted.

2. The method according to claim 1, characterized in that, Image preprocessing is performed on the first image to obtain the image matrix of the lesion to be predicted, including: The first image is subjected to N4 bias field correction, resampling and intensity normalization to obtain the third image; the third images of different sequences are stitched together according to channels to obtain the image matrix of the lesion to be predicted.

3. The method according to claim 1, characterized in that, The second image is preprocessed to obtain the foreground image of the lesion to be predicted, including: The second image is color-corrected to obtain a fourth image; the foreground image of the lesion to be predicted is separated from the fourth image using the Otsu method.

4. The method according to claim 1, characterized in that, Based on the connectivity relationships between image patches, the multiple image patches of the foreground image are represented as graph structure data, including: Multiple image patches belonging to the same foreground image are input into a pre-trained CTransPath network to obtain the image features of each image patch output by the CTransPath network. Graph structure data is generated based on the image features of each image patch and the connectivity of each image patch in the foreground image.

5. The method according to claim 1, characterized in that, Based on the first MRI feature and the first WSI feature, a prognostic prediction is made for the lesion site to be predicted, including: The first MRI feature and the first WSI feature are respectively subjected to noise reduction processing to obtain the second MRI feature and the second WSI feature; The outer product of the matrix of the second MRI feature and the second WSI feature is calculated to obtain the first fusion feature; The first fused feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result generated by the multilayer perceptron.

6. The method according to claim 1, characterized in that, Also includes: Acquire sample data, which includes first and second sample images of multiple lesion samples and prognostic results of the lesion samples, wherein the first sample image is an MRI image and the second sample image is a WSI image; The first sample image is preprocessed to obtain the sample image matrix of the lesion sample; the second sample image is preprocessed to obtain the foreground sample image of the lesion sample; the foreground sample image is divided into multiple image blocks by a sliding window of fixed size; and the multiple image blocks of the foreground sample image are represented as sample image structure data according to the connectivity between the image blocks. Through multiple iterations, the similarity between the sample image matrix and sample image structure data of the lesion samples is learned. In each iteration, a sample image matrix and a sample image structure data are randomly selected and input into the multimodal contrast learning framework to obtain the third MRI feature corresponding to the sample image matrix and the third WSI feature corresponding to the sample image structure data. The similarity between the third MRI feature and the third WSI feature is calculated to obtain the similarity between the lesion sample corresponding to the third MRI feature and the lesion sample corresponding to the third WSI feature, and the similarity matrix between each lesion sample is updated. Based on the similarity matrix and the label matrix corresponding to the similarity matrix, the loss function of the multimodal contrast learning framework is constructed and the loss value is calculated. Based on the loss value, the multimodal contrast learning framework is optimized until the preset training termination condition is reached to obtain the trained multimodal contrast learning framework.

7. The method according to claim 6, characterized in that, Also includes: The sample image matrix and sample image structure data of the same lesion sample are input into the trained multimodal contrast learning framework to obtain the fourth MRI feature corresponding to the sample image matrix and the fourth WSI feature corresponding to the sample image structure data. The fourth MRI feature and the fourth WSI feature are respectively subjected to noise reduction processing to obtain the fifth MRI feature and the fifth WSI feature; The outer product of the matrices of the fifth MRI feature and the fifth WSI feature is calculated to obtain the second fusion feature; The second fusion feature is converted into a one-dimensional matrix and input into a pre-trained multilayer perceptron to obtain the prognostic prediction result of the lesion sample generated by the multilayer perceptron; Based on the prognostic prediction results and prognostic results of the lesion samples, the multilayer perceptron is trained to obtain a trained multilayer perceptron.

8. A disease prognosis prediction system based on image data, characterized in that, include: The acquisition module is used to acquire a first image and a second image of the lesion to be predicted, wherein the first image is at least one magnetic resonance imaging (MRI) image and the second image is at least one full-view digital pathology image (WSI). The preprocessing module is used to perform image preprocessing on the first image to obtain the image matrix of the lesion to be predicted; The second image is preprocessed to obtain the foreground image of the lesion to be predicted. The foreground image is divided into multiple image blocks by a sliding window of fixed size. Based on the connectivity between the image blocks, the multiple image blocks of the foreground image are represented as graph structure data. A feature conversion module is used to input the image matrix and the graph structure data into a pre-trained multimodal contrastive learning framework to obtain a first MRI feature corresponding to the image matrix and a first WSI feature corresponding to the graph structure data. The multimodal contrastive learning framework includes a 3D medical image pre-training model, a first linear projection layer, a graph neural network, and a second linear projection layer. The 3D medical image pre-training model and the first linear projection layer generate MRI features based on the input image matrix, and the graph neural network and the second linear projection layer generate WSI features based on the input graph structure data. The prediction module is used to predict the prognosis of the lesion to be predicted based on the first MRI feature and the first WSI feature.

9. The system according to claim 8, characterized in that, The preprocessing module is also used for: Multiple image patches belonging to the same foreground image are input into a pre-trained CTransPath network to obtain the image features of each image patch output by the CTransPath network. Graph structure data is generated based on the image features of each image patch and the connectivity of each image patch in the foreground image.

10. A disease prognosis prediction system based on image data, characterized in that, include: Transceiver, processor, memory, and programs or instructions stored in the memory and executable on the processor; When the processor executes the program or instructions, it implements the steps of the method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images