Tumor risk prediction method and device and storage medium
By fusion of multimodal features of nuclear magnetic image sequences and pathological images, and using deep learning neural networks to predict tumor risk, the problem of low complexity and accuracy of tumor diagnosis in the existing technology is solved, and fully automated, fast and accurate is achieved tumor risk assessment.
Patent Information
- Application Number
- CN202410950063.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
The existing tumor risk prediction methods are complex, inefficient and low-accurate in diagnosis, making it difficult to achieve fully automatic, fast and accurate tumor diagnosis, especially in the scenarios of brain glioma.
Through feature extraction based on nuclear magnetic image sequence and pathological image, multimodal feature fusion is carried out, and deep learning neural network is used for classification to obtain tumor risk prediction results. Combined with attention scores and mutual attention mechanisms, the rapid and accurate risk of tumors is achieved. Evaluate.
It improves the efficiency and accuracy of medical diagnosis, can more accurately indicate the degree of tumor risk, reduces the possibility of misdiagnosis and misjudgment, and achieves fully automated tumor risk prediction.
Smart Images

Figure CN118919073A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a tumor risk prediction method, device and storage medium. Background Art
[0002] With the development of information technology, artificial intelligence technology has been widely used in the medical field. Computer methods based on deep learning can assist doctors in disease prediction, greatly improving the efficiency and accuracy of medical diagnosis. In the scenario of risk diagnosis of tumors (such as gliomas), due to the short diagnosis time, heavy tasks, high risks, and high difficulty in surgery, it is very easy to misdiagnose and misjudge. Therefore, manual diagnosis is difficult and automatic machine diagnosis is urgently needed.
[0003] However, current computational pathology methods often focus on the final diagnosis based on conventional pathology images, and pay more attention to the integration of pathology images with clinical information, molecular characteristics and other modal data. In addition, tumor risk prediction based on existing methods is complex and inefficient, not suitable for clinical diagnosis, and the results are less accurate. Therefore, a new tumor risk prediction scheme is urgently needed to achieve fast and accurate diagnosis of fully automatic tumors (such as gliomas). Summary of the invention
[0004] In view of this, the present disclosure proposes a tumor risk prediction method, device and storage medium.
[0005] According to one aspect of the present disclosure, a method for predicting tumor risk is provided. The method comprises:
[0006] Extracting features based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature;
[0007] Extracting features based on the pathological image containing the target tumor location to obtain a second image feature;
[0008] Performing multimodal feature fusion on the first image feature and the second image feature to obtain multimodal fused features;
[0009] Classification is performed based on the features after multimodal fusion to obtain the tumor risk prediction result, which is used to indicate the risk level of the tumor.
[0010] In a possible implementation, the tumor includes a glioma, and feature extraction is performed based on a nuclear magnetic resonance image sequence including a target tumor location to obtain a first image feature, including:
[0011] Performing gradient field correction on each nuclear magnetic image in the nuclear magnetic image sequence to obtain an image sequence after gradient field correction;
[0012] Performing skull removal and cropping processing on the image after gradient field correction to obtain a processed nuclear magnetic resonance image sequence;
[0013] Feature extraction is performed on each nuclear magnetic image in the processed nuclear magnetic image sequence to obtain the feature of each nuclear magnetic image as the first image feature.
[0014] In a possible implementation, feature extraction is performed based on a pathological image containing a target tumor location to obtain a second image feature, including:
[0015] The image corresponding to the tissue part in the pathological image is divided into multiple image blocks;
[0016] Feature extraction is performed on the multiple image blocks respectively, and the features corresponding to the respective image blocks are obtained as the second image features.
[0017] In a possible implementation, performing multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature includes:
[0018] Processing the first image feature based on the first linear mapping parameter to obtain a query feature;
[0019] Processing the second image feature based on the second linear mapping parameter to obtain a key feature;
[0020] Processing the second image feature based on the third linear mapping parameter to obtain a value feature;
[0021] Based on the query features and key features, an attention score is calculated, and the attention score represents the degree of correlation between the image blocks of the MRI image sequence and the pathological image;
[0022] Based on the attention score and value feature, the mutual attention value between the query feature, the key feature and the value feature is calculated as the feature after multimodal fusion.
[0023] In a possible implementation, classification is performed based on the features after multimodal fusion to obtain tumor risk prediction results, including:
[0024] Input the multimodal fused features and classification labels into a deep learning neural network to obtain an output vector of the classification labels, which is a semantic representation of the multimodal fused features;
[0025] The output vector of the classification label is input into the fully connected layer classifier to obtain the tumor risk prediction result, which is a binary classification result.
[0026] In a possible implementation, the method further includes:
[0027] Input the multimodal fused features and classification tags into the deep learning neural network to obtain the attention weights;
[0028] A preset number of nuclear magnetic resonance images in the nuclear magnetic resonance image sequence are selected in descending order of corresponding attention weights, where the preset number of nuclear magnetic resonance images are nuclear magnetic resonance images containing tumors;
[0029] A preset number of nuclear magnetic resonance images are used as the tumor localization result of the nuclear magnetic resonance image sequence.
[0030] In a possible implementation, the method further includes:
[0031] Based on the attention scores, a preset number of pathological images corresponding to the MRI images are selected for visualization to obtain the lesion areas of the pathological images.
[0032] According to another aspect of the present disclosure, a tumor risk prediction device is provided. The device comprises:
[0033] A first feature extraction module, configured to extract features based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature;
[0034] A second feature extraction module, used to extract features based on the pathological image containing the target tumor location to obtain a second image feature;
[0035] A multimodal feature fusion module, used to perform multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature;
[0036] The classification module is used to perform classification based on the features after multimodal fusion to obtain a tumor risk prediction result, which is used to indicate the risk level of the tumor.
[0037] In a possible implementation, the tumor includes a glioma, and the first feature extraction module is used to:
[0038] Performing gradient field correction on each nuclear magnetic image in the nuclear magnetic image sequence to obtain an image sequence after gradient field correction;
[0039] Performing skull removal and cropping processing on the image after gradient field correction to obtain a processed nuclear magnetic resonance image sequence;
[0040] Feature extraction is performed on each nuclear magnetic image in the processed nuclear magnetic image sequence to obtain the feature of each nuclear magnetic image as the first image feature.
[0041] In a possible implementation, the second feature extraction module is used to:
[0042] The image corresponding to the tissue part in the pathological image is divided into multiple image blocks;
[0043] Feature extraction is performed on the multiple image blocks respectively, and the features corresponding to the respective image blocks are obtained as the second image features.
[0044] In a possible implementation, the multimodal feature fusion module is used to:
[0045] Processing the first image feature based on the first linear mapping parameter to obtain a query feature;
[0046] Processing the second image feature based on the second linear mapping parameter to obtain a key feature;
[0047] Processing the second image feature based on the third linear mapping parameter to obtain a value feature;
[0048] Based on the query features and key features, an attention score is calculated, and the attention score represents the degree of correlation between the image blocks of the MRI image sequence and the pathological image;
[0049] Based on the attention score and value feature, the mutual attention value between the query feature, the key feature and the value feature is calculated as the feature after multimodal fusion.
[0050] In a possible implementation, the classification module is used to:
[0051] Input the multimodal fused features and classification labels into a deep learning neural network to obtain an output vector of the classification labels, which is a semantic representation of the multimodal fused features;
[0052] The output vector of the classification label is input into the fully connected layer classifier to obtain the tumor risk prediction result, which is a binary classification result.
[0053] In a possible implementation, the device further includes:
[0054] The input module is used to input the multimodal fused features and classification labels into the deep learning neural network to obtain the attention weights;
[0055] A selection module is used to select a preset number of nuclear magnetic resonance images in the nuclear magnetic resonance image sequence in descending order of corresponding attention weights, wherein the preset number of nuclear magnetic resonance images are nuclear magnetic resonance images containing tumors;
[0056] The determination module is used to use a preset number of nuclear magnetic resonance images as the tumor localization result of the nuclear magnetic resonance image sequence.
[0057] In a possible implementation, the device further includes:
[0058] The visualization processing module is used to select a preset number of pathological images corresponding to the nuclear magnetic resonance images for visualization processing based on the attention scores to obtain the lesion areas of the pathological images.
[0059] According to another aspect of the present disclosure, a tumor risk prediction device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.
[0060] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.
[0061] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0062] According to an embodiment of the present application, by performing feature extraction based on a sequence of MRI images and a pathological image containing the location of the target tumor, respectively, to obtain a first image feature and a second image feature, and performing multimodal feature fusion on the first image feature and the second image feature, image data of two different modalities with large differences in the amount of information can be combined to quickly perform risk prediction, and the obtained multimodal fused features fuse the pathological state and differentiation degree information of the tumor in the pathological image and the tumor location and tumor morphological change information in the MRI image. Therefore, by performing classification based on the multimodal fused features, a tumor risk prediction result can be obtained, which can more accurately indicate the risk level of the tumor, greatly improving the efficiency and accuracy of medical diagnosis.
[0063] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
[0065] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application.
[0066] Figure 2 A flow chart of a tumor risk prediction method according to an embodiment of the present application is shown.
[0067] Figure 3 A schematic diagram of a process for predicting tumor risk according to an embodiment of the present application is shown.
[0068] Figure 4A schematic diagram showing preprocessing of a nuclear magnetic resonance image according to an embodiment of the present application is shown.
[0069] Figure 5 A schematic diagram showing the tumor risk prediction results according to an embodiment of the present application.
[0070] Figure 6 A schematic diagram showing tumor localization using magnetic resonance imaging according to an embodiment of the present application is shown.
[0071] Figure 7 A schematic diagram showing lesion detection according to an embodiment of the present application is shown.
[0072] Figure 8 A structural diagram of a tumor risk prediction device according to an embodiment of the present application is shown.
[0073] Fig. 9 It is a block diagram of a device 1900 for tumor risk prediction according to an exemplary embodiment. DETAILED DESCRIPTION
[0074] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0075] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0076] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0077] With the development of information technology, artificial intelligence technology has been widely used in the medical field. Computer methods based on deep learning can assist doctors in disease prediction, greatly improving the efficiency and accuracy of medical diagnosis. In the scenario of tumor (such as glioma) risk diagnosis, due to the short diagnosis time, heavy tasks, high risks, and high difficulty in surgery, it is very easy to misdiagnose and misjudge. Therefore, manual diagnosis is difficult and there is an urgent need for automatic machine diagnosis. However, current computational pathology methods often focus on the final diagnosis based on conventional pathological images, and pay more attention to the fusion of pathological images with clinical information, molecular characteristics and other modal data. In addition, tumor risk prediction based on existing methods is complex and inefficient, not suitable for clinical diagnosis, and the results are less accurate. Therefore, there is an urgent need for a new tumor risk prediction scheme to achieve rapid and accurate diagnosis of fully automatic tumors (such as brain gliomas).
[0078] In view of this, the present application proposes a tumor risk prediction method, device and storage medium. The method extracts features based on the nuclear magnetic resonance image sequence and pathological image containing the target tumor location, obtains the first image feature and the second image feature, and performs multimodal feature fusion on the first image feature and the second image feature. The image data of two different modalities with large differences in information volume can be combined to quickly predict risks, and the obtained multimodal fused features fuse the pathological state and differentiation degree information of the tumor in the pathological image and the tumor location and tumor morphological change information in the nuclear magnetic resonance image. Therefore, by classifying based on the multimodal fused features, the tumor risk prediction result can be obtained, which can more accurately indicate the risk level of the tumor, greatly improving the efficiency and accuracy of medical diagnosis.
[0079] Figure 1 A schematic diagram of an application scenario according to an embodiment of the present application is shown. The tumor risk prediction system of the present application can be used in the scenario of risk prediction of brain glioma, such as Figure 1 As shown, the tumor risk prediction system of the present application can obtain pathological images obtained by rapid freezing and sectioning tissue samples, as well as brain magnetic resonance imaging (MRI) sequences, and after multimodal fusion of the features of the pathological images and MRI sequences, fully automated rapid diagnosis of brain gliomas can be performed based on the fused features to obtain tumor risk prediction results, tumor localization results for MRI sequences, and lesion areas for pathological images, etc., so that the present application can be applied to scenarios such as rapid intraoperative diagnosis, tumor localization by magnetic resonance imaging, and lesion detection by frozen sections.
[0080] The tumor risk prediction system of the present application can be used in a terminal device or a server, and the terminal device can be any one or more of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and a vehicle-mounted device. The present disclosure embodiment does not impose any special restrictions on the specific type of the terminal device, and it can have a wired or wireless communication function.
[0081] The server can be located locally or in the cloud, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. The wireless communication function can be realized, for example, through mobile communication technologies such as 2G / 3G / 4G / 5G, as well as Wi-Fi, Bluetooth, frequency modulation (FM), digital radio, satellite communication, etc. Communication can also be carried out through wired connection to achieve interaction with other devices.
[0082] Figure 2 FIG. 1 is a flow chart showing a tumor risk prediction method according to an embodiment of the present application. The method can be used in the above-mentioned tumor risk prediction system, such as Figure 2 As shown, the method may include:
[0083] Step S201, performing feature extraction based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature.
[0084] The tumor may include a glioma. In glioma resection surgery, doctors need to rely on quick judgment of cut tissue samples during the operation to accurately determine the range of resection. The pathological images obtained after rapid freezing and sectioning of tissue samples can help with diagnosis and determine the pathological state and degree of differentiation of the tumor. The MRI image sequence can be used to locate the tumor and determine changes in tumor morphology, which helps to determine surgical boundaries and specify surgical plans. It can also provide supplementary information for tumor risk prediction and improve the accuracy of diagnosis. In this application, MRI image sequences and pathological images including gliomas are obtained, and the features of the two are multimodally fused to combine the rich information in the two for tumor risk prediction.
[0085] The nuclear magnetic image sequence is the above-mentioned MRI sequence, which can be obtained by scanning the MRI image using preset scanning parameters when performing nuclear magnetic resonance imaging examination on the brain. In this application, the nuclear magnetic image sequence of T1-weighted imaging and axial direction (axial, AX) in the MRI image can be selected. Each nuclear magnetic image sequence can include one or more nuclear magnetic slice images (hereinafter referred to as nuclear magnetic images).
[0086] Figure 3 FIG. 1 is a flow chart showing a process for predicting tumor risk according to an embodiment of the present application. The obtained nuclear magnetic resonance image sequence can be as follows: Figure 3 shown.
[0087] Each nuclear magnetic resonance image in the nuclear magnetic resonance image sequence can be preprocessed to facilitate more accurate feature extraction later.
[0088] In step S201, you can:
[0089] Performing gradient field correction on each nuclear magnetic image in the nuclear magnetic image sequence to obtain an image sequence after gradient field correction;
[0090] Performing skull removal and cropping processing on the image after gradient field correction to obtain a processed nuclear magnetic resonance image sequence;
[0091] Feature extraction is performed on each nuclear magnetic image in the processed nuclear magnetic image sequence to obtain the feature of each nuclear magnetic image as the first image feature.
[0092] The gradient field correction may be N4 gradient field correction, and the clipping process may be center clipping.
[0093] See also Figure 4 , showing a schematic diagram of preprocessing a nuclear magnetic resonance image according to an embodiment of the present application. Figure 4 As shown in the figure, due to the change of magnetic field during the scanning process of different nuclear magnetic scanners, the intensity of the nuclear magnetic image sequence may be uneven, so the N4 gradient field correction can be performed on each nuclear magnetic image in the nuclear magnetic image sequence. Gradient field correction usually refers to the process of correcting the gradient field of the magnetic field in magnetic resonance imaging (MRI). The purpose of gradient field correction is to reduce image distortion or data error caused by magnetic field inhomogeneity.
[0094] Secondly, since the skull may affect the feature extraction of the brain tissue, the image after gradient field correction can be de-skulled to extract the brain tissue from the inside of the skull.
[0095] Next, the image may be center cropped, where the center cropping may be to crop a sub-image of a specific size from the center area of the image as the processed image.
[0096] like Figure 3 As shown in FIG. 1 , after obtaining the processed nuclear magnetic resonance image sequence, the pre-trained magnetic resonance network (MRNet) can be used to perform deep feature extraction on each nuclear magnetic resonance image in the processed nuclear magnetic resonance image sequence to obtain the first image feature. The first image feature can represent the texture, shape, signal strength, etc. in the image. The first image feature of each nuclear magnetic resonance image in the nuclear magnetic resonance image sequence can be recorded as l is the identifier of the MRI image.
[0097] The features extracted from a MRI sequence of a patient can be packaged as As the first image feature of the nuclear magnetic resonance image sequence, L is the length of the nuclear magnetic resonance image sequence, and d is the dimension of the feature.
[0098] Step S202: extracting features based on the pathological image containing the target tumor location to obtain second image features.
[0099] Pathological images can be obtained by rapidly freezing and slicing tissue samples. In order to use the Whole Slide Image (WSI) level labels for diagnosis, we need to use a package integration method of multi-instance learning, where each package can contain features extracted from a pathological image divided into multiple patches. See below, in step S202, you can:
[0100] The image corresponding to the tissue part in the pathological image is divided into multiple image blocks;
[0101] Feature extraction is performed on the multiple image blocks respectively, and the features corresponding to the respective image blocks are obtained as the second image features.
[0102] Each image patch may be of the same size.
[0103] like Figure 3 As shown in Figure 1, the pre-trained Resnet50 can be used to extract features from each image block to obtain the second image features corresponding to each image block. The second image features can reflect the tissue structure, cell morphology, staining pattern and other information in the image block. The second image features of each image block can be recorded as j is the identifier of the image block.
[0104] Features from a pathology image can be packaged as J is the total number of patches obtained by segmenting a pathological image, and d is the dimension size of the feature.
[0105] Step S203, performing multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature.
[0106] Due to the large difference in the magnitude of modal information between pathological images and MRI images, it is difficult to effectively fuse the information of the two modalities. In this application, the mutual attention mechanism is used to fuse the information of the two modalities, in which the features of the MRI image sequence are used as a guide (as a query feature) for fusion.
[0107] In step S203, you can:
[0108] Processing the first image feature based on the first linear mapping parameter to obtain a query feature;
[0109] Processing the second image feature based on the second linear mapping parameter to obtain a key feature;
[0110] Processing the second image feature based on the third linear mapping parameter to obtain a value feature;
[0111] Calculate the attention score based on the query features and key features;
[0112] Based on the attention score and the value feature, the mutual attention value between the query feature and the key feature and the value feature is calculated as the feature after multimodal fusion.
[0113] Among them, the first linear mapping parameter W q , the second linear mapping parameter W k and the third linear mapping parameter W v can be different trainable parameters, which can be Figure 3 The trainable parameters in the tumor risk prediction model shown can be obtained by pre-training the model. Case-level weak supervision labels can be used during training. The training is performed based on the loss between the prediction result and the label obtained after the sample image is input into the model. The training method can be implemented based on existing technology.
[0114] W q , W k and W v It can be used for linear mapping of the first image feature, the second image feature, and the second image feature to obtain the query feature (query) Key feature (key) Sum value feature (value) Examples of Q, K, and V can be found in Figure 3 Among them, one way to calculate Q, K and V can be referred to the following formula (1), formula (2) and formula (3) respectively:
[0115] Q=W q ·Hmri Formula (1)
[0116] K=W k ·H patch Formula (2)
[0117] V=W v ·H patch Formula (3)
[0118] The key information in the MRI image, such as the location, size, shape, etc. of the tumor, can be encoded in the query feature, which is used to interact with the feature identifiers of the pathological image to find the pathological feature most relevant to the key information; the key feature can be used to match and calculate similarity with the query feature to determine the lesion area in the pathological image that is most relevant to the query feature; the value feature can be information associated with the key feature. When the key feature most relevant to the query feature is determined, the value feature can be used to generate the final mutual attention value.
[0119] One way to calculate the attention score can be found in formula (4):
[0120]
[0121] in, is the attention score, softmax(·) indicates normalization, and d is the preset feature dimension. The attention score can represent the correlation between the MRI image sequence (including L MRI images) and the image blocks of the pathological image (a total of J). It can be seen that by calculating the attention score, the number of features of the pathological image can be reduced from J to L, thereby reducing the difficulty and complexity of obtaining the output vector of the classification label based on the features after multimodal fusion.
[0122] One way to calculate the features after multimodal fusion can be seen in formula (5):
[0123]
[0124] Among them, H fusion It can represent the features after multimodal fusion, and CoA(Q, K, V) can represent the mutual attention value between the query feature and the key feature and value feature.
[0125] Step S204, classifying based on the multimodal fused features to obtain a tumor risk prediction result.
[0126] Among them, the tumor risk prediction result can be used to indicate the risk level of the tumor. The tumor risk prediction result can be a classification result. The higher the risk level, the more likely the tumor (glioma) is to be a malignant tumor.
[0127] According to an embodiment of the present application, by performing feature extraction based on a sequence of MRI images and a pathological image containing the location of the target tumor, respectively, to obtain a first image feature and a second image feature, and performing multimodal feature fusion on the first image feature and the second image feature, image data of two different modalities with large differences in the amount of information can be combined to quickly perform risk prediction, and the obtained multimodal fused features fuse the pathological state and differentiation degree information of the tumor in the pathological image and the tumor location and tumor morphological change information in the MRI image. Therefore, by performing classification based on the multimodal fused features, a tumor risk prediction result can be obtained, which can more accurately indicate the risk level of the tumor, greatly improving the efficiency and accuracy of medical diagnosis.
[0128] In step S204, you can:
[0129] Input the multimodal fused features and classification labels into the deep learning neural network to obtain the output vector of the classification label;
[0130] The output vector of the classification label is input into the fully connected layer classifier to obtain the tumor risk prediction result.
[0131] Among them, deep learning neural network is for example Transformer network (see Figure 3 ), the classification tag (clstoken) can be a special input tag [CLS] in the Transformer network. This tag is a preset tag in the Transformer network and is usually added to the beginning of the input sequence (i.e., the feature after multimodal fusion) to capture the semantic information of the entire input sequence for subsequent classification tasks. The output vector of the classification tag encoded by the Transformer network can be the semantic representation of the feature after multimodal fusion.
[0132] Transformer networks and fully connected classifiers (such as Figure 3 The Classifier in the example can be obtained through pre-training. During training, weak supervision labels at the case level can be used. The loss between the classification result and the label obtained by inputting the multimodal fused features and classification labels corresponding to the sample images into the Transformer network and the fully connected classifier is used for training. The training method can be implemented based on existing technologies.
[0133] The fully connected classifier can be a binary classifier, and the final tumor risk prediction result can be a binary classification result, for example, indicating that brain glioma is high risk / low risk.
[0134] Figure 5 A schematic diagram showing the tumor risk prediction results according to an embodiment of the present application is shown. Figure 5 As shown, the model can be expressed Figure 3 The tumor risk prediction model in , the tumor risk prediction result can indicate the probability of the tumor being high risk / low risk, such as Figure 5 The probability of high risk in this example is 0.86, and the probability of low risk is 0.24, indicating that the tumor is more likely to be a high-risk tumor.
[0135] After experimental verification, the tumor risk prediction model using this method on a multimodal dataset can achieve a joint improvement in accuracy, F1-score (the harmonic mean of precision and recall), and AUC (Area Under the Receiver Operating Characteristic Curve), with the accuracy being improved by more than 5% compared to the existing single-modal method.
[0136] This application also constructs an interpretable result analysis method. By locating the MRI images containing tumors and detecting lesions on pathological images, the method of this application also achieves the interpretability of tumor risk prediction, thereby better assisting diagnosis. Among them, the method may also include:
[0137] Input the multimodal fused features and classification tags into the deep learning neural network to obtain the attention weights;
[0138] Select a preset number of nuclear magnetic resonance images in the nuclear magnetic resonance image sequence in descending order of corresponding attention weights;
[0139] A preset number of nuclear magnetic resonance images are used as the tumor localization result of the nuclear magnetic resonance image sequence.
[0140] The deep learning neural network may be the above-mentioned Transformer model. The attention weight may be an intermediate result output by the Transformer model, obtained based on the self-attention mechanism within the Transformer model, indicating the degree of attention paid by the Transformer model to each nuclear magnetic resonance image in the nuclear magnetic resonance image sequence, wherein the higher the degree of attention, the more helpful the nuclear magnetic resonance image is for the prediction of the classifier.
[0141] A preset number of MRI images (for example, the top K MRI images, K is a positive integer) can be used as MRI images containing tumors, that is, the largest K attention weights are selected, and the K MRI images corresponding to the K attention weights are determined, so that the MRI images where the tumor is located can be located.
[0142] Figure 6 FIG. 1 is a schematic diagram showing tumor localization using a nuclear magnetic resonance image according to an embodiment of the present application. Figure 6 As shown in the figure, by selecting the MRI images in the MRI image sequence through the attention weight, the main tumors (such as Figure 6The area indicated by the orange box in the figure is located in the NMR slice.
[0143] In order to perform further visualization analysis to determine the lesion area, the method may further include:
[0144] Based on the attention scores, a preset number of pathological images corresponding to the MRI images are selected for visualization to obtain the lesion areas of the pathological images.
[0145] The attention score may be the aforementioned AttnScore, and the lesion region is a portion of the tumor tissue region.
[0146] For example, a preset number (for example, N, where N is a positive integer) of image blocks of pathological images corresponding to MRI images may be selected in descending order of AttnScore to perform heat map formatting processing to obtain heat maps corresponding to multiple image blocks of pathological images. Figure 7 , shows a schematic diagram of lesion detection according to an embodiment of the present application. Figure 7 The left side shows a pathological image. Based on the attention scores of the image blocks corresponding to N MRI images in the pathological image, we can get the following: Figure 7 The heat map shown on the right shows that the darker the color (red), the stronger the positive correlation between the area and the lesion area in the pathological image.
[0147] Figure 8 FIG. 2 shows a structural diagram of a tumor risk prediction device according to an embodiment of the present application. Figure 8 As shown, the device comprises:
[0148] A first feature extraction module 801 is used to extract features based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature;
[0149] A second feature extraction module 802, configured to extract features based on the pathological image containing the target tumor location to obtain a second image feature;
[0150] The multimodal feature fusion module 803 is used to perform multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature;
[0151] The classification module 804 is used to perform classification based on the features after multimodal fusion to obtain a tumor risk prediction result, and the tumor risk prediction result is used to indicate the risk level of the tumor.
[0152] In a possible implementation, the tumor includes a glioma, and the first feature extraction module 801 is used to:
[0153] Performing gradient field correction on each nuclear magnetic image in the nuclear magnetic image sequence to obtain an image sequence after gradient field correction;
[0154] Performing skull removal and cropping processing on the image after gradient field correction to obtain a processed nuclear magnetic resonance image sequence;
[0155] Feature extraction is performed on each nuclear magnetic image in the processed nuclear magnetic image sequence to obtain the feature of each nuclear magnetic image as the first image feature.
[0156] In a possible implementation, the second feature extraction module 802 is used to:
[0157] The image corresponding to the tissue part in the pathological image is divided into multiple image blocks;
[0158] Feature extraction is performed on the multiple image blocks respectively, and the features corresponding to the respective image blocks are obtained as the second image features.
[0159] In a possible implementation, the multimodal feature fusion module 803 is used to:
[0160] Processing the first image feature based on the first linear mapping parameter to obtain a query feature;
[0161] Processing the second image feature based on the second linear mapping parameter to obtain a key feature;
[0162] Processing the second image feature based on the third linear mapping parameter to obtain a value feature;
[0163] Based on the query features and key features, an attention score is calculated, and the attention score represents the degree of correlation between the image blocks of the MRI image sequence and the pathological image;
[0164] Based on the attention score and value feature, the mutual attention value between the query feature, the key feature and the value feature is calculated as the feature after multimodal fusion.
[0165] In a possible implementation, the classification module 804 is used to:
[0166] Input the multimodal fused features and classification labels into a deep learning neural network to obtain an output vector of the classification labels, which is a semantic representation of the multimodal fused features;
[0167] The output vector of the classification label is input into the fully connected layer classifier to obtain the tumor risk prediction result, which is a binary classification result.
[0168] In a possible implementation, the device further includes:
[0169] The input module is used to input the multimodal fused features and classification labels into the deep learning neural network to obtain the attention weights;
[0170] A selection module is used to select a preset number of nuclear magnetic resonance images in the nuclear magnetic resonance image sequence in descending order of corresponding attention weights, wherein the preset number of nuclear magnetic resonance images are nuclear magnetic resonance images containing tumors;
[0171] The determination module is used to use a preset number of nuclear magnetic resonance images as the tumor localization result of the nuclear magnetic resonance image sequence.
[0172] In a possible implementation, the device further includes:
[0173] The visualization processing module is used to select a preset number of pathological images corresponding to the nuclear magnetic resonance images for visualization processing based on the attention scores to obtain the lesion areas of the pathological images.
[0174] According to an embodiment of the present application, by performing feature extraction based on a sequence of MRI images and a pathological image containing the location of the target tumor, respectively, to obtain a first image feature and a second image feature, and performing multimodal feature fusion on the first image feature and the second image feature, image data of two different modalities with large differences in the amount of information can be combined to quickly perform risk prediction, and the obtained multimodal fused features fuse the pathological state and differentiation degree information of the tumor in the pathological image and the tumor location and tumor morphological change information in the MRI image. Therefore, by performing classification based on the multimodal fused features, a tumor risk prediction result can be obtained, which can more accurately indicate the risk level of the tumor, greatly improving the efficiency and accuracy of medical diagnosis.
[0175] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0176] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.
[0177] The disclosed embodiment also proposes a tumor risk prediction device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.
[0178] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0179] Fig. 91 is a block diagram of a device 1900 for tumor risk prediction according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Fig. 9 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0180] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.
[0181] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.
[0182] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0183] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0184] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0185] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0186] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0187] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0188] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0189] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0190] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for predicting tumor risk, characterized in that: The method comprises: Extracting features based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature; Extracting features based on the pathological image containing the target tumor location to obtain a second image feature; Performing multimodal feature fusion on the first image feature and the second image feature to obtain multimodal fused features; Classification is performed based on the features after the multimodal fusion to obtain a tumor risk prediction result, and the tumor risk prediction result is used to indicate the risk level of the tumor.
2. The method according to claim 1, characterized in that The tumor includes a glioma, and the feature extraction based on the nuclear magnetic resonance image sequence containing the target tumor position to obtain the first image feature includes: Performing gradient field correction on each nuclear magnetic image in the nuclear magnetic image sequence to obtain an image sequence after gradient field correction; Performing skull removal and cropping processing on the image after gradient field correction to obtain a processed nuclear magnetic resonance image sequence; Feature extraction is performed on each nuclear magnetic resonance image in the processed nuclear magnetic resonance image sequence to obtain the feature of each nuclear magnetic resonance image as the first image feature.
3. The method according to claim 1, characterized in that The step of extracting features based on the pathological image containing the target tumor location to obtain the second image features includes: Cutting the image corresponding to the tissue part in the pathological image into a plurality of image blocks; Feature extraction is performed on the multiple image blocks respectively, and the features corresponding to the respective image blocks are obtained as the second image features.
4. According to the method of claim 1, the performing multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature comprises: Processing the first image feature based on a first linear mapping parameter to obtain a query feature; Processing the second image feature based on a second linear mapping parameter to obtain a key feature; Processing the second image feature based on a third linear mapping parameter to obtain a value feature; Based on the query feature and the key feature, calculating an attention score, wherein the attention score represents the degree of correlation between the image blocks of the nuclear magnetic resonance image sequence and the pathological image; Based on the attention score and the value feature, the mutual attention value between the query feature and the key feature and the value feature is calculated as the feature after the multimodal fusion.
5. The method according to claim 4, characterized in that The classification based on the multimodal fused features to obtain a tumor risk prediction result includes: Inputting the multimodal fused features and classification labels into a deep learning neural network to obtain an output vector of the classification labels, wherein the output vector of the classification labels is a semantic representation of the multimodal fused features; The output vector of the classification mark is input into the fully connected layer classifier to obtain a tumor risk prediction result, which is a binary classification result.
6. The method according to claim 4, characterized in that The method further comprises: Inputting the multimodal fused features and classification labels into a deep learning neural network to obtain attention weights; Selecting a preset number of nuclear magnetic resonance images in the nuclear magnetic resonance image sequence in a descending order of the attention weights, wherein the preset number of nuclear magnetic resonance images are nuclear magnetic resonance images containing tumors; The preset number of nuclear magnetic resonance images is used as the tumor localization result of the nuclear magnetic resonance image sequence.
7. The method according to claim 4, characterized in that The method further comprises: Based on the attention score, a preset number of pathological images corresponding to the nuclear magnetic resonance images are selected for visualization processing to obtain lesion areas of the pathological images.
8. A tumor risk prediction device, characterized in that: The device comprises: A first feature extraction module, configured to extract features based on a nuclear magnetic resonance image sequence containing a target tumor location to obtain a first image feature; A second feature extraction module, used to extract features based on the pathological image containing the target tumor location to obtain a second image feature; A multimodal feature fusion module, used to perform multimodal feature fusion on the first image feature and the second image feature to obtain a multimodal fused feature; A classification module is used to perform classification based on the features after the multimodal fusion to obtain a tumor risk prediction result, and the tumor risk prediction result is used to indicate the risk level of the tumor.
9. A tumor risk prediction device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.
10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Bladder cancer lymph node metastasis prediction method, device and equipment and storage medium
CN120013952A
Bladder Cancer Lymph Node Metastasis Prediction Method, Device, Equipment and Storage Medium
CN120013952B
Device for cerebral arterial thrombosis prognosis prediction and storage medium
CN120932045A
