A multimodal pre-training method, device and apparatus for cancer survival prediction

Abstract

The present application discloses a multimodal pre-training method, device and equipment for cancer survival prediction, which relates to the field of artificial intelligence, including: obtaining pathological images and multi-omics data of patient samples, and pre-processing them respectively; encoding the processed pathological images and processed multi-omics data respectively to obtain a first code and a second code; determining the contrast loss based on the first code and the second code, inputting the second code into a preset self-attention module to obtain a self-attention value, performing cross-attention calculation based on the self-attention value and the first code, and determining the target code according to the calculation result; determining the cross-entropy loss based on the target code and the matching result between the pathological image and the multi-omics data, back-propagating the contrast loss and the cross-entropy loss to optimize the model parameters, and completing the pre-training of the model based on the optimized model parameters. The present application can fully interact multimodal information and improve the accuracy of cancer survival prediction tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multimodal pre-training method, device and equipment for cancer survival prediction. Background Art

[0002] In order to provide cancer patients with the best treatment options and a better quality of life, clinicians need to accurately predict the survival rate of cancer patients. In recent years, multimodal integration methods based on histopathological images and multi-omics data have made certain progress in cancer survival prediction. However, the existing multimodal methods ignore the alignment of information between the two modalities before integrating pathological images and multi-omics data, that is, making the pathological images and multi-omics data of the same patient as close as possible in the sample space, while making the pathological images and multi-omics data of different patients as far apart as possible in the sample space. Therefore, the quality of the multimodal encoding obtained by directly integrating multimodal information without alignment is not high, resulting in the accuracy of cancer survival prediction still needs to be further improved. How to improve the accuracy of cancer survival prediction has become a problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal pre-training method, device and equipment for cancer survival prediction, which can make full use of multimodal information and improve the accuracy of cancer survival prediction tasks. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a multimodal pre-training method for cancer survival prediction, comprising:

[0005] Acquiring a pathological image and multi-omics data of a patient sample, and preprocessing the pathological image and the multi-omics data respectively to obtain a processed pathological image and processed multi-omics data;

[0006] Encoding the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data;

[0007] Determine a contrast loss based on the first code and the second code, input the second code into a preset self-attention module to obtain a self-attention value, perform a cross-attention calculation based on the self-attention value and the first code, and determine a target code according to the corresponding calculation result;

[0008] The cross entropy loss is determined based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, and the model parameters are optimized by backpropagation according to the contrast loss and the cross entropy loss, and the model pre-training is completed based on the optimized model parameters.

[0009] Optionally, the preprocessing of the pathological image and the multi-omics data respectively to obtain a processed pathological image and processed multi-omics data includes:

[0010] Using the CLAM algorithm to identify the pathological image, and extracting the target high-value area from the corresponding identified pathological image;

[0011] Cutting the target high-value area into a plurality of slices of target size and without overlapping to obtain a processed pathological image;

[0012] Calculating the variance value of each feature or gene expression in the multi-omics data based on each omics type corresponding to the multi-omics data, and filtering the features or genes with the variance value of 0 to determine filtered data;

[0013] Perform differential expression analysis based on the filtered data to determine analysis results, so as to retain features or genes that meet preset conditions based on the analysis results, so as to determine retained features or genes;

[0014] The importance of the retained features or genes is calculated using a random survival forest algorithm, and the corresponding calculation results are sorted according to a rule from large to small to obtain sorted results;

[0015] Features or genes with preset rankings are extracted from the sorted results, so as to determine processed multi-omics data based on the corresponding extracted results.

[0016] Optionally, the using corresponding encoders to encode the processed pathological image and the processed multi-omics data respectively to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data includes:

[0017] Inputting a plurality of the slices into a target visual Transformer in sequence to obtain a first code corresponding to the processed pathological image; wherein the target visual Transformer includes a self-attention mechanism and a feedforward neural network;

[0018] splicing the processed multi-omics data with predetermined CLS nodes to obtain a target sequence;

[0019] The target sequence is encoded using a target fully connected neural network and the self-attention mechanism respectively to obtain a second encoding corresponding to the processed multi-omics data.

[0020] Optionally, determining the contrast loss based on the first encoding and the second encoding includes:

[0021] Inputting the first code into the global attention pooling module for aggregation processing to determine the pathology map code according to the corresponding aggregation result;

[0022] Selecting a code corresponding to a predetermined CLS node from the second code to determine a multi-omics code according to the corresponding selected code;

[0023] The contrast loss is determined based on the pathology map code and the multi-omics code.

[0024] Optionally, performing cross attention calculation based on the self-attention value and the first code, and determining the target code according to the corresponding calculation result, includes:

[0025] Performing a cross-attention calculation based on the self-attention value and the first encoding to obtain a calculation result;

[0026] Inputting the calculation result into a feedforward neural network for processing to obtain an output code;

[0027] A code corresponding to a predetermined CLS node is selected from the output codes, and a target code is determined according to the corresponding selection result.

[0028] Optionally, before determining the cross entropy loss based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, the method further includes:

[0029] Determining whether the patient corresponding to the pathological image is the same as the patient corresponding to the multi-omics data;

[0030] If so, determining that the pathological image matches the multi-omics data;

[0031] If not, it is determined that the pathological image does not match the multi-omics data.

[0032] Optionally, determining the cross entropy loss based on the target encoding and a predetermined matching result between the pathological image and the multi-omics data includes:

[0033] Inputting the target code into a target fully connected neural network to obtain a target logical operator;

[0034] A cross entropy loss is determined based on the target logical operator and a matching result between the pathological image and the multi-omics data.

[0035] Optionally, the back propagating according to the contrast loss and the cross entropy loss to optimize the model parameters includes:

[0036] Adding the contrast loss and the cross entropy loss to determine a total loss according to the corresponding addition results;

[0037] The model parameters are optimized based on the total loss.

[0038] Optionally, after back-propagating the contrast loss and the cross entropy loss to optimize the model parameters, the method further includes:

[0039] Inputting the target code into a target fully connected neural network to obtain a target survival risk value;

[0040] Obtaining the survival time and death events of the patient samples;

[0041] Inputting the target survival risk value, the survival time, and the death event into a preset average negative log-likelihood loss function to obtain a target output;

[0042] The target output is back-propagated to optimize the model parameters, and the model is adjusted based on the currently optimized model parameters.

[0043] In a second aspect, the present application discloses a multimodal pre-training device for cancer survival prediction, comprising:

[0044] An image and data processing module, used for acquiring pathological images and multi-omics data of patient samples, and preprocessing the pathological images and the multi-omics data respectively to obtain processed pathological images and processed multi-omics data;

[0045] An encoding module, used to encode the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data;

[0046] A self-attention value acquisition module, used to determine the contrast loss based on the first code and the second code, and input the second code into a preset self-attention module to obtain a self-attention value;

[0047] A target encoding determination module, configured to perform a cross-attention calculation based on the self-attention value and the first encoding, and determine a target encoding according to a corresponding calculation result;

[0048] A model parameter optimization module is used to determine the cross entropy loss based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, back-propagate the contrast loss and the cross entropy loss to optimize the model parameters, and complete the pre-training of the model based on the optimized model parameters.

[0049] In a third aspect, the present application discloses an electronic device, comprising:

[0050] Memory, used to store computer programs;

[0051] A processor is used to execute the computer program to implement the aforementioned multimodal pre-training method for cancer survival prediction.

[0052] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned multimodal pre-training method for cancer survival prediction.

[0053] In the present application, when performing multimodal pre-training for cancer survival prediction, the pathological image and multi-omics data of the patient sample are first obtained, and the pathological image and the multi-omics data are pre-processed respectively to obtain a processed pathological image and the processed multi-omics data; then the processed pathological image and the processed multi-omics data are encoded respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data; then the contrast loss is determined based on the first code and the second code, the second code is input into a preset self-attention module to obtain a self-attention value, and a cross-attention calculation is performed based on the self-attention value and the first code, and a target code is determined according to the corresponding calculation result; finally, the cross-entropy loss is determined based on the target code and the predetermined matching result between the pathological image and the multi-omics data, and the model parameters are optimized by back-propagation according to the contrast loss and the cross-entropy loss, and the model pre-training is completed based on the optimized model parameters. It can be seen that the present application achieves the alignment of information between the two modalities by determining the contrast loss between the pathological images and the multi-omics data, and combines the matching results between the pathological images and the multi-omics data to enable full interaction of multimodal information and improve the accuracy of downstream cancer survival prediction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 A flow chart of a multimodal pre-training method for cancer survival prediction disclosed in this application;

[0056] Figure 2 A schematic diagram of the structure of a pathology-omics multimodal pre-training model disclosed in this application;

[0057] Figure 3 A schematic diagram of the structure of a pathology-omics multimodal fine-tuning model disclosed in this application;

[0058] Figure 4 This is a schematic diagram of the structure of a multimodal pre-training device for cancer survival prediction disclosed in this application;

[0059] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Existing multimodal methods ignore the alignment of information between the two modalities before integrating pathological images and multi-omics data, that is, making the pathological images and multi-omics data of the same patient as close as possible in the sample space, while making the pathological images and multi-omics data of different patients as far apart as possible in the sample space. Therefore, the quality of the multimodal encoding obtained by directly integrating multimodal information without alignment is not high, resulting in the accuracy of cancer survival prediction still needs to be further improved. In order to solve the above technical problems, the present application discloses a multimodal pre-training method for cancer survival prediction, which can achieve the alignment of information between two modalities and improve the accuracy of downstream cancer survival prediction tasks.

[0062] See also Figure 1 As shown, an embodiment of the present invention discloses a multimodal pre-training method for cancer survival prediction, comprising:

[0063] Step S11, obtaining a pathological image and multi-omics data of a patient sample, and preprocessing the pathological image and the multi-omics data respectively to obtain a processed pathological image and processed multi-omics data.

[0064] In this embodiment, clinical data of a patient sample is first given, including pathological images, multi-omics data, survival time, death events, etc. Then, the pathological image is preprocessed, and the high-value area of ​​the pathological image is first identified and extracted using the CLAM algorithm, and the background area is filtered; the high-value area is cut into N slices of size 256x 256 and non-overlapping. At the same time, the multi-omics data is processed, and the variance value of each feature or gene expression in the multi-omics data is calculated based on each omics type corresponding to the multi-omics data, and the features or genes with the variance value of 0 are filtered to determine the filtered data; differential expression analysis is performed based on the filtered data to determine the analysis results, so that the features or genes that meet the preset conditions are retained based on the analysis results to determine the retained features or genes; the importance of the retained features or genes is calculated using the random survival forest algorithm, and the corresponding calculation results are sorted according to the rule from large to small to obtain the sorted results; the features or genes with preset positions are extracted from the sorted results, so as to determine the processed multi-omics data based on the corresponding extracted results. In a specific embodiment, finally, each omics extracts the top 300 genes, and the dimension of the output sequence of the multi-omics image preprocessing module is 3x 300.

[0065] Step S12: Encode the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data.

[0066] In this embodiment, Figure 2 As shown in the figure, using the pathology image encoder, N slices are input one by one into the 6-layer visual Transformer, each layer has two submodules, namely self-attention and feedforward neural network. Finally, the pathology image outputs the encoding corresponding to N slices, that is, the first encoding corresponding to the processed pathology image. When encoding the multi-omics data using the multi-omics encoder, first add a [CLS] (Concept Learning System, an early decision tree learning algorithm) node of size 1x 300 and fill it with random initial values, and splice the [CLS] node with the three nodes of multi-omics to form a sequence of dimension 4x 300. Then, it is encoded by the fully connected neural network and self-attention respectively, and finally the encoding results corresponding to the 4 nodes are output, so as to obtain the second encoding corresponding to the processed multi-omics data.

[0067] Step S13, determine the contrast loss based on the first code and the second code, input the second code into a preset self-attention module to obtain a self-attention value, perform cross-attention calculation based on the self-attention value and the first code, and determine the target code according to the corresponding calculation results.

[0068] In this embodiment, before the pathological image and multi-omics data are integrated, a pathological-omics comparative learning task is introduced for alignment. In a specific embodiment, the comparative learning task includes: inputting the encoding of N slices into the global attention pooling module, and outputting the encoding representing the entire pathological image after aggregation, with a dimension of 1x 300. For the multi-omics encoding, only the encoding corresponding to the [CLS] node is selected, representing the encoding of the entire multi-omics modality, with a dimension of 1x 300. According to the output pathological image encoding and multi-omics encoding, the pathological-omics comparative loss L1 is calculated. In this way, the information between the two modalities is aligned before the pathological image and multi-omics data are integrated.

[0069] Afterwards, the second code is input into a preset self-attention module to obtain a self-attention value.

[0070] Specifically, the multimodal encoder consists of 6 layers of networks, each of which includes the following 3 submodules:

[0071] 1. Input the multi-omics codes of the four nodes output in step S12 into the self-attention submodule and output the self-attention value;

[0072] 2. Perform cross-attention calculation on the output self-attention value and the encoding of N slices, where the output self-attention value is set as query and the encoding of N slices is set as key and value;

[0073] 3. The result of the cross-attention calculation is passed through a feedforward neural network and finally the output of the multimodal encoder is encoded with a dimension of 4x 300.

[0074] Then, the code corresponding to the predetermined CLS node is selected from the output code, and the target code is determined according to the corresponding selection result as the final code for the fusion of pathological images and multi-omics data.

[0075] Step S14, determining the cross entropy loss based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, back-propagating the contrast loss and the cross entropy loss to optimize the model parameters, and completing the pre-training of the model based on the optimized model parameters.

[0076] In this embodiment, before determining the cross entropy loss based on the target code and the predetermined matching result between the pathological image and the multi-omics data, it is determined whether the patient corresponding to the pathological image is the same as the patient corresponding to the multi-omics data; if so, it is determined that the pathological image matches the multi-omics data; if not, it is determined that the pathological image does not match the multi-omics data. The target code is input into the target fully connected neural network to obtain a target logical operator; the cross entropy loss is determined based on the target logical operator and the matching result between the pathological image and the multi-omics data. The contrast loss and the cross entropy loss are added to determine the total loss according to the corresponding addition results; the model parameters are optimized based on the total loss, that is, the total loss is taken as the optimization target, and the model parameters are optimized by back propagation.

[0077] The above steps have now completed the pre-training of the model, and the model needs to be fine-tuned afterwards, wherein the target code is input into the target fully connected neural network to obtain the target survival risk value; the survival time and death events of the patient samples are obtained; the target survival risk value, the survival time and the death events are input into the preset average negative log-likelihood loss function to obtain the target output; based on the target output, backpropagation is performed to optimize the model parameters, and the model is adjusted based on the current optimized model parameters, thereby improving the accuracy of the downstream cancer survival prediction task.

[0078] In summary, when performing multimodal pre-training for cancer survival prediction, the present application first obtains pathological images and multi-omics data of patient samples, and pre-processes the pathological images and the multi-omics data respectively to obtain processed pathological images and processed multi-omics data; then uses corresponding encoders to encode the processed pathological images and the processed multi-omics data respectively to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data; then determines the contrast loss based on the first code and the second code, inputs the second code into a preset self-attention module to obtain a self-attention value, and performs a cross-attention calculation based on the self-attention value and the first code, and determines the target code according to the corresponding calculation results; finally, determines the cross-entropy loss based on the target code and the predetermined matching result between the pathological image and the multi-omics data, back-propagates according to the contrast loss and the cross-entropy loss to optimize the model parameters, and completes the pre-training of the model based on the optimized model parameters. It can be seen that the present application realizes the alignment of information between the two modalities through a series of processing of pathological images and multi-omics data, and combines the matching results between pathological images and the multi-omics data to enable full interaction of multimodal information and improve the accuracy of downstream cancer survival prediction tasks.

[0079] Based on the previous embodiment, it can be seen that after implementing multimodal pre-training, the present application will enter the model fine-tuning stage to further optimize the model. Next, the specific implementation process of model fine-tuning will be described in detail.

[0080] Since the model fine-tuning stage only cares about downstream tasks, the upstream tasks of the pre-trained model should be removed, so the model fine-tuning stage is different from the model pre-training process. Figure 3 As shown, firstly, clinical data of a patient sample is given, including pathological images, multi-omics data, survival time, death events, etc. Then, the pathological image preprocessing module includes two sub-steps: first, the CLAM algorithm is used to identify and extract high-value areas of the pathological image, and the background area is filtered; the high-value area is cut into N slices of size 256x256 and non-overlapping. At the same time, the multi-omics data preprocessing module is synchronized with the pathological image preprocessing module, including three sub-modules: for each omics type, the variance value of each feature / gene expression is calculated, and the gene with filtering method 0 is filtered. Differential expression analysis is performed, and only genes with large differences are retained; the importance of each feature / gene is calculated using the random survival forest algorithm and sorted from large to small. Finally, in a specific embodiment, each omics extracts the top 300 genes. It can also be the top 400, and the number here is not specifically limited, and it can be determined according to the actual situation.

[0081] After that, the pathological images and multi-omics data are encoded. Pathological image encoder: Input N slices one by one into the 6-layer visual Transformer. Each layer has two submodules, namely self-attention and feedforward neural network. Finally, the pathological image outputs the encoding corresponding to N slices. Multi-omics encoder: First, add a [CLS] node of size 1x 300 and fill it with random initial values. Splice the [CLS] node with the three nodes of multi-omics to form a sequence of dimension 4x 300. Then encode it through the fully connected neural network and self-attention respectively, and finally output the encoding results corresponding to the 4 nodes. Then input the multi-omics encoding of the output 4 nodes into the self-attention submodule, and output the self-attention value. The output self-attention value and the encoding of the N slices are cross-attention calculated, where the output self-attention value is set as query, and the encoding of the N slices is set as key and value. Finally, the output after the cross-attention calculation is passed through the feedforward neural network. The output encoding of the multimodal encoder has a dimension of 4x300, and then the multimodal encoding corresponding to the [CLS] node is selected as the final encoding for the fusion of pathological images and multi-omics data.

[0082] The final code of the output that can represent the integration of multimodal data is passed through a layer of fully connected neural network to obtain the target survival risk value; and the survival time and death events of the patient samples are obtained; the target survival risk value, the survival time and the death event are input into the preset average negative log-likelihood loss function to obtain the target output; finally, backpropagation is performed based on the target output to optimize the model parameters, and the model is adjusted based on the current optimized model parameters.

[0083] In this way, this application further optimizes the model, thereby improving the accuracy of downstream cancer survival prediction tasks.

[0084] See also Figure 4 As shown, an embodiment of the present invention discloses a multimodal pre-training device for cancer survival prediction, comprising:

[0085] An image and data processing module 11, used for acquiring pathological images and multi-omics data of patient samples, and preprocessing the pathological images and the multi-omics data respectively to obtain processed pathological images and processed multi-omics data;

[0086] An encoding module 12, configured to encode the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data;

[0087] A self-attention value acquisition module 13, used to determine the contrast loss based on the first code and the second code, and input the second code into a preset self-attention module to obtain a self-attention value;

[0088] A target coding determination module 14 is used to perform a cross attention calculation based on the self-attention value and the first coding, and determine a target coding according to a corresponding calculation result;

[0089] The model parameter optimization module 15 is used to determine the cross entropy loss based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, back-propagate the contrast loss and the cross entropy loss to optimize the model parameters, and complete the pre-training of the model based on the optimized model parameters.

[0090] In summary, when performing multimodal pre-training for cancer survival prediction, the present application first obtains pathological images and multi-omics data of patient samples, and pre-processes the pathological images and the multi-omics data respectively to obtain processed pathological images and processed multi-omics data; then uses corresponding encoders to encode the processed pathological images and the processed multi-omics data respectively to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data; then determines the contrast loss based on the first code and the second code, inputs the second code into a preset self-attention module to obtain a self-attention value, and performs a cross-attention calculation based on the self-attention value and the first code, and determines the target code according to the corresponding calculation results; finally, determines the cross-entropy loss based on the target code and the predetermined matching result between the pathological image and the multi-omics data, back-propagates according to the contrast loss and the cross-entropy loss to optimize the model parameters, and completes the pre-training of the model based on the optimized model parameters. It can be seen that the present application realizes the alignment of information between the two modalities through a series of processing of pathological images and multi-omics data, and combines the matching results between pathological images and the multi-omics data to enable full interaction of multimodal information and improve the accuracy of downstream cancer survival prediction tasks.

[0091] In some specific embodiments, the image and data processing module 11 may be specifically used to recognize the pathological image using the CLAM algorithm, and extract the target high-value area from the corresponding recognized pathological image;

[0092] Cutting the target high-value area into a plurality of slices of target size and without overlapping to obtain a processed pathological image;

[0093] Calculating the variance value of each feature or gene expression in the multi-omics data based on each omics type corresponding to the multi-omics data, and filtering the features or genes with the variance value of 0 to determine filtered data;

[0094] Perform differential expression analysis based on the filtered data to determine analysis results, so as to retain features or genes that meet preset conditions based on the analysis results, so as to determine retained features or genes;

[0095] The importance of the retained features or genes is calculated using a random survival forest algorithm, and the corresponding calculation results are sorted according to a rule from large to small to obtain sorted results;

[0096] Features or genes with preset rankings are extracted from the sorted results, so as to determine processed multi-omics data based on the corresponding extracted results.

[0097] In some specific embodiments, the encoding module 12 may specifically include:

[0098] A first code acquisition unit, used for sequentially inputting the plurality of slices into a target visual Transformer to obtain a first code corresponding to the processed pathological image; wherein the target visual Transformer includes a self-attention mechanism and a feedforward neural network;

[0099] A target sequence acquisition unit, used for splicing the processed multi-omics data with a predetermined CLS node to obtain a target sequence;

[0100] The second encoding acquisition unit is used to encode the target sequence using the target fully connected neural network and the self-attention mechanism respectively to obtain a second encoding corresponding to the processed multi-omics data.

[0101] In some specific embodiments, the self-attention value acquisition module 13 may specifically include:

[0102] A pathology image code determination unit, used for inputting the first code into a global attention pooling module for aggregation processing, so as to determine the pathology image code according to the corresponding aggregation result;

[0103] a multi-omics code determination unit, configured to select a code corresponding to a predetermined CLS node from the second code, so as to determine the multi-omics code according to the corresponding selected code;

[0104] A contrast loss determination unit is used to determine the contrast loss based on the pathology image code and the multi-omics code.

[0105] In some specific embodiments, the target encoding determination module 14 may specifically include:

[0106] A calculation result acquisition unit, used for performing a cross attention calculation based on the self-attention value and the first code to obtain a calculation result;

[0107] An output code acquisition unit, used for inputting the calculation result into a feedforward neural network for processing to obtain an output code;

[0108] The target code determination unit is used to select a code corresponding to a predetermined CLS node from the output code, and determine the target code according to the corresponding selection result.

[0109] In some specific embodiments, the device may further include:

[0110] A determination module, used to determine whether the patient corresponding to the pathological image is the same as the patient corresponding to the multi-omics data;

[0111] a first determination result execution module, configured to determine that the pathological image matches the multi-omics data;

[0112] The second determination result execution module is used to determine that the pathological image does not match the multi-omics data if no.

[0113] In some specific embodiments, the model parameter optimization module 15 may specifically include:

[0114] A target logic operator acquisition unit, used for inputting the target code into a target fully connected neural network to obtain a target logic operator;

[0115] A cross entropy loss determination unit is used to determine the cross entropy loss based on the target logical operator and the matching result between the pathological image and the multi-omics data.

[0116] In some specific embodiments, the model parameter optimization module 15 may specifically include:

[0117] a total loss determining unit, configured to add the contrast loss and the cross entropy loss to determine a total loss according to corresponding addition results;

[0118] A model parameter optimization unit is used to optimize the model parameters based on the total loss back propagation.

[0119] In some specific embodiments, the device may further include:

[0120] A target survival risk value acquisition module, used for inputting the target code into the target fully connected neural network to obtain a target survival risk value;

[0121] A survival time and death event acquisition module, used to obtain the survival time and death event of the patient sample;

[0122] A target output acquisition module, used to input the target survival risk value, the survival time and the death event into a preset average negative log-likelihood loss function to obtain a target output;

[0123] The model adjustment module is used to optimize the model parameters based on the target output back propagation, and complete the adjustment of the model based on the current optimized model parameters.

[0124] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0125] Figure 5A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multimodal pre-training method for cancer survival prediction disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0126] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0127] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0128] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the multimodal pre-training method for cancer survival prediction performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0129] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the multimodal pre-training method for cancer survival prediction disclosed above is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0130] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0131] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0133] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0134] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal pre-training method for cancer survival prediction, characterized in that: include: Acquiring a pathological image and multi-omics data of a patient sample, and preprocessing the pathological image and the multi-omics data respectively to obtain a processed pathological image and processed multi-omics data; Encoding the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data; Determine a contrast loss based on the first code and the second code, input the second code into a preset self-attention module to obtain a self-attention value, perform a cross-attention calculation based on the self-attention value and the first code, and determine a target code according to the corresponding calculation result; Determine a cross entropy loss based on the target encoding and a predetermined matching result between the pathological image and the multi-omics data, perform back propagation to optimize model parameters according to the contrast loss and the cross entropy loss, and complete pre-training of the model based on the optimized model parameters; The method of encoding the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data includes: Inputting the processed pathological images into a target visual Transformer in sequence to obtain a first code corresponding to the processed pathological images; wherein the target visual Transformer includes a self-attention mechanism and a feedforward neural network; splicing the processed multi-omics data with predetermined CLS nodes to obtain a target sequence; Encoding the target sequence using a target fully connected neural network and the self-attention mechanism respectively to obtain a second encoding corresponding to the processed multi-omics data; The determining of the contrast loss based on the first encoding and the second encoding includes: Inputting the first code into the global attention pooling module for aggregation processing to determine the pathology map code according to the corresponding aggregation result; Selecting a code corresponding to a predetermined CLS node from the second code to determine a multi-omics code according to the corresponding selected code; determining the contrast loss based on the pathology map code and the multi-omics code; The performing cross attention calculation based on the self-attention value and the first code, and determining the target code according to the corresponding calculation result, includes: Performing a cross-attention calculation based on the self-attention value and the first encoding to obtain a calculation result; Inputting the calculation result into a feedforward neural network for processing to obtain an output code; A code corresponding to a predetermined CLS node is selected from the output codes, and a target code is determined according to the corresponding selection result.

2. The multimodal pre-training method for cancer survival prediction according to claim 1, characterized in that: The preprocessing of the pathological image and the multi-omics data respectively to obtain the processed pathological image and the processed multi-omics data comprises: Using the CLAM algorithm to identify the pathological image, and extracting the target high-value area from the corresponding identified pathological image; Cutting the target high-value area into a plurality of slices of target size and without overlapping to obtain a processed pathological image; Calculating the variance value of each feature or gene expression in the multi-omics data based on each omics type corresponding to the multi-omics data, and filtering the features or genes with the variance value of 0 to determine filtered data; Perform differential expression analysis based on the filtered data to determine analysis results, so as to retain features or genes that meet preset conditions based on the analysis results, so as to determine retained features or genes; The importance of the retained features or genes is calculated using a random survival forest algorithm, and the corresponding calculation results are sorted according to a rule from large to small to obtain sorted results; Features or genes with preset rankings are extracted from the sorted results, so as to determine processed multi-omics data based on the corresponding extracted results.

3. The multimodal pre-training method for cancer survival prediction according to claim 1, characterized in that: Before determining the cross entropy loss based on the target encoding and the predetermined matching result between the pathological image and the multi-omics data, the method further includes: Determining whether the patient corresponding to the pathological image is the same as the patient corresponding to the multi-omics data; If so, determining that the pathological image matches the multi-omics data; If not, it is determined that the pathological image does not match the multi-omics data.

4. The multimodal pre-training method for cancer survival prediction according to claim 1, characterized in that: The determining of the cross entropy loss based on the target encoding and a predetermined matching result between the pathological image and the multi-omics data comprises: Inputting the target code into a target fully connected neural network to obtain a target logical operator; A cross entropy loss is determined based on the target logical operator and a matching result between the pathological image and the multi-omics data.

5. The multimodal pre-training method for cancer survival prediction according to claim 1, characterized in that: The back propagation according to the contrast loss and the cross entropy loss to optimize the model parameters includes: Adding the contrast loss and the cross entropy loss to determine a total loss according to the corresponding addition results; The model parameters are optimized based on the total loss.

6. The multimodal pre-training method for cancer survival prediction according to any one of claims 1 to 5, characterized in that: After back propagating the contrast loss and the cross entropy loss to optimize the model parameters, the method further includes: Inputting the target code into a target fully connected neural network to obtain a target survival risk value; Obtaining the survival time and death events of the patient samples; Inputting the target survival risk value, the survival time, and the death event into a preset average negative log-likelihood loss function to obtain a target output; The target output is back-propagated to optimize the model parameters, and the model is adjusted based on the currently optimized model parameters.

7. A multimodal pre-training device for cancer survival prediction, characterized in that: include: An image and data processing module, used for acquiring pathological images and multi-omics data of patient samples, and preprocessing the pathological images and the multi-omics data respectively to obtain processed pathological images and processed multi-omics data; An encoding module, used to encode the processed pathological image and the processed multi-omics data respectively using corresponding encoders to obtain a first code corresponding to the processed pathological image and a second code corresponding to the processed multi-omics data; A self-attention value acquisition module, used to determine the contrast loss based on the first code and the second code, and input the second code into a preset self-attention module to obtain a self-attention value; A target encoding determination module, configured to perform a cross-attention calculation based on the self-attention value and the first encoding, and determine a target encoding according to a corresponding calculation result; A model parameter optimization module, configured to determine a cross entropy loss based on the target encoding and a predetermined matching result between the pathological image and the multi-omics data, perform back propagation to optimize the model parameters according to the contrast loss and the cross entropy loss, and complete the pre-training of the model based on the optimized model parameters; The encoding module is further used to sequentially input the processed pathological images into the target visual Transformer to obtain a first encoding corresponding to the processed pathological images; wherein the target visual Transformer includes a self-attention mechanism and a feedforward neural network; the processed multi-omics data is spliced ​​with a predetermined CLS node to obtain a target sequence; the target sequence is encoded using a target fully connected neural network and the self-attention mechanism respectively to obtain a second encoding corresponding to the processed multi-omics data; The self-attention value acquisition module is further used to input the first code into the global attention pooling module for aggregation processing, so as to determine the pathology map code according to the corresponding aggregation result; select the code corresponding to the predetermined CLS node from the second code, so as to determine the multi-omics code according to the corresponding selected code; determine the contrast loss based on the pathology map code and the multi-omics code; Among them, the target coding determination module is used to perform cross-attention calculation based on the self-attention value and the first coding to obtain a calculation result; input the calculation result into a feedforward neural network for processing to obtain an output coding; select the coding corresponding to the predetermined CLS node from the output coding, and determine the target coding according to the corresponding selection result.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the multimodal pre-training method for cancer survival prediction as described in any one of claims 1 to 6.

Images