Tumor prognosis prediction model training method, and tumor prognosis prediction method and device

Abstract

The invention provides a tumor prognosis prediction model training method, a tumor prognosis prediction method and a tumor prognosis prediction device. The tumor prognosis prediction model training method comprises the following steps: inputting normalized image data of each piece of full-section pathological image data into an auto-encoder to obtain each piece of target feature map data; and combining each target feature map data with the overall lifetime data and the survival state data to obtain each data sample, and training a multi-instance learning model based on an attention mechanism to obtain a tumor prognosis prediction model for predicting the lifetime corresponding to the full-slice pathological image data. The method can improve the recognition and attention accuracy of the diseased region in the model training process, can solve the problem that the number of full-slice image tags is small, can improve the effective region of the extracted pathological image, can improve the training efficiency, accuracy and reliability of the tumor prognosis prediction model, and can improve the prediction efficiency of the tumor prognosis prediction model. Therefore, the effectiveness and accuracy of the result of tumor prognosis prediction by using the tumor prognosis prediction model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of tumor prognosis prediction, and in particular to a tumor prognosis prediction model training method, a tumor prognosis prediction method and a tumor prognosis prediction device. Background Art

[0002] Tumor prognosis prediction aims to predict the risk of progression of tumor-related diseases and ultimately predict the patient's survival, providing a reliable reference for the selection of treatment plans for tumor-related patients. Tumor prognosis prediction can estimate the probability of survival of tumor-related patients within a certain period of time by comprehensively analyzing the patient's pathological images, genomic information, clinical data, and other biomarkers, combined with doctor experience, statistics, and machine learning methods. Through these predictions, doctors can better understand the patient's risk of disease progression and thus develop more personalized and targeted treatment plans for patients.

[0003] With the development of reliable full-width pathology scanners and high-performance computer vision technology, quantitative pathology analysis based on medical images has become feasible. However, the current method of using deep learning technology to predict tumor prognosis cannot process the entire tumor pathology image at once and cannot guarantee the accuracy of feature extraction when processing images containing up to billions of pixels, such as colorectal cancer pathology images. This will consume a lot of time and cost in the feature extraction stage and affect the accuracy of subsequent tumor prognosis prediction results. At the same time, tumor prognosis prediction also faces the problems of lack of training data and easy neglect of feature distribution differences between different pathology data sets, which will also lead to the inability to guarantee the accuracy and reliability of tumor prognosis prediction results.

[0004] Based on this, there is an urgent need to design a method that can improve the efficiency and accuracy of feature extraction of tumor pathology images and improve the accuracy and reliability of tumor prognosis prediction results. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a tumor prognosis prediction model training method, a tumor prognosis prediction method and a device to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present application provides a method for training a tumor prognosis prediction model, comprising:

[0007] Inputting normalized image data of each full-slice pathology image data into a preset autoencoder respectively, so as to obtain target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder;

[0008] Combining the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data, respectively, to obtain data samples corresponding to each of the full-slice pathology image data;

[0009] Each of the data samples is used to train a preset multi-instance learning model based on the attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, it constitutes a tumor prognosis prediction model.

[0010] In some embodiments of the present application, before the normalized image data of each full-slice pathological image data is respectively input into a preset autoencoder, the method further includes:

[0011] Cutting each full-slice pathological image data into a plurality of image blocks of the same size;

[0012] Performing filtering processing on each of the image blocks;

[0013] Each of the image blocks after filtering is normalized to obtain each normalized image block corresponding to each of the full-slice pathology image data, wherein each of the normalized image blocks belonging to the same full-slice pathology image data together constitutes the normalized image data corresponding to the full-slice pathology image data.

[0014] In some embodiments of the present application, the filtering process on each of the image blocks includes:

[0015] Each of the image blocks is converted from the RGB space to the optical density space, and among the image blocks converted to the optical density space, the image blocks whose optical density values ​​are less than a preset optical density threshold are eliminated.

[0016] In some embodiments of the present application, the normalizing of each of the image blocks after filtering to obtain each normalized image block corresponding to each of the full-slice pathological image data includes:

[0017] Performing singular value decomposition on each of the image blocks converted to the optical density space to obtain a eigenvector corresponding to the optical density space of each image block, then sorting the eigenvalues ​​of each eigenvector in descending order, and selecting the first two from the sorted eigenvectors to form a plane of the optical density space of the image block;

[0018] Mapping the pixel points of each of the image blocks after filtering to the plane of the image block optical density space and normalizing them to a preset unit length;

[0019] Determine the angle of each pixel point on the plane of the image block optical density space relative to a preset direction, and search for an extreme angle among the angles;

[0020] Each of the extreme value angles is converted back to the optical density space to obtain normalized image data corresponding to each of the normalized image blocks.

[0021] In some embodiments of the present application, the autoencoder includes an encoder and a decoder connected to each other;

[0022] Correspondingly, the normalized image data of each full-slice pathology image data are respectively input into a preset autoencoder to obtain the target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder, including:

[0023] Inputting each of the normalized image blocks into the encoder respectively, so that the encoder maps each of the normalized image blocks into a preset feature space respectively, and enables the decoder to perform feature reconstruction on each of the normalized images mapped into the feature space respectively, so as to obtain feature reconstruction image data corresponding to each of the normalized image blocks;

[0024] Respectively splicing the characteristic reconstruction map data corresponding to the same full-slice pathology image data to obtain target characteristic map data corresponding to each full-slice pathology image data;

[0025] The autoencoder is pre-trained using each normalized image sample and a mean square error loss function.

[0026] In some embodiments of the present application, the multi-instance learning model based on the attention mechanism includes: an attention pooling layer and a linear activation layer connected to each other;

[0027] Correspondingly, the preset multi-instance learning model based on the attention mechanism is trained by using each of the data samples, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitutes a tumor prognosis prediction model, including:

[0028] Input each of the data samples into the attention pooling layer respectively, so that the attention pooling layer downsamples each of the input target feature map data based on the attention mechanism, and enables the linear activation layer to obtain the survival period prediction result data corresponding to the target feature map data after downsampling based on a preset activation function, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitutes a tumor prognosis prediction model;

[0029] The survival prediction result data includes: shape parameters and scale parameters used to represent the Weibull distribution model.

[0030] The second aspect of the present application provides a method for predicting tumor prognosis, comprising:

[0031] Acquiring normalized image data of target full-slice pathological image data;

[0032] The normalized image data of the target full-slice pathology image data is input into a preset tumor prognosis prediction model so that the autoencoder in the tumor prognosis prediction model outputs target feature map data corresponding to the target full-slice pathology image data, and the target feature map data is input into the multi-instance learning model based on the attention mechanism in the tumor prognosis prediction model so that the multi-instance learning model based on the attention mechanism outputs survival prediction result data corresponding to the target full-slice pathology image data, wherein the tumor prognosis prediction model is pre-trained based on the tumor prognosis prediction model training method.

[0033] The third aspect of the present application provides a tumor prognosis prediction model training device, comprising:

[0034] An autoencoder module, used for inputting normalized image data of each full-slice pathology image data into a preset autoencoder, so as to obtain target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder;

[0035] A sample combination module, used for combining the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data, so as to obtain a data sample corresponding to each of the full-slice pathology image data;

[0036] A weakly supervised learning module is used to use each of the data samples to train a preset multi-instance learning model based on an attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitute a tumor prognosis prediction model.

[0037] The fourth aspect of the present application provides a tumor prognosis prediction device, comprising:

[0038] A normalization module, used for obtaining normalized image data of target full-slice pathological image data;

[0039] A survival prediction module is used to input the normalized image data of the target full-slice pathology image data into a preset tumor prognosis prediction model so that the autoencoder in the tumor prognosis prediction model outputs the target feature map data corresponding to the target full-slice pathology image data, and input the target feature map data into the multi-instance learning model based on the attention mechanism in the tumor prognosis prediction model so that the multi-instance learning model based on the attention mechanism outputs the survival prediction result data corresponding to the target full-slice pathology image data, wherein the tumor prognosis prediction model is pre-trained based on the tumor prognosis prediction model training method.

[0040] The fifth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the tumor prognosis prediction model training method and / or the tumor prognosis prediction method.

[0041] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tumor prognosis prediction model training method and / or the tumor prognosis prediction method.

[0042] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the tumor prognosis prediction model training method and / or the tumor prognosis prediction method.

[0043] The tumor prognosis prediction model training method provided in the present application inputs the normalized image data of each full-slice pathology image data into a preset autoencoder respectively, so as to obtain the target feature map data corresponding to each of the full-slice pathology image data based on the output of the autoencoder; the target feature map data corresponding to each of the full-slice pathology image data is respectively combined with the overall survival data and the survival status data corresponding to each of the full-slice pathology image data to obtain the data samples corresponding to each of the full-slice pathology image data; and each of the data samples is used to train a preset multi-instance learning model based on an attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the full-slice pathology image data corresponding to the target feature map data. The method can improve the recognition and attention accuracy of diseased areas during model training based on the attention mechanism, solve the problem of fewer labels in full-slice images, and link pathological features with image features extracted by machine learning models. It can also improve the effective area of ​​pathological image extraction based on normalization processing, thereby assisting the attention mechanism to quickly focus on the lesion area during model training, reduce model training time and noise interference, and improve the accuracy of the attention mechanism, thereby improving the training efficiency, accuracy and reliability of the tumor prognosis prediction model, so as to improve the effectiveness and accuracy of the results of tumor prognosis prediction using the tumor prognosis prediction model.

[0044] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0045] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0047] Figure 1 This is a schematic diagram of a first flow chart of a tumor prognosis prediction model training method in one embodiment of the present application.

[0048] Figure 2 Schematic diagram of the architecture of a tumor prognosis prediction model in one embodiment of the present application.

[0049] Figure 3 Schematic diagram of a second flow chart of a tumor prognosis prediction model training method in one embodiment of the present application.

[0050] Figure 4 This is a third flow chart of the tumor prognosis prediction model training method in one embodiment of the present application.

[0051] Figure 5 Schematic diagram of the execution logic architecture of a tumor prognosis prediction model in an application example of the present application.

[0052] Figure 6 This is a schematic diagram showing an example of comparison between normalized image blocks obtained before and after normalization of an image block in an application example of the present application.

[0053] Figure 7 This is a schematic diagram showing an example of comparison between a normalized image block in an application example of the present application and its feature reconstructed image data obtained after passing through an autoencoder.

[0054] Figure 8 Schematic diagram of the process of predicting tumor prognosis in one embodiment of the present application.

[0055] Fig. 9 Schematic diagram of the structure of a tumor prognosis prediction model training device in one embodiment of the present application.

[0056] Fig.10 Schematic diagram of the structure of a tumor prognosis prediction device in one embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0058] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0059] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0060] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0061] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0062] In order to use deep learning technology to achieve tumor prognosis prediction for tumor pathology images, the whole slice pathology image data can be cut first, and the image slices containing cancer cells can be screened out. Then, neural networks such as convolutional neural networks (CNN), fully convolutional neural networks (FCN) or ResNet networks are used to extract image slice features, and then the features are input into the downstream pooling network for prognosis prediction. The downstream pooling network is generally a simple fully connected network, and the output prognosis prediction is the parameters of the Weibull distribution or the result data used to represent the three-year or five-year survival rate.

[0063] However, in the process of executing the above-mentioned tumor prognosis prediction method, the designers of the present application found that when processing colorectal cancer pathology images containing up to billions of pixels, the mean and standard deviation of the colorectal cancer pathology images are quite different from the natural image data sets of traditional computer vision such as ImageNet. At the same time, unlike the tasks in the traditional computer vision field, typical high-resolution colorectal cancer tissue digital pathology images contain up to billions of pixels. Therefore, when directly using convolutional neural networks (CNN), fully convolutional neural networks (FCN) or ResNet networks to extract features of image data containing up to billions of pixels such as colorectal cancer pathology images, not only can the accuracy of feature extraction not be guaranteed, but the entire pathology image cannot be processed at one time. Therefore, it will consume more time and cost in the feature extraction stage and affect the accuracy of subsequent tumor prognosis prediction results.

[0064] In addition, due to the "black box" characteristics of deep learning models, it is very difficult to link the model paradigm obtained by data drive with biological information. Due to the domain gap between natural images and medical images, the extension of data-driven machine learning models to medical images remains a major challenge for many computational pathology models. In clinical practice, the lack of extensive validation in different patient populations may reduce the clinical applicability of machine learning models. When processing sliced ​​images, traditional deep neural networks use the same network and parameters for feature extraction from different regions, resulting in invalid features. In addition, existing networks usually use ResNet networks pre-trained on natural images to extract pathological image features, ignoring the differences in feature distribution between different data sets, resulting in inaccurate features and prediction results.

[0065] Based on this, in order to solve the problem that the feature extraction stage in the model training process of the existing tumor prognosis prediction method consumes a lot of time cost and affects the accuracy of the subsequent tumor prognosis prediction results, which easily leads to the accuracy and reliability of the tumor prognosis prediction results cannot be guaranteed, etc., the embodiments of the present application respectively provide a tumor prognosis prediction model training method, a tumor prognosis prediction model training device for executing the tumor prognosis prediction model training method, a tumor prognosis prediction method, a tumor prognosis prediction device for executing the tumor prognosis prediction method, an electronic device, a computer-readable storage medium and a computer program product. Different from the existing prognosis prediction deep learning method, the present application adds multiple instance learning (Multiple Instance Learning, MIL) of the attention mechanism to solve the problem of fewer whole slice image (WSI) labels. In order to identify human pathological features and associate the pathological features with the image features extracted by the machine learning model, the present application introduces an attention mechanism in the multiple instance learning model to identify and focus on the diseased area. At the same time, in response to the problem that the mean and standard deviation of colorectal cancer pathology images found by the designers of this application are quite different from natural image data sets of traditional computer vision such as ImageNet, this application uses an autoencoder (AE) to replace pre-trained convolutional neural networks for feature extraction. In addition, this application uses the normalized image data of the full-slice pathology image data as the feature extraction basis of the autoencoder, which helps to assist the attention mechanism to quickly focus on the lesion area during the training process of the tumor prognosis prediction model, which can reduce training time and noise interference, and can improve the accuracy of the attention mechanism.

[0066] The details are described in detail through the following examples.

[0067] Based on this, the present application embodiment provides a tumor prognosis prediction model training method that can be implemented by a tumor prognosis prediction model training device, see Figure 1 The tumor prognosis prediction model training method specifically includes the following contents:

[0068] Step 100: inputting normalized image data of each full-slice pathology image data into a preset autoencoder respectively, so as to obtain target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder.

[0069] In one or more embodiments of the present application, whole-slice pathological image data refers to a whole-slice image WSI (whole scanning image) of a patient with a tumor-related disease that can be acquired by a device such as a WSI scanner. It is a pathological image specifically used to display an image of a pathological slice of a patient with a tumor-related disease, and may also be referred to as a whole-slice scanning image.

[0070] It can be understood that the tumor type in the tumor prognosis prediction in the present application can be set according to the actual application requirements. Correspondingly, the present application can adopt full-slice pathology image data of various tumor types, especially when processing full-slice pathology image data containing up to billions of pixels such as colorectal cancer pathology image data, which can significantly improve the accuracy of feature extraction and processing efficiency.

[0071] In one or more embodiments of the present application, in order to improve the domain adaptability of the encoder on the training data set, the present application first uses an autoencoder (AE) to replace the pre-trained convolutional neural network for pre-trained encoding, wherein the function of the autoencoder (AE) is to perform representation learning on the input information by taking the input information as a learning target.

[0072] On this basis, step 100 of the present application also uses the normalized image data of each full-slice pathological image data as the feature extraction basis of the autoencoder, which helps the attention mechanism to quickly focus on the lesion, can reduce training time and noise interference, and can improve the accuracy of the attention mechanism.

[0073] Step 200: Combining the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data, respectively, to obtain data samples corresponding to each of the full-slice pathology image data.

[0074] In step 200, the overall survival data and survival status data corresponding to each of the full-slice pathological image data can be manually acquired in advance and sent to the tumor prognosis prediction model training device for storage. After the tumor prognosis prediction model training device acquires the target feature map data corresponding to each of the full-slice pathological image data, the target feature map data of each of the full-slice pathological image data and the pre-stored overall survival data and survival status data corresponding to the target feature map data can be combined respectively, so that each of the data samples contains a target feature map data corresponding to the full-slice pathological image data, the overall survival data (Overall Survival Time, OS) and survival status data.

[0075] It can be understood that the survival status data is used to indicate whether the tumor-related disease patient to which the corresponding full-slice pathological image data belongs has died; the overall survival data (Overall Survival Time, OS) is used to indicate the time from randomization of a tumor-related disease patient to the death of the tumor-related disease patient due to any reason, which can be simply understood as the survival time of a tumor-related disease patient after treatment.

[0076] In step 200, the overall survival time (OS) data and the survival status data can be acquired by artificial preset and sent to the tumor prognosis prediction model training device for storage. After acquiring the target feature map data corresponding to each of the full-slice pathology image data, the tumor prognosis prediction model training device can combine the target feature map data corresponding to each of the full-slice pathology image data with the overall survival time data and survival status data corresponding to the full-slice pathology image data, respectively, to obtain the data samples corresponding to each of the full-slice pathology image data. In other words, each of the data samples contains the target feature map data, overall survival time data and survival status data corresponding to a full-slice pathology image data.

[0077] Step 300: Use each of the data samples to train a preset multi-instance learning model based on the attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathology image data, and together with the autoencoder, it constitutes a tumor prognosis prediction model.

[0078] It can be understood that the Multiple Instance Learning (MIL) model is a method evolved from a supervised learning algorithm, which defines a "package" as a collection of multiple instances and has a wide range of applications. Correspondingly, in this application, each of the data samples corresponds to a package, and a data sample is a package, wherein the feature reconstruction map data constituting the target feature map data are the above-mentioned instances; the overall survival data and survival status data in each of the data samples are the labels of the target feature map data.

[0079] It should be noted that the multi-instance learning model based on the attention mechanism refers to adding an attention mechanism to the pooling layer of the multi-instance learning model to form a pooling layer based on the attention mechanism, which can be called an attention pooling layer or a gated attention layer.

[0080] In one or more embodiments of the present application, the autoencoder and the multi-instance learning model based on the attention mechanism together constitute a complete network architecture for tumor prognosis prediction, that is, a tumor prognosis prediction model, see Figure 2 .

[0081] In the process of training the multi-instance learning model based on the attention mechanism, the trained autoencoder can be directly used to directly extract the target feature map data corresponding to each of the full-slice pathology image data, or the autoencoder can be trained first, and then the target feature map data corresponding to each of the full-slice pathology image data extracted by the autoencoder can be used to train the multi-instance learning model based on the attention mechanism.

[0082] From the above description, it can be seen that the tumor prognosis prediction model training method provided in the embodiment of the present application can improve the recognition and attention accuracy of the diseased area during the model training process based on the attention mechanism, and can solve the problem of fewer labels in the whole slice image and can link the pathological features with the image features extracted by the machine learning model. It can also improve the effective area of ​​​​the extracted pathological image based on normalization processing, and then assist the attention mechanism to quickly focus on the lesion area during the model training process, and can reduce the model training time and noise interference and improve the accuracy of the attention mechanism, and then improve the training efficiency, accuracy and reliability of the tumor prognosis prediction model, so as to improve the effectiveness and accuracy of the results of tumor prognosis prediction using the tumor prognosis prediction model.

[0083] In order to further improve the application effectiveness and accuracy of the normalized image data of each full-slice pathological image data, and to further assist the attention mechanism to quickly focus on the lesion area during the training process of the tumor prognosis prediction model, in a tumor prognosis prediction model training method provided in an embodiment of the present application, see Figure 3 The tumor prognosis prediction model training method further specifically includes the following contents before step 100:

[0084] Step 010: Cut each full-slice pathology image data into multiple image blocks of the same size.

[0085] Step 020: Filter each of the image blocks.

[0086] Step 030: Normalize each of the image blocks after filtering to obtain each normalized image block corresponding to each of the full-slice pathology image data, wherein each of the normalized image blocks belonging to the same full-slice pathology image data together constitutes the normalized image data corresponding to the full-slice pathology image data.

[0087] Specifically, the tumor prognosis prediction model training device first cuts (Tiling) the input full-slice pathology image data into multiple image blocks (patches) of the same size, filters these image blocks, removes some image blocks with too low or abnormal optical density values, and then normalizes (Normalize) all the image blocks.

[0088] In order to further improve the effectiveness and reliability of filtering each of the image blocks, and to further improve the accuracy and reliability of tumor prognosis prediction model training, in a tumor prognosis prediction model training method provided in an embodiment of the present application, see Figure 4 , step 020 in the tumor prognosis prediction model training method specifically includes the following contents:

[0089] Step 021: Convert each of the image blocks from the RGB space to the optical density space, and remove the image blocks whose optical density values ​​are less than a preset optical density threshold value from the image blocks converted to the optical density space.

[0090] Specifically, the tumor prognosis prediction model training device converts each of the image blocks obtained after cutting the full-slice pathological image data from the RGB space to the optical density (OD) space. Images with optical density values ​​(i.e., OD values) less than a preset optical density threshold are removed. The calculation formula for optical density OD is as follows:

[0091] OD=-log 10 (I)

[0092] Where I is an RGB color vector normalized to [0,1].

[0093] It should be noted that RGB is a color standard that obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other.

[0094] In order to further assist the attention mechanism to quickly focus on the lesion area during the training process of the tumor prognosis prediction model, further reduce the training time and noise interference and further improve the accuracy of the attention mechanism, in a tumor prognosis prediction model training method provided in an embodiment of the present application, see Figure 4 Step 030 in the tumor prognosis prediction model training method specifically includes the following contents:

[0095] Step 031: Perform singular value decomposition on each of the image blocks converted to the optical density space to obtain the eigenvectors corresponding to the optical density space of each image block, then sort the eigenvalues ​​of each eigenvector in descending order, and select the first two from each of the sorted eigenvectors to form a plane of the image block optical density space.

[0096] The image block optical density space is a pixel value of the optical density space of the image block.

[0097] The general steps of step 031 are to dye the image RGB space, optical density space, and decompose the feature vector according to SVD to obtain the color space composed of two bases (the mathematical principle is similar to the idea of ​​dimensionality reduction by principal component analysis).

[0098] Step 032: Map the pixel points of each of the image blocks after filtering to the plane of the image block optical density space and normalize them to a preset unit length.

[0099] Step 033: Determine the angle of each pixel point on the plane of the image block optical density space relative to a preset direction, and search for an extreme angle among the angles.

[0100] Step 034: Convert each of the extreme angles back to the optical density space to obtain normalized image data corresponding to each of the normalized image blocks.

[0101] Specifically, the tumor prognosis prediction model training device maps the pixels of each of the filtered image blocks to a plane of the image block optical density space formed by the eigenvectors corresponding to the two largest eigenvalues ​​obtained by singular value decomposition (SVD). The present application projects all data into the plane of the above-mentioned image block optical density space and normalizes them to unit length. The angle of each pixel relative to the first SVD direction is then calculated, and this angle can be used to distinguish different staining components. By finding the α percentile and (100-α) percentile of the angle as robust extreme values, these extreme angles are used to represent the directions that are most likely to represent different staining components. These extreme angles are converted back to the optical density (OD) space to obtain the optimal staining vector. Finally, the entire image is regenerated, that is, the normalized image data corresponding to each of the normalized image blocks. Among them, SVD decomposition, taking the first two eigenvectors, is similar to the dimensionality reduction idea of ​​principal component analysis.

[0102] That is to say, the embodiment of the present application screens out tissue areas based on the optical density of the slices, which helps to assist the attention mechanism to quickly focus on the lesion area during the training process of the tumor prognosis prediction model, can reduce training time and noise interference, and can improve the accuracy of the attention mechanism.

[0103] In order to further improve the efficiency, extraction accuracy and effectiveness of the target feature map data corresponding to each of the full-slice pathological image data, in a tumor prognosis prediction model training method provided in an embodiment of the present application, the autoencoder includes an encoder and a decoder connected to each other; see Figure 4 , step 100 in the tumor prognosis prediction model training method specifically includes the following contents:

[0104] Step 110: Input each of the normalized image blocks into the encoder respectively, so that the encoder maps each of the normalized image blocks to a preset feature space respectively, and enables the decoder to reconstruct each of the normalized images mapped to the feature space respectively, so as to obtain feature reconstruction image data corresponding to each of the normalized image blocks.

[0105] Step 120: splicing the feature reconstruction map data corresponding to the same full-slice pathology image data respectively to obtain target feature map data corresponding to each full-slice pathology image data; wherein the autoencoder is pre-trained using each normalized image sample and a mean square error loss function.

[0106] It can be understood that the normalized image sample refers to the historical normalized image data used in the training stage of the autoencoder, and the historical normalized image data has the same data format as the normalized image data, and the extraction method can also be the same.

[0107] That is to say, the embodiment of the present application uses an autoencoder as a feature extractor, rather than a pre-trained convolutional neural network (such as ResNet50 without the last layer) to ensure the feature extraction effect on medical images. The autoencoder of the present application consists of an encoder and a decoder. The former maps an image block to the feature space required by the present application, and the latter reconstructs the features into original data. The autoencoder is trained by minimizing the error between the original data and the reconstructed data.

[0108] Encoder z=f(x)

[0109]

[0110] where x and They respectively represent the true value (i.e., the normalized image block) and the predicted value (i.e., the feature reconstruction map data) of an image block, z means the feature in the latent space, f and g are the mapping functions of the encoder and decoder, respectively.

[0111] The embodiment of the present application can use mean squared error (MSE) loss as the loss function of the autoencoder to reconstruct features. When the autoencoder performs feature learning and data reconstruction, it measures the square average of the difference between the predicted value and the true value. mse The mathematical expression is as follows:

[0112]

[0113] Where n is the number of normalized image patches, x i is the i-th normalized image block (i.e., true value), It is the feature reconstruction graph data (i.e., prediction value) of the i-th normalized image block by the autoencoder.

[0114] In order to further improve the accuracy of identifying and focusing on diseased areas during model training, solve the problem of fewer labels in full-slice images, and link pathological features with image features extracted by machine learning models, in a tumor prognosis prediction model training method provided in an embodiment of the present application, the multi-instance learning model based on the attention mechanism includes: an attention pooling layer and a linear activation layer connected to each other; see Figure 4 , step 300 in the tumor prognosis prediction model training method specifically includes the following contents:

[0115] Step 310: Input each of the data samples into the attention pooling layer respectively, so that the attention pooling layer downsamples each of the input target feature map data based on the attention mechanism, and enables the linear activation layer to obtain the survival prediction result data corresponding to the target feature map data after downsampling based on a preset activation function, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathology image data, and together with the autoencoder constitutes a tumor prognosis prediction model; wherein the survival prediction result data includes: shape parameters and scale parameters used to represent the Weibull distribution model.

[0116] Specifically, in order to effectively utilize the full-slice image features in the case of few labels. This application uses a multiple instance learning (Multiple Instance Learning, MIL) model with an attention mechanism for weakly supervised prognosis prediction. Specifically, the feature map of each image block of the entire pathological image can be spliced, and the feature can be combined with the overall survival time (Overall Survival Time, OS) and survival status (i.e., whether it is dead) of the case to form a bag of multiple instance learning, that is, target feature map data and labels. Each bag is input into the multiple instance learning model for training, and the multiple instance learning model includes an attention pooling layer and a linear activation layer and a linear activation layer based on the Sigmoid activation function. The output of the multi-instance learning model based on the attention mechanism is a shape parameter (kappa) and a scale parameter (lambda), which can be used to represent the Weibull distribution model (Weibull Module). The predicted survival of patients with tumor diseases to which the full-slice pathological image data belongs can be obtained through the multi-instance learning model based on the attention mechanism.

[0117] Specifically, the attention pooling layer is used to calculate the attention score corresponding to each of the feature reconstruction graph data in each of the data samples;

[0118] The calculation formula of the attention score is as follows:

[0119]

[0120] Wherein, z represents the attention score of the feature reconstruction graph data; h j is the feature reconstruction graph data in the data sample, j is any positive integer between 1 and K, K is the number of feature reconstruction graph data in the data sample, and K is a positive integer greater than 1; data sample H = {h1, h2, ..., h K},h1,h2,…,h K Reconstruct graph data for each of the features in the data sample respectively.

[0121] α k Represents the attention pooling formula;

[0122] The attention pooling formula is as follows:

[0123]

[0124] Wherein, w and V are weights with different values; h k is the number of samples in the data except h jAnother feature reconstruction graph data is reconstructed, k is any positive integer between 1 and K, tanh represents hyperbolic tangent, and hyperbolic tangent (tanh) is used to process element by element to include negative and positive values ​​for proper gradient flow.

[0125] To further illustrate the above embodiment, the present application also provides a specific application example of a tumor prognosis prediction model training method. The execution logic architecture of the tumor prognosis prediction model is shown in Figure 5 The specific application examples of the tumor prognosis prediction model training method include the following contents:

[0126] S1: In order to improve the domain adaptability of the encoder on the training data set, this application example first uses an autoencoder (AE) to replace the pre-trained convolutional neural network for pre-trained encoding.

[0127] The network of this application example first cuts (Tiling) the input WSI pathology image into multiple image patches of the same size, filters these image patches, removes some image patches with too low optical density values ​​or abnormalities, and then normalizes all image patches. This application example designs an image reconstruction task to train the autoencoder of this application, and each image patch is converted into a feature map after passing through the encoder.

[0128] Specifically, this application example first converts the image blocks cut from the pathological image from the RGB space to the OD (Optical Density) space, and removes images with optical density values ​​less than a certain threshold.

[0129] OD=-log 10 (I)

[0130] Where I is an RGB color vector normalized to [0,1].

[0131] The OD-converted pixels are mapped to a plane formed by the eigenvectors corresponding to the two largest eigenvalues ​​obtained by singular value decomposition (SVD). This application projects all data into the above plane and normalizes them to unit length. The angle of each point relative to the first SVD direction is then calculated, and this angle can be used to distinguish different staining components. By finding the α percentile and (100-α) percentile of the angle as robust extreme values, these extreme values ​​represent the directions that are most likely to represent different staining components. These extreme angles are converted back to OD space to obtain the optimal staining vector. Finally, the entire image is regenerated. Figure 6 The figure is a schematic diagram showing an example of comparison between a normalized image block before and after normalization.

[0132] In addition, this application example uses an autoencoder as a feature extractor instead of a pre-trained convolutional neural network (such as ResNet50 with the last layer removed) to ensure the feature extraction effect on medical images.

[0133] The autoencoder of this application example consists of an encoder (Encoder, abbreviated as En) and a decoder (Decoder, abbreviated as De). The former maps an image block to the feature space required by this application, and the latter reconstructs the features into original data. The autoencoder is trained by minimizing the error between the original data and the reconstructed data.

[0134] Encoder z=f(x)

[0135]

[0136] where x and They represent the true value and predicted value of an image block respectively, z means the feature in the latent space, f and g are the mapping functions of the encoder and decoder respectively.

[0137] This application example uses the mean squared error (MSE) loss as the reconstruction loss function, which measures the square average of the difference between the predicted value and the true value when the autoencoder performs feature learning and data reconstruction. Its mathematical expression is as follows:

[0138]

[0139] Where n is the number of samples, x i is the true value of the i-th sample, is the model's predicted value for the i-th sample.

[0140] in, Figure 7 The figure is a comparison diagram of a normalized image block and its feature reconstruction data obtained after passing through the autoencoder. In this application example, the output of the encoder is used as the low-dimensional feature of each image block.

[0141] S2: In order to effectively utilize the full-slice image features in the case of few labels, this application example uses multiple instance learning (MIL) with an attention mechanism for weakly supervised prognosis prediction.

[0142] The feature map of each image block of the entire pathological image (i.e., full-slice pathological image data) is spliced, and the feature is combined with the overall survival time (OS) and survival status (i.e., whether the patient is dead) of the case to form a bag for multi-instance learning.

[0143] Each package is input into the multi-instance learning model for training. The multi-instance learning model includes a gated attention layer and a sigmoid as an activation layer. The output of the model is a shape parameter β (kappa) and a scale parameter α (lambda). These two parameters can represent the Weibull distribution model (Weibull Module). Through this model, this application example can obtain the predicted survival period of the case as long-term survival or short-term survival.

[0144] The multi-instance learning model of this application example consists of an attention pooling layer and a linear activation layer. The introduction of the attention mechanism in the pooling layer solves the difficulty in processing ultra-high pixel medical images in terms of computational complexity. The attention mechanism makes the model of this application have a higher weight in the diseased area, in other words, it pays more attention to the cells and tissue areas that should be processed. The principle of the attention pooling layer is as follows:

[0145]

[0146] Where w and V can be understood as two different linear layer weights in the model of this application. Hyperbolic tangent (tanh) is used element by element to include negative and positive values ​​for proper gradient flow. H = {h1,h2,…,h K} is a bag containing K feature maps (i.e., a case represented by a pathological image).

[0147] This application example uses the following formula to calculate the attention score of this application:

[0148]

[0149] This application example can obtain the importance of low-dimensional features extracted from each image block in a package based on the attention score.

[0150] That is to say, this application example proposes multiple instance learning (Multiple Instance Learning, MIL) with the addition of an attention mechanism to solve the problem of fewer whole slice image (WSI) labels. In order to identify human pathological features and link the pathological features with the image features extracted by the machine learning model, this application introduces an attention mechanism in the multiple instance learning model to identify and focus on the diseased area. In the feature extraction step, this application found that the mean and standard deviation of colorectal cancer pathology images are quite different from those of natural image datasets of traditional computer vision such as ImageNet. Therefore, this application uses an autoencoder (AutoEncoder, AE) to replace the pre-trained convolutional neural network for feature extraction.

[0151] In summary, the tumor prognosis prediction model training method provided by the application example of this application is different from the existing deep learning method for prognosis prediction. This application example adds multiple instance learning (Multiple Instance Learning, MIL) of the attention mechanism to solve the problem of fewer labels in the whole slice image (WSI). In order to identify human pathological features and link the pathological features with the image features extracted by the machine learning model, this application example introduces an attention mechanism in the multiple instance learning model to identify and focus on the diseased area. At the same time, this application example screens out tissue areas based on the slice optical density, which helps the attention mechanism to quickly focus on the lesion, reduces training time and noise interference, and improves the accuracy of the attention mechanism. In addition, in the feature extraction step, this application example finds that the mean and standard deviation of colorectal cancer pathology images are quite different from natural image data sets of traditional computer vision such as ImageNet. Therefore, this application example uses an autoencoder (AutoEncoder, AE) to replace the pre-trained convolutional neural network for feature extraction.

[0152] Based on the above-mentioned embodiment of the tumor prognosis prediction model training method, the present application also provides a tumor prognosis prediction method that can be implemented by a tumor prognosis prediction device, see Figure 8 The tumor prognosis prediction method specifically includes the following contents:

[0153] Step 400: Acquire normalized image data of target whole-slice pathological image data.

[0154] Step 500: Input the normalized image data of the target full-slice pathology image data into a preset tumor prognosis prediction model so that the autoencoder in the tumor prognosis prediction model outputs the target feature map data corresponding to the target full-slice pathology image data, and input the target feature map data into the multi-instance learning model based on the attention mechanism in the tumor prognosis prediction model so that the multi-instance learning model based on the attention mechanism outputs the survival prediction result data corresponding to the target full-slice pathology image data, wherein the tumor prognosis prediction model is pre-trained based on the tumor prognosis prediction model training method.

[0155] The tumor prognosis prediction model in step 400 and step 500 is trained in advance using the tumor prognosis prediction model training method in the aforementioned embodiment.

[0156] It can be seen from the above description that the tumor prognosis prediction method provided in the embodiment of the present application can improve the effectiveness and accuracy of the results of tumor prognosis prediction using a tumor prognosis prediction model.

[0157] The present application also provides a tumor prognosis prediction model training device for executing all or part of the tumor prognosis prediction model training method, see Fig. 9 The tumor prognosis prediction model training device specifically includes the following contents:

[0158] The autoencoder module 10 is used to input the normalized image data of each full-slice pathology image data into a preset autoencoder, so as to obtain the target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder.

[0159] The sample combination module 20 is used to combine the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data to obtain data samples corresponding to each of the full-slice pathology image data.

[0160] The weakly supervised learning module 30 is used to use each of the data samples to train a preset multi-instance learning model based on the attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, it constitutes a tumor prognosis prediction model.

[0161] The embodiment of the tumor prognosis prediction model training device provided in the present application can be specifically used to execute the processing flow of the embodiment of the tumor prognosis prediction model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned tumor prognosis prediction model training method embodiment.

[0162] The part of the tumor prognosis prediction model training device that performs tumor prognosis prediction model training can be executed in the server or completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of tumor prognosis prediction model training.

[0163] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0164] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0165] From the above description, it can be seen that the tumor prognosis prediction model training device provided in the embodiment of the present application can improve the recognition and attention accuracy of the diseased area during the model training process based on the attention mechanism, and can solve the problem of fewer labels in the whole slice image and can link the pathological features with the image features extracted by the machine learning model. It can also improve the effective area of ​​​​the extracted pathological image based on normalization processing, and then assist the attention mechanism to quickly focus on the lesion area during the model training process, and can reduce the model training time and noise interference and improve the accuracy of the attention mechanism, and then improve the training efficiency, accuracy and reliability of the tumor prognosis prediction model, so as to improve the effectiveness and accuracy of the results of tumor prognosis prediction using the tumor prognosis prediction model.

[0166] The present application also provides a tumor prognosis prediction device for executing all or part of the tumor prognosis prediction method, see Fig.10 , the tumor prognosis prediction device specifically includes the following contents:

[0167] The normalization module 40 is used to obtain normalized image data of the target full-slice pathological image data.

[0168] The survival prediction module 50 is used to input the normalized image data of the target full-slice pathological image data into a preset tumor prognosis prediction model so that the autoencoder in the tumor prognosis prediction model outputs the target feature map data corresponding to the target full-slice pathological image data, and input the target feature map data into the multi-instance learning model based on the attention mechanism in the tumor prognosis prediction model so that the multi-instance learning model based on the attention mechanism outputs the survival prediction result data corresponding to the target full-slice pathological image data, wherein the tumor prognosis prediction model is pre-trained based on the tumor prognosis prediction model training method.

[0169] The embodiment of the tumor prognosis prediction device provided in the present application can be specifically used to execute the processing flow of the embodiment of the tumor prognosis prediction method in the above-mentioned embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the above-mentioned tumor prognosis prediction method embodiment.

[0170] The part of the tumor prognosis prediction device that performs tumor prognosis prediction can be executed in the server or in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of tumor prognosis prediction.

[0171] It can be seen from the above description that the tumor prognosis prediction device provided in the embodiment of the present application can improve the effectiveness and accuracy of the results of tumor prognosis prediction using a tumor prognosis prediction model.

[0172] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the tumor prognosis prediction model training method and / or tumor prognosis prediction method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via wired or wireless means.

[0173] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0174] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the tumor prognosis prediction model training method and / or the program instructions / modules corresponding to the tumor prognosis prediction method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, realizing the tumor prognosis prediction model training method and / or the tumor prognosis prediction method in the above method embodiment.

[0175] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The one or more modules are stored in the memory, and when executed by the processor, perform the tumor prognosis prediction model training method and / or tumor prognosis prediction method in the embodiment.

[0177] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0178] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0179] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0180] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the aforementioned tumor prognosis prediction model training method and / or tumor prognosis prediction method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0181] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the tumor prognosis prediction model training method and / or the tumor prognosis prediction method.

[0182] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0183] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0184] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0185] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A tumor prognosis prediction model training method, characterized in that: include: Inputting normalized image data of each full-slice pathology image data into a preset autoencoder respectively, so as to obtain target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder; Combining the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data, respectively, to obtain data samples corresponding to each of the full-slice pathology image data; Each of the data samples is used to train a preset multi-instance learning model based on the attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, it constitutes a tumor prognosis prediction model.

2. The tumor prognosis prediction model training method according to claim 1, characterized in that: Before the normalized image data of each full-slice pathological image data are input into a preset autoencoder respectively, the method further includes: Cutting each full-slice pathological image data into a plurality of image blocks of the same size; Performing filtering processing on each of the image blocks; Each of the image blocks after filtering is normalized to obtain each normalized image block corresponding to each of the full-slice pathology image data, wherein each of the normalized image blocks belonging to the same full-slice pathology image data together constitutes the normalized image data corresponding to the full-slice pathology image data.

3. The tumor prognosis prediction model training method according to claim 2, characterized in that: The filtering process for each of the image blocks comprises: Each of the image blocks is converted from the RGB space to the optical density space, and among the image blocks converted to the optical density space, the image blocks whose optical density values ​​are less than a preset optical density threshold are eliminated.

4. The tumor prognosis prediction model training method according to claim 3, characterized in that: The step of normalizing each of the filtered image blocks to obtain each normalized image block corresponding to each of the full-slice pathological image data comprises: Performing singular value decomposition on each of the image blocks converted to the optical density space to obtain a eigenvector corresponding to the optical density space of each image block, then sorting the eigenvalues ​​of each eigenvector in descending order, and selecting the first two from the sorted eigenvectors to form a plane of the optical density space of the image block; Mapping the pixel points of each of the image blocks after filtering to the plane of the image block optical density space and normalizing them to a preset unit length; Determine the angle of each pixel point on the plane of the image block optical density space relative to a preset direction, and search for an extreme angle among the angles; Each of the extreme value angles is converted back to the optical density space to obtain normalized image data corresponding to each of the normalized image blocks.

5. The tumor prognosis prediction model training method according to claim 2, characterized in that: The autoencoder comprises an encoder and a decoder connected to each other; Correspondingly, the normalized image data of each full-slice pathology image data are respectively input into a preset autoencoder to obtain the target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder, including: Inputting each of the normalized image blocks into the encoder respectively, so that the encoder maps each of the normalized image blocks into a preset feature space respectively, and enables the decoder to perform feature reconstruction on each of the normalized images mapped into the feature space respectively, so as to obtain feature reconstruction image data corresponding to each of the normalized image blocks; Respectively splicing the characteristic reconstruction map data corresponding to the same full-slice pathology image data to obtain target characteristic map data corresponding to each full-slice pathology image data; The autoencoder is pre-trained using each normalized image sample and a mean square error loss function.

6. The tumor prognosis prediction model training method according to claim 1, characterized in that: The multi-instance learning model based on the attention mechanism includes: an attention pooling layer and a linear activation layer connected to each other; Correspondingly, the preset multi-instance learning model based on the attention mechanism is trained by using each of the data samples, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitutes a tumor prognosis prediction model, including: Input each of the data samples into the attention pooling layer respectively, so that the attention pooling layer downsamples each of the input target feature map data based on the attention mechanism, and enables the linear activation layer to obtain the survival period prediction result data corresponding to the target feature map data after downsampling based on a preset activation function, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitutes a tumor prognosis prediction model; The survival prediction result data includes: shape parameters and scale parameters used to represent the Weibull distribution model.

7. A method for predicting tumor prognosis, characterized in that: include: Acquiring normalized image data of target full-slice pathological image data; The normalized image data of the target full-slice pathology image data is input into a preset tumor prognosis prediction model so that the autoencoder in the tumor prognosis prediction model outputs target feature map data corresponding to the target full-slice pathology image data, and the target feature map data is input into the multi-instance learning model based on the attention mechanism in the tumor prognosis prediction model so that the multi-instance learning model based on the attention mechanism outputs survival prediction result data corresponding to the target full-slice pathology image data, wherein the tumor prognosis prediction model is pre-trained based on the tumor prognosis prediction model training method described in any one of claims 1 to 6.

8. A tumor prognosis prediction model training device, characterized in that: include: An autoencoder module, used for inputting normalized image data of each full-slice pathology image data into a preset autoencoder, so as to obtain target feature map data corresponding to each full-slice pathology image data based on the output of the autoencoder; A sample combination module, used for combining the target feature map data corresponding to each of the full-slice pathology image data with the overall survival data and survival status data corresponding to each of the full-slice pathology image data, so as to obtain a data sample corresponding to each of the full-slice pathology image data; A weakly supervised learning module is used to use each of the data samples to train a preset multi-instance learning model based on an attention mechanism, so that the multi-instance learning model based on the attention mechanism is used to predict the survival period corresponding to the full-slice pathological image data, and together with the autoencoder, constitute a tumor prognosis prediction model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the tumor prognosis prediction model training method as described in any one of claims 1 to 6, and / or implements the tumor prognosis prediction method as described in claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the tumor prognosis prediction model training method as described in any one of claims 1 to 6, and / or implements the tumor prognosis prediction method as described in claim 7.

Images