A tumor prognosis analysis method and system

Abstract

The present invention proposes a tumor prognosis analysis method and system, comprising the following steps: acquiring a full-field pathology image; obtaining a typical region within the full-field pathology image; and inputting the typical region image into a prognostic analysis subnetwork, which performs prognostic analysis based on clinical information and accurately identified typical region images. This method directly uses the full-field pathology image as input and, combined with clinical information, can effectively predict a patient's prognosis. The prognostic analysis subnetwork extracts and aggregates pathology image features and clinical information from the typical region to predict the patient's prognosis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a tumor prognosis analysis method and system. Background Art

[0002] Ovarian cancer is one of the most lethal cancers of the female reproductive system. High-grade serous adenocarcinomas (HGSA) of the ovary account for 75% of ovarian cancer cases and 80-90% of deaths, and have the worst prognosis of all ovarian cancer subtypes. Even with the same treatment, patients with advanced-stage HGSA have significant variability in prognosis. While a small percentage of patients experience short-term relapse and long survival after treatment, the majority (>75%) experience recurrence within two years. Studies have shown that the prognosis of ovarian HGSA is associated with a range of clinical indicators (such as age, lymph node metastasis, unilateral / bilateral tumor location, and postoperative FIGO staging), as well as pathological, genomic, transcriptomic, and immunological profiles. However, there are currently no effective methods to distinguish between those at high risk of recurrence and those with a poor prognosis, posing a challenge to precision medicine. Accurately predicting the prognosis of HGSA patients remains challenging, even for experienced pathologists.

[0003] However, the prognosis prediction of ovarian HGSA has important clinical value. Therefore, with the advancement of science and technology in recent years, technicians in this field have added artificial intelligence algorithms to perform prognosis prediction.

[0004] The application of artificial intelligence algorithms in ovarian HGSA analysis focuses on bioinformation mining. Wang et al. analyzed the density differences of cytotoxic T cells within tumor cell nests and tumor stroma in ovarian cancer pathological tissue sections and used machine learning algorithms to classify ovarian cancer into three immune phenotypes. Madabuhushi et al. used whole slide images (WSIs) of tumor tissue from 926 malignant tumors, including ovarian and cervical cancers, in the TCGA dataset and employed artificial intelligence algorithms to validate the correlation between the spatial distribution characteristics of tumor-infiltrating lymphocytes (TILs) and prognosis in solid tumors, including ovarian cancer. In recent years, deep learning methods, which can automatically learn from images and quickly obtain excellent feature representations, have achieved great success in the field of computer vision. Simultaneously, advances in medical imaging and data storage technologies have led to the widespread application of deep learning in computer-aided diagnosis. However, despite the great potential of deep learning, its application to pathological image analysis faces significant challenges.

[0005] Due to the high resolution of full-field pathology images, one difficulty in using computer-assisted analysis in this field is that these full-field histopathology images, with hundreds of millions or even billions of pixels, cannot be directly input into existing deep learning networks for training and inference. Consequently, there is limited research on establishing prognostic analysis systems based on the pathological features of ovarian HGSA. Summary of the Invention

[0006] In order to overcome the above-mentioned defects in the prior art, the purpose of the present invention is to provide a tumor prognosis analysis method and system.

[0007] In order to achieve the above-mentioned object of the present invention, the present invention provides a tumor prognosis analysis method, comprising the following steps:

[0008] Acquire full-field pathology images;

[0009] Acquire typical areas of pathological full-field images;

[0010] The typical region image is input into a prognostic analysis subnetwork, and the prognostic analysis subnetwork performs prognostic analysis based on clinical information and the accurately identified typical region image.

[0011] This method directly takes the full-field pathological image as input and combines it with clinical information to effectively predict the patient's prognosis. The prognosis analysis subnetwork predicts the patient's prognosis by extracting and summarizing the pathological image features and clinical information of typical areas.

[0012] In an optional solution of the tumor prognosis analysis method, the prognosis analysis subnetwork includes a pathology image feature extraction module, a clinical feature extraction module and a prognosis analysis module;

[0013] The pathological image feature extraction module extracts pathological image features from pathological image blocks and constructs feature aggregation. The clinical feature extraction module extracts clinical information. The prognosis analysis module cross-modally fuses pathological image features with clinical information and predicts the prognosis status.

[0014] This optional solution extracts and summarizes pathological image features and combines them with clinical information to predict the patient's prognosis, thereby improving the accuracy of prognosis prediction.

[0015] In an optional solution of the tumor prognosis analysis method, the pathological image feature extraction module includes a heavyweight feature extraction unit, a lightweight feature extraction unit and a feature aggregation unit;

[0016] The weight feature extraction unit is used to obtain high-order features of the pathological image, the lightweight feature extraction unit is used to obtain shallow basic features of the pathological image, and the feature aggregation unit first unifies the number of input feature channels for the high-order features and shallow basic features of the pathological image, and then uses dimensional addition to construct feature aggregation;

[0017] The weight feature extraction unit and the lightweight feature extraction unit are both deep convolutional neural networks, and the convolution layer of the weight feature extraction unit is larger than the convolution layer of the lightweight feature extraction unit.

[0018] This optional solution uses heavyweight feature extraction units and lightweight feature extraction units at the same time, taking into account both low-order and high-order features in the image and enhancing feature learning capabilities.

[0019] In an optional solution of the tumor prognosis analysis method, the prognosis analysis subnetwork extracts and summarizes the pathological image features image, combines clinical information clinical, and predicts the patient's prognosis status

[0020]

[0021] PAS(·) is the output of the sub-network, θ P It is a set of trainable network parameters.

[0022] This optional solution can effectively integrate key features from different types of data, including histopathology images and clinical data of tumor patients (for example, age, lymph node metastasis, unilateral / bilateral tumor, postoperative FIGO staging, whether neoadjuvant treatment was used before surgery, etc.); by simultaneously using heavyweight feature extraction units and lightweight feature extraction units for the input typical histopathology image area, it takes into account both low-order and high-order features in the typical histopathology image, thereby enhancing the feature learning ability of the sub-network.

[0023] In an optional solution of the tumor prognosis analysis method, the parameter set θ of the prognosis analysis subnetwork is P The training process adopts the minimization problem:

[0024] in, is the loss function of the prognostic analysis subnetwork, Q is the total number of case data in the case data set, q represents the qth case data in Q, θ P is the parameter set of the prognostic analysis sub-network, L unfavor is the cross entropy between the predicted value and the actual state of the poor prognosis case, L favor is the cross entropy between the predicted value and the actual state of the case with good prognosis, Ω is a very small positive number, η is the weight, P(·) is the prognostic state output by the tumor prognostic analysis system, which has two types: favor and unfavor, and Y favor For cases with a true prognosis of favor, Y unfavor is the case with the true prognostic status of unfavor, L(·) is the cross entropy, and there are cross entropies of the two states: favor and unfavor.

[0025] This option enables the prognostic analysis system to quickly and conveniently adjust its sensitivity to the prognostic status of a certain type of case by adjusting the weight of the cross entropy of cases with a good prognosis (favor) and cases with a poor prognosis (unfavor) in the loss function.

[0026] In an optional solution of the tumor prognosis analysis method, a method for obtaining a typical region of a pathological full-field image includes:

[0027] Inputting the pathological full-field image into a configurable hierarchical sampling subnetwork, wherein the configurable hierarchical sampling subnetwork is used to perform hierarchical sampling on the pathological full-field image to screen out typical areas;

[0028] The obtained typical regions are input into the pathological image precise analysis subnetwork to achieve precise recognition of the typical region images.

[0029] The configurable hierarchical sampling subnetwork in this optional solution can effectively process large-resolution full-field pathology images and screen out typical areas; the pathology image precision analysis subnetwork combines the feature separation module and the feature re-fusion module to make the output typical areas more precise and the fine-grained image recognition accuracy higher and more accurate.

[0030] In an optional solution of the tumor prognosis analysis method, the configurable hierarchical sampling subnetwork includes cascaded multi-stage sampling modules;

[0031] The first-level sampling module screens typical regions on the low-resolution image of the pathological full-field image and maps the regions onto the original-resolution pathological full-field image, thereby obtaining a plurality of original-resolution pathological image blocks with the same range as the screened regions, i.e., first-level sub-image blocks;

[0032] The j-th level sampling module performs typical area screening on the low-resolution image of the sub-image block output by the sampling module of the previous level and maps it to the pathological full-field image of the original resolution, obtaining multiple pathological image blocks of the original resolution with the same area range as the currently screened area, namely, the j-th level sub-image block, where j is an integer greater than 1.

[0033] The configurable layered sampling subnetwork in this optional scheme emphasizes the ability to process a small number of full-resolution image patches of the input image, which can significantly reduce peak GPU memory usage and have higher sampling accuracy. For very large images, such as mega- to gigapixel images, the number of selected image patches is much smaller than the size of the sample space.

[0034] In an optional solution of the tumor prognosis analysis method, the sampling modules each include a feature extraction unit, an attention weighting unit, and a classification unit;

[0035] The feature extraction unit extracts features from the pathological full-field image or the sub-image block output by the previous sampling module, and the classification unit classifies the features extracted by the feature extraction unit based on the weights of the attention weighting unit to obtain sub-image blocks.

[0036] This optional solution achieves the effect of directly using deep neural networks to screen typical feature areas from ultra-large images with hundreds of millions or even billions of pixels (for example, full-field tissue pathology images).

[0037] In an optional solution of the tumor prognosis analysis method, the function of the j-th level sampling module is:

[0038]

[0039] in, is the classification function of the j-th level sampling module, is the feature extraction function of the j-th level sampling module, is the attention function of the j-th level sampling module, c (j) is the input image x of the j-th level sampling module j The ratio is s j The low-resolution sampling of ∈(0,1) corresponds to the coordinate mapping of the pathological full-field image, and C is the coordinate mapping of all c (j) The set of components;

[0040] j-1th level sampling function is the input image x to the j-1th level sampling module j-1 The ratio is s j-1 ∈(0,1) low-resolution sampling to obtain coordinates c (j-1) And c (j-1) Mapped to the j-1th level sampling module input image x j-1 middle, is the j-th level sampling function, the input image x of the j-th level sampling module j for

[0041] This optional solution adopts a multi-layer step-by-step sampling mode, so that the sampling module of each layer only processes the area output by the previous layer, without processing other areas; through the coordinate c (j-1) And the scaling ratio can ensure that the output of the sampling module at each layer can be mapped back to the original image (pathological full-field image).

[0042] In an optional solution of the tumor prognosis analysis method, after Monte Carlo approximation, the function of the j-th level sampling module is: Among them, M refers to the number of sub-image blocks output by each sampling module, c (j-1) ∈C.

[0043] This alternative can effectively avoid j ,c) calculate features in all image blocks to improve the operation speed.

[0044] In an optional solution of the tumor prognosis analysis method, the loss function of the configurable hierarchical sampling subnetwork is

[0045] in, is the weighted sum of the cross entropy of sampling modules at all levels, is the cross entropy of the j-th level sampling module, α (j) is the weight of the j-th level sampling module,

[0046] L RZ =max{||CMLS-SAP1-AM(·)||2,||CMLS-SAP2-AM(·)||2,...,||CMLS-SAP N-AM(·)||2},||CMLS-SAP j -AM(·)||2 is the second-order norm of the network weight of the attention unit of the j-th sampling module, β is the weight coefficient of the regularization term, and N is the total number of sampling modules.

[0047] In this optional solution, a regularization term is introduced into the loss function, which allows the model to keep the model simple while minimizing the training error, while improving the model's generalization performance and preventing overfitting.

[0048] In an alternative embodiment of the tumor prognosis analysis method, the weight ω1 (j) is the area ratio or quantity ratio of the output image blocks of all levels of sampling modules to the output image blocks of the jth level, ω2 (j) It is the difference between the ratio of the sum of the gradient values of the image block with the smallest sum of gradient information and the image block with the largest sum of gradient information among all image blocks output by the j-th level sampling module and 1,

[0049] Among them, M (j) is the total number of output image blocks of the j-th level sampling module,

[0050] Among them, tile (j) is the gradient information in the image block output by the j-th level sampling module, and m is the sequence number of the image block output by the j-th level sampling module.

[0051] The smaller it is, the larger the weight is given to it; the ratio of the number of image blocks output by the l-th level sampling module to the sum of the number of image blocks output by all sampling modules can be directly used to achieve fast approximate calculation. The larger the value is, the higher the degree of information consistency of the output sampled image block is, and a larger weight is given to this part of the output.

[0052] In an optional solution of the tumor prognosis analysis method, the pathology image precise analysis subnetwork includes a feature separation module and a feature re-fusion module;

[0053] The feature separation module separates image features through a feature extraction unit specific to the image category;

[0054] The feature re-fusion module re-fuses the image features based on a channel selection mechanism to achieve accurate recognition of typical area images.

[0055] In this optional solution, the feature separation module can alleviate the negative feature transfer generated by images of different categories, and the feature fusion module can enhance the positive feature transfer of specific image categories.

[0056] In an optional solution of the tumor prognosis analysis method, the feature separation module uses multiple category feature maps to obtain specific category features, and uses specific category classifiers to constrain different category features to be separable.

[0057] In an optional solution of the tumor prognosis analysis method, the expression of the pathological image accurate analysis subnetwork is:

[0058] AAOH-FSM(·) is the action expression of the feature separation module.

[0059]

[0060] Among them, f n Represents the features of the nth image category, image features after separation from common features x n is a sample of the nth image category, Conv n (·) 1×1 convolution is used to extract the specific features of the nth image category, and the input and output channels are the same as f n The number of feature channels is the same, BN n (·) is the batch normalization layer, ReLU is the nonlinear activation function, G n (·) is the classifier for the nth image category, is a set of features of a specific image category, and P is the total number of image categories;

[0061] AAOH-FRM(·) is the action expression of the feature re-fusion module,

[0062] Represents the re-fusion feature, express τth th channels, τ∈[1,T], the total number of channels is T, Cat(·) represents the concatenation of multiple channel-level features, FC(·) represents the fully connected layer with an output dimension equal to the image category n,

[0063] The fused features F are passed to the joint classifier G(·) to predict the joint classification result.

[0064] In this optional solution, the feature separation module decouples mixed features, embeds specific features of different categories, and eliminates feature negative transfer; the feature re-fusion module fuses feature arrays based on the channel selection mechanism to enhance the feature positive transfer of specific image categories.

[0065] In an optional solution of the tumor prognosis analysis method, the loss function of the pathological image accurate analysis subnetwork is Loss all (·)=LossFSM (·)+Loss FRM (·),

[0066] Loss FSM (·) is the loss function of the feature separation module:

[0067]

[0068] Loss FRM (·) is the loss function of the feature re-fusion module: Loss FRM (·) = L(G(F), y), where L(·) is the cross entropy loss, y represents the true category, and G(F) represents the category predicted by the module.

[0069] In this optional solution, the ovarian tissue pathology image precise analysis subnetwork accurately identifies the input typical image area and is optional in the workflow of the entire prognostic analysis system, wherein: the feature separation module separates features through a feature extraction unit specific to the image category, thereby alleviating the negative migration of features generated by images of different categories; the feature re-fusion module re-fuses features specific to the image category based on the channel selection mechanism, thereby enhancing the positive migration of features of specific image categories.

[0070] The present invention also proposes a tumor prognosis analysis system, including an image receiving module, a processing module and a storage module. The image receiving module receives images for training or evaluation and sends the received images to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform operations corresponding to the above-mentioned tumor prognosis analysis method based on the image it receives.

[0071] The beneficial effects of the present invention are:

[0072] The present invention can effectively process pathology full-field-of-view images, use low-resolution views to indicate key information areas in the reduced view for the purpose of prognostic analysis, and then extract key features in the high-resolution view (pathology full-field-of-view image) based on the key information areas, without the need to obtain the entire pathology full-field-of-view image.

[0073] Through the careful design of each module, the present invention can extract key features that effectively characterize the prognostic status and effectively predict the prognostic status of tumor patients. The prediction results can provide an important reference for the analysis of the prognostic status of tumor patients.

[0074] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0076] Figure 1 It is a principle block diagram of embodiment 1;

[0077] Figure 2 It is a schematic diagram of the structure of the configurable hierarchical sampling sub-network;

[0078] Figure 3 It is a schematic diagram of the structure of the j-th level sampling module in the configurable hierarchical sampling subnetwork;

[0079] Figure 4 This is a schematic diagram of the structure of the pathology image precise analysis subnetwork. The figure takes an image containing three image categories as an example. Sub-tiles are preliminary typical region images. F(.) represents the feature set of these three image category features (f1(), f2(), f3()). Feature Separation is the feature separation module. G1(·), G2(·), and G3(·) are the classifiers of these three image categories, respectively. Feature Refusion is the feature refusion module. G(·) is the joint classifier. Fine-grained sub-tiles are highly fine-grained typical region images.

[0080] Figure 5 It is a schematic diagram of the structure of the prognostic analysis sub-network;

[0081] Figure 6 It is a structural diagram of the AGG module;

[0082] Figure 7a It is a schematic diagram comparing the analytical performance of experimental results of different methods;

[0083] Figure 7b It is a schematic diagram comparing the analytical stability of experimental results of different methods;

[0084] Figure 8a This is a performance comparison diagram of the ablation experiment;

[0085] Figure 8b This is a schematic diagram of the stability comparison of the ablation experiment;

[0086] Figure 9a This is a schematic diagram comparing the analytical performance of the experimental results of the prognostic analysis sub-network using different feature extraction models;

[0087] Figure 9b This is a schematic diagram comparing the analytical stability of the experimental results of the prognostic analysis sub-network using different feature extraction models;

[0088] Figure 10This is a schematic diagram of the analysis results of the prognostic analysis system for six typical ovarian HGSA cases. DETAILED DESCRIPTION

[0089] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0090] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0091] Example 1

[0092] like Figure 1 As shown, the present invention provides an embodiment of a tumor prognosis analysis method, which is used for the prognosis analysis of ovarian HGSA.

[0093] The specific steps include:

[0094] Acquire full-field pathology images.

[0095] This example collects data from patients with ovarian HGSA who were admitted and diagnosed at West China Second Hospital, Sichuan University from 2016 to 2019. Each patient's data includes a full-field pathological image of the primary ovarian tumor tissue, clinical information, and prognosis. The inclusion criteria for the cases are as follows:

[0096] (1) Pathological diagnosis confirmed as ovarian HGSA by senior doctors;

[0097] (2) The clinical data of the patients during the initial visit and treatment period were complete;

[0098] (3) FIGO stage II-IV;

[0099] (4) The follow-up period was >3 years.

[0100] This study included 450 patients, including 225 with poor prognosis (recurrence within 2 years after complete staging) and 225 with good prognosis (recurrence or no recurrence more than 2 years after complete staging). Poor prognosis was set to 1 and good prognosis was set to 0.

[0101] Ovarian HGSA cases with good and poor prognoses were designated as negative and positive samples, respectively. All experiments used three-fold cross-validation, using 300 data points (150 positive and 150 negative) as the training set and 150 data points (75 positive and 75 negative) as the test set.

[0102] Then, a typical region of the full-field pathology image is obtained. Specifically, the method of obtaining the typical region of the full-field pathology image can be, but is not limited to, using existing methods such as uniform block segmentation, overlap block segmentation, multi-instance deep learning analysis methods, and self-supervised learning-based analysis methods. Alternatively, the new method of this embodiment can be used, which is specifically achieved by the following steps:

[0103] The full-field pathology image is input into the configurable hierarchical sampling sub-network CMLS that has been pre-trained according to the screening requirements. The configurable hierarchical sampling sub-network CMLS is used to perform hierarchical sampling on the full-field pathology image to screen out typical areas. The obtained typical areas are input into the pathology image accurate analysis sub-network AAOH that has been pre-trained according to the fine-grained requirements to achieve accurate recognition of the typical area image.

[0104] Finally, the typical region images after accurate identification are input into the prognostic analysis sub-network PAS which has been pre-trained as required. The prognostic analysis sub-network PAS performs prognostic analysis based on clinical information and the typical region images after accurate identification.

[0105] During pre-training, the hierarchical sampling sub-network CMLS, the pathological image accurate analysis sub-network AAOH and the prognosis analysis sub-network PAS can be configured to be trained separately.

[0106] The configurable layered sampling sub-network CMLS includes cascaded multi-stage sampling modules. In this embodiment, the structure of the configurable layered sampling sub-network CMLS is as follows: Figure 2 As shown, it includes the first-level sampling module CMLS-SAP1(·), the second-level sampling module CMLS-SAP2(·), and the N-level sampling module CMLS-SAP n The number N of the sampling modules is configurable. In this embodiment, the value range of N is 1-5, preferably 2 or 3.

[0107] The first-level sampling module CMLS-SAP1(·) screens typical regions on the low-resolution pathology full-field image and maps them onto the original-resolution pathology full-field image, obtaining TN1 first-level sub-image blocks of the original resolution with the same range as the screened regions.

[0108] The sampling modules at each level are cascaded in sequence and perform typical area screening on the low-resolution image of the sub-image block output by the sampling module at the previous level and map it to the pathological full-field image at the original resolution, thereby obtaining multiple sub-image blocks at each level with the same area range as the screened area.

[0109] That is, the second-level sampling module CMLS-SAP2(·) screens typical regions on the low-resolution image of TN1 first-level sub-image blocks and maps them onto the full-field pathology image of the original resolution, obtaining TN2 second-level sub-image blocks of the original resolution with the same regional range as the currently screened region;

[0110] Sequentially, the j-th level sampling module CMLS-SAP j (·) Perform typical area screening on the low-resolution sub-image blocks output by the previous sampling module and map them onto the full-field pathology image at the original resolution to obtain multiple j-th sub-image blocks at the original resolution with the same area range as the currently screened area, where j is an integer greater than 1.

[0111] Until the Nth level sampling module CMLS-SAP n (·) In TN n-1 The typical area is screened on the low-resolution image of the (N-1)th level sub-image block and mapped to the original resolution pathological full-field image to obtain the original resolution TN with the same regional range as the current screened area. n n-th level sub-image blocks, which contain image detail information for subsequent analysis. The low resolutions collected by different sampling modules in the configurable layered sampling subnetwork CMLS can be the same or different, and are all between 0 and 1, preferably but not limited to increasing step by step. It emphasizes the ability to process a small number of full-resolution image blocks of the input image, which can significantly reduce the peak GPU memory usage and have higher sampling accuracy. For very large images (such as mega-pixels to gigapixels), the number of selected image blocks is much smaller than the sample space || S sampling The size of ||.

[0112] like Figure 3 As shown, each level of the sampling module of the configurable layered sampling subnetwork CMLS includes a feature extraction unit CMLS-SAP j -FE(·), attention weighted unit CMLS-SAP j -AM(·) and taxon CMLS-SAP j -CF(·), (1≤j≤N). Feature extraction unit CMLS-SAP j -FE(·) extracts features from the full-field pathology image or the sub-image block output by the previous sampling module, and the classification unit CMLS-SAP j-CF(·) is based on the attention weighted unit CMLS-SAP j The weight of -AM(·) is used to classify the features extracted by the feature extraction unit to obtain sub-image blocks.

[0113] In this embodiment, the feature extraction unit CMLS-SAP j -FE(·) is composed of trainable parameters A typical example of a deep convolutional neural network for characterization is shown in Table 1.

[0114] Attention Weighted Unit CMLS-SAP j -AM(·) is composed of trainable parameters A smaller deep convolutional neural network represented by

[15] is used to obtain the attention weight matrix, as shown in Table 2, a typical example of which.

[0115] Taxonomic unit CMLS-SAP j -CF(·) is composed of trainable parameters A single fully connected layer fc-n representing class .

[0116] Table 1 Feature extraction unit CMLS-SAP j A typical example of -FE(·)

[0117] layer type 1 Conv(3,1,1,8)+ReLU() 2 Conv(3,1,1,16)+ReLU() 3 Conv(3,1,1,32)+ReLU() 4 Conv(3,1,1,16)+ReLU() 5 Conv(3,1,1,8)+ReLU() 6 Conv(3,1,1,16)+ReLU() 7 Conv(3,1,1,32)+ReLU() 8 Conv(3,1,1,1)+ReLU() 9 GlobalAveragePooling2D() 10 SoftMax()

[0118] Table 2 Attention weighted unit CMLS-SAP j A typical example of -AM(·)

[0119] layer type 1 Conv(3,1,1,8)+ReLU() 2 Conv(3,1,1,16)+ReLU() 3 Conv(3,1,1,32)+ReLU() 4 Conv(3,1,1,1)+SoftMax() 5 SoftMax()

[0120] The action function CMLS-SAP1(x) of the first level (i.e., when j=1) sampling module of the configurable layered sampling subnetwork CMLS is as follows, which is given by θ={θ f ,θ a ,θ c}Parameterized convolutional neural network:

[0121]

[0122] in, is the classification function of the classification unit of the first-level sampling module, It is the feature extraction function of the feature extraction unit of the first-level sampling module, and the sampling function From the input image x1∈R H×W Extract the first-level sub-image block of size h1×w1, which corresponds to the low-resolution view V(x,s1)∈R with a scale of s1∈(0,1) h×wThe coordinates c1 = {i, j} in the sampling function, c1 represents the corresponding coordinates of the thumbnail view processed by the first-level sampling module (the low-resolution view after the original image (the pathological full-field image) is reduced) and the original image (the pathological full-field image), (i, j) is its horizontal and vertical coordinates in the original image, and the sampling function pass Maps coordinate c1 to a location in image x1 and returns a subimage patch of size h1×w1, where H×W is the resolution of image x1. and h is the height of the thumbnail view of the input 1st level sampling module, w is the width of the thumbnail view of the input 1st level sampling module, It is a floor operation.

[0123] The attention mechanism is introduced in CMLS-SAP1(x), and the convolutional neural network of the attention mechanism is α=a θ (V(x,s1)), where a θ (·) represents the trainable parameters The smaller deep convolutional neural network represented is used to obtain the attention weight matrix and obtain the attention weighted network:

[0124]

[0125] Among them, g Θ Generally refers to the classification function in the sampling module, here refers to the classification function in the l-th level sampling module f Θ Generally refers to the feature extraction function in the sampling module, here refers to the feature extraction function in the lth layer sampling module is the attention matrix, c = (i, j) represents the corresponding coordinates of the thumbnail view processed by the sampling module of this layer and the original image (pathological full-field image), (i, j) represents its horizontal and vertical coordinates in the original image, C is the set of c in the sampling modules of each layer, α c Refers to the convolutional neural network with attention mechanism α=a θ (V(x,s1,c)),||S sampling || is the set of all index pairs of the low-resolution view V(x,s1) of image x1, ||S sampling ||=h×w.

[0126] In order to avoid computing features from all sub-image patches of the low-resolution view V(x,s1), the formula Perform Monte Carlo approximation to obtain the function of the first-level sampling module: Where C={(i,j)~a θ (V(x,s1))|i=1 ,2,...,M} is the index pair M<<||S of the low-resolution view V(x,s1) sampling CMLS-SAP1(x) uses the low-resolution view V(x,s1) of image x1 to calculate the attention weight, obtains M sub-image blocks by sampling, and The average is used to obtain the aggregated features, P refers to the total number of image categories, z n Refers to the features unique to the nth image category.

[0127] Similarly, the action function of the j-th level sampling module of the configurable layered sampling subnetwork CMLS is is given by θ={θ f ,θ a ,θ c}Parameterized convolutional neural network.

[0128] in, is the classification function of the classification unit of the j-th level sampling module, is the feature extraction function of the feature extraction unit of the j-th level sampling module, View V(x,s j ,c)∈R u×v is the sampling function Based on scale S j ∈(0,1), c can map the thumbnail view of the layer back to the original image (pathological full field image). CMLS-SAP j (x) introduces the view V(x,s j ,c) Attention mechanism CMLS-SAP j -AM(·),CMLS-SAP j -AM(·)=V(x,s j ,c)∈R u×v , we get the attention weighted network:

[0129] Where x is the input image, is the attention function CMLS-SAP of the j-th level sampling module j -AM(·), the j-1th level sampling function is the input image x to the j-1th level sampling module j-1 The ratio is s j-1 ∈(0,1) low-resolution sampling to obtain coordinates c (j-1) And c (j-1) Mapped to the j-1th level sampling module input image x j-1 middle, is the j-th level sampling function, the input image x of the j-th level sampling module j for c(j) is the input image x of the j-th level sampling module j The ratio is s j The low-resolution sampling of ∈(0,1) corresponds to the coordinate mapping of the pathological full-field image, and C is the coordinate mapping of all c (j) The collection composed of.

[0130] The low-resolution sampling ratio of the input image for each level of sampling function is set in advance.

[0131] right Perform Monte Carlo approximation to obtain the function of the j-th level sampling module:

[0132]

[0133] Among them, M refers to the number of sub-image blocks output by each sampling module, c (j-1) ∈C,c (j) ~CMLS-SAP j-1 (V(x,s j-1 ,c (j-1) ))From the distribution Corresponding position extraction, CMLS-SAP j-1 (·) represents the function of the j-1th level sampling module, CMLS-SAP j (x) Use image x j Low-resolution or full-resolution sub-image blocks, from and c (j) The j-level discrete distribution is determined by the j-level low-resolution view V(x,s j ,c (j) ) is obtained. When j = 1, CMLS-SAP j (x) is the selection of typical feature areas on the low-resolution image; subsequently, the typical feature areas are magnified step by step and then sampled until j=N, and then the full-resolution image (pathological full-field image) is performed.

[0134] The loss function of the configurable layered sampling subnetwork CMLS is:

[0135]

[0136] in, is the weighted sum of the cross entropy of the sampling modules at all levels, L RZ is the regularization term, β is the weight coefficient of the regularization term, and the default value is 1.0.

[0137] The weighted sum of cross entropy of sampling modules at all levels Contains the cross entropy of the j-th level sampling module and the weight α of the j-th level sampling module (j). Weight α (j) It consists of two parts, including the area ratio ω1 of the output / input image of the j-th level sampling module (j) , and the difference between the ratio of the sum of the gradient values of the image block with the smallest sum of gradient information and the image block with the largest sum of gradient information in all image blocks of the output j-th level and 1

[0138] Among them, M (j) The total number of image blocks output by the j-th level sampling module,

[0139] Among them, tile (j) is the gradient information in the image block output by the j-th level sampling module, and m is the sequence number of the image block output by the j-th level sampling module.

[0140] Weight ω1 (j) Represents the area ratio of the output / input image of the j-th sampling module; the smaller the area ratio, the larger the weight is assigned; the ratio of the number of image blocks output by the j-th sampling module to the sum of the number of image blocks output by all sampling modules can be directly used to achieve fast approximate calculation. Weight ω2 (j) Represents the ratio of the minimum to maximum value of the gradient information in the output image block of the current j-th layer sampling module. The larger the value, the higher the degree of information consistency of the output sampling image block, and the greater the weight given to this part of the output.

[0141] The purpose of introducing a regularization term into the loss function is to keep the model simple while minimizing the training error, while improving the generalization performance of the model and preventing overfitting.

[0142] L RZ =max{||CMLS-SAP1-AM(·)||2,||CMLS-SAP2-AM(·)||2,...,||CMLS-SAP N -AM(·)||2}, where ||CMLS-SAP j −AM(·)||2 is the second-order norm of the network weights of the attention unit of the j-th sampling module.

[0143] The structure of the pathological image accurate analysis subnetwork AAOH is as follows Figure 4 As shown, the feature separation module AAOH-FSM and the feature fusion module AAOH-FRM.

[0144] The feature separation module AAOH-FSM separates features through feature extraction units specific to image categories, alleviating the negative feature transfer generated by images of different categories; the feature re-fusion module AAOH-FRM re-fuses image features based on the channel selection mechanism, achieves accurate recognition of typical area images, and enhances the positive feature transfer of specific image categories.

[0145] The function expression of the pathological image accurate analysis sub-network AAOH is: The loss function of the pathological image accurate analysis sub-network is Loss all (·)=Loss FSM (·)+Loss FRM (·),

[0146] Among them, AAOH-FSM(·) is the action expression of the feature separation module AAOH-FSM, Loss FSM (·) is the loss function of the feature separation module AAOH-FSM, AAOH-FRM(·) is the function expression of the feature fusion module AAOH-FRM, Loss FRM (·) is the loss function of the feature re-fusion module AAOH-FRM.

[0147] The feature separation module AAOH-FSM decouples mixed features, embeds specific features of different categories, eliminates feature negative transfer, uses multiple category feature maps to obtain specific category features, and uses specific category classifiers to constrain different category features to be separable.

[0148] The action expression of the feature separation module AAOH-FSM is:

[0149] Among them, f n Represents the features of the nth image category. The category of the specific image can be obtained through the training of the feature separation module, and the image features after separation from the common features x n is a sample of the nth image category, Conv n (·) 1×1 convolution is used to extract the specific features of the nth image category, and the input and output channels are the same as f n The number of feature channels is the same as that of BN. n (·) is the batch normalization layer, ReLU is the nonlinear activation function, G n (·) is the classifier for the nth image category, which is only used in the training phase and will be discarded in the inference phase. It is a collection of features of a specific image category, and P is the total number of image categories.

[0150] Loss function of feature separation module AAOH-FSM FSM (·)for:

[0151] L refers to the cross entropy operation, Refers to the classification head for features The predicted output value of y n Refers to the nth image block of a specific category, which is GroundTruth; Pointer pair y n Find the cross entropy.

[0152] The feature re-fusion module AAOH-FRM(·) fuses feature arrays based on the channel selection mechanism, and its function expression is:

[0153] in, Represents the re-fusion feature, express τth th channels, τ∈[1,T], the total number of channels is T, Cat(·) represents the cascade of multiple feature channels, FC(·) represents the fully connected layer with output dimension equal to image category n. Finally, the fusion feature F τ Passed to the joint classifier G(·) to predict the joint classification result.

[0154] Loss function of the feature re-fusion module AAOH-FRM FRM (·) is: Loss FRM (·) = L(G(F), y), where L(·) is the cross entropy loss, y represents the true category, and G(F) represents the category predicted by the module.

[0155] The structure of the prognostic analysis subnetwork PAS(·) is as follows Figure 5 As shown, it includes a pathological image feature extraction module PAS-PFEM(·), a clinical feature extraction module PAS-CFEM(·) and a prognosis analysis module PAS-PAM(·).

[0156] The prognostic analysis subnetwork PAS(·) extracts and summarizes pathological image features image, combines clinical information clinical, and predicts the patient's prognosis The process of action is:

[0157] Among them, PAS(·) represents the output of the sub-network, θ P It is a set of trainable network parameters. The pathological image feature extraction module PAS-PFEM(·) extracts pathological image features from pathological image blocks and constructs feature aggregation.

[0158] Specifically, the pathology image feature extraction module PAS-PFEM(·) includes a heavyweight feature extraction unit (HFEU), a lightweight feature extraction unit (LFEU), and a feature aggregation unit (AGG(·). The heavyweight feature extraction unit (HFEU) is used to obtain high-order features of pathology images, while the lightweight feature extraction unit (LFEU) is used to obtain low-order features of pathology images. The feature aggregation unit (AGG(·)) first unifies the number of input channels for both high-order and low-order features of pathology images and then constructs feature aggregation by dimensional addition. The specific execution process is as follows: The pathology image feature extraction module PAS-PFEM(·) extracts pathology image features using the lightweight feature extraction units (LFEU) and the heavyweight feature extraction units (HFEU), and the feature aggregation unit (AGG(·)) constructs the feature aggregation. The simultaneous use of the lightweight feature extraction units (LFEU) and the heavyweight feature extraction units (HFEU) is intended to balance low-order and high-order features in the image and enhance feature learning capabilities.

[0159] Among them, the weight feature extraction unit HFEU is composed of trainable parameters Q HEFU A deeper (10-16 layers) deep convolutional neural network was characterized to extract high-order features in typical case image regions, and Table 3 shows a 12-layer implementation thereof.

[0160] Table 3 An embodiment of the heavyweight feature extraction unit HEFU

[0161] Layer Type 1 Conv(3,1,1,8)+ReLU() 2 Conv(3,1,1,16)+ReLU() 3 Conv(3,1,1,32)+ReLU() 4 Conv(3,1,1,16)+ReLU() 5 Conv(3,1,1,8)+ReLU() 6 Conv(3,1,1,16)+ReLU() 7 Conv(3,1,1,32)+ReLU() 8 Conv(3,1,1,16)+ReLU() 9 Conv(3,1,1,8)+ReLU() 10 Conv(3,1,1,16)+ReLU() 11 Conv(3,1,1,32)+ReLU() 12 Conv(3,1,1,1)+ReLU()

[0162] The lightweight feature extraction unit LFEU is composed of trainable parameters Q LEFU A shallower (4-8 layers) deep convolutional neural network is characterized to extract low-level features in typical case image regions, and Table 3 shows a 6-layer embodiment thereof.

[0163] Table 4 An embodiment of a lightweight feature extraction unit LEFU

[0164]

[0165]

[0166] The convolutional layers of the heavy feature extraction unit HFEU are larger than those of the light feature extraction unit LFEU.

[0167] The feature aggregation unit AGG(·) fuses the features from the lightweight feature extraction unit LFEU and the heavy feature extraction unit HFEU. It first uses a 1×1 convolution layer to compress the channels of the input features and then aggregates these features using the feature dimension addition operation. In particular, when two features of different dimensions from different modules are used as input, the deep features need to be upsampled first and then the feature dimensions are added. The AGG module structure is as follows: Figure 6 shown.

[0168] The clinical feature extraction module PAS-CFEM(·) consists of two fully connected layers with 32 neurons, which is used to extract clinical information such as age, lymph node metastasis, unilateral / bilateral tumor, postoperative FIGO stage, and whether neoadjuvant therapy was performed.

[0169] The prognostic analysis module PAS-PAM(·) cross-modally fuses pathological image features with clinical information and predicts prognostic status.

[0170] The parameter set θ of the prognostic analysis subnetwork PAS(·) P The training process can be expressed as the following minimization problem:

[0171]

[0172] in, is the loss function of the prognostic analysis subnetwork, Q is the total number of case data in the case data set, q represents the qth case data in Q, and L unfavor is the cross entropy between the predicted value and the actual state of the poor prognosis case, L favor is the cross entropy between the predicted value and the actual state of the case with good prognosis, Ω is a very small positive number, η is the weight, and adjusting η adjusts L favor 、L unfavor The default value is 1.0. P(·) is the prognostic status output by the tumor prognostic analysis system, which has two types: favor and unfavor. favor For cases with a true prognosis of favor, Y unfavor is the case with the true prognostic status of unfavor, L(·) is the cross entropy, and there are cross entropies of the two states: favor and unfavor.

[0173] In this embodiment, the configurable hierarchical sampling sub-network, the pathological image precise analysis sub-network, and the prognosis analysis sub-network are trained separately.

[0174] Prognostic analysis is performed on the pathological full-field images to be analyzed based on the trained prognostic analysis model.

[0175] To compare the performance of prognostic analysis in this example, we used recall, accuracy, and precision. The larger the mean of these metrics, the better the system performance; the smaller the standard deviation, the greater the system stability. tp and tn represent true positive and true negative samples, respectively, while fp and fn represent false positive and false negative samples, respectively.

[0176] The recall rate represents the proportion of correctly predicted positive samples to all positive samples.

[0177] Accuracy represents the proportion of samples that are predicted correctly.

[0178] Precision represents the proportion of correctly predicted positive samples to all predicted positive samples.

[0179] Comparison of prognostic analysis performance of ovarian HGSA

[0180] Table 5 shows the results of the prognostic status analysis of ovarian HGSA using the proposed method and six existing methods. In order to visually compare the performance and stability of different methods, a bar chart is used to display the analysis results, such as Figure 7a 、 Figure 7b shown.

[0181] Table 5 Experimental results of different methods

[0182]

[0183] In Table 5, the first column is the method type, and [1], [2], [3], [4], [5], and [6] represent the results obtained using the methods of reference [1], reference [2], reference [3], reference [4], reference [5], and reference [6], respectively. Among them, references [1]-[6] are the following references:

[0184] [1]Xin Liao,Liang Sun,Kaixuan Yang,et al(2018)Prognosis Evaluation ofOvarian Granulosa Cell Tumor Based on Co-forest Intelligence Model[J].Journalof Engineering Science and Technology Review,2018,11(2):135-142.https: / / doi.org / 10.25103 / jestr.112.19

[0185] [2]Xin Liao,Liang Sun,Kaixuan Yang,et al(2017)Prognostic EvaluationMethod of Ovarian Granulosa Cell Tumor Based on Semi-supervised CollaborativeIntelligence Model.Journal of Engineering Science and Technology Review,2017.10(6):96-103.https: / / doi.org / 10.25103 / jestr.106.13

[0186] [3]Babu T,Singh T,Gupta D,et al.Colon cancer prediction onhistological images using deep learning features and Bayesian optimized SVM[J].Journal of Intelligent and Fuzzy Systems,2021,221(Jun.7):1-12.https: / / doi.org / 10.1016 / j.knosys.2021.106965.

[0187] [4]Nikhilanand Arya,Sriparna Saha.Multi-modal advanced deep learningarchitectures for breast cancer survival prediction[J].Knowledge-BasedSystems,2021,221:106965.1-106965.11.https: / / doi.org / 10.1016 / j.knosys.2021.106965

[0188] [5]Tan K, Huang W, Liu

[0189] [6]Zeng H, Chen L, et al. Integration of histopathological images and multi-dimensional omics analyzes predicts molecular features and prognosis in high-grade serous ovarian cancer. Gynecol Oncol.2021,163(1):171-180.https: / / doi.org / 10.1016 / j.ygyno.2021.07.015

[0190] The second to fourth columns are recall rate, accuracy rate, and precision rate, respectively. The data items (MEAN, STD) are the mean and standard deviation of the corresponding indicators determined by three-fold cross validation. Figure 7a The height of the rectangle in represents the mean of the data set. The rectangles in the first column, the second column, and the third column represent the mean of recall rate, accuracy rate, and precision, respectively. Figure 7b The height of the rectangle in represents the standard deviation of the data set. The rectangles in the first column, the second column, and the third column represent the standard deviations of recall rate, accuracy rate, and precision, respectively.

[0191] According to Table 5 and Figure 7a It can be seen that the average recall rate, accuracy rate and precision rate of the HS-FGC method in this paper are significantly better than those of the methods in references [1] and [2], and slightly better than those in references [3], [4], [5] and [6]. That is, the method has better prognostic analysis performance. Figure 7bIt can be seen that the standard deviation of the accuracy and precision of the proposed method are significantly lower than those of the methods in references [1], [2], [3], [4], [5], and [6]; the standard deviation of the recall rate of the proposed method is significantly lower than that of the methods in references [2], [3], [4], and [6], but slightly higher than that of the methods in references [1] and [5]. Based on the above information, it can be seen that the proposed method has good stability in prognostic analysis. The methods in references [1] and [2] have obvious deficiencies in terms of comprehensiveness in processing patient information and feature mining capabilities. The image feature extraction networks of the methods in references [3], [4], and [5] are designed for other tumor types. When directly applied to ovarian HGSA, performance degradation will occur. And like the method in reference [6], the applicability of case data is not fully considered. The proposed method fully considers the diversity of ovarian HGSA case data. At the same time, we have carefully designed each module in the system and can extract key features that effectively represent the prognostic status. Its prediction results can provide an important reference for the prognostic analysis of ovarian HGSA patients.

[0192] Ablation experiments

[0193] In order to verify the necessity and effectiveness of each intermediate step in the proposed prognostic analysis system, relevant ablation experiments were designed, and the experimental results are shown in Table 6. At the same time, a bar chart was used for comparison, such as Figure 8a and Figure 8b shown.

[0194] Table 6 Ablation experiment results

[0195]

[0196]

[0197] In Table 6, the first column is the method type: HS-FGC represents the method of this paper; HS(1)-FGC represents that the method of this paper only performs single-stage sampling on the pathological image; HS(0)-FGC represents that the method of this paper only randomly selects some areas from the pathological image; HS-FGC(0) represents that the method of this paper does not use the ovarian tissue pathological image precise analysis subnetwork; HS-FGC-NC represents that the method of this paper does not use clinical information; HS-FGC(0)-NC represents that the method of this paper does not use the ovarian tissue pathological image precise analysis subnetwork and does not use clinical information. Figure 8a The height of the rectangle in represents the mean value of the group of indicators. Figure 8b The height of the rectangle in represents the standard deviation of the group of indicators.

[0198] From Table 6 and Figure 8aIt can be seen that the average values of recall rate, accuracy rate and precision rate of this method are significantly better than those of methods HS(1)-FGC, HS(0)-FGC, HS-FGC(0), HS-FGC-NC and HS-FGC(0)-NC, that is, method HS-FGC has better prognostic analysis performance. Figure 8b It can be seen that the standard deviation of the recall rate, accuracy rate and precision rate of this method is lower than that of the methods HS(1)-FGC, HS-FGC-NC and HS-FGC(0)-NC, and slightly higher than that of the HS(0)-FGC method; the standard deviation of the accuracy rate and precision rate of this method is slightly higher than that of the method HS-FGC(0). Based on the above information, it can be seen that the method of this paper has good prognostic analysis stability. The results of the ablation experiment show that the method of this paper can effectively improve the prognostic analysis performance of ovarian HGSA (obtaining a higher mean of the recall rate, accuracy rate and precision rate indicators) by screening typical areas from pathological images through configurable stratified sampling, and then performing accurate image analysis on the typical areas, combined with clinical information, and improve the analysis stability (obtaining a smaller standard deviation of the recall rate, accuracy rate and precision rate indicators). The above conclusions provide an experimental basis for the rationality and necessity of each intermediate step in the method of this paper.

[0199] Performance comparison of the prognostic analysis subnetwork using different feature extraction models

[0200] The feature extraction units LFEN and HFEN in the prognostic analysis subnetwork can use a combination of different models. In order to select the best model combination, a comparative experiment was conducted, and the experimental results are shown in Table 7. At the same time, a bar chart is used to display the experimental results, as shown in Table 7. Figure 9a and Figure 9b shown.

[0201] Table 7 Performance when selecting different feature extraction models

[0202]

[0203] In Table 7, the first column is the method type, where (RN34+RN34), (RN50+RN50), (RN101+RN101), (RN34+RN101), and (RN50+RN101) represent different combinations of LFEN and HFEN implemented using the ResNet-34, ResNet-50, and ResNet-101 models, respectively. Figure 9a The height of the rectangle in represents the mean value of the group of indicators. Figure 9b The height of the rectangle in represents the standard deviation of the group of indicators.

[0204] From Table 7 and Figure 9aIt can be seen that the average recall rate, accuracy rate, and precision rate of (RN34+RN50) are better than those of other combinations, and it has better prognostic analysis performance. Figure 9b As can be seen, the mean square error of accuracy and precision for (RN34+RN50) is significantly lower than for the other combinations; the standard deviation of recall is slightly higher than that for (RN50+RN50), but lower than for the other combinations. This information demonstrates that our method exhibits good stability in prognostic analysis. The feature extraction capability of the prognostic analysis subnetwork will affect analysis performance. This capability, in turn, depends on the specific models used in the LFEN and HFEN. These experimental results provide experimental support for using the ResNet-34 and ResNet-50 model structures as the LFEN and HFEN models.

[0205] Figure 10 The results of the proposed prognostic analysis system for six typical cases of ovarian HGSA are presented. The first column shows the full-field histopathological image of the ovarian tumor. Columns 2 through 6 provide information such as the patient's age at diagnosis, lymph node metastasis, unilateral / bilateral tumor status, postoperative FIGO stage, and whether neoadjuvant therapy was administered. Column 7 shows the actual prognostic status obtained through follow-up. Column 8 presents the analysis results of our method.

[0206] The present invention also provides a tumor prognosis analysis system, including an image receiving module, a processing module and a storage module. The image receiving module receives images for training or evaluation and sends the received images to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform operations corresponding to the above-mentioned tumor prognosis analysis method based on the image it receives.

[0207] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0208] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A tumor prognosis analysis method, characterized in that: The following steps are involved: Acquire full-field pathology images; Acquire typical areas of pathological full-field images; Inputting the typical region image into a prognostic analysis subnetwork, wherein the prognostic analysis subnetwork performs prognostic analysis based on clinical information and the accurately identified typical region image; The prognosis analysis subnetwork includes a pathology image feature extraction module, a clinical feature extraction module and a prognosis analysis module; The pathological image feature extraction module extracts pathological image features from pathological image blocks and constructs feature aggregation. The clinical feature extraction module extracts clinical information. The prognosis analysis module cross-modally fuses pathological image features with clinical information and predicts prognosis. The prognostic analysis subnetwork extracts and summarizes pathological image features, combines them with clinical information, and predicts the patient's prognosis. PAS(·) is the output of the sub-network, θ P It is a set of trainable network parameters; The parameter set θ of the prognostic analysis subnetwork P The training process adopts the minimization problem: in, is the loss function of the prognostic analysis subnetwork, Q is the total number of case data in the case data set, q represents the qth case data in Q, θ P is the parameter set of the prognostic analysis sub-network, L unfavor is the cross entropy between the predicted value and the actual state of the poor prognosis case, L favor is the cross entropy between the predicted value and the actual state of the case with good prognosis, Ω is a very small positive number, η is the weight, P(·) is the prognostic state output by the tumor prognostic analysis system, which has two types: favor and unfavor, and Y favor For cases with a true prognosis of favor, Y unfavor is the case with the true prognostic status of unfavor, L(·) is the cross entropy, and there are cross entropies of the two states: favor and unfavor.

2. The tumor prognosis analysis method according to claim 1, characterized in that: The pathological image feature extraction module includes a heavyweight feature extraction unit, a lightweight feature extraction unit and a feature aggregation unit; The weight feature extraction unit is used to obtain high-order features of the pathological image, the lightweight feature extraction unit is used to obtain shallow basic features of the pathological image, and the feature aggregation unit first unifies the number of input feature channels for the high-order features and shallow basic features of the pathological image, and then uses dimensional addition to construct feature aggregation; The weight feature extraction unit and the lightweight feature extraction unit are both deep convolutional neural networks, and the convolution layer of the weight feature extraction unit is larger than the convolution layer of the lightweight feature extraction unit.

3. The tumor prognosis analysis method according to claim 1, characterized in that: Methods for acquiring typical regions of the full field of view pathology image include: Inputting the pathological full-field image into a configurable hierarchical sampling subnetwork, wherein the configurable hierarchical sampling subnetwork is used to perform hierarchical sampling on the pathological full-field image to screen out typical areas; The obtained typical regions are input into the pathological image precise analysis subnetwork to achieve precise recognition of the typical region images.

4. The tumor prognosis analysis method according to claim 3, characterized in that: The configurable hierarchical sampling subnetwork includes cascaded multi-stage sampling modules; The first-level sampling module screens typical regions on the low-resolution image of the pathological full-field image and maps the regions onto the original-resolution pathological full-field image, thereby obtaining a plurality of original-resolution pathological image blocks with the same range as the screened regions, i.e., first-level sub-image blocks; The j-th level sampling module performs typical area screening on the low-resolution image of the sub-image block output by the sampling module of the previous level and maps it to the pathological full-field image of the original resolution, obtaining multiple pathological image blocks of the original resolution with the same area range as the currently screened area, namely, the j-th level sub-image block, where j is an integer greater than 1.

5. The tumor prognosis analysis method according to claim 4, characterized in that: The sampling modules each include a feature extraction unit, an attention weighting unit, and a classification unit; The feature extraction unit extracts features from the pathological full-field image or the sub-image block output by the previous sampling module, and the classification unit classifies the features extracted by the feature extraction unit based on the weights of the attention weighting unit to obtain sub-image blocks.

6. The tumor prognosis analysis method according to claim 4, characterized in that: The function of the j-th level sampling module is: in, is the classification function of the j-th level sampling module, is the feature extraction function of the j-th level sampling module, is the attention function of the j-th level sampling module, c (j) is the input image x of the j-th level sampling module j The ratio is S j The low-resolution sampling of ∈(0,1) corresponds to the coordinate mapping of the pathological full-field image, and C is the coordinate mapping of all c (j) The set of components; j-1th level sampling function is the input image x to the j-1th level sampling module j -1 is the ratio Sj-1 ∈(0,1) low-resolution sampling to obtain coordinates c (j-1) And c (j-1) Mapped to the j-1th level sampling module input image x j-1 middle, is the j-th level sampling function, the input image x of the j-th level sampling module j for 7. The tumor prognosis analysis method according to claim 6, characterized in that: After Monte Carlo approximation, the function of the j-th level sampling module is: Among them, M refers to the number of sub-image blocks output by each level of sampling module, c (j-1) ∈C.

8. The tumor prognosis analysis method according to claim 4, characterized in that: The loss function of the configurable layered sampling subnetwork is in, is the weighted sum of the cross entropy of sampling modules at all levels, is the cross entropy of the j-th level sampling module, α (j) is the weight of the j-th level sampling module, L RZ =max{||CMLS-SAP1-AM(·)||2,||CMLS-SAP2-AM(·)||2,......,||CMLS-SAP N -AM(·)||2}, ||CMLS-SAP j -AM(·)||2 is the second-order norm of the network weight of the attention unit of the j-th sampling module, β is the weight coefficient of the regularization term, and N is the total number of sampling modules.

9. The tumor prognosis analysis method according to claim 8, characterized in that: Weight ω1 (j) is the area ratio or quantity ratio of the output image blocks of all levels of sampling modules to the output image blocks of the jth level, ω2 (j) It is the difference between the ratio of the sum of the gradient values of the image block with the smallest sum of gradient information and the image block with the largest sum of gradient information among all image blocks output by the j-th level sampling module and 1, Among them, M (j) is the total number of output image blocks of the j-th level sampling module, Among them, tile (j) is the gradient information in the image block output by the j-th level sampling module, and m is the sequence number of the image block output by the j-th level sampling module.

10. The tumor prognosis analysis method according to claim 1, characterized in that: The pathological image precise analysis subnetwork includes a feature separation module and a feature re-fusion module; The feature separation module separates image features through a feature extraction unit specific to the image category; The feature re-fusion module re-fuses the image features based on a channel selection mechanism to achieve accurate recognition of typical area images.

11. The tumor prognosis analysis method according to claim 10, characterized in that: The feature separation module uses multiple category feature maps to obtain specific category features, and uses specific category classifiers to constrain different category features to be separable.

12. The tumor prognosis analysis method according to claim 10, characterized in that: The expression of the pathological image precise analysis subnetwork is: AAOH-FSM(·) is the action expression of the feature separation module. Among them, f n Represents the features of the nth image category, image features after separation from common features x n is a sample of the nth image category, Conv n (·) 1×1 convolution is used to extract the specific features of the nth image category, and the input and output channels are the same as f n The number of feature channels is the same, BN n (·) is the batch normalization layer, ReLU is the nonlinear activation function, G n (·) is the classifier for the nth image category, is a set of features of a specific image category, and P is the total number of image categories; AAOH-FRM(·) is the action expression of the feature re-fusion module, Represents the re-fusion feature, express τth th channels, τ∈[1,T], the total number of channels is T, Cat(·) represents the concatenation of multiple channel-level features, FC(·) represents the fully connected layer with an output dimension equal to the image category n, The fusion feature F τ Passed to the joint classifier G(·) to predict the joint classification result.

13. The tumor prognosis analysis method according to claim 10, characterized in that: The loss function of the pathological image accurate analysis sub-network is Loss all (·)=Loss FSM (·)+Loss FRM (·), Loss FSM (·) is the loss function of the feature separation module: Loss FRM (·) is the loss function of the feature re-fusion module: Loss FRM (·) = L(G(F), y), where L(·) is the cross entropy loss, y represents the true category, and G(F) represents the category predicted by the module.

14. A tumor prognosis analysis system, characterized in that: The method comprises an image receiving module, a processing module and a storage module. The image receiving module receives images for training or evaluation and sends the received images to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform an operation corresponding to the tumor prognosis analysis method according to any one of claims 1 to 13 based on the received image.

Images