A task-agnostic feature learning method based on whole pathological section images

Abstract

The present invention discloses a task-agnostic feature learning method based on whole-pathology section images, comprising the following steps: preprocessing the whole-slide pathology images to extract multi-scale features of tumor and tumor-infiltrating lymphocyte regions; constructing a task-agnostic feature learning machine, automatically optimizing the network structure through neural architecture search, and updating the architecture parameters by using the Gumbel-Softmax reparameterization technique; through a task-level meta-learning algorithm, jointly optimizing the network weights and architecture parameters to extract cross-task reusable features; and through a small amount of task-specific fine-tuning, adapting to unseen clinical tasks and verifying the model performance. By adopting the above-mentioned task-agnostic feature learning method based on whole-pathology section images, the present invention applies meta-learning to pathological image analysis for the first time, and significantly improves the generalization ability of the model for unknown tasks through automated architecture search and feature decoupling, providing efficient and scalable technical support for clinical diagnosis and prognosis analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and particularly to a task-agnostic feature learning method based on whole-pathology slide images. Background Art

[0002] Whole-Slide Images (WSIs) are the gold standard for cancer diagnosis and prognosis. In recent years, with the development of digital pathology technology, computer-aided diagnosis systems based on WSIs have gradually become a research hotspot. Multi-Task Learning (MTL) can improve the generalization ability of the model by sharing feature representations of different tasks.

[0003] Existing methods usually solve the adaptation problem of unseen tasks in the following two ways: one is to add a new task to the existing network and train from scratch, but this method ignores the learned shared features, resulting in resource waste and negative transfer problems; the other is to construct a general feature extractor, but these features are often too close to the patterns of training tasks and are difficult to adapt to new tasks.

[0004] Therefore, the existing MTL methods have poor adaptability to unseen tasks and are difficult to be directly applied to new clinical tasks. Especially when the new task is quite different from the known tasks, the performance drops significantly.

[0005] Therefore, there is an urgent need for a task-agnostic feature learning method based on whole-pathology slide images to overcome the defects of the existing technology. Summary of the Invention

[0006] The purpose of the present invention is to provide a task-agnostic feature learning method based on whole-pathology slide images, which solves the problem of poor adaptability to unseen tasks in the existing technology and is difficult to be directly applied to new clinical tasks, especially when the new task is quite different from the known tasks and the performance drops significantly.

[0007] To achieve the above purpose, the present invention provides a task-agnostic feature learning method based on whole-pathology slide images, including the following steps:

[0008] Step S1: Preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions;

[0009] Step S2: Construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and adopt the Gumbel-Softmax reparameterization technique to realize continuous update of the architecture parameters;

[0010] Step S3: Through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features;

[0011] Step S4: Adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance.

[0012] Preferably, in step S1, preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions. The specific process is as follows:

[0013] Step S11: Divide the whole-slide pathology image into non-overlapping image patches of 512×512 pixels;

[0014] Step S12: Use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocyte region;

[0015] Step S13: Calculate the area ratio of the tumor and tumor-infiltrating lymphocytes in each image patch, and select the top 100 image patches with the highest area ratio;

[0016] Step S14: Use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches;

[0017] Step S15: Through the multi-instance learning method, regard each multi-instance learning as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

[0018] Preferably, in step S2, construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and adopt the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters. The specific process is as follows:

[0019] Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of the task-agnostic feature learner;

[0020] Step S22: Based on the DARTS algorithm, combine the Gumbel-Softmax reparameterization technique to make the discrete architecture parameters continuous, achieve differentiable architecture search, and then optimize the architecture parameters;

[0021] Step S23: Stack the optimal cells obtained from the search 8 times to construct the backbone network of the task-agnostic feature learner for multi-scale feature fusion; among them, downsampling layers with a stride of 2 are inserted at the 3rd and 6th unit positions during the unit stacking.

[0022] Preferably, in step S21, the search space includes separable convolution, dilated convolution, skip connection, and Transformer nodes;

[0023] Define the search space, and design the unit structure of the network as a directed acyclic graph containing 6 nodes, specifically including:

[0024] (1) Two input nodes, used to receive the output features of the predecessor unit;

[0025] (2) Two intermediate nodes, and the supported operations include separable convolution, dilated convolution, and skip connection;

[0026] (3) One Transformer node, adopting a standard Transformer module to enhance global feature interaction;

[0027] (4) One output node, used to aggregate the features of all intermediate nodes and the Transformer node to generate the output of the unit.

[0028] Preferably, in step S22, based on the DARTS algorithm, combined with the Gumbel-Softmax reparameterization technique, the discrete architecture parameters are made continuous to achieve differentiable architecture search, and then the architecture parameters are optimized as follows:

[0029] ;

[0030] Among them, is the smoothed architecture parameter; is the categorical probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations.

[0031] Preferably, in step S3, through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features. The specific process is as follows:

[0032] Step S31: Divide the training dataset into a support set and a query set;

[0033] Step S32: In the inner loop, quickly fine-tune the specific head of each task through the support set to adapt to the requirements of different tasks;

[0034] Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features.

[0035] Preferably, in step S31, the training dataset includes: cancer subtype classification, TNM staging, and survival prediction;

[0036] Each task is randomly divided into a support set and a query set; among them, 80% of the training dataset is the support set, which is used for inner-loop fine-tuning; 20% of the training dataset is the query set, which is used for outer-loop optimization.

[0037] Preferably, in step S32, the specific head of each task is quickly fine-tuned through the support set in the inner loop to adapt to the requirements of different tasks;

[0038] In the inner loop, only the weight parameters of the model are updated, and the update formula is as follows:

[0039] ;

[0040] Where, is the network weight of task at the step; is the inner-loop learning rate; represents the gradient calculation of the weight ; is the support set loss function of task ; represents the forward calculation process of the learner of task ; is the support set of task ; is the specific head parameter of task ;

[0041] Preferably, in step S33, in the outer loop, the network weights and architecture parameters are jointly optimized through the query set to extract cross-task reusable features, and the joint optimization objective function is as follows:

[0042] ;

[0043] Where, represents the global network weight parameters; represents the global architecture parameters; is the outer-loop learning rate of the network weights; is the outer-loop learning rate of the architecture parameters; represents the gradient calculation of the joint optimization objective function ; represents the query set loss function of task ; is the query set of task ; is the network weight of task at the step; is the weight coefficient of the entropy regularization term; represents the total number of nodes in the unit; It represents the node to the node of the smoothed architecture parameters, which is a continuous probability distribution after Gumbel-Softmax reparameterization; is the entropy regularization term;

[0044] Here, an entropy regularization constraint on the architecture parameters is introduced, and its definition is as follows:

[0045] ;

[0046] Among them, represents the sampling probability of the architecture parameters, indicating the connection probability from the node to the node ;

[0047] Preferably, in step S4, through a small amount of task-specific fine-tuning, it adapts to unseen clinical tasks and verifies the model performance. The specific process is as follows:

[0048] Step S41: Fix the parameters of the task-agnostic feature learning backbone network to ensure that its parameters do not change during subsequent fine-tuning;

[0049] Step S42: Use task-specific samples to fine-tune the classification layer and update the parameters of the task-specific head;

[0050] Step S43: Output the prediction results of the model and visually display the key feature regions.

[0051] Therefore, the present invention adopts the above-mentioned task-agnostic feature learning method based on whole-pathology slide images, automatically optimizes the network structure through neural architecture search, reduces the cost of manually designing the network; extracts cross-task reusable features through task-level meta-learning algorithms, reducing the need for fine-tuning on unseen tasks; and improves the training stability and generalization ability of the model through entropy regularization constraints.

[0052] Next, through the accompanying drawings and embodiments, the technical solutions of the present invention will be further described in detail. Description of the Drawings

[0053] Figure 1 is a schematic flowchart of a task-agnostic feature learning method based on whole-pathology slide images according to the present invention;

[0054] Figure 2 is a schematic framework diagram of a task-agnostic feature learning device according to the present invention;

[0055] Figure 3 is a schematic diagram of the NAS unit structure according to the present invention. Detailed Embodiments

[0056] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0057] As Figure 1 shown, a task-agnostic feature learning method based on whole-slide pathology images includes the following steps:

[0058] Step S1, preprocess the whole-slide pathology image (WSI), extract multi-scale features of the tumor and tumor-infiltrating lymphocytes (TILs) regions, and provide high-quality input for subsequent tasks;

[0059] Step S2, construct a task-agnostic feature learner (TAFL) to automatically optimize the network structure through neural architecture search (NAS), and use the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters.

[0060] Step S3, through a task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters, extract cross-task reusable features, and significantly improve the generalization ability of the model for unseen tasks;

[0061] Step S4, through a small amount of task-specific fine-tuning, quickly adapt to unseen clinical tasks and verify the model performance.

[0062] Embodiment

[0063] Step S1, preprocess the whole-slide pathology image (WSI), extract multi-scale features of the tumor and tumor-infiltrating lymphocytes (TILs) regions, and provide high-quality input for subsequent tasks.

[0064] Step S11, divide the whole-slide pathology image into non-overlapping image patches of 512×512 pixels.

[0065] Step S12, use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocytes (TILs) region.

[0066] Step S13, calculate the area ratio of the tumor and TILs in each image patch, and select the top 100 image patches with the highest area ratio as high-information regions.

[0067] Step S14, use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches.

[0068] Step S15, through the multi-instance learning (MIL) method, regard each WSI as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

[0069] Step S2, construct a task-agnostic feature learner (TAFL) as Figure 2As shown, the network structure is automatically optimized through Neural Architecture Search (NAS), and the Gumbel-Softmax reparameterization technique is used to achieve continuous update of the architecture parameters.

[0070] Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of TAFL.

[0071] The search space includes convolutional operations, skip connections, and Transformer nodes. Define the search space, and design the unit structure of the network as a directed acyclic graph containing 6 nodes, specifically including:

[0072] (1) Two input nodes for receiving the output features of the previous unit;

[0073] (2) Two intermediate nodes, and the supported operations include separable convolution (1×3 or 1×5), dilated convolution (dilation rate = 2), and skip connections;

[0074] (3) One Transformer node, adopting a standard Transformer module (number of heads = 8, hidden layer dimension = 512) to enhance global feature interaction;

[0075] (4) One output node for pooling the features of all intermediate nodes and the Transformer node to generate the output of the unit.

[0076] Step S22: Based on the DARTS algorithm, combined with the Gumbel-Softmax reparameterization technique, make the discrete architecture parameters continuous, achieve differentiable architecture search, and then optimize the architecture parameters as follows:

[0077] ;

[0078] Among them, is the smoothed architecture parameter; is the categorical probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations.

[0079] Step S23: Stack the optimal units obtained from the search 8 times to construct the TAFL backbone network. Insert downsampling layers (stride = 2) at the 3rd and 6th unit positions to achieve multi-scale feature fusion. Among them, the neural architecture search (NAS) unit structure is as Figure 3 shown.

[0080] Step S3: Through the task-level meta-learning algorithm, namely Algorithm 1, jointly optimize the network weights and architecture parameters, extract cross-task reusable features, and significantly improve the generalization ability of the model for unseen tasks.

[0081] Step S31: Divide the training dataset.

[0082] The training dataset includes: cancer subtype classification (such as LUAD / LUSC of NSCLC), TNM staging (early / late stage), survival prediction (Cox model).

[0083] For each task, randomly divide 80% as the support set for inner-loop fine-tuning; 20% as the query set for outer-loop optimization.

[0084] Step S32: In the inner loop, quickly fine-tune the specific head of each task through the support set to adapt to the requirements of different tasks.

[0085] To accelerate training, only update the weight parameters of the model in the inner loop, and the update formula is as follows:

[0086] ;

[0087] where is the network weight of task at step ; is the inner-loop learning rate; represents the gradient calculation of the weight ; is the support set loss function of task ; represents the forward calculation process of the learner of task ; is the support set of task ; is the specific head parameter of task such as classification layer or regression layer parameters.

[0088] Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set, extract cross-task reusable features, and the joint optimization objective function is as follows:

[0089] ;

[0090] where represents the global network weight parameters; represents the global architecture parameters; is the outer-loop learning rate of the network weights; is the outer-loop learning rate of the architecture parameters; represents the gradient calculation of the joint optimization objective function Gradient calculation; Represents a task Query set loss function; Is the task Query set; Is the task At the Step network weights; Is the weight coefficient of the entropy regularization term; Represents the total number of nodes in the unit; Is to represent the node To node Smoothing architecture parameters, continuous probability distribution after Gumbel-Softmax reparameterization; Is the entropy regularization term.

[0091] Here, an entropy regularization constraint architecture parameter is introduced to reduce the gradient variance and improve the training stability. Its definition is as follows:

[0092] ;

[0093] Among them, Represents the sampling probability of the architecture parameter, representing the connection probability from node To node ;

[0094] Algorithm 1: Task-level meta-learning algorithm, as follows:

[0095] 1. Randomly initialize And ;

[0096] 2. While not converged:

[0097] For each task :

[0098] Initialize = ,[[]] ;

[0099] For To :

[0100] Calculate the support set loss ;

[0101] Update the model weights ;

[0102] Calculate the query set loss ;

[0103] Jointly update And 。

[0104] Step S4: Quickly adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance.

[0105] Step S41: Fix the parameters of the TAFL backbone network to ensure that its parameters do not change during subsequent fine-tuning.

[0106] Step S42: Fine-tune the classification layer (such as the fully connected layer) or the Cox regression layer using task-specific samples and update the parameters of these task-specific heads.

[0107] Step S43: Output the prediction results of the model, such as cancer stage, subtype classification, survival prediction, etc., and visually display the key feature regions.

[0108] Therefore, the present invention adopts the above-mentioned task-agnostic feature learning method based on whole-pathology slide images, automatically optimizes the network structure through neural architecture search, reduces the cost of manually designing the network; extracts cross-task reusable features through the task-level meta-learning algorithm, reduces the fine-tuning requirements for unseen tasks; and improves the training stability and generalization ability of the model through entropy regularization constraints.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A task-agnostic feature learning method based on whole pathological section images, characterized in that, It includes the following steps: Step S1: Preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions; Step S2: Construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and adopt the Gumbel-Softmax reparameterization technique to achieve continuous update of architecture parameters. The specific process is as follows: Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of the task-agnostic feature learner; The search space includes separable convolution, dilated convolution, skip connection, and Transformer nodes; Define the search space, and design the unit structure of the network as a directed acyclic graph containing 6 nodes, specifically including: (1) 2 input nodes for receiving the output features of the predecessor unit; (2) 2 intermediate nodes, and the supported operations include separable convolution, dilated convolution, and skip connection; (3) 1 Transformer node, using the standard Transformer module to enhance global feature interaction; (4) 1 output node for pooling the features of all intermediate nodes and Transformer nodes to generate the output of the unit; Step S22: Based on the DARTS algorithm, combined with the Gumbel-Softmax reparameterization technique, make the discrete architecture parameters continuous, achieve differentiable architecture search, and then optimize the architecture parameters, as follows: ； Among them, are the smoothed architecture parameters; is the class probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations; Step S23: Stack the optimal units obtained by the search 8 times to construct the backbone network of the task-agnostic feature learner for multi-scale feature fusion; among them, downsampling layers with a stride of 2 are inserted at the 3rd and 6th unit positions during unit stacking; Step S3: Through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features; Step S4: Through a small amount of task-specific fine-tuning, adapt to unseen clinical tasks and verify the model performance.

2. The task-agnostic feature learning method based on whole pathological section images according to claim 1, wherein In step S1, preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions. The specific process is as follows: Step S11: Divide the whole-slide pathology image into non-overlapping image patches of 512×512 pixels; Step S12: Use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocyte region; Step S13: Calculate the area ratios of the tumor and tumor-infiltrating lymphocytes in each image patch, and select the top 100 image patches with the largest area ratios; Step S14: Use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches; Step S15: Through the multi-instance learning method, regard each multi-instance learning as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

3. A task-agnostic feature learning method based on whole pathological section images according to claim 1, characterized in that In step S3, through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features. The specific process is as follows: Step S31: Divide the training data set into a support set and a query set; Step S32: In the inner loop, perform fast fine-tuning on the specific head of each task through the support set to meet the requirements of different tasks; Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features.

4. A task-agnostic feature learning method based on whole pathological section images according to claim 3, characterized in that In step S31, the training dataset includes: cancer subtype classification, TNM staging, and survival prediction; Each task is randomly divided into a support set and a query set; among them, 80% of the training dataset is the support set for inner-loop fine-tuning; 20% of the training dataset is the query set for outer-loop optimization.

5. The task-agnostic feature learning method based on whole pathology slide images according to claim 3, wherein In step S32, in the inner loop, perform fast fine-tuning on the specific head of each task through the support set to meet the requirements of different tasks; In the inner loop, only update the weight parameters of the model, and the update formula is as follows: ； Among them, is the task at the network weight of step; is the inner loop learning rate; represents the gradient calculation of the weight ; is the support set loss function of the task ; represents the forward calculation process of the learner of the task ; is the support set of the task ; is the specific head parameter of the task ; 6. The task-agnostic feature learning method based on whole pathological section images according to claim 3, wherein In step S33, in the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features, and the joint optimization objective function is as follows: ； Among them, represents the global network weight parameter; represents the global architecture parameter; is the outer-loop learning rate of the network weights; is the outer-loop learning rate of the architecture parameters; represents the gradient calculation of the joint optimization objective function ; represents the query set loss function of the task ; is the query set of the task ; is the network weight of the task at the th step; is the weight coefficient of the entropy regularization term; represents the total number of nodes in the unit; is the smoothed architecture parameter representing the node to the node , which is a continuous probability distribution after Gumbel-Softmax reparameterization; is the entropy regularization term; Here, introduce entropy regularization to constrain the architecture parameters, and its definition is as follows: ； Among them, represents the sampling probability of the architecture parameters, representing the node to the node connection probability.

7. A task-agnostic feature learning method based on whole pathological section images according to claim 1, characterized in that In step S4, adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance. The specific process is as follows: Step S41: Fix the parameters of the task-agnostic feature learning backbone network to ensure that its parameters do not change during subsequent fine-tuning; Step S42: Use task-specific samples to fine-tune the classification layer and update the parameters of the task-specific head; Step S43: Output the prediction results of the model and visualize the key feature regions.

Images