Task-independent feature learning method based on full-pathological section image

Abstract

The invention discloses a task-independent feature learning method based on a full-section pathological image, and the method comprises the following steps: carrying out the preprocessing of the full-section pathological image, and extracting the multi-scale features of a tumor and a tumor infiltration lymphocyte region; a task independent feature learning device is constructed, a network structure is automatically optimized through neural architecture search, and architecture parameters are updated by adopting a Gumbel-Softmax re-parameterization technology; through a task-level meta-learning algorithm, network weights and architecture parameters are jointly optimized, and cross-task reusable features are extracted; through specific fine tuning of a small number of tasks, the method adapts to unseen clinical tasks and verifies model performance. According to the task-independent feature learning method based on the full pathological section image, meta-learning is applied to pathological image analysis for the first time, the generalization ability of a model to unknown tasks is remarkably improved through automatic framework search and feature decoupling, and efficient and extensible technical support is provided for clinical diagnosis and prognosis analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and particularly relates to a task-agnostic feature learning method based on whole-pathology section images. Background Art

[0002] Whole-Slide Images (WSIs) are the gold standard for cancer diagnosis and prognosis. In recent years, with the development of digital pathology technology, computer-aided diagnosis systems based on WSIs have gradually become a research hotspot. Multi-Task Learning (MTL) can improve the generalization ability of a model by sharing feature representations of different tasks.

[0003] Existing methods usually solve the adaptation problem of unseen tasks in the following two ways: one is to add a new task to the existing network and train from scratch, but this method ignores the learned shared features, resulting in resource waste and negative transfer problems; the other is to construct a general feature extractor, but these features are often too close to the patterns of training tasks and are difficult to adapt to new tasks.

[0004] Therefore, the existing MTL methods have poor adaptability to unseen tasks and are difficult to be directly applied to new clinical tasks. Especially when the new task is quite different from the known tasks, the performance drops significantly.

[0005] Therefore, there is an urgent need for a task-agnostic feature learning method based on whole-pathology section images to overcome the defects of the existing technology. Summary of the Invention

[0006] The purpose of the present invention is to provide a task-agnostic feature learning method based on whole-pathology section images, so as to solve the problems in the existing technology that the adaptability to unseen tasks is poor and it is difficult to be directly applied to new clinical tasks. Especially when the new task is quite different from the known tasks, the performance drops significantly.

[0007] To achieve the above purpose, the present invention provides a task-agnostic feature learning method based on whole-pathology section images, including the following steps: Step S1: Preprocess the whole-pathology section image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions; Step S2: Construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and adopt the Gumbel-Softmax reparameterization technique to realize continuous update of the architecture parameters; Step S3: Through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features; Step S4: Adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance.

[0008] Preferably, in step S1, preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions. The specific process is as follows: Step S11: Segment the whole-slide pathology image into non-overlapping image patches of 512×512 pixels. Step S12: Use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocyte region. Step S13: Calculate the area ratios of the tumor and tumor-infiltrating lymphocytes in each image patch, and select the top 100 image patches with the highest area ratios. Step S14: Use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches. Step S15: Through the multi-instance learning method, regard each multi-instance learning as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

[0009] Preferably, in step S2, construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and use the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters. The specific process is as follows: Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of the task-agnostic feature learner. Step S22: Based on the DARTS algorithm, combined with the Gumbel-Softmax reparameterization technique, make the discrete architecture parameters continuous, achieve differentiable architecture search, and then optimize the architecture parameters. Step S23: Stack the optimal units obtained by the search 8 times to construct the backbone network of the task-agnostic feature learner for multi-scale feature fusion. Among them, downsampling layers with a stride of 2 are inserted at the 3rd and 6th unit positions during the unit stacking.

[0010] Preferably, in step S21, the search space includes separable convolution, dilated convolution, skip connection, and Transformer nodes. Define the search space and design the unit structure of the network as a directed acyclic graph containing 6 nodes, specifically including: (1) 2 input nodes for receiving the output features of the previous unit. (2) 2 intermediate nodes, and the supported operations include separable convolution, dilated convolution, and skip connection. (3) One Transformer node, adopting a standard Transformer module to enhance global feature interaction; (4) One output node, used to aggregate the features of all intermediate nodes and Transformer nodes to generate the output of the unit.

[0011] Preferably, in step S22, based on the DARTS algorithm and combined with the Gumbel-Softmax reparameterization technique, the discrete architecture parameters are made continuous to achieve differentiable architecture search, and then the architecture parameters are optimized as follows: ; Among them, are the smoothed architecture parameters; is the class probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations.

[0012] Preferably, in step S3, through the task-level meta-learning algorithm, the network weights and architecture parameters are jointly optimized to extract cross-task reusable features. The specific process is as follows: Step S31: Divide the training dataset into a support set and a query set; Step S32: In the inner loop, quickly fine-tune the specific head of each task through the support set to adapt to the requirements of different tasks; Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features.

[0013] Preferably, in step S31, the training dataset includes: cancer subtype classification, TNM staging, and survival prediction; Each task randomly divides the support set and the query set; among them, 80% of the training dataset is the support set for inner loop fine-tuning; 20% of the training dataset is the query set for outer loop optimization.

[0014] Preferably, in step S32, in the inner loop, quickly fine-tune the specific head of each task through the support set to adapt to the requirements of different tasks; In the inner loop, only update the weight parameters of the model. The update formula is as follows: ; Among them, is the network weight of task at the th step; is the inner-loop learning rate; represents the gradient calculation for the weights ; is the support set loss function of the task ; represents the forward calculation process of the task learner; is the support set of the task ; is the specific head parameter of the task ;

[0015] Preferably, in step S33, the outer loop jointly optimizes the network weights and architecture parameters through the query set, extracts cross-task reusable features, and jointly optimizes the objective function as follows: ; where represents the global network weight parameter; represents the global architecture parameter; is the outer-loop learning rate of the network weights; is the outer-loop learning rate of the architecture parameters; represents the gradient calculation for the jointly optimized objective function ; represents the query set loss function of the task ; is the query set of the task ; is the network weight of the task at the th step; is the weight coefficient of the entropy regularization term; represents the total number of nodes in the unit; is the smoothed architecture parameter representing the node to the node after reparameterization by Gumbel-Softmax, a continuous probability distribution; is the entropy regularization term.

[0016] Here, an entropy regularization constraint on the architecture parameters is introduced, and its definition is as follows: ; where represents the sampling probability of the architecture parameters, indicating the connection probability from node to node ;

[0017] Preferably, in step S4, through a small amount of task-specific fine-tuning, the model is adapted to unseen clinical tasks and the model performance is verified. The specific process is as follows: Step S41: Fix the parameters of the task-agnostic feature learner backbone network to ensure that its parameters do not change during subsequent fine-tuning; Step S42: Fine-tune the classification layer using task-specific samples to update the parameters of the task-specific head; Step S43: Output the prediction results of the model and visually display the key feature regions.

[0018] Therefore, the present invention adopts the above-mentioned task-agnostic feature learning method based on whole-pathology slide images, automatically optimizes the network structure through neural architecture search, reduces the cost of manually designing the network; extracts cross-task reusable features through the task-level meta-learning algorithm, and reduces the fine-tuning requirements for unseen tasks; and improves the training stability and generalization ability of the model through entropy regularization constraints.

[0019] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. Description of the Drawings

[0020] Figure 1 It is a schematic flowchart of a task-agnostic feature learning method based on whole-pathology slide images of the present invention; Figure 2 It is a schematic framework diagram of the task-agnostic feature learner of the present invention; Figure 3 It is a schematic diagram of the NAS unit structure of the present invention. Detailed Embodiments

[0021] The technical solution of the present invention will be further described below with reference to the drawings and embodiments.

[0022] As Figure 1 shown, a task-agnostic feature learning method based on whole-pathology slide images includes the following steps: Step S1: Preprocess the whole-slide pathology image (WSI) to extract multi-scale features of the tumor and tumor-infiltrating lymphocytes (TILs) region, providing high-quality input for subsequent tasks; Step S2: Construct a task-agnostic feature learner (TAFL) to automatically optimize the network structure through neural architecture search (NAS), and use the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters.

[0023] Step S3: Through the task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters, extract cross-task reusable features, and significantly improve the generalization ability of the model for unseen tasks; Step S4: Quickly adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance.

[0024] Embodiment Step S1: Preprocess the whole-slide pathology image (WSI), extract multi-scale features of the tumor and tumor-infiltrating lymphocytes (TILs) regions, and provide high-quality input for subsequent tasks.

[0025] Step S11: Divide the whole-slide pathology image into non-overlapping image patches of 512×512 pixels.

[0026] Step S12: Use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocytes (TILs) region.

[0027] Step S13: Calculate the area ratios of the tumor and TILs in each image patch, and select the top 100 image patches with the highest area ratios as high-information regions.

[0028] Step S14: Use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches.

[0029] Step S15: Through the multi-instance learning (MIL) method, regard each WSI as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

[0030] Step S2: Construct a task-agnostic feature learner (TAFL) as Figure 2 shown, automatically optimize the network structure through neural architecture search (NAS), and use the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters.

[0031] Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of TAFL.

[0032] The search space includes convolutional operations, skip connections, and Transformer nodes. Define the search space, and design the unit structure of the network as a directed acyclic graph with 6 nodes, specifically including: (1) Two input nodes for receiving the output features of the predecessor units; (2) Two intermediate nodes, and the supported operations include separable convolution (1×3 or 1×5), dilated convolution (dilation rate = 2), and skip connections; (3) One Transformer node, using a standard Transformer module (number of heads = 8, hidden layer dimension = 512) to enhance global feature interaction; (4) One output node for pooling the features of all intermediate nodes and Transformer nodes to generate the output of the unit.

[0033] Step S22: Based on the DARTS algorithm and combined with the Gumbel-Softmax reparameterization technique, the discrete architecture parameters are made continuous to achieve differentiable architecture search, and then the architecture parameters are optimized as follows: ; Among them, are the smoothed architecture parameters; is the categorical probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations.

[0034] Step S23: Stack the optimal cells obtained from the search 8 times to construct the TAFL backbone network. Downsampling layers (stride = 2) are inserted at the 3rd and 6th cell positions to achieve multi-scale feature fusion. Among them, the neural architecture search (NAS) cell structure is as Figure 3 shown.

[0035] Step S3: Through the task-level meta-learning algorithm, i.e., Algorithm 1, jointly optimize the network weights and architecture parameters to extract cross-task reusable features, significantly improving the generalization ability of the model for unseen tasks.

[0036] Step S31: Divide the training dataset.

[0037] The training dataset includes: cancer subtype classification (such as LUAD / LUSC of NSCLC), TNM staging (early / late stage), survival prediction (Cox model).

[0038] For each task, 80% is randomly divided as the support set for inner-loop fine-tuning; 20% is the query set for outer-loop optimization.

[0039] Step S32: In the inner loop, quickly fine-tune the specific head of each task through the support set to adapt to the requirements of different tasks.

[0040] To accelerate training, only the weight parameters of the model are updated in the inner loop, and the update formula is as follows: ; Among them, is the network weight of task at the th step; is the inner-loop learning rate; represents the gradient calculation for the weight ; is the support set loss function of task ; Represents the forward calculation process of the learner for the task ; Is the support set of the task ; Is the specific head parameter (such as classification layer or regression layer parameter) of the task ;

[0041] Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set, extract cross-task reusable features, and jointly optimize the objective function as follows: ; Wherein, Represents the global network weight parameter; Represents the global architecture parameter; Is the outer loop learning rate of the network weights; Is the outer loop learning rate of the architecture parameters; Represents the gradient calculation of the jointly optimized objective function ; Represents the query set loss function of the task ; Is the query set of the task ; Is the network weight of the task At the step; Is the weight coefficient of the entropy regularization term; Represents the total number of nodes in the unit; Is to represent the node To node The smoothed architecture parameter, a continuous probability distribution after Gumbel-Softmax reparameterization; Is the entropy regularization term.

[0042] Here, introduce the entropy regularization to constrain the architecture parameters, reduce the gradient variance, and improve the training stability. Its definition is as follows: ; Wherein, Represents the sampling probability of the architecture parameters, representing the connection probability from node To node ;

[0043] Algorithm 1: Task-level meta-learning algorithm, as follows: 1. Randomly initialize and ; 2. while not converged: for each task : Initialization , ; For to : Calculate the support set loss ; Update the model weights ; Calculate the query set loss ; Joint update and .

[0044] Step S4: Quickly adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance.

[0045] Step S41: Fix the parameters of the TAFL backbone network to ensure that its parameters do not change during subsequent fine-tuning.

[0046] Step S42: Fine-tune the classification layer (such as the fully connected layer) or the Cox regression layer using task-specific samples and update the parameters of these task-specific heads.

[0047] Step S43: Output the prediction results of the model, such as cancer staging, subtype classification, survival prediction, etc., and visually display the key feature regions.

[0048] Therefore, the present invention adopts the above-mentioned task-agnostic feature learning method based on whole-pathology slide images, automatically optimizes the network structure through neural architecture search, reduces the cost of manually designing the network; extracts cross-task reusable features through the task-level meta-learning algorithm, reduces the need for fine-tuning of unseen tasks; and improves the training stability and generalization ability of the model through entropy regularization constraints.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A task-agnostic feature learning method based on whole pathology slide images, characterized in that, It includes the following steps: Step S1: Preprocess the whole-slide pathology image and extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions; Step S2: Construct a task-agnostic feature learner, automatically optimize the network structure through neural architecture search, and adopt the Gumbel-Softmax reparameterization technique to achieve continuous update of the architecture parameters; Step S3: Through a task-level meta-learning algorithm, jointly optimize the network weights and architecture parameters to extract cross-task reusable features; Step S4: Through a small amount of task-specific fine-tuning, adapt to unseen clinical tasks and verify the model performance.

2. The task-agnostic feature learning method based on whole pathological section images according to claim 1, wherein In Step S1, the whole-slide pathology image is preprocessed to extract multi-scale features of the tumor and tumor-infiltrating lymphocyte regions. The specific process is as follows: Step S11: Divide the whole-slide pathology image into non-overlapping image patches of 512×512 pixels; Step S12: Use a pre-trained U-Net model to perform semantic segmentation on the image patches to identify the tumor region and the tumor-infiltrating lymphocyte region; Step S13: Calculate the area ratio of the tumor and tumor-infiltrating lymphocytes in each image patch, and select the top 100 image patches with the largest area ratio; Step S14: Use the ResNet-50 model pre-trained on ImageNet to extract the deep features of the selected image patches; Step S15: Through the multi-instance learning method, regard each multi-instance learning as a bag, and the image patches it contains as instances for subsequent clinical task prediction.

3. A task-agnostic feature learning method based on whole pathological section images according to claim 1, characterized in that In Step S2, a task-agnostic feature learner is constructed. Through neural architecture search, the network structure is automatically optimized, and the Gumbel-Softmax reparameterization technique is adopted to achieve continuous update of the architecture parameters. The specific process is as follows: Step S21: Adopt the search strategy of the DARTS algorithm based on gradient optimization to achieve continuous architecture parameter sampling and automatically optimize the network topology of the task-agnostic feature learner; Step S22: Based on the DARTS algorithm, combined with the Gumbel-Softmax reparameterization technique, make the discrete architecture parameters continuous, realize differentiable architecture search, and then optimize the architecture parameters; Step S23: Stack the optimal units obtained by the search 8 times to construct the backbone network of the task-agnostic feature learner for multi-scale feature fusion; among them, downsampling layers with a stride of 2 are inserted at the 3rd and 6th unit positions during unit stacking.

4. A task-agnostic feature learning method based on whole pathological section images according to claim 3, characterized in that, In Step S21, the search space includes separable convolution, dilated convolution, skip connection, and Transformer nodes; Define the search space and design the unit structure of the network as a directed acyclic graph containing 6 nodes, specifically including: (1) 2 input nodes for receiving the output features of the previous unit; (2) 2 intermediate nodes, and the supported operations include separable convolution, dilated convolution, and skip connection; (3) 1 Transformer node, using the standard Transformer module to enhance global feature interaction; (4) 1 output node for aggregating the features of all intermediate nodes and Transformer nodes to generate the output of the unit.

5. A task-agnostic feature learning method based on whole pathology slide images according to claim 3, characterized in that In step S22, based on the DARTS algorithm and combined with the Gumbel-Softmax reparameterization technique, the discrete architecture parameters are made continuous to achieve differentiable architecture search, and then the architecture parameters are optimized as follows: ； Among them, are the smoothed architecture parameters; is the class probability of the th candidate operation; is the noise sampled from the Gumbel distribution; is the temperature parameter; represents the candidate operation index; represents the total number of candidate operations.

6. The task-agnostic feature learning method based on whole pathology slide images according to claim 1, wherein In step S3, through the task-level meta-learning algorithm, the network weights and architecture parameters are jointly optimized to extract cross-task reusable features. The specific process is as follows: Step S31: Divide the training dataset into a support set and a query set. Step S32: In the inner loop, quickly fine-tune the specific head of each task through the support set to meet the requirements of different tasks. Step S33: In the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features.

7. A task-agnostic feature learning method based on whole pathological section images according to claim 6, wherein In step S31, the training dataset includes cancer subtype classification, TNM staging, and survival prediction. The support set and the query set are randomly divided for each task; among them, 80% of the training dataset is the support set for inner-loop fine-tuning, and 20% of the training dataset is the query set for outer-loop optimization.

8. A task-agnostic feature learning method based on whole pathology slide images according to claim 6, characterized in that In step S32, in the inner loop, quickly fine-tune the specific head of each task through the support set to meet the requirements of different tasks. In the inner loop, only the weight parameters of the model are updated, and the update formula is as follows: ； Among them, is the task at the step network weight; is the inner loop learning rate; represents the gradient calculation of the weight ; is the task support set loss function; represents the forward calculation process of the task learner; is the task support set; is the task specific head parameter.

9. A task-agnostic feature learning method based on whole pathological section images according to claim 6, wherein, In step S33, in the outer loop, jointly optimize the network weights and architecture parameters through the query set to extract cross-task reusable features, and the joint optimization objective function is as follows: ； Among them, represents the global network weight parameter; represents the global architecture parameter; is the outer-loop learning rate of the network weights; is the outer-loop learning rate of the architecture parameters; represents the gradient calculation of the joint optimization objective function ; represents the query set loss function of task ; is the query set of task ; is the network weight of task at the th step; is the weight coefficient of the entropy regularization term; represents the total number of nodes in the unit; is the smoothed architecture parameter representing node to node , which is a continuous probability distribution after Gumbel-Softmax reparameterization; is the entropy regularization term; Here, an entropy regularization constraint is introduced for the architecture parameters, and its definition is as follows: ； Among them, represents the sampling probability of architecture parameters, representing the node to node connection probability.

10. A task-agnostic feature learning method based on whole pathology slide images according to claim 1, wherein, In step S4, adapt to unseen clinical tasks through a small amount of task-specific fine-tuning and verify the model performance. The specific process is as follows: Step S41: Fix the parameters of the task-agnostic feature learning backbone network to ensure that its parameters do not change during subsequent fine-tuning. Step S42: Use task-specific samples to fine-tune the classification layer and update the parameters of the task-specific head. Step S43: Output the prediction results of the model and visually display the key feature regions.

Images