Pathological image analysis model training method and device, equipment and storage medium

Abstract

The invention discloses a pathological image analysis model training method and device, equipment and a storage medium, and the method comprises the steps: constructing an initial pathological image analysis model which comprises a shared aggregator, a first expert network, a second expert network and a consistency constraint unit; respectively processing the original distribution data and the weight balance distribution data through a sharing aggregator, and generating a corresponding first image embedding representation and a corresponding second image embedding representation; respectively inputting the first image embedded representation and the second image embedded representation into a first expert network and a second expert network for processing to obtain a first prediction result and a second prediction result; determining a consistency constraint loss through a consistency constraint unit based on the first prediction result and the second prediction result; and performing parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model. Compared with the prior art, the tail pathology category identification accuracy of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for training a pathological image analysis model. Background Technology

[0002] With the rapid development of digital pathology, computer-aided analysis methods based on whole-slide images (WSI) have become important tools for cancer classification, grading, and rare lesion detection. Existing technologies typically employ a multiple instance learning (MIL) framework, which divides high-resolution WSIs into several image patches, aggregates them into single-slide-level embeddings after feature encoding, and finally completes diagnostic prediction through a classifier.

[0003] However, real-world clinical data often exhibits an extreme long-tail distribution: common lesions (such as Gleason grade 3 prostate adenocarcinoma without lymph node metastasis) have a large sample size, while rare lesions (such as Gleason grade 5 and isolated tumor cell ITC) account for less than 4%. Traditional MIL models perform well in head categories, but their accuracy in identifying tail categories is often below 30%, making it difficult to meet the clinical need for accurate diagnosis of rare subtypes.

[0004] Therefore, there is an urgent need for a training method for pathological image analysis models that can effectively improve the accuracy of the model in identifying tail pathological categories. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for training a pathological image analysis model, aiming to solve the technical problem of low accuracy in identifying tail pathological categories in existing pathological image analysis models.

[0006] To achieve the above objectives, the present invention provides a method for training a pathological image analysis model, the method comprising the following steps: An initial pathological image analysis model is constructed, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and the rebalanced distributed data respectively to generate corresponding first image embedding representations and second image embedding representations. The first image embedding representation and the second image embedding representation are respectively input into the first expert network and the second expert network for processing to obtain the first prediction result and the second prediction result; Based on the first prediction result and the second prediction result, the consistency constraint loss is determined by the consistency constraint unit. The parameters of the initial pathological image analysis model are optimized based on the consistency constraint loss to obtain the target pathological image analysis model.

[0007] Optionally, the initial pathological image analysis model further includes a feature extraction module. Before the step of processing the original distribution data and rebalanced distribution data through the shared aggregator to generate the corresponding first image embedding representation and second image embedding representation, the model further includes: Obtain whole-section pathological images and segment the whole-section pathological images to obtain multiple image blocks; The feature extraction module is used to extract features from each of the image blocks to generate an instance-level embedding set; The instance-level embedding set is subjected to original distribution sampling and inverse frequency rebalancing sampling respectively to obtain original distribution data and rebalanced distribution data.

[0008] Optionally, the step of determining the consistency constraint loss based on the first prediction result and the second prediction result through the consistency constraint unit includes: The consistency constraint unit inputs the first image embedding representation and the second image embedding representation into the second expert network and the first expert network, respectively, to obtain the third prediction result and the fourth prediction result. Determine a first difference between the first prediction result and the fourth prediction result, and a second difference between the second prediction result and the third prediction result; The consistency constraint loss is determined based on the first difference and the second difference.

[0009] Optionally, the method further includes: The first image embedding representation and the second image embedding representation are respectively input into the adapter for spatial mapping to obtain the corresponding first mapping embedding and second mapping embedding; A learnable prompting engine generates text feature vectors corresponding to the pathological categories of the whole-slice pathological images; The similarity between the first mapping embedding and the second mapping embedding and the text feature vector is determined, and the distillation loss is determined based on the similarity.

[0010] Optionally, the step of generating a text feature vector corresponding to the pathological category of the whole-slice pathological image through a learnable prompting engine includes: Obtain the embedding vector of the preset category description text, and initialize a trainable parameter matrix with the same dimension as the embedding vector; The embedding vector is concatenated with the trainable parameter matrix to form a dynamic prompt vector; The dynamic cue vector is input into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole-slice pathological image.

[0011] Optionally, the step of optimizing the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model includes: Obtain the true labels of the whole-slice pathological images, and determine the standard cross-entropy loss of the first prediction result and the second prediction result with respect to the true labels; The classification loss is determined based on the standard cross-entropy loss, and the classification loss, distillation loss, and consistency constraint loss are weighted and summed to obtain the total loss. Using the backpropagation algorithm, the parameters of the shared aggregator, the first expert network, and the second expert network in the initial pathological image analysis model are optimized based on the total loss to obtain the target pathological image analysis model.

[0012] Optionally, after the step of optimizing the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model, the method further includes: The whole-section pathological image to be analyzed is segmented to obtain multiple image blocks to be analyzed. The feature extraction module in the target pathological image analysis model is used to extract features from each of the image blocks to be analyzed, generating a set of feature vectors. The set of feature vectors is input into the shared aggregator in the target pathological image analysis model to obtain the inference image embedding representation; The inference image embedding representation is input into the first expert network in the target pathological image analysis model to obtain the pathological classification result.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a pathological image analysis model training device, the device comprising: The model building module is used to build an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The data processing module is used to process the original distributed data and the rebalanced distributed data through the shared aggregator to generate corresponding first image embedding representations and second image embedding representations. The result output module is used to input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing to obtain a first prediction result and a second prediction result. The loss determination module is used to determine the consistency constraint loss based on the first prediction result and the second prediction result through the consistency constraint unit. The parameter optimization module is used to optimize the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a pathological image analysis model training device, the device comprising: a memory, a processor, and a pathological image analysis model training program stored in the memory and executable on the processor, the pathological image analysis model training program being configured to implement the steps of the pathological image analysis model training method described above.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a pathological image analysis model training program, wherein the pathological image analysis model training program, when executed by a processor, implements the steps of the pathological image analysis model training method described above.

[0016] This invention discloses the construction of an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and rebalanced distributed data respectively, generating corresponding first and second image embedding representations. The first and second image embedding representations are then input into the first and second expert networks for processing, respectively, to obtain a first prediction result and a second prediction result. Based on the first and second prediction results, the consistency constraint unit determines a consistency constraint loss. Based on the consistency constraint loss, the parameters of the initial pathological image analysis model are optimized to obtain a target pathological image analysis model. Because this invention processes the original distributed data and rebalanced distributed data in parallel using the first and second expert networks, and determines the consistency constraint loss based on the first and second prediction results using the consistency constraint unit, compared to existing technologies, this invention effectively improves the accuracy of the model in identifying tail pathological categories. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis model training method of the present invention; Figure 2 This is a flowchart illustrating the second embodiment of the pathological image analysis model training method of the present invention; Figure 3 This is a flowchart illustrating the third embodiment of the pathological image analysis model training method of the present invention; Figure 4This is a structural block diagram of the first embodiment of the pathological image analysis model training device of the present invention; Figure 5 This is a schematic diagram of the structure of a pathological image analysis model training device in the hardware operating environment involved in the embodiments of the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] This invention provides a method for training a pathological image analysis model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis model training method of the present invention.

[0021] In this embodiment, the pathological image analysis model training method includes steps S10 to S50: Step S10: Construct an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit.

[0022] It should be noted that the executing entity in this embodiment can be a computer server device with data processing, network communication, and program execution functions applied in pathological image analysis scenarios, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the above functions (such as a pathological image analysis model training device). The following uses a system containing a pathological image analysis model training device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0023] It should be explained that the shared aggregator can be a Transformer aggregation network with shared parameters. The output of the shared aggregator is connected to the input of the first expert network and the second expert network respectively, forming a "two-stream-two-head" structure.

[0024] It should be noted that the initial pathological image analysis model also includes a feature extraction module, and before step S10, steps S01 to S03 are also included: Step S01: Obtain a whole-section pathological image and segment the whole-section pathological image to obtain multiple image blocks.

[0025] Step S02: Use the feature extraction module to extract features from each of the image blocks to generate an instance-level embedding set.

[0026] Step S03: Perform original distribution sampling and inverse frequency rebalancing sampling on the instance-level embedding set to obtain original distribution data and rebalanced distribution data.

[0027] Understandably, a whole-slide pathological image can refer to a single high-resolution digital image obtained by scanning a complete pathological tissue section (including the field of view of the coverslip) in one go using a digital slide scanner, and is usually stored in WSI (Whole Slide Image) format.

[0028] It should be explained that the instance-level embedding set can refer to a fixed-dimensional vector set obtained by performing forward reasoning on each image block after the whole slice pathological image is segmented into several image blocks. Each vector in the set is called an instance embedding, corresponding to a local region on the whole slice pathological image.

[0029] Understandably, the aforementioned feature extraction module can be a feature encoder (such as the CONCH model) that has been pre-trained on a large number of pathological images.

[0030] It should be noted that the original distribution sampling can refer to random sampling in the training batch construction stage according to the class frequency of the whole slice pathological images in the natural dataset, so that the probability of rare class samples being sampled is low and the probability of common class samples being sampled is high, thereby maintaining the "long tail" distribution characteristics.

[0031] Furthermore, inverse frequency rebalancing sampling refers to repeatedly sampling the instance-level embedding set after inverse frequency weighting by category: categories with smaller sample sizes receive higher sampling weights, so that the frequency of each category appearing within a training period tends to be consistent, thereby alleviating the problem of insufficient training signals for tail categories. The sampling probability of inverse frequency rebalancing sampling... ,in, This indicates the number of samples in the original data for a particular category. For example, if a rare category has only 10 samples ( Its sampling probability is The common category has 1000 samples ( ), sampling probability is In this way, the number of samples of each type in the rebalanced distribution data tends to be balanced, solving the problem of rare categories being ignored in the original long-tail distribution.

[0032] Understandably, raw distribution data can refer to training batch data formed after sampling from the raw distribution, with the same class proportions as naturally collected datasets, used to preserve true clinical distribution information. Rebalanced distribution data can refer to training batch data formed after inverse frequency rebalancing sampling, with nearly uniform class proportions, used to enhance the model's ability to discriminate tail classes.

[0033] Step S20: The original distributed data and the rebalanced distributed data are processed by the shared aggregator to generate the corresponding first image embedding representation and second image embedding representation.

[0034] It should be noted that the core function of the shared aggregator is to take a comprehensive view of all local features and intelligently allocate different attention weights, ultimately merging them into a comprehensive feature vector that can represent the entire slice.

[0035] It needs to be explained that the first image embedding representation ( () can refer to a single vector output by a shared aggregator after performing self-attention aggregation on instance-level embedding sequences in the original distributed data, used to characterize the global features of the entire slice from the perspective of the original long-tailed distribution.

[0036] Accordingly, the second image embedding representation ( () can refer to another single vector output by the same shared aggregator after performing self-attention aggregation on instance-level embedding sequences in rebalanced distributed data, which is used to characterize the global features of the entire slice from the perspective of class balance.

[0037] Step S30: Input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing to obtain the first prediction result and the second prediction result.

[0038] It should be noted that the first expert network and the second expert network can be two independent fully connected classifiers.

[0039] It should be understood that the first expert network ( ) can be dedicated to processing the first image embedding representation ( A learnable classifier whose parameters are updated only in the original distribution data path, outputting a class logits vector (i.e., the first prediction result) corresponding to the original distribution perspective. Second expert network ( ) can be dedicated to processing second image embedding representations ( A learnable classifier, with the same structure as the first expert network but independent parameters, is updated only in the rebalanced distribution data path, outputting a class logits vector corresponding to the rebalanced distribution perspective (i.e., the second prediction result). ).

[0040] Step S40: Based on the first prediction result and the second prediction result, determine the consistency constraint loss through the consistency constraint unit.

[0041] It should be noted that the above-mentioned consistency constraint unit can be implemented by exchanging the expert network with the input distribution and calculating the mean square error (MSE) of its output logic value.

[0042] In the specific implementation, step S40 includes steps S401 to S403: Step S401: Through the consistency constraint unit, the first image embedding representation and the second image embedding representation are respectively input to the second expert network and the first expert network to obtain the third prediction result and the fourth prediction result.

[0043] Step S402: Determine the first difference between the first prediction result and the fourth prediction result and the second difference between the second prediction result and the third prediction result.

[0044] Step S403: Determine the consistency constraint loss based on the first difference and the second difference.

[0045] It should be noted that the third prediction result can be the category logits vector obtained by feeding the first image embedding representation into the second expert network, used to measure the response of the original distribution features to the rebalanced expert. The fourth prediction result can be the category logits vector obtained by feeding the second image embedding representation into the first expert network, used to measure the response of the rebalanced distribution features to the original expert.

[0046] It should be understood that the first difference can refer to the element-wise difference between the fourth prediction result and the first prediction result (both from the perspective of the original distribution), usually taken as the L2 norm or MSE, quantifying the discriminative consistency bias between the two experts on the rebalanced feature. The second difference can refer to the element-wise difference between the third prediction result and the second prediction result (both from the perspective of the rebalanced distribution), quantifying the discriminative consistency bias between the two experts on the original feature.

[0047] It needs to be explained that the consistency constraint loss The corresponding formula is as follows: ; In the formula, These are the first and second image embedding representations generated by the shared aggregator, respectively. These are the first expert network and the second expert network, respectively. It is the square of the L2 norm (i.e., the mean square error MSE). It is used for constraints In processing balanced data (i.e., the second image embedding representation), When ), its output should be the same as The baseline output is consistent; It is used for constraints In processing the raw data (i.e., the first image embedding representation), When ), its output should be the same as The baseline output is consistent.

[0048] It should be noted that the aforementioned consistency constraint loss forces the two expert networks to make consistent judgments on the same content, enabling the shared aggregator to learn more generalized feature representations that are independent of distribution.

[0049] Step S50: Optimize the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model.

[0050] It should be noted that the above parameter optimization can refer to the iterative process of using the backpropagation algorithm to calculate gradients and update all learnable parameters (e.g., shared aggregator, first expert network, second expert network, etc.) in the initial pathological image analysis model with the goal of minimizing consistency constraint loss (joint classification loss, distillation loss).

[0051] This embodiment discloses the construction of an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and rebalanced distributed data respectively, generating corresponding first and second image embedding representations. The first and second image embedding representations are then input into the first and second expert networks for processing, respectively, to obtain a first prediction result and a second prediction result. Based on the first and second prediction results, the consistency constraint unit determines a consistency constraint loss. Based on the consistency constraint loss, the parameters of the initial pathological image analysis model are optimized to obtain a target pathological image analysis model. Because this embodiment processes the original distributed data and rebalanced distributed data in parallel using the first and second expert networks, and determines the consistency constraint loss based on the first and second prediction results using the consistency constraint unit, compared to existing technologies, this embodiment effectively improves the accuracy of the model in identifying tail pathological categories.

[0052] refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the pathological image analysis model training method of the present invention.

[0053] Based on the first embodiment described above, in this embodiment, the method further includes steps S404 to S406: Step S404: Input the first image embedding representation and the second image embedding representation into the adapter for spatial mapping to obtain the corresponding first mapping embedding and second mapping embedding.

[0054] Step S405: Generate a text feature vector corresponding to the pathological category of the whole-slice pathological image using a learnable prompting engine.

[0055] Step S406: Determine the similarity between the first mapping embedding and the second mapping embedding and the text feature vector, and determine the distillation loss based on the similarity.

[0056] It should be noted that the adapter can be a trainable subnetwork consisting of a "multilayer perceptron (MLP) + linear projection layer" to map the image embedding space (e.g., 1024 dimensions) to the text embedding space (e.g., 768 dimensions), eliminating cross-modal dimensional and distribution differences, so that subsequent similarity calculations can be performed in the same space.

[0057] Understandably, the first mapping embedding represents a vector transformed by the adapter, with the same dimension as the text feature vector, used for comparison with the text-side features. Correspondingly, the second image embedding represents a vector transformed by the same adapter.

[0058] In the specific implementation, step S405 includes steps S4051 to S4053: Step S4051: Obtain the embedding vector of the preset category description text, and initialize a trainable parameter matrix with the same dimension as the embedding vector.

[0059] Step S4052: Concatenate the embedding vector with the trainable parameter matrix to form a dynamic cue vector.

[0060] Step S4053: Input the dynamic prompt vector into the pre-trained text encoder to obtain the text feature vector corresponding to the pathological category of the whole slice pathological image.

[0061] It should be explained that the learnable suggestion engine can be a dynamic suggestion generation module composed of "embedded vectors of pre-defined category description texts + a trainable parameter matrix". No manual annotation is required, and the parameters are updated synchronously with the network during training. The embedded vectors of the pre-defined category description texts can refer to static word embedding vectors obtained by converting pre-set, non-trained category description strings in one go using a pre-trained text encoder (such as CONCH). This vector is only used as a template in subsequent processes and concatenated with the trainable parameter matrix; the weights are no longer updated.

[0062] It should be understood that the pathological category of a whole-section pathological image can be the diagnostic label to which the section belongs, such as "Gleason grade 5 prostate adenocarcinoma" or "breast micrometastasis", which is used to index the corresponding preset category description text.

[0063] It should be noted that dynamic cue vectors can refer to cue vectors formed by concatenating the embedding vector with the trainable parameter matrix bit by bit, and updated in real time with the gradient during training. Their parameters change continuously during backpropagation, and they can produce text features with different semantics for the same category.

[0064] It should be understood that a pre-trained text encoder can refer to a text transformation network that has been pre-trained on a large-scale general corpus or medical text, with fixed weights and does not participate in the update of downstream tasks. It is used to map dynamic cue vectors into high-dimensional semantic vectors, which serve as teacher signals in the distillation loss, so that image-side features can absorb the medical prior knowledge implied by the text modality with zero annotation through similarity alignment.

[0065] It should be explained that the distillation loss mentioned above can be the cross-entropy loss, which measures the cosine similarity deviation between the "global features of the image after adapter mapping" and the "text features generated by the learnable cue engine." Its calculation process completely mimics the "teacher-student" alignment method in knowledge distillation, except that the teacher signal is provided by the text modality. By distilling the semantic knowledge contained in the text into the image analysis model through distillation loss, zero-label cross-modal knowledge injection can be achieved without additional manual text annotation, solely through learnable cue, thereby improving the accuracy of tail category recognition.

[0066] Accordingly, in order to further improve the accuracy of the target pathological image analysis model in identifying tail categories, step S50 includes steps S501 to S503: Step S501: Obtain the true label of the whole slide pathological image, and determine the standard cross-entropy loss of the first prediction result and the second prediction result with respect to the true label; Step S502: Determine the classification loss based on the standard cross-entropy loss, and then perform a weighted summation of the classification loss, the distillation loss, and the consistency constraint loss to obtain the total loss; Step S503: Using the backpropagation algorithm, optimize the parameters of the shared aggregator, the first expert network, and the second expert network in the initial pathological image analysis model based on the total loss to obtain the target pathological image analysis model.

[0067] It should be noted that the total loss The corresponding formula is as follows: ; In the formula, Represents classification loss. Indicates distillation loss, This represents the loss due to consistency constraints. and These represent the weighting coefficients corresponding to distillation loss and consistency constraint loss, respectively, used to balance the impact of different losses. For example... , .

[0068] In the specific implementation, the backpropagation algorithm is used to optimize the parameters of the shared aggregator, first expert network, second expert network, adapter, and learnable prompting engine in the initial pathological image analysis model based on the total loss, thereby obtaining the target pathological image analysis model. Multi-objective optimization of the model is achieved by jointly optimizing the classification loss, distillation loss, and consistency constraint loss: ensuring the accuracy of the basic classification (…). ); medical prior knowledge fused with text modality ( ); Enhance the robustness of the model to changes in distribution ( ).

[0069] This embodiment discloses a method for spatial mapping by inputting the first image embedding representation and the second image embedding representation into an adapter to obtain corresponding first and second mapping embeddings; obtaining embedding vectors of preset category description text and initializing a trainable parameter matrix with the same dimension as the embedding vectors; concatenating the embedding vectors with the trainable parameter matrix to form dynamic cue vectors; inputting the dynamic cue vectors into a pre-trained text encoder to obtain text feature vectors corresponding to the pathological categories of the whole-slice pathological image; determining the similarity between the first and second mapping embeddings and the text feature vectors, and determining the distillation loss based on the similarity. Compared with the prior art, this embodiment maps two distributions of image embeddings to the text space through an adapter, eliminating modality and dimensionality differences, allowing consistency constraints and distillation loss to be jointly optimized in the same space, improving training stability, and transferring medical semantics from the pre-trained text encoder to the image feature space without manual writing or annotation of text, significantly reducing annotation costs.

[0070] refer to Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the pathological image analysis model training method of the present invention.

[0071] Based on the above embodiments, in this embodiment, after step S50, steps S60 to S90 are further included: Step S60: Segment the whole-section pathological image to be analyzed to obtain multiple image blocks to be analyzed.

[0072] Step S70: Use the feature extraction module in the target pathological image analysis model to extract features from each of the image blocks to be analyzed, and generate a set of feature vectors.

[0073] Step S80: Input the set of feature vectors into the shared aggregator in the target pathological image analysis model to obtain the inference image embedding representation.

[0074] Step S90: The inference image embedding representation is input into the first expert network in the target pathological image analysis model to obtain the pathological classification result.

[0075] Understandably, the whole-slice pathological image to be analyzed can refer to a newly input, unlabeled digital slice (WSI) with an unknown pathological category, which needs to be automatically determined by the model.

[0076] It should be understood that the feature vector set can be an instance-level embedding sequence obtained by the feature extraction module in the target pathological image analysis model after the whole slice pathological image to be analyzed is segmented into image blocks, and each vector corresponds to a local region.

[0077] Furthermore, the inference image embedding representation can be a single-size vector output by a shared aggregator after performing self-attention aggregation on the feature vector set, used to condense the discriminative information of the entire slice.

[0078] It should be noted that the pathological classification result can be the category logits vector or category probability distribution output by the first expert network after performing forward computation on the inference image embedding representation. The final diagnostic label (such as Gleason classification, micrometastasis, etc.) can be obtained by argmax (a mathematical operation that represents the value of the independent variable that makes the function reach its maximum value).

[0079] After obtaining the target pathological image analysis model, during the deployment phase, inputting a whole-slice pathological image to be analyzed only requires passing through the feature extraction module, the shared aggregator, and the first expert network to obtain the final pathological classification result. The balanced branching and multimodal distillation used during training do not participate in inference, thus improving the efficiency of pathological classification.

[0080] This embodiment discloses a method for segmenting a whole-slice pathological image to be analyzed into multiple image blocks; using a feature extraction module in the target pathological image analysis model to extract features from each image block, generating a set of feature vectors; inputting the set of feature vectors into a shared aggregator in the target pathological image analysis model to obtain an inferred image embedding representation; and inputting the inferred image embedding representation into a first expert network in the target pathological image analysis model to obtain a pathological classification result. Compared to existing technologies, this embodiment obtains the final pathological classification result through the feature extraction module, shared aggregator, and first expert network, thus improving the efficiency of pathological classification.

[0081] Furthermore, this embodiment of the invention also proposes a storage medium storing a pathological image analysis model training program, which, when executed by a processor, implements the steps of the pathological image analysis model training method described above.

[0082] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the pathological image analysis model training device of the present invention.

[0083] like Figure 4 As shown, the pathological image analysis model training device proposed in this embodiment of the invention includes: a model construction module 601, a data processing module 602, a result output module 603, a loss determination module 604, and a parameter optimization module 605.

[0084] The model building module 601 is used to build an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit.

[0085] The data processing module 602 is used to process the original distributed data and the rebalanced distributed data through the shared aggregator to generate corresponding first image embedding representations and second image embedding representations.

[0086] The result output module 603 is used to input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing to obtain a first prediction result and a second prediction result.

[0087] The loss determination module 604 is used to determine the consistency constraint loss based on the first prediction result and the second prediction result through the consistency constraint unit.

[0088] The parameter optimization module 605 is used to optimize the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model.

[0089] The model building module 601 is further configured to acquire whole-slice pathological images, segment the whole-slice pathological images to obtain multiple image blocks; extract features from each image block using the feature extraction module to generate an instance-level embedding set; and perform original distribution sampling and inverse frequency rebalancing sampling on the instance-level embedding set to obtain original distribution data and rebalanced distribution data.

[0090] The loss determination module 604 is further configured to input the first image embedding representation and the second image embedding representation into the second expert network and the first expert network respectively through the consistency constraint unit to obtain a third prediction result and a fourth prediction result; determine a first difference between the first prediction result and the fourth prediction result and a second difference between the second prediction result and the third prediction result; and determine the consistency constraint loss based on the first difference and the second difference.

[0091] This embodiment of the device discloses the construction of an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and rebalanced distributed data respectively, generating corresponding first and second image embedding representations. The first and second image embedding representations are then input into the first and second expert networks for processing, respectively, to obtain a first prediction result and a second prediction result. Based on the first and second prediction results, the consistency constraint unit determines a consistency constraint loss. Based on the consistency constraint loss, the parameters of the initial pathological image analysis model are optimized to obtain a target pathological image analysis model. Because this embodiment processes the original distributed data and rebalanced distributed data in parallel using the first and second expert networks, and determines the consistency constraint loss based on the first and second prediction results using the consistency constraint unit, compared to existing technologies, this embodiment effectively improves the accuracy of the model in identifying tail pathological categories.

[0092] Based on the first embodiment of the pathological image analysis model training device of the present invention, a second embodiment of the pathological image analysis model training device of the present invention is proposed.

[0093] In this embodiment, the loss determination module 604 is further configured to input the first image embedding representation and the second image embedding representation into the adapter for spatial mapping to obtain the corresponding first mapping embedding and second mapping embedding; generate a text feature vector corresponding to the pathological category of the whole-slice pathological image through a learnable prompting engine; determine the similarity between the first mapping embedding and the second mapping embedding and the text feature vector, and determine the distillation loss based on the similarity.

[0094] The loss determination module 604 is further configured to obtain the embedding vector of the preset category description text and initialize a trainable parameter matrix with the same dimension as the embedding vector; concatenate the embedding vector with the trainable parameter matrix to form a dynamic prompt vector; input the dynamic prompt vector into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole slice pathological image.

[0095] The parameter optimization module 605 is further configured to obtain the true label of the whole-slice pathological image, and determine the standard cross-entropy loss of the first prediction result and the second prediction result with respect to the true label; determine the classification loss based on the standard cross-entropy loss, and perform a weighted summation of the classification loss, the distillation loss and the consistency constraint loss to obtain the total loss; and optimize the parameters of the shared aggregator, the first expert network and the second expert network in the initial pathological image analysis model based on the total loss using the backpropagation algorithm to obtain the target pathological image analysis model.

[0096] Other embodiments or specific implementations of the pathological image analysis model training device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0097] This application provides a pathological image analysis model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pathological image analysis model training method in the above embodiment 1.

[0098] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a pathological image analysis model training device suitable for implementing embodiments of this application. The pathological image analysis model training device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The pathological image analysis model training device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0099] like Figure 5As shown, the pathological image analysis model training device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the pathological image analysis model training device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the pathological image analysis model training device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows pathological image analysis model training devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0100] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0101] The pathological image analysis model training device provided in this application, employing the pathological image analysis model training method described in the above embodiments, can solve the technical problem of low accuracy in tail pathological category recognition of existing pathological image analysis models. Compared with the prior art, the beneficial effects of the pathological image analysis model training device provided in this application are the same as those of the pathological image analysis model training method provided in the above embodiments, and other technical features in this pathological image analysis model training device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0103] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0105] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0107] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the protection scope of the present invention.

Claims

1. A method for training a pathological image analysis model, characterized in that, The method includes: An initial pathological image analysis model is constructed, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and the rebalanced distributed data respectively to generate corresponding first image embedding representations and second image embedding representations. The first image embedding representation and the second image embedding representation are respectively input into the first expert network and the second expert network for processing to obtain the first prediction result and the second prediction result; Based on the first prediction result and the second prediction result, the consistency constraint loss is determined by the consistency constraint unit. The parameters of the initial pathological image analysis model are optimized based on the consistency constraint loss to obtain the target pathological image analysis model.

2. The pathological image analysis model training method as described in claim 1, characterized in that, The initial pathological image analysis model further includes a feature extraction module. Before the step of processing the original distribution data and rebalanced distribution data through the shared aggregator to generate the corresponding first image embedding representation and second image embedding representation, the model further includes: Obtain whole-section pathological images and segment the whole-section pathological images to obtain multiple image blocks; The feature extraction module is used to extract features from each of the image blocks to generate an instance-level embedding set; The instance-level embedding set is subjected to original distribution sampling and inverse frequency rebalancing sampling respectively to obtain original distribution data and rebalanced distribution data.

3. The pathological image analysis model training method as described in claim 1, characterized in that, The step of determining the consistency constraint loss based on the first prediction result and the second prediction result through the consistency constraint unit includes: The consistency constraint unit inputs the first image embedding representation and the second image embedding representation into the second expert network and the first expert network, respectively, to obtain the third prediction result and the fourth prediction result. Determine a first difference between the first prediction result and the fourth prediction result, and a second difference between the second prediction result and the third prediction result; The consistency constraint loss is determined based on the first difference and the second difference.

4. The pathological image analysis model training method as described in claim 2, characterized in that, The method further includes: The first image embedding representation and the second image embedding representation are respectively input into the adapter for spatial mapping to obtain the corresponding first mapping embedding and second mapping embedding; A learnable prompting engine generates text feature vectors corresponding to the pathological categories of the whole-slice pathological images; The similarity between the first mapping embedding and the second mapping embedding and the text feature vector is determined, and the distillation loss is determined based on the similarity.

5. The pathological image analysis model training method as described in claim 4, characterized in that, The step of generating a text feature vector corresponding to the pathological category of the whole-slice pathological image through a learnable prompting engine includes: Obtain the embedding vector of the preset category description text, and initialize a trainable parameter matrix with the same dimension as the embedding vector; The embedding vector is concatenated with the trainable parameter matrix to form a dynamic prompt vector; The dynamic cue vector is input into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole-slice pathological image.

6. The pathological image analysis model training method as described in claim 4, characterized in that, The step of optimizing the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model includes: Obtain the true labels of the whole-slice pathological images, and determine the standard cross-entropy loss of the first prediction result and the second prediction result with respect to the true labels; The classification loss is determined based on the standard cross-entropy loss, and the classification loss, distillation loss, and consistency constraint loss are weighted and summed to obtain the total loss. Using the backpropagation algorithm, the parameters of the shared aggregator, the first expert network, and the second expert network in the initial pathological image analysis model are optimized based on the total loss to obtain the target pathological image analysis model.

7. The pathological image analysis model training method as described in claim 2, characterized in that, After the step of optimizing the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model, the method further includes: The whole-section pathological image to be analyzed is segmented to obtain multiple image blocks to be analyzed; The feature extraction module in the target pathological image analysis model is used to extract features from each of the image blocks to be analyzed, generating a set of feature vectors. The set of feature vectors is input into the shared aggregator in the target pathological image analysis model to obtain the inference image embedding representation; The inference image embedding representation is input into the first expert network of the target pathological image analysis model to obtain the pathological classification result.

8. A training device for a pathological image analysis model, characterized in that, The device includes: The model building module is used to build an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The data processing module is used to process the original distributed data and the rebalanced distributed data through the shared aggregator to generate corresponding first image embedding representations and second image embedding representations. The result output module is used to input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing to obtain a first prediction result and a second prediction result. The loss determination module is used to determine the consistency constraint loss based on the first prediction result and the second prediction result through the consistency constraint unit. The parameter optimization module is used to optimize the parameters of the initial pathological image analysis model based on the consistency constraint loss to obtain the target pathological image analysis model.

9. A training device for a pathological image analysis model, characterized in that, The device includes: a memory, a processor, and a pathological image analysis model training program stored in the memory and executable on the processor, the pathological image analysis model training program being configured to implement the steps of the pathological image analysis model training method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a pathological image analysis model training program, which, when executed by a processor, implements the steps of the pathological image analysis model training method as described in any one of claims 1 to 7.

Images