Pathological image analysis model training method and device, equipment and storage medium

Abstract

The application discloses a pathological image analysis model training method and device, equipment and a storage medium. The method comprises the following steps: constructing an initial pathological image analysis model, wherein the initial pathological image analysis model comprises a shared aggregator, a first expert network, a second expert network and a consistency constraint unit; processing original distribution data and rebalanced distribution data through the shared aggregator respectively to generate corresponding first image embedding representations and second image embedding representations; inputting the first image embedding representations and the second image embedding representations into the first expert network and the second expert network respectively for processing to obtain first prediction results and second prediction results; determining a consistency constraint loss through the consistency constraint unit based on the first prediction results and the second prediction results; and performing parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model. Compared with the prior art, the accuracy of model recognition of tail pathological categories is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a pathological image analysis model training method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of digital pathology, computer-aided analysis methods based on whole slide images (WSI) have become an important means for cancer typing, grading and rare lesion detection. The existing technology usually adopts a multiple instance learning (MIL) framework, divides high-resolution WSI into a plurality of image blocks, aggregates the single slide-level embedding after feature encoding, and finally completes the diagnosis prediction through a classifier.

[0003] However, real clinical data generally has an extremely long-tailed distribution: common lesions (such as prostate adenocarcinoma Gleason 3, no lymph node metastasis) have a large sample size, while rare lesions (such as Gleason 5, isolated tumor cells ITC) account for less than 4%. Traditional MIL models perform well on head classes, but the recognition accuracy on tail classes is often less than 30%, making it difficult to meet the needs of precise diagnosis of rare subtypes in clinical practice.

[0004] Therefore, there is an urgent need for a pathological image analysis model training method that can effectively improve the accuracy of model recognition of tail pathological classes. SUMMARY

[0005] The main purpose of the present application is to provide a pathological image analysis model training method, device, equipment and storage medium, which aims to solve the technical problem of low accuracy of existing pathological image analysis model for tail pathological class recognition.

[0006] To achieve the above-mentioned purpose, the present application provides a pathological image analysis model training method, which comprises the following steps:

[0007] An initial pathological image analysis model is constructed, which includes a shared aggregator, a first expert network, a second expert network and a consistency constraint unit;

[0008] The shared aggregator processes the original distribution data and the rebalanced distribution data respectively to generate corresponding first image embedding representations and second image embedding representations;

[0009] The first image embedding representations and the second image embedding representations are input into the first expert network and the second expert network respectively for processing to obtain first prediction results and second prediction results;

[0010] determine, by the consistency constraint unit, a consistency constraint loss based on the first prediction result and the second prediction result;

[0011] perform parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model.

[0012] Optionally, the initial pathological image analysis model further comprises a feature extraction module, and before the step of processing the original distribution data and the re-balanced distribution data by the shared aggregator respectively to generate corresponding first image embedding representation and second image embedding representation, the method further comprises:

[0013] obtain a plurality of image blocks by segmenting the whole-slide pathological image.

[0014] perform feature extraction on each of the image blocks by the feature extraction module to generate a set of instance-level embeddings.

[0015] perform original distribution sampling and inverse frequency re-balancing sampling on the set of instance-level embeddings respectively to obtain original distribution data and re-balanced distribution data.

[0016] Optionally, the step of determining, by the consistency constraint unit, a consistency constraint loss based on the first prediction result and the second prediction result comprises:

[0017] input the first image embedding representation and the second image embedding representation into the second expert network and the first expert network respectively by the consistency constraint unit to obtain a third prediction result and a fourth prediction result.

[0018] determine a first difference between the first prediction result and the fourth prediction result and a second difference between the second prediction result and the third prediction result.

[0019] determine a consistency constraint loss according to the first difference and the second difference.

[0020] Optionally, the method further comprises:

[0021] input the first image embedding representation and the second image embedding representation into an adapter for spatial mapping to obtain corresponding first mapped embedding and second mapped embedding.

[0022] generate a text feature vector corresponding to the pathological category of the whole-slide pathological image by a learnable prompt engine.

[0023] determine a similarity between the first mapped embedding and the second mapped embedding and the text feature vector, and determine a distillation loss based on the similarity.

[0024] Optionally, the step of generating, by the learnable prompt engine, a text feature vector corresponding to the pathological category of the whole-slide pathological image comprises:

[0025] An embedding vector of a preset category description text is obtained, and a trainable parameter matrix consistent with the dimension of the embedding vector is initialized;

[0026] The embedding vector and the trainable parameter matrix are spliced to form a dynamic prompt vector;

[0027] The dynamic prompt vector is input into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole-slide pathological image.

[0028] Optionally, the step of optimizing the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model comprises:

[0029] A true label of the whole-slide pathological image is obtained, and a standard cross-entropy loss of the first prediction result and the second prediction result with the true label is determined;

[0030] A classification loss is determined according to the standard cross-entropy loss, and the classification loss, the distillation loss and the consistency constraint loss are weighted and summed to obtain a total loss;

[0031] The shared aggregator, the first expert network and the second expert network in the initial pathological image analysis model are optimized in parameters according to the total loss by a back propagation algorithm to obtain a target pathological image analysis model.

[0032] Optionally, after the step of optimizing the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model, the method further comprises:

[0033] The whole-slide pathological image to be analyzed is segmented to obtain a plurality of image blocks to be analyzed;

[0034] The feature extraction module in the target pathological image analysis model is used to extract features of each of the image blocks to be analyzed to generate a set of feature vectors;

[0035] The set of feature vectors is input into the shared aggregator in the target pathological image analysis model to obtain an inference image embedding representation;

[0036] The inference image embedding representation is input into the first expert network in the target pathological image analysis model to obtain a pathological classification result.

[0037] In addition, to achieve the above object, the application further provides a pathological image analysis model training device, which comprises:

[0038] a model construction module, configured to construct an initial pathological image analysis model, wherein the initial pathological image analysis model comprises a shared aggregator, a first expert network, a second expert network and a consistency constraint unit;

[0039] a data processing module, configured to process original distribution data and rebalanced distribution data respectively through the shared aggregator, and generate corresponding first image embedding representation and second image embedding representation;

[0040] a result output module, configured to input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing, and obtain a first prediction result and a second prediction result;

[0041] a loss determination module, configured to determine a consistency constraint loss through the consistency constraint unit based on the first prediction result and the second prediction result;

[0042] a parameter optimization module, configured to perform parameter optimization on the initial pathological image analysis model based on the consistency constraint loss, and obtain a target pathological image analysis model.

[0043] In addition, to achieve the above object, the application further provides a pathological image analysis model training device, which comprises a memory, a processor and a pathological image analysis model training program stored in the memory and executable on the processor, wherein the pathological image analysis model training program is configured to implement the steps of the pathological image analysis model training method as described above.

[0044] In addition, to achieve the above object, the application further provides a storage medium, wherein the storage medium stores a pathological image analysis model training program, and the pathological image analysis model training program is executable on a processor to implement the steps of the pathological image analysis model training method as described above.

[0045] This invention discloses the construction of an initial pathological image analysis model, which includes a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit. The shared aggregator processes the original distributed data and rebalanced distributed data respectively, generating corresponding first and second image embedding representations. The first and second image embedding representations are then input into the first and second expert networks for processing, respectively, to obtain a first prediction result and a second prediction result. Based on the first and second prediction results, the consistency constraint unit determines a consistency constraint loss. Based on the consistency constraint loss, the parameters of the initial pathological image analysis model are optimized to obtain a target pathological image analysis model. Because this invention processes the original distributed data and rebalanced distributed data in parallel using the first and second expert networks, and determines the consistency constraint loss based on the first and second prediction results using the consistency constraint unit, compared to existing technologies, this invention effectively improves the accuracy of the model in identifying tail pathological categories. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis model training method of the present invention;

[0047] Figure 2 This is a flowchart illustrating the second embodiment of the pathological image analysis model training method of the present invention;

[0048] Figure 3 This is a flowchart illustrating the third embodiment of the pathological image analysis model training method of the present invention;

[0049] Figure 4 This is a structural block diagram of the first embodiment of the pathological image analysis model training device of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of a pathological image analysis model training device in the hardware operating environment involved in the embodiments of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] This invention provides a method for training a pathological image analysis model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis model training method of the present invention.

[0054] In this embodiment, the pathological image analysis model training method comprises steps S10-S50.

[0055] Step S10: An initial pathological image analysis model is constructed, which comprises a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit.

[0056] It should be noted that the execution subject of the present embodiment can be a computer server device with data processing, network communication, and program running functions applied to pathological image analysis scenarios, such as servers, tablets, personal computers, etc., or an electronic device capable of realizing the above functions (such as a pathological image analysis model training device) and the like. The following takes a system containing a pathological image analysis model training device (hereinafter referred to as a system) as an example to illustrate the present embodiment and the following embodiments.

[0057] It should be explained that the shared aggregator can be a parameter-shared Transformer aggregation network, and the output end of the shared aggregator is connected to the input end of the first expert network and the second expert network respectively, forming a "double-flow-double-head" structure.

[0058] It should be noted that the initial pathological image analysis model further comprises a feature extraction module, and before step S10, steps S01-S03 are further included.

[0059] Step S01: Obtain a whole slide pathological image, and segment the whole slide pathological image to obtain a plurality of image blocks.

[0060] Step S02: Use the feature extraction module to perform feature extraction on each of the image blocks to generate an instance-level embedding set.

[0061] Step S03: Perform original distribution sampling and inverse frequency rebalancing sampling on the instance-level embedding set respectively to obtain original distribution data and rebalancing distribution data.

[0062] It can be understood that the whole slide pathological image can refer to a single high-resolution digital image obtained by scanning a complete pathological tissue section (including the range of the coverslip field) in one time by a digital slide scanner, which is usually stored in WSI (Whole Slide Image) format.

[0063] It should be explained that the instance-level embedding set can refer to a fixed-dimension vector set obtained by using the feature extraction module to perform forward inference on each image block after the whole slide pathological image is segmented into a plurality of image blocks, and each vector in the set is called an instance embedding, corresponding to a local region on the whole slide pathological image.

[0064] It can be understood that the feature extraction module described above can be a feature encoder pre-trained on a large number of pathological images (such as the CONCH model).

[0065] It should be noted that the original distribution sampling can refer to, in the training batch construction stage, random sampling according to the category frequency of the whole slice pathological image in the natural data set, so that the probability of sampling the rare category sample is low, and the probability of sampling the common category sample is high, thereby maintaining the "long tail" distribution characteristics.

[0066] Further, the inverse frequency rebalancing sampling can refer to repeatedly sampling the instance-level embedding set after classifying the inverse frequency weighting: the less the sample amount of a category, the higher the sampling weight, so that the number of occurrences of each category in a training cycle tends to be consistent, thereby alleviating the problem of insufficient training signal of the tail category. The sampling probability of the inverse frequency rebalancing sampling is , wherein, represents the number of samples of a certain category in the original data. For example, if a rare category has only 10 samples ( ), the sampling probability is ; and a common category has 1000 samples ( ), the sampling probability is . In this way, the number of samples of each type in the rebalanced distribution data tends to be balanced, solving the problem that rare categories are ignored in the original long-tail distribution.

[0067] It can be understood that the original distribution data can refer to the training batch data formed after the original distribution sampling, which has the same category ratio as the naturally collected data set, and is used to retain the true clinical distribution information. The rebalanced distribution data can refer to the training batch data formed after the inverse frequency rebalancing sampling, which has a category ratio close to uniform, and is used to enhance the discrimination ability of the model for the tail category.

[0068] Step S20: processing the original distribution data and the rebalanced distribution data respectively through the shared aggregator to generate corresponding first image embedding representation and second image embedding representation.

[0069] It should be noted that the core role of the shared aggregator is to overview all local features and intelligently allocate different attention weights, and finally fuse into a comprehensive feature vector that can represent the whole slice.

[0070] It should be explained that the first image embedding representation ( ) can refer to a single vector output by the shared aggregator after self-attention aggregation of the instance-level embedding sequence in the original distribution data, which is used to represent the global features of the whole slice in the original long-tail distribution perspective.

[0071] Correspondingly, the second image embedding representation ( ) can refer to another single vector output by the same shared aggregator after self-attention aggregation of the instance-level embedding sequence in the rebalanced distribution data, used to represent the global features of the entire slice in the class balance perspective.

[0072] Step S30: input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing, to obtain a first prediction result and a second prediction result.

[0002] It should be noted that the first expert network and the second expert network can be two independent fully connected classifiers.

[0074] It should be understood that the first expert network ( ) can be a learnable classifier dedicated to processing the first image embedding representation ( ), and the parameters thereof are only updated in the original distribution data path, and a class logits vector corresponding to the original distribution perspective (i.e., the first prediction result, ) is output. The second expert network ( ) can be a learnable classifier dedicated to processing the second image embedding representation ( ), which has the same structure as the first expert network but independent parameters, and is only updated in the rebalanced distribution data path, and outputs a class logits vector corresponding to the rebalanced distribution perspective (i.e., the second prediction result, ).

[0075] Step S40: based on the first prediction result and the second prediction result, determining a consistency constraint loss by the consistency constraint unit.

[0076] It should be noted that the consistency constraint unit can be realized by exchanging the expert network and the input distribution, and calculating the mean square error (MSE) of the output logic value.

[0077] In a specific implementation, step S40 includes steps S401-S403:

[0078] Step S401: input the first image embedding representation and the second image embedding representation into the second expert network and the first expert network respectively by the consistency constraint unit, to obtain a third prediction result and a fourth prediction result.

[0079] Step S402: determine a first difference between the first prediction result and the fourth prediction result, and a second difference between the second prediction result and the third prediction result.

[0080] Step S403: determine a consistency constraint loss according to the first difference and the second difference.

[0081] It should be noted that the third prediction result can be a category logits vector obtained after the first image embedding representation is input into the second expert network, which is used to measure the response of the original distribution feature on the re-balanced expert. The fourth prediction result can be a category logits vector obtained after the second image embedding representation is input into the first expert network, which is used to measure the response of the re-balanced distribution feature on the original expert.

[0082] It should be understood that the first difference can refer to the element-by-element difference between the fourth prediction result and the first prediction result (both of which are original distribution perspectives), usually taking L2 norm or MSE, to quantify the consistent deviation of the two experts in judging the re-balanced feature. The second difference can refer to the element-by-element difference between the third prediction result and the second prediction result (both of which are re-balanced distribution perspectives), to quantify the consistent deviation of the two experts in judging the original feature.

[0083] It should be explained that the consistency constraint loss The corresponding formula is as follows:

[0084] ;

[0085] In the formula, are the first image embedding representation and the second image embedding representation generated by the shared aggregator respectively, are the first expert network and the second expert network respectively; is the square of the L2 norm (i.e. mean square error MSE); is used to constrain When processing balanced data (i.e. the second image embedding representation, ), its output should be consistent with the benchmark output of ; is used to constrain When processing original data (i.e. the first image embedding representation, ), its output should be consistent with the benchmark output of .

[0086] It should be noted that the consistency constraint loss described above forces the judgments of the two expert networks on the same content to be consistent, so that the shared aggregator learns a distribution-independent, more generalized feature representation.

[0087] Step S50: parameter optimization is performed on the initial pathological image analysis model based on the consistency constraint loss, to obtain a target pathological image analysis model.

[0088] It should be noted that the above parameter optimization can refer to an iterative process of calculating gradients and updating all learnable parameters (e.g., shared aggregator, first expert network, and second expert network) in the initial pathological image analysis model by using a backpropagation algorithm to minimize the consistency constraint loss (joint classification loss, distillation loss) as the goal.

[0089] The embodiment discloses constructing an initial pathological image analysis model including a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit; processing original distribution data and rebalanced distribution data through the shared aggregator to generate corresponding first image embedding representations and second image embedding representations; inputting the first image embedding representations and the second image embedding representations into the first expert network and the second expert network for processing to obtain first prediction results and second prediction results; determining a consistency constraint loss based on the first prediction results and the second prediction results through the consistency constraint unit; and performing parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model. Since the embodiment processes original distribution data and rebalanced distribution data in parallel through the first expert network and the second expert network, and determines a consistency constraint loss based on first prediction results and second prediction results through a consistency constraint unit, compared with the prior art, the embodiment effectively improves the accuracy of model recognition of tail pathological categories.

[0090] Reference Figure 2 , Figure 2 The flowchart of the second embodiment of the pathological image analysis model training method of the present application is shown.

[0091] Based on the above first embodiment, in the present embodiment, the method further comprises steps S404-S406:

[0092] Step S404: input the first image embedding representation and the second image embedding representation into an adapter for spatial mapping to obtain corresponding first mapping embedding and second mapping embedding.

[0093] Step S405: generate a text feature vector corresponding to the pathological category of the whole slice pathological image through a learnable prompt engine.

[0094] Step S406: determine the similarity between the first mapping embedding and the second mapping embedding and the text feature vector, and determine a distillation loss based on the similarity.

[0095] It should be noted that the adapter can be a trainable subnetwork composed of a "multi-layer perceptron (MLP) + linear projection layer" for mapping the image embedding space (e.g., 1024 dimensions) to the text embedding space (e.g., 768 dimensions), eliminating the cross-modal dimension and distribution differences, and enabling subsequent similarity calculations in the same space.

[0096] It can be understood that the first mapping embedding represents a vector transformed by the adapter, with the same dimension as the text feature vector, for comparison with the text side features. Accordingly, the second image embedding represents a vector transformed by the same adapter.

[0097] In a specific implementation, step S405 includes steps S4051-S4053:

[0098] Step S4051: Obtain an embedding vector of a preset category description text and initialize a trainable parameter matrix with the same dimension as the embedding vector.

[0099] Step S4052: Concatenate the embedding vector and the trainable parameter matrix to form a dynamic prompt vector.

[0100] Step S4053: Input the dynamic prompt vector into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole-slice pathological image.

[0101] It should be noted that the learnable prompt engine can be a dynamic prompt generation module formed by concatenating the embedding vector of the preset category description text and the trainable parameter matrix, without the need for manual annotation. The parameters are updated simultaneously with the network during training. The embedding vector of the preset category description text can be a static word embedding vector obtained by one-time conversion of a pre-set, non-training category description string through a pre-trained text encoder (CONCH, etc.). This vector is only used as a template to concatenate with the trainable parameter matrix in the subsequent process, and the weight is no longer updated.

[0102] It should be understood that the pathological category of the whole-slice pathological image can be a diagnostic label to which the slice belongs, such as "prostate adenocarcinoma Gleason 5", "breast micrometastasis", etc., for indexing the corresponding preset category description text.

[0103] It should be noted that the dynamic prompt vector can be a prompt vector formed by concatenating the embedding vector and the trainable parameter matrix bit by bit and updated in real time with the gradient during training. The parameters in the backpropagation process are continuously changing, and different semantics of the text features can be generated for the same category.

[0104] It should be understood that the pre-trained text encoder can refer to a text conversion network that is pre-trained on a large-scale general corpus or medical text, the weights of which are fixed and do not participate in the update of the downstream task, which is used to map the dynamic prompt vector into a high-dimensional semantic vector as a teacher signal in the distillation loss, so that the image side features can absorb the medical prior knowledge contained in the text modality through similarity alignment.

[0105] It should be explained that the above distillation loss can be a cross-entropy loss that measures the cosine similarity deviation between the image global features mapped by the adapter and the text features generated by the learnable prompt engine, and the calculation process completely simulates the "teacher-student" alignment method in knowledge distillation, except that the teacher signal is provided by the text modality. Through the distillation loss, the semantic knowledge contained in the text is distilled into the image analysis model, and no additional manual annotation of the text is required. Zero-labeled cross-modal knowledge injection can be achieved only through a learnable prompt, thereby improving the tail class recognition accuracy.

[0106] Correspondingly, in order to further improve the tail class recognition accuracy of the target pathological image analysis model, step S50 includes steps S501-S503:

[0107] Step S501: obtaining the true label of the whole slide pathological image, and determining the standard cross-entropy loss of the first prediction result and the second prediction result with the true label respectively;

[0108] Step S502: determining a classification loss according to the standard cross-entropy loss, and performing weighted summation on the classification loss, the distillation loss and the consistency constraint loss to obtain a total loss;

[0109] Step S503: performing parameter optimization on the shared aggregator, the first expert network and the second expert network in the initial pathological image analysis model according to the total loss through a back propagation algorithm, to obtain a target pathological image analysis model.

[0110] It should be noted that the total loss The corresponding formula is as follows:

[0111] ;

[0112] In the formula, represents the classification loss, represents the distillation loss, represents the consistency constraint loss, and respectively represent the weight coefficients corresponding to the distillation loss and the consistency constraint loss, which are used to balance the influence of different losses, for example, , .

[0113] In a specific implementation, the shared aggregator, the first expert network, the second expert network, the adapter, and the learnable prompt engine in the initial pathological image analysis model are optimized in parameters according to the total loss by means of a back propagation algorithm, and a target pathological image analysis model is obtained. By jointly optimizing the classification loss, the distillation loss, and the consistency constraint loss, multi-objective optimization of the model is realized: ensuring the basic classification accuracy (1 ); fusing the medical prior knowledge in the text mode (2 ); and enhancing the robustness of the model to distribution changes (3 ).

[0114] The first image embedding representation and the second image embedding representation are respectively input into an adapter for spatial mapping to obtain corresponding first mapping embedding and second mapping embedding; an embedding vector of a preset category description text is obtained, and a trainable parameter matrix consistent in dimension with the embedding vector is initialized; the embedding vector and the trainable parameter matrix are spliced to form a dynamic prompt vector; the dynamic prompt vector is input into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole slice pathological image; similarity between the first mapping embedding and the second mapping embedding and the text feature vector is determined, and a distillation loss is determined based on the similarity. Compared with the prior art, the adapter is used to map the image embeddings of the two distributions to the text space in the embodiment, the modal and dimensional differences are eliminated, the consistency constraint and the distillation loss can be jointly optimized in the same space, the training stability is improved, and the medical semantics in the pre-trained text encoder can be transferred to the image feature space without manual writing or labeling of the text, thereby significantly reducing the labeling cost.

[0115] Reference Figure 3 , Figure 3 is a flowchart of a third embodiment of the pathological image analysis model training method.

[0116] Based on the above embodiments, in the present embodiment, after the step S50, the steps S60-S90 are further included:

[0117] Step S60: The whole slice pathological image to be analyzed is segmented to obtain a plurality of image blocks to be analyzed.

[0118] Step S70: The feature extraction module in the target pathological image analysis model is used to extract features of each of the image blocks to be analyzed to generate a set of feature vectors.

[0119] Step S80: The set of feature vectors is input into the shared aggregator in the target pathological image analysis model to obtain an inference image embedding representation.

[0120] Step S90: input the inference image embedding representation into a first expert network in the target pathological image analysis model to obtain a pathological classification result.

[0121] It can be understood that the whole slice pathological image to be analyzed can refer to a newly input digital slice (WSI) without any annotation, and the pathological category thereof is unknown and needs to be automatically determined by the model.

[0122] It should be understood that the set of feature vectors can be an instance-level embedding sequence obtained by a feature extraction module in the target pathological image analysis model after the whole slice pathological image to be analyzed is segmented into image blocks, and each vector corresponds to a local region.

[0123] Further, the inference image embedding representation can be a single tensor-level vector output by the shared aggregator after performing self-attention aggregation on the set of feature vectors, which is used to condense the discrimination information of the whole slice.

[0124] It should be noted that the pathological classification result can be a class logits vector or a class probability distribution output by the first expert network after forward calculation on the inference image embedding representation, and the final diagnosis label (such as Gleason grading, micro-metastasis, etc.) can be obtained by argmax (a mathematical operation representing the value of the independent variable that makes the function maximum).

[0125] After obtaining the target pathological image analysis model, in the deployment stage, inputting a whole slice pathological image to be analyzed, only passing through the path of the feature extraction module, the shared aggregator and the first expert network, the final pathological classification result can be obtained. The balanced branch and the multi-modal distillation used in training do not participate in inference, which improves the efficiency of pathological classification.

[0126] The embodiment discloses segmenting a whole slice pathological image to be analyzed to obtain a plurality of image blocks to be analyzed; using a feature extraction module in the target pathological image analysis model to extract features of each image block to be analyzed to generate a set of feature vectors; inputting the set of feature vectors into a shared aggregator in the target pathological image analysis model to obtain an inference image embedding representation; and inputting the inference image embedding representation into a first expert network in the target pathological image analysis model to obtain a pathological classification result. Compared with the prior art, the embodiment can obtain the final pathological classification result by passing through the path of the feature extraction module, the shared aggregator and the first expert network, thereby improving the efficiency of pathological classification.

[0127] In addition, the embodiment of the present application also proposes a storage medium, wherein the storage medium stores a pathological image analysis model training program, and the pathological image analysis model training program is executed by a processor to realize the steps of the pathological image analysis model training method as described above.

[0128] Reference Figure 4 , Figure 4 is a structural block diagram of a pathological image analysis model training device according to an embodiment of the present application.

[0129] As shown in Figure 4 , the pathological image analysis model training device according to the embodiment of the present application comprises a model construction module 601, a data processing module 602, a result output module 603, a loss determination module 604, and a parameter optimization module 605.

[0130] The model construction module 601 is configured to construct an initial pathological image analysis model, wherein the initial pathological image analysis model comprises a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit.

[0131] The data processing module 602 is configured to process original distribution data and re-balanced distribution data respectively through the shared aggregator to generate corresponding first image embedding representation and second image embedding representation.

[0132] The result output module 603 is configured to input the first image embedding representation and the second image embedding representation into the first expert network and the second expert network respectively for processing to obtain a first prediction result and a second prediction result.

[0133] The loss determination module 604 is configured to determine a consistency constraint loss through the consistency constraint unit based on the first prediction result and the second prediction result.

[0134] The parameter optimization module 605 is configured to perform parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model.

[0135] The model construction module 601 is further configured to acquire a whole slice pathological image, segment the whole slice pathological image to obtain a plurality of image blocks, perform feature extraction on each of the image blocks by using the feature extraction module to generate an instance-level embedding set, and perform original distribution sampling and inverse frequency re-balanced sampling on the instance-level embedding set respectively to obtain original distribution data and re-balanced distribution data.

[0136] The loss determination module 604 is further configured to input the first image embedding representation and the second image embedding representation into the second expert network and the first expert network respectively through the consistency constraint unit to obtain a third prediction result and a fourth prediction result, determine a first difference between the first prediction result and the fourth prediction result and a second difference between the second prediction result and the third prediction result, and determine a consistency constraint loss according to the first difference and the second difference.

[0137] The device embodiment discloses constructing an initial pathological image analysis model, which comprises a shared aggregator, a first expert network, a second expert network, and a consistency constraint unit; the shared aggregator is used to process original distribution data and rebalanced distribution data respectively to generate corresponding first image embedding representation and second image embedding representation; the first image embedding representation and the second image embedding representation are input into the first expert network and the second expert network respectively for processing to obtain first prediction result and second prediction result; based on the first prediction result and the second prediction result, the consistency constraint unit is used to determine a consistency constraint loss; and based on the consistency constraint loss, the initial pathological image analysis model is optimized in parameters to obtain a target pathological image analysis model. Since the device embodiment processes original distribution data and rebalanced distribution data in parallel through the first expert network and the second expert network, and determines a consistency constraint loss through the consistency constraint unit based on the first prediction result and the second prediction result, compared with the prior art, the device embodiment effectively improves the accuracy of model recognition of tail pathological categories.

[0138] Based on the first embodiment of the pathological image analysis model training device of the present application, the second embodiment of the pathological image analysis model training device of the present application is proposed.

[0139] In the present embodiment, the loss determination module 604 is further configured to input the first image embedding representation and the second image embedding representation into an adapter for spatial mapping to obtain corresponding first mapping embedding and second mapping embedding; generate a text feature vector corresponding to the pathological category of the whole slice pathological image through a learnable prompt engine; determine the similarity between the first mapping embedding and the second mapping embedding and the text feature vector, and determine a distillation loss based on the similarity.

[0140] The loss determination module 604 is further configured to obtain an embedding vector of a preset category description text, and initialize a trainable parameter matrix consistent with the dimension of the embedding vector; splice the embedding vector and the trainable parameter matrix to form a dynamic prompt vector; input the dynamic prompt vector into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole slice pathological image.

[0141] The parameter optimization module 605 is further configured to obtain a true label of the whole slide pathological image, and determine a standard cross-entropy loss of the first prediction result and the second prediction result with the true label respectively; determine a classification loss according to the standard cross-entropy loss, and obtain a total loss by weighted summation of the classification loss, the distillation loss and the consistency constraint loss; and perform parameter optimization on the shared aggregator, the first expert network and the second expert network in the initial pathological image analysis model according to the total loss by a back propagation algorithm, to obtain a target pathological image analysis model.

[0142] Other embodiments or specific implementations of the pathological image analysis model training apparatus can refer to the above-mentioned method embodiments, and will not be described here.

[0143] The present application provides a pathological image analysis model training device, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pathological image analysis model training method in Embodiment I.

[0144] Reference will now be made to the drawings, and specific examples relating to pathological image analysis model training devices suitable for use in implementing embodiments of the present application will be described herein. Figure 5 The pathological image analysis model training device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The pathological image analysis model training device shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0145] As Figure 5As shown, the pathological image analysis model training device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. Various programs and data required for the operation of the pathological image analysis model training device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the pathological image analysis model training device to communicate with other devices wirelessly or by wire to exchange data. Although the pathological image analysis model training device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0146] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication apparatus, or installed from the storage apparatus 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0147] The pathological image analysis model training device provided by the present disclosure adopts the pathological image analysis model training method in the above-mentioned embodiments, and can solve the technical problem of low accuracy of existing pathological image analysis models in recognizing tail pathological categories. Compared with the prior art, the pathological image analysis model training device provided by the present disclosure has the same beneficial effects as the pathological image analysis model training method provided by the above-mentioned embodiments, and other technical features in the pathological image analysis model training device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0148] It should be understood that various parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above description of embodiments, specific functional, structural, material or characteristic features are combined in a manner that is appropriate for the particular embodiment.

[0149] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that are obvious to those skilled in the art are intended to be within the scope of the application. The scope of the application is defined by the claims.

[0150] It should be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or systems that comprise a list of elements do not include only those elements, but can also include other elements not expressly listed or inherent to such processes, methods, articles, or systems. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system that includes the element.

[0151] The above-mentioned embodiment numbers of the application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0152] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk) and includes a number of instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0153] The above is only the preferred embodiment of the present application, and does not limit the scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the protection scope of the present application.

Claims

1. A pathological image analysis model training method, characterized by, The method comprises: constructing an initial pathological image analysis model, the initial pathological image analysis model comprising a shared aggregator, a first expert network, a second expert network and a consistency constraint unit; processing original distribution data and rebalanced distribution data through the shared aggregator respectively to generate corresponding first image embedding representations and second image embedding representations; inputting the first image embedding representations and the second image embedding representations into the first expert network and the second expert network respectively for processing to obtain first prediction results and second prediction results; determining a consistency constraint loss through the consistency constraint unit based on the first prediction results and the second prediction results; performing parameter optimization on the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model; The initial pathological image analysis model further comprises a feature extraction module, and before the step of processing original distribution data and rebalanced distribution data through the shared aggregator respectively to generate corresponding first image embedding representations and second image embedding representations, the method further comprises: obtaining a full slice pathological image, and segmenting the full slice pathological image to obtain a plurality of image blocks; extracting features of each of the image blocks using the feature extraction module to generate a set of instance-level embeddings; performing original distribution sampling and inverse frequency rebalancing sampling on the set of instance-level embeddings respectively to obtain original distribution data and rebalanced distribution data; The method further comprises: inputting the first image embedding representations and the second image embedding representations into an adapter for spatial mapping to obtain corresponding first mapped embeddings and second mapped embeddings; generating a text feature vector corresponding to the pathological category of the full slice pathological image through a learnable prompt engine; determining the similarity between the first mapped embeddings and the second mapped embeddings and the text feature vector, and determining a distillation loss based on the similarity; The step of generating a text feature vector corresponding to the pathological category of the full slice pathological image through a learnable prompt engine comprises: obtaining an embedding vector of a preset category description text, and initializing a trainable parameter matrix consistent in dimension with the embedding vector; concatenating the embedding vector and the trainable parameter matrix to form a dynamic prompt vector; inputting the dynamic prompt vector into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the full slice pathological image.

2. The pathological image analysis model training method of claim 1, wherein, The step of determining a consistency constraint loss through the consistency constraint unit based on the first prediction results and the second prediction results comprises: inputting the first image embedding representations and the second image embedding representations into the second expert network and the first expert network respectively through the consistency constraint unit to obtain third prediction results and fourth prediction results; determining a first difference between the first prediction results and the fourth prediction results and a second difference between the second prediction results and the third prediction results; determining a consistency constraint loss according to the first difference and the second difference.

3. The pathological image analysis model training method of claim 1, wherein, The step of parameter optimization of the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model comprises: obtaining a true label of the whole-slide pathological image, and determining a standard cross-entropy loss of the first prediction result and the second prediction result with the true label respectively; determining a classification loss according to the standard cross-entropy loss, and performing weighted summation on the classification loss, the distillation loss and the consistency constraint loss to obtain a total loss; parameter optimization of the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model.

4. The pathological image analysis model training method of claim 1, wherein, After the step of parameter optimization of the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model, the method further comprises: segmenting a whole-slide pathological image to be analyzed to obtain a plurality of image blocks to be analyzed; performing feature extraction on each of the image blocks to be analyzed by using a feature extraction module in the target pathological image analysis model to generate a feature vector set; inputting the feature vector set into a shared aggregator in the target pathological image analysis model to obtain an inference image embedding representation; inputting the inference image embedding representation into a first expert network in the target pathological image analysis model to obtain a pathological classification result. 5.A pathological image analysis model training apparatus, characterized by comprising: The device comprises: a model construction module configured to construct an initial pathological image analysis model, wherein the initial pathological image analysis model comprises a shared aggregator, a first expert network, a second expert network and a consistency constraint unit, and the initial pathological image analysis model further comprises a feature extraction module; a data processing module configured to process original distribution data and rebalanced distribution data by the shared aggregator respectively to generate corresponding first image embedding representations and second image embedding representations; a result output module configured to input the first image embedding representations and the second image embedding representations into the first expert network and the second expert network respectively for processing to obtain first prediction results and second prediction results; a loss determination module configured to determine a consistency constraint loss by the consistency constraint unit based on the first prediction results and the second prediction results; a parameter optimization module configured to perform parameter optimization of the initial pathological image analysis model based on the consistency constraint loss to obtain a target pathological image analysis model; the model construction module is further configured to obtain a whole-slide pathological image, segment the whole-slide pathological image to obtain a plurality of image blocks, perform feature extraction on each of the image blocks by using the feature extraction module to generate an instance-level embedding set, and perform original distribution sampling and inverse frequency rebalancing sampling on the instance-level embedding set respectively to obtain original distribution data and rebalanced distribution data. The loss determination module is further configured to input the first image embedding representation and the second image embedding representation into an adapter for spatial mapping respectively to obtain corresponding first mapping embedding and second mapping embedding; generate a text feature vector corresponding to the pathological category of the whole slide pathological image through a learnable prompt engine; determine a similarity between the first mapping embedding and the second mapping embedding and the text feature vector, and determine a distillation loss based on the similarity; The loss determination module is further configured to obtain an embedding vector of a preset category description text, and initialize a trainable parameter matrix consistent with a dimension of the embedding vector; concatenate the embedding vector and the trainable parameter matrix to form a dynamic prompt vector; input the dynamic prompt vector into a pre-trained text encoder to obtain a text feature vector corresponding to the pathological category of the whole slide pathological image. 6.A pathological image analysis model training apparatus, comprising: The device comprises a memory, a processor, and a pathological image analysis model training program stored on the memory and executable on the processor, and the pathological image analysis model training program is configured to implement the steps of the pathological image analysis model training method according to any one of claims 1 to 4.

7. A storage medium, characterized by The storage medium stores a pathological image analysis model training program, and the pathological image analysis model training program implements the steps of the pathological image analysis model training method according to any one of claims 1 to 4 when executed by the processor.

Images