Breast pathology visual model establishing method based on multi-model fusion and combined distillation

Abstract

The invention relates to the field of artificial intelligence and medical image processing, in particular to a mammary gland pathology visual model establishing method based on multi-model fusion and combined distillation, which comprises the following steps: performing tissue segmentation and dyeing normalization on a full-slice image; inputting the image blocks into a pre-training teacher model of a plurality of freezing parameters in parallel to extract high-dimensional features, and generating unified enhanced features through a learnable feature fusion network; constructing a student model, and performing end-to-end training by using a joint loss function including feature simulation, logic output distillation and multi-task supervision; and connecting a plurality of task specific prediction heads to the student model, and realizing full-slice-level multi-task diagnosis and treatment prediction through an aggregation strategy. According to the technical scheme, efficient knowledge migration and multi-task cooperation can be achieved, and the accuracy, generalization ability and reasoning efficiency of mammary gland pathology analysis are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and medical image processing, in particular to a breast pathological visual model establishment method based on multi-model fusion and joint distillation. BACKGROUND

[0002] With the deep integration of digital pathology and artificial intelligence technology, breast pathological intelligent analysis based on whole slide image (WSI) has become a key direction to improve the efficiency and accuracy of diagnosis. Current research generally relies on deep learning models to extract features and predict tasks from high-resolution pathological images, aiming to realize the automatic mapping from histomorphology to clinical diagnosis and treatment indicators. However, breast pathological images have high heterogeneity, significant staining differences, and multiple clinical tasks (such as grading, typing, and staging), which pose severe challenges to the generalization ability, representation richness, and multi-task collaboration of the model.

[0003] Among them, the existing technology mainly develops along three paths: one is to fine-tune a single pre-trained model (such as ResNet or Vision Transformer) on a specific breast pathological dataset, which achieves task-specific optimization, but is limited by the limited data size and difficult to migrate to other related tasks; two is to use a multi-instance learning framework (such as CLAM) to aggregate block-level features of WSI, which reduces the dependence on labeling, but the feature extraction backbone is still a single general model, which fails to integrate the complementary prior knowledge contained in multiple advanced pathological base models; three is to directly call emerging general pathological base models (such as UNI v2 or Virchow 2) as feature extractors, which shows improvement in some tasks, but falls into the dilemma of "choosing one and abandoning the rest", unable to integrate the differentiated advantages of different models in macroscopic tissue structure and microscopic cell morphology.

[0004] The existing technology generally has three major defects: first, the model architecture is isolated, resulting in the need to maintain multiple specialized models in clinical deployment, system redundancy, and low inference efficiency; second, there is a lack of unified fusion mechanism for multi-source heterogeneous pathological knowledge, making it difficult to perform consistency checking and joint reasoning on different model outputs; third, an end-to-end multi-task learning paradigm has not been established, which cannot generate multi-dimensional clinical indicators covering diagnosis, prognosis, and treatment response from a single WSI input, severely restricting the actual landing of "diagnosis and treatment integrated" intelligent systems. Therefore, there is an urgent need for a breast pathological visual model establishment method that can deeply integrate multi-model prior knowledge, achieve knowledge transfer through joint distillation, and support multi-task collaborative prediction. SUMMARY

[0005] To address the shortcomings of existing technologies, this invention provides a method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation. By integrating the complementary prior knowledge of multiple pre-trained teacher models through a multi-source feature extraction and fusion module, and utilizing a joint distillation training mechanism to achieve efficient knowledge transfer, a single student model with rich representation capabilities is constructed.

[0006] To achieve the above objectives, this invention provides a method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation, comprising: Step 1: Input and preprocessing. Receive the original whole-slice image, automatically detect and segment the tissue region, cut the tissue region into image blocks of fixed size, and manually select a whole-slice image with good staining and crop a region from it as a staining normalization template. Perform color normalization using existing technology to eliminate staining differences. Step 2: Multi-source feature extraction and fusion. Image patches are input in parallel into multiple pre-trained teacher models to extract high-dimensional feature vectors. Then, these features are integrated into a unified enhanced feature representation through a learnable feature fusion network. Step 3: Joint distillation training. Construct a student model by using the joint distillation loss function to force the student model to simultaneously mimic the output behavior of all teacher models and learn their fused feature representations. Step 4: Multi-task prediction. Multiple parallel task-specific prediction heads are connected on top of the fused features output by the student model to perform various diagnostic tasks. During inference, the prediction results of all image patches are processed by the system and then aggregated to form the final full-slice image-level prediction.

[0007] Preferably, in step 1, the image patch is cropped to a fixed size of 256 pixels by 256 pixels at a magnification of 20x, the tissue region detection adopts a semantic segmentation algorithm based on deep learning with a segmentation accuracy of greater than or equal to 95%, and the color normalization processing adopts a color transfer technique based on generative adversarial networks with a color consistency error of less than 5%.

[0008] Preferably, in step 2, the teacher model includes at least two pre-trained pathological basic models. The model parameters are frozen and do not participate in training. Each teacher model outputs a high-dimensional feature vector with feature dimensions of 1024 and 2048, respectively. These feature vectors come from the model's global average pooling layer or the layer before the classification head.

[0009] Preferably, the feature fusion network includes a feature alignment module, which projects each teacher feature to the same dimension through a fully connected layer. The projected feature dimension is uniformly 512-dimensional. The projection layer uses a linear activation function, and the weights are initialized using a Kamming uniform distribution.

[0010] Preferably, the feature fusion network further includes a feature splicing module, which splices the projected feature vectors along the channel dimension, resulting in a spliced ​​feature dimension of 1024. Before the splicing operation, each feature vector is subjected to L2 normalization, with a normalization coefficient of 0.0001.

[0011] Preferably, the feature fusion network further includes a nonlinear fusion module, which inputs the spliced ​​features into a learnable neural network consisting of two fully connected layers. The first fully connected layer has an output dimension of 512 and uses a Gaussian error linear unit activation function, while the second fully connected layer has an output dimension of 256 and uses a hyperbolic tangent activation function. A random deactivation layer with a dropout rate of 0.2 is set between the two layers.

[0012] Preferably, in step 3, the student model adopts a visual transformer architecture, which includes 12 encoder layers, each containing 8 attention heads, a hidden layer dimension of 768, an internal dimension of 3072 for the feedforward network, and uses learnable one-dimensional position embedding for position encoding.

[0013] Preferably, the joint distillation loss function consists of three parts: feature imitation loss, logic output distillation loss, and multi-task supervision loss. The feature imitation loss uses mean squared error to calculate the distance between the student model's output features and the fused teacher features. The logic output distillation loss uses KL divergence to calculate the difference between the teacher model and the student model's task output logic values. The multi-task supervision loss uses cross-entropy loss and mean squared error loss respectively, depending on the task type.

[0014] Preferably, the feature imitation loss is calculated as the sum of squares of the feature vector differences, the distillation temperature in the logical output distillation loss is set to 2.5, and the classification task in the multi-task supervision loss uses label smoothing technology with a smoothing parameter set to 0.1.

[0015] Preferably, the weight hyperparameters in the joint distillation loss function are determined by grid search, the feature imitation loss weight is set to 0.4, the logical output distillation loss weight is set to 0.3, the multi-task supervision loss weight is set to 0.3, the total loss function is optimized using the adaptive moment estimation algorithm, the initial learning rate is set to 0.0001, and the batch size is 32.

[0016] Preferably, the task-specific prediction head in step 4 includes a breast lesion grading prediction head, a molecular subtyping prediction head, a tumor staging prediction head, and a treatment response prediction head. Each prediction head consists of two fully connected layers. The first layer has an output dimension of 128 and uses a modified linear unit activation function. The second layer has an output dimension set according to task requirements and uses a linear activation function.

[0017] Preferably, the aggregation strategy includes average pooling aggregation and attention-weighted aggregation. Average pooling aggregation takes the arithmetic mean of the prediction results for all image patches. Attention-weighted aggregation learns the importance weights of each image patch and performs a weighted summation. The attention weights are calculated by a single-layer perceptron, with an output dimension of 1, and are normalized using the softmax function.

[0018] Preferably, the method further includes a model validation module, which uses five-fold cross-validation to evaluate model performance. The training set and validation set are split in each fold to maintain a consistent class distribution. The model performance metrics include accuracy, precision, recall, and F1 score, and all metrics are required to reach 90% or higher.

[0019] Preferably, the method further includes an online update mechanism, which triggers incremental learning of the model when new labeled data reaches a certain scale. The incremental learning adopts an elastic weight consolidation algorithm, the importance weight is calculated through the Fisher information matrix, the regularization coefficient is set to 0.8, and the update cycle is once every 3 months.

[0020] Preferably, the method is deployed on a distributed computing platform, using a graphics processing unit cluster for model training. The training time for a single session does not exceed 24 hours. During the inference phase, it supports parallel processing of multiple full-slice images, with a single image processing time of less than 3 minutes. The system throughput reaches 20 full-slice images per hour.

[0021] Compared with the closest existing technology, the present invention has the following advantages: By integrating complementary prior knowledge from multiple pre-trained teacher models through a multi-source feature extraction and fusion module, and utilizing a joint distillation training mechanism to achieve efficient knowledge transfer, a single student model with rich representational capabilities is constructed. This model supports multi-task collaborative prediction, significantly improving the accuracy and generalization ability of breast pathology analysis, while reducing system complexity and inference resource consumption, thus providing a reliable technical foundation for the practical implementation of integrated diagnosis and treatment intelligent systems. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation provided by the present invention; Figure 2 This is a flowchart illustrating the practical application of the method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation provided by this invention. Detailed Implementation

[0023] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1 This invention provides a method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation, such as... Figure 1 As shown, it includes: By utilizing a joint distillation training mechanism to achieve efficient knowledge transfer, a single student model with rich representation capabilities is constructed. This model supports multi-task collaborative prediction, significantly improving the accuracy and generalization ability of breast pathology analysis, while reducing system complexity and inference resource consumption. It is applied to a method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation.

[0026] In the above-mentioned method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation, step (1) involves receiving the original whole-slice image, automatically performing tissue region detection and segmentation, cutting the tissue region into image blocks of fixed size, and simultaneously manually selecting a well-stained whole-slice image and cropping a region from it as a staining normalization template. Color normalization is then performed using existing techniques to eliminate staining differences. Specifically, in step (1), the original whole-slice image is generated by a digital pathology scanner, typically acquired at 40x or 20x optical magnification, with a pixel size of approximately 0.25 micrometers or 0.5 micrometers per pixel. The system first loads the whole-slice image and performs semantic segmentation through a built-in tissue region detection module. This module adopts a deep learning-based U-Net architecture, with its encoder consisting of a ResNet-34 backbone network and its decoder containing skip connections and upsampling operations. The final output is a binary mask of the same size as the input image, used to identify the effective tissue region and the background region. The segmentation algorithm achieves a segmentation accuracy of over 95% on the internal validation set, ensuring that subsequent processing focuses only on regions containing pathological information and avoiding unnecessary computation. Subsequently, the system uses this mask to segment the tissue region into a series of fixed-size image blocks. Each image block is cropped into a 256-pixel by 256-pixel square region at 20x magnification, a size that balances computational efficiency with preservation of pathological details. All image blocks are uniformly stored in lossless compressed TIFF format, along with their coordinate metadata from the original whole-slice image, for spatial alignment in the subsequent aggregation stage. Simultaneously, to eliminate color shifts caused by differences in staining protocols between different batches and laboratories, the system introduces a staining normalization process. This process begins with a pathologist manually selecting a high-quality, clearly structured whole-slice image as a reference, and then manually cropping a representative region containing typical breast tissue (such as ducts, lobules, and stroma) from it. This region serves as the staining normalization template. All image blocks to be processed are mapped to the color space of this template using a generative adversarial network-based color transfer technique. This generative adversarial network consists of a generator and a discriminator. The generator uses a CycleGAN architecture, constraining the reversibility of color transitions through cycle consistency loss, while the discriminator uses a PatchGAN structure to ensure the realism of local textures. After this processing, the color consistency error of all image patches in the system is controlled within 5%, significantly improving the model's robustness to color variations.

[0027] In the above-mentioned method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation, step (2) involves inputting image patches in parallel into multiple pre-trained teacher models to extract high-dimensional feature vectors. These features are then integrated into a unified enhanced feature representation through a learnable feature fusion network. Specifically, in step (2), the system configures at least two pre-trained pathology models as teacher models, such as UNI v2 and Virchow 2. Both models have been pre-trained on large-scale general pathology datasets and are adept at capturing macroscopic tissue structure patterns and microscopic cell morphology details, respectively. During training, the parameters of all teacher models are frozen and do not participate in gradient updates; they are used only as feature extractors. Each image patch is simultaneously fed into these two teacher models, which extract high-dimensional feature vectors from the last feature map before their global average pooling layer or classification head, respectively. These vectors have dimensions of 1024 and 2048. These raw feature vectors are then fed into the feature fusion network for integration. This feature fusion network first includes a feature alignment module, which consists of two independent fully connected layers, corresponding to the outputs of the two teacher models. Each fully connected layer projects the high-dimensional input features to a unified intermediate dimension, 512. The projection layer uses a linear activation function, and its weight matrix is ​​initialized using a Kamming uniform distribution to ensure the stability of gradient flow in the early stages of training. After feature alignment, the fusion network enters the feature concatenation module. Before the concatenation operation, to eliminate scale differences in the output features of different teacher models, the system performs L2 normalization on each 512-dimensional projected feature vector, with the normalization coefficient set to 0.0001 to prevent numerical instability. Subsequently, the two normalized feature vectors are concatenated along the channel dimension to form a 1024-dimensional concatenated feature vector. Finally, this concatenated feature is fed into the nonlinear fusion module, which is a learnable neural network consisting of two fully connected layers. The first fully connected layer maps the 1024-dimensional input to a 512-dimensional output and uses a Gaussian error linear unit as the activation function to introduce nonlinear expressive power; the second fully connected layer further compresses the 512-dimensional features to 256 dimensions and uses a hyperbolic tangent activation function for the final nonlinear transformation. To prevent overfitting, a random deactivation layer with a dropout rate of 0.2 is placed between the two layers, and 20% of the neurons are randomly reset to zero during training. After these operations, the feature fusion network finally outputs a 256-dimensional enhanced feature representation, which integrates complementary knowledge from multiple teacher models and forms the target for subsequent joint distillation training.

[0028] In the above method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation, step (3) constructs a student model, which is forced to simultaneously imitate the output behavior of all teacher models and learn their fused feature representations through the joint distillation loss function. Specifically, in step (3), the student model adopts a visual transformer architecture, which contains 12 encoder layers, each containing 8 attention heads, a hidden layer dimension of 768, and an internal dimension of 3072 for the feedforward network. The positional encoding adopts a learnable one-dimensional positional embedding to preserve the spatial order information of the image patch. The student model receives the image patch preprocessed in step (1) as input and outputs a 256-dimensional feature vector, which is strictly aligned with the output dimension of the fusion network in step (2). The joint distillation loss function consists of a weighted sum of three parts to comprehensively guide the learning process of the student model. The first part is the feature imitation loss, which aims to make the output features of the student model as close as possible to the enhanced feature representations of the teacher models after fusion. This loss uses mean squared error to calculate the distance between the two, and its mathematical expression is as follows: Where N is the batch size, f δ (i) and f t (i) The first part represents the student model output feature and the teacher model fusion feature for the i-th sample, respectively. The second part is the logistic output distillation loss, which aims to make the student model mimic the soft probabilistic output of the teacher model in various diagnostic tasks. Since the teacher model itself does not directly output multi-task logistic values, the system first attaches a lightweight task header to each teacher model to obtain its logistic output in each task. Then, these logistic outputs are weighted and averaged or voted to form the final teacher soft label. The difference between the student model's logistic output and this soft label is measured by KL divergence, and the distillation temperature is set to 2.5 to smooth the probability distribution and make knowledge transfer more effective. The third part is the multi-task supervision loss, which directly uses real labeled data to supervise the student model. For classification tasks (such as lesion grading and molecular subtyping), cross-entropy loss with label smoothing technology is used, and the smoothing parameter is set to 0.1 to alleviate overfitting; for regression tasks (such as treatment response scoring), mean squared error loss is used. The weight hyperparameters for these three loss components were determined on the validation set using grid search, and were ultimately set as follows: feature imitation loss weight 0.4, logistic output distillation loss weight 0.3, and multi-task supervision loss weight 0.3. The total loss function was optimized using the adaptive moment estimation algorithm, with an initial learning rate of 0.0001 and a batch size of 32. Training was performed on a distributed graphics processing unit cluster, with a single complete training cycle not exceeding 24 hours.

[0029] In the above-mentioned method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation, step (4) connects multiple parallel task-specific prediction heads on the fusion features output by the student model to perform various diagnostic and treatment tasks. During inference, after all image patches are processed by the system, their prediction results are aggregated to form the final full-slice image-level prediction. Specifically, in step (4), the task-specific prediction heads include a breast lesion grading prediction head, a molecular subtyping prediction head, a tumor staging prediction head, and a treatment response prediction head, totaling four independent prediction branches. Each prediction head consists of two fully connected layers: the first layer maps the 256-dimensional features output by the student model to 128 dimensions and uses a modified linear unit activation function; the second layer sets the output dimension according to the needs of the specific task. For example, if the lesion grading is 4 categories, the output dimension is 4; if the molecular subtyping is 3 categories, the output dimension is 3; if the treatment response is a continuous value, the output dimension is 1. The second layer uniformly uses a linear activation function. During the inference phase, a full-slice image is segmented into hundreds to thousands of image patches. Each patch is independently processed by the student model and its corresponding task head to obtain its own patch-level prediction result. To obtain the final diagnostic conclusion at the full-slice level, the system employs two aggregation strategies. The first is average pooling aggregation, which takes the arithmetic mean of the prediction results (whether probability distributions or regression values) of all image patches on the same task. This method is simple and effective, suitable for scenarios where lesions are relatively evenly distributed. The second is attention-weighted aggregation, a more refined strategy that learns the importance weights of each image patch through an additional attention network. This attention network is a single-layer perceptron, taking the 256-dimensional features of the student model as input and outputting a dimension of 1. Its output is normalized by a softmax function to obtain the attention weights of all image patches. The final full-slice prediction result is the weighted sum of the prediction results of each image patch and its corresponding attention weight. The system can dynamically select an appropriate aggregation strategy based on task characteristics or user needs. Furthermore, to ensure the reliability and continuous evolution capability of the model, this invention also includes a model validation module and an online update mechanism. The model validation module employs five-fold cross-validation to evaluate model performance. The training and validation sets for each fold are split using stratified sampling to maintain consistent class distribution. Evaluation metrics include accuracy, precision, recall, and F1 score; all metrics must reach at least 90% for deployment. The online update mechanism continuously monitors the accumulation of newly labeled data during system operation. When the amount of new data reaches a preset threshold (e.g., equivalent to 10% of the initial training set), an incremental learning process is automatically triggered. This process uses an elastic weight consolidation algorithm, calculating the importance weights of each parameter in the network using the Fisher information matrix. During training on new data, stronger regularization constraints are applied to important parameters, with a regularization coefficient set to 0.8. This effectively protects old knowledge while learning new knowledge, preventing catastrophic forgetting.The model's complete update cycle is set to once every 3 months to balance model performance and maintenance costs. The entire system is deployed on a distributed computing platform, supporting parallel processing of multiple whole-slice images during the inference phase. The processing time for a single image is less than 3 minutes, and the system throughput reaches 20 whole-slice images per hour, meeting the timeliness requirements of practical clinical applications.

[0030] To illustrate the practical application effect of the present invention more specifically, an application scenario example is constructed. Assume that the pathology department of a tertiary hospital needs to process about 50 whole-slice images of breast cancer patients every day, and the tasks cover Nottingham classification, ER / PR / HER2 molecular subtyping, TNM staging, and pathological complete response (pCR) prediction after neoadjuvant chemotherapy. After deploying the method described in this invention, the system first performs the preprocessing of step (1) on all whole-slice images of the day to generate a standardized image block sequence. Subsequently, on the student model built and trained in steps (2) and (3), the multi-task prediction of step (4) is performed on each image block. For the Nottingham classification task, the system adopts attention-weighted aggregation because high-grade tumors are often concentrated in specific areas, and the attention mechanism can effectively focus on these key areas; while for the molecular subtyping task, since its expression is relatively uniform, the system chooses average pooling aggregation. Finally, the system outputs the prediction results of four clinical indicators for each whole-slice image within 3 minutes, and attaches a visual heatmap to indicate the spatial distribution of each prediction result on the tissue slice. According to internal testing, the system achieved F1 scores of 92.5%, 91.8%, 90.3% and 89.7% on the four tasks, respectively, all exceeding the preset threshold of 90%. Moreover, its reasoning speed far surpasses that of the traditional manual review process, significantly improving diagnostic efficiency and consistency.

[0031] In this embodiment, a method for establishing a visual model of breast pathology based on multi-model fusion and joint distillation is described, such as... Figure 2 As shown, the specific implementation method is as follows: (1) Input and preprocessing module: Receives the raw WSI, automatically performs tissue region detection and segmentation, and then cuts the tissue region into fixed-size image patches (e.g., cropped into 256x256 pixel patches at 20x magnification). Simultaneously, a WSI with good staining is manually selected, and a region is cropped from it as a staining normalization template. Color normalization is performed using existing RestainNet techniques to eliminate staining differences.

[0032] (2) Multi-source feature extraction and fusion module: 2.1 Input the image patches in parallel into multiple pre-trained teacher models (with fixed parameters, including at least UNI v2 and Virchow 2) to extract high-dimensional feature vectors.

[0033] Teacher models: These include at least two pre-trained models, UNI and Virchow, whose parameters are frozen and do not participate in training. They represent different, advanced prior knowledge for understanding pathological images.

[0034] Process: The same image patch is input into all teacher models in parallel.

[0035] Output: Each teacher model outputs a high-dimensional feature vector. For example, UNI outputs a 1024-dimensional vector, while Virchow might output a 2048-dimensional vector. These feature vectors typically come from the model's global average pooling layer or the layer before the classification head.

[0036] 2.2 Subsequently, these features are integrated into a unified, enhanced feature representation through a learnable feature fusion network. The specific operation of the feature fusion model is as follows: Step 1: Feature Alignment: The calculation formula for the features extracted by each teacher model is as follows: Among them, h k T is the feature vector extracted from the k-th teacher model. k Let x be the feature extraction function for the k-th teacher model, and let x be the original input of the model. Since the feature dimensions (d1, d2, ...) output by different teacher models may be inconsistent, we first project each teacher feature to the same dimension d using a simple fully connected layer (linear projection). common .

[0037] Among them, F k The output vector of the features of the k-th teacher model after projection, MLP k This is the fully connected projection layer corresponding to the k-th teacher model; Step 2: Feature concatenation / aggregation: Concatenate the projected feature vectors along the channel dimension.

[0038] Among them, F concat This represents the final aggregated feature after concatenating the projected features of all teacher models along the channel dimension. `Concat` is the channel-dimensional concatenation operation. [F1, F2, ..., F...] k This is a list of projected features arranged in order of the teacher model. Step 3: Nonlinear Fusion: Input the concatenated features into a small, learnable neural network. This network typically consists of fully connected layers, activation functions (such as ReLU, GELU), and Dropout layers.

[0039] (3) Joint Distillation Training Module: This is the core of model training. We construct a student model (e.g., a Vision Transformer), whose architecture may differ from that of the teacher model. Through the joint distillation loss function, the student model is forced to simultaneously mimic the output behavior of all teacher models and learn their fused feature representations.

[0040] In step 3, "Combined Distillation Training," the early stopping method execution logic is added, and the specific process is as follows: Validation set partitioning: Before model training, the total dataset (including labeled whole-slice image patches and corresponding multi-task labels) is divided into a training set and an independent validation set in a 3:1 ratio. This ensures that the case distribution, pathological type, and staining style of the validation set are consistent with those of the training set (using stratified sampling) to avoid data distribution bias affecting the reliability of early intervention judgment. The validation set must cover all categories of samples for the four tasks: breast lesion grading, molecular subtyping, tumor staging, and treatment response prediction. The validation sample size for each task should not be less than 20% of the total sample size for that task.

[0041] The monitoring indicator is defined as follows: the "multi-task weighted average F1 score" is selected as the core indicator for early intervention monitoring. Its weight is consistent with the weight of the multi-task supervision loss in the combined distillation loss function (i.e., allocated according to the clinical importance of each task, such as 0.35 for tumor subtyping, 0.3 for molecular subtyping, 0.2 for lesion grading, and 0.15 for treatment response prediction). Simultaneously, the total loss value of combined distillation on the validation set is monitored. Early intervention is triggered when the core indicator stagnates and the total loss value continues to decrease.

[0042] Early stop parameter configuration: Patience: Set to 15 training epochs. This means that an early stop warning is triggered when the multi-task weighted average F1 score on the validation set does not exceed the current optimal value for 15 consecutive epochs. This parameter is set based on the sample complexity of breast pathology data (a single WSI corresponds to thousands of image patches, resulting in a long training cycle) to avoid false stoppages caused by short-term index fluctuations (such as fluctuations caused by annotation errors in individual image patches).

[0043] Minimum improvement threshold (MinDelta): Set to 0.001 (i.e. 0.1%). The "optimal model" record is updated and the patience counter is reset only when the core metric of the validation set improves by more than 0.001 from the current best value; if the improvement is less than or equal to 0.001, it is considered that the metric has not improved effectively, and the patience counter is incremented by 1.

[0044] Optimal model saving mechanism: During training, after each training cycle, the validation set core metric and the total joint distillation loss are calculated. If the core metric exceeds the historical best value, all parameters of the current student model (including encoder layer weights, attention head parameters, and task-specific prediction head parameters) and the hyperparameter configuration of the joint distillation loss function are immediately saved. The saved file is named "student_model_best_epoch_{epoch}f1{score}.pth", and metadata such as the corresponding training cycle, training set loss, and validation set loss are also recorded.

[0045] Early stop triggering and recovery mechanism: When the patience counter reaches 15, an early stop is triggered, immediately terminating the joint distillation training process, abandoning subsequent training cycles, and directly loading the parameters of the last saved "optimal model" as the final training result.

[0046] If the core metrics temporarily drop (less than 15 cycles) during training due to data noise or abnormal samples, the patience counter will be reset to 0 and the optimal model parameters will be updated and saved when the metrics recover and exceed the historical best value in subsequent cycles.

[0047] Training epoch cap: The maximum training epoch is set to 100 epochs. Even if the validation set metrics are still slowly improving (without triggering early stopping), training will automatically stop when the training epoch reaches 100 epochs to avoid wasting computational resources due to overtraining. This cap is based on previous experimental results (the student model of this invention typically reaches peak generalization ability within 80-90 epochs).

[0048] III. Synergistic Adaptation of Early Stopping Method and Combined Distillation Training Collaborative loss function: The validation set loss monitored by the early stopping method is the joint distillation total loss (feature imitation loss + logical output distillation loss + multi-task supervision loss), ensuring that the early stopping judgment considers both the knowledge distillation effect and the multi-task prediction accuracy, avoiding the imbalance of "good distillation effect but low task prediction accuracy" or vice versa.

[0049] Gradient update adaptation: Early stopping does not affect the gradient calculation and parameter update logic during training. It only performs index evaluation and judgment after each training cycle, which is consistent with the optimization rhythm of the adaptive moment estimation algorithm (Adam) (the validation index is calculated after each epoch).

[0050] Model stability guarantee: If the validation set metrics fluctuate drastically during training (e.g., a drop of more than 5% in a single period), it is considered an abnormal situation. At this time, the early stop counter is paused and the metric evaluation of that period is automatically skipped to avoid accidental stoppages caused by labeling errors, abnormal data preprocessing, or other random factors.

[0051] (4) Multi-task prediction module: After training, the student model becomes our "major visual model of breast pathology". On top of its output fusion features, multiple parallel task-specific prediction heads (usually fully connected layers) are connected to perform the aforementioned diagnostic and treatment tasks.

[0052] Methodological Details (1) Multi-source feature extraction and adaptive fusion Let the input image patch be x, and we have k pre-trained teacher models. The features extracted by each teacher model are: ,in .

[0053] Simple concatenation or weighted averaging may not be optimal. This invention employs a lightweight adaptive fusion module, as shown in the following equation, resulting in the following fused features: (2) Combined distillation based on DINOv2 and multi-task output Student Model The output for the same image patch x is .

[0054] Combined distillation loss It consists of three parts: Feature imitation loss: The formula for calculating the ability of the student model to learn the representation of the fused teacher features is as follows: Logical Output Distillation Loss: For each labeled task (e.g., classification), we compute the KL divergence of the logistic values ​​(logits) output by the teacher and student models for that task. Let the logits obtained by the teacher model through its prediction head be... The student model is .

[0055] Among them, L t logit The distillation loss is the logical output of the t-th task. It is the softmax function. It is the distillation temperature, D KL Let KL divergence be denoted as KL divergence.

[0056] Multi-task supervised training loss: The student model is directly trained under supervised supervision using available labeled data. For classification tasks, cross-entropy loss is used; for regression tasks (such as T-stages), mean squared error loss is used. .

[0057] Among them, L sup The sum of multi-task supervision losses for all tasks. Let L be the supervision loss weight for the t-th task. t task Let be the single-task supervision loss for the t-th task; The overall combined distillation loss function is: in, , , It is a hyperparameter used to balance the weights of various losses.

[0058] (3) Design of multi-task prediction head After the student model is trained, we attach several simple prediction heads to it. Each head Using the features output by the student model as input, the system outputs the prediction results for the corresponding task through one or more fully connected layers. During inference, all image patches of a WSI are processed by the system, and the prediction results are aggregated using strategies such as averaging and attention weighting to form the final WSI-level prediction.

[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation, characterized in that, include: Step 1: Input and preprocessing. Receive the original whole-slice image, automatically detect and segment the tissue region, cut the tissue region into image blocks of fixed size, and manually select a whole-slice image with good staining and crop a region from it as a staining normalization template. Eliminate staining differences through color normalization processing. Step 2: Multi-source feature extraction and fusion. Image patches are input in parallel into multiple pre-trained teacher models to extract high-dimensional feature vectors. Then, these features are integrated into a unified enhanced feature representation through a learnable feature fusion network. Step 3: Joint distillation training. Construct a student model by using the joint distillation loss function to force the student model to simultaneously mimic the output behavior of all teacher models and learn their fused feature representations. Step 4: Multi-task prediction. Multiple parallel task-specific prediction heads are connected on top of the fused features output by the student model to perform various diagnostic tasks. During inference, the prediction results of all image patches are processed by the system and then aggregated to form the final full-slice image-level prediction.

2. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 1, characterized in that: In step 1, the image patch is cropped to a fixed size of 256 pixels by 256 pixels at a magnification of 20x. The tissue region detection adopts a semantic segmentation algorithm based on deep learning, with a segmentation accuracy of greater than or equal to 95%. The color normalization processing adopts a color transfer technique based on generative adversarial networks, with a color consistency error of less than 5%.

3. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 1, characterized in that: In step 2, the teacher model includes at least two pre-trained pathological basic models. The model parameters are frozen and do not participate in training. Each teacher model outputs a high-dimensional feature vector with feature dimensions of 1024 and 2048 respectively. These feature vectors come from the model's global average pooling layer or the layer before the classification head.

4. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 3, characterized in that: The feature fusion network includes a feature alignment module, which projects each teacher's feature to the same dimension through a fully connected layer. The projected feature dimension is unified to 512 dimensions. The projection layer uses a linear activation function, and the weights are initialized using a Kamming uniform distribution.

5. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 4, characterized in that: The feature fusion network also includes a feature splicing module, which splices the projected feature vectors along the channel dimension. The spliced ​​feature dimension is 1024. Before the splicing operation, each feature vector is subjected to L2 normalization, and the normalization coefficient is 0.0001.

6. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 5, characterized in that: The feature fusion network further includes a nonlinear fusion module, which inputs the spliced ​​features into a learnable neural network consisting of two fully connected layers. The first fully connected layer has an output dimension of 512 and uses the Gaussian error linear unit activation function, while the second fully connected layer has an output dimension of 256 and uses the hyperbolic tangent activation function. A random deactivation layer with a dropout rate of 0.2 is set between the two layers.

7. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 1, characterized in that: In step 3, the student model adopts a visual transformer architecture, which includes 12 encoder layers, each containing 8 attention heads, a hidden layer dimension of 768, an internal dimension of 3072 for the feedforward network, and a learnable one-dimensional position embedding for position encoding.

8. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 1, characterized in that: The combined distillation loss function consists of three parts. It includes feature imitation loss, logistic output distillation loss, and multi-task supervision loss. Feature imitation loss uses mean squared error to calculate the distance between the student model's output features and the fused teacher features. Logistic output distillation loss uses KL divergence to calculate the difference between the teacher model and the student model's output logistic values ​​in the task. Multi-task supervision loss uses cross-entropy loss and mean squared error loss respectively, depending on the task type.

9. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 8, characterized in that: The feature imitation loss is calculated as the sum of squares of the differences between feature vectors. The distillation temperature in the logical output distillation loss is set to 2.

5. The classification task in the multi-task supervision loss uses label smoothing technology with a smoothing parameter set to 0.

1. The weight hyperparameters in the joint distillation loss function are determined through grid search. The weight of the feature imitation loss is set to 0.4, the weight of the logical output distillation loss is set to 0.3, and the weight of the multi-task supervision loss is set to 0.

3. The total loss function is optimized using the adaptive moment estimation algorithm, with an initial learning rate of 0.0001 and a batch size of 32.

10. The method for establishing a visual model of breast pathology based on multi-model fusion and combined distillation according to claim 1, characterized in that: In step 4, the task-specific prediction heads include a breast lesion grading prediction head, a molecular subtyping prediction head, a tumor staging prediction head, and a treatment response prediction head. Each prediction head consists of two fully connected layers. The first layer has an output dimension of 128 and uses a modified linear unit activation function. The output dimension of the second layer is set according to the task requirements and uses a linear activation function. The aggregation strategy includes average pooling aggregation and attention-weighted aggregation. Average pooling aggregation takes the arithmetic mean of the prediction results for all image patches. Attention-weighted aggregation learns the importance weights of each image patch and performs a weighted summation. The attention weights are calculated through a single-layer perceptron, with an output dimension of 1, and are normalized using the softmax function.

Images