Three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation

Abstract

The application discloses a three-dimensional microscopic image recognition model training method and system based on cross-dimension knowledge distillation. The three-dimensional microscopic image recognition model to be trained is taken as a student model, a pre-trained two-dimensional microscopic image recognition model is taken as a teacher model, hard labels are obtained by manually labeling strong supervision training three-dimensional microscopic image sample data in the three-dimensional microscopic image sample data, soft labels are obtained by identifying the supervision training three-dimensional microscopic image sample data by using the teacher model, and the supervision training three-dimensional microscopic image sample data is used for mixed supervision knowledge distillation training, so that the student model converges in training and is used as the three-dimensional microscopic image recognition model. The application realizes cross-dimension knowledge migration from a two-dimensional pre-trained image recognition large model to a three-dimensional student model, effectively utilizes three-dimensional information to improve the reasoning capability of the three-dimensional student model, and thus breaks through the performance bottleneck of the three-dimensional microscopic image recognition model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional image, more particularly, to a three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation. BACKGROUND

[0002] The evolution of microscopic imaging technology from two-dimensional to three-dimensional is a technological revolution spanning hundreds of years in the history of science. The earliest microscopic imaging can be traced back to the single-lens optical microscope invented by Dutch scientist Leeuwenhoek in the 17th century, which first revealed the two-dimensional morphology of cells and microorganisms. In the late 20th century, the birth of confocal microscopy marked the beginning of three-dimensional imaging, which used pinhole filtering technology to achieve optical sectioning and reconstruct three-dimensional structures through layer-by-layer scanning, but was mainly applied to fixed samples due to speed limitations. In the 21st century, two-photon microscopy, super-resolution fluorescence microscopy, and optical sectioning microscopy have been developed and applied.

[0003] Microscopic imaging technology has achieved revolutionary applications in the fields of life sciences and medicine by breaking through the limits of resolution and dynamic observation. For example, microscopic imaging technology applied to auxiliary diagnosis takes the microscopic image of the sample as input and uses an image recognition model to identify the pathological sample, thereby helping doctors make a diagnosis. In theory, three-dimensional images contain more information and are more conducive to precise identification of samples by microscopic image recognition models. However, due to longer development time and more widespread application, artificially annotated two-dimensional microscopic imaging data far exceeds three-dimensional microscopic imaging data in scale and quality. Limited by the scale and quality of artificially annotated data, the recognition accuracy of three-dimensional microscopic image recognition models needs to be improved, and the scope of application is limited.

[0004] The traditional two-dimensional-based microscopic image recognition model has obvious dimensional defects. Although two-dimensional microscopic images are relatively easy to obtain and have lower annotation costs, their inherent planar characteristics cannot accurately reflect the true three-dimensional spatial distribution characteristics, such as tumor morphology and distribution, resulting in systematic bias in model application. Image recognition models based on three-dimensional microscopic images using 3D pathological data for end-to-end training face severe challenges. Due to the difficulty of pathological data annotation and limited sample size, models trained from scratch often struggle to break through performance bottlenecks and have obvious deficiencies in feature extraction capability and generalization.

[0005] Diba et al. proposed a "spatio-temporal channel correlation (STC)" module to model the spatio-temporal correlation between 3D CNN channels through spatial and temporal branches to enhance feature representation, and introduced a knowledge transfer method from pre-trained 2D CNN to 3D CNN, realizing cross-dimension knowledge transfer. However, this method must introduce time dimension information and cannot be applied to knowledge transfer from static 2D image recognition models to 3D image recognition models. SUMMARY

[0006] In order to solve the above defects or improvement needs of the prior art, the present application provides a three-dimensional microscopic image recognition model training method and system based on cross-dimension knowledge distillation, which aims to use a two-dimensional microscopic image recognition model trained by a large amount of data as a teacher model, adopt three-dimensional microscopic image sample data, and combine data soft labels and hard labels for hybrid supervised knowledge distillation training, thereby solving the technical problems that the three-dimensional microscopic recognition model cannot break through the inference ability limitation of the two-dimensional microscopic image recognition model as the teacher model, and the three-dimensional microscopic recognition model cannot well utilize the information of three-dimensional images and limit the inference ability due to insufficient three-dimensional annotation data.

[0007] To achieve the above-mentioned purpose, according to one aspect of the present application, a three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation is provided, comprising the following steps:

[0008] The three-dimensional microscopic image recognition model to be trained is used as a student model, and the pre-trained two-dimensional microscopic image recognition model is used as a teacher model. The hard labels are obtained by manually labeling the strong supervised training three-dimensional microscopic image sample data in the three-dimensional microscopic image sample data for supervised training. The soft labels are obtained by using the teacher model to recognize the three-dimensional microscopic image sample data for supervised training. The hybrid supervised knowledge distillation training is performed using the three-dimensional microscopic image sample data for supervised training, and the student model trained to converge is used as the three-dimensional microscopic image recognition model.

[0009] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation acquires training data according to the following method:

[0010] (1) Data hard label preparation: the strong supervised training three-dimensional microscopic image sample data is manually labeled to obtain the class of the three-dimensional microscopic image sample data, which is used as the hard label of the strong supervised training three-dimensional microscopic image sample data; the strong supervised training three-dimensional microscopic image sample is the manually labeled sample.

[0011] (2) Data soft label preparation: the three-dimensional microscopic image sample data for supervised training including the strong supervised training three-dimensional microscopic image sample data is simulated and compressed into multiple two-dimensional microscopic image samples by two-dimensional microscopic imaging; the feature vectors of the multiple two-dimensional microscopic image samples are obtained by using the teacher model to extract features of the multiple two-dimensional microscopic image samples; the feature vectors of the multiple two-dimensional image samples are mean-pooled and fused into the feature vector of the three-dimensional microscopic image, which is used as the data soft label of the three-dimensional microscopic image sample data.

[0012] The supervised training three-dimensional microscopic image sample data is composed of strong supervised training three-dimensional microscopic image sample data and weak supervised training three-dimensional microscopic image sample data, the strong supervised training three-dimensional microscopic image sample data has hard labels and soft labels, and the weak supervised training three-dimensional microscopic image sample data only has soft labels.

[0013] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation has a strong supervised training three-dimensional microscopic image sample data amount not less than 1 / 10 of a weak supervised training three-dimensional microscopic image sample data amount, and a preferred ratio of the strong supervised training three-dimensional microscopic image sample data amount to the weak supervised training three-dimensional microscopic image sample data amount is between 1:2 and 1:10.

[0014] When the supervised training three-dimensional microscopic image sample data is a pathological sample, the strong supervised training three-dimensional microscopic image sample data amount is greater than or equal to 20,000, and the weak supervised training three-dimensional microscopic image sample data amount is greater than or equal to 40,000.

[0015] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation trains the student model with the loss function minimization as the target; the loss function is used to represent the comprehensive difference between the three-dimensional microscopic image recognition model as the student model and the two-dimensional microscopic image recognition model as the teacher model and the difference between the student model and the artificial annotation result.

[0016] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation has the loss function represented as a weighted sum of hard label loss and soft label loss; the loss function LOSS is recorded as:

[0017] LOSS = λLOSS h + LOSS s

[0018] wherein, LOSS h is the hard label loss, i.e. the binary cross-entropy loss; λ is the weight coefficient of the hard label loss, and the value is between [0, 1]; LOSS s is the soft label loss.

[0019] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation has LOSS s as the soft label loss, and the calculation method is as follows:

[0020] LOSS s = αD KL + (1-α)MSE

[0021] wherein, α is a weight coefficient, and the value range is (0, 1), DKL For divergence soft label loss, MSE is MSE soft label loss, which is calculated according to the following method respectively:

[0022]

[0023] During the training process, the KL divergence between the feature vectors P output by the teacher model and the feature vectors Q output by the student model for the three-dimensional microscopic image sample data used for supervised training is calculated; wherein x represents all dimensions of each image feature vector, z is a certain dimension of the feature vector, P(x) and Q(x) are the values corresponding to the dimension x in the feature vector respectively; n is the number of samples;

[0024]

[0025] Wherein, P is the feature vector output by the teacher model for the three-dimensional microscopic image sample data used for supervised training, Q is the feature vector output by the student model for the three-dimensional microscopic image sample data used for supervised training, ||·||2 is the Euclidean norm, and n is the number of samples.

[0026] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimensional knowledge distillation adopts progressive training for knowledge distillation training, that is, the weight of the prediction difference of the data soft label sample and the prediction difference of the student model for the data hard label sample is adjusted according to the training degree during the knowledge distillation training process: the higher the training degree of the student model, the smaller the proportion of the weight of the prediction difference of the data soft label sample in the loss function.

[0027] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimensional knowledge distillation, the student model and the teacher model are based on the same architecture.

[0028] Preferably, the three-dimensional microscopic image recognition model training method based on cross-dimensional knowledge distillation, the student model specifically includes an image preprocessing module for mapping the input image to an embedding space to obtain a dense vector of high-dimensional semantic features; the image preprocessing module includes a one-dimensional position encoder and / or an adapter;

[0029] The one-dimensional position code introduced is learnable, and the dimension is the same as the embedding vector, and the position code output is directly superimposed on the extracted feature vector;

[0030] The input of the adapter is the dense vector of the encoded input image, and the output is a dense vector of the same dimension, which includes a lower projection layer, a deep 3D convolution layer, and an upper projection layer connected in turn, and has trainable parameters.

[0031] According to another aspect of the present application, a three-dimensional microscopic image recognition model training system based on cross-dimension knowledge distillation is provided for an electronic device or a non-transitory computer readable storage medium;

[0032] The electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation provided by the present application when executing the program.

[0033] The non-transitory computer readable storage medium has a computer program stored thereon, and the computer program is executable on a processor to implement the steps of the three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation provided by the present application.

[0034] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0035] The three-dimensional microscopic image recognition model training method and system based on cross-dimension knowledge distillation provided by the present application use a two-dimensional microscopic image recognition model trained by a large amount of data as a teacher model, make soft labels and hard labels for three-dimensional microscopic image sample data respectively, and perform strong-weak mixed supervised knowledge distillation training to realize cross-dimension knowledge transfer from a two-dimensional pre-trained image recognition large model to a three-dimensional student model and effectively utilize three-dimensional information to improve the reasoning ability of the three-dimensional student model, thereby breaking through the performance bottleneck of a three-dimensional microscopic image recognition model, especially a pathological model, which is difficult to obtain a three-dimensional microscopic image recognition model with reasoning ability superior to an existing two-dimensional microscopic image recognition model due to the scarcity of manually annotated training data.

[0036] In a preferred embodiment, the present application uses a multi-dimensional loss function and a dynamic training strategy, innovatively combines a multi-dimensional loss function of soft label loss and hard label loss, and uses a dynamic loss switching mechanism to adjust the relative weights of soft label loss and hard label loss as the training progresses, avoids knowledge forgetting problems, significantly improves the model convergence speed and training stability, and enables the model to maintain excellent reasoning performance under the condition of limited supervised training with three-dimensional microscopic image sample data.

[0037] In a preferred embodiment, the student model increases an adapter to realize dimension adaptation from a two-dimensional model to a three-dimensional model, has lightweight parameters, and only needs to fine-tune a small number of parameters to realize dimension conversion, significantly reduces the calculation cost, avoids the overfitting risk brought by full parameter training, and enables the model to efficiently learn the feature representation of three-dimensional images under the condition of limited supervised training with three-dimensional microscopic image sample data. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1A three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation provided by the present application is shown in the flowchart;

[0039] Figure 2 A student model structure diagram provided by an embodiment of the present application is shown in the figure;

[0040] Figure 3 A student model recognition accuracy test result provided by embodiment 1 of the present application is shown in the table;

[0041] Figure 4 A student model recognition accuracy test result provided by embodiment 2 of the present application is shown in the table. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0043] The three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation provided by the present application comprises the following steps:

[0044] The three-dimensional microscopic image recognition model to be trained is used as a student model, and the pre-trained two-dimensional microscopic image recognition model is used as a teacher model. The hard label is obtained by manually labeling the strong supervision training three-dimensional microscopic image sample data in the supervision training three-dimensional microscopic image sample data. The soft label is obtained by using the teacher model to recognize the supervision training three-dimensional microscopic image sample data. The mixed supervision knowledge distillation training is performed using the supervision training three-dimensional microscopic image sample data to obtain the training converged student model for use as a three-dimensional microscopic image recognition model. Specifically:

[0045] The training data is obtained by the following method:

[0046] (1) Data hard label preparation: the strong supervision training three-dimensional microscopic image sample data is manually labeled to obtain the class of the three-dimensional microscopic image sample data, which is used as the hard label of the strong supervision training three-dimensional microscopic image sample data; the strong supervision training three-dimensional microscopic image sample is the manually labeled sample.

[0047] (2) Data soft label preparation: the three-dimensional microscopic image sample data for supervised training, including the three-dimensional microscopic image sample data for strong supervision training, is simulated and compressed into multiple two-dimensional microscopic image samples; a teacher model is used to extract features of the multiple two-dimensional microscopic image samples to obtain feature vectors of the multiple two-dimensional microscopic image samples, and the feature vectors of the multiple two-dimensional image samples are mean-pooled and fused into a feature vector of the three-dimensional microscopic image as a data soft label of the three-dimensional microscopic image sample data.

[0048] The three-dimensional microscopic image sample data for supervised training is composed of three-dimensional microscopic image sample data for strong supervision training and three-dimensional microscopic image sample data for weak supervision training, the three-dimensional microscopic image sample data for strong supervision training has hard labels and soft labels, and the three-dimensional microscopic image sample data for weak supervision training only has soft labels.

[0049] The three-dimensional microscopic image sample data for strong supervision training needs to be manually labeled and is relatively limited in quantity, and the three-dimensional microscopic image sample data for weak supervision training is greater than the three-dimensional microscopic image sample data for supervised training. In order to better utilize three-dimensional information, the amount of three-dimensional microscopic image sample data for strong supervision training is not less than 1 / 10 of the amount of three-dimensional microscopic image sample data for weak supervision training. In the preferred scheme, the ratio of the amount of three-dimensional microscopic image sample data for strong supervision training to the amount of three-dimensional microscopic image sample data for weak supervision training is between 1:2 and 1:10, in order to balance the training effect of large amount of data learning and 3D information. In order to improve the accuracy of training, the amount of three-dimensional microscopic image sample data for strong supervision training is greater than or equal to 20,000, and the amount of three-dimensional microscopic image sample data for weak supervision training is greater than or equal to 40,000.

[0050] The student model is trained with the objective of minimizing the loss function; the loss function is used to represent the comprehensive difference between the three-dimensional microscopic image recognition model as the student model and the two-dimensional microscopic image recognition model as the teacher model and the difference between the student model and the artificial labeling result; specifically, the loss function is represented as the weighted sum of hard label loss and soft label loss; using the comprehensive difference as the loss function makes the three-dimensional student model not only capable of distilling the teacher model to achieve rapid training, but also capable of learning three-dimensional information to achieve cross-latitude distillation, break through the limitation of the training effect of the teacher model on the student model, and achieve the training effect of the student model surpassing the teacher model.

[0051] To match the data property difference between hard label and soft label, the hard label loss is the binary cross-entropy loss of the student model on the three-dimensional microscopic image sample data for strong supervision training, and the soft label loss is the weighted sum of the KL divergence and MSE loss of the student model on the three-dimensional microscopic image sample for supervision training. The hard label is characterized by non-black or white, limited information, and only forces the student model to fit the real distribution. Cross-entropy forces the student model to approach 1 for the prediction probability of class k and approach 0 for other classes, which meets the "hard classification" requirement of the hard label. The soft label is characterized by carrying the "implicit knowledge" (such as class correlation) of the teacher model, guiding the student model to learn more general feature representation. Among them, the KL divergence emphasizes "fitting the teacher distribution P from the student distribution Q", which does not symmetrically punish the difference between Q and P, and the KL divergence is sensitive to the "shape" of the distribution, which can effectively capture the relative confidence between classes in the teacher model, and is suitable for transmitting the implicit knowledge of the teacher model; while the MSE symmetrically punishes the difference of all classes, which is suitable for the scene that needs to uniformly fit the probability of all classes, but may pay too much attention to the low confidence class, resulting in dilution of the transmission of high confidence knowledge. Therefore, the soft label loss is composed of the KL divergence and the MSE, which achieves a complementary effect. The soft label loss composed of the KL divergence and the MSE loss avoids over-learning of the prediction results of the teacher model, absorbs the implicit knowledge of the teacher model, and realizes the cross-latitude distillation of the two-dimensional image recognition model to the three-dimensional image recognition model in combination with the hard label loss.

[0052] Specifically, the loss function LOSS is denoted as:

[0053] LOSS = λLOSS h + LOSS s

[0054] wherein, LOSS h is the hard label loss, i.e. the binary cross-entropy loss; λ is the weight coefficient of the hard label loss, and the value is between [0, 1]; LOSS s is the soft label loss, and the calculation method is as follows:

[0055] LOSS s = αD KL + (1-α)MSE

[0056] wherein, α is the weight coefficient, the value range is (0, 1), D KL is the divergence soft label loss, and MSE is the MSE soft label loss, which is calculated according to the following method:

[0057]

[0058] During the training process, the KL divergence between the feature vector P output by the teacher model for the three-dimensional microscopic image sample data for supervised training and the feature vector Q output by the student model is calculated; wherein, represents all dimensions of each image feature vector, x is a certain dimension of the feature vector, P(x), Q(x) are the values corresponding to the dimension x in the feature vector respectively; n is the number of samples;

[0059]

[0060] wherein, P is the feature vector output by the teacher model for the three-dimensional microscopic image sample data for supervised training, Q is the feature vector output by the student model for the three-dimensional microscopic image sample data for supervised training, ||·||2 is the Euclidean norm, and n is the number of samples.

[0061] In the preferred scheme, progressive training is adopted for knowledge distillation training, that is, the weight of the prediction difference of the data soft label sample and the prediction difference of the student model for the data hard label sample is adjusted according to the training degree during the knowledge distillation training: the higher the training degree of the student model, the smaller the proportion of the weight of the prediction difference of the data soft label sample in the loss function. In the preferred scheme, only the soft label loss is used to constrain the model in the initial stage of the training, that is, the weight coefficient λ of the hard label loss is 0, and as the number of training rounds increases, the hard label loss constraint is started and the weight coefficient λ of the hard label loss is continuously increased in the subsequent training. In order to ensure that the model will not forget the knowledge learned from the teacher model due to over-learning of the hard label, the weight of the hard label is usually not more than the weight of the soft label loss.

[0062] Dynamic adjustment of the weights of the hard label loss and the soft label loss in the knowledge distillation training can accelerate the convergence of the model and avoid the performance bottleneck of the student model due to insufficient ability or excessive dependence on the teacher. The main adjustment strategy is "emphasis on soft label in early stage and emphasis on hard label in later stage", that is, "learn ideas in early stage and learn judgments in later stage". In the early stage of knowledge distillation, the student model is insufficient in ability and needs to quickly absorb the experience (such as feature representation, class association, etc.) of the teacher model through the soft label, so the soft label is the main source of knowledge at this time. In the later stage of knowledge distillation, the student model has certain ability and needs to correct the potential errors of the teacher model to form an independent decision-making ability from the teacher, so the hard label is the key calibration signal at this time. The student model, based on the same architecture as the teacher model, generally includes a convolutional layer, and unlike the two-dimensional convolution kernel of the convolutional layer of the teacher model, the convolutional layer of the student model is correspondingly expanded to a three-dimensional convolution kernel; the student model specifically includes an image preprocessing module, and generally also includes a feature extraction module such as a Transformer encoder and a classifier:

[0063] The image preprocessing module is configured to map an input image to an embedding space, obtain a dense vector of high-dimensional semantic features, and output the dense vector, wherein the output dense vector is used for feature extraction and classification.

[0064] In a preferred embodiment, the image preprocessing module comprises a one-dimensional position encoder (3D Position Embedding) that introduces a learnable one-dimensional position encoding with the same dimension as the embedding vector, and the output position encoding is directly superimposed on the extracted feature vector to compensate for the characteristics of the Transformer encoder that is not sensitive to the input order, retain the differences brought by the three-dimensional image information, and facilitate cross-latitude distillation.

[0065] In a preferred embodiment, the image preprocessing module comprises an adapter, wherein the input of the adapter is the dense vector of the encoded input image, and the output of the adapter is a dense vector with the same dimension, which comprises a lower projection layer, a deep 3D convolution layer, and an upper projection layer connected in sequence, and has trainable parameters; the adapter enhances the information fusion of the 3D image through the deep 3D convolution layer, so that the student model suitable for the three-dimensional image and the teacher model suitable for the two-dimensional image adopt the same architecture, ensuring the distillation training effect while realizing the cross-latitude information preservation and training of the two-dimensional image to the three-dimensional image, combining the joint training of soft labels and hard labels, and realizing the cross-latitude distillation.

[0066] The Transformer encoder is configured to extract a feature sequence containing global information, and comprises:

[0067] a plurality of Transformer sublayers, wherein each layer comprises a multi-head self-attention layer for self-attention encoding, and a feedforward network connected in series with the multi-head self-attention layer for nonlinear transformation of the features;

[0068] In a preferred embodiment, the multi-head self-attention layer and the feedforward network are connected in series with a residual connection for linking the input and output of each sublayer, and a layer normalization layer for alleviating gradient vanishing.

[0069] The classifier is configured to identify the category of the input three-dimensional image according to the feature sequence.

[0070] The following is an example:

[0071] Example 1 is a three-dimensional microscopic image recognition model for breast cancer lymph node metastasis prediction

[0072] Accurate assessment of tumor metastasis characteristics is a key challenge in the diagnosis and treatment of various solid tumors. Lymph node metastasis, as one of the main routes of spread for various malignant tumors such as breast cancer, is not only a core basis for determining cancer stage and developing treatment plans, but also a key indicator for assessing patient prognosis. Traditional two-dimensional pathological sections cannot fully reflect the three-dimensional spatial characteristics of tumors, which may lead to missed diagnosis of small lymph node metastases, thus delaying the best treatment opportunity. Three-dimensional pathological data obtained by high-resolution light microscopy breaks through this limitation, its non-destructive imaging characteristics preserve the complete spatial structure of the tumor, significantly improve the utilization rate of samples, and can more accurately identify early-stage small metastatic lesions that are difficult to detect by traditional methods, which is beneficial to the early detection of small lesions. In the detection of key sites such as breast cancer sentinel lymph nodes, it shows unique diagnostic advantages. Combined with innovative cross-dimensional knowledge distillation technology, this method not only makes full use of the prior knowledge of existing two-dimensional data, but also improves the diagnostic accuracy through three-dimensional feature analysis, providing a reliable basis for early diagnosis of lymph node metastasis, and can better assess the degree of tumor infiltration and spatial distribution characteristics, providing important reference for prognosis analysis and individualized treatment of various solid cancers, and showing broad clinical application prospects.

[0073] The three-dimensional microscopic image recognition model training method based on cross-dimensional knowledge distillation provided in the embodiment includes the following steps:

[0074] Selection of teacher model: the teacher model selects the Virchow2.0 large model pre-trained on a large amount of two-dimensional pathological data. The model is based on the Vision Transformer architecture and has 632 million parameters. It has been pre-trained on 1.5 million whole-section histopathology images of about 100,000 patients, has strong feature extraction capability, and can achieve the most advanced performance in various computational pathology downstream tasks.

[0075] Structure of student model: the student model selects the same VIT architecture as the teacher model and uses a three-dimensional embedding layer. The student model of the embodiment is shown in Figure 2 , and specifically includes an image preprocessing module, a Transformer encoder, and a classifier:

[0076] The image preprocessing module is used to map the input image to the embedding space to obtain a dense vector of high-dimensional semantic features and input the Transformer encoder; the image preprocessing module includes an embedding layer, a one-dimensional position encoder, and an adapter;

[0077] The embedding layer linearly projects the flattened vector of the input image block obtained by three-dimensional blocking of the input image to the embedding space. The embedding layer has a learnable weight matrix E∈R 768×D(D is the embedding dimension, default 768) is implemented to obtain a sequence with dimension NxD, N is the number of input image blocks obtained by three-dimensional blocking of the input image.

[0078] The input image of the embodiment (such as the size of 56x224x224x3, 3 represents the RGB channel) is uniformly divided into fixed-size image blocks (Patches). In this embodiment, each block has a size of 7x14x14x3, and a total of N=56 / 7x224 / 14x224 / 14=2048 blocks are generated. Each block is flattened to obtain a one-dimensional vector 7x14x14x3 with a length of 4116, and then the vector is mapped to the embedding space through linear projection. This process is implemented through a learnable weight matrix E∈R 4116×D (D is the embedding dimension, default 1280) is implemented to obtain a sequence with dimension NxD.

[0079] A one-dimensional position encoder (3D Position Embedding) is introduced, which introduces a learnable one-dimensional position encoding with the same dimension as the embedding vector. The position encoding output is directly superimposed on the image block sequence projected by the Transformer encoder.

[0080] The input of the adapter is the dense vector of the encoded input image, and the output is a dense vector with the same dimension, which includes a lower projection layer, a deep 3D convolution layer, and an upper projection layer connected in sequence. It has trainable parameters, and the deep 3D convolution layer enhances the information fusion of 3D images, so that the student model suitable for three-dimensional images and the teacher model suitable for two-dimensional images use the same architecture, ensuring the distillation training effect while realizing the cross- latitude information preservation and training of two-dimensional images to three-dimensional images, combining the joint training of soft labels and hard labels, and realizing cross-latitude distillation.

[0081] The adapter is a lightweight module, and its core goal is to enable pre-trained models to efficiently adapt to different tasks or data modalities by inserting a small number of trainable parameters without modifying most of the parameters of the original model. Our model architecture is to distill a 3D student model from a 2D large model, and the adapter module is added in our student model to increase the adaptability of the student model to different dimensional data. In general, the adapter module and the student model architecture as similar as possible to the teacher model together determine that our architecture has more advantages in cross-dimension distillation.

[0082] The Transformer encoder is used to extract a feature sequence containing global information, including:

[0083] A plurality of Transformer sub-layers, wherein each layer comprises a multi-head self-attention layer for self-attention encoding, and a feed-forward network in series with the multi-head self-attention layer for non-linear transformation of the features; the multi-head self-attention layer and the feed-forward network are followed by a residual connection for linking the input and output of each sub-layer, and a layer normalization layer for alleviating gradient vanishing.

[0084] Multi-Head Self-Attention (MSA) splits the input into h heads (e.g. h = 12), respectively calculates the attention weight and splices the result, and integrates the multi-view features through linear projection. This mechanism allows the model to capture the long-range dependencies between image blocks.

[0085] Feed-Forward Network (FFN): composed of two fully connected layers, using GELU activation function in between, for non-linear transformation of the features at each position.

[0086] Residual connection and layer normalization layer: the output of each sub-layer (MSA, FFN) is added to the input through a residual connection, and a layer normalization layer (LayerNorm) is applied to alleviate the problem of gradient vanishing.

[0087] The embedded Token sequence is input into a Transformer encoder stacked by L layers (12 layers in this embodiment). The final output of the encoder is a feature sequence containing global information, in which the mean-pooled result of the Token sequence is used as the overall representation of the image.

[0088] The classifier is used to identify the category of the input three-dimensional image according to the feature sequence, and in this embodiment, an MLP classification head is used.

[0089] After all Tokens are processed by all encoding layers, they are sent to the classification head (MLP Head) through the layer normalization layer. The classification head in this embodiment is composed of two fully connected layers, containing Dropout and activation function in between, and the final output dimension is consistent with the number of target categories (e.g. sick or not sick classification of pathological images), completing the classification task.

[0090] The training data is obtained by the following method:

[0091] The present embodiment adopts a high-resolution light sheet microscope system to obtain three-dimensional pathological data of lymphoma tissue. This imaging technology can realize three-dimensional whole imaging with subcellular resolution, and completely retains the spatial structure information of tumor tissue. According to the characteristics of the original data, the anisotropy correction algorithm is used to eliminate the resolution difference of different axial directions. In order to adapt to the input requirements of the deep learning model, the large volume data is cut into standardized three-dimensional block data, which is about 60000 image blocks as the three-dimensional microscopic image sample data for supervised training. The image specification is 56x224x224 pixels.

[0092] (1) Data hard label preparation: three-dimensional microscopic image sample data for supervised training, of which 20000 image blocks are used as three-dimensional image samples for supervised training, and artificial labeling is performed. For breast cancer lymph node metastasis, the hard label value of the three-dimensional microscopic image sample for supervised training is 1, and the hard label value of other three-dimensional microscopic image samples for supervised training is 0.

[0093] (2) Data soft label preparation: input all three-dimensional microscopic image sample data for supervised training into the teacher model to obtain soft labels, and the specific steps are as follows:

[0094] The three-dimensional microscopic image sample data for supervised training with a specification of 56x224x224 pixels is sliced along the Z axis (light sheet scanning direction) to obtain 56 two-dimensional microscopic images with a specification of 224x224 pixels, which are input into the teacher model for feature extraction to obtain the feature vectors of the 56 two-dimensional microscopic images. The mean pooling fusion is performed to obtain the feature vector of the three-dimensional microscopic image sample data for supervised training, that is, the soft label of the three-dimensional microscopic image sample data for supervised training.

[0095] The student model is trained with the objective of minimizing the loss function. The loss function used in the present embodiment is:

[0096] The loss function LOSS is denoted as:

[0097] LOSS = λLOSS h + LOSS s

[0098] Wherein, LOSS h is the hard label loss, that is, the binary cross entropy loss; λ is the weight coefficient of the hard label loss, and the value is between [0, 1]; LOSS s is the soft label loss, and the calculation method is as follows:

[0099] LOSS s = αD KL +(1-α)MSE

[0100] Wherein, α is the weight coefficient, and the value range is (0, 1); D KLFor divergence soft label loss, MSE is MSE soft label loss, which is calculated as follows:

[0101]

[0102] During the training process, the KL divergence between the feature vectors P output by the teacher model and the feature vectors Q output by the student model for the three-dimensional microscopic image sample data used for supervised training is calculated; wherein, represents all dimensions of each image feature vector, x is a certain dimension of the feature vector, P(x) and Q(x) are the values corresponding to dimension x in the feature vector respectively; n is the number of samples;

[0103]

[0104] wherein P is the feature vector output by the teacher model for the three-dimensional microscopic image sample data used for supervised training, Q is the feature vector output by the student model for the three-dimensional microscopic image sample data used for supervised training, ||·||2 is the Euclidean norm, and n is the number of samples.

[0105] Kullback-Leibler divergence (KL divergence) is an index for measuring the difference between two probability distributions, widely used in information theory, statistics and machine learning. It quantifies the amount of information lost when one probability distribution Q is used to approximate another probability distribution P. KL divergence is commonly used as a loss function in machine learning to optimize model distribution (such as generative model, variational autoencoder, etc.) to approximate the real data distribution.

[0106] MSE is a commonly used index to measure the difference between predicted values and true values, widely used in regression tasks. It calculates the average of the square of the difference between predicted values and true values, emphasizing large errors (square effect) and being sensitive to outliers.

[0107] In this embodiment, α is 0.5. Progressive training is used for knowledge distillation training: in the initial stage of 0-50 rounds, the weight coefficient λ of the hard label loss is 0, in the 51st-99th round, the weight coefficient λ of the hard label loss is 0.02, and in the final stage after 100 rounds, the weight coefficient λ of the hard label loss is 0.2.

[0108] Example 2: Training of three-dimensional microscopic image recognition model for structure grading prediction of three-dimensional prostate biopsy samples

[0109] Accurate assessment of the histological features of prostate cancer is an important challenge in clinical diagnosis and treatment. Unlike breast cancer, which focuses on micro lesions, the diagnosis of prostate cancer relies more on accurate identification of glandular structure heterogeneity and spatial arrangement patterns. Traditional two-dimensional pathological sections cannot fully present the three-dimensional structural features of prostate tissue, which may lead to the loss of key diagnostic information. Three-dimensional pathological data obtained by high-resolution light sheet microscopy breaks through this limitation, which fully preserves the spatial topological relationship of glandular structure and significantly improves the utilization rate of tissue samples, and can accurately identify subtle structural abnormalities that are difficult to find by traditional methods. It has unique advantages in key diagnostic aspects of prostate cancer.

[0110] The three-dimensional microscopic image recognition model training method based on cross-dimension knowledge distillation provided by the embodiment includes the following steps:

[0111] Selection of teacher model: the teacher model adopts a large model pre-trained on a large amount of two-dimensional pathological data, such as Prov-GigaPath model, or the same Virchow model as in Embodiment 1. The Virchow model used in this embodiment has strong feature extraction capability.

[0112] Structure of student model: the student model is based on a three-dimensional visual Transformer architecture, which has the same image preprocessing module as the student model in Embodiment 1.

[0113] The training data is obtained according to the following method:

[0114] In this study, high-resolution light sheet microscopy system was used to obtain three-dimensional pathological data of prostate tissue. This imaging technology realizes three-dimensional whole imaging of glandular structure, and fully preserves the key histological features.

[0115] (1) Data hard label preparation: three-dimensional microscopic image sample data for supervised training, about 10,000 image blocks are used as three-dimensional image samples for strong supervision training, and artificial labeling is performed. Based on the gold standard of Gleason grading of prostate biopsy samples, we divide the sample structure into low risk, grade 3, grade 4, and grade 5, and obtain the hard label of the image.

[0116] (2) Data soft label preparation: same as Embodiment 1.

[0117] The student model is trained with the goal of minimizing the loss function. The loss function, training process used in this embodiment are similar to those in Embodiment 1:

[0118] In this embodiment, the value of a is 0.5. Progressive training is used for knowledge distillation training: in the initial stage of 0-50 rounds, the weight coefficient λ of the hard label loss is 0, in the 51st-99th round, the weight coefficient λ of the hard label loss is 0.02, and in the final stage after 100 rounds, the weight coefficient λ of the hard label loss is 0.2.

[0119] As shown in Figure 3 , 4 In embodiments 1 and 2, the accuracy of the knowledge distillation model on the test set is better than that of the two-dimensional teacher model alone and the three-dimensional student model directly obtained by strong supervision training. In addition, the adapter improves the performance of the knowledge distillation model. The data of embodiments 1 and 2 show that using high-resolution 3D pathological data instead of traditional 2D slices completely retains the spatial structure information of tumor tissues and more truly reflects the three-dimensional distribution characteristics of lesions. This innovation breaks through the information limitation of traditional 2D pathological analysis, provides more accurate spatial morphological basis for cancer prediction, and significantly improves the reliability of diagnosis.

[0120] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation, characterized in that, Includes the following steps: The three-dimensional microscopic image recognition model to be trained is used as the student model, and the pre-trained two-dimensional microscopic image recognition model is used as the teacher model. Hard labels are obtained by manually annotating the strongly supervised three-dimensional microscopic image sample data in the supervised training data. Soft labels are obtained by recognizing the supervised three-dimensional microscopic image sample data using the teacher model. Hybrid supervised knowledge distillation training is performed using the supervised three-dimensional microscopic image sample data to obtain a converged student model, which is then used as the three-dimensional microscopic image recognition model. The student model and the teacher model are based on the same architecture. The training data was obtained using the following method: (1) Data hard label preparation: The three-dimensional microscopic image sample data for strong supervision training is manually labeled to obtain the category of the three-dimensional microscopic image sample data, which is used as the hard label of the three-dimensional microscopic image sample data for strong supervision training; the three-dimensional microscopic image sample for strong supervision training is the manually labeled sample. (2) Data soft label preparation: The supervised training three-dimensional microscopic image sample data, including the three-dimensional microscopic image sample data for strongly supervised training, is compressed into multiple two-dimensional microscopic image samples through two-dimensional microscopic imaging simulation; the feature vector of the multiple two-dimensional microscopic image samples is obtained by feature extraction using the teacher model; the feature vector of the multiple two-dimensional microscopic image samples is averaged and pooled to form the feature vector of the three-dimensional microscopic image, which is used as the data soft label of the three-dimensional microscopic image sample data. The supervised training 3D microscopic image sample data consists of strongly supervised training 3D microscopic image sample data and weakly supervised training 3D microscopic image sample data. The strongly supervised training 3D microscopic image sample data has hard labels and soft labels, while the weakly supervised training 3D microscopic image sample data only has soft labels. Progressive training is used for knowledge distillation training. During the knowledge distillation training process, the weights of the prediction difference of the soft-label data samples and the prediction difference of the student model for the hard-label data samples are adjusted according to the training level: the higher the training level of the student model, the smaller the weight of the prediction difference of the soft-label data samples in the loss function.

2. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 1, characterized in that, The amount of 3D microscopic image sample data used for strongly supervised training shall not be less than 1 / 10 of the amount of 3D microscopic image sample data used for weakly supervised training.

3. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 2, characterized in that, The ratio of the amount of 3D microscopic image sample data used for strongly supervised training to that used for weakly supervised training is between 1:2 and 1:

10. When the supervised training three-dimensional microscopic image sample data is a pathological sample, the amount of three-dimensional microscopic image sample data for strong supervision training is greater than or equal to 20,000, and the amount of three-dimensional microscopic image sample data for weak supervision training is greater than or equal to 40,000.

4. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 1, characterized in that, The student model is trained with the goal of minimizing the loss function; the loss function is used to characterize the comprehensive difference between the three-dimensional microscopic image recognition model as the student model and the two-dimensional microscopic image recognition model as the teacher model, as well as the difference between the student model and the manually labeled results.

5. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 4, characterized in that, The loss function is expressed as a weighted sum of hard-label loss and soft-label loss; loss function Notation: ； in, This is the hard-label loss, i.e., the binary cross-entropy loss; represents the weighting coefficients for the hard label loss, with values ​​between [0, 1]. This is the soft label loss.

6. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 5, characterized in that, The soft-label loss is calculated as follows: ； in, This is the weighting coefficient, with a value range of (0, 1). For divergence soft label loss, The MSE soft label loss is calculated using the following methods: ； During training, the feature vector output by the teacher model for supervised training using 3D microscopic image sample data is calculated. and the feature vectors output by the student model The KL divergence between them; where, Represents all dimensions of each image feature vector. Let be a certain dimension of the feature vector. , The dimensions of the feature vectors are respectively The corresponding value; ； in, For supervised training using 3D microscopic image sample data, the feature vector output by the teacher model is given. For supervised training using 3D microscopic image sample data, the feature vector output by the student model is given. For the Euclidean norm, This represents the number of samples.

7. The training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in claim 1, characterized in that, The student model specifically includes an image preprocessing module, which maps the input image to the embedding space to obtain a dense vector of high-dimensional semantic features; the image preprocessing module includes a one-dimensional position encoder and / or an adaptor; A learnable one-dimensional positional encoding is introduced, which has the same dimension as the embedding vector, and its output positional encoding is directly superimposed on the extracted feature vector. The adaptive takes a dense vector of an encoded input image as input and outputs a dense vector of the same dimension, comprising a lower projection layer, a deep 3D convolutional layer, and an upper projection layer connected in series, and has trainable parameters.

8. A training system for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation, characterized in that, For electronic devices or non-transitory computer-readable storage media; The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in any one of claims 1 to 7. The non-transitory computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the training method for a three-dimensional microscopic image recognition model based on cross-dimensional knowledge distillation as described in any one of claims 1 to 7.

Images