Knowledge distillation method and device based on mean value teacher model and probability label

Abstract

The invention provides a knowledge distillation method and device based on a mean teacher model and probability tags, and the method comprises the steps: obtaining probability tags corresponding to original training images based on a plurality of original training images in a preset training data set through a probability prediction model, carrying out the binarization processing of the probability tags, and obtaining the probability tags corresponding to the original training images; obtaining a target probability label; for each training period, inputting a target training image obtained by processing the original training image into a mean value teacher model to obtain a prediction result; calculating a loss value of each training period based on the prediction result, the target probability label and a dynamic weight corresponding to the probability label; and performing parameter updating on a student model in the mean value teacher model by using the loss value to obtain a target student model after the student model is trained by the probability prediction model through knowledge distillation. Through the method, the accuracy of knowledge distillation is improved, and the overall classification performance and generalization ability of the student model in the teacher model are averaged.

Description

Technical Field

[0001] The present application relates to the technical field of knowledge distillation, and in particular to a knowledge distillation method and device based on a mean teacher model and probability labels. Background Art

[0002] In image classification tasks, knowledge distillation provides an effective solution to the contradiction between model complexity and computing resources in image classification tasks, while also achieving model compression while maintaining or even improving model performance.

[0003] At present, traditional knowledge distillation methods usually rely on fixed labels and static loss functions, which cannot fully deal with the uncertainty problems existing in samples, reduce the accuracy of knowledge distillation, and thus reduce the performance of the student model. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a knowledge distillation method and device based on a mean teacher model and probability labels, wherein the original training images are predicted by a probability prediction model to obtain probability labels and are binarized, and each original training image is processed to be input into the mean teacher model to obtain a prediction result, and each original training image and its probability label are adaptively adjusted using a dynamic weight adjustment mechanism to supervise the proportion of loss and consistency loss during model training, so that the student model in the mean teacher model can better utilize the loss value to strengthen the learning of sample features of probability labels, while avoiding the problem of over-training of samples that are easy to classify, thereby improving the accuracy of knowledge distillation, and thereby improving the overall classification performance and generalization ability of the student model in the mean teacher model.

[0005] The present application embodiment provides a knowledge distillation method based on a mean teacher model and probability labels, the knowledge distillation method comprising:

[0006] Based on multiple original training images in a preset training data set, a preset probability prediction model is used to obtain probability labels corresponding to the original training images respectively, and the probability labels are binarized to obtain target probability labels corresponding to the original training images respectively;

[0007] The multiple original training images are processed respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, random noise is added to the target training images and then input into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model;

[0008] Based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label, the loss value of each training cycle is calculated using a preset loss function expression;

[0009] The loss value is used to update the parameters of the student model, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

[0010] Furthermore, the binarization processing of the probability labels to obtain the target probability labels corresponding to the original training images includes:

[0011] Compare the probability value corresponding to each of the probability labels with the preset probability threshold to obtain a comparison result;

[0012] Determine the probability label whose comparison result is that the probability value is less than the preset probability threshold as the first target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the first target probability label as the first preset value;

[0013] The probability label whose comparison result is that the probability value is greater than or equal to the preset probability threshold is determined as the second target probability label among the target probability labels corresponding to the original training images, and the label value corresponding to the second target probability label is set to a second preset value.

[0014] Furthermore, the processing of the multiple original training images to obtain multiple target training images includes:

[0015] Resampling the plurality of original training images respectively by using a preset interpolation method to obtain a plurality of first training images, so as to adjust the pixel spacing values ​​of the original training images to a preset distance value;

[0016] The first training images are respectively cropped and normalized to obtain a plurality of second training images, and each of the second training images is intercepted using a preset first size range for a centroid point of a target object in each of the second training images to obtain a plurality of third training images;

[0017] Data enhancement processing is performed on the third training image to obtain a plurality of target training images.

[0018] Furthermore, the performing data enhancement processing on the third training image to obtain a plurality of target training images includes:

[0019] Filling the third training images with pixels according to a preset second size range, and randomly cropping the third training images after pixel filling using the first size range to obtain a plurality of fourth training images;

[0020] Each of the fourth training images is flipped horizontally, vertically and vertically in sequence, and any area is selected from the flipped fourth training image and set to be empty according to a preset third size range, so as to obtain a plurality of fifth training images;

[0021] The fifth training image is normalized using pre-calibrated mean standard deviation data to obtain a plurality of target training images.

[0022] Furthermore, the loss value of each training cycle is calculated based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label using a preset loss function expression, including:

[0023] Based on the first prediction result and the second prediction result, a consistency loss value of each training cycle is calculated using a preset mean square error loss function;

[0024] Based on the second prediction result and the target probability label, a cross entropy loss value of each training cycle is calculated using a preset cross entropy loss function;

[0025] Based on the probability value corresponding to the probability label, determine the dynamic weight value corresponding to the probability label using a preset dynamic weight expression;

[0026] Based on the consistency loss value, the cross entropy loss value and the dynamic weight value, the loss value of each training cycle is calculated using a preset loss function expression.

[0027] Further, the using the loss value to update the parameters of the student model, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through the parameter update of the multiple training cycles, includes:

[0028] Using the loss value to update the parameters of the student model to obtain an updated student model for each training cycle;

[0029] Using a preset validation data set to perform a performance evaluation on the updated student model, and optimizing the updated student model based on the evaluation result to obtain an optimized student model for each training cycle;

[0030] By updating the parameters of the optimized student model for multiple training cycles, a target student model is obtained after the probability prediction model trains the student model through knowledge distillation.

[0031] Furthermore, the method further comprises:

[0032] Based on the parameters corresponding to the target student model, the parameters corresponding to the teacher model are updated by exponential moving average calculation to obtain the target teacher model.

[0033] The embodiment of the present application also provides a knowledge distillation device based on a mean teacher model and probability labels, the knowledge distillation device comprising:

[0034] A data processing module is used to obtain probability labels corresponding to the original training images based on a plurality of original training images in a preset training data set by using a preset probability prediction model, and to perform binarization processing on the probability labels to obtain target probability labels corresponding to the original training images;

[0035] A model output module, used to process the multiple original training images respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, add random noise to the target training images and input them into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model;

[0036] A loss function module, configured to calculate the loss value of each training cycle using a preset loss function expression based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label;

[0037] The first updating module is used to update the parameters of the student model using the loss value, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

[0038] Furthermore, when the data processing module is used to perform binarization processing on the probability labels to obtain the target probability labels corresponding to the original training images, the data processing module is used to:

[0039] Compare the probability value corresponding to each of the probability labels with the preset probability threshold to obtain a comparison result;

[0040] Determine the probability label whose comparison result is that the probability value is less than the preset probability threshold as the first target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the first target probability label as the first preset value;

[0041] The probability label whose comparison result is that the probability value is greater than or equal to the preset probability threshold is determined as the second target probability label among the target probability labels corresponding to the original training images, and the label value corresponding to the second target probability label is set to a second preset value.

[0042] Furthermore, when the model output module is used to process a plurality of original training images in a preset training data set to obtain a plurality of target training images, the model output module is used to:

[0043] Resampling the plurality of original training images respectively by using a preset interpolation method to obtain a plurality of first training images, so as to adjust the pixel spacing values ​​of the original training images to a preset distance value;

[0044] The first training images are respectively cropped and normalized to obtain a plurality of second training images, and each of the second training images is intercepted using a preset first size range for a centroid point of a target object in each of the second training images to obtain a plurality of third training images;

[0045] Data enhancement processing is performed on the third training image to obtain a plurality of target training images.

[0046] Furthermore, when the model output module is used to perform data enhancement processing on the third training image to obtain multiple target training images, the model output module is used to:

[0047] Filling the third training images with pixels according to a preset second size range, and randomly cropping the third training images after pixel filling using the first size range to obtain a plurality of fourth training images;

[0048] Each of the fourth training images is flipped horizontally, vertically and vertically in sequence, and any area is selected from the flipped fourth training image and set to be empty according to a preset third size range, so as to obtain a plurality of fifth training images;

[0049] The fifth training image is normalized using pre-calibrated mean standard deviation data to obtain a plurality of target training images.

[0050] Further, when the loss function module is used to calculate the loss value of each training cycle based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label using a preset loss function expression, the loss function module is used to:

[0051] Based on the first prediction result and the second prediction result, a consistency loss value of each training cycle is calculated using a preset mean square error loss function;

[0052] Based on the second prediction result and the target probability label, a cross entropy loss value of each training cycle is calculated using a preset cross entropy loss function;

[0053] Based on the probability value corresponding to the probability label, determine the dynamic weight value corresponding to the probability label using a preset dynamic weight expression;

[0054] Based on the consistency loss value, the cross entropy loss value and the dynamic weight value, the loss value of each training cycle is calculated using a preset loss function expression.

[0055] Further, when the first updating module is used to update the parameters of the student model using the loss value, so as to obtain the target student model after the probability prediction model trains the student model through knowledge distillation through the parameter update of the multiple training cycles, the first updating module is used to:

[0056] Using the loss value to update the parameters of the student model to obtain an updated student model for each training cycle;

[0057] Using a preset validation data set to perform a performance evaluation on the updated student model, and optimizing the updated student model based on the evaluation result to obtain an optimized student model for each training cycle;

[0058] By updating the parameters of the optimized student model for multiple training cycles, a target student model is obtained after the probability prediction model trains the student model through knowledge distillation.

[0059] Furthermore, the knowledge distillation device further includes a second updating module, and the second updating module is used to:

[0060] Based on the parameters corresponding to the target student model, the parameters corresponding to the teacher model are updated by exponential moving average calculation to obtain the target teacher model.

[0061] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the knowledge distillation method based on the mean teacher model and probability labels are performed as described above.

[0062] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the knowledge distillation method based on the mean teacher model and probability labels are executed as described above.

[0063] The embodiments of the present application provide a knowledge distillation method and device based on a mean teacher model and probability labels. Compared with the knowledge distillation method in the prior art that relies on fixed labels and static loss functions, the knowledge distillation method predicts the original training images through a probability prediction model to obtain probability labels and performs binarization processing, and processes each original training image to input it into the mean teacher model to obtain a prediction result. A dynamic weight adjustment mechanism is used to adaptively adjust each original training image and its probability label to supervise the proportion of loss and consistency loss during the model training process, so that the student model in the mean teacher model can better utilize the loss value to strengthen the learning of sample features of the probability label, while avoiding the problem of over-training of samples that are easy to classify, thereby improving the accuracy of knowledge distillation, and thereby improving the overall classification performance and generalization ability of the student model in the mean teacher model.

[0064] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 One of the flowcharts of a knowledge distillation method based on a mean teacher model and probability labels provided in an embodiment of the present application;

[0067] Figure 2 The second flowchart of a knowledge distillation method based on a mean teacher model and probability labels provided in an embodiment of the present application;

[0068] Figure 3 A knowledge distillation data flow chart based on a mean teacher model and probability labels provided in an embodiment of the present application;

[0069] Figure 4 One of the structural schematic diagrams of a knowledge distillation device based on a mean teacher model and probability labels provided in an embodiment of the present application;

[0070] Figure 5 The second structural schematic diagram of a knowledge distillation device based on a mean teacher model and probability labels provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0073] Research has found that at present, traditional knowledge distillation methods usually rely on fixed labels and static loss functions, which cannot fully deal with the uncertainty problems existing in samples, reduce the accuracy of knowledge distillation, and thus reduce the performance of the student model.

[0074] Based on this, an embodiment of the present application provides a knowledge distillation method based on a mean teacher model and probability labels, in which the original training images are predicted by a probability prediction model to obtain probability labels and are binarized, and each original training image is processed to be input into the mean teacher model to obtain a prediction result, and a dynamic weight adjustment mechanism is used to adaptively adjust each original training image and its probability label to supervise the proportion of loss and consistency loss during the model training process, so that the student model in the mean teacher model can better utilize the loss value to strengthen the learning of sample features of the probability labels, while avoiding the problem of over-training of samples that are easy to classify, thereby improving the accuracy of knowledge distillation, and thereby improving the overall classification performance and generalization ability of the student model in the mean teacher model.

[0075] See also Figure 1 , Figure 1 This is one of the flow charts of a knowledge distillation method based on a mean teacher model and probability labels provided in an embodiment of the present application. Figure 1 As shown in , the knowledge distillation method based on the mean teacher model and probability labels provided in the embodiment of the present application includes:

[0076] S101. Based on a plurality of original training images in a preset training data set, a preset probability prediction model is used to obtain probability labels corresponding to the original training images respectively, and the probability labels are binarized to obtain target probability labels corresponding to the original training images respectively.

[0077] In an embodiment of the present application, the preset training data set is used to provide training data support for the model training process of the preset probability prediction model performing knowledge distillation on the preset mean teacher model, and the preset training data set includes at least a plurality of original training images. In this step, the plurality of original training images are input into the preset probability prediction model to obtain the probability label corresponding to each original training image output by the probability prediction model; wherein each probability label corresponds to a probability value.

[0078] Among them, the probability prediction model is a fixed and pre-trained model, which is usually a model with relatively large and complex data and structure. The output is the probability distribution corresponding to the original training image, that is, the soft label in the knowledge distillation process. The probability prediction model is used to guide the student model in the mean teacher model to learn through soft labels so as to transfer knowledge of the soft labels to the student model.

[0079] Exemplarily, the preset training data set may include multiple lung nodule images, and each lung nodule image can obtain the malignancy risk probability of the lung nodule image using the nodule risk prediction algorithm set in the preset probability prediction model, that is, a probability label corresponding to each lung nodule image can be obtained.

[0080] Furthermore, the probability label corresponding to each original training image is binarized using a preset probability threshold to divide the probability label into a first probability label and its first preset value, and a second probability label and its second preset value, that is, the benign or malignant classification of the lung nodules in the original training image is performed to process the binary classification task of the image.

[0081] In one embodiment of the present application, during specific implementation, the step of binarizing the probability labels in step S101 to obtain the target probability labels corresponding to the original training images may include:

[0082] S1011. Compare the probability value corresponding to each of the probability labels with a preset probability threshold to obtain a comparison result.

[0083] In an embodiment of the present application, the preset probability threshold may be set to 0.5, or may be specifically calibrated according to the field and category of the specific target object included in the preset training data set and the specific needs of the user.

[0084] In this step, the probability values ​​corresponding to the probability labels corresponding to each original training image output by the preset probability prediction model are numerically compared with the preset probability thresholds to obtain a comparison result showing the numerical relationship between the probability values ​​and the preset probability thresholds.

[0085] S1012. Determine the probability label whose comparison result is that the probability value is less than the preset probability threshold as the first target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the first target probability label to a first preset value.

[0086] In an embodiment of the present application, the target probability labels corresponding to the multiple original training images respectively include a first target probability label and a second target probability label; wherein the first target probability label refers to a probability label whose probability value is less than a preset probability threshold; the second target probability label refers to a probability label whose probability value is greater than or equal to the preset probability threshold.

[0087] In this step, a probability label whose probability value is less than a preset probability threshold is determined as a first probability label, and a label value corresponding to the first probability label is set to a first preset value; wherein the first probability label can be represented as a binary label.

[0088] In the embodiment of the present application, the first preset value is set to 0.

[0089] S1013. Determine the probability label whose comparison result is that the probability value is greater than or equal to the preset probability threshold as the second target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the second target probability label to a second preset value.

[0090] In this step, a probability label whose probability value is greater than or equal to a preset probability threshold is determined as a second target probability label, and the label value corresponding to the second probability label is set to a second preset value; wherein the second probability label can be represented as a binary label.

[0091] In the embodiment of the present application, the second preset value is set to 1.

[0092] S102. Process the multiple original training images respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, add random noise to the target training images and input them into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model.

[0093] In this step, before inputting into the mean teacher model, first, a plurality of original training images in a preset training data set are sequentially resampled and normalized; then, based on the centroid point of the target object in the obtained image, each obtained image is cropped using a preset first size range; finally, the cropped images are subjected to data enhancement processing to obtain a plurality of target training images.

[0094] In one embodiment of the present application, in a specific implementation, the step of processing the multiple original training images to obtain multiple target training images in step S102 may include:

[0095] S1021. Resample the plurality of original training images respectively using a preset interpolation method to obtain a plurality of first training images, so as to adjust the pixel spacing value of the original training image to a preset distance value.

[0096] In one possible implementation of the present application, the preset distance value is generally set to 1 mm, that is, the pixel spacing value of the original training image represented as a three-dimensional image is adjusted to "1 mm×1 mm×1 mm".

[0097] In the embodiment of the present application, the preset interpolation method may include but is not limited to any one of trilinear interpolation and nearest neighbor interpolation.

[0098] Among them, the trilinear interpolation method is a method for interpolation in three-dimensional space, which is often used for volume data resampling in medical image processing, such as CT (computed tomography) and MRI (magnetic resonance imaging) images. When the original image needs to be converted to a different resolution, direction or position, trilinear interpolation can provide a smooth transition effect and reduce artifacts caused by discretization.

[0099] The nearest neighbor interpolation method is generally used for various image processing including CT images. For each pixel or voxel in the target image, the nearest corresponding point in the source image is found, and the gray value of the point is directly assigned to the target point to achieve resampling of the image.

[0100] S1022. Crop and normalize the first training images to obtain a plurality of second training images, and intercept each of the second training images using a preset first size range for a centroid point of a target object in each of the second training images to obtain a plurality of third training images.

[0101] In this step, during the specific implementation, first, the first training image is HU cropped according to a preset range interval to limit the HU value of the first training image within the preset range interval; then, the cropped first training image is normalized to obtain a plurality of second training images; finally, for the centroid point of the target object in each second training image, each second training image is intercepted using a preset first size range to obtain a plurality of third training images.

[0102] Here, the HU clipping may refer to a windowed clipping or an intensity clipping. The Hounsfield unit (HU) in a CT image is a way to measure the density of a substance, based on the attenuation coefficient of water. HU clipping can help highlight tissues within a specific range while suppressing unnecessary information.

[0103] For example, the HU value of a first training image represented as a CT image is limited to between -1200 and 600, that is, the first training image is HU cropped at [-1200, 600] to effectively focus on the lung and bone structures of the CT image, because the range of [-1200, 600] generally covers the density variation from lung tissue (negative value) to bone tissue (positive value).

[0104] The preset range interval may be specifically calibrated according to the field and category of the specific target object included in the first training image and the specific needs of the user.

[0105] In an embodiment of the present application, the target object refers to an object corresponding to the image content included in the first training image. For example, when the first training image is a lung nodule image, the target object is a lung nodule; when the first training image is a kidney image, the target object may be a kidney stone.

[0106] Here, the preset first size range may be set to a "32×32×32" pixel value, and may also be specifically calibrated according to the field and category of the specific target object included in the first training image and the specific needs of the user.

[0107] S1023: Perform data enhancement processing on the third training image to obtain multiple target training images.

[0108] In the embodiment of the present application, performing data enhancement processing on the third training image can prevent overfitting of the training image, improve the robustness of the model, and enhance the performance of the model.

[0109] In one implementation of the present application, during specific implementation, step S1023 may include:

[0110] S10231. Fill the third training images with pixels according to a preset second size range, and randomly crop the third training images after the pixels are filled using the first size range to obtain a plurality of fourth training images.

[0111] In this step, during the specific implementation, first, each third training image is pixel-filled to a preset second size range by creating a 3D group and simulating tissue air; then, a position is randomly selected to place the third training image in the pixel-filled space; finally, the pixel-filled space is randomly cropped according to the first size range to obtain multiple fourth training images.

[0112] The second size range is larger than the first size range; illustratively, the first size range may be set to a pixel value of "32×32×32"; and the second size range may be set to a pixel value of "36×36×36".

[0113] S10232. Each of the fourth training images is flipped horizontally, vertically, and vertically in sequence, and any area is selected from the flipped fourth training image and set to be empty according to a preset third size range, so as to obtain a plurality of fifth training images.

[0114] In this step, during the specific implementation, each fourth training image is flipped in the width direction (horizontally), height direction (vertically), and depth direction (vertically); then, a position is randomly selected, and all pixel values ​​within the area of ​​the third size range centered at the position are set to 0, to obtain multiple fifth training images.

[0115] The third size range may be set to a "4×4×4" pixel value, and may also be specifically calibrated according to the field and category of the specific target object included in the first training image and the specific needs of the user.

[0116] S10233. Normalize the fifth training image using pre-calibrated mean standard deviation data to obtain multiple target training images.

[0117] In the embodiment of the present application, the pre-calibrated mean standard deviation data refers to the mean standard deviation data obtained by performing mean calculation and standard deviation calculation on a preset training set in advance.

[0118] In this step, normalizing and standardizing the fifth training image helps accelerate the convergence of the model and improves model performance. When the mean and standard deviation are used to normalize the data, it actually refers to the standardization process, that is, each feature is converted to zero mean and unit variance by subtracting the mean and dividing by the standard deviation.

[0119] In an embodiment of the present application, the preset mean teacher model is a technology used for semi-supervised learning. Through semi-supervised learning, a sliding average version of a teacher model can be maintained. The teacher model and student model in the mean teacher model share the initial weights and set update rules.

[0120] Exemplarily, the mean teacher model may include a deep learning framework for lung nodule detection and diagnosis (e.g., a model based on the DeepLung framework), which combines a multi-scale 3D convolutional neural network (CNN) and other techniques to improve the accuracy of lung nodule detection.

[0121] Here, in the process of knowledge distillation in the embodiment of the present application, the preset probability prediction model is used to generate soft labels in the knowledge distillation, and the preset mean teacher model is used to guide the training of the probability prediction model through knowledge distillation. The purpose of performing knowledge distillation in the scenario of the embodiment of the present application is to solve the problem of insufficient training data for the mean teacher model.

[0122] In one possible implementation of the present application, the preset mean teacher model includes a student and a teacher model. In an embodiment of the present application, in each training cycle, the model parameters of the student model in the mean teacher model are updated and optimized so as to use the model parameters of the student model in the mean teacher model during the inference or application stage using the mean teacher model.

[0123] The student model and the teacher model have the same network structure (e.g., resnet structure) and use the same random initialization parameters. The teacher model can be dynamically updated by the student model through exponential moving average for label guidance in the semi-supervised learning process and to promote consistency learning of the student model in semi-supervised learning.

[0124] For step S102, for each training cycle in multiple training cycles, random noise is added to the target training image and then input into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model.

[0125] In this step, knowledge distillation of multiple training cycles is performed on the mean teacher model. For each of the multiple training cycles, standard normal distribution random noise with the same data shape is added to the target training image of each training cycle, and its amplitude is scaled to 0.1; wherein, the noise of the target training image input to the student model is inconsistent with that of the target training image input to the teacher model. In each training cycle, the noise corresponding to each input to the student model and the teacher model is generated separately.

[0126] Furthermore, the target training image to which the first random noise is added is input into the teacher model included in the mean teacher model to obtain a first prediction result output by the teacher model; the target training image to which the second random noise is added is input into the student model included in the mean teacher model to obtain a second prediction result output by the student model.

[0127] S103. Based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label, a preset loss function expression is used to calculate the loss value of each training cycle.

[0128] In an embodiment of the present application, a dynamic weight adjustment mechanism corresponding to the probability label is used to adaptively adjust the proportion of supervision loss and consistency loss during the training process for the probability label of each target training image. Through this adjustment mechanism, the model can better utilize the consistency loss to enhance the learning of sample features, while avoiding over-training of samples that are easy to classify, ultimately improving the overall classification performance and generalization ability of the mean teacher model.

[0129] In one implementation of the present application, during specific implementation, step S103 may include:

[0130] S1031. Based on the first prediction result and the second prediction result, a preset mean square error loss function is used to calculate the consistency loss value of each training cycle.

[0131] In this step, the first prediction result output by the teacher model and the second prediction result output by the student model are respectively input into the expression of the preset mean square error loss function to calculate the consistency loss value of each training cycle.

[0132] Among them, the mean squared error (MSE) loss can be used to measure the average of the squared differences between the predicted values ​​and the true values.

[0133] In an embodiment of the present application, the expression of the preset mean square error loss function is as follows.

[0134]

[0135] Among them, L sup represents the consistency loss value of each training cycle; n represents the number of target training images; T p Indicates the first prediction result; S P Represents the second prediction result.

[0136] S1032. Based on the second prediction result and the target probability label, a preset cross entropy loss function is used to calculate the cross entropy loss value of each training cycle.

[0137] Here, the cross entropy loss function is used to measure the difference between the probability distribution predicted by the student model and the probability distribution of the target probability label output by the preset probability prediction model.

[0138] In this step, the second prediction result of the student model and the label value of the target probability label corresponding to each target training image are respectively input into the expression of the preset cross entropy loss function, and the cross entropy loss value of each training cycle is calculated.

[0139] In an embodiment of the present application, the expression of the preset cross entropy loss function is as follows.

[0140] L con =-∑ i TProb log(S P ).

[0141] Among them, L con represents the cross entropy loss value of each training cycle; TProb represents the label value of the target probability label corresponding to each target training image; S P Represents the second prediction result.

[0142] S1033. Based on the probability value corresponding to the probability label, determine the dynamic weight value corresponding to the probability label using a preset dynamic weight expression.

[0143] In this step, the probability value corresponding to the probability label is input into a preset dynamic weight expression to calculate the dynamic weight value corresponding to the probability label.

[0144] In the embodiment of the present application, the preset dynamic weight expression is as follows.

[0145] W=1-|prob-0.5|×2.

[0146] Among them, W represents the dynamic weight value corresponding to the probability label of each target training image; prob represents the probability value corresponding to the probability label of each target training image.

[0147] S1034. Based on the consistency loss value, the cross entropy loss value and the dynamic weight value, a preset loss function expression is used to calculate the loss value of each training cycle.

[0148] In this step, the consistency loss value, cross entropy loss value, and dynamic weight value are respectively input into the preset loss function expression to calculate the loss value of each training cycle.

[0149] In the embodiment of the present application, the preset loss function expression is as follows.

[0150] L=W×L con +(1-W)×L sup .

[0151] Where L represents the loss value of each training cycle; W represents the dynamic weight value corresponding to the probability label of each target training image; L con Represents the cross entropy loss value of each training cycle; L sup Represents the consistency loss value for each training cycle.

[0152] S104. Utilize the loss value to update the parameters of the student model, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

[0153] In an embodiment of the present application, the loss value is used to update the parameters of the student model in each training cycle, that is, the student model is back-propagated and parameters including weights are updated; wherein, back-propagation (BP) is a key step for training neural network models, which adjusts the parameters in the model by calculating the gradient of the loss function with respect to the network parameters (weights and biases) to minimize the prediction error. When the mean teacher model is trained by knowledge distillation using a preset probability prediction model, the knowledge of a larger and more complex probability prediction model is transferred to a student model in a mean teacher model. In this process, only the model parameters of the student model are back-propagated and the weights are updated.

[0154] In one implementation of the present application, during specific implementation, step S104 may include:

[0155] S1041. Use the loss value to update the parameters of the student model to obtain an updated student model for each training cycle.

[0156] In this step, when the loss value of each training cycle is calculated, based on the preset learning rate, the loss value is used to backpropagate the gradients of the parameters of each layer of the student model. At this stage, the parameters of the mean teacher model are regarded as constants and will not be affected in any way, thereby obtaining an updated student model for each training cycle.

[0157] In the embodiment of the present application, the preset learning rate can be set to 0.001.

[0158] S1042. Use a preset verification data set to perform performance evaluation on the updated student model, and optimize the updated student model based on the evaluation result to obtain an optimized student model for each training cycle.

[0159] In this step, in each training cycle, the performance of the updated student model is evaluated using a preset validation data set, and model parameters, such as learning rate, temperature parameter, and balance factor, are adjusted according to the evaluation results to optimize the updated student model and obtain an optimized student model for each training cycle.

[0160] S1043. By updating the parameters of the optimized student model for multiple training cycles, a target student model is obtained after the probability prediction model trains the student model through knowledge distillation.

[0161] In this step, after the parameters of the student model are updated for multiple training cycles, a target student model is obtained after the probability prediction model is trained on the student model through knowledge distillation.

[0162] For example, after 1000 training cycles of knowledge distillation, a preset validation data set is used to find the model with the best AUC performance in the preset validation data set as the target student model, or the optimized student model obtained after 1000 training cycles of knowledge distillation training is used as the target student model; wherein AUC performance refers to the area under the ROC curve, which is an important indicator for evaluating the performance of a binary classification model. It measures the ability of the model to distinguish between positive and negative classes without being affected by class imbalance. The value range of AUC is from 0 to 1.

[0163] As an implementation example of a target student model and a preset probability model, the probability prediction model and the target student model were tested using a preset test data set and a test data set provided by a hospital respectively. The AUC performance result of the probability prediction model was 0.75, and the AUC performance result of the target student model was 0.82. It can be seen that the AUC performance result of the target student model is better than that of the probability prediction model. Therefore, after the knowledge distillation method provided in the embodiment of the present application is performed on the student model in the mean teacher model, the overall classification performance and generalization ability of the student model in the mean teacher model are improved.

[0164] Optional, see Figure 2 , Figure 2 This is the second flowchart of a knowledge distillation method for a mean teacher model provided in an embodiment of the present application.

[0165] like Figure 2 As shown in, in addition to the knowledge distillation method based on the mean teacher model and probability labels described in steps S101 to S104, the embodiment of the present application also includes step S105. Specifically, step S105 is used to illustrate a method for updating parameters of the teacher model in the mean teacher model using the target student model to improve the overall performance of the mean teacher model.

[0166] S105. Based on the parameters corresponding to the target student model, the parameters corresponding to the teacher model are updated by exponential moving average calculation to obtain a target teacher model.

[0167] In the embodiment of the present application, the exponential moving average (EMA) is a weighted averaging method that assigns higher weights to newer observations while older observations gradually decay. In neural network training, EMA is used to track the long-term trend of model parameters rather than relying solely on the latest gradient update.

[0168] In this step, when using exponential moving average to update the parameters of the teacher model, that is, in self-distillation or collaborative distillation scenarios, the exponential moving average can smooth the learning process of the student model, so that the teacher model can better capture the learning dynamics of the student model, and can be used as a regularization method to improve the generalization ability of the model.

[0169] For example, see Figure 3 , Figure 3 A knowledge distillation data flow chart based on a mean teacher model and probability labels provided in an embodiment of the present application. Figure 3 As shown in , the first random noise (random noise 1) and the second random noise (random noise 2) are added to the target training image respectively, and then input into the teacher model and the student model respectively to obtain the first prediction result and the second prediction result. The consistency loss value can be calculated based on the first prediction result and the second prediction result, the cross entropy loss value can be calculated based on the second prediction result and the label value of the target probability label, and the dynamic weight value can be calculated based on the probability label corresponding to the target training image.

[0170] The knowledge distillation method based on the mean teacher model and probability labels provided in the embodiment of the present application predicts the original training images through the probability prediction model to obtain probability labels and performs binarization processing, and processes each original training image to input it into the mean teacher model to obtain a prediction result, and uses a dynamic weight adjustment mechanism to adaptively adjust each original training image and its probability label to supervise the proportion of loss and consistency loss during the model training process, so that the student model in the mean teacher model can better utilize the loss value to strengthen the learning of sample features of the probability label, while avoiding the problem of over-training of samples that are easy to classify, thereby improving the accuracy of knowledge distillation, and thereby improving the overall classification performance and generalization ability of the student model in the mean teacher model.

[0171] See also Figure 4 , Figure 5 , Figure 4 This is one of the structural schematic diagrams of a knowledge distillation device based on a mean teacher model and probability labels provided in an embodiment of the present application. Figure 5 The second structural diagram of a knowledge distillation device based on a mean teacher model and probability labels provided in an embodiment of the present application. Figure 4 As shown in , the knowledge distillation device 400 includes:

[0172] The data processing module 410 is used to obtain the probability labels corresponding to the original training images based on the plurality of original training images in the preset training data set by using the preset probability prediction model, and to perform binarization processing on the probability labels to obtain the target probability labels corresponding to the original training images respectively;

[0173] A model output module 420 is used to process the multiple original training images respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, add random noise to the target training images and input them into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model;

[0174] A loss function module 430, configured to calculate the loss value of each training cycle using a preset loss function expression based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label;

[0175] The first updating module 440 is used to update the parameters of the student model using the loss value, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

[0176] Furthermore, when the data processing module 410 is used to perform binarization processing on the probability labels to obtain the target probability labels corresponding to the original training images, the data processing module 410 is used to:

[0177] Compare the probability value corresponding to each of the probability labels with the preset probability threshold to obtain a comparison result;

[0178] Determine the probability label whose comparison result is that the probability value is less than the preset probability threshold as the first target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the first target probability label as the first preset value;

[0179] The probability label whose comparison result is that the probability value is greater than or equal to the preset probability threshold is determined as the second target probability label among the target probability labels corresponding to the original training images, and the label value corresponding to the second target probability label is set to a second preset value.

[0180] Furthermore, when the model output module 420 is used to process a plurality of original training images in a preset training data set to obtain a plurality of target training images, the model output module 420 is used to:

[0181] Resampling a plurality of original training images in a preset training data set respectively by using a preset interpolation method to obtain a plurality of first training images, so as to adjust the pixel spacing value of the original training image to a preset distance value;

[0182] The first training images are respectively cropped and normalized to obtain a plurality of second training images, and each of the second training images is intercepted using a preset first size range for a centroid point of a target object in each of the second training images to obtain a plurality of third training images;

[0183] Data enhancement processing is performed on the third training image to obtain a plurality of target training images.

[0184] Furthermore, when the model output module 420 is used to perform data enhancement processing on the third training image to obtain multiple target training images, the model output module 420 is used to:

[0185] Filling the third training images with pixels according to a preset second size range, and randomly cropping the third training images after pixel filling using the first size range to obtain a plurality of fourth training images;

[0186] Each of the fourth training images is flipped horizontally, vertically and vertically in sequence, and any area is selected from the flipped fourth training image and set to be empty according to a preset third size range, so as to obtain a plurality of fifth training images;

[0187] The fifth training image is normalized using pre-calibrated mean standard deviation data to obtain a plurality of target training images.

[0188] Further, when the loss function module 430 is used to calculate the loss value of each training cycle based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label using a preset loss function expression, the loss function module 430 is used to:

[0189] Based on the first prediction result and the second prediction result, a consistency loss value of each training cycle is calculated using a preset mean square error loss function;

[0190] Based on the second prediction result and the target probability label, a cross entropy loss value of each training cycle is calculated using a preset cross entropy loss function;

[0191] Based on the probability value corresponding to the probability label, determine the dynamic weight value corresponding to the probability label using a preset dynamic weight expression;

[0192] Based on the consistency loss value, the cross entropy loss value and the dynamic weight value, the loss value of each training cycle is calculated using a preset loss function expression.

[0193] Further, when the first updating module 440 is used to update the parameters of the student model using the loss value, so as to obtain the target student model after the probability prediction model trains the student model through knowledge distillation through the parameter update of the multiple training cycles, the first updating module 440 is used to:

[0194] Using the loss value to update the parameters of the student model to obtain an updated student model for each training cycle;

[0195] Using a preset validation data set to perform a performance evaluation on the updated student model, and optimizing the updated student model based on the evaluation result to obtain an optimized student model for each training cycle;

[0196] By updating the parameters of the optimized student model for multiple training cycles, a target student model is obtained after the probability prediction model trains the student model through knowledge distillation.

[0197] Further, such as Figure 5 As shown in , the knowledge distillation device 400 further includes a second updating module 450, and the second updating module 450 is used to:

[0198] Based on the parameters corresponding to the target student model, the parameters corresponding to the teacher model are updated by exponential moving average calculation to obtain the target teacher model.

[0199] The knowledge distillation device based on the mean teacher model and probability labels provided in the embodiment of the present application predicts the original training images through the probability prediction model to obtain probability labels and performs binarization processing, and processes each original training image to input it into the mean teacher model to obtain a prediction result, and uses a dynamic weight adjustment mechanism to adaptively adjust each original training image and its probability label to supervise the proportion of loss and consistency loss during the model training process, so that the student model in the mean teacher model can better utilize the loss value to strengthen the learning of sample features of the probability label, while avoiding the problem of over-training of samples that are easy to classify, thereby improving the accuracy of knowledge distillation, and thereby improving the overall classification performance and generalization ability of the student model in the mean teacher model.

[0200] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in , the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0201] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 1 as well as Figure 2 The steps of the knowledge distillation method based on the mean teacher model and probability labels in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0202] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the knowledge distillation method based on the mean teacher model and probability labels in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0205] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0206] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A knowledge distillation method based on mean teacher model and probability label, characterized in that: The knowledge distillation method comprises: Based on multiple original training images in a preset training data set, a preset probability prediction model is used to obtain probability labels corresponding to the original training images respectively, and the probability labels are binarized to obtain target probability labels corresponding to the original training images respectively; The multiple original training images are processed respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, random noise is added to the target training images and then input into a preset mean teacher model to obtain a first prediction result and a second prediction result respectively output by a teacher model and a student model included in the mean teacher model; Based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label, the loss value of each training cycle is calculated using a preset loss function expression; The loss value is used to update the parameters of the student model, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

2. The method according to claim 1, characterized in that The binarization of the probability labels to obtain target probability labels corresponding to the original training images respectively includes: Compare the probability value corresponding to each of the probability labels with the preset probability threshold to obtain a comparison result; Determine the probability label whose comparison result is that the probability value is less than the preset probability threshold as the first target probability label among the target probability labels corresponding to the original training images, and set the label value corresponding to the first target probability label as the first preset value; The probability label whose comparison result is that the probability value is greater than or equal to the preset probability threshold is determined as the second target probability label among the target probability labels corresponding to the original training images, and the label value corresponding to the second target probability label is set to a second preset value.

3. The method according to claim 1, characterized in that The processing of the plurality of original training images to obtain a plurality of target training images includes: Resampling the plurality of original training images respectively by using a preset interpolation method to obtain a plurality of first training images, so as to adjust the pixel spacing values ​​of the original training images to a preset distance value; The first training images are respectively cropped and normalized to obtain a plurality of second training images, and each of the second training images is intercepted using a preset first size range for a centroid point of a target object in each of the second training images to obtain a plurality of third training images; Data enhancement processing is performed on the third training image to obtain a plurality of target training images.

4. The method according to claim 3, characterized in that The performing data enhancement processing on the third training image to obtain a plurality of target training images comprises: Filling the third training images with pixels according to a preset second size range, and randomly cropping the third training images after pixel filling using the first size range to obtain a plurality of fourth training images; Each of the fourth training images is flipped horizontally, vertically and vertically in sequence, and any area is selected from the flipped fourth training image and set to be empty according to a preset third size range, so as to obtain a plurality of fifth training images; The fifth training image is normalized using pre-calibrated mean standard deviation data to obtain a plurality of target training images.

5. The method according to claim 1, characterized in that The loss value of each training cycle is calculated based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label using a preset loss function expression, including: Based on the first prediction result and the second prediction result, a consistency loss value of each training cycle is calculated using a preset mean square error loss function; Based on the second prediction result and the target probability label, a cross entropy loss value of each training cycle is calculated using a preset cross entropy loss function; Based on the probability value corresponding to the probability label, determine the dynamic weight value corresponding to the probability label using a preset dynamic weight expression; Based on the consistency loss value, the cross entropy loss value and the dynamic weight value, the loss value of each training cycle is calculated using a preset loss function expression.

6. The method according to claim 1, characterized in that The method of updating the parameters of the student model by using the loss value to obtain a target student model after the probability prediction model trains the student model by knowledge distillation through the parameter updates of the multiple training cycles includes: Using the loss value to update the parameters of the student model to obtain an updated student model for each training cycle; Using a preset validation data set to perform a performance evaluation on the updated student model, and optimizing the updated student model based on the evaluation result to obtain an optimized student model for each training cycle; By updating the parameters of the optimized student model for multiple training cycles, a target student model is obtained after the probability prediction model trains the student model through knowledge distillation.

7. The method according to claim 1, characterized in that The method further comprises: Based on the parameters corresponding to the target student model, the parameters corresponding to the teacher model are updated by exponential moving average calculation to obtain the target teacher model.

8. A knowledge distillation device based on a mean teacher model and probability labels, characterized in that: The knowledge distillation device comprises: A data processing module is used to obtain probability labels corresponding to the original training images based on a plurality of original training images in a preset training data set by using a preset probability prediction model, and to perform binarization processing on the probability labels to obtain target probability labels corresponding to the original training images; A model output module, used to process the multiple original training images respectively to obtain multiple target training images, and for each training cycle in the multiple training cycles, add random noise to the target training images and input them into a preset mean teacher model to obtain a first prediction result outputted by the teacher model and the student model respectively included in the mean teacher model and a second prediction result outputted by the student model; A loss function module, configured to calculate the loss value of each training cycle using a preset loss function expression based on the first prediction result, the second prediction result, the target probability label and the dynamic weight corresponding to the probability label; The first updating module is used to update the parameters of the student model using the loss value, so as to obtain a target student model after the probability prediction model trains the student model through knowledge distillation through parameter updates of the multiple training cycles.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor when running to execute the steps of the knowledge distillation method based on the mean teacher model and probability labels as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the knowledge distillation method based on the mean teacher model and probability labels as described in any one of claims 1 to 7 are performed.

Images