A model processing method, apparatus and device

Abstract

The embodiment of the specification discloses a model processing method, device and equipment, the method comprises: obtaining a first batch of image samples from a plurality of batches of image samples for training a student model, a plurality of different image samples are included in the first batch of image samples, and a mask image after a mask processing of the image sample; inputting the plurality of image samples into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample respectively, and inputting the plurality of mask images into the student model to obtain a second prediction result corresponding to each mask image respectively; determining the association relationship information between the plurality of image samples based on the first prediction result, and modulating and processing the second prediction result based on the association relationship information between the plurality of image samples to obtain prediction coding information of a reconstructed image corresponding to each mask image; training the student model based on the first prediction result and the prediction coding information.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method, apparatus and device for processing a model. Background Technology

[0002] As people become increasingly concerned about the privacy of their data, the security of visual data is becoming increasingly important. Typically, large amounts of unlabeled visual data can be used for self-supervised learning and representation learning to pre-train visual models. However, how to fully utilize model information to achieve efficient model training is a crucial issue in the field of vision. Therefore, there is a need for a technical solution that can more efficiently train models and improve the visual model's understanding of relationships between image samples, thereby enhancing the downstream generalization ability of the trained visual model. Summary of the Invention

[0003] The purpose of the embodiments in this specification is to provide a technical solution that can perform model training more efficiently and improve the visual model's understanding of the relationships between image samples, thereby further enhancing the downstream generalization ability of the trained visual model.

[0004] To achieve the above technical solution, the embodiments in this specification are implemented as follows:

[0005] This specification provides a model processing method, comprising: obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and masked images of the image samples after masking processing; inputting the multiple image samples into a teacher model corresponding to the student model to obtain a first prediction result for each image sample; and inputting the multiple masked images into the student model to obtain a second prediction result for each masked image; determining the correlation information between the multiple image samples based on the first prediction results, and modulating the second prediction result for each masked image based on the correlation information to obtain predictive coding information for the reconstructed image corresponding to each masked image; and training the student model based on the first prediction results and the predictive coding information for the reconstructed image corresponding to each masked image to obtain a trained student model.

[0006] This specification provides a model processing apparatus, comprising: a sample acquisition module, which acquires a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and masked images of the image samples after masking processing; a prediction module, which inputs the multiple image samples into a teacher model corresponding to the student model to obtain a first prediction result for each image sample, and inputs the multiple masked images into the student model to obtain a second prediction result for each masked image; a modulation module, which determines the correlation information between the multiple image samples based on the first prediction results, and modulates the second prediction result for each masked image based on the correlation information, to obtain predictive coding information for the reconstructed image corresponding to each masked image; and a model training module, which trains the student model based on the first prediction results and the predictive coding information for the reconstructed image corresponding to each masked image, to obtain a trained student model.

[0007] This specification provides a model processing device comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to: acquire a first batch of image samples from a plurality of batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and masked images of the image samples after masking; input the plurality of image samples into a teacher model corresponding to the student model to obtain a first prediction result for each image sample; and input the plurality of masked images into the student model to obtain a second prediction result for each masked image; based on the first prediction results corresponding to the plurality of image samples, determine the correlation information between the plurality of image samples, and modulate the second prediction result corresponding to each masked image based on the correlation information between the plurality of image samples to obtain predictive coding information for the reconstructed image corresponding to each masked image; and train the student model based on the first prediction results corresponding to each image sample and the predictive coding information for the reconstructed image corresponding to each masked image to obtain a trained student model.

[0008] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: First batch of image samples are obtained from multiple batches of image samples used to train a student model. The first batch of image samples includes multiple different image samples and masked images of the image samples after masking. The multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result for each image sample. The multiple masked images are then input into the student model to obtain a second prediction result for each masked image. Based on the first prediction results of the multiple image samples, correlation information between the multiple image samples is determined. Based on the correlation information between the multiple image samples, the second prediction result for each masked image is modulated to obtain predictive coding information for the reconstructed image corresponding to each masked image. Based on the first prediction results and the predictive coding information for the reconstructed image corresponding to each masked image, the student model is trained to obtain a trained student model.

[0009] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the following process: Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and masked images of the image samples after masking. Inputting the multiple image samples into a teacher model corresponding to the student model to obtain a first prediction result for each image sample, and inputting the multiple masked images into the student model to obtain a second prediction result for each masked image. Based on the first prediction results corresponding to the multiple image samples, determining the correlation information between the multiple image samples, and modulating the second prediction result corresponding to each masked image based on the correlation information between the multiple image samples to obtain predictive coding information for the reconstructed image corresponding to each masked image. Training the student model based on the first prediction results corresponding to each image sample and the predictive coding information for the reconstructed image corresponding to each masked image to obtain a trained student model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an embodiment of a model processing method described in this specification;

[0012] Figure 2 This is a schematic diagram illustrating the processing procedure of one model in this specification;

[0013] Figure 3 This is a schematic diagram illustrating the processing procedure of another model in this specification;

[0014] Figure 4 This is an embodiment of another model processing method described in this specification;

[0015] Figure 5 This is a schematic diagram of an information modulation and processing procedure described in this specification;

[0016] Figure 6 This is yet another embodiment of the model processing method described in this specification;

[0017] Figure 7 This is an embodiment of a processing device for a model described in this specification;

[0018] Figure 8 This is an embodiment of a processing device for one model in this specification. Detailed Implementation

[0019] This specification provides a method, apparatus, and device for processing a model.

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] This specification provides a self-supervised learning mechanism based on feature distillation. As people increasingly value their privacy data, the security of visual data is becoming increasingly important. Typically, large amounts of unlabeled visual data can be used for self-supervised learning and representation learning to pre-train visual models. Visual masking has been proven to be an efficient training method. The basic approach involves occluding the visual data input to the visual model according to a certain proportion, and then using the encoded representation of the masked image to reconstruct the prediction result corresponding to the complete visual data, thus achieving the pre-training purpose. However, the above methods all use a self-supervised pre-trained visual model as a teacher model, but only consider using the unmasked portion of the same image as cues for their own masking portion, thus insufficiently utilizing the teacher model's information and hindering efficient training. This specification provides a feature distillation algorithm that can perform model training more efficiently. This algorithm not only considers reconstruction based on image samples but also uses the relationships between image samples from the teacher model as guidance for the reconstruction task, thereby improving the student model's understanding of the relationships between image samples and further enhancing the downstream generalization ability of the trained visual model. Specific processing details can be found in the following embodiments.

[0022] like Figure 1 As shown in the embodiments of this specification, a method for processing a model is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial business or online shopping business, or a backend server for an application, etc. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps:

[0023] In step S102, the first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked.

[0024] The student model can be any model required in any business. Specifically, the student model can be a risk control model or a biometric model in financial businesses such as payment and transfer services. The risk control model can be used to identify risks in user-input images or collected images to determine whether the image will attack the system or the user's account. The biometric model can be used for facial recognition, fingerprint recognition, palm print recognition, etc. In addition, the student model can also be a risk identification model in instant messaging services. This risk identification model can identify the information contained in the user-input image to determine whether it contains specified risks (such as fraud risk or privacy leakage risk). The specific settings can be set according to the actual situation, and this specification does not limit this. Furthermore, depending on the model's purpose or function, the student model can be a model for any purpose or function. For example, the learning model can be a model for classification (such as binary classification, multi-class classification, etc.), a model for data prediction, a model for risk detection, a model for semantic analysis, etc. The structure of the learning model can be diverse; it can be built using convolutional neural networks, classification algorithms, ResNet, etc., depending on the specific circumstances. This specification does not limit this in the embodiments. Multiple batches of image samples, i.e., multiple batches of image samples, are typically divided into multiple batches before training the model. Then, the model is trained sequentially using the training samples from each batch. The first batch of image samples can be any batch of image samples from the multiple batches. Image sample masking can involve completely or partially occluding the image samples.

[0025] In implementation, such as Figure 2As shown, considering the advantages of the knowledge distillation mechanism, model training can be performed through knowledge distillation. This involves using a self-supervised pre-trained visual model as the teacher model to train a student model. Multiple batches of image samples for training the student model can be obtained in various ways. For example, a certain number of image samples can be obtained from a specified database and then divided into multiple batches. Alternatively, a certain number of image samples can be crawled from the internet using a web crawler and then divided into multiple batches. Or, images provided by different users can be obtained as image samples, and these samples can be divided into multiple batches. The specific methods can be tailored to the actual situation, and this embodiment does not limit the specific methods described in this specification.

[0026] Each batch of image samples mentioned above can include multiple different image samples and a mask image after each image sample has been masked. In other words, each batch of image samples can include multiple different image samples and multiple different mask images. After obtaining multiple batches of image samples in the above manner, the first batch of image samples can be obtained from these batches to train the student model. For details, please refer to the relevant content below.

[0027] In step S104, multiple image samples are input into the teacher model corresponding to the student model to obtain the first prediction result for each image sample, and multiple mask images are input into the student model to obtain the second prediction result for each mask image.

[0028] The teacher model can be any model. For example, it can be a model for classification (such as binary or multi-class classification), a model for data prediction, a model for risk detection, or a model for semantic analysis. The structure of the teacher model can be diverse; it can be built using convolutional neural networks, classification algorithms, or ResNet, etc., depending on the specific circumstances. This embodiment does not limit this. The teacher model and the aforementioned student model can be trained using knowledge distillation. Knowledge distillation is a mechanism that transfers knowledge from a deep model or ensemble model with many parameters (i.e., the teacher model) to a relatively small, shallow model (i.e., the student model). Based on this, the teacher model and the aforementioned student model have the same function or role. For example, if the student model is a biometric model in financial services, then the teacher model can also be applied to biometrics in financial services; if the student model is a risk identification model in instant messaging services, then the teacher model can also be applied to risk identification in instant messaging services. The specific configuration can be determined according to the actual situation. In this embodiment, the model parameters of the teacher model can be frozen, meaning the model parameters of the teacher model remain fixed.

[0029] In implementation, such as Figure 2 As shown, based on complete image samples, the teacher model can provide the target and clues for the prediction results of the student model. Based on this, multiple image samples from the first batch of image samples can be input into the teacher model. The teacher model processes each image sample to obtain the corresponding output result (i.e., the first prediction result corresponding to each image sample). The multiple image samples input into the teacher model can be all the image samples contained in the first batch of image samples, or they can be a partial image sample. Specifically, they can be 2 image samples from the multiple image samples in the first batch of image samples, or 3 or 5 image samples from the multiple image samples in the first batch of image samples, etc. The specific settings can be determined according to the actual situation.

[0030] Since the above processing of the teacher model is to provide the target and clues for the prediction results of the student model, the samples input into the student model should correspond to the image samples input into the teacher model. That is, if there are two image samples input into the teacher model, then the samples input into the student model should be the mask images corresponding to the two image samples. For example, if there are three image samples input into the teacher model, namely image sample 1, image sample 2 and image sample 3, then the samples input into the student model should be the mask images corresponding to image sample 1, image sample 2 and image sample 3. Based on this, multiple mask images composed of the mask images corresponding to each image sample can be input into the student model. The student model processes each mask image to obtain the corresponding output result (i.e., the second prediction result corresponding to each mask image).

[0031] It should be noted that, in order to simplify data processing and storage, the first prediction result corresponding to each image sample can be represented by an aggregated data. For example, a matrix or vector can be used to represent the first prediction results corresponding to multiple image samples (i.e., a matrix or vector represents multiple first prediction results). Correspondingly, the second prediction result corresponding to each mask image can be represented by an aggregated data. For example, a matrix or vector can be used to represent multiple second prediction results. In practical applications, different data can also be used to represent the first prediction result corresponding to each image sample or the second prediction result corresponding to each mask image, etc. The specific data can be set according to the actual situation, and this specification does not limit this.

[0032] In step S106, based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0033] The reconstructed image can be obtained by restoring the image that was obscured in the masked image.

[0034] In implementation, such as Figure 2As shown, the above-mentioned processing methods only consider using the unmasked portion of the same image as clues for the masked portion, without considering the correlation between other image samples from the same batch and the current image. This results in insufficient utilization of teacher model information, hindering efficient training. Therefore, the correlation between other image samples from the same batch and the current image can be considered to modulate the student model's direct prediction results, thereby making fuller use of teacher model information and improving training efficiency. Specifically, the similarity between any two image samples in the first prediction results of multiple image samples can be calculated. Based on the calculated similarity information, the correlation information between multiple image samples can be determined through specified algorithms (such as weighted average algorithms, clustering algorithms, fusion algorithms, etc.) or mapping relationships. The correlation information between multiple image samples can be fused into the second prediction result corresponding to each masked image to improve the accuracy of training. The second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image. The above-mentioned adjustment process of fusing the correlation information between multiple image samples into the second prediction result corresponding to each mask image can be varied. For example, an encoder can be pre-built to encode the correlation information between multiple image samples and the second prediction result corresponding to each mask image, respectively, to obtain the coding information corresponding to the correlation information between multiple image samples and the coding information of the second prediction result corresponding to each mask image. Then, the above two coding information can be calculated by a preset fusion algorithm (such as matrix or vector multiplication algorithm, addition algorithm, etc.), and the calculation result is used as the prediction coding information of the reconstructed image corresponding to each mask image. In addition, the prediction coding information of the reconstructed image corresponding to each mask image can also be obtained through other different methods. The specific method can be set according to the actual situation, and the embodiments in this specification do not limit this.

[0035] It should be noted that the predictive coding information of the reconstructed image corresponding to each mask image can also be represented by an aggregated data. For example, a matrix or vector can be used to represent the predictive coding information of the reconstructed image corresponding to multiple mask images (i.e., a matrix or vector represents multiple predictive coding information). The specific settings can be made according to the actual situation, and the embodiments in this specification do not limit this.

[0036] In step S108, the student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image, to obtain the trained student model.

[0037] In implementation, the encoder described above can be used to encode the first prediction result corresponding to each image sample to obtain the encoding information of the first prediction result corresponding to each image sample. Then, based on the encoding information of the first prediction result corresponding to each image sample and the prediction encoding information of the reconstructed image corresponding to each mask image, the corresponding loss information can be calculated through a preset loss function. The loss information can be used for gradient backpropagation to update the model parameters of the student model. Then, the above processing can be continued using image samples from other batches of multiple batches of image samples to update the model parameters of the student model until the number of iterations reaches a preset threshold or the above loss function converges. Finally, the trained student model can be obtained. Alternatively, a decoder corresponding to the encoder described above can be constructed. This decoder can decode the prediction encoding information of the reconstructed image corresponding to each mask image to obtain the prediction result of the reconstructed image corresponding to each mask image. Based on the first prediction result corresponding to each image sample and the prediction result of the reconstructed image corresponding to each mask image, the corresponding loss information can be calculated using a preset loss function. This loss information can be used for gradient backpropagation to update the model parameters of the student model. Then, the above processing can be continued using image samples from other batches of multiple batches of image samples to update the model parameters of the student model until the number of iterations reaches a preset threshold or the preset loss function converges. Finally, the trained student model can be obtained. Alternatively, the student model can be trained in other ways, depending on the actual situation. This specification does not limit this aspect in the embodiments.

[0038] This specification provides a model processing method. It involves obtaining a first batch of image samples from multiple batches of image samples used to train a student model. This first batch includes multiple different image samples and masked images of these samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model, obtaining a first prediction result for each image sample. The multiple masked images are then input into the student model, obtaining a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on this correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image for each masked image. The predictive coding information of the images is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, based on the correlation information between multiple image samples, the second prediction result corresponding to each mask image is modulated, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effectiveness of model training.

[0039] In practical applications, multiple mask images include a mask image selected based on a first preset ratio and the remaining mask images. In order to improve the performance and generalization ability of the student model, the mask image selected based on the first preset ratio is a mask image obtained after completely occluding the corresponding image sample, and the remaining mask images are mask images obtained after occluding the corresponding image sample with a preset occlusion ratio.

[0040] The first preset ratio can be set according to actual conditions, such as 20% or 50%. The remaining mask images are the mask images other than the mask image selected based on the first preset ratio from multiple mask images. Complete occlusion masking can be equivalent to completely occluding the image sample relative to the image processing module in the student model. Therefore, in practical applications, complete occlusion masking can be masking by completely occluding the image sample with a mask, or it can be masking by occluding a portion of the image sample except for one or more portions of a specified size, where the unoccluded images cannot provide effective relevance information in the image processing module of the student model. In this case, it is also considered complete occlusion masking, i.e., equivalent to completely occluding the image sample. The preset occlusion ratio can be the proportion of an image or a portion of an image that needs to be occluded. The preset occlusion ratio can be set according to actual conditions, such as 50% or 60%.

[0041] In practical applications, the student model may include modules built based on Vision Transformer. Based on the division of the above multiple mask images, multiple mask images can be obtained in the following way, as detailed in steps A2 to A8 below.

[0042] In step A2, each image sample is divided into multiple image blocks.

[0043] In implementation, each image sample can be divided equally. For example, each image sample can be divided into multiple image blocks of the same size. Specifically, each image sample can be divided into 4 image blocks of the same size or 8 image blocks of the same size. Alternatively, each image sample can be divided according to various specified division rules. For example, a specified area in each image sample can be divided into multiple image blocks of the same size, and the remaining area can be randomly divided to obtain multiple image blocks. The specific division can be set according to the actual situation, and the embodiments in this specification do not limit this.

[0044] In step A4, a first number of image samples are selected from multiple different image samples based on a first preset ratio.

[0045] The first preset ratio corresponds to the first quantity. For example, if there are 100 different image samples and the first preset ratio is 50%, then the first quantity can be 50. Or, if there are 10 different image samples and the first preset ratio is 40%, then the first quantity can be 4, etc. The specific setting can be determined according to the actual situation.

[0046] In step A6, an image block is selected from each image sample in the first number of image samples, and the unselected image blocks in each image sample are masked to obtain a masked image selected at a first preset ratio.

[0047] In implementation, since the module built based on Vision Transformer cannot provide effective correlation information for the only remaining image block, an image block is selected from each image sample in the first number of image samples, and the unselected image blocks in each image sample are masked, which is equivalent to masking the image sample completely, thereby obtaining the mask image selected by the first preset ratio.

[0048] In step A8, from each of the remaining image samples (excluding the first number of image samples) from multiple different image samples, an image block is selected with a preset occlusion ratio, and the selected image block in each image sample is masked to obtain the remaining mask image.

[0049] In practical applications, the specific processing method for determining the correlation information between multiple image samples based on the first prediction results corresponding to multiple image samples in step S106 above can be varied. The following provides another optional processing method, which may include the following: inputting the first prediction results corresponding to multiple image samples into the target model to obtain the correlation information between multiple image samples. The target model is used to project the vector used for predicting category information in the first prediction results corresponding to multiple image samples onto the first dimension, and calculate the similarity between the first prediction results after different projections. Based on the calculated similarity, the correlation information between multiple image samples is determined. The first dimension is less than a preset dimension threshold.

[0050] The target model can be constructed in various ways. For example, it can be constructed using a neural network, or using a specified algorithm (such as a classification algorithm, clustering algorithm, etc.), or it can be constructed using a neural network combined with a specified algorithm, etc. The specific method can be set according to the actual situation, and this specification does not limit this. The preset dimension threshold can be set according to the actual situation, such as 5 or 4.

[0051] In implementation, such as Figure 3As shown, a lightweight projection head (i.e., the target model) can be added to the output of the teacher model to calculate the relationship between multiple image samples, depending on the actual situation. Specifically, considering that the first prediction result corresponding to each image sample will contain a vector for predicting category information, the dimension of this vector may be high. In this case, the vector can be dimensionality reduced to project the vector to a lower dimension. In addition, in order to determine the correlation between multiple image samples, the above determination process can be completed by combining similarity. Based on this, the corresponding target model can be constructed through the above-mentioned processing or functions, and the target model can be trained using training samples to obtain the trained target model.

[0052] The first prediction results corresponding to multiple image samples can be input into the target model. The target model extracts vectors for predicting category information (usually CLS tokens, which can be located at a specified position in the first prediction results) from the first prediction results. The target model can then project the vectors for predicting category information from the first prediction results corresponding to multiple image samples into a lower-dimensional (i.e., the first dimension) latent space. Then, the similarity between the first prediction results after different projections can be calculated. Based on the calculated similarity, the association information between multiple image samples can be obtained through a specified algorithm (such as weighted calculation or fusion algorithm) or transformation relationship. Alternatively, the calculated similarity can be directly used as the association information between multiple image samples, etc. The specific settings can be configured according to the actual situation.

[0053] In practical applications, in addition to constructing the target model in the above way, there are many other ways to construct the target model. The following is another model structure for the target model, which includes a linear layer, a ReLU nonlinear activation layer and a normalization layer.

[0054] In implementation, the first prediction results corresponding to multiple image samples can be input into the target model. Through the linear layer in the target model, the vectors used to predict category information (usually CLS Tokens) in the first prediction results corresponding to multiple image samples can be projected into the lower first-dimensional latent space. Then, through the ReLU nonlinear activation layer and the normalization layer, the similarity between the first prediction results after different projections can be calculated. Based on the calculated similarity, the association information between multiple image samples can be obtained through the specified algorithm or transformation relationship.

[0055] In practical applications, the specific processing method for modulating the second prediction result corresponding to each mask image based on the correlation information between multiple image samples in step S106 above to obtain the prediction coding information of the reconstructed image corresponding to each mask image can be varied. Another optional processing method is provided below, such as... Figure 4 As shown, the process may specifically include the following steps S1062 to S1066.

[0056] In step S1062, the correlation information between multiple image samples and the location information of the mask information corresponding to the mask image are fused with the second prediction result corresponding to each mask image to obtain a fused representation.

[0057] In implementation, such as Figure 5 As shown, the second prediction results corresponding to multiple masked images can be represented by aggregated information, i.e. Figure 5 The small box at the top left can expand the second prediction results corresponding to multiple mask images to the size of the mask image. Then, the correlation information between multiple image samples (specifically, the embedded features of the correlation information between multiple image samples) and the location information of the mask information corresponding to the mask image (specifically, the embedded features of the location information) can be fused with the expanded second prediction results to obtain the fused representation.

[0058] In step S1064, the vector used to predict category information in the fusion representation is recombined using the correlation information between multiple image samples to obtain recombined information for predicting category information.

[0059] In implementation, such as Figure 5 As shown, based on the correlation information between multiple image samples, the Softmax function can be used to calculate the corresponding Softmax value. Vectors for predicting category information can be extracted from the fused representation. These vectors can then be recombined using the Softmax value (i.e.,...). Figure 5 (Multiplication calculation in the process) yields recombined information used to predict category information.

[0060] In step S1066, the remaining vectors in the fusion representation, excluding the vectors used to predict category information, are aggregated and randomly shuffled according to a preset distribution rule to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0061] In implementation, such as Figure 5As shown, in addition to the vectors used to predict category information in the fusion representation, the remaining vectors (i.e., the vectors corresponding to the remaining tokens other than CLS Tokens) are aggregated and then randomly shuffled according to the uniform distribution rule. These shuffled information are readjusted to the original dimensions to obtain the predictive coding information of the reconstructed image corresponding to each mask image.

[0062] In practical applications, the specific processing method of step S108 above can vary. The following provides another optional processing method, such as... Figure 6 As shown, the specific process may include the following steps S1082 to S1086.

[0063] In step S1082, the prediction coding information of the reconstructed images corresponding to multiple mask images is input into the decoder to obtain the third prediction result of the reconstructed image corresponding to each mask image.

[0064] In step S1084, based on the third prediction result of the reconstructed image corresponding to each mask image and the first prediction result corresponding to each image sample, the corresponding loss information is obtained through a preset loss function.

[0065] The loss function mentioned above can include various types, such as the mean squared error loss function and the cross-entropy loss function, which can be set according to the actual situation.

[0066] In step S1086, the model parameters of the student model and the decoder are adjusted based on the above loss information to train the student model until the above loss function converges, thus obtaining the trained student model.

[0067] In practical applications, based on the target model described above, step S1086 may further include: adjusting the model parameters of the student model, decoder, and target model based on the loss information to jointly train the student model, decoder, and target model until the loss function converges, thereby obtaining the trained student model.

[0068] In practical applications, the loss function mentioned above may include the Smooth L1 loss function. The number and width of network layers in the decoder are smaller than the number and width of network layers in the encoder corresponding to the decoder. The encoder is used to generate predictive coding information for the reconstructed image corresponding to each mask image.

[0069] In implementation, an asymmetric encoder-decoder can be used, meaning that the number of network layers in the decoder is less than the number of network layers in the encoder, and the width of the network layers in the decoder is less than the width of the network layers in the encoder.

[0070] This specification provides a model processing method. It involves obtaining a first batch of image samples from multiple batches of image samples used to train a student model. This first batch includes multiple different image samples and masked images of these samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model, obtaining a first prediction result for each image sample. The multiple masked images are then input into the student model, obtaining a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on this correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image for each masked image. The predictive coding information of the images is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, based on the correlation information between multiple image samples, the second prediction result corresponding to each mask image is modulated, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effectiveness of model training.

[0071] The above describes the model processing method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a model processing device, such as... Figure 7 As shown.

[0072] The processing unit of the model includes: a sample acquisition module 701, a prediction module 702, a modulation module 703, and a model training module 704, wherein:

[0073] The sample acquisition module 701 acquires the first batch of image samples from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked.

[0074] The prediction module 702 inputs multiple image samples into the teacher model corresponding to the student model to obtain a first prediction result for each image sample, and inputs multiple mask images into the student model to obtain a second prediction result for each mask image.

[0075] The modulation module 703 determines the correlation information between the multiple image samples based on the first prediction results corresponding to the multiple image samples, and modulates the second prediction result corresponding to each mask image based on the correlation information between the multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0076] The model training module 704 trains the student model based on the first prediction result corresponding to each image sample and the prediction encoding information of the reconstructed image corresponding to each mask image, thereby obtaining the trained student model.

[0077] In the embodiments of this specification, the plurality of mask images include a mask image selected based on a first preset ratio and the remaining mask images. The mask image selected based on the first preset ratio is a mask image obtained after completely occluding the corresponding image sample. The remaining mask images are mask images obtained after occluding the corresponding image sample with a preset occlusion ratio.

[0078] In the embodiments described in this specification, the student model includes a module built based on Vision Transformer, and the device further includes:

[0079] The segmentation module divides each image sample into multiple image blocks;

[0080] The selection module selects a first number of image samples from multiple different image samples based on the first preset ratio;

[0081] The first masking module selects an image block from each image sample in the first number of image samples, and performs masking processing on the unselected image blocks in each image sample to obtain the masked image selected by the first preset ratio.

[0082] The second masking module selects image blocks from each of the remaining image samples (excluding the first number of image samples) from multiple different image samples, according to a preset occlusion ratio, and performs masking processing on the selected image blocks in each image sample to obtain the remaining masked image.

[0083] In this embodiment of the specification, the modulation module 703 inputs the first prediction results corresponding to multiple image samples into the target model to obtain the association information between the multiple image samples. The target model is used to project the vector used for predicting category information in the first prediction results corresponding to multiple image samples onto a first dimension, calculate the similarity between the first prediction results after different projections, and determine the association information between the multiple image samples based on the calculated similarity. The first dimension is less than a preset dimension threshold.

[0084] In the embodiments described in this specification, the target model includes a linear layer, a ReLU nonlinear activation layer, and a normalization layer.

[0085] In this embodiment of the specification, the modulation module 703 includes:

[0086] The fusion unit fuses the correlation information between multiple image samples and the location information of the mask information corresponding to the mask image with the second prediction result corresponding to each mask image to obtain a fused representation;

[0087] The recombination unit reassembles the vector used to predict category information in the fusion representation using the correlation information between multiple image samples to obtain recombined information for predicting category information.

[0088] The processing unit aggregates the remaining vectors in the fusion representation, excluding the vectors used to predict category information, and randomly shuffles them according to a preset distribution rule to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0089] In this embodiment of the specification, the model training module 704 includes:

[0090] The decoding unit inputs the prediction coding information of the reconstructed images corresponding to multiple mask images into the decoder to obtain the third prediction result of the reconstructed image corresponding to each mask image;

[0091] The loss determination unit obtains the corresponding loss information based on the third prediction result of the reconstructed image corresponding to each mask image and the first prediction result corresponding to each image sample through a preset loss function.

[0092] The training unit adjusts the model parameters of the student model and the decoder based on the loss information to train the student model until the loss function converges, thereby obtaining the trained student model.

[0093] In the embodiments of this specification, the loss function includes the Smooth L1 loss function, the number and width of the network layers in the decoder are smaller than the number and width of the network layers in the encoder corresponding to the decoder, and the encoder is used to generate predictive coding information for the reconstructed image corresponding to each mask image.

[0094] This specification provides a model processing apparatus that obtains a first batch of image samples from multiple batches of image samples used to train a student model. The first batch of image samples includes multiple different image samples and masked images of these image samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result for each image sample. The multiple masked images are then input into the student model to obtain a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on the correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image for each masked image. The predictive coding information of the images is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, based on the correlation information between multiple image samples, the second prediction result corresponding to each mask image is modulated, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effectiveness of model training.

[0095] The above describes the model processing apparatus provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a model processing device, such as... Figure 8 As shown.

[0096] The processing device for the model can be a terminal device or a server, as described in the above embodiments.

[0097] The processing device of the model can vary considerably due to differences in configuration or performance, and may include one or more processors 801 and memory 802. Memory 802 may store one or more application programs or data. Memory 802 may be temporary or persistent storage. The application programs stored in memory 802 may include one or more modules (not shown in the figures), each module including a series of computer-executable instructions for the model's processing device. Furthermore, processor 801 may be configured to communicate with memory 802 and execute the series of computer-executable instructions in memory 802 on the model's processing device. The model's processing device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, and one or more keyboards 806.

[0098] Specifically, in this embodiment, the processing device for the model includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the processing device of the model, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0099] The first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked.

[0100] Multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result for each image sample. Multiple mask images are input into the student model to obtain a second prediction result for each mask image.

[0101] Based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0102] The student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain the trained student model.

[0103] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for processing the model are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0104] This specification provides a model processing device that obtains a first batch of image samples from multiple batches of image samples used to train a student model. The first batch of image samples includes multiple different image samples and masked images of these image samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result for each image sample. The multiple masked images are then input into the student model to obtain a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on the correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image for each masked image. The predictive coding information of the images is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, based on the correlation information between multiple image samples, the second prediction result corresponding to each mask image is modulated, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effectiveness of model training.

[0105] Furthermore, based on the above Figures 1 to 6 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instruction information. In one specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process:

[0106] The first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked.

[0107] Multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result for each image sample. Multiple mask images are input into the student model to obtain a second prediction result for each mask image.

[0108] Based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0109] The student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain the trained student model.

[0110] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0111] This specification provides a storage medium that obtains a first batch of image samples from multiple batches of image samples used to train a student model. The first batch of image samples includes multiple different image samples and masked images of these image samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result for each image sample. The multiple masked images are then input into the student model to obtain a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on the correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image corresponding to each masked image. The predictive coding information is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image sample (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation information between multiple image samples, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effect of model training.

[0112] Furthermore, based on the above Figures 1 to 6 The method shown in this specification, along with one or more embodiments, also provides a computer program product including a computer program that, when executed by a processor, performs the following process:

[0113] The first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked.

[0114] Multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result for each image sample. Multiple mask images are input into the student model to obtain a second prediction result for each mask image.

[0115] Based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0116] The student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain the trained student model.

[0117] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0118] This specification provides a computer program product that obtains a first batch of image samples from multiple batches of image samples used to train a student model. The first batch of image samples includes multiple different image samples and masked images of these image samples after masking. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result for each image sample. The multiple masked images are then input into the student model to obtain a second prediction result for each masked image. Based on the first prediction results, the correlation information between the multiple image samples is determined. Based on this correlation information, the second prediction result for each masked image is modulated to obtain a reconstructed image for each masked image. The predictive coding information of the images is used to train the student model. Based on the first prediction result corresponding to each image sample and the predictive coding information of the reconstructed image corresponding to each mask image, the student model is trained. In this way, the first prediction result input by the teacher model is used to calculate the correlation between image samples in the same batch, which is used as auxiliary information for the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, based on the correlation information between multiple image samples, the second prediction result corresponding to each mask image is modulated, which can promote the information interaction between multiple image samples, thereby improving the student model's understanding of the relationship between image samples, enhancing the downstream generalization ability of the trained student model, and improving the efficiency and effectiveness of model training.

[0119] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0120] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0121] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0122] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0123] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0124] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0130] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0135] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for processing a model, the method comprising: The first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked. Multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result for each image sample. Multiple mask images are input into the student model to obtain a second prediction result for each mask image. Based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image. The student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain the trained student model.

2. The method according to claim 1, wherein the plurality of mask images includes a mask image selected based on a first preset ratio and the remaining mask images, wherein the mask image selected based on the first preset ratio is a mask image obtained after performing a masking process that completely occludes the corresponding image sample, and the remaining mask images are mask images obtained after performing a masking process that occludes the corresponding image sample with a preset occlusion ratio.

3. The method according to claim 2, wherein the student model includes a module built based on Vision Transformer, and the method further includes: Each image sample is divided into multiple image blocks; Based on the first preset ratio, a first number of image samples are selected from multiple different image samples; One image block is selected from each image sample in the first number of image samples, and the unselected image blocks in each image sample are masked to obtain the masked image selected by the first preset ratio. From each of the remaining image samples (excluding the first number of image samples) from multiple different image samples, an image block is selected with a preset occlusion ratio, and the selected image block in each image sample is masked to obtain the remaining masked image.

4. The method according to claim 1, wherein determining the association information between the multiple image samples based on the first prediction results corresponding to the multiple image samples includes: The first prediction results corresponding to multiple image samples are input into the target model to obtain the association information between the multiple image samples. The target model is used to project the vector used to predict category information in the first prediction results corresponding to multiple image samples onto a first dimension, calculate the similarity between the first prediction results after different projections, and determine the association information between the multiple image samples based on the calculated similarity. The first dimension is less than a preset dimension threshold.

5. The method according to claim 4, wherein the target model includes a linear layer, a ReLU nonlinear activation layer, and a normalization layer.

6. The method according to claim 1, wherein the step of modulating the second prediction result corresponding to each mask image based on the correlation information between multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image includes: The association information between multiple image samples and the location information of the mask information corresponding to the mask image are fused with the second prediction result corresponding to each mask image to obtain a fused representation; The vector used to predict category information in the fusion representation is recombined using the correlation information between multiple image samples to obtain recombined information for predicting category information. The remaining vectors in the fusion representation, excluding the vectors used to predict category information, are aggregated and randomly shuffled according to a preset distribution rule to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

7. The method according to claim 1 or 6, wherein training the student model based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images to obtain the trained student model includes: The prediction coding information of the reconstructed images corresponding to multiple mask images is input into the decoder to obtain the third prediction result of the reconstructed image corresponding to each mask image; Based on the third prediction result of the reconstructed image corresponding to each mask image and the first prediction result corresponding to each image sample, the corresponding loss information is obtained through a preset loss function. The model parameters of the student model and the decoder are adjusted based on the loss information to train the student model until the loss function converges, thus obtaining the trained student model.

8. The method according to claim 7, wherein the loss function includes a Smooth L1 loss function, the number and width of network layers in the decoder are both smaller than the number and width of network layers in the encoder corresponding to the decoder, and the encoder is used to generate predictive coding information for the reconstructed image corresponding to each mask image.

9. A model processing apparatus, the apparatus comprising: The sample acquisition module acquires the first batch of image samples from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked. The prediction module inputs multiple image samples into the teacher model corresponding to the student model to obtain a first prediction result for each image sample, and inputs multiple mask images into the student model to obtain a second prediction result for each mask image. The modulation module determines the correlation information between the multiple image samples based on the first prediction results corresponding to the multiple image samples, and modulates the second prediction result corresponding to each mask image based on the correlation information between the multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image. The model training module trains the student model based on the first prediction result corresponding to each image sample and the prediction encoding information of the reconstructed image corresponding to each mask image, thereby obtaining the trained student model.

10. A model processing apparatus, the model processing apparatus comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: The first batch of image samples is obtained from multiple batches of image samples used to train the student model. The first batch of image samples includes multiple different image samples and a mask image after the image samples have been masked. Multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result for each image sample. Multiple mask images are input into the student model to obtain a second prediction result for each mask image. Based on the first prediction results corresponding to multiple image samples, the correlation information between the multiple image samples is determined, and based on the correlation information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image. The student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain the trained student model.

Images