Model processing method, device and equipment

By using the prediction results and association information of the teacher model in visual model training, the mask image samples are modulated and processed to generate reconstruction encoding information, which solves the problem of insufficient information utilization in the prior art and improves training efficiency and generalization ability.

CN119006957AActive Publication Date: 2024-11-22ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411124386.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The prior art fails to fully utilize teacher model information in visual model training, resulting in insufficient training efficiency and generalization ability, especially in terms of relationship cognition between image samples.

Method used

By obtaining masked image samples from multiple batches of image samples, using the teacher model to perform prediction results and association relationship information for modulation processing, the prediction encoding information of the reconstructed image is generated, and the student model is trained based on this to enhance the relationship between image samples. Cognition.

Benefits of technology

The training efficiency and downstream generalization ability of visual models are improved, the understanding of the relationship between image samples is enhanced, and more efficient model training effect is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006957A_ABST
    Figure CN119006957A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a model processing method, device and equipment, and the method comprises the steps: obtaining a first batch of image samples from a plurality of batches of image samples used for training a student model, wherein the first batch of image samples comprise a plurality of different image samples and mask images after the image samples are subjected to mask processing; inputting the plurality of image samples into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and inputting the plurality of mask images into the student model to obtain a second prediction result corresponding to each mask image; based on the first prediction result, determining association relationship information among the plurality of image samples, and performing modulation processing on the second prediction result based on the association relationship information among the plurality of image samples to obtain prediction coding information of a reconstructed image corresponding to each mask image; and training a student model based on the first prediction result and the prediction coding information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a model processing method, device and equipment. Background Art

[0002] As people pay more and more attention to their privacy data, the security of visual data is becoming more and more important. Usually, a large amount of unlabeled visual data can be used for self-supervised learning and representation learning to pre-train visual models, but how to make full use of model information to achieve efficient training of the model is an important topic in the field of vision. To this end, it is necessary to provide a technical solution that can train the model more efficiently and improve the visual model's cognition of the relationship between image samples to further enhance the downstream generalization ability of the trained visual model. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a technical solution that can perform model training more efficiently and improve the visual model that needs to be trained's understanding of the relationship between image samples, so as to further enhance the downstream generalization ability of the trained visual model.

[0004] In order to implement the above technical solution, the embodiments of this specification are implemented as follows:

[0005] The embodiment of the present specification provides a method for processing a model, the method comprising: obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked. Input the multiple image samples into the teacher model corresponding to the student model, respectively obtain the first prediction results corresponding to each image sample, and input the multiple mask images into the student model, respectively obtain the second prediction results corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, determine the association relationship information between the multiple image samples, and modulate the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image. Based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image, train the student model to obtain the trained student model.

[0006] The embodiment of the present specification provides a model processing device, the device comprising: a sample acquisition module, which acquires a first batch of image samples from a plurality of image samples used to train a student model, the first batch of image samples including a plurality of different image samples and a mask image after the image samples are masked. A prediction module, which inputs a plurality of the image samples into a teacher model corresponding to the student model, obtains a first prediction result corresponding to each of the image samples, and inputs a plurality of the mask images into the student model, obtains a second prediction result corresponding to each of the mask images. A modulation module, which determines the association relationship information between the plurality of image samples based on the first prediction results corresponding to the plurality of image samples, and modulates the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain the prediction coding information of the reconstructed image corresponding to each of the mask images. A model training module, which trains the student model based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, to obtain a trained student model.

[0007] The embodiment of the present specification provides a model processing device, the model processing device includes: a processor; and a memory arranged to store computer executable instructions, the executable instructions when being executed, make the processor: obtain a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples includes multiple different image samples, and mask images after the image samples are masked. Input multiple image samples into the teacher model corresponding to the student model, respectively obtain the first prediction result corresponding to each image sample, and input multiple mask images into the student model, respectively obtain the second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, determine the association relationship information between the multiple image samples, and modulate the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image. Based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image, train the student model to obtain the trained student model.

[0008] The embodiment of the present specification also provides a storage medium, the storage medium is used to store computer executable instructions, and the executable instructions implement the following process when executed by the processor: obtain a first batch of image samples from multiple batches of image samples used to train the student model, the first batch of image samples includes multiple different image samples, and mask images after the image samples are masked. Input multiple image samples into the teacher model corresponding to the student model, respectively obtain the first prediction result corresponding to each image sample, and input multiple mask images into the student model, respectively obtain the second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, determine the association relationship information between the multiple image samples, and modulate the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples to obtain the prediction coding information of the reconstructed image corresponding to each mask image. Based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image, train the student model to obtain the trained student model.

[0009] The embodiment of the present specification also provides a computer program product, including a computer program, which implements the following process when executed by a processor: obtaining a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked. Inputting multiple image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each image sample, and inputting multiple mask images into the student model, respectively obtaining a second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, determining the association relationship information between the multiple image samples, and modulating the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples, obtaining the prediction coding information of the reconstructed image corresponding to each mask image. Based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image, the student model is trained to obtain a trained student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0011] Figure 1 This is an embodiment of a processing method of a model in this specification;

[0012] Figure 2 A schematic diagram of the processing process of a model of this specification;

[0013] Figure 3 A schematic diagram of the processing process of another model of this specification;

[0014] Figure 4 This is another processing method embodiment of the present specification;

[0015] Figure 5 A schematic diagram of a modulation process of information in this specification;

[0016] Figure 6 This is another processing method embodiment of a model of this specification;

[0017] Figure 7 This is a processing device embodiment of a model of this specification;

[0018] Figure 8 This is a processing device embodiment of a model of this specification. DETAILED DESCRIPTION

[0019] The embodiments of this specification provide a method, device and equipment for processing a model.

[0020] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0021] The embodiments of this specification provide a self-supervised learning mechanism based on feature distillation. As people pay more and more attention to their privacy data, the security of visual data is becoming more and more important. Generally, a large amount of unlabeled visual data can be used for self-supervised learning and representation learning to pre-train visual models. Among them, the visual mask learning method has been verified as an efficient training method. The basic method is to block the visual data before inputting into the visual model according to a certain proportion, and then use the encoded representation of the mask image to reconstruct the prediction result corresponding to the complete visual data to achieve the purpose of pre-training. However, the above methods all use a visual model that has been pre-trained by self-supervision as a teacher model, but only consider the use of the unmasked part of the same image as the clue of the mask part of itself, so that the information of the teacher model is not fully utilized, which is not conducive to efficient training. The embodiments of this specification provide a feature distillation algorithm that can perform model training more efficiently. The algorithm not only considers the reconstruction based on the image sample, but also uses the relationship between the image samples from the teacher model as a guide for the reconstruction task, thereby improving the student model that needs to be trained for the relationship between the image samples. The downstream generalization ability of the trained visual model is further enhanced. For specific processing, please refer to the specific content in the following embodiments.

[0022] like Figure 1 As shown, an embodiment of this specification provides a method for processing a model, and the execution subject of the method may be a terminal device or a server, etc., wherein the terminal device may be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically a smart watch, a vehicle-mounted device, etc.), etc., wherein the server may be an independent server, or a server cluster composed of multiple servers, etc., and the server may be a background server for a financial service or an online shopping service, or a background server for an application, etc. In this embodiment, a detailed description is given by taking the execution subject as a server as an example. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps:

[0023] In step S102, a first batch of image samples is obtained from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked.

[0024] Among them, the student model can be a model that needs to be used in any business. Specifically, the student model can be a risk prevention and control model, a biometric model, etc. in financial businesses such as payment services and transfer services. The risk prevention and control model can be used to identify risks of images input by users or collected images to determine whether the image will attack the system or user's account, etc. The biometric model can be used for facial recognition, fingerprint recognition, palm print recognition and other processing. In addition, the student model can also be a risk identification model in instant messaging services. The risk identification model can identify the information contained in the image input by the user to determine whether it contains specified risks (such as fraud risks or privacy leakage risks, etc.), etc. It can be set specifically according to actual conditions, and the embodiments of this specification do not limit this. In addition, according to the purpose or function of the model, the student model can be a model of any purpose or function. For example, the learning model can be a model for classification (such as binary classification, multi-classification, etc.), the learning model can also be a model for data prediction, the learning model can also be a model for risk detection, the learning model can also be a model for semantic analysis, etc. The structure of the learning model can be various. The learning model can be constructed by a convolutional neural network, or by a classification algorithm, or by ResNet, etc. It can be set according to actual conditions, and this specification does not limit this. Multiple batches of image samples are multiple Batches image samples. Before training the model, the training samples are usually divided into multiple batches, and then the model is trained by using the training samples of each batch in turn. The first batch of image samples can be any batch of image samples in multiple batches of image samples. The masking of image samples can be a process in which the image samples are completely or partially blocked.

[0025] In implementation, Figure 2As shown, considering the advantages of the knowledge distillation mechanism, model training can be performed by means of knowledge distillation, that is, a visual model that has undergone self-supervised pre-training is used as a teacher model to train a student model that needs to be trained, wherein multiple batches of image samples for training the student model can be obtained in a variety of different ways. For example, a certain number of image samples can be obtained from a specified database, and then a certain number of image samples can be divided into multiple different batches to obtain multiple batches of image samples, or a certain number of image samples can be crawled from the Internet through a web crawler, and then a certain number of image samples can be divided into multiple different batches to obtain multiple batches of image samples, or images provided by the user can be obtained from different users, and the images provided by the user can be used as image samples, thereby obtaining a certain number of image samples, and then a certain number of image samples can be divided into multiple different batches to obtain multiple batches of image samples, etc. The specific method can be set according to the actual situation, and the embodiments of this specification are not limited to this.

[0026] Among them, each batch of image samples in the above-mentioned multiple batches of image samples may include multiple different image samples, and a mask image after each image sample is masked, that is, each batch of image samples may include multiple different image samples, and multiple different mask images. After obtaining multiple batches of image samples in the above manner, the first batch of image samples can be obtained from the multiple batches of image samples to perform model training on the student model. For details, please refer to the following related content.

[0027] In step S104, multiple image samples are input into the teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and multiple mask images are input into the student model to obtain a second prediction result corresponding to each mask image.

[0028] Among them, the teacher model can be any model, for example, the teacher model can be a model for classification (specifically such as binary classification, multi-classification, etc.), or a model for data prediction, or a model for risk detection, or a model for semantic analysis, etc. The structure of the teacher model can be various, and the teacher model can be constructed by a convolutional neural network, or by a classification algorithm, or by ResNet, etc., which can be set specifically according to actual conditions, and this specification embodiment does not limit this. The teacher model and the above-mentioned student model can be trained by knowledge distillation. Knowledge distillation can be a mechanism for migrating the knowledge of a deep model or an integrated model (i.e., a teacher model) with many parameters to a relatively small shallow model (i.e., a student model). Based on this, the teacher model has the same function or role as the above-mentioned student model. For example, if the student model is a biometric model in a financial business, then the teacher model can also be applied to biometrics in financial business, and if the student model is a risk identification model in an instant messaging business, then the teacher model can also be applied to risk identification in an instant messaging business, which can be set specifically according to actual conditions. In this embodiment, the model parameters of the teacher model can be in a frozen state, that is, the model parameters of the teacher model are fixed.

[0029] In implementation, Figure 2 As shown, the teacher model can be used to provide the student model with the goal and clues of the prediction results based on the complete image samples. Based on this, multiple image samples in the first batch of image samples can be input into the teacher model, and each image sample is processed by the teacher model to obtain the corresponding output result (i.e., the first prediction result corresponding to each image sample). Among them, the multiple image samples simultaneously input into the teacher model can be all the image samples included in the first batch of image samples, or can be part of the image samples, for example, it can be 2 image samples among the multiple image samples in the first batch of image samples, or it can be 3 or 5 image samples among the multiple image samples in the first batch of image samples, etc., which can be set according to the actual situation.

[0030] Since the above processing of the teacher model is to provide the student model with the goal and clues of the prediction results, the samples input into the student model should correspond to the image samples input into the teacher model, that is, if 2 image samples are input into the teacher model, then the samples input into the student model should be the mask images corresponding to the above 2 image samples. For example, if 3 image samples are input into the teacher model, namely image sample 1, image sample 2 and image sample 3, then the samples input into the student model should be the mask image corresponding to image sample 1, the mask image corresponding to image sample 2 and the mask image corresponding to image sample 3. Based on this, multiple mask images consisting of the mask image corresponding to each image sample in the multiple image samples can be input into the student model, and each mask image is processed by the student model to obtain the corresponding output result (i.e., the second prediction result corresponding to each mask image).

[0031] It should be noted that in order to simplify data processing and data storage, the first prediction result corresponding to each of the above-mentioned image samples can be represented by an aggregated data. For example, a matrix or vector can be used to represent the first prediction results corresponding to multiple image samples (that is, one matrix or vector represents multiple first prediction results). Correspondingly, the second prediction result corresponding to each of the above-mentioned mask images can be represented by an aggregated data. For example, a matrix or vector can be used to represent multiple second prediction results. In actual applications, different data can also be used to represent the first prediction result corresponding to each image sample or the second prediction result corresponding to each mask image, etc. The specific setting can be based on actual conditions, and the embodiments of this specification are not limited to this.

[0032] In step S106, based on the first prediction results corresponding to the multiple image samples, the association relationship information between the multiple image samples is determined, and based on the association relationship information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image.

[0033] The reconstructed image may be an image obtained by restoring an image blocked in the mask image.

[0034] In implementation, Figure 2As shown, since the above-mentioned related processing method only considers using the unmasked part of the same image as the clue of the masked part of the image itself, and does not consider the association between other image samples from the same batch and the current image, it leads to insufficient utilization of the teacher model information, which is not conducive to efficient training. For this reason, the association between other image samples from the same batch and the current image can be considered to modulate the direct prediction results of the student model, so as to more fully utilize the information of the teacher model and improve the training efficiency. Specifically, the similarity between the first prediction results corresponding to any two image samples of the first prediction results corresponding to multiple image samples can be calculated. Based on the calculated similarity information, the association relationship information between the multiple image samples can be determined by a specified algorithm (such as a weighted average algorithm, a clustering algorithm, a fusion algorithm, etc.) or a mapping relationship. The association relationship information between the multiple image samples can be fused to the second prediction result corresponding to each mask image to obtain the association relationship information for each mask image. The second prediction result corresponding to each mask image is modulated to obtain the prediction coding information of the reconstructed image corresponding to each mask image. The adjustment processing of fusing the association relationship information between multiple image samples into the second prediction result corresponding to each mask image can be varied. For example, an encoder can be pre-constructed, and the association relationship information between multiple image samples and the second prediction result corresponding to each mask image are respectively encoded by the encoder to obtain the encoding information corresponding to the association relationship information between multiple image samples and the encoding information of the second prediction result corresponding to each mask image. Then, the above two encoding information can be calculated by a preset fusion algorithm (such as a matrix or vector multiplication algorithm, an addition algorithm, etc.), and the calculation result obtained is used as the prediction coding information of the reconstructed image corresponding to each mask image. In addition, the prediction coding information of the reconstructed image corresponding to each mask image can be obtained by many other different methods, which can be set according to actual conditions, and the embodiments of this specification do not limit this.

[0035] It should be noted that the prediction coding information of the reconstructed image corresponding to each mask image can also be represented by an aggregated data. For example, a matrix or vector can be used to represent the prediction coding information of the reconstructed images corresponding to multiple mask images (that is, one matrix or vector represents multiple prediction coding information). The specific setting can be based on actual conditions, and the embodiments of this specification do not limit this.

[0036] In step S108, the student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image to obtain a trained student model.

[0037] In implementation, the first prediction result corresponding to each image sample can be encoded by the above-mentioned encoder to obtain the encoding information of the first prediction result corresponding to each image sample. Then, the corresponding loss information can be calculated by a preset loss function based on the encoding information of the first prediction result corresponding to each image sample and the prediction encoding information of the reconstructed image corresponding to each mask image. The loss information can be used to perform gradient backpropagation to update the model parameters of the student model. Then, the above-mentioned processing can be continued using image samples from other batches of multiple batches of image samples to update the model parameters of the student model until the number of iterations reaches a preset threshold or the above-mentioned loss function converges. Finally, the trained student model can be obtained. Alternatively, a decoder corresponding to the above-mentioned encoder can be constructed, and the decoder can be used to decode the predicted coding information of the reconstructed image corresponding to each mask image to obtain the predicted result of the reconstructed image corresponding to each mask image. The corresponding loss information can be calculated according to the first prediction result corresponding to each image sample and the predicted result of the reconstructed image corresponding to each mask image through a preset loss function. The loss information can be used to perform gradient backpropagation to update the model parameters of the student model. Then, the above-mentioned processing can be continued using image samples from other batches of multiple batches of image samples to update the model parameters of the student model until the number of iterations reaches a preset threshold or the above-mentioned preset loss function converges. Finally, a trained student model can be obtained, or the student model can be trained by other methods, which can be set specifically according to actual conditions and are not limited to this in the embodiments of this specification.

[0038] The embodiment of the present specification provides a model processing method, by obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked, then, inputting the multiple image samples into a teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each image sample, and inputting the multiple mask images into the student model, respectively obtaining a second prediction result corresponding to each mask image, and determining association relationship information between the multiple image samples based on the first prediction results corresponding to the multiple image samples, and modulating the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples to obtain a reconstructed image corresponding to each mask image. The student model is trained based on the prediction coding information of the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0039] In actual applications, the multiple mask images include mask images selected based on a first preset ratio and remaining mask images. In order to improve the performance and generalization ability of the student model, the mask images selected based on the first preset ratio are mask images obtained after mask processing of the corresponding image samples is completely occluded, and the remaining mask images are mask images obtained after mask processing of the corresponding image samples is occluded at a preset occlusion ratio.

[0040] Among them, the first preset ratio can be set according to actual conditions, such as 20% or 50%. The remaining mask images are mask images other than the mask images selected based on the first preset ratio in the multiple mask images. The mask processing of complete occlusion can be equivalent to the mask processing of completely occluding the image sample relative to the image processing module in the student model. Based on this, in practical applications, the mask processing of complete occlusion can be the mask processing performed by completely occluding the image sample by the mask, or the mask processing of complete occlusion can also be the mask processing performed by masking the partial image except one or more specified sizes of partial images in the image sample, and the image not occluded by the mask cannot provide effective correlation information in the image processing module in the student model. At this time, it is also recognized as the mask processing of complete occlusion, that is, equivalent to the mask processing of completely occluding the image sample. The preset occlusion ratio can be the ratio of a certain image or a partial image in a certain image that needs to be occluded. The preset occlusion ratio can be set according to actual conditions, such as 50% or 60%.

[0041] In practical applications, the student model may include a module built based on Vision Transformer. Based on the division of the above-mentioned multiple mask images, multiple mask images can be obtained in the following manner. For details, please refer to the processing of the following steps A2 to A8.

[0042] In step A2, each image sample is divided into a plurality of image blocks.

[0043] In implementation, each image sample can be divided evenly. For example, each image sample can be divided evenly into multiple image blocks of the same size. Specifically, each image sample can be divided evenly into 4 image blocks of the same size or each image sample can be divided evenly into 8 image blocks of the same size, etc. For another example, each image sample can be divided according to a plurality of different specified division rules. For example, a specified area in each image sample can be divided evenly into multiple image blocks of the same size, and the remaining area can be randomly divided, and finally a plurality of image blocks can be obtained, etc. The specific setting can be based on the actual situation, and the embodiments of this specification do not limit this.

[0044] In step A4, a first number of image samples are selected from a plurality of different image samples based on a first preset ratio.

[0045] Among them, the first preset ratio corresponds to the first quantity. For example, the multiple different image samples are 100 image samples, the first preset ratio is 50%, and the first quantity can be 50. For another example, the multiple different image samples are 10 image samples, the first preset ratio is 40%, and the first quantity can be 4, etc. The specific setting can be based on actual conditions.

[0046] In step A6, an image block is selected from each image sample in the first number of image samples, and the unselected image blocks in each image sample are subjected to mask processing to obtain a mask image selected at a first preset ratio.

[0047] In implementation, since the module built based on the Vision Transformer cannot provide effective correlation information for the only remaining image block, at this time, one image block is selected from each image sample in the first number of image samples, and the unselected image blocks in each image sample are masked, which is equivalent to masking the image sample completely, thereby obtaining a mask image selected at a first preset ratio.

[0048] In step A8, an image block is selected from each image sample of the remaining image samples except the first number of image samples from the multiple different image samples at a preset occlusion ratio, and the selected image block in each image sample is masked to obtain a remaining mask image.

[0049] In practical applications, the specific processing methods for determining the association relationship information between multiple image samples based on the first prediction results corresponding to multiple image samples in the above step S106 can be varied. An optional processing method is provided below, which may specifically include the following: inputting the first prediction results corresponding to multiple image samples into the target model to obtain the association relationship information between the multiple image samples, the target model is used to project the vector used to predict the category information in the first prediction results corresponding to the multiple image samples to the first dimension, and calculate the similarity between the first prediction results after different projections, and determine the association relationship information between the multiple image samples based on the calculated similarity, and the first dimension is less than a preset dimension threshold.

[0050] Among them, the target model can be constructed in a variety of ways, for example, the target model can be constructed by a neural network, or the target model can be constructed by a specified algorithm (such as a classification algorithm, a clustering algorithm, etc.), or the target model can also be constructed by a neural network in combination with a specified algorithm, etc., which can be set according to actual conditions, and this specification embodiment does not limit this. The preset dimension threshold can be set according to actual conditions, such as 5 or 4, etc.

[0051] In implementation, Figure 3As shown, according to actual conditions, a lightweight projection head (i.e., target model) can be added to the output end of the teacher model to calculate the relationship between multiple image samples. Specifically, considering that the first prediction result corresponding to each image sample will contain a vector for predicting category information, the dimension of the vector may be relatively high. At this time, the vector can be subjected to dimensionality reduction processing to project the vector to a lower dimension. In addition, in order to determine the association relationship between multiple image samples, the above determination processing can be completed in combination with the similarity. Based on this, the corresponding target model can be constructed through the above-mentioned processing or functions that need to be completed, and the target model can be trained using training samples to obtain a trained target model.

[0052] The first prediction results corresponding to the multiple image samples can be input into the target model, and the vector for predicting the category information (usually CLS Tokens, which can be located at a specified position in the first prediction result) can be extracted from the first prediction results through the target model. The vector for predicting the category information in the first prediction results corresponding to the multiple image samples can be projected into a latent space of a lower dimension (i.e., the first dimension) through the target model. Then, the similarity between the first prediction results after different projections can be calculated. Based on the calculated similarity, the association relationship information between the multiple image samples can be finally obtained through a specified algorithm (such as weighted calculation or fusion algorithm, etc.) or transformation relationship. Alternatively, the calculated similarity can be directly used as the association relationship information between the multiple image samples, etc., which can be set specifically according to actual conditions.

[0053] In practical applications, in addition to constructing the target model in the above manner, the target model can also be constructed in many other ways. The following is a model structure of a target model, as follows: the target model includes a linear layer, a ReLU nonlinear activation layer and a normalization layer.

[0054] In implementation, the first prediction results corresponding to multiple image samples can be input into the target model. Through the linear layer in the target model, the vectors used to predict category information (usually CLS Tokens) in the first prediction results corresponding to the multiple image samples can be projected into a latent space of a lower first dimension. Then, the similarity between the first prediction results after different projections can be calculated through the ReLU nonlinear activation layer and the normalization layer. Based on the calculated similarity, the association relationship information between the multiple image samples can be finally obtained through a specified algorithm or transformation relationship.

[0055] In practical applications, the specific processing methods for modulating the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples in the above step S106 to obtain the prediction coding information of the reconstructed image corresponding to each mask image can be various. The following is an optional processing method, such as Figure 4 As shown, the processing may specifically include the following steps S1062 to S1066.

[0056] In step S1062, the association relationship information between the multiple image samples and the position information of the position where the mask information corresponding to the mask image is located are fused with the second prediction result corresponding to each mask image to obtain a fused representation.

[0057] In implementation, Figure 5 As shown, the second prediction results corresponding to multiple mask images can be represented by aggregated information, that is, Figure 5 The small box on the upper left side of the figure can expand the second prediction results corresponding to the multiple mask images to the size of the mask image. Then, the association relationship information between the multiple image samples (specifically, it can be the embedded features of the association relationship information between the multiple image samples) and the position information of the location where the mask information corresponding to the mask image is located (specifically, it can be the embedded features of the position information) can be fused with the expanded second prediction results to obtain a fused representation.

[0058] In step S1064, the vector used for predicting the category information in the fusion representation is reorganized by using the association relationship information between the multiple image samples to obtain reorganized information used for predicting the category information.

[0059] In implementation, Figure 5 As shown, based on the association relationship information between multiple image samples, the Softmax function can be used to calculate and obtain the corresponding Softmax value. The vector used to predict the category information can be extracted from the above fusion representation, and the vector used to predict the category information in the fusion representation can be reorganized through the above Softmax value (i.e. Figure 5 ), and obtain the reorganized information for predicting category information.

[0060] In step S1066, the remaining vectors in the fusion representation except the vector used to predict the category information are aggregated and randomly shuffled according to a preset distribution rule to obtain the predicted coding information of the reconstructed image corresponding to each mask image.

[0061] In implementation, Figure 5As shown in the figure, the remaining vectors except the vector used to predict the category information in the fusion representation (i.e., the vectors corresponding to the remaining tokens Tokens except the CLS Tokens) are aggregated and then randomly shuffled according to the uniform distribution rule. The shuffled information is readjusted to the original dimension to obtain the predicted coding information of the reconstructed image corresponding to each mask image.

[0062] In practical applications, the specific processing methods of the above step S108 can be varied. An optional processing method is provided below, such as Figure 6 As shown, the processing may specifically include the following steps S1082 to S1086.

[0063] In step S1082, the prediction coding information of the reconstructed images corresponding to the multiple mask images is input into the decoder to obtain a third prediction result of the reconstructed image corresponding to each mask image.

[0064] In step S1084, based on the third prediction result of the reconstructed image corresponding to each mask image and the first prediction result corresponding to each image sample, corresponding loss information is obtained through a preset loss function.

[0065] Among them, the above loss function can include multiple types, such as mean square error loss function, cross entropy loss function, etc., which can be set according to actual conditions.

[0066] In step S1086, the model parameters of the student model and the decoder are adjusted based on the above loss information to train the student model until the above loss function converges to obtain a trained student model.

[0067] In practical applications, based on the situation of the above-mentioned target model, the processing of the above-mentioned step S1086 may also include: adjusting the model parameters of the student model, decoder and target model based on the above-mentioned loss information to jointly train the student model, decoder and target model until the loss function converges to obtain the trained student model.

[0068] In practical applications, the above loss function may include a Smooth L1 loss function, the number and width of the network layers included in the decoder are smaller than the number and width of the network layers included in the encoder corresponding to the decoder, and the encoder is used to generate prediction coding information of the reconstructed image corresponding to each mask image.

[0069] In implementation, an asymmetric encoder-decoder may be selected, that is, the number of network layers included in the decoder is smaller than the number of network layers included in the encoder, and the width of the network layers included in the decoder is smaller than the width of the network layers included in the encoder.

[0070] The embodiment of the present specification provides a model processing method, by obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked, then, inputting the multiple image samples into a teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each image sample, and inputting the multiple mask images into the student model, respectively obtaining a second prediction result corresponding to each mask image, and determining association relationship information between the multiple image samples based on the first prediction results corresponding to the multiple image samples, and modulating the second prediction result corresponding to each mask image based on the association relationship information between the multiple image samples to obtain a reconstructed image corresponding to each mask image. The student model is trained based on the prediction coding information of the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0071] The above is a processing method for a model provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides a processing device for a model, such as Figure 7 shown.

[0072] The processing device of the model includes: a sample acquisition module 701, a prediction module 702, a modulation module 703 and a model training module 704, wherein:

[0073] A sample acquisition module 701 acquires a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked;

[0074] The prediction module 702 inputs the plurality of image samples into the teacher model corresponding to the student model to obtain a first prediction result corresponding to each of the image samples, and inputs the plurality of mask images into the student model to obtain a second prediction result corresponding to each of the mask images;

[0075] The modulation module 703 determines the association relationship information between the plurality of image samples based on the first prediction results corresponding to the plurality of image samples, and performs modulation processing on the second prediction result corresponding to each mask image based on the association relationship information between the plurality of image samples to obtain prediction coding information of the reconstructed image corresponding to each mask image;

[0076] The model training module 704 trains the student model based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images to obtain a trained student model.

[0077] In an embodiment of the present specification, the multiple mask images include mask images selected based on a first preset ratio and remaining mask images, the mask images selected based on the first preset ratio are mask images obtained after mask processing is performed on the corresponding image samples in a completely occluded manner, and the remaining mask images are mask images obtained after mask processing is performed on the corresponding image samples in a preset occlusion ratio.

[0078] In the embodiment of this specification, the student model includes a module built based on Vision Transformer, and the device also includes:

[0079] A division module, dividing each of the image samples into a plurality of image blocks;

[0080] A selection module, which selects a first number of image samples from a plurality of different image samples based on the first preset ratio;

[0081] A first mask module selects an image block from each image sample in the first number of image samples, and performs mask processing on the unselected image blocks in each image sample to obtain a mask image selected at the first preset ratio;

[0082] The second mask module selects an image block at a preset occlusion ratio from each image sample of the remaining image samples except the first number of image samples from a plurality of different image samples, and performs mask processing on the selected image block in each image sample to obtain the remaining mask image.

[0083] In an embodiment of the present specification, the modulation module 703 inputs the first prediction results corresponding to the multiple image samples into the target model to obtain the association relationship information between the multiple image samples. The target model is used to project the vectors used to predict category information in the first prediction results corresponding to the multiple image samples to a first dimension, and calculate the similarity between the first prediction results after different projections, and determine the association relationship information between the multiple image samples based on the calculated similarity, and the first dimension is less than a preset dimension threshold.

[0084] In the embodiment of this specification, the target model includes a linear layer, a ReLU nonlinear activation layer and a normalization layer.

[0085] In the embodiment of this specification, the modulation module 703 includes:

[0086] A fusion unit, fusing the association relationship information between the plurality of image samples and the position information of the position where the mask information corresponding to the mask image is located with the second prediction result corresponding to each mask image to obtain a fusion representation;

[0087] a reorganization unit, which reorganizes the vector used for predicting the category information in the fusion representation through the association relationship information between the plurality of image samples to obtain reorganized information used for predicting the category information;

[0088] The processing unit aggregates the remaining vectors in the fusion representation except the vector used to predict the category information, and randomly shuffles them according to a preset distribution rule to obtain the prediction coding information of the reconstructed image corresponding to each of the mask images.

[0089] In the embodiment of this specification, the model training module 704 includes:

[0090] A decoding unit, inputting the prediction coding information of the reconstructed images corresponding to the plurality of mask images into a decoder, and obtaining a third prediction result of the reconstructed image corresponding to each mask image;

[0091] A loss determination unit, based on a third prediction result of a reconstructed image corresponding to each of the mask images and a first prediction result corresponding to each of the image samples, obtains corresponding loss information through a preset loss function;

[0092] A training unit adjusts the model parameters of the student model and the decoder based on the loss information to train the student model until the loss function converges, thereby obtaining a trained student model.

[0093] In an embodiment of the present specification, the loss function includes a Smooth L1 loss function, the number and width of the network layers included in the decoder are smaller than the number and width of the network layers included in the encoder corresponding to the decoder, and the encoder is used to generate prediction coding information of the reconstructed image corresponding to each mask image.

[0094] The embodiment of the present specification provides a model processing device, which obtains a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and the multiple mask images are input into the student model to obtain a second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, the association relationship information between the multiple image samples is determined, and the second prediction result corresponding to each mask image is modulated based on the association relationship information between the multiple image samples to obtain a reconstructed image corresponding to each mask image. The student model is trained based on the prediction coding information of the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0095] The above is a processing device for a model provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides a processing device for a model, such as Figure 8 shown.

[0096] The processing device of the model may provide a terminal device or a server, etc. for the above-mentioned embodiment.

[0097] The processing device of the model may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and one or more storage applications or data may be stored in the memory 802. Among them, the memory 802 may be a short-term storage or a persistent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the processing device of the model. Furthermore, the processor 801 may be configured to communicate with the memory 802, and execute a series of computer executable instructions in the memory 802 on the processing device of the model. The processing device of the model may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input and output interfaces 805, and one or more keyboards 806.

[0098] Specifically in this embodiment, the processing device of the model includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the processing device of the model, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following:

[0099] Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked;

[0100] Inputting a plurality of the image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting a plurality of the mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images;

[0101] Based on the first prediction results corresponding to the plurality of image samples, determining the association relationship information between the plurality of image samples, and performing modulation processing on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain prediction coding information of the reconstructed image corresponding to each of the mask images;

[0102] Based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, the student model is trained to obtain a trained student model.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the processing device embodiment of the model, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0104] The embodiment of the present specification provides a model processing device, which obtains a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and the multiple mask images are input into the student model to obtain a second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, the association relationship information between the multiple image samples is determined, and the second prediction result corresponding to each mask image is modulated based on the association relationship information between the multiple image samples to obtain a reconstructed image corresponding to each mask image. The student model is trained based on the prediction coding information of the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0105] Furthermore, based on the above Figures 1 to 6 In one embodiment, the present specification further provides a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0106] Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked;

[0107] Inputting a plurality of the image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting a plurality of the mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images;

[0108] Based on the first prediction results corresponding to the plurality of image samples, determining the association relationship information between the plurality of image samples, and performing modulation processing on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain prediction coding information of the reconstructed image corresponding to each of the mask images;

[0109] Based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, the student model is trained to obtain a trained student model.

[0110] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0111] An embodiment of the present specification provides a storage medium, which obtains a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and the multiple mask images are input into the student model to obtain a second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, the association relationship information between the multiple image samples is determined, and based on the association relationship information between the multiple image samples, the second prediction result corresponding to each mask image is modulated to obtain a reconstructed image corresponding to each mask image. The prediction coding information of the teacher model is then used, and then the student model is trained based on the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0112] Furthermore, based on the above Figures 1 to 6 In one or more embodiments of the present specification, a computer program product is provided, including a computer program. When the computer program in the computer program product is executed by a processor, the following process can be implemented:

[0113] Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked;

[0114] Inputting a plurality of the image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting a plurality of the mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images;

[0115] Based on the first prediction results corresponding to the plurality of image samples, determining the association relationship information between the plurality of image samples, and performing modulation processing on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain prediction coding information of the reconstructed image corresponding to each of the mask images;

[0116] Based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, the student model is trained to obtain a trained student model.

[0117] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0118] An embodiment of the present specification provides a computer program product, which obtains a first batch of image samples from multiple batches of image samples used to train a student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked. Then, the multiple image samples are input into a teacher model corresponding to the student model to obtain a first prediction result corresponding to each image sample, and the multiple mask images are input into the student model to obtain a second prediction result corresponding to each mask image. Based on the first prediction results corresponding to the multiple image samples, the association relationship information between the multiple image samples is determined, and the second prediction result corresponding to each mask image is modulated based on the association relationship information between the multiple image samples to obtain a reconstructed image corresponding to each mask image. The student model is trained based on the prediction coding information of the first prediction result corresponding to each image sample and the prediction coding information of the reconstructed image corresponding to each mask image. In this way, the correlation between the image samples in the same batch is calculated using the first prediction result input by the teacher model, and this is used as auxiliary information of the prediction result of the student model to reconstruct the features of the masked image samples (i.e., the mask image). Moreover, the second prediction result corresponding to each mask image is modulated based on the correlation relationship information between the multiple image samples, thereby promoting information interaction between the multiple image samples, and then improving the student model that needs to be trained's cognition of the relationship between image samples, so as to enhance the downstream generalization ability of the trained student model and improve the efficiency and effect of model training.

[0119] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0121] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0122] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0123] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0124] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0125] The embodiments of this specification are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable fraud case serial and parallel device to produce a machine, so that the instructions executed by the processor of the computer or other programmable fraud case serial and parallel device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable fraud case serial and parallel device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0133] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] The above description is only an embodiment of this specification and is not intended to limit this document. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this specification.

Claims

1. A method for processing a model, the method comprising: Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked; Inputting a plurality of the image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting a plurality of the mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images; Based on the first prediction results corresponding to the plurality of image samples, determining the association relationship information between the plurality of image samples, and performing modulation processing on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain prediction coding information of the reconstructed image corresponding to each of the mask images; Based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, the student model is trained to obtain a trained student model.

2. According to the method according to claim 1, the multiple mask images include mask images selected based on a first preset ratio and remaining mask images, the mask images selected based on the first preset ratio are mask images obtained after mask processing of completely occluding the corresponding image samples, and the remaining mask images are mask images obtained after mask processing of occluding the corresponding image samples at a preset occlusion ratio.

3. The method according to claim 2, wherein the student model includes a module built based on Vision Transformer, and the method further comprises: Dividing each of the image samples into a plurality of image blocks; Selecting a first number of image samples from a plurality of different image samples based on the first preset ratio; Selecting an image block from each image sample in the first number of image samples, and performing mask processing on the unselected image blocks in each image sample to obtain a mask image selected at the first preset ratio; From each image sample of the remaining image samples except the first number of image samples from a plurality of different image samples, an image block is selected at a preset occlusion ratio, and the selected image block in each image sample is masked to obtain the remaining mask image.

4. The method according to claim 1, wherein determining the association relationship information between the plurality of image samples based on the first prediction results corresponding to the plurality of image samples comprises: The first prediction results corresponding to the multiple image samples are input into the target model to obtain the association relationship information between the multiple image samples. The target model is used to project the vectors used to predict category information in the first prediction results corresponding to the multiple image samples to a first dimension, and calculate the similarity between the first prediction results after different projections, and determine the association relationship information between the multiple image samples based on the calculated similarity, and the first dimension is less than a preset dimension threshold.

5. According to the method of claim 4, the target model includes a linear layer, a ReLU nonlinear activation layer and a normalization layer.

6. The method according to claim 1, wherein the modulating process is performed on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples to obtain the prediction coding information of the reconstructed image corresponding to each of the mask images, comprising: fusing the association relationship information between the plurality of image samples and the position information of the position where the mask information corresponding to the mask image is located with the second prediction result corresponding to each mask image to obtain a fused representation; Recombining the vector used for predicting the category information in the fusion representation through the association relationship information between the plurality of image samples to obtain recombined information used for predicting the category information; The remaining vectors in the fusion representation except the vector used to predict the category information are aggregated and randomly shuffled according to a preset distribution rule to obtain the predicted coding information of the reconstructed image corresponding to each of the mask images.

7. The method according to claim 1 or 6, wherein the student model is trained based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images to obtain the trained student model, comprising: Inputting the prediction coding information of the reconstructed images corresponding to the plurality of mask images into a decoder to obtain a third prediction result of the reconstructed image corresponding to each mask image; Based on the third prediction result of the reconstructed image corresponding to each of the mask images and the first prediction result corresponding to each of the image samples, obtaining corresponding loss information through a preset loss function; The model parameters of the student model and the decoder are adjusted based on the loss information to train the student model until the loss function converges, thereby obtaining a trained student model.

8. According to the method of claim 7, the loss function includes a Smooth L1 loss function, the number and width of the network layers included in the decoder are smaller than the number and width of the network layers included in the encoder corresponding to the decoder, and the encoder is used to generate predictive coding information of the reconstructed image corresponding to each of the mask images.

9. A model processing device, the device comprising: A sample acquisition module, which acquires a first batch of image samples from multiple batches of image samples used to train the student model, wherein the first batch of image samples includes multiple different image samples and mask images after the image samples are masked; A prediction module, inputting the plurality of image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting the plurality of mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images; A modulation module, based on the first prediction results corresponding to the plurality of image samples, determines the association relationship information between the plurality of image samples, and performs modulation processing on the second prediction result corresponding to each mask image based on the association relationship information between the plurality of image samples, so as to obtain prediction coding information of the reconstructed image corresponding to each mask image; The model training module trains the student model based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images to obtain a trained student model.

10. A model processing device, the model processing device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Obtaining a first batch of image samples from multiple batches of image samples used to train a student model, the first batch of image samples including multiple different image samples and mask images after the image samples are masked; Inputting a plurality of the image samples into the teacher model corresponding to the student model, respectively obtaining a first prediction result corresponding to each of the image samples, and inputting a plurality of the mask images into the student model, respectively obtaining a second prediction result corresponding to each of the mask images; Based on the first prediction results corresponding to the plurality of image samples, determining the association relationship information between the plurality of image samples, and performing modulation processing on the second prediction result corresponding to each of the mask images based on the association relationship information between the plurality of image samples, to obtain prediction coding information of the reconstructed image corresponding to each of the mask images; Based on the first prediction result corresponding to each of the image samples and the prediction coding information of the reconstructed image corresponding to each of the mask images, the student model is trained to obtain a trained student model.

Citation Information

Patent Citations

  • Text supervision semantic segmentation algorithm based on multi-view consistency

    CN115797639A

  • Prediction model training method and device, equipment, medium and program product

    CN117217368A

  • Method and device for spatial modeling of image class relationship based on prototype network

    CN118447340A

  • Teaching student network for end-to-end semi-supervised object detection

    US20230281980A1