Model training method and device based on difficult sample mining and storage medium
By mining hard samples from image patches and iteratively updating the masking strategy, the dependence of self-supervised masking learning on manually predefined strategies is resolved, thereby improving the model's representation ability and accuracy in the field of image perception.
Patent Information
- Application Number
- CN202310370230.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-07
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-04-07
AI Technical Summary
The performance of self-supervised mask learning heavily relies on manually predefined masking strategies, which affects the accuracy in the field of image perception, especially for the detection and classification of difficult-to-identify targets or categories.
By using a hard sample mining method, the input image is mined in units of image patches to generate a target mask strategy. The mask strategy is iteratively updated to avoid manual predefinition and improve the transferability of the model's feature representation.
It improves the representational ability of pre-trained models, reduces reliance on manually predefined masking strategies, and enhances the detection and classification accuracy in the field of image perception.
Smart Images
Figure CN116704217B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training method based on difficult sample mining, a device and a storage medium. BACKGROUND
[0002] Self-supervised learning is a research hotspot in the field of computer vision in recent years, which intends to train a pre-training model with strong representation extraction capability without relying on any manual annotation. Mask learning, as a self-supervised learning method, can learn more transferable feature representation, and has attracted strong attention from researchers due to its strong representation learning ability.
[0003] However, current research shows that the performance of self-supervised mask learning is strongly dependent on the manually predefined mask strategy. Therefore, the manually defined mask strategy directly affects the representation ability of the self-supervised mask learning pre-training model, and further affects the performance indicators such as accuracy rate in the image perception field to which it is applied, such as image classification, target detection and semantic segmentation. For example, the greater the accuracy rate of the larger volume class in the same image is correctly classified, the greater the accuracy rate of the target with clearer boundary or greater difference between individuals is successfully detected, and conversely, the smaller the accuracy rate of the target or class that is difficult to identify is successfully detected or correctly classified.
[0004] Therefore, how to avoid the dependence on the manually predefined mask strategy when using self-supervised mask learning to train the model in the image perception field has become a technical problem to be solved in the field. SUMMARY
[0005] In view of the technical problems in the prior art, the present application provides a model training method based on difficult sample mining, a device and a storage medium.
[0006] In a first aspect, the present application provides a model training method based on difficult sample mining, comprising:
[0007] obtaining model parameters obtained by training a first model based on a first image in a current iteration process; the first model is used to reconstruct each image block in the input first image and determine the predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in an original image;
[0008] updating the model parameters of a second model based on the model parameters of the first model;
[0009] determining the predicted reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameters;
[0010] generate a target mask strategy for performing a mask operation on each image block in the first image based on the prediction reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement;
[0011] update the first image based on the target mask strategy for training the first model in a next iteration process until a loss function of the first model reaches convergence or a number of iterations reaches a preset total number of iterations.
[0012] Optionally, before the obtaining the model parameter based on the first image for training the first model in the current iteration process, the method comprises:
[0013] obtain a first feature vector corresponding to each image block in the first image based on a feature extraction network of the first model;
[0014] determine the first image after reconstruction based on an image reconstruction network of the first model and the first feature vector;
[0015] determine a prediction reconstruction loss of the first image based on a loss prediction network of the first model and the first feature vector;
[0016] determine a loss function of the first model based on the first image, the first image after reconstruction and the prediction reconstruction loss of the first image;
[0017] train the image reconstruction network and the loss prediction network by using a back propagation and gradient descent algorithm.
[0018] Optionally, the determining the loss function of the first model based on the first image, the first image after reconstruction and the prediction reconstruction loss of the first image comprises:
[0019] determine a reconstruction loss function of the first image based on the first image after reconstruction and an original image corresponding to the first image;
[0020] determine the loss function of the first model based on the reconstruction loss function of the first image and the prediction reconstruction loss of the first image.
[0021] Optionally, the determining the prediction reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameter comprises:
[0022] obtain a second feature vector corresponding to each image block in the original image based on a feature extraction network of the second model;
[0023] determine, based on a loss prediction network of the second model after updating model parameters and the second feature vector, a predicted reconstruction loss of each image block in the original image.
[0024] Optionally, the target mask strategy for performing the mask operation on each image block in the first image is generated based on the predicted reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement, and the target mask strategy comprises:
[0025] In the predicted reconstruction loss corresponding to each image block in the original image, the predicted reconstruction loss meeting the preset difficult sample requirement is selected as a target predicted reconstruction loss; the preset difficult sample requirement is that the predicted reconstruction loss is greater than or equal to a preset threshold or the predicted reconstruction loss meets the top N% after sorting by size; the N is a positive number.
[0026] Based on the correspondence between each image block in the original image and the predicted reconstruction loss, the image block corresponding to the target predicted reconstruction loss is determined as the image block of the difficult sample.
[0027] Based on the random mask information and the mask information corresponding to the image block of the difficult sample, a target mask strategy for performing the mask operation on each image block in the first image is determined by using a weighted summation method.
[0028] Optionally, the model parameters of the second model are updated based on the model parameters of the first model, and the updating comprises:
[0029] Based on the parameters of the feature extraction network and the parameters of the loss prediction network obtained by training the first model in the current iteration number, the parameters of the feature extraction network and the parameters of the loss prediction network included in the second model are updated by using an exponential smoothing method.
[0030] Optionally, the loss function of the first model is determined based on a binary cross-entropy algorithm or a mean square error algorithm.
[0031] In a second aspect, the present application further provides a model training device based on difficult sample mining, comprising:
[0032] The acquisition module is configured to acquire model parameters obtained by training a first model based on a first image in a current iteration process; the first model is configured to reconstruct each image block in an input first image and determine a predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in an original image.
[0033] The updating module is configured to update model parameters of a second model based on model parameters of the first model.
[0034] determining, based on the second model after updating the model parameters, a prediction reconstruction loss corresponding to each image block in the original image;
[0035] generating, based on the prediction reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement, a target mask strategy for performing a mask operation on each image block in the first image;
[0036] updating, based on the target mask strategy, the first image for training the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iterations reaches a preset total number of iterations.
[0037] In a third aspect, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the model training method based on difficult sample mining according to the first aspect when executing the program.
[0038] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the model training method based on difficult sample mining according to the first aspect.
[0039] In a fifth aspect, the present application also provides a computer program product, which comprises a computer program, and the computer program is executable on a processor to implement the model training method based on difficult sample mining according to the first aspect.
[0040] The model training method, device and storage medium based on difficult sample mining provided by the present application can help the model to learn more transferable feature representation, improve the representation ability of the pre-training model, and thus avoid the dependence of the pre-training model of the self-supervised mask learning on the artificial predefined mask rule. BRIEF DESCRIPTION OF DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.
[0042] Figure 1is a flowchart of a model training method based on difficult sample mining provided by an embodiment of the present application.
[0043] Figure 2 is a schematic diagram of a specific implementation process of the model training method based on difficult sample mining provided by an embodiment of the present application.
[0044] Figure 3 is a structural schematic diagram of the model training device based on difficult sample mining provided by an embodiment of the present application.
[0045] Figure 4 is a schematic diagram of the physical structure of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0046] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.
[0047] In the embodiments of the present application, the term "a plurality of" means two or more, and other quantifiers are similar.
[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0049] Figure 1 is a flowchart of a model training method based on difficult sample mining provided by an embodiment of the present application, as Figure 1 shown, the method comprises:
[0050] Step 101, obtaining model parameters obtained by training a first model based on a first image in a current iteration process; the first model is used for reconstructing each image block in the input first image and determining the predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in an original image;
[0051] Specifically, the original image is segmented according to a preset pixel unit to obtain a plurality of image blocks corresponding to the original image, each image block has a corresponding position in the original image, a mask is added to one or more image blocks in the original image to obtain a first image, that is, an original image to which the mask is added, the first image is input into a first model, the first model is used to reconstruct each image block in the first image and further determine a predicted reconstruction loss corresponding to each image block. The predicted reconstruction loss refers to a loss determined by a predicted reconstruction image and a corresponding original image. After the first image is input into the first model, a reconstruction image of each image block in the first image is output, and a predicted reconstruction loss corresponding to each image block in the first image is output. The reconstruction results corresponding to each image block of the first image constitute a final reconstruction image corresponding to the first image.
[0052] Further, a loss function corresponding to the first model can be determined according to the reconstruction results of each image block in the first image and / or the predicted reconstruction loss corresponding to each image block in the first image.
[0053] In the current iteration process, the model parameters obtained after the training of the first model based on the first image are acquired.
[0054] In the current iteration process, the model parameters obtained after the training of the first model based on the first image are acquired.
[0055] Step 102, updating the model parameters of the second model based on the model parameters of the first model;
[0056] Step 103, determining a predicted reconstruction loss corresponding to each image block in the original image based on the second model after the model parameters are updated;
[0057] The model parameters of the first model are used to update the model parameters of the second model, so that the second model can also determine a predicted reconstruction loss corresponding to an input image based on the input image. Of course, the second model can also include more functions, but the parameters of part of the functions in the second model are updated by the model parameters of the first model.
[0058] The original image is input into the second model after the model parameters are updated, at this time, the second model has the model parameters obtained in the current iteration process, so that the second model can also predict a predicted reconstruction loss corresponding to each image block in the input original image, that is, determine the value of the predicted reconstruction loss corresponding to each image block in the original image. The greater the value of the predicted reconstruction loss is, the more difficult the image block is to reconstruct. Conversely, the smaller the value of the predicted reconstruction loss is, the less difficult the image block is to reconstruct. If it is necessary to determine a difficult sample, the image block with a larger value of the predicted reconstruction loss is preferentially selected.
[0059] Step 104, based on the prediction reconstruction loss corresponding to each image block in the original image and the preset difficult sample requirement, a target mask strategy for performing a mask operation on each image block in the first image is generated;
[0060] Step 105, based on the target mask strategy, the first image is updated for training the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iteration processes reaches the preset total number of iterations.
[0061] Based on the second model after updating the model parameters, after determining the prediction reconstruction loss corresponding to each image block in the input original image, according to the preset difficult sample requirement, it is determined which image blocks with different prediction reconstruction loss values need to be masked, that is, the target mask strategy for performing a mask operation on each image block in the first image is determined;
[0062] Then, according to the target mask strategy, a mask operation is performed on the first image to obtain an updated first image, and the updated first image is input into the above-mentioned first model to perform the training process of the first model, that is, to perform the next iteration process. Through the above-mentioned iteration process, the first model can be improved to autonomously construct difficult problems, and the corresponding difficult samples can be mined.
[0063] The model training method based on difficult sample mining provided by the application mines difficult samples for input images in units of image blocks, and iteratively updates the mask strategy for input images based on the mask information corresponding to the mined difficult samples, without manually setting the mask strategy. In the iterative training process, the mask strategy is constantly updated, which can help the model to learn more transferable feature representations and improve the representation ability of the pre-training model, thereby avoiding the dependence of the pre-training model of the self-supervised mask learning on the artificial predefined mask rule.
[0064] Optionally, before obtaining the model parameters obtained by training the first model based on the first image in the current iteration process, the method comprises:
[0065] Based on the feature extraction network of the first model, a first feature vector corresponding to each image block in the first image is obtained;
[0066] Based on the image reconstruction network of the first model and the first feature vector, the first image after reconstruction is determined;
[0067] Based on the loss prediction network of the first model and the first feature vector, the prediction reconstruction loss of the first image is determined;
[0068] determine a loss function of the first model based on the first image, the reconstructed first image and a predicted reconstruction loss of the first image;
[0069] train the image reconstruction network and the loss prediction network by using a back propagation and a gradient descent algorithm.
[0070] Specifically, before obtaining the model parameters of the first model based on the first image in the current iteration process, the first model and the loss function corresponding to the first model need to be constructed, and the first model needs to be trained, which specifically includes:
[0071] construct the first model composed of a feature extraction network, an image reconstruction network and a loss prediction network, wherein the feature extraction network is used to extract a first feature vector corresponding to each image block in the first image, and the first feature vector can represent character features, correlations between pixels and other features corresponding to each image block, encoding features of each unit constituting the image block, etc.
[0072] The image reconstruction network is used to reconstruct the first image based on the extracted first feature vector, and determine the reconstructed first image, that is, the reconstructed first image determined after reconstructing one or more image blocks with added masks in the first image.
[0073] The loss prediction network is used to predict the predicted reconstruction loss of the first image based on the extracted first feature vector, wherein the predicted reconstruction loss is determined based on the predicted reconstructed image and the original image corresponding to the input image.
[0074] The image reconstruction network and the loss prediction network can be implemented by corresponding decoders.
[0075] determine the loss function of the first model based on the first image, the reconstructed first image and the predicted reconstruction loss of the first image. The loss function of the first model specifically includes the loss function corresponding to the image reconstruction network and the loss function corresponding to the loss prediction network.
[0076] train the image reconstruction network and the loss prediction network by using a back propagation and a gradient descent algorithm.
[0077] Optionally, the determination of the loss function of the first model based on the first image, the reconstructed first image and the predicted reconstruction loss of the first image includes:
[0078] determine the reconstruction loss function of the first image based on the reconstructed first image and the original image corresponding to the first image;
[0079] determine the loss function of the first model based on the reconstruction loss function of the first image and the predicted reconstruction loss of the first image.
[0080] Specifically, in the first model, the reconstruction loss function of the first image is determined based on the reconstructed first image and the original image corresponding to the first image, and a common loss function determination algorithm, such as a binary cross-entropy algorithm or a mean square error algorithm.
[0081] The loss function of the first model is determined based on the reconstruction loss function of the first image and the predicted reconstruction loss of the first image using a common loss function determination algorithm, such as a binary cross-entropy algorithm or a mean square error algorithm.
[0082] When the binary cross-entropy algorithm is used to determine the loss function of the first model, the corresponding formula can be represented as:
[0083]
[0084]
[0085]
[0086] where L2 represents the loss function of the first model, represents the predicted reconstruction loss corresponding to the i-th image block of the first image, represents the predicted reconstruction loss corresponding to the j-th image block of the first image, N represents the total number of image blocks included in the first image, and σ(·) represents a sigmoid function, i.e. and respectively represent whether the reconstruction loss of the i-th image block of the first image is greater than the reconstruction loss of the j-th image block, if L1(i)>L1(j), then and if L1(i)<L1(j), then and L1(i) represents the reconstruction loss function corresponding to the i-th image block of the first image, and L1(j) represents the reconstruction loss function corresponding to the j-th image block of the first image.
[0087] When the mean square error algorithm is used, taking the square loss as an example, the loss function of the first model is determined, and the corresponding formula can be represented as:
[0088] L2(X)=(L1(X)-g(X)) 2;
[0089] L1(X) = (X - f(X)) 2 ;
[0090] Wherein, L1(X) represents the loss function corresponding to the image reconstruction network of the first model, that is, the real image reconstruction loss, X represents the pixel value of the original image; f(X) represents the pixel value of the reconstructed image; L2(X) represents the loss function corresponding to the loss prediction network, and g(X) represents the predicted reconstruction loss of the first image. Here, the image reconstruction network of the first model can be trained first, and then the loss prediction network of the first model is trained, or the image reconstruction network and the loss prediction network of the first model can be jointly trained.
[0091] Optionally, the second model based on the updated model parameters determines the predicted reconstruction loss corresponding to each image block in the original image, comprising:
[0092] Based on the feature extraction network of the second model, the second feature vector corresponding to each image block in the original image is obtained;
[0093] Based on the loss prediction network of the second model after updating the model parameters and the second feature vector, the predicted reconstruction loss of each image block in the original image is determined.
[0094] Specifically, after the second model obtains the model parameters obtained by training the first model in the current iteration process, the second model can also extract features from the input image and determine the predicted reconstruction loss corresponding to the input image.
[0095] And the image input into the second model is the original image, based on the feature extraction network in the second model, the second feature vector corresponding to the original image is extracted, and the second feature vector is input into the loss prediction network in the second model, to determine the predicted reconstruction loss corresponding to all image blocks constituting the original image, and then determine which image block corresponds to a larger predicted reconstruction loss value, that is, the reconstruction difficulty of the image block is larger, and the image block is preferentially selected to set the corresponding mask to construct the difficult sample.
[0096] Wherein, each image block of the original image and the first image is one-to-one corresponding and has the same position information, and it is assumed that each image block of the original image P is represented as p i , and t represents the total number of image blocks included in the original image; the image block of the first image P c is represented as p ci , and Then p i and p ciThe same image block is represented. The specific rule for dividing the image into image blocks can be to construct an image block by a plurality of pixel units, or to determine an image block according to the size, for example, the width and height are w and h respectively, and the specific units of width and height are set according to the size of the image.
[0097] Optionally, the target mask strategy for performing the mask operation on each image block in the first image is generated based on the prediction reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement, comprising:
[0098] In the prediction reconstruction loss corresponding to each image block in the original image, the prediction reconstruction loss satisfying the preset difficult sample requirement is selected as the target prediction reconstruction loss; the preset difficult sample requirement is that the prediction reconstruction loss is greater than or equal to a preset threshold, or the prediction reconstruction loss satisfies the top N% in the size order; the N is a positive number;
[0099] Based on the correspondence between each image block in the original image and the prediction reconstruction loss, the image block corresponding to the target prediction reconstruction loss is determined as the image block of the difficult sample;
[0100] Based on the random mask information and the mask information corresponding to the image block of the difficult sample, a weighted sum is used to determine the target mask strategy for performing the mask operation on each image block in the first image.
[0101] Specifically, based on the second model after updating the model parameters, the second feature vector of the original image is extracted, and after determining the prediction reconstruction loss corresponding to each image block of the original image, the part satisfying the preset difficult sample requirement is selected according to the preset difficult sample requirement. Here, the preset difficult sample requirement can be a preset threshold, as long as the prediction reconstruction loss corresponding to an image block is greater than or equal to the preset threshold, the image block is selected out, and then the corresponding mask information is set according to the selected image block and the mask is set to the randomly selected image block in the original image. A weighted sum is used to determine the target mask strategy for performing the mask operation on each image block in the first image. For example, the selected image block is set in the full mask mode, the randomly selected image block is set in the partial mask mode, or the selected image block is set in the partial mask mode, and the randomly selected image block is set in the full mask mode. It can also be a combination of other mask modes. In addition, the above-mentioned preset difficult sample requirement can also be to sort the prediction reconstruction loss from large to small, and to select the top N% of the prediction reconstruction loss in the sorting result. The specific implementation mode and the preset threshold are similar, which will not be described here.
[0102] Optionally, the model parameters of the second model are updated based on the model parameters of the first model, comprising:
[0103] Based on the parameters of the feature extraction network and the parameters of the loss prediction network obtained by training the first model in the current iteration number, the parameters of the feature extraction network and the parameters of the loss prediction network included in the second model are updated in an exponential smoothing manner.
[0104] Specifically, the parameters of the feature extraction network and the parameters of the loss prediction network obtained by training the first model in the current iteration number are updated in an exponential smoothing manner to update the parameters of the feature extraction network and the parameters of the loss prediction network included in the second model. In this way, the second model can extract the second feature vector corresponding to each image block in the original image based on the updated feature extraction network, and determine the predicted reconstruction loss of each image block in the original image based on the updated loss prediction network and the second feature vector.
[0105] The model training method based on difficult sample mining provided by the present application mines difficult samples in units of image blocks for input images, and iteratively updates the mask strategy for input images based on the mask information corresponding to the mined difficult samples, without manually setting the mask strategy. In the iterative training process, the mask strategy is constantly updated, which can help the model learn more transferable feature representations and improve the representation ability of the pre-trained model, thereby avoiding the dependence of the pre-trained model of the self-supervised mask learning on the artificial predefined mask rule.
[0106] In order to more clearly illustrate the model training method based on difficult sample mining provided by the present application, specific examples are used for illustration.
[0107] Figure 2 is a specific implementation flowchart of the model training method based on difficult sample mining provided by the embodiment of the present application, as shown in Figure 2 , including:
[0108] In the initial iteration process, a first image used to train the first model is determined by a random mask and an original image;
[0109] The first image is divided into a plurality of image blocks corresponding to the first image by a preset pixel unit or a preset size;
[0110] The first feature vector corresponding to each image block in the first image is obtained by the feature extraction network in the first model;
[0111] The first feature vector is input into the image reconstruction network and the loss prediction network in the first model, respectively, to output the reconstructed first image and the predicted reconstruction loss corresponding to each image block in the first image;
[0112] According to the first image and the reconstructed first image, a loss function corresponding to the image reconstruction network of the first model is determined, and the image reconstruction network is trained by using a back propagation and gradient descent algorithm;
[0113] The loss function corresponding to the image reconstruction network of the first model can be expressed as:
[0114] L1(X)=(X-f(X)1 2 ; wherein L1(X) represents the loss function corresponding to the image reconstruction network of the first model, i.e. the real image reconstruction loss, and X represents the pixel value of the original image; f(X) represents the pixel value of the reconstructed image;
[0115] According to the predicted reconstruction loss corresponding to each image block in the first image and the real image reconstruction loss, a loss function corresponding to the loss prediction network of the first model is determined, and the loss prediction network is trained by using a back propagation and gradient descent algorithm;
[0116] The loss function corresponding to the loss prediction network of the first model can be expressed as:
[0117] L2(X)=(L1(X)-g(X)) 2 ; wherein L2(X) represents the loss function corresponding to the loss prediction network, L1(X) represents the loss function corresponding to the image reconstruction network of the first model, i.e. the real image reconstruction loss, and g(X) represents the predicted reconstruction loss of the first image.
[0118] The model parameters of the first model are updated to the second model by exponential smoothing in the current iteration process, and the parameters of the second model include the parameters of the feature extraction network and the parameters of the loss prediction network.
[0119] The second model with updated model parameters can determine the second feature vector corresponding to each image block of the original image and the predicted reconstruction loss corresponding to each image block of the original image based on the input original image.
[0120] According to the preset difficulty sample requirement, the predicted reconstruction loss greater than the preset threshold or the top N% after sorting is selected; the corresponding image block is selected as the image block of the difficult sample according to the correspondence between each image block and the predicted reconstruction loss.
[0121] According to the random mask and the mask information of the difficult sample, a target mask strategy for performing a mask operation on the first image is determined, and the first image is updated.
[0122] input the updated first image into the first model, start the next iteration process, train the first model until the loss function of the first model reaches convergence or the number of iteration processes reaches the preset total number of iterations.
[0123] Figure 3 is a structural schematic diagram of a model training device based on difficult sample mining provided by an embodiment of the present application, as shown in the figure, the device comprises: Figure 3
[0124] The acquisition module 301 is configured to acquire model parameters of the first model obtained by training the first model based on the first image in the current iteration process; the first model is configured to reconstruct each image block in the input first image and determine the prediction reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in an original image;
[0125] The update module 302 is configured to update the model parameters of the second model based on the model parameters of the first model.
[0126] The determination module 303 is configured to determine the prediction reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameters.
[0127] The mask module 304 is configured to generate a target mask strategy for performing a mask operation on each image block in the first image based on the prediction reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement.
[0128] The iteration module 305 is configured to update the first image based on the target mask strategy, train the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iteration processes reaches the preset total number of iterations.
[0129] Optionally, the device comprises a preprocessing module configured to:
[0130] acquire the first feature vector corresponding to each image block in the first image based on the feature extraction network of the first model;
[0131] determine the reconstructed first image based on the image reconstruction network of the first model and the first feature vector;
[0132] determine the prediction reconstruction loss of the first image based on the loss prediction network of the first model and the first feature vector;
[0133] determine the loss function of the first model based on the first image, the reconstructed first image and the prediction reconstruction loss of the first image.
[0134] The image reconstruction network and the loss prediction network are trained using a back propagation and gradient descent algorithm.
[0135] Optionally, in the process of determining the loss function of the first model based on the first image, the reconstructed first image, and the predicted reconstruction loss of the first image, the preprocessing module is specifically configured to:
[0136] determine a reconstruction loss function of the first image based on the reconstructed first image and an original image corresponding to the first image;
[0137] determine the loss function of the first model based on the reconstruction loss function of the first image and the predicted reconstruction loss of the first image.
[0138] Optionally, in the process of determining the predicted reconstruction loss of each image block in the original image based on the second model with updated model parameters, the determining module 303 is specifically configured to:
[0139] obtain a second feature vector corresponding to each image block in the original image based on a feature extraction network of the second model;
[0140] determine the predicted reconstruction loss of each image block in the original image based on a loss prediction network of the second model with updated model parameters and the second feature vector.
[0141] Optionally, in the process of generating a target mask strategy for performing a mask operation on each image block in the first image based on the predicted reconstruction loss of each image block in the original image and a preset difficult sample requirement, the mask module 304 is specifically configured to:
[0142] screen, from the predicted reconstruction loss of each image block in the original image, a target predicted reconstruction loss that meets the preset difficult sample requirement as the target predicted reconstruction loss; the preset difficult sample requirement is that the predicted reconstruction loss is greater than or equal to a preset threshold or the predicted reconstruction loss meets the top N% after sorting by size; the N is a positive number;
[0143] determine, based on a correspondence between each image block in the original image and the predicted reconstruction loss, an image block corresponding to the target predicted reconstruction loss as an image block of a difficult sample;
[0144] determine, based on random mask information and mask information corresponding to the image block of the difficult sample, a target mask strategy for performing a mask operation on each image block in the first image in a weighted summation manner.
[0145] Optionally, in the process of updating the model parameters of the second model based on the model parameters of the first model, the updating module 302 is specifically configured to:
[0146] Based on the parameters of the feature extraction network and the parameters of the loss prediction network obtained by training the first model in the current iteration number, the parameters of the feature extraction network and the parameters of the loss prediction network included in the second model are updated in an exponential smoothing manner.
[0147] Optionally, the loss function of the first model is determined based on a binary cross-entropy algorithm or a mean square error algorithm.
[0148] The model training device based on difficult sample mining provided by the embodiment of the application can execute the technical solutions of the model training method based on difficult sample mining in any of the above embodiments, and the implementation principles and beneficial effects thereof are similar to those of the model training method based on difficult sample mining. For details, refer to the implementation principles and beneficial effects of the model training method based on difficult sample mining, which will not be described here.
[0149] Figure 4 is the physical structure schematic diagram of the electronic device provided by the embodiment of the application, as Figure 4 shown, the electronic device can include a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420 and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke the logical instructions in the memory 430 to execute the model training method based on difficult sample mining, which includes:
[0150] Obtaining model parameters obtained by training the first model based on the first image in the current iteration process; the first model is used to reconstruct each image block in the input first image and determine the predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in an original image;
[0151] Updating the model parameters of the second model based on the model parameters of the first model;
[0152] Determining the predicted reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameters;
[0153] Based on the predicted reconstruction loss corresponding to each image block in the original image and the preset difficult sample requirement, generating a target mask strategy for performing a mask operation on each image block in the first image;
[0154] Based on the target mask strategy, the first image is updated for training the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iterations reaches the preset total number of iterations.
[0155] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0156] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the model training method based on difficult sample mining provided by the above-mentioned methods, which comprises:
[0157] Obtaining model parameters of the first model obtained by training the first model based on the first image in the current iteration process; the first model is used to reconstruct each image block in the input first image and determine the predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in the original image;
[0158] Updating the model parameters of the second model based on the model parameters of the first model;
[0159] Determining the predicted reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameters;
[0160] Generating a target mask strategy for performing a mask operation on each image block in the first image based on the predicted reconstruction loss corresponding to each image block in the original image and the preset difficult sample requirement;
[0161] Based on the target mask strategy, the first image is updated for training the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iteration processes reaches a preset total number of iterations.
[0162] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the model training method based on difficult sample mining provided by the above-mentioned methods, the method comprising:
[0163] obtaining model parameters of the first model obtained by training the first model based on the first image in the current iteration process; the first model is used to reconstruct each image block in the input first image and determine the predicted reconstruction loss corresponding to each image block in the first image; the first image is an image obtained by adding a mask to one or more image blocks in the original image;
[0164] updating the model parameters of the second model based on the model parameters of the first model;
[0165] determining the predicted reconstruction loss corresponding to each image block in the original image based on the second model after updating the model parameters;
[0166] generating a target mask strategy for performing a mask operation on each image block in the first image based on the predicted reconstruction loss corresponding to each image block in the original image and a preset difficult sample requirement;
[0167] updating the first image based on the target mask strategy for training the first model in the next iteration process until the loss function of the first model reaches convergence or the number of iteration processes reaches a preset total number of iterations.
[0168] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement it without creative labor.
[0169] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0170] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A model training method based on hard sample mining, characterized in that, include: Obtain the model parameters obtained by training the first model based on the first image during the current iteration; The first model is used to reconstruct each image patch in the input first image and determine the prediction reconstruction loss corresponding to each image patch in the first image; the first image is an image after adding a mask to one or more image patches in the original image; Update the model parameters of the second model based on the model parameters of the first model; Based on the second model with updated model parameters, the prediction reconstruction loss corresponding to each image patch in the original image is determined; Based on the prediction and reconstruction loss corresponding to each image patch in the original image and the preset hard sample requirements, a target masking strategy is generated for performing masking operations on each image patch in the first image. Based on the target masking strategy, the first image is updated for use in the next iteration to train the first model until the loss function of the first model converges or the number of iterations reaches the preset total number of iterations.
2. The model training method based on hard sample mining according to claim 1, characterized in that, Before obtaining the model parameters obtained by training the first model based on the first image during the current iteration, the process includes: Based on the feature extraction network of the first model, the first feature vector corresponding to each image block in the first image is obtained; Based on the image reconstruction network of the first model and the first feature vector, the reconstructed first image is determined; Based on the loss prediction network of the first model and the first feature vector, the prediction reconstruction loss of the first image is determined. Based on the first image, the reconstructed first image, and the predicted reconstruction loss of the first image, the loss function of the first model is determined. The image reconstruction network and the loss prediction network are trained using backpropagation and gradient descent algorithms.
3. The model training method based on hard sample mining according to claim 2, characterized in that, The step of determining the loss function of the first model based on the first image, the reconstructed first image, and the prediction and reconstruction loss of the first image includes: Based on the reconstructed first image and the original image corresponding to the first image, the reconstruction loss function of the first image is determined; The loss function of the first model is determined based on the reconstruction loss function of the first image and the prediction reconstruction loss of the first image.
4. The model training method based on hard sample mining according to claim 3, characterized in that, The second model, based on the updated model parameters, determines the prediction and reconstruction loss corresponding to each image patch in the original image, including: Based on the feature extraction network of the second model, the second feature vector corresponding to each image block in the original image is obtained; Based on the loss prediction network of the second model after updating the model parameters, and the second feature vector, the prediction reconstruction loss of each image patch in the original image is determined.
5. The model training method based on hard sample mining according to claim 4, characterized in that, The step of generating a target masking strategy for performing masking operations on each image patch in the first image, based on the prediction and reconstruction loss corresponding to each image patch in the original image and the preset hard sample requirements, includes: In the original image, the prediction reconstruction loss corresponding to each image patch is selected to meet the preset difficult sample requirement and is used as the target prediction reconstruction loss; the preset difficult sample requirement is that the prediction reconstruction loss is greater than or equal to a preset threshold, or the prediction reconstruction loss is among the top N% after being sorted by size; where N is a positive number. Based on the correspondence between each image patch in the original image and the prediction reconstruction loss, the image patch corresponding to the target prediction reconstruction loss is determined as the image patch of the hard sample; Based on the random mask information and the mask information corresponding to the image blocks that are considered as difficult samples, a weighted summation method is used to determine the target masking strategy for performing masking operations on each image block in the first image.
6. The model training method based on hard sample mining according to claim 5, characterized in that, The step of updating the model parameters of the second model based on the model parameters of the first model includes: Based on the parameters of the feature extraction network and the loss prediction network obtained from training the first model in the current iteration, the parameters of the feature extraction network and the loss prediction network included in the second model are updated using an exponential smoothing method.
7. The model training method based on hard sample mining according to any one of claims 1 to 3, characterized in that, The loss function of the first model is determined based on the binary cross-entropy algorithm or the mean squared error algorithm.
8. A model training device based on hard sample mining, characterized in that, include: The acquisition module is used to acquire the model parameters obtained by training the first model based on the first image during the current iteration. The first model is used to reconstruct each image patch in the input first image and determine the prediction reconstruction loss corresponding to each image patch in the first image; the first image is an image after adding a mask to one or more image patches in the original image; The update module is used to update the model parameters of the second model based on the model parameters of the first model. The determination module is used to determine the prediction reconstruction loss corresponding to each image patch in the original image based on the second model after updating the model parameters; The masking module is used to generate a target masking strategy for performing masking operations on each image block in the first image based on the prediction and reconstruction loss corresponding to each image block in the original image and the preset hard sample requirements. An iterative module is used to update the first image based on the target masking strategy, and to train the first model in the next iteration until the loss function of the first model converges or the number of iterations reaches a preset total number of iterations.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model training method based on hard sample mining as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method based on hard sample mining as described in any one of claims 1 to 7.