Model training and image processing method, device, electronic device and storage medium

Abstract

The present application discloses a model training and image processing method, apparatus, electronic device, and storage medium. The method may include inputting a plurality of training samples into a model to obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types; updating model parameters of the model according to the loss values, and determining the sample types to which at least some of the training samples belong based on the loss values; storing the at least some of the training samples in sample sets corresponding to the sample types to which they belong; and training the model based on the training samples included in the sample sets.

Description

Technical Field

[0001] The present application relates to computer technology, and specifically to model training and image processing methods, devices, electronic devices, and storage media. Background Art

[0002] In the field of neural networks, training sample sets are often used to train neural network models. However, in real-world scenarios, certain specific scenarios may only include a small amount of sample data. Therefore, the constructed training sample set may lack relevant samples for these specific scenarios. As a result, when training a neural network based on these training sample sets, the model may not be able to fully learn the relevant information for these specific scenarios due to the imbalance of sample data included in the training sample set, resulting in poor performance in these specific scenarios.

[0003] For example, in the field of face recognition, it is often necessary to train a face recognition model based on a set of face images (training sample set) in the hope that the face recognition model has a good face recognition effect.

[0004] However, since the facial image data included in the above-mentioned facial image set may not be balanced, it may only include a small amount of image data for specific scenes such as children or people wearing masks, resulting in the facial recognition model being unable to learn the relevant information for recognizing faces in such specific scenes, causing the model to perform poorly in specific scenes such as children or people wearing masks. Summary of the Invention

[0005] In view of this, the present application discloses at least one model training method, which includes:

[0006] Input a number of training samples into the model to obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types;

[0007] updating the model parameters of the model according to the loss value, and determining the sample types of at least some of the training samples based on the loss value;

[0008] The at least part of the training samples are stored in sample sets corresponding to the sample types to which they belong, and the model is trained based on the training samples included in the sample sets.

[0009] In some of the illustrated embodiments, determining the sample types corresponding to at least some of the training samples in the plurality of training samples based on the loss value includes:

[0010] Determine the difficult samples among the plurality of training samples based on the loss value;

[0011] The sample type to which the difficult sample belongs is determined according to the sample features corresponding to the difficult sample.

[0012] In some of the illustrated embodiments, the method further includes:

[0013] Inputting the reference images corresponding to each sample type into the above model respectively to obtain the feature centers corresponding to each sample type; wherein the feature centers are used to determine the sample type to which the difficult sample belongs;

[0014] Establish a sample set corresponding to each feature center.

[0015] In some of the illustrated embodiments, each sample type corresponds to M reference images; wherein M is a positive integer;

[0016] The reference images corresponding to each sample type are input into the above model to obtain the feature centers corresponding to each sample type, including:

[0017] Input the M reference images corresponding to each sample type into the model to obtain M reference features corresponding to each sample type;

[0018] The M benchmark features corresponding to each sample type are weighted averaged to obtain the feature center corresponding to each sample type.

[0019] In some of the illustrated embodiments, determining the sample type of the difficult sample according to the sample features corresponding to the difficult sample includes:

[0020] Determine the similarity between the sample features obtained after the difficult sample is input into the above model and the centers of each feature;

[0021] The highest similarity among the above similarities is determined, and the sample type corresponding to the feature center corresponding to the highest similarity is determined as the sample type to which the difficult sample belongs.

[0022] In some of the illustrated embodiments, the method further includes:

[0023] Comparing the sample features of each of the difficult samples obtained by the above model to obtain the similarity of the difficult samples;

[0024] Based on the similarity of the difficult samples, the categories to which different sample data in the difficult samples belong are classified.

[0025] In some of the illustrated embodiments, storing the at least part of the training samples into sample sets corresponding to the sample types to which they belong includes:

[0026] Calculate the average similarity of difficult samples in the sample set corresponding to each category, and obtain the feature center of each of the above sample sets;

[0027] The sample features of the newly obtained difficult samples are compared with the feature centers of each of the above sample sets for similarity, and the newly obtained difficult samples are merged into the above sample sets of their corresponding categories.

[0028] In some of the illustrated embodiments, determining the difficult samples among the plurality of training samples based on the loss value includes:

[0029] Determine the N largest loss values ​​among the loss values ​​corresponding to each training sample; where N is a positive integer;

[0030] The training samples corresponding to the above N loss values ​​are determined as the above difficult samples.

[0031] In some of the illustrated embodiments, determining the difficult samples among the plurality of training samples based on the loss value includes:

[0032] Determine whether the loss value corresponding to each training sample reaches a second preset threshold;

[0033] If the loss value corresponding to any training sample reaches the second preset threshold, the training sample is determined as the difficult sample.

[0034] In some of the illustrated embodiments, the method further includes:

[0035] After the number of stored difficult samples reaches a first preset threshold and the stored difficult samples are input into the above-mentioned model for training, the difficult samples corresponding to the P loss values ​​corresponding to each difficult sample obtained in this training are stored in the sample set corresponding to the sample type to which each difficult sample belongs.

[0036] In some of the illustrated embodiments, the method further includes:

[0037] Before inputting several training samples into the model, batch data is constructed based on the above training samples;

[0038] The above inputs several training samples into the model and obtains the loss value corresponding to each training sample, including:

[0039] Input the above batch data into the model to obtain the loss value corresponding to each training sample in the batch data.

[0040] In some of the illustrated embodiments, the training of the model based on the training samples included in the sample set includes:

[0041] Determining whether the number of difficult samples in each sample set reaches a first preset threshold;

[0042] If the requirement is met, the difficult samples of this type will be input into the above model for training and the above model parameters will be updated; otherwise, the difficult samples will continue to be accumulated.

[0043] In some of the illustrated embodiments, the first preset threshold is the number of samples included in the batch data.

[0044] In some of the illustrated embodiments, the method further includes:

[0045] Before using the above training samples to train the model, the model is pre-trained using pre-training samples; wherein the above pre-training samples include pre-training samples of multiple sample types.

[0046] The present application also discloses an image processing method, which comprises:

[0047] Acquire the target image;

[0048] Performing image processing on the target image through an image processing model to obtain an image processing result corresponding to the target image;

[0049] The above-mentioned image processing model includes a model trained based on the model training method shown in any of the aforementioned embodiments.

[0050] The present application also discloses a model training device, which includes: an input module for inputting a plurality of training samples into the model to obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types;

[0051] A determination module, configured to update the model parameters of the model according to the loss value, and determine the difficult samples in each training sample based on the loss value;

[0052] an updating and determining module, configured to update the model parameters of the model according to the loss value, and determine the sample types to which at least some of the training samples belong based on the loss value;

[0053] The storage and training module is used to store at least part of the training samples in sample sets corresponding to the sample types to which they belong, and to train the model based on the training samples included in the sample sets.

[0054] The present application also discloses an image processing device, comprising:

[0055] An acquisition module, used to acquire a target image;

[0056] An image processing module is used to perform image processing on the target image through an image processing model to obtain an image processing result corresponding to the target image;

[0057] The above-mentioned image processing model includes a model trained based on the model training method shown in any of the aforementioned embodiments.

[0058] The present application also discloses an electronic device, comprising:

[0059] processor;

[0060] A memory for storing the processor-executable instructions;

[0061] The processor is configured to call the executable instructions stored in the memory to implement the aforementioned model training method or weighted image processing method.

[0062] The present application also discloses a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to implement the aforementioned model training method or weight image processing method.

[0063] In the present application, during the model training process, the above scheme can determine the difficult samples based on the loss value obtained during training. After determining the difficult samples, the above training samples can be classified and stored, and the above model can be trained based on the difficult samples in the above sample set, and the above model parameters can be updated. Therefore, on the one hand, difficult samples can be screened out and trained on the difficult samples during the training of the model using the training samples, so that there is no need to construct a separate training set for the difficult samples for independent training, reducing the workload of developers; on the other hand, various types of difficult samples can be used to train the model, thereby increasing the number of optimization times of the model for each type of difficult sample, so that the model performs better in each specific scenario.

[0064] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0066] Figure 1 A flow chart of a traditional model training method shown in this application;

[0067] Figure 2 A flow chart of a model training method shown in this application;

[0068] Figure 3 A flow chart of a model training method shown in this application;

[0069] Figure 4 A flow chart of a model training method shown in this application;

[0070] Figure 5 A schematic diagram of the internal structure of a memory unit shown in this application;

[0071] Figure 6 This is a schematic structural diagram of a model training device shown in this application;

[0072] Figure 7 This is a schematic diagram of the hardware structure of an electronic device shown in this application. DETAILED DESCRIPTION

[0073] The following exemplary embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0074] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used herein, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0075] Before introducing the embodiments of this application, we first introduce the traditional model training method. The following uses the field of face recognition as an example to illustrate the embodiments.

[0076] See Figure 1 , Figure 1 This is a flow chart of a traditional model training method shown in this application. It should be noted that, Figure 1 The process description shown is only a schematic description of the model training method process and can be fine-tuned in actual applications.

[0077] like Figure 1 As shown, when performing model training, it is usually necessary to first execute S102 to prepare training samples.

[0078] In the field of face recognition, the training samples mentioned above can generally be a collection of multiple face images labeled with person objects. When preparing the training samples, the original images can generally be annotated with true values ​​using manual or machine-assisted annotation. For example, after acquiring the original image, image annotation software can be used to annotate the person objects indicated by the faces included in the original image, thereby obtaining several training samples. It should be noted that one-hot encoding and other methods can be used for annotation during the true value annotation, and this application does not limit the specific annotation method.

[0079] After obtaining a plurality of training samples, S104 may be executed to generate batch data from the plurality of training samples by random sampling during each training process. After obtaining the batch data, the batch data may be input into the model for training.

[0080] The above-mentioned batch data may specifically include a number of training samples. Among them, the above-mentioned training samples need to be input into the above-mentioned model for training during this round of iterative training to update the parameters of the above-mentioned model. It can be understood that in the field of face recognition, the above-mentioned training samples can be face images that are labeled with true values. It should be noted that this application does not specifically limit the number of samples included in the batch data. The present application can also adopt a single data model training scheme, which can refer to the batch data scheme and will not be described in detail here.

[0081] In this step, the forward propagation method can be used in the above model to obtain the feature map corresponding to each training sample, and the face recognition results corresponding to each training sample obtained in this training are output through the connected classifier.

[0082] After obtaining the above face recognition results, S106 may be executed to input each face recognition result obtained through training and the true value corresponding to the training sample corresponding to each recognition result into a preset loss function to calculate the loss value.

[0083] The preset loss function may be a loss function commonly used in the field of face recognition and is not particularly limited here.

[0084] After obtaining the loss value corresponding to each training sample, S108 may be executed to update the above model parameters by back-propagating the gradient using the gradient descent method.

[0085] The above-mentioned gradient descent method can be stochastic gradient descent (SGD), batch gradient descent (BGD), or mini-batch gradient descent (MBGD), without particular limitation. In this application, BGD is used.

[0086] After one training session is completed, the above steps S102 to S108 may be repeated until the above model converges.

[0087] The above is the traditional model training method. It is not difficult to find that since the facial image data included in the above training samples may not be balanced, only a small amount of image data (i.e., difficult samples) may be included for specific scenes such as children or people wearing masks, resulting in the face recognition model being unable to learn the relevant information for recognizing faces in such specific scenes, causing the model to perform poorly in specific scenes such as children or people wearing masks (i.e., difficult samples).

[0088] In order to improve the performance of face recognition models for certain specific scenarios, traditional technologies usually require constructing specific types of training samples related to specific scenarios and continuing to train the trained model several times to achieve the effect of fine-tuning the model.

[0089] It is not difficult to find that the training of models in related technologies not only requires sample construction for different scenarios, but also requires multiple independent training of the model. It can be seen that the related technologies are relatively cumbersome and not very friendly to developers.

[0090] Based on this, the present application proposes a model training method. This method screens out difficult samples from the training sample set during the model training process using training samples, and forms batch data with the type of difficult samples as the dimension to conduct centralized training and learning on the model. Thus, on the one hand, difficult samples can be screened out during the training process and trained on these difficult samples, eliminating the need to construct a separate training set for difficult samples for independent training, thus reducing the workload of developers; on the other hand, various types of difficult samples can be used to train the model, thereby increasing the number of optimizations of the model for each type of difficult sample, making the model perform better in various specific scenarios.

[0091] See Figure 2 , Figure 2 This is a flow chart of a model training method shown in this application. Figure 2 As shown, the above method may include:

[0092] S202, inputting several training samples into the model to obtain the loss value corresponding to each training sample; wherein,

[0093] The above training samples include training samples of multiple sample types;

[0094] S204, updating the model parameters of the model according to the loss value, and determining the sample types of at least some of the training samples based on the loss value;

[0095] S206: Store at least part of the training samples in sample sets corresponding to the sample types to which they belong, and train the model based on the training samples included in the sample sets.

[0096] The above-mentioned model training method can be applied to electronic devices. Among them, the above-mentioned electronic devices can execute the above-mentioned model training method by being equipped with a software system corresponding to the model training method. In the embodiment of the present application, the type of the above-mentioned electronic device can be a laptop computer, a computer, a server, a mobile phone, a PAD terminal, etc., which is not particularly limited in this application.

[0097] It is understandable that the above-mentioned model training method can be executed solely by the terminal device or the server device, or can be executed in coordination with the terminal device and the server device.

[0098] For example, the above model training method can be integrated into the client. After receiving the model training request, the terminal device equipped with the client can provide computing power through its own hardware environment to execute the above model training method.

[0099] For another example, the above model training method can be integrated into a system platform. After receiving a model training request, the server device equipped with the system platform can provide computing power through its own hardware environment to execute the above model training method.

[0100] For example, the model training method can be divided into two tasks: constructing a training sample set and training the model based on the training sample set. Constructing the training sample set can be integrated into the client and installed on the terminal device. The model training task can be integrated into the server and installed on the server device. After constructing the training sample set, the terminal device can initiate a model training request to the server device. Upon receiving the model training request, the server device can respond to the request and train the model based on the training sample set.

[0101] The following description will be made by taking an electronic device (hereinafter referred to as the device) as an example.

[0102] The above-mentioned model can be a model built based on a neural network. In different fields, the above-mentioned model can be a model with different structures and uses. For example, in the field of face recognition, the above-mentioned model can be a face recognition model built based on a convolutional network (hereinafter referred to as the "model"). For another example, in the field of autonomous driving, the above-mentioned model can be an image processing model built based on LSTM (long short-term memory model). For another example, the above-mentioned model can be a human body recognition model built based on a convolutional network, etc. The following embodiment is described using the field of face recognition as an example.

[0103] The aforementioned model parameters specifically refer to various parameters within the aforementioned model that require adjustment. It is understood that model training is actually a process of continuously adjusting these model parameters. When the model converges, the model parameters are considered to be optimally adjusted.

[0104] The above-mentioned model convergence refers to the model reaching a certain preset convergence condition during the training process. It is understandable that the model convergence can be considered to have completed this training. This application does not specifically limit the specific conditions for model convergence.

[0105] Before using the training samples to train the model, the model is pre-trained using pre-training samples, wherein the pre-training samples include pre-training samples of multiple sample types. This can accelerate the model convergence speed and improve the model training efficiency.

[0106] At least some of the above-mentioned several training samples may refer to difficult samples. The above-mentioned difficult samples specifically refer to training samples with large loss values ​​that appear during the training process (i.e., difficult-to-learn samples). It is understandable that difficult samples can usually represent data from scenarios that do not appear frequently. Therefore, the predictions of the model obtained by training data from common scenarios for difficult samples are usually inaccurate. It can be seen that it is feasible to determine difficult samples by the loss value obtained by model training in this application.

[0107] For example, in the field of face recognition, difficult samples can be specific types of image data such as face images of people wearing masks, face images of children, and face images of the elderly.

[0108] When determining the sample classes corresponding to at least some of the training samples based on the loss value, difficult samples among the training samples can be determined based on the loss value. The sample types to which the difficult samples belong can then be determined based on the sample features corresponding to the difficult samples.

[0109] The first preset threshold may be a value set based on experience. In some examples, to facilitate model operation, the first preset threshold may be the same as the number of samples included in the batch data.

[0110] The following describes the embodiments of the present application in conjunction with the field of face recognition.

[0111] See Figure 3 , Figure 3 This is a flow chart of a model training method shown in this application. It should be noted that, Figure 3 The process description shown is only a schematic description of the model training method process and can be fine-tuned in actual applications. Figure 3 The process of backpropagation to update model parameters is not shown.

[0112] Figure 3 The memory unit shown in is specifically a virtual unit, which can be implemented by carrying specific codes: storing difficult samples, and outputting the stored difficult samples when the number of difficult samples reaches a first preset threshold.

[0113] The memory unit may include a sample set, a counter, and an output subunit. The sample set may be used to store difficult samples. The counter may be used to indicate the number of difficult samples stored in the memory unit. The output subunit may be used to retrieve and output stored difficult samples from the sample set.

[0114] In some examples, to facilitate data access, the sample set may include a linear data structure, such as a queue. It will be appreciated that when the sample set is in the form of a queue, the maximum capacity of the queue can be set to the first preset threshold. In this case, when the queue is full, it can be considered that the number of stored difficult samples has reached the first preset threshold. Of course, in this case, the memory unit does not need to include the counter.

[0115] When performing model training, it is necessary to first execute S302 to prepare training samples.

[0116] In the field of face recognition, the above-mentioned training samples can generally be a collection of multiple face images labeled with person objects. When preparing the above-mentioned training samples, the original images can generally be labeled with true values ​​using manual labeling or machine-assisted labeling. For example, after obtaining the original image, image labeling software can be used to label the person objects indicated by the faces included in the original image, thereby obtaining several training samples. It should be noted that when constructing the training samples, one-hot encoding or other methods can be used for construction, and this application does not limit the specific method of constructing the training samples.

[0117] When preparing training samples, training samples including multiple sample types may be randomly sampled.

[0118] The above sample types are specifically used to indicate the scene type to which the sample belongs. For example, in the field of face recognition, if a sample image includes a child's face, the sample can be considered to be a child sample type. If a sample image includes an elderly person's face, the sample can be considered to be an elderly sample type. If a sample image includes a face wearing a mask, the sample can be considered to be a mask sample type. This ensures that the training samples include a variety of training samples, improving training effectiveness.

[0119] After obtaining a number of training samples, S202 may be executed to input the training samples into the model to obtain a loss value corresponding to each training sample.

[0120] In the above training process, S304 may be performed first to construct batch data based on a number of training samples, and then input the batch data into the model for training.

[0121] Specifically, during each training process, batch data is generated from the aforementioned training samples by random sampling. After the batch data is obtained, the batch data can be input into the aforementioned model for training.

[0122] In this step, the forward propagation method can be used in the above model to obtain the feature map corresponding to each training sample, and the face recognition results corresponding to each training sample obtained in this training are output through the connected classifier.

[0123] It should be noted that this application does not specifically limit the number of samples included in the batch data. This application can also adopt a single data model training solution, which can refer to the batch data solution and will not be described in detail here.

[0124] After obtaining the above face recognition results, S306 may be executed to input each face recognition result obtained through training and the true value corresponding to the training sample corresponding to each recognition result into a preset loss function to calculate the loss value.

[0125] The preset loss function may be a loss function commonly used in the field of face recognition and is not particularly limited here.

[0126] After obtaining the loss value corresponding to each training sample, S204 may be executed to update the model parameters of the model according to the loss value, and determine the difficult samples in each training sample based on the loss value.

[0127] Specifically, on the one hand, S308 can be executed to back-propagate the gradient through the gradient descent method to update the above model parameters; on the other hand, S310 can be executed to determine the difficult samples included in the above training samples based on the loss values ​​corresponding to each training sample obtained after forward propagation.

[0128] When determining difficult samples, in some examples, N loss values ​​with larger values ​​among the loss values ​​corresponding to each training sample can be determined. Wherein, the above-mentioned N is a positive integer. For example, the loss values ​​corresponding to each training sample can be sorted in order from large to small. After the sorting is completed, the loss values ​​ranked in the top N positions can be determined as the N loss values ​​with larger values. It should be noted here that the above-mentioned N can be a value set based on experience. This application does not specifically limit the value of N.

[0129] After determining N loss values ​​with larger numerical values, the training samples corresponding to the N loss values ​​can be determined as the difficult samples.

[0130] In other examples, it is determined whether the loss value corresponding to each training sample reaches a second preset threshold.

[0131] The second preset threshold may be a value set based on experience. Reaching the second preset threshold includes at least two conditions: being greater than or equal to the second preset threshold. The second preset threshold may be a reference line for determining whether a training sample is a difficult sample. If the loss value corresponding to any training sample reaches the second preset threshold, the training sample is determined to be a difficult sample.

[0132] After determining the difficult sample, S204 may be executed to determine the sample type of the difficult sample.

[0133] The sample type of the aforementioned difficult sample is specifically used to indicate the scene type to which the difficult sample belongs. For example, in the field of face recognition, when the difficult sample includes an image of a child's face, the difficult sample can be considered to belong to the child sample type. When the difficult sample includes an image of an elderly person's face, the difficult sample can be considered to belong to the elderly sample type. When the difficult sample includes an image of a face wearing a mask, the difficult sample can be considered to belong to the mask sample type.

[0134] When determining the sample type of a difficult sample, the feature center of the difficult sample extracted by the above model can be matched with the feature center of each sample type extracted by the above model, and the sample type corresponding to the matching feature center can be determined as the sample type of the above difficult sample.

[0135] The above sample sets can be used to store difficult samples. In some examples, reference images corresponding to each sample type can be input into the above model to obtain feature centers corresponding to each sample type; these feature centers are used to determine the sample type to which the difficult sample belongs. Sample sets corresponding to each feature center are then established.

[0136] When training the model based on the difficult samples in the sample set, it can be determined whether the number of difficult samples in each sample set reaches a first preset threshold. If so, the difficult samples of that type are input into the model for training and the model parameters are updated; otherwise, difficult samples continue to be accumulated.

[0137] Please continue to see Figure 3 After determining the difficult samples, S206 may be executed to store the difficult samples in a sample set corresponding to the sample type to which the difficult samples belong, and to train the model based on the difficult samples in the sample set.

[0138] In some examples, the difficult samples may be stored in the memory unit.

[0139] The memory unit may periodically or upon receiving each difficult sample, determine whether the number of stored difficult samples has reached the first preset threshold. If so, the stored difficult samples are fed into the model for training and the model parameters are updated. If not, no action is taken.

[0140] In some examples, the first preset threshold is the number of samples included in the batch data. When the stored difficult samples are input into the model for training and the model parameters are updated, S312 can be executed to construct the stored difficult samples into batch data and input them into the model for training and updating the model parameters.

[0141] Since the number of difficult samples output by the above memory unit is consistent with the number of samples included in the batch data constructed when training the model, it is convenient for the model to perform calculations.

[0142] After the model parameters are updated once, the steps S304 to S312 may be repeated until the model converges.

[0143] During model training, the above solution can identify difficult samples based on the loss value obtained during training. After determining the difficult samples, the training samples can be classified and stored, and the model can be trained based on the difficult samples in the sample set, and the model parameters can be updated. Therefore, on the one hand, difficult samples can be screened out and trained on these difficult samples during the training process of the model using training samples, eliminating the need to construct a separate training set for difficult samples and conduct independent training, reducing the workload of developers; on the other hand, various types of difficult samples can be used to train the model, thereby increasing the number of times the model is optimized for each type of difficult sample, making the model perform better in specific scenarios.

[0144] In some embodiments, in order to enable the model to learn relevant sample information in a variety of specific scenarios, thereby improving the performance of the model in a variety of scenarios, when executing S206 to store difficult samples, the sample type (i.e., the scenario) to which the difficult samples belong can be determined first. After determining the sample type to which the difficult samples belong, the difficult samples are classified and stored.

[0145] The above sample type is specifically used to indicate the scene type to which the difficult sample belongs. For example, in the field of face recognition, when the difficult sample includes an image of a child's face, it can be considered that the difficult sample belongs to the child sample type. When the difficult sample includes an image of an elderly person's face, it can be considered that the difficult sample belongs to the elderly sample type. When the difficult sample includes an image of a face wearing a mask, it can be considered that the difficult sample belongs to the mask sample type.

[0146] See Figure 4 , Figure 4 This is a flow chart of a model training method shown in this application. It should be noted that, Figure 4 The process description shown is only a schematic description of the model training method process and can be fine-tuned in actual applications. Figure 4 The process of backpropagation to update model parameters is not shown.

[0147] Figure 4 The memory unit shown in is specifically a virtual unit, which can be implemented by carrying specific codes: classifying and storing difficult samples, and when the number of difficult samples of any type reaches a first preset threshold, inputting the difficult samples of that type into the above-mentioned model for training and updating the above-mentioned model parameters.

[0148] The memory unit may include several sample sets corresponding to the sample types, a counter, and an output subunit.

[0149] The plurality of sample sets corresponding to the sample types are used to store various types of difficult sample data. The counter can be used to indicate the number of difficult samples stored in each sample set. The output subunit is used to obtain and output the stored difficult samples from the sample sets that meet the conditions.

[0150] In some examples, to facilitate data access, the sample set may include a linear data structure, such as a queue. It will be appreciated that when the sample set is in the form of a queue, the maximum capacity of the queue can be set to the first preset threshold. In this case, when the queue is full, it can be considered that the number of stored difficult samples has reached the first preset threshold. Of course, in this case, the memory unit does not need to include the counter.

[0151] During model training, memory cells usually need to be initialized. The following describes the initialization process of memory cells by introducing their internal structure.

[0152] See Figure 5 , Figure 5 This is a schematic diagram of the internal structure of a memory unit shown in this application. It should be noted that, Figure 5 The internal diagram shown is only a schematic illustration and may be fine-tuned in actual application.

[0153] like Figure 5 As shown, the memory unit may include multiple sample types. The sample types included in the memory unit can be pre-set according to actual business needs. For example, when business needs require improving the model's face recognition capabilities for the elderly and children, the memory unit may be set with an elderly sample type and a child sample type. For another example, when business needs require prompting the model's face recognition capabilities for the elderly, children, and people wearing masks, the memory unit may be set with an elderly sample type, a child sample type, and a mask-wearing sample type. The following description will be made using the example of a memory unit including an elderly sample type and a child sample type.

[0154] It should be noted that since a large amount of the training samples mentioned above are still sample data from common scenarios, in order to improve the compatibility of the memory unit and enable the model to learn relevant information about various difficult samples, the memory unit may also include a normal sample type. The normal sample type is used to store difficult samples from common scenarios (i.e., non-specific scenarios).

[0155] For example, when the training samples include three types: elderly, adults, and children, adults are the type in the common scene. In this case, in addition to the difficult sample data of the elderly and children types, the memory unit can also include difficult sample data representing the common scene of adults.

[0156] Please continue to see Figure 5 , where the first sample type may indicate a normal type; the second sample type may indicate an elderly type; and the third sample type may indicate a child type.

[0157] In the above solution, since the difficult sample data in conventional scenarios are also stored, the number of optimization times of the model for difficult samples in such scenarios can be increased, so that the model performs better for difficult samples in such scenarios.

[0158] After determining the sample types included in the memory unit, a corresponding sample set can be created in the memory unit for each sample type. The sample set is in the form of a queue. The maximum capacity corresponding to each queue can be set to the first preset threshold (batch data size). When any queue is full, it can be considered that the number of difficult samples stored in the queue has reached the first preset threshold.

[0159] Please continue to see Figure 5 , where the first sample type queue can be used to store difficult samples of the normal type; the second sample type queue can be used to store difficult samples of the elderly type; and the first sample type queue can be used to store difficult samples of the child type.

[0160] After determining the sample types included in the memory unit, the feature centers corresponding to each sample type can also be determined.

[0161] After determining the feature center corresponding to each sample type, the reference image corresponding to each sample type can be input into the above model to obtain the feature center corresponding to each sample type.

[0162] The feature center is specifically used to determine the sample type to which the difficult sample belongs. In some examples, the feature center can be labeled with pixels of the feature vector.

[0163] It can be understood that by determining the feature center that is most similar to the sample feature corresponding to the difficult sample, the sample type to which the difficult sample belongs can be determined.

[0164] The above-mentioned sample features specifically refer to features obtained after performing operations such as convolution and pooling on difficult samples. In some examples, the above-mentioned sample features can be represented in the form of feature vectors.

[0165] In this step, a face image belonging to each sample type can be selected first. For example, when the memory unit includes an elderly sample type and a child sample type, a child face image and an elderly face image can be selected as the reference image.

[0166] After determining the reference image, the reference image corresponding to each sample type can be input into the above model for forward propagation to obtain the feature center corresponding to each sample type.

[0167] In some examples, to determine more accurate feature centers, M reference images may be selected for each sample type when selecting precise images. M is a positive integer. It is understood that in some examples, the number of reference images selected for each sample type may vary. For example, 10 images may be selected for the elderly and 8 for the children. The following explanation assumes the same number of reference images for each sample type.

[0168] Please continue to see Figure 5 , where the first reference image set may include M reference images of normal type; the second reference image set may include M reference images of elderly type; and the third reference image set may include M reference images of children type.

[0169] At this time, when determining the feature center, the M reference images corresponding to each sample type may be respectively input into the model to obtain M reference features corresponding to each sample type.

[0170] The aforementioned reference features may include features obtained by performing operations such as convolution and pooling on the reference image. In some examples, the aforementioned reference features may be represented in the form of feature vectors.

[0171] After obtaining the M benchmark features corresponding to each sample type, the M benchmark features corresponding to each sample type are weighted averaged to obtain the feature center corresponding to each sample type.

[0172] The above M is an empirical threshold and is not particularly limited here.

[0173] The above-mentioned reference features are feature maps obtained by performing feature extraction (such as several convolution operations) on the reference image through the above-mentioned model.

[0174] It should be noted that the weight used in the weighted average is not particularly limited in this application. For example, the weight can be 1.

[0175] In some cases, the sample types included in the memory unit may not be determined. In this case, a K-MEANS clustering algorithm may be used to cluster the obtained difficult samples to obtain the sample types included in the memory unit.

[0176] Specifically, the sample features of each of the difficult samples obtained by the above model can be compared to obtain the similarity of the difficult samples. Then, based on the similarity of the difficult samples, the categories to which different sample data in the difficult samples belong can be classified.

[0177] The above difficult samples may include several unknown sample types. The above clustering algorithm can be used to reasonably classify the difficult samples and obtain several sample types.

[0178] After determining the sample types included in the memory unit, the feature center of each sample set can be obtained by calculating the average similarity of difficult samples in the sample set corresponding to each category. Therefore, when a new difficult sample is obtained, the sample features of the newly obtained difficult sample can be compared with the feature center of each sample set for similarity, and the newly obtained difficult sample can be merged into the sample set of its corresponding category.

[0179] In the sample type determination method disclosed in the above example, manual determination of sample types can be avoided. Instead, unsupervised clustering is performed based on the actual situation of difficult samples to obtain sample types that are more in line with the actual difficult samples, thereby improving the model prediction effect.

[0180] Please continue to see Figure 5 , where the first feature center can be the feature center corresponding to the normal type; the second feature center can be the feature center corresponding to the elderly type; and the third feature center can be the feature center corresponding to the child type. Using these three feature maps, we can determine the sample type to which the target difficult sample belongs.

[0181] Please continue to see Figure 4 After completing the initialization of the memory unit, the formal training of the model can begin. When performing model training, it is necessary to first execute S402 to prepare training samples.

[0182] After determining a number of training samples, the process may proceed to step S404 to construct batch data based on the training samples, and input the batch data into the model for training.

[0183] After obtaining the prediction results corresponding to the training samples included in the batch data, S406 can be executed to input the true values ​​of the face recognition results obtained through training and the training samples corresponding to the recognition results into a preset loss function to calculate the loss value.

[0184] After obtaining the loss value corresponding to each training sample, on the one hand, S408 can be executed to back-propagate the gradient through the gradient descent method to update the above model parameters; on the other hand, S410 can be executed to determine the difficult samples included in the above training samples based on the loss value corresponding to each training sample obtained after forward propagation.

[0185] After the difficult sample is determined, S412 may be executed to determine the sample type to which the difficult sample belongs.

[0186] In this step, the similarity between the sample features obtained after the difficult sample is input into the model and the centers of each feature can be determined.

[0187] For example, in the present application, the similarity between the sample features and each feature center can be determined using a similarity calculation scheme such as cosine distance or Mahalanobis distance. When calculating the similarity between the sample features and each feature center, the corresponding relationship between the feature center and the similarity calculated based on the feature center can also be maintained.

[0188] After determining the similarities between the sample features and the feature centers, the highest similarity among the similarities may be determined, and the sample type corresponding to the feature center corresponding to the highest similarity may be determined as the sample type to which the difficult sample belongs.

[0189] For example, in this application, the determined similarities can be sorted in descending order, and the similarity at the top can be determined as the highest similarity. After determining the highest similarity, the feature center corresponding to the highest similarity can be determined by querying the maintained correspondence relationship. After determining the feature center corresponding to the highest similarity, the sample type corresponding to the feature center can be determined as the sample type to which the difficult sample belongs.

[0190] After determining the sample type of the difficult sample, S414 may be executed to store the difficult sample in a sample set corresponding to the sample type to which the difficult sample belongs.

[0191] In this step, the difficult samples may be stored in a queue corresponding to the sample type to which the difficult samples belong.

[0192] Please continue to see Figure 5 , assuming that the sample type corresponding to the target difficult sample is the child type, the image data corresponding to the difficult sample can be inserted into the child type queue (ie, the third sample type queue).

[0193] When the number of difficult samples of any type reaches a first preset threshold, S416 may be executed to input the difficult samples of that type into the above model for training, and update the above model parameters.

[0194] In this step, if any sample dataset queue included in the memory unit is full, it can be considered that the number of difficult samples stored in the queue has reached a first preset threshold. At this point, the difficult samples stored in the queue can be extracted to construct batch data. After the batch data is constructed, the batch data can be input into the model for training and updating the model parameters.

[0195] In the above scheme, since multiple sample types can be flexibly set, and difficult samples can be classified, stored, and trained, on the one hand, the model can be trained on multiple specific types of difficult samples, so that the trained model can have better performance in multiple scenarios of this specific type; on the other hand, there is no need to establish training samples specifically for multiple types, which reduces the workload of developers.

[0196] In some embodiments, in order to perform multiple trainings on difficult samples, after the number of stored difficult samples reaches a first preset threshold and the stored difficult samples are input into the above-mentioned model for training, the difficult samples corresponding to the P loss values ​​corresponding to each difficult sample obtained in this training can also be stored in the sample set corresponding to the sample type to which each difficult sample belongs.

[0197] Wherein, P is a positive integer set based on experience.

[0198] In this step, after the number of stored difficult samples reaches a first preset threshold and the stored difficult samples are input into the above-mentioned model for training to obtain the loss value corresponding to each difficult sample, P loss values ​​with larger values ​​among the loss values ​​can be determined.

[0199] After determining the above P loss values, the above sample types of the difficult samples corresponding to the P loss values ​​can be determined, and the difficult samples corresponding to the P loss values ​​can be stored in a sample set corresponding to the sample type to which each difficult sample belongs.

[0200] In the above scheme, after the number of stored difficult samples reaches the first preset threshold and the stored difficult samples are input into the above model for training, the difficult samples corresponding to the P loss values ​​corresponding to each difficult sample obtained in this training with larger values ​​can be stored in the sample set corresponding to the sample type to which each difficult sample belongs. Therefore, difficult samples with larger loss values ​​can be stored multiple times and the model can be trained multiple times, thereby increasing the number of times the model is optimized through this type of difficult samples, so that the model performs better for this type of difficult samples.

[0201] This application also proposes an image processing method. This method can be applied to any electronic device. This method performs image processing using an image processing model trained using the training method described in any of the aforementioned embodiments. This ensures that the image processing model not only performs well in conventional scenarios, but also in different specific scenarios, thereby improving image processing performance.

[0202] Specifically, the above method may include:

[0203] A target image is acquired and image processing is performed on the target image through an image processing model to obtain an image processing result corresponding to the target image.

[0204] The target image can be any image that requires image processing. For example, in a face recognition scenario, the target image can be an image containing a face. The image processing model can be any model that requires image processing. For example, in a face recognition scenario, the image processing model can be a face recognition model.

[0205] Corresponding to any of the above embodiments, the present application also proposes a model training device.

[0206] See Figure 6 , Figure 6 This is a structural diagram of a model training device shown in this application.

[0207] like Figure 6 As shown, the apparatus 600 may include: an input module 610, configured to input a plurality of training samples into the model to obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types;

[0208] An updating and determining module 620 is configured to update the model parameters of the model according to the loss value, and determine the sample types of at least some of the training samples based on the loss value;

[0209] The storage and training module 630 is configured to store at least part of the training samples in sample sets corresponding to the sample types to which they belong, and to train the model based on the training samples included in the sample sets.

[0210] In some of the illustrated embodiments, the updating and determining module 620 includes:

[0211] A first determining module is configured to determine difficult samples among the plurality of training samples based on the loss value;

[0212] The second determining module is used to determine the sample type to which the difficult sample belongs according to the sample features corresponding to the difficult sample.

[0213] In some of the illustrated embodiments, the apparatus 600 further includes:

[0214] Establish a module to input the reference images corresponding to each sample type into the above model to obtain the feature centers corresponding to each sample type; wherein the feature centers are used to determine the sample type to which the difficult sample belongs;

[0215] Establish a sample set corresponding to each feature center.

[0216] In some embodiments shown, each sample type corresponds to M reference images, wherein M is a positive integer; the establishment module is specifically used to:

[0217] Input the M reference images corresponding to each sample type into the model to obtain M reference features corresponding to each sample type;

[0218] The M benchmark features corresponding to each sample type are weighted averaged to obtain the feature center corresponding to each sample type.

[0219] In some of the illustrated embodiments, the updating and determining module 620 is specifically configured to:

[0220] Determine the similarity between the sample features obtained after the difficult sample is input into the above model and the centers of each feature;

[0221] The highest similarity among the above similarities is determined, and the sample type corresponding to the feature center corresponding to the highest similarity is determined as the sample type to which the difficult sample belongs.

[0222] In some of the illustrated embodiments, the apparatus 600 further includes:

[0223] The classification module compares the sample features of each of the difficult samples obtained by the above model to obtain the similarity of the difficult samples;

[0224] Based on the similarity of the difficult samples, the categories to which different sample data in the difficult samples belong are classified.

[0225] In some of the illustrated embodiments, the updating and determining module 620 is specifically configured to:

[0226] Calculate the average similarity of difficult samples in the sample set corresponding to each category, and obtain the feature center of each of the above sample sets;

[0227] The sample features of the newly obtained difficult samples are compared with the feature centers of each of the above sample sets for similarity, and the newly obtained difficult samples are merged into the above sample sets of their corresponding categories.

[0228] In some of the illustrated embodiments, the updating and determining module 620 is specifically configured to:

[0229] Determine the N largest loss values ​​among the loss values ​​corresponding to each training sample; where N is a positive integer;

[0230] The training samples corresponding to the above N loss values ​​are determined as the above difficult samples.

[0231] In some of the illustrated embodiments, the updating and determining module 620 is specifically configured to:

[0232] Determine whether the loss value corresponding to each training sample reaches a second preset threshold;

[0233] If the loss value corresponding to any training sample reaches the second preset threshold, the training sample is determined as the difficult sample.

[0234] In some of the illustrated embodiments, the apparatus 600 further includes:

[0235] The storage module stores the difficult samples corresponding to the P loss values ​​corresponding to the difficult samples obtained in this training, and stores them in the sample set corresponding to the sample type to which each difficult sample belongs, after the number of stored difficult samples reaches a first preset threshold and the stored difficult samples are input into the above-mentioned model for training.

[0236] In some of the illustrated embodiments, the apparatus 600 further includes:

[0237] A batch processing module that constructs batch data based on several training samples before inputting them into the model;

[0238] The above inputs several training samples into the model and obtains the loss value corresponding to each training sample, including:

[0239] Input the above batch data into the model to obtain the loss value corresponding to each training sample in the batch data.

[0240] In some embodiments shown, the storage and training module 630 is specifically used to:

[0241] Determining whether the number of difficult samples in each sample set reaches a first preset threshold;

[0242] If the requirement is met, the difficult samples of this type will be input into the above model for training and the above model parameters will be updated; otherwise, the difficult samples will continue to be accumulated.

[0243] In some of the illustrated embodiments, the first preset threshold is the number of samples included in the batch data.

[0244] In some of the illustrated embodiments, the apparatus 600 further includes:

[0245] The pre-training module uses the pre-training samples to pre-train the model before using the above training samples to train the model; wherein the above pre-training samples include pre-training samples of multiple sample types.

[0246] The present application also proposes an image processing device, which may include:

[0247] The acquisition module is used to acquire the target image.

[0248] The image processing module is used to perform image processing on the target image through an image processing model to obtain an image processing result corresponding to the target image.

[0249] Among them, the above-mentioned image processing model includes a model obtained based on the model training method shown in any of the above-mentioned embodiments.

[0250] The embodiments of the model training device or image processing device shown in this application can be applied to electronic devices.

[0251] Accordingly, the present application discloses an electronic device, which may include: a processor.

[0252] Memory used to store processor-executable instructions.

[0253] Among them, the above-mentioned processor is configured to call the executable instructions stored in the above-mentioned memory to implement the model training method or image processing method as shown above.

[0254] See Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of an electronic device shown in this application.

[0255] like Figure 7 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing corresponding instructions of the model training device or the image processing device.

[0256] The embodiments of the above-mentioned device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running it. From the hardware level, in addition to Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0257] It is understandable that in order to improve the processing speed, the corresponding instructions of the model training device or the image processing device can also be directly stored in the memory, which is not limited here.

[0258] The present application proposes a computer-readable storage medium, which stores a computer program. The computer program is used to execute the model training method or image processing method as shown above.

[0259] It will be understood by those skilled in the art that one or more embodiments of the present application may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0260] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0261] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the data processing device embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0262] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0263] Embodiments of the subject matter and functional operations described in this application may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this application may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0264] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0265] A computer suitable for executing a computer program may include, for example, a general-purpose and / or special-purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such large-capacity storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0266] Computer-readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media, and storage devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0267] Although the application includes many specific implementation details, these should not be interpreted as limiting the scope of any disclosure or the scope of protection claimed, but are mainly used to describe the features of the specific disclosed embodiments. Certain features described in multiple embodiments of the application can also be combined and implemented in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented in any suitable sub-combination. In addition, although features can work in some combinations as described above and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0268] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0269] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0270] The above are merely preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Inputting a plurality of training samples into the model to obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types, the training samples include face images annotated with true values, and the model includes a face recognition model; updating model parameters of the model according to the loss value, and determining sample types of at least some of the training samples based on the loss value, wherein the sample type is used to indicate a scene type to which the sample belongs, and the at least some of the training samples include difficult samples; Storing at least part of the training samples in sample sets corresponding to the sample types to which they belong, and training the model based on the training samples included in the sample sets; The determining, based on the loss value, the sample types corresponding to at least some of the training samples in the plurality of training samples includes: Determining a difficult sample among the plurality of training samples based on the loss value; Determining the sample type to which the difficult sample belongs according to the sample features corresponding to the difficult sample; The training of the model based on the training samples included in the sample set includes: Constructing the stored difficult samples into batch data, and inputting the batch data into the model to train the model; The step of constructing the stored difficult samples into batch data and inputting the batch data into the model to train the model includes: When the number of difficult samples of any sample type reaches a first preset threshold, the difficult samples of any sample type are input into the model for training.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the reference images corresponding to each sample type into the model respectively to obtain the feature centers corresponding to each sample type; wherein the feature centers are used to determine the sample type to which the difficult sample belongs; Establish a sample set corresponding to each feature center.

3. The method according to claim 2, characterized in that M reference images corresponding to each sample type; wherein M is a positive integer; The step of inputting the reference images corresponding to the respective sample types into the model to obtain the feature centers corresponding to the respective sample types includes: Inputting the M reference images corresponding to each sample type into the model respectively to obtain M reference features corresponding to each sample type; The M benchmark features corresponding to each sample type are weighted averaged to obtain the feature center corresponding to each sample type.

4. The method according to claim 2 or 3, characterized in that The determining the sample type of the difficult sample according to the sample features corresponding to the difficult sample includes: Determine the similarity between the sample features obtained after the difficult sample is input into the model and the centers of each feature; The highest similarity among the similarities is determined, and the sample type corresponding to the feature center corresponding to the highest similarity is determined as the sample type to which the difficult sample belongs.

5. The method according to claim 1, wherein The method further comprises: Comparing the sample features of each of the difficult samples obtained by the model to obtain the similarity of the difficult samples; Based on the similarity of the difficult samples, the categories to which different sample data in the difficult samples belong are classified.

6. The method according to claim 5, characterized in that The storing the at least part of the training samples into sample sets corresponding to the sample types to which they belong respectively includes: Calculate the average similarity of difficult samples in the sample set corresponding to each category to obtain the feature center of each sample set; The sample features of the newly obtained difficult samples are compared with the feature centers of each of the sample sets for similarity, and the newly obtained difficult samples are merged into the sample sets of their corresponding categories.

7. The method according to any one of claims 1 to 3, characterized in that: The determining of difficult samples among the plurality of training samples based on the loss value includes: Determine N loss values ​​with larger values ​​among the loss values ​​corresponding to the training samples; wherein N is a positive integer; The training samples corresponding to the N loss values ​​are determined as the difficult samples.

8. The method according to any one of claims 1 to 3, characterized in that: The determining of difficult samples among the plurality of training samples based on the loss value includes: Determine whether the loss value corresponding to each training sample reaches a second preset threshold; If the loss value corresponding to any training sample reaches the second preset threshold, the training sample is determined as the difficult sample.

9. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: After the number of stored difficult samples reaches a first preset threshold and the stored difficult samples are input into the model for training, the difficult samples corresponding to the P loss values ​​corresponding to the difficult samples obtained in this training with larger values ​​are stored in the sample set corresponding to the sample type to which each difficult sample belongs.

10. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Before inputting a plurality of training samples into the model, constructing batch data based on the training samples; The step of inputting several training samples into the model and obtaining the loss value corresponding to each training sample includes: The batch data is input into the model to obtain the loss value corresponding to each training sample in the batch data.

11. The method according to claim 10, characterized in that The training of the model based on the training samples included in the sample set includes: Determining whether the number of difficult samples in each sample set reaches a first preset threshold; If the requirement is met, the difficult samples of this type will be input into the model for training and the model parameters will be updated; otherwise, the difficult samples will continue to be accumulated.

12. The method according to claim 11, characterized in that The first preset threshold is the number of samples included in the batch data.

13. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Before using the training samples to train the model, the model is pre-trained using pre-training samples; wherein the pre-training samples include pre-training samples of multiple sample types.

14. An image processing method, characterized in that: The method comprises: Acquire the target image; Performing image processing on the target image through an image processing model to obtain an image processing result corresponding to the target image; Wherein, the image processing model includes a model obtained based on the model training method described in any one of claims 1-13.

15. A model training device, characterized in that: The device comprises: An input module, configured to input a plurality of training samples into the model and obtain a loss value corresponding to each training sample; wherein the training samples include training samples of multiple sample types, the training samples include face images annotated with true values, and the model includes a face recognition model; an updating and determining module, configured to update model parameters of the model according to the loss value, and determine, based on the loss value, sample types to which at least some of the training samples belong, wherein the sample types are used to indicate the scene types to which the samples belong, and wherein at least some of the training samples include difficult samples; a storage and training module, configured to store at least part of the training samples in sample sets corresponding to the sample types to which they belong, and to train the model based on the training samples included in the sample sets; When the updating and determining module is used to determine the sample types corresponding to at least some of the training samples based on the loss value, the updating and determining module is specifically used to: Determining a difficult sample among the plurality of training samples based on the loss value; Determining the sample type to which the difficult sample belongs according to the sample features corresponding to the difficult sample; When the storage and training module is used to train the model based on the training samples included in the sample set, it is specifically used to: Constructing the stored difficult samples into batch data, and inputting the batch data into the model to train the model; The storage and training module is used to construct the stored difficult samples into batch data and input the batch data into the model to train the model, specifically for: When the number of difficult samples of any sample type reaches a first preset threshold, the difficult samples of any sample type are input into the model for training.

16. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire a target image; An image processing module is used to perform image processing on the target image through an image processing model to obtain an image processing result corresponding to the target image; Wherein, the image processing model includes a model obtained based on the model training method described in any one of claims 1-13.

17. An electronic device, characterized in that: The device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the model training method according to any one of claims 1 to 13 or the image processing method according to claim 14.

18. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is used to execute the model training method according to any one of claims 1 to 13 or the image processing method according to claim 14.

Images