Model training methods, devices, electronic equipment, and storage media based on balanced samples

Abstract

This application provides a model training method, apparatus, electronic device, and storage medium based on balanced samples. The method calculates accumulated coefficients based on the same type of expression coefficients from multiple training samples, and constructs an accumulated vector from these coefficients. A target accumulated coefficient is determined based on the distribution of the accumulated coefficients in the accumulated vector. A portion of the target accumulated coefficient representing the target expression coefficient is assigned a first weight, and the remaining expression coefficients are assigned a second weight, resulting in a weight matrix composed of the first and second weights. The model is adjusted and trained further under the guidance of a loss function based on the weight matrix until preset requirements are met. In this scheme, the distribution of expression coefficients in the training samples is used to assign different weights to the expression coefficients, and training is performed under the guidance of a weighted loss function. This ensures that the training is based on information from uniform expression coefficients in the samples, optimizing the model training effect based on samples that conform to real-world scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model training method, apparatus, electronic device, and storage medium based on balanced samples. Background Technology

[0002] In the field of artificial intelligence, model training is a common process. Model training involves iterative optimization using samples and guided by certain metrics. The goal of model training is to make the model's output as close as possible to the true labels of the samples. The samples used for training are crucial because the model optimizes by learning the features of the samples. However, the collected samples often suffer from imbalance, with some common samples having a large proportion while others have a small proportion. This can easily lead to underfitting of the model to the insufficient sample data during training, affecting the model's training performance.

[0003] In existing methods, more samples are typically generated by using the model or by oversampling or undersampling. However, these methods are prone to producing virtual samples that do not fully match the actual scenario, thus affecting the training effect of the model. Summary of the Invention

[0004] The purpose of this application includes, for example, providing a model training method, apparatus, electronic device, and storage medium based on balanced samples, which can optimize model training performance based on samples that conform to real-world scenarios.

[0005] The embodiments of this application can be implemented as follows:

[0006] In a first aspect, this application provides a model training method based on balanced samples, the method comprising:

[0007] Multiple training samples are obtained, each training sample having a label vector, and the label vector containing multiple expression coefficients;

[0008] The cumulative coefficients are calculated based on the same type of expression coefficients from the multiple training samples, and a cumulative vector is constructed from the obtained cumulative coefficients.

[0009] The target cumulative coefficient is determined based on the distribution of the cumulative coefficients in the cumulative vector. A portion of the target expression coefficients of the target cumulative coefficients are assigned a first weight, and the other expression coefficients are assigned a second weight, resulting in a weight matrix composed of all the first and second weights.

[0010] The constructed model is trained using the multiple training samples, and the model is adjusted and trained under the guidance of the loss function composed of the weight matrix until the preset requirements are met.

[0011] In an optional implementation, the step of obtaining multiple training samples includes:

[0012] Obtain multiple raw samples. For each type of expression coefficient in the multiple raw samples, construct multiple arrays with an initial empty state for that type of expression coefficient. Each array corresponds to a sub-interval divided by the interval to which the expression coefficient belongs.

[0013] For each type of expression coefficient, iterate through each original sample and place each original sample into the array corresponding to the sub-interval to which the expression coefficient belongs;

[0014] For each type of expression coefficient, original samples are selected from the corresponding array as training samples until a preset number of training samples are obtained.

[0015] In an optional implementation, the sub-intervals to which the various expression coefficients belong are distributed in ascending order;

[0016] Before the step of selecting original samples from the corresponding array as training samples for various expression coefficients, the step of obtaining multiple training samples further includes:

[0017] For each type of expression coefficient, delete the array with the smallest corresponding sub-interval;

[0018] Delete empty arrays from multiple arrays.

[0019] In an optional implementation, before the step of selecting original samples from the corresponding array as training samples for various expression coefficients, the step of obtaining multiple training samples further includes:

[0020] For each type of expression coefficient, determine the corresponding array containing an array with a number of original samples less than a set number;

[0021] An array containing fewer than a set number of original samples is merged with its neighboring arrays so that the number of original samples in the merged array is greater than or equal to the set number.

[0022] In an optional implementation, the step of selecting original samples from the corresponding array as training samples for various expression coefficients until a preset number of training samples are obtained includes:

[0023] For each type of expression coefficient, a raw sample is selected from its corresponding array in turn, and it is checked whether the selected raw sample is the same as any of the currently determined training samples. If they are the same, the next array is selected. If they are not the same, the selected raw sample is used as the training sample.

[0024] The selection process stops when the number of training samples reaches the preset number.

[0025] In an optional implementation, the step of obtaining multiple training samples further includes:

[0026] Copy each of the obtained arrays to get the corresponding copied arrays;

[0027] The step of selecting original samples from the corresponding array as training samples for various expression coefficients until a preset number of training samples are obtained includes:

[0028] For each type of expression coefficient, if there is an empty array in the array corresponding to that type of expression coefficient, then the original sample in the copy array corresponding to that array is copied and placed in that array.

[0029] In an optional implementation, the step of determining the target accumulation coefficient based on the distribution of accumulation coefficients in the accumulation vector includes:

[0030] By comparing the magnitudes of the accumulation coefficients in the accumulation vector, the accumulation coefficients that meet the set requirements are determined;

[0031] The accumulation coefficient in the accumulation vector that exceeds a preset multiple of the accumulation coefficient that meets the set requirements is determined as the target accumulation coefficient.

[0032] In an optional implementation, the step of assigning a first weight to a portion of the target expression coefficients and a second weight to the other expression coefficients includes:

[0033] Arrange the target expression coefficients corresponding to the target cumulative coefficients in ascending order;

[0034] Each target expression coefficient after being arranged is assigned a first weight in turn until the cumulative target coefficient does not exceed a preset multiple of the cumulative coefficient that meets the set requirements;

[0035] All target expression coefficients other than those assigned the first weight, as well as all expression coefficients of other accumulated coefficients besides the target accumulated coefficients, are assigned the second weight.

[0036] In an optional implementation, the loss function is the product of the absolute value of the difference between the label vector of the training sample and the output vector of the model, and the weight matrix.

[0037] Secondly, this application provides a model training apparatus based on balanced samples, the apparatus comprising:

[0038] The acquisition module is used to acquire multiple training samples, each of which has a label vector, and the label vector contains multiple expression coefficients.

[0039] The module is used to calculate the cumulative coefficients based on the same type of expression coefficients of the multiple training samples, and to construct a cumulative vector from the obtained multiple cumulative coefficients;

[0040] The determination module is used to determine the target cumulative coefficient based on the distribution of the cumulative coefficients in the cumulative vector, and assign a first weight to a portion of the target expression coefficients of the target cumulative coefficients, and assign a second weight to the other expression coefficients, to obtain a weight matrix composed of all the first weights and the second weights.

[0041] The training module is used to train the constructed model using the multiple training samples, and to adjust the model and continue training under the guidance of the loss function composed of the weight matrix until the preset requirements are met and training stops.

[0042] Thirdly, this application provides an electronic device including one or more storage media and one or more processors communicating with the storage media. The one or more storage media store machine-executable instructions that can be executed by the processor. When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any of the foregoing embodiments.

[0043] Fourthly, this application provides a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the method steps described in any of the foregoing embodiments.

[0044] The beneficial effects of the embodiments of this application include, for example:

[0045] This application provides a model training method, apparatus, electronic device, and storage medium based on balanced samples. The method involves acquiring multiple training samples and calculating accumulated coefficients based on the same type of expression coefficients from these samples. An accumulated vector is then constructed from these accumulated coefficients. A target accumulated coefficient is determined based on the distribution of the accumulated coefficients within the accumulated vector. A portion of the target accumulated coefficient representing the target expression coefficient is assigned a first weight, while the remaining expression coefficients are assigned a second weight, resulting in a weight matrix composed of the first and second weights. When training the model using multiple training samples, the model is adjusted and trained further under the guidance of a loss function based on the weight matrix until a preset requirement is met. In this scheme, the distribution of expression coefficients in the training samples is used to assign different weights to the expression coefficients, and training is then performed under the guidance of a loss function that includes the weights. This ensures that the training is based on information from uniform expression coefficients in the samples, optimizing the model training effect based on samples that conform to real-world scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram illustrating an application scenario of the model training method provided in the embodiments of this application;

[0048] Figure 2 A flowchart of the model training method provided in the embodiments of this application;

[0049] Figure 3 A schematic diagram of the expression base provided in the embodiments of this application;

[0050] Figure 4 for Figure 2 A flowchart of the sub-steps included in step S101;

[0051] Figure 5 for Figure 4 A flowchart of the sub-steps included in step S1013;

[0052] Figure 6 for Figure 2 A flowchart of the sub-steps included in step S103;

[0053] Figure 7 for Figure 2 Another flowchart of the sub-steps included in step S103;

[0054] Figure 8A structural block diagram of the electronic device provided in the embodiments of this application;

[0055] Figure 9 A functional block diagram of the model training device provided in the embodiments of this application.

[0056] Icons: 100 - Live Streaming Provider; 200 - Live Streaming Server; 300 - Live Streaming Receiver; 410 - Memory; 420 - Processor; 430 - Model Training Device Based on Balanced Samples; 431 - Acquisition Module; 432 - Construction Module; 433 - Determination Module; 434 - Training Module. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0060] In the description of this application, it should be noted that the terms "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0061] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0062] The balanced sample-based model training method provided in this application can be applied to various scenarios. It is primarily used to train a model based on samples containing face images, and then utilize the trained model to achieve face recognition processing of images. For example, it can be used to train the model using surveillance images collected in a monitoring scenario, or it can be used to train the model using images of broadcasters collected in a live streaming scenario.

[0063] Please see Figure 1This is a schematic diagram of a possible application scenario provided by an embodiment of this application. The application scenario includes a live streaming provider 100, a live streaming receiver 300, and a live streaming server 200 that is communicatively connected to the live streaming provider 100 and the live streaming receiver 300, respectively.

[0064] The live streaming provider 100 can be a terminal device (such as a smartphone, tablet, or personal computer) used by the broadcaster during the live stream, and the live streaming receiver 300 can be a terminal device (such as a smartphone, tablet, or personal computer) used by the viewer while watching the live stream.

[0065] In this embodiment, the scenario may further include an image acquisition device for capturing video and images. This image acquisition device can be directly installed or integrated into the live streaming provider 100. For example, the image acquisition device can be a camera configured on the live streaming provider 100, and other modules or components in the live streaming provider 100 can receive image information sent from the image acquisition device via an internal bus. Alternatively, the image acquisition device can also be independent of the live streaming provider 100, with communication between the two occurring via wired or wireless means.

[0066] The live streaming provider 100 can send the live video stream to the live streaming server 200, which then pushes it to the live streaming receiver 300. The live streaming provider 100 can also analyze and process image frames contained in the captured images or videos, such as obtaining facial expression information. Alternatively, the processing of facial expression information in the images can also be performed in the live streaming server 200. Therefore, in this embodiment, the training of the model used to process facial expression information can be performed either in the live streaming provider 100 or in the live streaming server 200.

[0067] It should be noted that the above is only one possible application scenario of the model training method of this application, and does not limit the application scenario of the model training method to this.

[0068] Please see Figure 2 This application also provides a model training method based on balanced samples applicable to electronic devices, which can be either the aforementioned live streaming provider 100 or the aforementioned live streaming server 200. The method steps defined in the process related to the model training method based on balanced samples can be implemented by the electronic device. The following will describe... Figure 2 The specific process shown will be explained in detail.

[0069] S101, Obtain multiple training samples. Each training sample has a label vector, which contains multiple expression coefficients.

[0070] S102, calculate the cumulative coefficient based on the same type of expression coefficient of the multiple training samples, and construct the cumulative vector from the obtained multiple cumulative coefficients.

[0071] S103, determine the target cumulative coefficient according to the distribution of the cumulative coefficients in the cumulative vector, assign a first weight to a portion of the target expression coefficients of the target cumulative coefficients, and assign a second weight to the other expression coefficients, to obtain a weight matrix composed of all the first weights and the second weights.

[0072] S104, the constructed model is trained using the multiple training samples, and the model is adjusted and trained under the guidance of the loss function composed of the weight matrix until the preset requirements are met.

[0073] In this embodiment, the acquired training samples are historically generated samples containing face images. Each training sample is labeled with a label vector, which contains multiple expression coefficients. Multiple expression bases are pre-set, each of which exhibits one or more facial feature state changes relative to a standard expression template. For example, the standard expression template can be a face template in a neutral state, such as... Figure 3 The first emoji template is shown. For example... Figure 3 The expression base in the second row and first column is the expression base with the right eye state changed relative to the standard expression template. In this expression base, the right eye of the face is closed.

[0074] Each training sample's label vector contains expression coefficients that correspond to different expression bases. The expression coefficients reflect the correlation between the facial expressions in the training samples and their corresponding expression bases. For example, the more correlated the expressions, the closer the expression coefficient is to 1; the less correlated they are, the closer the expression coefficient is to 0. The final facial expressions presented in the training samples are the result of a weighted sum of multiple expression bases according to their correlation.

[0075] Among the multiple training samples obtained, each training sample has a label vector, and each label vector has multiple expression coefficients, such as 52 expression coefficients.

[0076] In conventionally collected training samples, there are a large number of samples with frequently occurring expressions, such as open-eyed expressions and closed-mouthed expressions, while there are fewer samples with less common expressions, such as closed-eye expressions and open-mouthed expressions. If the model is trained directly using conventionally collected training samples, the model will learn more feature information of common expressions, but there is a deficiency of undersampling when learning from a small number of samples.

[0077] In this embodiment, the aim is to train the model based on the information of the balanced distribution of samples, and to use the processing results of the balanced distribution of samples as a standard for judging the quality of the model training.

[0078] Each training sample has a multi-dimensional label vector containing multiple expression coefficients that can be arranged in a fixed order, for example, according to their respective correspondences as follows: Figure 3 The expression bases shown are arranged in order. The expression coefficients corresponding to the same expression base in each label vector can be used as the same type of expression coefficients. The same type of expression coefficients from multiple training samples can be summed to obtain the summed coefficient.

[0079] For example, by summing the first expression coefficients in each label vector, we obtain a summed coefficient. In this way, we can obtain a summed vector constructed from multiple summed coefficients.

[0080] Each cumulative coefficient can be seen as a combination of the correlation between multiple training samples and a certain expression base. For example, the more training samples that are correlated with a certain expression base, the larger the corresponding cumulative coefficient will be; or the higher the correlation between the training samples and a certain expression base, the larger the corresponding cumulative coefficient will be.

[0081] Therefore, the distribution of the accumulation coefficients in the accumulation vector can reflect the overall correlation between multiple training samples and each expression base.

[0082] Therefore, in this embodiment, the problem of training the model using samples with balanced distribution can be transformed into the problem of balancing the distribution of the accumulated vector.

[0083] In this embodiment, the target accumulation coefficient can be determined based on the distribution of the accumulation coefficients in the accumulation vector. The target accumulation coefficient can be an accumulation coefficient with a relatively prominent value in the accumulation vector, such as an accumulation coefficient with a large value. There can be one or more target accumulation coefficients.

[0084] Each target cumulative coefficient corresponds to a class of expression coefficients. This class of expression coefficients contains multiple target expression coefficients belonging to each training sample. Taking the above example, there are 52 target expression coefficients, each being the first element in each label vector.

[0085] To achieve a balanced distribution of the accumulated vector, a method can be adopted where a first weight is assigned to a portion of the target expression coefficients, while a second weight is assigned to the other expression coefficients. The first and second weights are different, and the first weight can be less than the second weight. For example, the first weight can be 0, and the second weight can be 1. The selection of the target expression coefficients assigned the first weight can be based on the distribution of all target expression coefficients. For instance, some target expression coefficients with smaller values ​​can be assigned the first weight, while others retain their default weights, i.e., the second weight.

[0086] By using the above method, a weight matrix consisting of all the first and second weights can be obtained.

[0087] Building upon the above, the constructed model can be trained using multiple training samples. This model can include, but is not limited to, a convolutional neural network model. The loss function used during model training includes the aforementioned weight matrix, where each element corresponds to a weight of a specific expression coefficient in the training samples. By incorporating this weight matrix into the loss function, the effective information from the training samples and the effective information from the model output, when guided by the loss function, are based on the weights contained in the weight matrix. In other words, the effective information used is based on the balanced information within the training samples.

[0088] Thus, by training the model under the guidance of the loss function containing the weight matrix, the information used to train the actual model is the information of the distribution balance in the training samples, which can optimize the model training effect based on samples that conform to the real scene.

[0089] In this embodiment, the training samples obtained above can be directly acquired, unprocessed image samples, or samples that have undergone certain processing based on directly acquired image samples. Please refer to... Figure 4 In one implementation, the acquisition of the multiple training samples described above can be achieved through the following steps:

[0090] S1011, Obtain multiple original samples, and for each type of expression coefficient in the multiple original samples, construct multiple arrays with an initial empty state for that type of expression coefficient, with each array corresponding to a sub-interval divided by the interval to which the expression coefficient belongs.

[0091] S1012: For each type of expression coefficient, iterate through each original sample and place each original sample into the array corresponding to the sub-interval to which the expression coefficient belongs.

[0092] S1013: For each type of expression coefficient, select original samples from the corresponding array as training samples until a preset number of training samples are obtained.

[0093] In this embodiment, multiple raw samples can be acquired. These raw samples can be directly acquired image samples, such as images of the broadcaster acquired by the image acquisition device of the live streaming provider 100, or images containing faces acquired through other means. Similarly, each raw sample has a label vector, and each label vector contains multiple expression coefficients.

[0094] For each type of expression coefficient, multiple arrays with an initial empty state can be constructed. Taking the above example, each label vector contains 52 expression coefficients, that is, it contains 52 types of expression coefficients. Assuming there are a total of 500 original samples, 10 arrays with an initial empty state can be constructed for each type of expression coefficient, so 10*500 arrays can be constructed in advance.

[0095] Among them, the 10 arrays for various expression coefficients correspond to multiple sub-intervals divided by the interval to which the expression coefficient belongs. For example, if the interval to which various expression coefficients belong is [0,1], that is, the value of each expression coefficient is between 0 and 1, then the sub-intervals corresponding to the 10 arrays can be [0,0.1), [0,1,0.2)...[0.9,1].

[0096] For each type of expression coefficient, the original samples can be iterated through, and the training samples can be placed into the corresponding arrays according to the sub-interval to which the expression coefficient belongs. For example, for the first type of expression coefficient, when iterating through the first original sample, if the first expression coefficient of the original sample is 0.05, then the original sample is placed into the array with the corresponding sub-interval [0, 0.1). When each original sample has 52 expression coefficients, then each original sample needs to perform 52 placement operations.

[0097] By using the above placement actions, for each type of expression coefficient, the corresponding array can be used to represent the distribution of each original sample from the perspective of each expression coefficient.

[0098] Thus, for various expression coefficients, original samples can be selected from the corresponding array as training samples. The selection method can be set according to needs, such as selecting those with moderate numerical distribution or selecting according to the order of the array. The selection process can be stopped when the number of selected training samples reaches a preset number.

[0099] By placing the original samples into different arrays according to the values ​​of their various expression coefficients, the original samples can be divided into different arrays based on these expression coefficients. Then, a predetermined number of training samples can be obtained by sequentially selecting from these arrays. In this way, selection can be performed at the granularity of the expression coefficients contained within the original samples, rather than at the granularity of the original samples themselves, thus achieving a balance in terms of expression coefficients.

[0100] Please see Figure 5 In one possible implementation, the selection of training samples can be achieved in the following way:

[0101] S10131, for each type of expression coefficient, select an original sample from its corresponding array in turn.

[0102] S10132, check whether the selected original sample is the same as any of the currently determined training samples. If so, repeat step S10133; otherwise, repeat step S10134.

[0103] S10133, execute the selection of the next array.

[0104] S10134, the selected original samples are used as training samples.

[0105] S10135, when the number of training samples obtained reaches the preset number, stop the selection action.

[0106] In this embodiment, when selecting training samples, they can be selected sequentially according to the order of the expression coefficients in the label vector, or multiple indices can be randomly generated, such as random indices from 1 to 52, and the indices can be attached to various expression coefficients. The selection is then performed sequentially in ascending order of the indices.

[0107] When selecting a certain type of expression coefficient, there should be multiple arrays corresponding to that expression coefficient. The selection can be performed sequentially according to the order of the arrays, but only one original sample is selected in each round. For example, an original sample can be selected from the first array corresponding to it, and then the selection of the next expression coefficient can be performed until the first array of the last expression coefficient is selected, thus completing one round of selection.

[0108] In the next round of selection, the selection begins from the second array of the first type of expression coefficients, and continues until the second array of the last type of expression coefficients is selected, thus completing the second round of selection. This process is repeated until a predetermined number of training samples are obtained, at which point the selection process stops.

[0109] When selecting each original sample, since each original sample may appear repeatedly in multiple arrays, the original sample selected from a certain array may have been pre-selected and determined as a training sample. Therefore, when selecting an original sample, it is possible to check for repetition. If repetition indicates that it has been pre-selected, the selection of that array in that round can be abandoned, and the selection of the next array can be performed subsequently.

[0110] Furthermore, since each array is selected in multiple rounds sequentially, if the preset number of training samples has not yet been obtained, but all original samples in a certain array are selected and the array becomes empty, in this case, given that multiple arrays have been obtained as described above, the multiple arrays can be copied in advance to obtain corresponding copied arrays.

[0111] Based on this, when selecting original samples from the array as training samples, for each type of expression coefficient, if there is an empty array in the array corresponding to that expression coefficient, the original samples in the corresponding copy array are copied and placed into that array. Then, the samples are selected from the copied array.

[0112] In this embodiment, before selecting training samples, the obtained array can be pre-processed. As mentioned above, the sub-intervals corresponding to the arrays for each type of expression coefficient are distributed from smallest to largest. The magnitude of the expression coefficient reflects the correlation between facial expressions and the expression base; the smaller the expression coefficient, the lower the correlation, and the larger the expression coefficient, the higher the correlation. In conventionally collected samples, facial expressions are generally neutral, meaning the expression coefficients are generally small. Therefore, a large proportion of conventionally collected samples have small expression coefficients.

[0113] To avoid the situation where a large proportion of the original samples with relatively small expression coefficients are more likely to be selected later, thus affecting the sample balance, we can delete the array with the smallest sub-interval for each type of expression coefficient. For example, we can delete the array with the sub-interval [0, 0.1).

[0114] The resulting array may contain some arrays that do not contain any samples. To avoid meaningless operations during subsequent selection, empty arrays can be deleted from multiple arrays.

[0115] Furthermore, due to the large number of arrays obtained, some original samples may be selected multiple times during the subsequent selection of training samples, thus requiring multiple detection and judgment actions. Therefore, in this embodiment, for each type of expression coefficient, an array containing fewer than a set number of original samples can be determined. The array containing fewer than the set number of original samples is then merged with its adjacent arrays so that the number of original samples in the merged array is greater than or equal to the set number.

[0116] The number of elements can be set based on requirements, such as three, four, or any other number. When merging, adjacent arrays can be merged according to the order in which they were distributed.

[0117] In this way, arrays containing a small number of original samples can be merged, thus avoiding the repeated selection of some original samples in subsequent selections.

[0118] In this embodiment, by grouping the original samples based on the original samples with expression coefficients as the granularity, and selecting a preset number of training samples from them, a balanced training sample can be obtained from the granularity of expression coefficients.

[0119] Based on the obtained training samples, the cumulative vector is obtained according to the expression coefficients in the training samples as described above. The weight matrix is ​​obtained based on the distribution of the cumulative vector, and the model is trained based on the weight matrix.

[0120] In this embodiment, when constructing the accumulated vector based on the obtained training samples, each training sample has a label vector. Assuming that each label vector contains 52 expression coefficients, the label vector can be denoted as B = [b1, b2, ... b...]. 52 ], b i ∈[0,1]. If there are n training samples, the label vector of the n training samples can be denoted as [B1,B2,…B…]. n ].

[0121] The accumulation vector can be defined as P = [p1, p2, ... p2]. 52 ],in That is, each cumulative coefficient in the cumulative vector is the cumulative value of the same type of expression coefficients in n training samples.

[0122] Please see Figure 6 In step S103 above, when determining the target accumulation coefficient based on the distribution of accumulation coefficients in the accumulation vector, it can be achieved in the following way:

[0123] S1031, compare the magnitudes of the accumulation coefficients in the accumulation vector to determine the accumulation coefficients that meet the set requirements.

[0124] S1032, the accumulation coefficient in the accumulation vector that exceeds a preset multiple of the accumulation coefficient that meets the set requirements is determined as the target accumulation coefficient.

[0125] When evaluating whether the data of a batch of training samples is uniform, this can be achieved by determining whether the differences between the accumulation coefficients in the accumulation vector formed by the batch of training samples are controlled within a certain range. In this embodiment, an accumulation coefficient that meets the set requirements can be found among multiple accumulation coefficients. This accumulation coefficient that meets the set requirements can be the second smallest accumulation coefficient or the third smallest accumulation coefficient, etc.

[0126] Based on the cumulative coefficient that meets the set requirements, the cumulative coefficient that exceeds the cumulative coefficient that meets the set requirements by a preset multiple is determined as the target cumulative coefficient, such as exceeding the cumulative coefficient that meets the set requirements by 1.5 times or 2 times.

[0127] The determined target cumulative coefficient may be one or more. Based on this, in step S103 above, when assigning weights to the target expression coefficients corresponding to the target cumulative coefficients and to other expression coefficients, this can be achieved in the following way (please refer to the relevant documentation). Figure 7 :

[0128] S1033, Arrange the target expression coefficients corresponding to the target cumulative coefficients in ascending order.

[0129] S1034, assign a first weight to each of the arranged target expression coefficients in sequence until the target cumulative coefficient does not exceed the preset multiple of the cumulative coefficient that meets the set requirements.

[0130] S1035, assign the second weight to all target expression coefficients other than the first weight in the target cumulative coefficient, as well as all expression coefficients of other cumulative coefficients other than the target cumulative coefficient.

[0131] With n training samples, each accumulated coefficient will correspond to n expression coefficients, representing expression coefficients belonging to the same class in the n training samples, i.e., expression coefficients corresponding to the same expression base. The target expression coefficients corresponding to the target accumulated coefficient can be arranged in ascending order. When assigning weights to the target expression coefficients, they can be assigned the first weight sequentially in ascending order, for example, 0. Furthermore, after assigning the first weight to each target expression coefficient in each round, the multiple between the target accumulated coefficient and the accumulated coefficient meeting the preset requirements is recalculated. If the target accumulated coefficient is no longer greater than the preset multiple of the accumulated coefficient meeting the set requirements, the assignment of the first weight to the target expression coefficient corresponding to that target accumulated coefficient can be stopped. Then, the other target expression coefficients corresponding to that target accumulated coefficient that were not assigned the first weight are assigned the second weight.

[0132] Among the multiple cumulative coefficients, those other than the target cumulative coefficient can be assigned a second weight, such as 1, to their corresponding expression coefficients.

[0133] In this embodiment, the reason for assigning weights of 0 to the target expression coefficients in ascending order is that the expression coefficients contained in common training samples are generally small in value. Therefore, assigning weights of 0 to these small expression coefficients can reduce their importance in subsequent model training.

[0134] Using the above method, a weight matrix of the following form can be obtained:

[0135]

[0136] When training the model, if the loss function includes the weight matrix mentioned above, the loss of the expression coefficients corresponding to m(i,j) being 0 in the training samples will not be included, thus not affecting the model's learning.

[0137] In this embodiment, the constructed loss function can be the product of the absolute value of the difference between the label vector of the training sample and the output vector of the model, and the aforementioned weight matrix.

[0138] In one implementation, the loss function can be constructed as follows:

[0139]

[0140] Where n represents the number of training samples, MASK represents the matrix weights, and Y represents the model's output vector. This represents the label vector of the training sample.

[0141] Guided by the aforementioned loss function, multiple batches of training samples can be used for iterative training. After each iteration, the model parameters are adjusted and training continues until a preset requirement is met, at which point training stops. This preset requirement could be, for example, the loss function reaching convergence, the number of iterations reaching a set maximum, or the training duration reaching a set duration.

[0142] The model trained in the above manner can be used in the application stage for the recognition and processing of images. The image to be processed can be imported into the trained model. After the model's recognition and processing, an output vector of the image to be processed can be obtained. This output vector contains multiple expression coefficients corresponding to the facial expressions in the image to be processed. These multiple expression coefficients correspond to, respectively, as shown in the figure below. Figure 3 The diagram shows multiple facial expression bases. The resulting output vectors can be used for other purposes requiring analysis and processing of facial expressions.

[0143] The model training method provided in this embodiment can construct an accumulated vector that represents the overall training samples at the granularity of expression coefficients, and then assign different weights to the expression coefficients in the accumulated vector. Thus, during model training, each expression coefficient can have a different impact on the model according to its weight, so that the model can be trained under the influence of uniform information, meet the sample information in real-world scenarios, and improve the training effect of the model.

[0144] Furthermore, the training samples are selected based on the original samples collected, with the expression coefficient as the granularity. This allows for the preprocessing of the original samples to achieve a certain degree of uniform distribution, thereby improving the balance effect that can be achieved by the subsequent matrix weights.

[0145] Please see Figure 8 This application also provides an electronic device. This electronic device can be a live streaming device, for example, a terminal device used by a broadcaster during live streaming (such as the live streaming provider 100 mentioned above), or a server that communicates with the terminal device used by the broadcaster during live streaming (such as the live streaming server 200 mentioned above).

[0146] In detail, the electronic device may include a memory 410, a processor 420, and a model training device 430 based on balanced samples. The memory 410 and the processor 420 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, they can be electrically connected via one or more communication buses or signal lines. The model training device 430 based on balanced samples includes at least one software functional module that can be stored in the memory 410 in the form of software or firmware. The processor 420 is used to execute executable computer programs stored in the memory 410, such as the software functional modules and computer programs included in the model training device 430 based on balanced samples, to achieve model training of balanced samples with the expression coefficients of the training samples as the granularity.

[0147] The memory 410 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0148] The memory 410 stores the program, and the processor 420 executes the program after receiving an execution instruction. The processor 420 may be an integrated circuit chip with signal processing capabilities. For example, it may be a central processing unit (CPU), a network processor (NP), a system-on-chip (SoC), a digital signal processor (DSP), etc., to implement or execute the methods and steps disclosed in the embodiments of this application.

[0149] Understandable. Figure 8 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown may also include, for example, a communication unit for exchanging information with other live streaming devices. Figure 8 The components shown can be implemented using hardware, software, or a combination thereof.

[0150] Please see Figure 9 This is a functional block diagram of a balanced sample-based model training device 430 provided in an embodiment of this application. The balanced sample-based model training device 430 includes an acquisition module 431, a construction module 432, a determination module 433, and a training module 434.

[0151] The acquisition module 431 is used to acquire multiple training samples, each of which has a label vector containing multiple expression coefficients.

[0152] In this embodiment, the acquisition module 431 can be used to perform... Figure 2 The relevant content regarding the acquisition module 431 in step S101 shown can be found in the previous description of step S101.

[0153] Construction module 432 is used to calculate the cumulative coefficient based on the same type of expression coefficients of the multiple training samples, and construct the cumulative vector from the obtained multiple cumulative coefficients;

[0154] In this embodiment, the construction module 432 can be used to execute Figure 2 The relevant content regarding the construction module 432 in step S102 shown can be found in the previous description of step S102.

[0155] The determination module 433 is used to determine the target cumulative coefficient according to the distribution of the cumulative coefficients in the cumulative vector, and assign a first weight to a portion of the target expression coefficients of the target cumulative coefficients, and assign a second weight to the other expression coefficients, to obtain a weight matrix composed of all the first weights and the second weights.

[0156] In this embodiment, the determining module 433 can be used to perform... Figure 2 The relevant content regarding the determining module 433 in step S103 shown can be found in the previous description of step S103.

[0157] The training module 434 is used to train the constructed model using the multiple training samples, and to adjust the model under the guidance of the loss function composed of the weight matrix and continue training until the preset requirements are met and training stops.

[0158] In this embodiment, the training module 434 can be used to perform... Figure 2 The relevant content regarding the training module 434 in step S104 shown can be found in the previous description of step S104.

[0159] In one possible implementation, the aforementioned acquisition module 431 can be specifically used for:

[0160] Obtain multiple raw samples. For each type of expression coefficient in the multiple raw samples, construct multiple arrays with an initial empty state for that type of expression coefficient. Each array corresponds to a sub-interval divided by the interval to which the expression coefficient belongs.

[0161] For each type of expression coefficient, iterate through each original sample and place each original sample into the array corresponding to the sub-interval to which the expression coefficient belongs;

[0162] For each type of expression coefficient, original samples are selected from the corresponding array as training samples until a preset number of training samples are obtained.

[0163] In one possible implementation, the sub-intervals corresponding to the various expression coefficients are distributed in ascending order. The aforementioned acquisition module 431 can also be used for:

[0164] For each type of expression coefficient, delete the array with the smallest corresponding sub-interval;

[0165] Delete empty arrays from multiple arrays.

[0166] In one possible implementation, the acquisition module 431 described above can also be used for:

[0167] For each type of expression coefficient, determine the corresponding array containing an array with a number of original samples less than a set number;

[0168] An array containing fewer than a set number of original samples is merged with its neighboring arrays so that the number of original samples in the merged array is greater than or equal to the set number.

[0169] In one possible implementation, the aforementioned acquisition module 431 can be specifically used for:

[0170] For each type of expression coefficient, a raw sample is selected from its corresponding array in turn, and it is checked whether the selected raw sample is the same as any of the currently determined training samples. If they are the same, the next array is selected. If they are not the same, the selected raw sample is used as the training sample.

[0171] The selection process stops when the number of training samples reaches the preset number.

[0172] In one possible implementation, the acquisition module 431 described above can also be used for:

[0173] Copy each of the obtained arrays to get the corresponding copied arrays;

[0174] For each type of expression coefficient, if there is an empty array in the array corresponding to that type of expression coefficient, then the original sample in the copy array corresponding to that array is copied and placed in that array.

[0175] In one possible implementation, the determining module 433 described above can be used to:

[0176] By comparing the magnitudes of the accumulation coefficients in the accumulation vector, the accumulation coefficients that meet the set requirements are determined;

[0177] The accumulation coefficient in the accumulation vector that exceeds a preset multiple of the accumulation coefficient that meets the set requirements is determined as the target accumulation coefficient.

[0178] In one possible implementation, the determining module 433 described above can be used to:

[0179] Arrange the target expression coefficients corresponding to the target cumulative coefficients in ascending order;

[0180] Each target expression coefficient after being arranged is assigned a first weight in turn until the cumulative target coefficient does not exceed a preset multiple of the cumulative coefficient that meets the set requirements;

[0181] All target expression coefficients other than those assigned the first weight, as well as all expression coefficients of other accumulated coefficients besides the target accumulated coefficients, are assigned the second weight.

[0182] In one possible implementation, the loss function is the product of the absolute value of the difference between the label vector of the training samples and the output vector of the model, and the weight matrix.

[0183] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0184] In this application embodiment, corresponding to the above-described model training method based on balanced samples, a computer-readable storage medium is also provided, which stores a computer program that executes the various steps of the above-described model training method based on balanced samples when the computer program is run.

[0185] The steps executed by the aforementioned computer program during runtime will not be described in detail here, but can be found in the explanation of the balanced sample-based model training method described above.

[0186] In summary, the model training method, apparatus, electronic device, and storage medium based on balanced samples provided in this application acquire multiple training samples and calculate accumulated coefficients based on the same type of expression coefficients from the multiple training samples, constructing an accumulated vector from the multiple accumulated coefficients. A target accumulated coefficient is determined based on the distribution of the accumulated coefficients in the accumulated vector. A portion of the target accumulated coefficient representing the target expression coefficient is assigned a first weight, and the remaining expression coefficients are assigned a second weight, resulting in a weight matrix composed of the first and second weights. When training the model using multiple training samples, the model is adjusted and trained further under the guidance of a loss function composed of the weight matrix until a preset requirement is met. In this scheme, the distribution of expression coefficients in the training samples is used to assign different weights to the expression coefficients, and training is performed under the guidance of a loss function containing weights. This ensures that the training is based on the information of uniform expression coefficients in the samples, optimizing the model training effect based on samples that conform to real-world scenarios.

[0187] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method based on balanced samples, characterized in that, The method comprises: obtaining a plurality of training samples, each of the training samples having a label vector containing a plurality of expression coefficients, the training samples being samples containing face images generated in history, each of the expression coefficients corresponding to each expression base and embodying the correlation between the face expression in the training sample and the corresponding expression base; calculating an accumulated coefficient according to the same class expression coefficient of the plurality of training samples, constructing an accumulated vector from the obtained plurality of accumulated coefficients, and the distribution of the accumulated coefficients in the accumulated vector embodying the comprehensive information of the correlation of the plurality of training samples with respect to each expression base; determining a target accumulated coefficient according to the distribution of the accumulated coefficients in the accumulated vector, the target accumulated coefficient being one or more of the plurality of accumulated coefficients with larger values; assigning a first weight to part of the target expression coefficients of the target accumulated coefficient and a second weight to other expression coefficients to obtain a weight matrix composed of all the first weights and the second weights, the first weight being smaller than the second weight; training the constructed model using the plurality of training samples, adjusting and continuing to train the model under the guidance of the loss function composed of the weight matrix until the training is stopped when a preset requirement is met, so that the model is trained using the distribution-balanced information in the plurality of training samples, wherein the trained model is used to realize the recognition processing of the image containing the face. 2.The model training method based on balanced samples according to claim 1, characterized in that, The step of obtaining a plurality of training samples comprises: obtaining a plurality of original samples, for each class of expression coefficients in the plurality of original samples, constructing a plurality of arrays with an initial state of null for the class of expression coefficients, each of the arrays corresponding to each sub-interval divided by the sub-interval to which the class of expression coefficients belongs; traversing each original sample for each class of expression coefficients, and placing each original sample in the array corresponding to the sub-interval to which the expression coefficient belongs; for each class of expression coefficients, selecting original samples from the corresponding array as training samples until a preset number of training samples are obtained. 3.The model training method based on balanced samples according to claim 2, characterized in that, The sub-intervals corresponding to the arrays of each class of expression coefficients are distributed in order from small to large; Before the step of selecting original samples from the corresponding array as training samples for each class of expression coefficients, the step of obtaining a plurality of training samples further comprises: for each class of expression coefficients, deleting the array with the smallest sub-interval in the corresponding array; deleting the array with null in the plurality of arrays. 4.The method of claim 2, wherein, Before the step of selecting original samples from the corresponding array as training samples for each class of expression coefficients, the step of obtaining a plurality of training samples further comprises: for each class of expression coefficients, determining the array containing less than a set number of original samples in the corresponding array; merging the array containing less than a set number of original samples with its adjacent array so that the number of original samples in the merged array is greater than or equal to the set number. 5.The model training method based on balanced samples according to claim 2, characterized in that, The step of selecting original samples from the corresponding array as training samples for each class of expression coefficients until a preset number of training samples are obtained comprises: Selecting an original sample from each array corresponding to each type of expression coefficient in sequence, and detecting whether the selected original sample is repeated with any of the currently determined training samples, if repeated, selecting an original sample from the next array, if not repeated, taking the selected original sample as a training sample; Stopping the selecting action when the number of obtained training samples reaches a preset number.

6. The balanced sample based model training method of claim 5, wherein, The step of obtaining multiple training samples further includes: Copying each array to obtain a corresponding copied array; The step of selecting an original sample from each array corresponding to each type of expression coefficient as a training sample until a preset number of training samples are obtained further includes: If there is an empty array in the array corresponding to each type of expression coefficient, copying the original sample in the copied array corresponding to the array and placing it in the array. 7.The balanced sample based model training method of claim 1, wherein, The step of determining a target accumulation coefficient according to the distribution of accumulation coefficients in the accumulation vector includes: Comparing the size of the accumulation coefficients in the accumulation vector to determine the accumulation coefficients that meet the set requirements; Determining the accumulation coefficients in the accumulation vector that exceed the preset multiple of the accumulation coefficients that meet the set requirements as target accumulation coefficients. 8.The balanced sample based model training method of claim 7, wherein, The step of assigning a first weight to part of the target expression coefficients of the target accumulation coefficient and assigning a second weight to other expression coefficients includes: Arranging each target expression coefficient corresponding to the target accumulation coefficient in ascending order; Assigning a first weight to each target expression coefficient in sequence until the target accumulation coefficient does not exceed the preset multiple of the accumulation coefficients that meet the set requirements; Assigning a second weight to all expression coefficients other than the target expression coefficients of the target accumulation coefficient and other accumulation coefficients other than the target accumulation coefficient.

9. The balanced sample based model training method according to any one of claims 1-8, characterized in that, The loss function is the absolute value of the difference between the label vector of the training sample and the output vector output by the model, multiplied by the weight matrix. 10.A device for model training based on balanced samples, characterized in that, The device for implementing the model training method based on balanced samples according to any one of claims 1-9 includes: An acquisition module for acquiring multiple training samples, each training sample having a label vector containing multiple expression coefficients; A construction module for calculating accumulation coefficients according to the same type of expression coefficients of the multiple training samples, and constructing an accumulation vector from the obtained multiple accumulation coefficients; A determination module for determining a target accumulation coefficient according to the distribution of accumulation coefficients in the accumulation vector, and assigning a first weight to part of the target expression coefficients of the target accumulation coefficient and a second weight to other expression coefficients to obtain a weight matrix composed of all first weights and second weights; A training module for training the constructed model using the multiple training samples, adjusting and continuing to train the model under the guidance of the loss function composed of the weight matrix until the training stops when the preset requirements are met.

11. An electronic device, comprising: The electronic device comprises one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores machine executable instructions, and the machine executable instructions are executed to implement the method steps of any one of claims 1-9.

Images