Data processing method and device, computer equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, computer equipment and a computer readable storage medium, wherein the method comprises: performing first processing on first facial feature data corresponding to a target frame in a video, the first processing comprising: determining a target expression coefficient corresponding to the target frame according to the first facial feature data, a regression model and adjustment information; updating the adjustment information according to the target expression coefficient; wherein the first facial feature data is input data of the regression model; the target expression coefficient is output data obtained based on the regression model; the adjustment information affects the output data of the regression model; and sequentially taking multiple frame images in the video as target frames to perform the first processing.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] With the development of the metaverse, the applications of virtual and digital humans are becoming increasingly widespread and diverse, playing a vital role in social, gaming, and office scenarios. This also places higher demands on the accuracy and vividness of facial expression recognition for digital humans. Among related technologies, expression prediction models are used to predict facial expressions in images. However, due to significant individual differences in facial features and expressions, the accuracy of these prediction models remains relatively low. Summary of the Invention

[0003] In view of the above, embodiments of this application provide at least one data processing method, apparatus, computer device, and computer-readable storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0006] A first processing step is performed on the first facial feature data corresponding to a target frame in the video. The first processing step includes: determining a target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model.

[0007] The first process is performed on multiple frames of the video as target frames in sequence.

[0008] In some embodiments, the adjustment information is used for at least one of the following:

[0009] Adjust the output data of the regression model;

[0010] Adjust the output data of the regression model;

[0011] Adjustments are made to the intermediate calculation data of the regression model.

[0012] In some embodiments, updating the adjustment information based on the target expression coefficient includes:

[0013] If the target expression coefficient exceeds the preset output data range corresponding to the regression model, the adjustment information is updated.

[0014] In some embodiments, updating the adjustment information includes at least one of the following:

[0015] The adjustment information is updated based on the preset value;

[0016] The adjustment information is updated based on the target expression coefficient and the preset output data range.

[0017] In some embodiments, the adjustment information includes at least one of the following: translation parameters and scaling parameters; wherein,

[0018] The translation parameter is used to adjust the size of the output value of the regression model;

[0019] The scaling parameter is used to amplify or reduce the output value of the regression model by a specific factor.

[0020] In some embodiments, where the adjustment information includes the translation parameters...

[0021] The step of determining the target expression coefficient corresponding to the target frame based on the first facial feature data, regression model, and adjustment information includes:

[0022] Using the translation parameters before the update, the first facial feature data is translated to obtain the second facial feature data; wherein, the translation parameters before the update are determined based on the facial feature data corresponding to the first frame in the video, and the first frame is an image frame before the target frame.

[0023] Based on the second facial feature data and the regression model, the target expression coefficient is determined;

[0024] Updating the adjustment information based on the target expression coefficient includes:

[0025] Based on the minimum input value of the regression model, the first facial feature data, and the translation parameters before the update, a first adjustment factor is determined;

[0026] The translation parameters before the update are updated using the first adjustment factor to obtain the updated translation parameters.

[0027] In some embodiments, when the adjustment information includes the scaling parameter, the scaling parameter includes at least one of the following: a first scaling parameter and a second scaling parameter;

[0028] Updating the adjustment information based on the target expression coefficient includes at least one of the following:

[0029] If the target expression coefficient is greater than the maximum model output value of the regression model, the first scaling parameter is used as the scaling factor, and the first scaling parameter is determined based on the first facial feature data.

[0030] If the target expression coefficient is greater than the maximum value in the output dataset, the second scaling parameter is used as the scaling parameter, and the second scaling parameter is determined based on the target expression coefficient; wherein, the output dataset includes the target expression coefficient corresponding to at least one second image frame in the video, and each second image frame is an image frame preceding the target frame.

[0031] In some embodiments, determining the target expression coefficient corresponding to the target frame based on the first facial feature data, the regression model, and the adjustment information includes:

[0032] The first intermediate result of the regression model is obtained based on the translation parameters;

[0033] The second result of the regression model is obtained based on the first intermediate result and the scaling parameter;

[0034] The target expression coefficient is determined based on the second result.

[0035] On the other hand, embodiments of this application also provide a data processing apparatus, including:

[0036] A data processing module is used to perform a first processing on first facial feature data corresponding to a target frame in a video. The first processing includes: determining a target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model.

[0037] The data processing module is further configured to sequentially use multiple frame images from the video as target frames to perform the first processing.

[0038] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0039] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0040] The data processing method, apparatus, computer device, and computer-readable storage medium provided in this application embodiment perform a first processing on a target frame in a video. This first processing includes: determining a target expression coefficient corresponding to the target frame based on first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient, wherein the adjustment information affects the output data of the regression model; and then sequentially using multiple frame images in the video as target frames to perform the first processing. Thus, when using a regression model to predict expression coefficients on faces in a video, even if the faces in the video differ significantly from the training data used to train the regression model, by performing a limited number of updates to the adjustment information, the first facial feature data corresponding to the target frame can be dynamically mapped to the input range of the regression model, and output data within the inherent output range of the regression model can be obtained. This solves the problem of model adaptability on different faces efficiently and at low cost without expanding the model training dataset. Furthermore, since this application embodiment dynamically adjusts the data, virtual or digital faces can be generated during facial image capture without needing to obtain the user's expressionless state.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0043] Figure 1 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the application flow of a data processing method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0051] In related technologies, the realization of facial expressions in virtual or digital humans typically involves key technologies such as face capture, key point detection, 3D reconstruction, expression coefficient calculation, and animation binding and driving. Among these, facial key point detection and the calculation of basic facial expression coefficients (Blendshape) using regression methods have become one of the important methods for facial rendering in virtual or digital humans.

[0052] With a blank expression as the initial state, any expression can be represented as a superposition of the initial state and each local expression. Based on this, the expression coefficients are a set of predefined facial states and scaling constants used to represent facial expressions, as shown in the following formula (1):

[0053] f=ω0f0+...+ω i f i +...+ω n f n (1);

[0054] Where f represents the expression coefficient; ω i f represents the scaling constant for the i-th local expression; i This represents the i-th local expression; n represents the number of local expressions, where n is a positive integer.

[0055] In related technologies, expression coefficients are predicted based on the original facial mesh or facial key points using machine learning (e.g., regression operations) or deep learning methods. However, due to significant individual differences in facial features and expressions among different people, training the model presents considerable challenges. Therefore, when using expression prediction models to predict expression coefficients for different faces, one of the main challenges and bottlenecks in promoting the application of this facial calculation method is how to match the predicted expression coefficients of different faces as closely as possible to the inherent range of the base expression coefficients (e.g., the range of [0,1]).

[0056] One proposed solution to this problem is to expand the training dataset to improve the applicability of the expression prediction model. To achieve the desired expression prediction accuracy using this approach, it would be necessary to collect a large number of different face images and images of different facial expressions, obtain the facial key points for each image, and label the expression coefficients for each image. However, currently there are no large open-source datasets available for model training, and building a training dataset from scratch is costly. Therefore, this approach has low feasibility.

[0057] Based on this, embodiments of this application provide a data processing method. This method uses a regression model and adjustment information to determine the target expression coefficient corresponding to the target frame in the video, and uses the target expression coefficient to update the adjustment information. In the process of using the regression model to predict the expression coefficient, the adjustment information is continuously updated so that the output data of the regression model can match the inherent range of the base expression coefficient of the regression model. This makes the regression model highly adaptable to different faces and can obtain high accuracy in predicting expression coefficients for different faces.

[0058] The data processing method provided in this application embodiment can be executed by the processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device).

[0059] Figure 1 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps S101 to S102:

[0060] Step S101: Perform a first processing on the first facial feature data corresponding to the target frame in the video. The first processing includes: determining the target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model.

[0061] Here, the video can be captured in real time by a local camera device, or it can be a video that has already been captured and stored on a local storage device, or it can be received from other video storage devices through a communication interface.

[0062] A target frame is an image frame containing the face of a human or other organism. By recognizing the face in this target frame, the facial expression coefficients of the human or other organism can be determined.

[0063] The first facial feature data is facial feature data identified from the target frame, such as facial key point data. The first facial feature data may include one-dimensional data, two-dimensional data, or three-dimensional data. For example, in the case of blinking, only one-dimensional data from the upper eyelid to the lower eyelid can be used to represent the eye feature data; in the case of smiling, two-dimensional or three-dimensional data can be used to represent the eye feature data.

[0064] In some embodiments, the regression model can be a linear regression model, which can be abstractly represented by the following formula (2):

[0065] y = kx + b (2);

[0066] Where k is the weight value of the regression model; b is the function bias or offset of the regression model; x is the input data of the regression model, such as facial regression data; and y is the output value of the regression model, such as the predicted facial expression coefficient.

[0067] The adjustment information is used to adjust the input data, intermediate calculation results, and model output data of the regression model. Specifically, it adjusts the facial feature data or the intermediate results and model output data calculated by the regression model based on the facial feature data to obtain data that meets the input and output requirements of the regression model. After updating the adjustment information according to the target expression coefficient, the updated adjustment information is applied to the first processing of subsequent frames of the target frame, thereby achieving continuous dynamic adjustment of the adjustment information and gradually increasing the accuracy of the calculation of facial expression coefficients from front to back in the video. Optionally, the adjustment information includes multiple adjustment parameters, which adjust at least two of the input data, intermediate calculation results, and model output data of the regression model, thereby increasing the adjustment dimensions and improving the overall adjustment effect.

[0068] In practical applications, the adjustment information can be determined based on the first facial feature data, regression model, and adjustment information to determine the target expression coefficient corresponding to the target frame.

[0069] Since the faces in the current video may differ significantly from the facial data used to train the regression model, when using the regression model to predict the expression coefficients of faces in the video, it is necessary to update the adjustment information based on the target expression coefficients so that the facial feature data determined based on the updated adjustment information meets the input requirements of the regression model and the output expression coefficients can be mapped to the inherent output range of the regression model (e.g., the output predicted expression coefficients are in the range of [0,1]).

[0070] Step S102: The first processing is performed on multiple frame images in the video as target frames in sequence.

[0071] Here, the first processing is performed sequentially on multiple frames of images in the video to obtain the target expression coefficient corresponding to each frame, and then the expression changes of the virtual human or digital human are controlled based on the target expression coefficient corresponding to each frame.

[0072] In this embodiment, a first processing step is performed on a target frame in a video. This first processing step includes: determining the target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; and updating the adjustment information based on the target expression coefficient, wherein the adjustment information affects the output data of the regression model. Then, multiple frames in the video are sequentially used as target frames for the first processing. Thus, when using a regression model to predict the expression coefficient of a face in a video, even if the face in the video differs significantly from the training data used to train the regression model, by updating the adjustment information a limited number of times (e.g., 10 frames), the first facial feature data corresponding to the target frame can be dynamically mapped to the inherent input range of the regression model, and output data within the inherent output range of the regression model can be obtained. This solves the problem of model adaptability to different faces efficiently and at low cost without expanding the model training dataset. Furthermore, since the adjustment information is dynamically adjusted in this embodiment, virtual or digital faces can be generated during facial image capture without needing to obtain the user's expressionless state.

[0073] In some embodiments, the adjustment information is used for at least one of the following:

[0074] Adjust the input data for the regression model;

[0075] Adjust the output data of the regression model;

[0076] Adjustments are made to the intermediate calculation data of the regression model.

[0077] Here, the input data of the regression model is adjusted using adjustment information; that is, the first facial feature data corresponding to the target frame is adjusted using adjustment information.

[0078] In some embodiments, adjusting the first facial feature data using adjustment information may include at least one of the following:

[0079] The first facial feature data is translated to fit within the inherent input range of the regression model. In some embodiments, the facial feature data corresponding to the first frame input into the regression model can be translated to the starting point of the inherent input range of the regression model. Taking the inherent input range of the regression model as [0,1] as an example, translating the facial feature data corresponding to the first frame to the starting point of the inherent input range of the regression model means translating the facial feature data corresponding to the first frame to point 0.

[0080] The first facial feature data is scaled to reduce or increase its size to fit within the inherent input range of the regression model.

[0081] Adjusting the output data of a regression model using adjustment information is used to obtain the target expression coefficient. For example, when the output data range of the regression model is much larger or smaller than the inherent output range of the regression model, the adjustment information can be used to reduce or increase the output data of the regression model, thereby mapping the output data to the inherent output range of the regression model.

[0082] Intermediate calculation data refers to the data obtained after the regression model performs preliminary calculations based on the first facial feature data. For example, it could be the data obtained after performing a single translation or scaling iteration on the first facial feature data. By adjusting the intermediate calculation data of the regression model using adjustment information, multiple iterative adjustments to the input or output data of the regression model can be achieved, resulting in better prediction performance.

[0083] In some embodiments, updating the adjustment information based on the target expression coefficient includes:

[0084] If the target expression coefficient exceeds the preset output data range corresponding to the regression model, the adjustment information is updated.

[0085] Here, the preset output data range corresponding to the regression model can be the inherent output range of the regression model, that is, the output range set when training the regression model. The upper and lower limits of the preset output data range represent at least two states of an expression, such as the opening and closing of the eyes, and the downward and upward curve of the corners of the mouth. For example, the preset output data range of the eye expression coefficient can be set to [0,1], where an eye expression coefficient of 0 corresponds to the eyes being closed, an eye expression coefficient of 1 corresponds to the eyes being fully open, or an eye expression coefficient close to 0 corresponds to the eyes being closed, and an eye expression coefficient close to 1 corresponds to the eyes being fully open.

[0086] When the target expression coefficient exceeds the preset output data range of the regression model, it indicates a significant difference between the first facial feature data corresponding to the target frame and the facial feature data used for training the regression model. Furthermore, even after adjusting the first facial feature data or the intermediate calculation data and output data of the regression model using the adjustment information, it still cannot be mapped to the inherent output range of the regression model. Therefore, by updating the adjustment information, the adjustment results of the updated adjustment information on the first facial feature data or the intermediate calculation data and output data of the regression model can be optimized, thereby improving the prediction accuracy of the regression model.

[0087] In some embodiments, updating the adjustment information includes at least one of the following:

[0088] The adjustment information is updated based on the preset value;

[0089] The adjustment information is updated based on the target expression coefficient and the preset output data range.

[0090] Here, the preset value can be a pre-set step size for updating the adjustment information.

[0091] In some embodiments, the adjustment information is updated with a preset translation amount, such that the translation amount corresponding to the updated adjustment information increases or decreases by the preset translation amount with each update. In some embodiments, the adjustment information is updated with a preset scaling amount, such that the updated adjustment information shrinks or expands by a preset factor with each update. In this way, updating the adjustment information based on a preset update step size, without needing to update the adjustment information based on other factors, can accelerate the calculation speed of the regression model and save computational resources.

[0092] In some embodiments, updating the adjustment information based on the target expression coefficient and a preset output data range may include at least one of the following:

[0093] The information is updated and adjusted based on the fact that the target expression coefficient is less than the minimum value of the preset output data range of the regression model;

[0094] The information is updated and adjusted based on the fact that the target expression coefficient is greater than the maximum value of the preset output data range of the regression model;

[0095] The information is updated and adjusted based on the degree to which the target expression coefficient is less than or greater than the preset output data range of the regression model.

[0096] In this way, by updating the adjustment information based on the relationship between the target expression coefficient and the preset output data range, the adjustment information can be updated more accurately, enabling the regression model to quickly adapt to the faces in the current video based on the updated adjustment information.

[0097] In some embodiments, the adjustment information includes at least one of the following: translation parameters and scaling parameters; wherein,

[0098] The translation parameter is used to adjust the size of the output value of the regression model;

[0099] The scaling parameter is used to amplify or reduce the output value of the regression model by a specific factor.

[0100] Here, the translation parameter is used to adjust the size of the first facial feature data. For example, taking the first frame of the input regression model as an example, the facial feature data corresponding to the first frame is translated to the starting point of the inherent input range of the regression model. For example, the inherent input range of the regression model is [0,1], that is, x_min = 0 at the starting point; assuming that the facial feature data corresponding to the first frame is x = 0.5; then translating the facial feature data corresponding to the first frame to the starting point of the inherent input range of the regression model is equivalent to translating the facial feature data of the object in the first frame by 0.5, that is, the value of the translation parameter here is 0.5.

[0101] In practical applications, scaling parameters can be used to amplify or reduce the intermediate calculation results or output values ​​of a regression model by a specific factor. This means that scaling parameters can be used to amplify or reduce the values ​​of intermediate calculation data or output data of a regression model by a specific factor.

[0102] In this embodiment, by using translation parameters to adjust the size of the output value of the regression model, or by using scaling parameters to magnify or reduce the intermediate calculation results or output data of the regression model by a specific factor, it is possible to adjust the input data, intermediate calculation results and / or output data of the regression model from multiple dimensions, thereby improving the flexibility of data processing.

[0103] In some embodiments, when the adjustment information includes the translation parameters, the step S101 of determining the target expression coefficient corresponding to the target frame based on the first facial feature data, the regression model, and the adjustment information can be implemented through the following steps S1011 to S1012:

[0104] Step S1011: Using the translation parameters before the update, the first facial feature data is translated to obtain the second facial feature data; wherein, the translation parameters before the update are determined based on the facial feature data corresponding to the first frame in the video, and the first frame is an image frame before the target frame.

[0105] Step S1012: Determine the target expression coefficient based on the second facial feature data and the regression model;

[0106] The step S101, which involves updating the adjustment information based on the target expression coefficient, can be achieved through the following steps S1013 to S1014:

[0107] Step S1013: Determine the first adjustment factor based on the minimum input value of the regression model, the first facial feature data, and the translation parameters before the update;

[0108] Step S1014: Using the first adjustment factor, update the translation parameters before the update to obtain the updated translation parameters.

[0109] In this way, firstly, the first facial feature data is translated based on the translation parameters determined by the previous frame of the target frame to obtain the target expression coefficient corresponding to the target frame. Then, the translation parameters are updated based on the target expression coefficient to obtain the updated translation parameters. This allows the adjustment information to be updated based on the current target frame. That is, the adjustment information can be updated using multiple frames of images in the video, so that the target expression coefficient output by the regression model is closer to the real facial expression in the current video and can quickly adapt to the facial features in the current video.

[0110] In some embodiments, step S1013 can be implemented by the following steps:

[0111] Determine the first difference between the minimum model input value within the inherent input range of the regression model and the first facial feature data;

[0112] Determine the second difference between the first difference and the translation parameter before the update;

[0113] The first adjustment factor is determined based on the second difference.

[0114] Here, the first adjustment factor tmp can be determined by the following formula (3):

[0115] tmp=x_min-x_current-self.translation_vactor (3);

[0116] Where x_min represents the minimum model input value of the regression model; x_current represents the first facial feature data; and self.translation_vactor represents the translation parameter before the update.

[0117] Step S1014 can be achieved through the following steps:

[0118] The first adjustment factor is weighted to obtain the weighted first adjustment factor;

[0119] The second translation vector is adjusted using the weighted first adjustment factor to obtain the first translation vector.

[0120] Here, the first adjustment factor can be weighted based on preset weights. For example, a weight of 0.1 can be used to weight the first adjustment factor. In practical applications, the weight value can be set according to different needs. For example, if it is desired that the output of the regression model can respond to the adjustment of the translation parameter more quickly, a larger weight value can be set for the first adjustment factor; if it is desired that the output of the regression model can respond to the adjustment of the translation parameter more smoothly, a smaller weight value can be set for the first adjustment factor.

[0121] In some embodiments, the updated translation parameter tanslation_vactor can be determined by the following formula (4):

[0122] translation_vactor=self.translation_vactor+0.1*tmp (4);

[0123] Here, self.translation_vactor represents the translation parameters before the update; 0.1 is the weight coefficient.

[0124] In some embodiments, when the adjustment information includes the scaling parameter, the scaling parameter includes at least one of the following: a first scaling parameter and a second scaling parameter;

[0125] The step S101 of updating the adjustment information based on the target expression coefficient includes at least one of the following steps S1015 and S1016:

[0126] Step S1015: If the target expression coefficient is greater than the maximum model output value of the regression model, the first scaling parameter is used as the scaling factor, and the first scaling parameter is determined based on the first facial feature data.

[0127] Here, the maximum model output value is the maximum value in the inherent output range of the regression model; for example, the maximum model output value is 1.

[0128] When the target expression coefficient is greater than the maximum model output value, it indicates that the range of facial expressions in the current video is larger than the range of facial expressions used for model training. For example, taking the eyes as an example, a regression model is trained using training data with a smaller eye area. The inherent output range of the regression model is [0,1], where an output value of 0 indicates that the eyes are closed and an output value of 1 indicates that the eyes are fully open. If the range of eye expressions in the current video is large, then when using the linear regression model to predict the expression coefficient of the eyes in the video, based on the linear characteristics of the linear regression model, the prediction result may exceed the maximum model output value of 1. That is, when the eyes in the video are not fully open, the target expression coefficient output by the regression model is greater than 1. This will result in the inability to accurately map the facial expression features in the video to a virtual or digital human.

[0129] At this point, the first scaling parameter can be used to perform a scaling operation on the first facial feature data corresponding to the target frame, so as to map the first facial feature data into the inherent output range [0,1] of the regression model, thereby making the output value of the regression model within the inherent output range [0,1] of the regression model.

[0130] In some embodiments, determining the first scaling parameter based on the first facial feature data includes:

[0131] Determine the third difference between the maximum model input value and the minimum model input value of the regression model;

[0132] Determine the fourth difference between the minimum model input value of the regression model and the first facial feature data;

[0133] The first scaling parameter is determined based on the third difference and the fourth difference.

[0134] Here, the maximum and minimum model input values ​​refer to the maximum and minimum values ​​within the inherent input range of the regression model, respectively. For example, if the inherent input range of the regression model is [0,1], then the maximum model input value is 1 and the minimum model input value is 0.

[0135] In some embodiments, the first scaling factor s1 can be determined based on the following formula (5):

[0136] s1=((x_max-x_min) / (x_current-x_min)) (5);

[0137] Where x_max represents the maximum model input value; x_min represents the minimum model input value; and x_current represents the first facial feature data.

[0138] In practical applications, since the first scaling factor only needs to be updated when the output value of the regression model is greater than the maximum model output value, the value of the first scaling factor will become smaller and smaller as the number of updates increases.

[0139] Step S1016: If the target expression coefficient is greater than the maximum value in the output dataset, the second scaling parameter is used as the scaling parameter, and the second scaling parameter is determined based on the target expression coefficient; wherein, the output dataset includes the target expression coefficient corresponding to at least one second image frame in the video, and each second image frame is an image frame preceding the target frame.

[0140] In practical applications, the scenario requiring the use of the second scaling parameter is as follows: Taking the eyes as an example, the eyes in the regression model's training set are relatively large, while the eyes in the current video are smaller. That is, the range of facial expressions in the current video is smaller than the inherent output range of the regression model. This causes the regression model's output value to fail to reach its maximum output value when the eyes are fully open in the video, preventing the virtual or digital human from accurately mapping facial expressions in the video. Therefore, it is necessary to amplify the regression model's output value to adapt it to the facial features in the video.

[0141] If the target expression coefficient is less than the maximum model output value, the second scaling parameter is updated if the target expression coefficient is greater than the maximum value in the output dataset.

[0142] Here, an initial value can be set for the second scaling parameter, that is, the output value of the regression model is amplified by a predetermined amount using the second scaling parameter.

[0143] When the target expression coefficient is greater than the maximum value in the output dataset, it means that the eye expression coefficient in the target frame exceeds the maximum output value calculated based on the second scaling parameter. Therefore, the second scaling parameter needs to be updated to remap the eye expression coefficient in the current video to the inherent output range of the regression model.

[0144] In some embodiments, determining the second scaling parameter based on the target expression coefficient may include the following steps:

[0145] The second scaling parameter is determined based on the ratio of the maximum model output value of the regression model to the target expression coefficient.

[0146] In some embodiments, the second scaling parameter s2 can be determined by the following formula (6):

[0147] s2 = y_max / bs_max (6);

[0148] Where y_max represents the maximum model output value; bs_max represents the target expression coefficient.

[0149] In some embodiments, the step S101, which involves determining the target expression coefficient corresponding to the target frame based on the first facial feature data, the regression model, and the adjustment information, can be implemented through the following steps S1017 to S1019:

[0150] Step S1017: Obtain the first intermediate result of the regression model based on the translation parameters;

[0151] Step S1018: Obtain the second result of the regression model based on the first intermediate result and the scaling parameter;

[0152] Step S1019: Determine the target expression coefficient based on the second result.

[0153] Here, the first intermediate result can be the facial feature data obtained by translating the first facial feature data.

[0154] In some embodiments, the second result may be facial feature data obtained by scaling down the translated facial feature data based on the first scaling parameter. In some embodiments, the second result may also be the output value of the regression model obtained by scaling up the output data of the regression model based on the second scaling parameter.

[0155] In some embodiments, the second result can be used as the target expression coefficient. In some embodiments, the second result can also be further scaled to obtain the target expression coefficient.

[0156] Figure 2 The following is a flowchart illustrating an application embodiment of the data processing method provided in this application. Figure 2 The application example is described in detail below.

[0157] Step S201: Acquire video; then, proceed to step S202.

[0158] Here, the video is a video that includes facial images.

[0159] Step S202: Based on the translation parameter c, perform translation processing on the facial feature data corresponding to the first frame; then, execute step S203.

[0160] Here, taking the first frame as the baseline, the facial feature data corresponding to the first frame is shifted to the minimum model input data of the regression model.

[0161] Step S203: Determine the expression coefficient y1 corresponding to the first frame based on the regression model; then, proceed to step S204.

[0162] Here, the regression model is used to predict the expression coefficient for the facial feature data translated by the translation parameter, and the expression coefficient y1 corresponding to the first frame is obtained.

[0163] Step S204, determine whether y1 is greater than 1; if so, execute step S205; if not, execute step S206;

[0164] Step S205, update the first scaling parameter s1; then execute step S208;

[0165] Here, based on the facial feature data corresponding to the first frame, the maximum model input value and the minimum model input value of the regression model, the first scaling parameter s1 is determined.

[0166] Step S206, determine whether y1 is less than 0; if so, execute step S207; if not, execute step S209;

[0167] Step S207, update the translation parameter c; then execute step S209;

[0168] Here, based on the translation parameter before update, the minimum model input value of the regression model and the facial feature data corresponding to the first frame, the updated translation parameter c is determined.

[0169] Step S208, determine the expression coefficient y2 corresponding to the first frame based on the regression model and the updated first scaling parameter s1 and translation parameter c; then execute step S210;

[0170] Step S209, y2 = y1; then execute step S210;

[0171] Here, since it is confirmed in step S204 and step S206 that the value of y1 is within the inherent output range of the regression model (i.e., 0 < y1 < 1), therefore, the value of y1 is used as the value of y2 for subsequent operations.

[0172] Step S210, determine whether y2 is the maximum value in the output data; if so, execute step S211; if not, execute step S213;

[0173] Step S211, update the second scaling parameter S2; then execute step S212;

[0174] Here, based on the ratio of the maximum model output value of the regression model and y2, the second scaling parameter s2 is determined.

[0175] Step S212, determine the expression coefficient y3 corresponding to the first frame based on the regression model and the updated s1, s2 and c; then execute step S213;

[0176] Here, based on the updated s1, s2 and c, the expression coefficient of the first frame is re-predicted using a regression model to obtain the expression coefficient y3 corresponding to the first frame.

[0177] Step S213: Determine the target expression coefficient.

[0178] Here, the target expression coefficient corresponding to the first frame is determined based on the expression coefficient y3 of the object in the first frame; or when the value of y2 is not the maximum value in the output data, the target expression coefficient corresponding to the first frame is determined based on y2.

[0179] Based on the foregoing embodiments, this application provides a data processing device, which includes various units and modules included in each unit. It can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0180] Figure 3 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the data processing device 300 includes a data processing module 310, wherein:

[0181] Data processing module 310 is used to perform a first processing on first facial feature data corresponding to a target frame in a video. The first processing includes: determining a target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model.

[0182] The data processing module 310 is further configured to sequentially use multiple frame images in the video as target frames to perform the first processing.

[0183] In some embodiments, the adjustment information is used for at least one of the following:

[0184] Adjust the input data for the regression model;

[0185] Adjust the output data of the regression model;

[0186] Adjustments are made to the intermediate calculation data of the regression model.

[0187] In some embodiments, the data processing module 310 is further configured to:

[0188] If the target expression coefficient exceeds the preset output data range corresponding to the regression model, the adjustment information is updated.

[0189] In some embodiments, the data processing module 310 is further configured to perform at least one of the following:

[0190] The adjustment information is updated based on the preset value;

[0191] The adjustment information is updated based on the target expression coefficient and the preset output data range.

[0192] In some embodiments, the adjustment information includes at least one of the following: translation parameters and scaling parameters; wherein,

[0193] The translation parameter is used to adjust the size of the output value of the regression model;

[0194] The scaling parameter is used to amplify or reduce the output value of the regression model by a specific factor.

[0195] In some embodiments, where the adjustment information includes the translation parameters...

[0196] The data processing module 310 is also used for:

[0197] Using the translation parameters before the update, the first facial feature data is translated to obtain the second facial feature data; wherein, the translation parameters before the update are determined based on the facial feature data corresponding to the first frame in the video, and the first frame is an image frame before the target frame.

[0198] Based on the second facial feature data and the regression model, the target expression coefficient is determined;

[0199] Based on the minimum input value of the regression model, the first facial feature data, and the translation parameters before the update, a first adjustment factor is determined;

[0200] The translation parameters before the update are updated using the first adjustment factor to obtain the updated translation parameters.

[0201] In some embodiments, when the adjustment information includes the scaling parameter, the scaling parameter includes at least one of the following: a first scaling parameter and a second scaling parameter;

[0202] The data processing module 310 is also configured to perform at least one of the following:

[0203] If the target expression coefficient is greater than the maximum model output value of the regression model, the first scaling parameter is used as the scaling factor, and the first scaling parameter is determined based on the first facial feature data.

[0204] If the target expression coefficient is greater than the maximum value in the output dataset, the second scaling parameter is used as the scaling parameter, and the second scaling parameter is determined based on the target expression coefficient; wherein, the output dataset includes the target expression coefficient corresponding to at least one second image frame in the video, and each second image frame is an image frame preceding the target frame.

[0205] In some embodiments, the data processing module 310 is further configured to:

[0206] The first intermediate result of the regression model is obtained based on the translation parameters;

[0207] The second result of the regression model is obtained based on the first intermediate result and the scaling parameter;

[0208] The target expression coefficient is determined based on the second result.

[0209] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0210] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0211] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0212] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0213] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0214] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the device, storage medium, and computer program embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the device, storage medium, computer program, and computer program product embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0215] It should be noted that, Figure 4 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 4 As shown, the hardware entity of the computer device 400 includes: a processor 401, a communication interface 402, and a memory 403, wherein:

[0216] Processor 401 typically controls the overall operation of computer device 400.

[0217] Communication interface 402 enables computer devices to communicate with other terminals or servers via a network.

[0218] The memory 403 is configured to store instructions and applications executable by the processor 401, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) of the processor 401 and various modules in the computer device 400. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 401, the communication interface 402, and the memory 403 can be performed via bus 404.

[0219] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0220] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0222] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0223] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0224] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0225] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0226] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, comprising: A first processing step is performed on the first facial feature data corresponding to a target frame in the video. The first processing step includes: determining a target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model. The first process is performed on multiple frames in the video as target frames in sequence. The adjustment information includes at least one of the following: translation parameters and scaling parameters; the translation parameters are used to adjust the size of the output value of the regression model; the scaling parameters are used to amplify or reduce the output value of the regression model by a specific factor.

2. The method according to claim 1, wherein the adjustment information is used for at least one of the following: Adjust the input data for the regression model; Adjust the output data of the regression model; Adjustments are made to the intermediate calculation data of the regression model.

3. The method according to claim 1, wherein updating the adjustment information based on the target expression coefficient includes: If the target expression coefficient exceeds the preset output data range corresponding to the regression model, the adjustment information is updated.

4. The method according to claim 3, wherein updating the adjustment information includes at least one of the following: The adjustment information is updated based on the preset value; The adjustment information is updated based on the target expression coefficient and the preset output data range.

5. The method according to claim 1, wherein, When the adjustment information includes the translation parameters, The step of determining the target expression coefficient corresponding to the target frame based on the first facial feature data, regression model, and adjustment information includes: Using the translation parameters before the update, the first facial feature data is translated to obtain the second facial feature data; wherein, the translation parameters before the update are determined based on the facial feature data corresponding to the first frame in the video, and the first frame is an image frame before the target frame. Based on the second facial feature data and the regression model, the target expression coefficient is determined; Updating the adjustment information based on the target expression coefficient includes: Based on the minimum input value of the regression model, the first facial feature data, and the translation parameters before the update, a first adjustment factor is determined; The translation parameters before the update are updated using the first adjustment factor to obtain the updated translation parameters.

6. The method according to claim 1, wherein, When the adjustment information includes the scaling parameter, the scaling parameter includes at least one of the following: a first scaling parameter and a second scaling parameter; Updating the adjustment information based on the target expression coefficient includes at least one of the following: If the target expression coefficient is greater than the maximum model output value of the regression model, the first scaling parameter is used as the scaling parameter, and the first scaling parameter is determined based on the first facial feature data. If the target expression coefficient is greater than the maximum value in the output dataset, the second scaling parameter is used as the scaling parameter, and the second scaling parameter is determined based on the target expression coefficient; wherein, the output dataset includes the target expression coefficient corresponding to at least one second image frame in the video, and each second image frame is an image frame preceding the target frame.

7. The method according to claim 1, wherein determining the target expression coefficient corresponding to the target frame based on the first facial feature data, the regression model, and the adjustment information comprises: The first intermediate result of the regression model is obtained based on the translation parameters; The second result of the regression model is obtained based on the first intermediate result and the scaling parameter; The target expression coefficient is determined based on the second result.

8. A data processing apparatus, comprising: A data processing module is used to perform a first processing on first facial feature data corresponding to a target frame in a video. The first processing includes: determining a target expression coefficient corresponding to the target frame based on the first facial feature data, a regression model, and adjustment information; updating the adjustment information based on the target expression coefficient; wherein the first facial feature data is the input data of the regression model; the target expression coefficient is obtained based on the output data of the regression model; and the adjustment information affects the output data of the regression model. The data processing module is further configured to sequentially use multiple frame images in the video as target frames to perform the first processing; The adjustment information includes at least one of the following: translation parameters and scaling parameters; the translation parameters are used to adjust the size of the output value of the regression model; the scaling parameters are used to amplify or reduce the output value of the regression model by a specific factor.

9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1 to 7.

Images