Expression feature parameter processing method, device and electronic equipment

Abstract

The present application provides a method, device and electronic device for processing expression feature parameters, the method comprising: obtaining a target single-frame recognition result of a first image frame in a video to be recognized and historical single-frame recognition results of multiple second image frames, and then obtaining a recognition offset corresponding to the first image frame according to the target single-frame recognition result and the historical single-frame recognition result through an offset prediction model, and then adjusting the target single-frame recognition result according to the recognition offset to obtain the adjusted expression feature parameters. In this way, the current target single-frame recognition result can be corrected according to the historical single-frame recognition result, and the data mutation error that may exist in the target single-frame recognition result can be offset, so that the adjusted expression feature parameters are more in line with the change trend of the historical single-frame recognition result, and the amount of calculation is relatively small, and the processing efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a method, device and electronic device for processing expression feature parameters. Background Art

[0002] With the continuous development of image processing technology, the generation technology of 3D face models has been used in various fields, such as video special effects overlay, virtual image generation, data analysis, etc. In order to make the generated 3D face vivid, a variety of different data are needed when generating the 3D face model. Taking the 3D Morphable Mode (3DMM) which is more commonly used in generating 3D face models as an example, the process of face reconstruction usually requires three types of feature parameters, namely: shape feature parameters (also known as shape coefficients) used to characterize the 3D shape of the face contour in the expressionless state, color feature parameters (also known as color coefficients) used to characterize the color or shadow on the face, and expression feature parameters (also known as expression coefficients) used to adjust the data on the reconstructed face model so that the model has different expressions.

[0003] In order to make the generated face model more realistic and natural, taking the expression feature parameters as an example, the 3D expression feature coefficients can be predicted by identifying and analyzing the 2D image. In some expression feature coefficient acquisition methods, only a single frame of 2D image is recognized and processed to obtain the expression feature coefficient recognition result of a single frame, but there may be a problem of high recognition error rate; in other expression feature coefficient extraction methods, feature extraction and feature processing are used for multiple frames of continuous 2D images to jointly obtain the recognition result of a single frame, but the processing calculation amount is large and the processing efficiency is poor. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a method for processing expression feature parameters, the method comprising:

[0005] Acquire a first image frame in a video to be recognized, perform expression feature parameter recognition on the first image frame using a single-frame recognition model, and obtain a target single-frame recognition result;

[0006] Acquire historical single-frame recognition results of a plurality of second image frames before the first image frame; the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame;

[0007] The target single-frame recognition result and the plurality of historical single-frame recognition results are input into an offset prediction model to obtain a recognition offset corresponding to the first image frame; the recognition offset represents the difference between the expression feature parameters predicted by the change trend of the plurality of historical single-frame recognition results and the target single-frame recognition result;

[0008] The target single-frame recognition result is adjusted according to the recognition offset to obtain adjusted expression feature parameters.

[0009] In a possible implementation, the single-frame recognition model includes at least one convolution unit and a fully connected unit; the step of performing expression feature parameter recognition on the first image frame through the single-frame recognition model to obtain a target single-frame recognition result includes:

[0010] Processing the first image frame by the at least one convolution unit to obtain an intermediate processing result;

[0011] The intermediate processing result is processed by the fully connected unit to obtain the target single-frame recognition result, and the target single-frame recognition result includes a plurality of data representing the states of different facial organs.

[0012] In a possible implementation, the step of inputting the target single-frame recognition result and the plurality of historical single-frame recognition results into an offset prediction model to obtain a recognition offset corresponding to the first image frame includes:

[0013] Based on the first model parameter, the second model parameter and the third model parameter obtained by pre-training in the offset prediction model, the recognition offset is calculated according to the target single-frame recognition result and the plurality of historical single-frame recognition results through an attention mechanism;

[0014] Among them, the first model parameter and the second model parameter are used to calculate the correlation between the target single-frame recognition result and multiple historical single-frame recognition results, and the third model parameter is used to calculate the change trend of multiple historical single-frame recognition results.

[0015] In a possible implementation, based on the first model parameter, the second model parameter and the third model parameter pre-trained in the offset prediction model, the step of calculating the recognition offset according to the target single-frame recognition result and the plurality of historical single-frame recognition results through an attention mechanism includes:

[0016] Determine a first calculation vector of an attention function according to the target single-frame recognition result and a first model parameter of the offset prediction model;

[0017] Determine a second calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset coding constant matrix and a second model parameter of the offset prediction model;

[0018] Determining a third calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset coding constant matrix and a third model parameter of the offset prediction model; the coding constant matrix is ​​used to indicate the order of the plurality of historical single-frame recognition results;

[0019] The recognition offset is calculated based on the first calculation vector, the second calculation vector and the third calculation vector through a self-attention function.

[0020] In a possible implementation, the step of determining the second calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset encoding constant matrix and a second model parameter of the offset prediction model includes:

[0021] Adding a historical recognition result matrix composed of a plurality of the historical single-frame recognition results to the coding constant matrix to obtain a recognition result adjustment matrix, and multiplying the recognition result adjustment matrix by the second model parameter to obtain the second calculation vector;

[0022] The step of determining the third calculation vector of the attention function according to the plurality of historical single-frame recognition results, the preset coding constant matrix and the third model parameter of the offset prediction model comprises:

[0023] Adding a historical recognition result matrix composed of a plurality of historical single-frame recognition results to the coding constant matrix to obtain a recognition result adjustment matrix, and multiplying the recognition result adjustment matrix by the third model parameter to obtain the third calculation vector;

[0024] Among them, in the coding constant matrix, the value of the data item corresponding to the second image frame that is close to the first image frame in time sequence is greater than the value of the data item corresponding to the second image frame that is far away from the first image frame in time sequence.

[0025] In a possible implementation, the method further includes:

[0026] Facial reconstruction is performed according to the adjusted expression feature parameters.

[0027] In a possible implementation, the method further includes:

[0028] Acquire a third image frame in the training video data, and an expression feature parameter label corresponding to the third image frame;

[0029] Inputting the third image frame into the single-frame recognition model to be trained, and obtaining the expression feature parameter prediction result output by the single-frame recognition model;

[0030] The model parameters of the single-frame recognition model are adjusted according to the expression feature parameter prediction result and the expression feature parameter label.

[0031] In a possible implementation, the method further includes:

[0032] Acquire historical single frame recognition results of a plurality of fourth image frames located before the third image frame in the training video data;

[0033] Inputting the historical single-frame recognition results of the plurality of fourth image frames into the offset prediction model to be trained, and obtaining the predicted offset output by the offset prediction model;

[0034] The actual offset is obtained by calculation according to the expression feature parameter label of the third image frame and the expression feature parameter prediction result;

[0035] The model parameters of the offset prediction model are adjusted according to the actual offset and the predicted offset.

[0036] Another object of the present application is to provide an expression feature parameter processing device, the expression feature parameter processing device comprising:

[0037] A data acquisition module, used to acquire a first image frame in a video to be recognized, and historical single-frame recognition results of a plurality of second image frames before the first image frame; the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame;

[0038] A single-frame recognition module, used to perform expression feature parameter recognition on the first image frame through a single-frame recognition model to obtain a target single-frame recognition result;

[0039] An offset determination module, used to input the target single-frame recognition result and the plurality of historical single-frame recognition results into an offset prediction model based on a self-attention mechanism to obtain a recognition offset corresponding to the first image frame;

[0040] The result adjustment module is used to adjust the target single-frame recognition result according to the recognition offset to obtain adjusted expression feature parameters.

[0041] Another object of the present application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the expression feature parameter processing method provided in the present application is implemented.

[0042] Another object of the present application is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by one or more processors, they implement the expression feature parameter processing method provided by the present application.

[0043] Compared with the prior art, this application has the following beneficial effects:

[0044] The present application provides an expression feature parameter processing method, device and electronic device, which determines the recognition offset to adjust the target single-frame recognition result of the first image frame based on the historical single-frame recognition results of multiple second images before the first image frame. In this way, the current target single-frame recognition result can be corrected according to the historical single-frame recognition results, and the data mutation errors that may exist in the target single-frame recognition results can be offset, so that the adjusted expression feature parameters are more consistent with the change trend of the historical single-frame recognition results, and the calculation amount is relatively small, and the processing efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a live broadcast system provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the steps of the expression feature processing method provided in the embodiment of the present application;

[0048] Figure 3 A schematic diagram of an expression feature recognition model provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a live broadcast server provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the functional modules of the facial expression feature processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0056] This embodiment provides an expression feature parameter recognition model, which can be applied to recognize and process 2D video images to obtain facial expression feature parameters. These expression feature parameters can be used to combine with a 3D model of a face without expression, so that the 3D model of the face can have an expression similar to that of the face in the 2D video image.

[0057] For example, the facial expression feature parameter recognition model 100 can be applied to a live video broadcast system, see Figure 1The video live broadcast system may include a live broadcast server 10, a video providing terminal 20 and a video receiving terminal 30. The video live broadcast system may be a service platform for Internet live broadcasting. The live broadcast server 10 may be respectively connected to the video providing terminal 20 and the video receiving terminal 30 for communication, and may provide live broadcast services for the video providing terminal 20 and the video receiving terminal 30, and may provide live broadcast services for related products during the live broadcasting process, such as game live broadcast services, etc., for the video providing terminal 20 and the video receiving terminal 30.

[0058] In some possible implementations, the video providing terminal 20 and the video receiving terminal 30 can be used interchangeably. For example, a host of the video providing terminal 20 can use the video providing terminal 20 to provide live video services to viewers, or watch live videos provided by other hosts as a viewer. For another example, a viewer of the video receiving terminal 30 can also use the video receiving terminal 30 to watch live videos provided by the host they are following, or provide live video services to other viewers as a host.

[0059] In this embodiment, the video providing terminal 20 and the video receiving terminal 30 may be, but are not limited to, a smart phone, a personal digital assistant, a tablet computer, a personal computer, a laptop computer, a virtual reality terminal device, an augmented reality terminal device, etc. In the specific implementation process, there may be zero, one or more video providing terminals 20 and video receiving terminals 30 connected to the live broadcast server 10. Figure 1 Among them, the video providing terminal 20 and the video receiving terminal 30 can be installed with a program product for providing an Internet live broadcast service. For example, the program product can be an application APP, a Web page, a small program, etc. related to the Internet live broadcast service used in a computer or a smart phone.

[0060] In this embodiment, the live broadcast server 10 can be a single physical server or a device cluster composed of multiple physical servers for performing different data processing functions. The device cluster can be centralized or distributed (for example, the live broadcast server 10 can be a distributed system). In some possible implementations, if the live broadcast server 10 uses a single physical server, different logical device clusters can be allocated to the physical server based on different live broadcast service functions.

[0061] Understandably, Figure 1 The live broadcast server 10 shown is only a feasible example. In other feasible embodiments, the live broadcast server 10 may also include only Figure 1 Some of the components shown may also include other components.

[0062] Specifically, the expression feature parameter recognition model 100 can be configured in the live broadcast server 10, and the live broadcast server 10 can also be configured with a face reconstruction model 300. The live broadcast server 10 can obtain the live broadcast video data to be processed from the video providing terminal 20, and recognize and process the face image in the live broadcast video data through the expression feature parameter recognition model 100 to obtain the expression feature parameters. Then, the expression feature parameters are combined with the 3D face model of the virtual image through the face reconstruction model 300, or the face is reconstructed according to the expression feature parameters to obtain the processed live broadcast video data. Then, the live broadcast server 10 pushes the processed live broadcast video data to each video receiving terminal 30.

[0063] Combine the following Figure 1 The application scenario shown in the figure is an exemplary description of the live data stream interaction method provided in the embodiment of the present application. Figure 2 The live data stream interaction method provided in this embodiment can be implemented by Figure 1 The live broadcast server 10 in the embodiment is executed. It should be understood that in other embodiments, the order of some steps in the live broadcast data stream interaction method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the live broadcast data stream interaction method executed by the live broadcast server 10 are described as follows.

[0064] Step S110, obtaining a first image frame in the video to be recognized, performing expression feature parameter recognition on the first image frame through a single-frame recognition model, and obtaining a target single-frame recognition result.

[0065] Please refer to Figure 3 In this embodiment, the expression feature parameter recognition model 100 may include a single frame recognition model and an offset prediction model. In step S110, the expression feature parameter recognition may be performed on the first image frame by the single frame recognition model to obtain the target single frame recognition result.

[0066] Optionally, the single-frame recognition model may include at least one convolution unit and a fully connected unit. In step S110, the first image frame is processed by the at least one convolution unit to obtain an intermediate processing result, and then the intermediate processing result is processed by the fully connected unit to obtain the target single-frame recognition result. Among them, the convolution unit may include network layer structures such as convolution layer, normalization layer, activation function, pooling layer, etc. Different convolution units may have different network layer results; the fully connected unit may include pooling layer, flattening layer, fully connected layer, etc.

[0067] Exemplarily, the single-frame recognition model may be similar to the model structure of Resnet-50, wherein some convolutional units may include convolutional layers, normalization layers, ReLU activation function layers and maximum pooling layers, some convolutional units may include convolutional layers, normalization layers and ReLU activation function layers, and fully connected units may include average pooling layers, flattening layers and fully connected layers.

[0068] The target single-frame recognition result may include multiple data representing the activity levels of different key parts of the face. The target single-frame recognition result may be a multi-dimensional vector, and each data item of the multi-dimensional vector may be used to represent the activity level of a key part of a face. For example, the target single-frame recognition result may be expressed as (R1, R2, ..., R N ), where R1 to R N The activity levels of multiple key facial parts such as the mouth center opening degree, the left eye center opening degree, the right eye center opening degree, etc. can be represented respectively. Accordingly, the historical single-frame recognition result and the target single-frame recognition result have the same data structure.

[0069] Step S120, obtaining historical single-frame recognition results of a plurality of second image frames before the first image frame, wherein the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame.

[0070] In this embodiment, the first image frame may be an image frame that currently needs to be recognized in the video to be recognized, and the plurality of second image frames may be a plurality of image frames that are located before the first image frame in time sequence in the video to be recognized. Optionally, in this embodiment, the plurality of second image frames may be a plurality of image frames that are continuous before the first image frame. The historical single-frame recognition result described in step S120 is also a result obtained by the single-frame recognition model recognizing each of the second image frames.

[0071] Step S130, inputting the target single-frame recognition result and the plurality of the historical single-frame recognition results into an offset prediction model to obtain a recognition offset corresponding to the first image frame. The recognition offset represents the difference between the expression feature parameters predicted by the change trend of the plurality of the historical single-frame recognition results and the target single-frame recognition result.

[0072] The inventors have discovered that in continuous video data, the activity levels of key parts of the face usually do not change suddenly in several adjacent frames. Even rapidly changing expressions in video data are composed of gradient effects in multiple image frames. In the absence of recognition errors, the target single-frame recognition result of the first image frame should conform to the changing trend of the historical single-frame recognition results of multiple second image frames.

[0073] Therefore, in this embodiment, the recognition offset is calculated based on the changing trend of the historical single-frame recognition results of multiple second image frames and the target single-frame recognition result of the first image frame. The recognition offset can be used to characterize the difference between the trend predicted by the historical single-frame recognition results of multiple second image frames and the actual target single-frame recognition result of the first image frame.

[0074] The recognition offset may be a multi-dimensional vector, and each dimension of the recognition offset is an adjustment value corresponding to the dimension of the corresponding dimension in the target single-frame recognition result.

[0075] Step S140, adjusting the target single-frame recognition result according to the recognition offset to obtain adjusted expression feature parameters.

[0076] It is understandable that if an error occurs during the recognition process for the first image frame, some data items in the target single-frame recognition result may be greatly mutated compared to the historical single-frame recognition results of the second image frames, which may lead to a larger absolute value of the data item corresponding to the recognition offset obtained in step S130. Therefore, adjusting the target single-frame recognition result according to the recognition offset can offset the erroneous data that has mutated in the target single-frame recognition result, so that the target single-frame recognition result is more consistent with the change trend of the historical single-frame recognition results.

[0077] Based on the above design, in this embodiment, the recognition offset is determined according to the data dependency and change trend in multiple historical single-frame recognition results, so that the data mutation error in the target single-frame recognition result can be corrected, thereby improving the accuracy of the output result of the entire model. In addition, the offset prediction model processes the historical single-frame recognition results and the target single-frame recognition results instead of directly processing the image data, thereby ensuring the efficiency of data processing, and is more suitable for scenarios such as live video broadcasts that have high requirements for processing timeliness.

[0078] In this embodiment, after obtaining the adjusted expression feature parameters, facial reconstruction can be performed according to the adjusted expression feature parameters. Specifically, the adjusted expression feature parameters can be combined with a pre-set 3D face model without expression, so as to adjust the position of key face parts on the 3D face model, so that the 3D face model has a corresponding expression posture.

[0079] In some possible implementations, in step S130, the recognition offset can be calculated based on the first model parameters, the second model parameters, and the third model parameters pre-trained in the offset prediction model, through an attention mechanism according to the target single-frame recognition result and multiple historical single-frame recognition results.

[0080] The first model parameter and the second model parameter are used to calculate the correlation between the target single-frame recognition result and multiple historical single-frame recognition results, and the third model parameter is used to calculate the change trend of multiple historical single-frame recognition results.

[0081] For example, the first model parameter can be used to calculate the Q (Query) vector required in the attention mechanism in combination with the target single-frame recognition result, the second model parameter can be used to calculate the K (Key) vector required in the attention mechanism in combination with the historical single-frame recognition result, and the third model parameter can be used to calculate the V (Value) vector required in the attention mechanism in combination with the historical single-frame recognition result. Then, the recognition offset is calculated based on the Q vector, K vector and V vector through the attention mechanism.

[0082] Specifically, step S130 may include the following sub-steps.

[0083] Step S131, determining a first calculation vector of the attention function according to the target single frame recognition result and the first model parameter of the offset prediction model. In this step, the first calculation vector can be a Q (Query) vector required for calculation of the self-attention mechanism.

[0084] Step S132, determining a second calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset coding constant matrix and a second model parameter of the offset prediction model. The coding constant matrix is ​​used to indicate the order of the plurality of historical single-frame recognition results.

[0085] In this step, the second calculation vector may be a K (Key) vector required for use in the calculation of the self-attention mechanism. Specifically, the historical recognition result matrix composed of a plurality of the historical single-frame recognition results may be added to the coding constant matrix to obtain a recognition result adjustment matrix, and the recognition result adjustment matrix may be multiplied by the second model parameter to obtain the second calculation vector. Step S133, determining the third calculation vector of the attention function according to the plurality of the historical single-frame recognition results, the preset coding constant matrix and the third model parameter of the offset prediction model.

[0086] In this step, the third calculation vector may be a V (Value) vector required for use in the calculation of the self-attention mechanism. The third calculation vector may be the same as the second calculation vector. Specifically, the recognition result adjustment matrix may be obtained by adding a historical recognition result matrix composed of a plurality of historical single-frame recognition results to the encoding constant matrix, and the recognition result adjustment matrix may be multiplied by the third model parameter to obtain the third calculation vector.

[0087] Step S134, calculating the recognition offset according to the first calculation vector, the second calculation vector and the third calculation vector through a self-attention function.

[0088] Specifically, in this embodiment, the vector corresponding to the target single frame recognition result is denoted as pred∈R N , N is the vector dimension. Then the first calculation vector Q, the second calculation vector K and the third calculation vector W are calculated as follows:

[0089] Q = pred i W Q ,K=(input i +pos)W K ,V=(input i +pos)W v

[0090] W Q ∈R 1＊N ,W K ∈R N*M ,W V ∈R N*M

[0091] Q∈R 1＊N ,K∈R N*M ,V∈R N*M

[0092] Among them, pred i is the target single frame recognition result of the i-th first image frame, W Q is the first model parameter of the offset prediction model, input i is the historical recognition result matrix composed of M historical single-frame recognition results, pos is the encoding constant matrix, input i +pos is the recognition result adjustment matrix; s K is the second model parameter of the offset prediction model, W V is the second model parameter of the offset prediction model; pos is the encoding constant matrix,

[0093] The recognition offset attention is calculated as follows:

[0094]

[0095] Where T is the matrix transpose symbol.

[0096] In this embodiment, in the coding constant matrix, a value of a data item corresponding to a second image frame that is close to the first image frame in time sequence is greater than a value of a data item corresponding to a second image frame that is far away from the first image frame in time sequence.

[0097] For example, the first image frame is the i-th image frame, and there are 5 second image frames involved in the calculation. The first row in the above coding constant matrix corresponds to the 5th second image frame before the first image frame (i.e., the i-5th frame). The coding constant matrix pos can be expressed as follows:

[0098]

[0099] The last row in the coding constant matrix corresponds to the first second image frame (i.e., the i-1th frame) before the first image frame, and the values ​​of the data items in the coding constant matrix increase from the last row to the first row. In this way, after adding the coding constant matrix to the historical recognition matrix, the data corresponding to the second image frame closer to the first image frame can be made larger, thereby increasing the influence of the output result of the second image frame closer to the first image frame, so that the data items in the obtained recognition result adjustment matrix have time sequence.

[0100] In addition, the method provided in this embodiment may include the following training steps for the single-frame recognition model.

[0101] Step S201, obtaining a third image frame in the training video data and an expression feature parameter label corresponding to the third image frame.

[0102] In this embodiment, the training video data may include multiple image frames, each of which may be pre-labeled with a corresponding expression feature parameter label. The expression feature parameter label may be a multi-dimensional vector, and the expression feature parameter label may have the same data structure as the target single frame recognition result.

[0103] Step S202: input the third image frame into the single-frame recognition model to be trained to obtain the expression feature parameter prediction result output by the single-frame recognition model.

[0104] Step S203, adjusting the model parameters of the single-frame recognition model according to the expression feature parameter prediction result and the expression feature parameter label.

[0105] In this embodiment, for each of the third image frames, a loss value representing the data difference between the expression feature parameter prediction result and the expression feature parameter label can be calculated, and multiple iterations are performed for the purpose of reducing the loss value, and the loss value is adjusted to adjust the model parameters of each convolution unit and the fully connected unit in the single-frame recognition model. When the data difference between the expression feature parameter prediction result and the expression feature parameter label is within a preset range, or the data difference is no longer reduced, or the number of iterations reaches a set number, the single-frame recognition model training can be considered to be completed.

[0106] In addition, the method provided in this embodiment may include the following steps of training the offset prediction model.

[0107] Step S301, obtaining historical single-frame recognition results of a plurality of fourth image frames located before the third image frame in the training video data.

[0108] Step S302: inputting historical single-frame recognition results of the plurality of fourth image frames into an offset prediction model to be trained to obtain a predicted offset output by the offset prediction model.

[0109] Step S303, calculating and obtaining the actual offset according to the expression feature parameter label and the expression feature parameter prediction result of the third image frame.

[0110] Step S304: adjusting the model parameters of the offset prediction model according to the actual offset and the predicted offset.

[0111] Specifically, in this embodiment, the i-th frame in the training video data can be used as the third image frame, and the i-1 to im frames can be used as the fourth image frame. i , and m historical single-frame recognition result inputs of the fourth image frame i ={pred i-m ,...,pred i-1 Then, calculate the actual offset delta i =exp i -pred i After the expression feature parameter prediction result of the third image frame and the historical single frame recognition results of the m fourth image frames are input into the offset prediction model to be trained, the predicted offset is obtained. Then, the data difference between the actual offset and the predicted offset is calculated, and the first model parameter W in the offset prediction model is adjusted according to the obtained data difference. Q , the second model parameter W K and the third model parameter W V.

[0112] Based on the same inventive concept, this embodiment also provides a Figure 1 The live broadcast server 10 shown, please refer to Figure 4 , Figure 4 A block diagram of the live broadcast server 10. The live broadcast server 10 includes an expression feature parameter processing device 110, a machine-readable storage medium 120 and a processor 130.

[0113] The machine-readable storage medium 120 and the processor 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The expression feature parameter processing device 110 includes a plurality of software function modules that can be stored in the machine-readable storage medium 120 in the form of software or firmware or solidified in the operating system (OS) of the live broadcast server 10. The processor 130 is used to execute the executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the expression feature parameter processing device 110.

[0114] The machine-readable storage medium 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store a program, and the processor 130 executes the program after receiving an execution instruction.

[0115] The processor 130 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 130 may be a general-purpose processor 130, including a central processing unit 130 (CPU), a network processor 130 (NP), etc.; it may also be a digital signal processor 130 (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor 130 may be a microprocessor 130 or the processor 130 may also be any conventional processor 130, etc.

[0116] Please refer to Figure 5 This embodiment further provides an expression feature parameter processing device 110, which includes a plurality of functional modules that can be stored in a machine-readable storage medium 120 in the form of software. Functionally, the expression feature parameter processing device 110 may include a data acquisition module 111, a single frame recognition module 112, an offset determination module 113, and a result adjustment module 114.

[0117] The data acquisition module 111 is used to acquire a first image frame in a video to be recognized, and historical single-frame recognition results of a plurality of second image frames before the first image frame; the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame;

[0118] In this embodiment, the data acquisition module 111 can be used to execute Figure 2 As shown in step S110, for a detailed description of the data acquisition module 111, please refer to the description of step S110.

[0119] The single frame recognition module 112 is used to perform expression feature parameter recognition on the first image frame through a single frame recognition model to obtain a target single frame recognition result;

[0120] In this embodiment, the single frame recognition module 112 can be used to perform Figure 2 As shown in step S120, for a detailed description of the single frame identification module 112, please refer to the description of step S120.

[0121] The offset determination module 113 is used to input the target single-frame recognition result and the multiple historical single-frame recognition results into the offset prediction model based on the self-attention mechanism to obtain the recognition offset corresponding to the first image frame;

[0122] In this embodiment, the offset determination module 113 may be used to perform Figure 2 As shown in step S130, for a detailed description of the offset determination module 113, please refer to the description of step S130.

[0123] The result adjustment module 114 is used to adjust the target single-frame recognition result according to the recognition offset to obtain adjusted expression feature parameters.

[0124] In this embodiment, the result adjustment module 114 can be used to perform Figure 2 As shown in step S140 , for a detailed description of the result adjustment module 114 , please refer to the description of step S140 .

[0125] In summary, the present application provides an expression feature parameter processing method, device and electronic device, which determines the recognition offset to adjust the target single-frame recognition result of the first image frame based on the historical single-frame recognition results of multiple second images before the first image frame. In this way, the current target single-frame recognition result can be corrected according to the historical single-frame recognition results, and the data mutation errors that may exist in the target single-frame recognition results can be offset, so that the adjusted expression feature parameters are more in line with the changing trend of the historical single-frame recognition results, and the amount of calculation is relatively small, and the processing efficiency is higher.

[0126] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0128] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0130] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing expression feature parameters, characterized in that: The method comprises: Acquire a first image frame in a video to be recognized, perform expression feature parameter recognition on the first image frame using a single-frame recognition model, and obtain a target single-frame recognition result; Acquire historical single-frame recognition results of a plurality of second image frames before the first image frame; the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame; The target single-frame recognition result and the multiple historical single-frame recognition results are input into an offset prediction model to obtain a recognition offset corresponding to the first image frame; the recognition offset is calculated based on the change trend of the multiple historical single-frame recognition results of the second image frames and the target single-frame recognition result of the first image frame; the recognition offset represents the difference between the expression feature parameters predicted by the change trend of the multiple historical single-frame recognition results and the target single-frame recognition result; The target single-frame recognition result is adjusted according to the recognition offset to obtain adjusted expression feature parameters.

2. The method according to claim 1, characterized in that: The single-frame recognition model includes at least one convolution unit and a fully connected unit; the step of performing expression feature parameter recognition on the first image frame through the single-frame recognition model to obtain a target single-frame recognition result includes: Processing the first image frame by the at least one convolution unit to obtain an intermediate processing result; The intermediate processing result is processed by the fully connected unit to obtain the target single-frame recognition result, and the target single-frame recognition result includes a plurality of data representing the states of different facial organs.

3. The method according to claim 1, characterized in that: The step of inputting the target single-frame recognition result and the plurality of historical single-frame recognition results into an offset prediction model to obtain a recognition offset corresponding to the first image frame comprises: Based on the first model parameter, the second model parameter and the third model parameter obtained by pre-training in the offset prediction model, the recognition offset is calculated according to the target single-frame recognition result and the plurality of historical single-frame recognition results through an attention mechanism; Among them, the first model parameter and the second model parameter are used to calculate the correlation between the target single-frame recognition result and multiple historical single-frame recognition results, and the third model parameter is used to calculate the change trend of multiple historical single-frame recognition results.

4. The method according to claim 3, characterized in that The step of calculating the recognition offset based on the target single-frame recognition result and the plurality of historical single-frame recognition results by an attention mechanism based on the first model parameter, the second model parameter and the third model parameter obtained by pre-training in the offset prediction model comprises: Determine a first calculation vector of an attention function according to the target single-frame recognition result and a first model parameter of the offset prediction model; Determine a second calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset coding constant matrix and a second model parameter of the offset prediction model; Determining a third calculation vector of the attention function according to the plurality of historical single-frame recognition results, a preset coding constant matrix and a third model parameter of the offset prediction model; the coding constant matrix is ​​used to indicate the order of the plurality of historical single-frame recognition results; The recognition offset is calculated based on the first calculation vector, the second calculation vector and the third calculation vector through a self-attention function.

5. The method according to claim 4, characterized in that The step of determining the second calculation vector of the attention function according to the plurality of historical single-frame recognition results, the preset coding constant matrix and the second model parameter of the offset prediction model comprises: Adding a historical recognition result matrix composed of a plurality of the historical single-frame recognition results to the coding constant matrix to obtain a recognition result adjustment matrix, and multiplying the recognition result adjustment matrix by the second model parameter to obtain the second calculation vector; The step of determining the third calculation vector of the attention function according to the plurality of historical single-frame recognition results, the preset coding constant matrix and the third model parameter of the offset prediction model comprises: Adding a historical recognition result matrix composed of a plurality of historical single-frame recognition results to the coding constant matrix to obtain a recognition result adjustment matrix, and multiplying the recognition result adjustment matrix by the third model parameter to obtain the third calculation vector; Among them, in the coding constant matrix, the value of the data item corresponding to the second image frame that is close to the first image frame in time sequence is greater than the value of the data item corresponding to the second image frame that is far away from the first image frame in time sequence.

6. The method according to claim 1, characterized in that The method further comprises: Facial reconstruction is performed according to the adjusted expression feature parameters.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a third image frame in the training video data, and an expression feature parameter label corresponding to the third image frame; Inputting the third image frame into the single-frame recognition model to be trained to obtain the expression feature parameter prediction result output by the single-frame recognition model; The model parameters of the single-frame recognition model are adjusted according to the expression feature parameter prediction result and the expression feature parameter label.

8. The method according to claim 7, characterized in that The method further comprises: Acquire historical single frame recognition results of a plurality of fourth image frames located before the third image frame in the training video data; Inputting the historical single-frame recognition results of the plurality of fourth image frames into the offset prediction model to be trained, and obtaining the predicted offset output by the offset prediction model; The actual offset is calculated based on the expression feature parameter label of the third image frame and the expression feature parameter prediction result; The model parameters of the offset prediction model are adjusted according to the actual offset and the predicted offset.

9. An expression feature parameter processing device, characterized in that: The expression feature parameter processing device comprises: A data acquisition module, used to acquire a first image frame in a video to be recognized, and historical single-frame recognition results of a plurality of second image frames before the first image frame; the historical single-frame recognition results include expression feature parameters obtained by performing expression feature parameter recognition on the second image frame; A single-frame recognition module, used to perform expression feature parameter recognition on the first image frame through a single-frame recognition model to obtain a target single-frame recognition result; An offset determination module, used to input the target single-frame recognition result and the plurality of historical single-frame recognition results into an offset prediction model based on a self-attention mechanism to obtain a recognition offset corresponding to the first image frame; The result adjustment module is used to adjust the target single-frame recognition result according to the recognition offset to obtain adjusted expression feature parameters.

10. An electronic device, characterized in that: The method comprises a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by one or more processors, the method according to any one of claims 1 to 8 is implemented.

Images