Data processing method, apparatus, device, medium, and program product

Abstract

The application discloses a data processing method and device, equipment, medium and program product. The method comprises the following steps: obtaining a face key point prediction model when target audio data is acquired, wherein the face key point prediction model at least comprises N first feature extraction networks, a first residual network and a key point prediction network, and the first feature extraction network comprises a multi-head attention mechanism; inputting target audio parameters into the N first feature extraction networks respectively, obtaining N groups of different first target audio features of the target audio parameters through the N first feature extraction networks, and the target audio parameters are parameters extracted from the target audio data; obtaining a first key audio feature by using the first residual network according to the N groups of first target audio features; and inputting the first key audio feature into the key point prediction network, and obtaining a face key point by using the key point prediction network. The accuracy of face key point prediction is high.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a data processing method, apparatus, device, medium and program product. Background Technology

[0002] Artificial intelligence and augmented reality (AR) technologies are developing rapidly, and internet users have increasingly diverse content consumption needs, with a growing demand for content such as animation, virtual digital humans, and AR.

[0003] In existing technologies, deep learning models such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) are typically used. Audio and video are used as input to these models, and the output is used to predict facial landmarks. However, deep learning models are highly complex, have many parameters, and require a large amount of training data. When training data is insufficient, the accuracy of the model's recognition is low.

[0004] Existing methods for predicting facial landmarks have low accuracy. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, medium, and program product that can improve the accuracy of facial landmark prediction.

[0006] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0007] Given the target audio data, a facial landmark prediction model is obtained. The facial landmark prediction model includes at least N first feature extraction networks, a first residual network, and a landmark prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism.

[0008] The target audio parameters are respectively input into the N first feature extraction networks, and N sets of first target audio features of the target audio parameters are obtained through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data.

[0009] Based on the N sets of first target audio features, the first key audio features are obtained using the first residual network;

[0010] The first key audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

[0011] On the other hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0012] The model acquisition module is used to acquire a facial key point prediction model when the target audio data is acquired. The facial key point prediction model includes at least: N first feature extraction networks, a first residual network, and a key point prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism.

[0013] The first feature extraction module is used to input the target audio parameters into the N first feature extraction networks respectively, and obtain N sets of first target audio features of the target audio parameters through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data.

[0014] The second feature extraction module is used to obtain the first key audio feature based on the N sets of first target audio features using the first residual network;

[0015] The prediction module is used to input the first key audio feature into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0016] In another aspect, embodiments of this application provide a data processing device, the device including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the data processing method described above.

[0017] In another aspect, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data processing method described above is implemented.

[0018] In another aspect, embodiments of this application provide a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform the data processing method described above.

[0019] The data processing method, apparatus, device, and computer storage medium of this application embodiment can acquire a facial landmark prediction model when target audio data is obtained. The facial landmark prediction model includes at least: N first feature extraction networks, a first residual network, and a landmark prediction network. The first feature extraction networks include a multi-head attention mechanism. Target audio parameters are input into the N first feature extraction networks respectively, and N different sets of first target audio features of the target audio parameters are obtained through the N first feature extraction networks. The target audio parameters are parameters extracted from the target audio data. Based on the N sets of first target audio features, the first residual network is used to obtain first key audio features. The first key audio features are input into the landmark prediction network, and facial landmarks are obtained through the landmark prediction network. By using N feature extraction networks including a multi-head attention mechanism, multiple different sets of audio features of the target audio parameters can be extracted. Then, the multiple sets of audio features are passed through the residual network to obtain key audio features. The facial landmarks corresponding to the audio parameters are predicted through the key audio features, resulting in high accuracy in facial landmark prediction. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of a data processing method provided in one embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a first feature extraction network provided in one embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a first feature extraction network provided in another embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a facial landmark prediction model provided in one embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this application. Detailed Implementation

[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0029] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0030] A fully connected layer, also known as a densely connected layer or a linear layer, is a common layer type in neural networks. Its function is to connect all neurons in the input layer or the previous layer with all neurons in the current layer.

[0031] Multi-head attention is an extension of self-attention, commonly used in sequence modeling tasks such as machine translation and language generation. It allows the model to use multiple self-attention mechanisms on different feature representations and fuse their outputs. By using multiple attention heads, the model can simultaneously focus on different features to better capture information from the input sequence. This parallel processing also helps improve computational efficiency and increase the model's expressive power.

[0032] Activation layers are a common type of layer in neural networks, used to introduce nonlinearity. They map the linearly transformed output to another space by performing a point-wise nonlinear transformation on the input data.

[0033] The inner product layer performs a linear transformation on the input data and maps it to the output space. The core operations of the inner product layer are matrix multiplication and bias addition. Inner product layers are commonly used in the feedforward process of neural networks. In deep neural networks, multiple consecutive inner product layers are typically used to progressively extract and combine features from the input data. Non-linear activation function layers, such as ReLU, Sigmoid, or Tanh, are often inserted between inner product layers to enhance the network's non-linear modeling capabilities.

[0034] Residual calculation refers to a computational method used in deep learning when employing residual connections. Specifically, it involves establishing skip connections using residual connections in deep learning. By adding the input to the output of the main nonlinear transformation, information is directly passed to subsequent layers. This technique helps address gradient vanishing and model degradation problems, improving the training and performance of deep neural networks.

[0035] Batch normalization is a neural network optimization technique designed to improve stability and convergence speed during training. It normalizes the input distribution across different layers of the network by standardizing each mini-batch of data. In deep neural networks, the input distribution of each layer can change as the signal propagates through the network; this is known as "internal covariate shift." Internal covariate shift can make the network difficult to converge, require a smaller learning rate, and be highly sensitive to the choice of initialization parameters. Batch normalization addresses internal covariate shift by calculating the mean and standard deviation on each mini-batch of data and then standardizing the data.

[0036] Merging and concatenation refers to the operation of merging or connecting multiple layers or feature maps in deep learning. Merging and concatenation can be used for different tasks and network architectures, such as multi-scale feature fusion, residual networks, and attention mechanisms.

[0037] In the process of cascaded mergers, there are generally two main approaches:

[0038] Concatenation: This involves joining the outputs of two or more layers along a dimensional axis. For example, if two feature maps have dimensions [h,w,c1] and [h,w,c2], the concatenated feature map will have dimensions [h,w,c1+c2]. This method allows for the concatenation of information from different layers or feature maps along the channel dimension, thus providing a richer feature representation.

[0039] Addition: This involves element-wise summing the outputs of two or more layers. In residual networks, a skip connection is added to add the original input to the transformed output, yielding the final output. This allows the network to learn the residual between the input and output, further improving model performance.

[0040] In the field of lip-sound recognition, using audio and video as input to a deep learning model to output digital human facial expressions and lip-sound synchronization technology presents the following problems in deep learning model training and optimization:

[0041] 1. Deep learning networks use convolutional neural networks or recurrent neural networks, which are relatively complex, slow to train, and have many model parameters.

[0042] 2. Deep learning networks using convolutional neural networks or recurrent neural networks do not perform well during training and require high computing power.

[0043] 3. The limited amount of training data in current systems for lip sound recognition and facial expression emotion recognition leads to insufficient adaptation of the model to new samples during final testing, resulting in poor model generalization ability.

[0044] Based on the above research, in order to solve the problems of the prior art, this application provides a data processing method, apparatus, device and computer storage medium.

[0045] The data processing method provided in the embodiments of this application will be described below.

[0046] Figure 1 A flowchart illustrating a data processing method provided in one embodiment of this application is shown. Figure 1 As shown, the data processing method provided in this application embodiment includes the following steps: S101 to S105:

[0047] S101. Given the target audio data, obtain a facial landmark prediction model. The facial landmark prediction model includes at least: N first feature extraction networks, a first residual network, and a landmark prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism.

[0048] S102. Input the target audio parameters into N first feature extraction networks respectively, and obtain N sets of first target audio features of the target audio parameters through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data.

[0049] S103. Based on the N sets of first target audio features, the first key audio features are obtained using the first residual network.

[0050] S104. Input the first key audio feature into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0051] The data processing method of this application embodiment can acquire a facial landmark prediction model when target audio data is obtained. The facial landmark prediction model includes at least: N first feature extraction networks, a first residual network, and a landmark prediction network. The first feature extraction networks include a multi-head attention mechanism. Target audio parameters are input into the N first feature extraction networks respectively, and N different sets of first target audio features are obtained through the N first feature extraction networks. The target audio parameters are parameters extracted from the target audio data. Based on the N sets of first target audio features, the first residual network is used to obtain first key audio features. The first key audio features are input into the landmark prediction network, and the facial landmarks are obtained through the landmark prediction network. By using N feature extraction networks including a multi-head attention mechanism, multiple different sets of audio features of the target audio parameters can be extracted. Then, the multiple sets of audio features are passed through the residual network to obtain key audio features. The facial landmarks corresponding to the audio parameters are predicted through the key audio features, resulting in high accuracy in facial landmark prediction.

[0052] It is understood that the methods provided in the embodiments of this application can be executed by a computer device, which includes, but is not limited to, a terminal device or a business server.

[0053] The steps of the data processing method provided in the embodiments of this application will be described below.

[0054] In S101, a facial landmark prediction model is obtained after acquiring the target audio data.

[0055] Specifically, taking a computer device as the executing entity as an example, the computer device can acquire the target audio data. When the target audio data is stored locally, it can be extracted from the file itself; alternatively, when the target audio data is stored on another computer device, it can be sent to the local device for processing by that other computer device. This application embodiment does not limit the method of acquiring the target audio data.

[0056] Furthermore, when acquiring target audio data, the terminal device can acquire a facial landmark prediction model for predicting facial landmarks in the target audio data.

[0057] The facial landmark prediction model includes at least: N first feature extraction networks, a first residual network, and a landmark prediction network, where N is a positive integer greater than 1, and the first feature extraction network includes a multi-head attention mechanism.

[0058] Multi-head attention is an attention mechanism used to enhance the representational capabilities of a model. Traditional attention mechanisms calculate the similarity between the query, key, and value to obtain the attention distribution, and then weighted summation of the value and the attention distribution yields the output. Multi-head attention introduces multiple attention heads, each with its own independent query, key, and value. The attention distribution is calculated in parallel, and the outputs of the different heads are concatenated or linearly transformed to obtain the final attention representation. By introducing multi-head attention, the feature extraction network can simultaneously focus on information at different locations and semantic levels, thus better capturing task-relevant features. Each attention head can learn different weight distributions, providing richer representational capabilities. Furthermore, multi-head attention can be computed in parallel, improving the training and inference efficiency of the model.

[0059] In some embodiments, the target audio parameters described above can be obtained through the following steps:

[0060] Acquire target audio data;

[0061] The target audio data is input into the linear predictive coding model, and the target audio parameters are extracted from the target audio data through the linear predictive coding model.

[0062] Linear Predictive Coding (LPC) is a speech signal compression technique used to encode speech signals into a more efficient representation to reduce storage space or transmission bandwidth. Therefore, the spectral features of the speech signal in the target audio data can be extracted using the LPC model, thereby allowing the extraction of the target audio parameters.

[0063] In one example, the target audio parameter can be a parameter related to the formant frequency, bandwidth, or other characteristics of the audio data.

[0064] In this embodiment, a linear prediction model can be used to parameterize the target audio data to obtain the target audio parameters of the target audio data.

[0065] In S102 above, the target audio parameters are input into N first feature extraction networks respectively, and N sets of first target audio features of the target audio parameters are obtained through the N first feature extraction networks.

[0066] The first feature extraction network is used to extract the first target audio feature from the target audio parameters of the target audio data.

[0067] Furthermore, by utilizing N first feature extraction networks, N sets of first target audio features can be obtained from the target audio parameters. Since each attention head in the multi-head attention of each first feature extraction network can calculate the corresponding attention distribution, multiple different sets of first target audio features can be obtained, thereby extracting richer audio features from the target audio data.

[0068] In one example, the first feature extraction network described above is as follows: Figure 2 The diagram shows a multi-head attention mechanism. The multi-head attention mechanism includes three attention mechanisms (attention mechanisms for the paths containing fully connected layer 2, fully connected layer 3, and fully connected layer 5); each attention mechanism is connected by a cascaded merging component 1. Taking the attention mechanism for the path containing fully connected layer 2 as an example, this attention mechanism includes fully connected layer 2, activation function layer 1 (example: Activation1(Softmax)), and inner product operation layer 1 (example: Multiply1(Activation1,Dense1)).

[0069] Fully connected layer 2 primarily receives and transmits feature information from fully connected layer 1. The target audio parameters output from fully connected layer 1 are input to fully connected layer 2. The attention mechanism within fully connected layer 2 uses a softmax operation in activation function layer 1 to normalize the data vector in the network layer into a probability distribution vector, where the sum of all probabilities is 1. Then, in inner product operation layer 1, the outputs of activation function layer 1 and fully connected layer 1 are multiplied by a matrix operation to obtain the output value of the attention mechanism. The output values ​​from each attention mechanism are input to the concatenation component 1, which performs a concatenation operation to merge the output values ​​of the three attention mechanisms to obtain the first target audio feature. Concatenation effectively focuses the attention of feature extraction on more critical and useful audio features.

[0070] In S103 above, the first key audio features are obtained by using the first residual network based on N sets of first target audio features.

[0071] The first residual network allows for residual calculation of the input values ​​(target audio parameters) and output values ​​(first target audio features) of the first feature extraction network to obtain key audio features. Residual calculation involves linear superposition operations within the network layers. By using residual calculation, the entire network only needs to learn the difference between the input and output, simplifying the learning objective and reducing the difficulty.

[0072] For example, the first residual network can include... Figure 3 The residual calculation components 1 and 2 may optionally include a batch standardization layer.

[0073] In some embodiments, the target audio parameters include N sub-audio parameters, the first residual network includes N residual calculation components, and S103 above may include:

[0074] Based on N sets of first target audio features, the first key audio features are obtained using the first residual network, including:

[0075] When i equals 1, the first sub-audio parameter and the first group of first target audio features are input into the first residual calculation component, and the first difference feature is obtained through the first residual calculation component.

[0076] Where i is a positive integer, N sub-audio parameters include the first sub-audio parameter, the first group of first target audio features are the audio features obtained by inputting the first sub-audio parameter into the first first feature extraction network, N residual calculation components include the first residual calculation component, and N first feature extraction networks include the first first feature extraction network.

[0077] When i is greater than 1 and less than or equal to N, the i-th sub-audio parameter, the i-th group of first target audio features, and the (i-1)-th difference feature are input into the i-th residual calculation component, and the i-th difference feature is obtained through the i-th residual calculation component.

[0078] Wherein, the first target audio feature of the i-th group is the first target audio feature obtained by inputting the i-th sub-audio parameter into the i-th first feature extraction network, the N first feature extraction networks include the i-th first feature extraction network, the i-1 difference feature is the sub-difference feature obtained by the i-1 residual calculation component, and the N residual calculation components include the i-1 residual calculation component and the i-th residual calculation component;

[0079] Update the value of i to i+1, and return to the step of inputting the i-th sub-audio parameter, the i-th group of audio features and the i-1-th difference feature into the i-th residual calculation component and obtaining the i-th sub-difference feature through the i-th residual calculation component, until i is greater than N;

[0080] The i-th difference feature is identified as the first key audio feature.

[0081] Specifically, a process for calculating differential features based on sub-audio parameters and target audio features is implemented. Each step depends on the output of the previous step and is executed iteratively by updating the value of i until the final first key audio feature is calculated. This addresses the problems of vanishing gradients and network degradation by combining a multi-head attention mechanism with multiple residual calculation components; and it also allows for the extraction of more key features from the audio data.

[0082] In one example, such as Figure 3 Assuming N is 2, then S102 to S103 above can specifically include the following steps:

[0083] S301: Employs parallel multi-feature input and obtains target audio parameters based on Linear Predictive Coding (LPC) features. Dimensionality reduction of the multi-target audio parameters can be achieved through a Global Average Pooling component. Global Average Pooling is a structured regularization operation that compresses the three-dimensional feature information (frame count, features, and channel count) from the preceding network layers into flattened one-dimensional data, facilitating weight adjustment in subsequent fully connected layers. The Global Average Pooling operation transforms the input three-dimensional data into one-dimensional data.

[0084] S302: Fully Connected Layer Component. This layer receives feature information from all previous layers and provides a better transition between network layer parameters and the network output label points. It has 256 data points. Weight optimization is performed through this fully connected layer, which receives global average pooling feature information from the previous layers. The fully connected layer effectively bridges the gap between the input training feature information and the multi-head attention mechanism.

[0085] The first feature extraction component includes a multi-head attention mechanism component. The multi-head attention mechanism consists of three attention mechanisms cascaded and merged through a Concat operation. Taking the attention mechanism of the path containing fully connected layer 2 as an example, this attention mechanism includes a fully connected layer 2, an activation function layer Activation1 (Softmax), and a Multiply1 (Activation1, fully connected layer 1) inner product operation. The fully connected layer 2 mainly obtains and passes feature information from fully connected layer 1.

[0086] S303. After merging cascaded layer 1, a fully connected layer is added for transition. Then, residual connection (i.e., the first residual component) is used to linearly superimpose fully connected layer 1 and fully connected layer 5. The residual calculation is used to learn the part of the difference between the input and output of fully connected layer 1 and fully connected layer 5. Finally, batch normalization 1 is performed to divide the data into small batches for stochastic gradient descent.

[0087] S304, Residual Component 2 (Second Residual Component). Residual Component 2 performs residual calculations on the outputs of Batch Normalization 1, Fully Connected Layer 6, and Fully Connected Layer 10 to obtain the first key audio feature. The first key audio feature can also be batch normalized (e.g., Batch Normalization 2).

[0088] In S104 above, the first key audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

[0089] A keypoint prediction network is a model capable of predicting facial keypoints from a given input (such as audio, video, or image). It analyzes audio or video data and generates corresponding facial keypoint coordinates. These keypoints can be used for tasks such as facial expression recognition, pose estimation, and face recognition. The keypoint prediction network maps the first key audio features to facial keypoints. Specific implementation methods vary depending on the network architecture and training dataset used. A keypoint prediction network can be tailored to the specific task requirements, and the model can be trained and used for prediction based on this network.

[0090] In some embodiments, the facial landmark prediction model further includes a first fusion component.

[0091] Following S103 above, it also includes:

[0092] Obtain the first target emotion parameter, which is a parameter extracted from the target audio data;

[0093] The first target emotion parameter and the first key audio feature are input into the first fusion component. The first fusion component performs a merging and concatenation operation on the first target emotion parameter and the first key audio feature to obtain the second target audio feature.

[0094] The above S104 includes:

[0095] The second target audio features are input into the keypoint prediction network, and the facial keypoints corresponding to the target audio data are obtained through the keypoint prediction network.

[0096] In this embodiment, a first emotion parameter of the same audio data is added; the first emotion parameter is merged with the first fusion feature to increase the emotion feature of the speech data.

[0097] The first emotion vector is input as Gaussian white noise distribution data of type [?, 258], where 258 is the dimension of the Gaussian white noise distribution data. In this application, it is designed to be 258, but this parameter can be modified according to different model requirements.

[0098] During model training, for input samples containing emotions, to ensure a more natural and fitting audio-to-facial-expression synchronization, the initial Gaussian white noise sampling data was randomly generated. During training, the weights in the network were continuously updated by combining the audio features, constantly adjusting and optimizing the subsequent emotion vector matrix. The training combined the emotion feature vectors learned from different emotion data to form an emotion state matrix, which was then merged and concatenated with the first key audio feature.

[0099] Optionally, emotional feature parameters can be analyzed and extracted using the acoustic features of audio data. Examples include tone analysis, energy analysis, spectrum analysis, speech rate analysis, and sound quality analysis. Furthermore, machine learning algorithms and sentiment analysis models can be used to extract more specific and accurate emotional features from audio.

[0100] In some embodiments, the facial landmark prediction model further includes a second feature extraction network and a second residual network, wherein the second feature extraction network includes a multi-head attention mechanism.

[0101] After inputting the first target emotion parameter and the first key audio feature into the first fusion component, and merging and concatenating the first target emotion parameter and the first key audio feature through the first fusion component to obtain the second target audio feature, the process further includes:

[0102] The second target audio features are input into the second feature extraction network, and the third target speech features are obtained through the second feature extraction network.

[0103] The third target speech features and the first target emotion parameters are input into the second residual network, and the residual calculation is performed using the second residual network to obtain the second key audio features.

[0104] The above-mentioned second target audio features are input into the keypoint prediction network. Through the keypoint prediction network, the facial keypoints corresponding to the target audio data are obtained, including:

[0105] The second key audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

[0106] In this embodiment, a second feature extraction network based on a multi-head attention mechanism and a second residual network are added to further extract key features from the second target speech features to obtain the third target speech features.

[0107] In one example, such as Figure 4 The diagram shows a partial structural representation of the facial landmark prediction model.

[0108] S401: Emotion Vector Input Component. During the training phase, it serves as the third parallel input in the network. The emotion vector input is Gaussian white noise distribution data of type [?, 258], where 258 represents the dimension of the Gaussian white noise distribution data. This network model is designed with 258, but this parameter can be modified according to different model requirements. For input samples with emotions, to make the audio synchronization with facial expressions more natural and fitting, the initial Gaussian white noise sampling data is randomly generated. During training, the weights are continuously updated in the network by combining with audio features, and the subsequent emotion vector matrix is ​​continuously adjusted and optimized. The training combines the emotion feature vectors learned from different emotion data to form an emotion state matrix, which is then merged and concatenated with the audio features (i.e., merged and concatenated through the first fusion component).

[0109] In some embodiments, the facial landmark prediction model further includes a second fusion component.

[0110] After inputting the second target speech features and the second target emotion parameters into the second residual network, and using the second residual network to perform residual processing to obtain the second key audio features, the process further includes:

[0111] The second key audio feature and the second target emotion parameter are input into the second fusion component, and the second fusion component is used to perform merging and cascading processing to obtain the third target audio feature. The second target emotion parameter is a parameter extracted from the target audio data.

[0112] The second key audio feature is input into the keypoint prediction network. Through the keypoint prediction network, the facial keypoints corresponding to the target audio data are obtained, including:

[0113] The third target audio feature is input into a keypoint prediction network, which then obtains the facial keypoints corresponding to the target audio data. Specifically, user credit data can be input into pre-trained supervised and unsupervised deep learning models respectively. Combining the supervised and unsupervised deep learning models determines the credit representation value of each user in the user dataset, improving the accuracy of data processing.

[0114] In this embodiment, after obtaining the second fused audio feature, the second fused audio feature and the second target emotion parameter are merged and concatenated to further increase the feature information.

[0115] In one example, such as Figure 5 Step S402:

[0116] The audio feature and emotion feature fusion cascade component (i.e., the first fusion component) merges and cascades the first key audio feature and the first emotion vector feature to achieve dual feature fusion. It is worth noting that the dimensions of the two features are consistent, which effectively increases the connection between different features during the fusion and cascade operation. After fusing the two features through the fusion operation, the feature information is effectively increased, further mitigating the gradient vanishing problem and suppressing overfitting.

[0117] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of a data processing apparatus. Please refer to the following embodiments.

[0118] First see Figure 6 The data processing apparatus 600 provided in this application embodiment includes the following units:

[0119] The model acquisition module 601 is used to acquire a facial key point prediction model when the target audio data is acquired. The facial key point prediction model includes at least: N first feature extraction networks, a first residual network, and a key point prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism.

[0120] The first feature extraction module 502 is used to input the target audio parameters into N first feature extraction networks respectively, and obtain N sets of first target audio features of the target audio parameters through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data.

[0121] The second feature extraction module 503 is used to obtain the first key audio features based on N sets of first target audio features using the first residual network;

[0122] The prediction module 504 is used to input the first key audio features into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0123] The data processing apparatus 500 provided in this application can acquire a facial landmark prediction model when target audio data is obtained. The facial landmark prediction model includes at least: N first feature extraction networks, a first residual network, and a landmark prediction network. The first feature extraction networks include a multi-head attention mechanism. Target audio parameters are input into the N first feature extraction networks respectively, and N different sets of first target audio features are obtained through the N first feature extraction networks. The target audio parameters are parameters extracted from the target audio data. Based on the N sets of first target audio features, the first residual network is used to obtain first key audio features. The first key audio features are input into the landmark prediction network, and the facial landmarks are obtained through the landmark prediction network. By using N feature extraction networks including a multi-head attention mechanism, multiple different sets of audio features of the target audio parameters can be extracted. Then, the multiple sets of audio features are passed through the residual network to obtain key audio features. The facial landmarks corresponding to the audio parameters are predicted through the key audio features, resulting in high accuracy in facial landmark prediction.

[0124] In some embodiments, the target audio parameters include N sub-audio parameters, the first residual network includes N residual calculation components, and the second feature extraction component 503 includes:

[0125] The first residual calculation unit is used to input the first sub-audio parameter and the first group of first target audio features into the first residual calculation component when i equals 1, and obtain the first difference feature through the first residual calculation component.

[0126] Where i is a positive integer, N sub-audio parameters include the first sub-audio parameter, the first group of first target audio features are the audio features obtained by inputting the first sub-audio parameter into the first first feature extraction network, N residual calculation components include the first residual calculation component, and N first feature extraction networks include the first first feature extraction network.

[0127] The second residual calculation unit is used to input the i-th sub-audio parameter, the i-th group of first target audio features, and the (i-1)-th difference feature into the i-th residual calculation component when i is greater than 1 and less than or equal to N, and obtain the i-th difference feature through the i-th residual calculation component.

[0128] Wherein, the first target audio feature of the i-th group is the first target audio feature obtained by inputting the i-th sub-audio parameter into the i-th first feature extraction network, the N first feature extraction networks include the i-th first feature extraction network, the i-1 difference feature is the sub-difference feature obtained by the i-1 residual calculation component, and the N residual calculation components include the i-1 residual calculation component and the i-th residual calculation component;

[0129] Update the value of i to i+1, and return to the step of inputting the i-th sub-audio parameter, the i-th group of audio features and the i-1-th difference feature into the i-th residual calculation component and obtaining the i-th sub-difference feature through the i-th residual calculation component, until i is greater than N;

[0130] The first determining unit is used to determine the i-th difference feature as the first key audio feature.

[0131] In some embodiments, the facial landmark prediction model further includes a first fusion component.

[0132] The aforementioned data processing apparatus 500 may further include:

[0133] The parameter acquisition module is used to acquire the first target emotion parameter, which is a parameter extracted from the target audio data;

[0134] The first fusion module is used to input the first target emotion parameter and the first key audio feature into the first fusion component, and to perform a merging and concatenation operation on the first target emotion parameter and the first key audio feature through the first fusion component to obtain the second target audio feature.

[0135] The aforementioned prediction module 504 includes:

[0136] The first prediction unit is used to input the second target audio features into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0137] In some embodiments, the facial landmark prediction model further includes a second feature extraction network and a second residual network, wherein the second feature extraction network includes a multi-head attention mechanism.

[0138] The aforementioned data processing apparatus 500 may further include:

[0139] The third feature extraction module is used to input the second target audio features into the second feature extraction network, and obtain the third target speech features through the second feature extraction network;

[0140] The fourth feature extraction module inputs the second target speech features and the first target emotion parameters into the second residual network, and uses the second residual network to perform residual calculation processing to obtain the second key audio features;

[0141] The aforementioned prediction module 504 also includes:

[0142] The second prediction unit is used to input the second key audio features into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0143] In some embodiments, the facial landmark prediction model further includes a second fusion component.

[0144] The aforementioned data processing apparatus 500 may further include:

[0145] The second fusion module is used to input the second key audio feature and the second target emotion parameter into the second fusion component, and use the second fusion component to perform merging and cascading processing to obtain the third target audio feature. The second target emotion parameter is a parameter extracted from the target audio data.

[0146] The aforementioned prediction module 504 also includes:

[0147] The third prediction unit is used to input the third target audio features into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

[0148] In some embodiments, the data processing apparatus 500 described above may further include:

[0149] The audio acquisition module is used to acquire target audio data;

[0150] The parameter extraction module is used to input the target audio data into the linear predictive coding model and extract the target audio parameters from the target audio data through the linear predictive coding model.

[0151] Figure 6 A schematic diagram of the hardware structure of the data processing device provided in an embodiment of this application is shown.

[0152] The data processing device may include a processor 601 and a memory 602 storing computer program instructions.

[0153] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0154] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0155] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the data processing methods according to any embodiment of this disclosure.

[0156] The processor 601 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0157] In one example, the data processing device may further include a communication interface 603 and a bus 610. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0158] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0159] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Module Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0160] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments.

[0161] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0162] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0163] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0164] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0165] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Given the target audio data, a facial landmark prediction model is obtained. The facial landmark prediction model includes at least N first feature extraction networks, a first residual network, and a landmark prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism. The target audio parameters are respectively input into the N first feature extraction networks, and N sets of first target audio features of the target audio parameters are obtained through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data. Based on the N sets of first target audio features, the first key audio features are obtained using the first residual network; The first key audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

2. The method according to claim 1, characterized in that, The target audio parameters include N sub-audio parameters, and the first residual network includes N residual calculation components. The step of obtaining the first key audio feature using the first residual network based on the N sets of first target audio features includes: When i equals 1, the first sub-audio parameter and the first group of first target audio features are input into the first residual calculation component, and the first difference feature is obtained through the first residual calculation component. Where i is a positive integer, the N sub-audio parameters include the first sub-audio parameter, the first group of first target audio features are audio features obtained by inputting the first sub-audio parameter into the first first feature extraction network, the N residual calculation components include the first residual calculation component, and the N first feature extraction networks include the first first feature extraction network; When i is greater than 1 and less than or equal to N, the i-th sub-audio parameter, the i-th group of first target audio features, and the (i-1)-th difference feature are input to the i-th residual calculation component, and the i-th difference feature is obtained through the i-th residual calculation component. Wherein, the i-th group of first target audio features is the first target audio feature obtained by inputting the i-th sub-audio parameters into the i-th first feature extraction network, the N first feature extraction networks include the i-th first feature extraction network, the (i-1)-th difference feature is the sub-difference feature obtained through the (i-1)-th residual calculation component, and the N residual calculation components include the (i-1)-th residual calculation component and the i-th residual calculation component; Update the value of i to i+1, and return to the step of inputting the i-th sub-audio parameter, the i-th group of audio features and the i-1-th difference feature into the i-th residual calculation component when i is greater than 1 and less than or equal to N, and obtaining the i-th sub-difference feature through the i-th residual calculation component, until i is greater than N; The i-th difference feature is determined as the first key audio feature.

3. The method according to claim 1, characterized in that, The facial landmark prediction model also includes a first fusion component. After obtaining the first key audio features using the first residual network based on the N sets of first target audio features, the process further includes: Obtain a first target emotion parameter, wherein the first target emotion parameter is a parameter extracted from the target audio data; The first target emotion parameter and the first key audio feature are input into the first fusion component. The first fusion component performs a merging and concatenation operation on the first target emotion parameter and the first key audio feature to obtain the second target audio feature. The step of inputting the first key audio feature into the key point prediction network, and obtaining the facial key points corresponding to the target audio parameters through the key point prediction network, includes: The second target audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

4. The method according to claim 3, characterized in that, The facial landmark prediction model further includes a second feature extraction network and a second residual network. The second feature extraction network includes a multi-head attention mechanism. After inputting the first target emotion parameter and the first key audio feature into the first fusion component, and merging and concatenating the first target emotion parameter and the first key audio feature through the first fusion component to obtain the second target audio feature, the method further includes: The second target audio features are input into the second feature extraction network, and the third target speech features are obtained through the second feature extraction network. The third target speech feature and the second target emotion parameter are input into the second residual network, and residual calculation is performed using the second residual network to obtain the second key audio feature. The second target emotion parameter is a parameter extracted from the target audio data. The step of inputting the second target audio features into the keypoint prediction network, and obtaining the facial keypoints corresponding to the target audio data through the keypoint prediction network, includes: The second key audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

5. The method according to claim 4, characterized in that, The facial landmark prediction model also includes a second fusion component. After inputting the third target speech features and the second target emotion parameters into the second residual network, and using the second residual network to perform residual calculation processing to obtain the second key audio features, the method further includes: The second key audio feature and the second target emotion parameter are input into the second fusion component, and the second fusion component is used to perform merging and concatenation processing to obtain the third target audio feature. The step of inputting the second key audio feature into the key point prediction network, and obtaining the facial key points corresponding to the target audio data through the key point prediction network, includes: The third target audio feature is input into the key point prediction network, and the facial key points corresponding to the target audio data are obtained through the key point prediction network.

6. The method according to claim 1, characterized in that, The method further includes: Acquire target audio data; The target audio data is input into a linear predictive coding model, and the target audio parameters are extracted from the target audio data through the linear predictive coding model.

7. A data processing apparatus, characterized in that, The device includes: The model acquisition module is used to acquire a facial key point prediction model when the target audio data is acquired. The facial key point prediction model includes at least: N first feature extraction networks, a first residual network, and a key point prediction network, where N is a positive integer greater than 1. The first feature extraction network includes a multi-head attention mechanism. The first feature extraction module is used to input the target audio parameters into the N first feature extraction networks respectively, and obtain N sets of first target audio features of the target audio parameters through the N first feature extraction networks. The N sets of first target audio features are different audio features, and the target audio parameters are parameters extracted from the target audio data. The second feature extraction module is used to obtain the first key audio feature based on the N sets of first target audio features using the first residual network. The prediction module is used to input the first key audio feature into the key point prediction network, and obtain the facial key points corresponding to the target audio data through the key point prediction network.

8. A data processing device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data processing method as described in any one of claims 1-6.

Images