A face image generation method and apparatus

Abstract

This application discloses a method and apparatus for generating facial images. The method includes: obtaining a speech to be tested; extracting at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein the multiple first speech feature data correspond to multiple facial dimension information; inputting the multiple first speech feature data into a speech recognition model; and outputting predicted values ​​of each facial dimension corresponding to the speech frame; wherein the predicted values ​​of the correlated facial dimensions satisfy constraint conditions; and generating a facial image corresponding to the speech frame based on the multiple predicted values.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method and apparatus for generating facial images. Background Technology

[0002] Current methods for generating facial expressions based on speech extract multiple speech feature data points from the speech, then input these features into a speech recognition model. The model then predicts the values ​​of all the speech feature data. Therefore, existing methods do not consider the correlation between the speech feature data, resulting in low accuracy of the predicted values. Summary of the Invention

[0003] This application provides a method and apparatus for generating facial images.

[0004] This application provides a method for generating a facial image, the method comprising:

[0005] Obtain the speech to be tested, and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information;

[0006] Multiple first speech feature data are input into a speech recognition model, and the predicted values ​​of each facial dimension corresponding to the speech frame are output; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0007] A facial image corresponding to the speech frame is generated based on multiple predicted values.

[0008] The method also includes:

[0009] Multiple speech samples are acquired, each speech sample containing multiple second speech feature data and label values ​​and correlation data for each second speech feature data, wherein the correlation data indicates the correlation between facial dimensions; wherein, the multiple second speech feature data correspond to multiple facial dimension information;

[0010] Input all the second speech feature data of the speech sample into the initial speech recognition model, and output the predicted value of each second speech feature data; wherein, the predicted values ​​of the related facial dimensions satisfy the constraint conditions;

[0011] The label values ​​of the second speech feature data are corrected based on the correlation data of the second speech feature data.

[0012] The loss value of the speech sample is calculated based on the predicted values ​​of all second speech feature data and the corrected label values ​​of the speech sample.

[0013] The parameters of the initial speech recognition model are optimized based on the loss value to obtain a speech recognition model.

[0014] The correlation includes symmetry, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0015] The predicted values ​​of symmetrical facial dimensions are constrained by a mean and an offset, wherein the mean is the average of the predicted values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the predicted value and the mean.

[0016] The correlation includes mutual exclusion, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0017] The predicted values ​​of mutually exclusive facial dimensions are constrained by their magnitude, and at most one of the predicted values ​​of the multiple mutually exclusive facial dimensions is not zero.

[0018] The correlation includes positive or negative correlation, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0019] The predicted values ​​of facial dimensions with positive or negative correlation are constrained by numerical comparison, and the values ​​of multiple predicted values ​​of facial dimensions with positive or negative correlation satisfy a specified proportional relationship.

[0020] The method also includes:

[0021] The multiple first speech feature data of the speech frame are classified according to multiple facial regions to obtain at least one first speech feature data for each facial region of the speech frame.

[0022] The step of generating the facial image corresponding to the speech frame based on multiple predicted values ​​includes:

[0023] A facial region image of the facial region in the speech frame is generated based on the predicted value of at least one first speech feature data of each facial region in the speech frame.

[0024] The facial image of the speech frame is generated based on multiple facial region images of the speech frame.

[0025] The method also includes:

[0026] The correlation data is a first identifier, which indicates the symmetry between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including:

[0027] Based on the first identifier of the second speech feature data, second speech feature data symmetrical to the second speech feature data is determined. Based on the label values ​​of the second speech feature data and the associated second speech feature data, the mean and offset of the label values ​​are determined. Based on the mean and offset, the label values ​​of the second speech feature data and the associated second speech feature data are corrected. The mean is the mean of the label values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the label value and the mean.

[0028] And / or,

[0029] The correlation data is a second identifier, which indicates the mutual exclusion between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including:

[0030] Based on the second identifier of the second speech feature data, a second speech feature data mutually exclusive with the second speech feature data is determined. Based on the label value of the second speech feature data and the associated second speech feature data, a valid value identifier is determined. Based on the valid value identifier, the label value of the second speech feature data and the associated second speech feature data is corrected. The valid value identifier indicates the validity of the label value of the second speech feature data.

[0031] And / or,

[0032] The correlation data is a third identifier, which indicates the positive or negative correlation between facial dimensions. The correction of the label value of the second voice feature data based on the correlation data of the second voice feature data includes:

[0033] Based on the third identifier of the second speech feature data, determine the second speech feature data that is positively or negatively correlated with the second speech feature data. Based on the label values ​​of the second speech feature data and the associated second speech feature data, determine the ratio. Based on the ratio, correct the label values ​​of the second speech feature data and the associated second speech feature data.

[0034] The speech sample contains multiple speech sample frames, and each speech sample frame contains multiple second speech feature data. The method further includes:

[0035] Traverse all speech sample frames of the speech sample, and determine the sub-loss value of the current speech sample frame based on the predicted value and label value of the current speech sample frame and the previous speech sample frame.

[0036] After the traversal is completed, the sub-loss values ​​of all speech sample frames are summed to obtain the corrected loss value of the speech sample.

[0037] The loss value of the speech sample is corrected based on the corrected loss value.

[0038] Another embodiment of this application provides a facial image generation apparatus, the apparatus comprising:

[0039] The acquisition module is used to acquire the speech to be tested and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information.

[0040] The deep learning module is used to input multiple first speech feature data into the speech recognition model and output the predicted values ​​of each facial dimension corresponding to the speech frame; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0041] The processing module is used to generate a facial image corresponding to the speech frame based on multiple predicted values. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of a facial image generation method according to an embodiment of this application is shown;

[0044] Figure 2 A flowchart of a facial image generation method according to another embodiment of this application is shown;

[0045] Figure 3 A flowchart of a facial image generation method according to another embodiment of this application is shown;

[0046] Figure 4 A flowchart of a facial image generation method according to another embodiment of this application is shown;

[0047] Figure 5 A schematic diagram of a facial image generation apparatus according to an embodiment of this application is shown. Detailed Implementation

[0048] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] To improve the accuracy of predicted values ​​from speech feature data, and thus make the facial expressions generated from the predicted values ​​more vivid, one embodiment of this application provides a facial image generation method, such as... Figure 1 As shown, the method includes:

[0050] Step 101: Obtain the speech to be tested, and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information.

[0051] For example, if 10 speech frames are extracted from a certain speech segment to be tested, and 64 facial dimensions are preset, then at least one first speech feature data corresponding to the 64 facial dimensions is extracted from each speech frame.

[0052] The first speech feature data includes facial dimension information corresponding to the facial dimension of the first speech feature data. For example, if the facial dimension corresponding to a certain first speech feature data is the left eye blinking, then the first speech feature data includes the facial dimension information of the left eye blinking.

[0053] Step 102: Input multiple first speech feature data into the speech recognition model and output the predicted values ​​of each facial dimension corresponding to the speech frame; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0054] In this embodiment, the speech recognition model used is the seq2seq (Sequence to Sequence) model. In other embodiments, any other speech recognition model may be used.

[0055] The predicted values ​​of the facial dimensions with correlations satisfy the constraints, and the correlations include symmetry, mutual exclusion, positive correlation, and negative correlation.

[0056] Step 103: Generate a facial image corresponding to the speech frame based on multiple predicted values.

[0057] Based on the predicted values ​​of multiple facial dimensions corresponding to the speech frame, a facial image corresponding to the speech frame is generated.

[0058] In the above scheme, multiple facial dimension-corresponding first speech feature data are extracted from each speech frame of the speech to be tested, and these first speech feature data are input into the speech recognition model to output predicted values ​​corresponding to multiple facial dimensions for each speech frame. Among the facial dimensions, different facial dimensions are usually correlated (e.g., left and right eye blinks are symmetrical, left and right pursed lips are mutually exclusive, and left and right eyebrow raising are positively correlated). Therefore, constraining the predicted values ​​corresponding to correlated facial dimensions can improve the accuracy of the predicted values, thereby making the facial expressions generated based on these predicted values ​​more vivid.

[0059] This application also provides a method for generating facial images, such as... Figure 2 As shown, the method includes:

[0060] Step 201: Obtain multiple speech samples. Each speech sample contains multiple second speech feature data and label values ​​and correlation data for each second speech feature data. The correlation data indicates the relationship between facial dimensions. The multiple second speech feature data correspond to multiple facial dimension information.

[0061] Each speech sample contains multiple second speech feature data and label values ​​and correlation data for each second speech feature data.

[0062] For example, a speech sample contains three second speech feature data points: {"Speech Feature Data": "Second Speech Feature Data A", "Label Value": 1, "Relationship Data": "Second Speech Feature Data B"}, {"Speech Feature Data": "Second Speech Feature Data B", "Label Value": 0, "Relationship Data": "Second Speech Feature Data A"}, and {"Speech Feature Data": "Second Speech Feature Data C", "Label Value": 1, "Relationship Data": "None"}. The relationship between second speech feature data A and second speech feature data B is "Second Speech Feature Data B", indicating a relationship between them. The relationship between second speech feature data C and second speech feature data B is "None", meaning there are no related second speech feature data points for second speech feature data C.

[0063] Correlation data can also be any other data that can characterize the correlation between speech feature data.

[0064] Each second speech feature data corresponds to a facial dimension information. For example, if facial dimension A and facial dimension B are set, then the speech sample contains second speech feature data A and second speech feature data B. Second speech feature data A corresponds to the facial dimension information of facial dimension A, and second speech feature data B corresponds to the facial dimension information of facial dimension B.

[0065] Step 202: Input all the second speech feature data of the speech sample into the initial speech recognition model and output the predicted value of each second speech feature data; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0066] In this embodiment, the initial speech recognition model uses the Conformer model plus an LSTM neural network. In other embodiments, any other speech recognition model may be used.

[0067] Step 203: Correct the label value of the second speech feature data based on the correlation data of the second speech feature data.

[0068] The label values ​​of the second speech feature data are corrected based on the correlation data. The correlation data can be identifiers, mapping relationship data, or any other data that can characterize the correlation.

[0069] Correcting the label values ​​of the second speech feature data based on correlation data can make the corrected label values ​​correlated.

[0070] Step 204: Calculate the loss value of the speech sample based on the predicted values ​​of all second speech feature data of the speech sample and the corrected label values.

[0071] The predicted values ​​of all second speech feature data and the corrected label values ​​are input into the loss function to calculate the loss value of the speech sample.

[0072] In this embodiment, the loss value L is calculated based on the following loss function:

[0073]

[0074] Where N is the total number of second speech feature data, y i Let i be the predicted value of the i-th second speech feature data. Let be the label value of the i-th second speech feature data.

[0075] In other implementations, the loss value can also be calculated based on any other loss function.

[0076] Step 205: Optimize the parameters of the initial speech recognition model based on the loss value to obtain the speech recognition model.

[0077] In the above scheme, the label values ​​of the second speech feature data are corrected based on correlation data, so that the corrected label values ​​are correlated. Then, a loss value is calculated based on the predicted value obtained after recognizing the second speech feature data by the initial speech recognition model and the corrected label values. The parameters of the initial speech recognition model are optimized based on the loss value, which enables the optimized speech recognition model to learn the ability to process the correlated second speech feature data, thereby improving the accuracy of the speech recognition model in recognizing the second speech feature data.

[0078] In one example of this application, the correlation includes symmetry, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0079] The predicted values ​​of symmetrical facial dimensions are constrained by a mean and an offset, wherein the mean is the average of the predicted values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the predicted value and the mean.

[0080] For example, the facial dimensions corresponding to predicted values ​​A and B are symmetrical. If the predicted values ​​A and B before constraint are 0.6 and 0.8 respectively, then the mean is 0.7 and the offset is 0.1. The predicted values ​​A and B are constrained based on this mean of 0.7 and the offset of 0.1.

[0081] For example, the facial dimensions corresponding to predicted values ​​C, D, and E exhibit symmetry. Before constraints, the predicted values ​​C, D, and E are 0.2, 0.3, and 0.4, respectively, resulting in a mean of 0.3. The offsets of the predicted values ​​C, D, and E from the mean are 0.1, 0, and 0.1, respectively. Since multiple offsets exist, in this embodiment, the maximum value among these offsets can be selected, resulting in a final offset of 0.1. In other embodiments, the maximum, minimum, average, or median value among multiple offsets can be selected, or other methods can be used to determine the final offset from multiple offsets. The predicted values ​​C, D, and E are constrained based on this mean of 0.3 and the offset of 0.1.

[0082] In this embodiment, when constraining the predicted value based on the mean and offset, the offset between the constrained predicted value and the mean needs to be controlled within 0.1.

[0083] For example, if the unconstrained predicted values ​​A and B are 0.3 and 0.7 respectively, with a mean of 0.5, then the offset is 0.2. After constraining predicted values ​​A and B based on the mean and offset, the constrained predicted value A becomes 0.4, and the constrained predicted value B becomes 0.6. Therefore, the mean of the constrained predicted values ​​A and B is now 0.5, and the offset becomes 0.1. The offset is now controlled within 0.1.

[0084] For example, the unconstrained predicted values ​​C, D, and E are 0.1, 0.4, and 0.4 respectively, with a mean of 0.3. Taking the maximum offset, the offset is 0.2. After constraining the predicted values ​​C, D, and E based on the mean and offset, the constrained predicted value C is 0.2, and the constrained predicted values ​​D and E are 0.35. Therefore, the mean of the constrained predicted values ​​C, D, and E is 0.3, while the maximum offset is 0.1, keeping the offset within 0.1.

[0085] Of course, in other implementations, a preset offset can be set based on specific needs. After constraining the predicted value based on the mean and the offset, the constrained offset is less than or equal to the preset offset.

[0086] Symmetrical facial dimensions, such as the left and right eye blinking, are typically reflected in facial expressions as the left and right eyes blink simultaneously. Therefore, the degree of closure between the left and right eyes during blinking is not significantly different. Existing methods do not consider the symmetry between facial dimensions, resulting in large discrepancies between predicted values ​​for symmetrical facial dimensions (i.e., the deviation of predicted values ​​from the mean of predicted values), leading to less expressive facial images generated based on these predictions.

[0087] In the above scheme, the mean of all predicted values ​​is determined, then the offset between the predicted value and the mean is determined, and finally the predicted values ​​are constrained based on the mean and the offset. This allows for a symmetrical relationship between the constrained predicted values, and the constrained offset is controlled within a preset offset, ensuring that these predicted values ​​are similar but not completely identical. This makes the predicted values ​​more accurate, and the facial images generated based on these predicted values ​​have more vivid expressions.

[0088] In one example of this application, for facial dimensions with symmetry, the speech recognition model can be changed from the predicted value after prediction constraints to the mean and offset value after prediction constraints.

[0089] For example, the constrained predicted value A is 0.4, the constrained predicted value B is 0.6, the mean of the constrained predicted values ​​A and B is 0.5, and the offset is 0.1. Originally, the speech recognition model predicted the value of the speech feature data after recognizing it. After modification, the speech recognition model predicts the mean and offset of the speech feature data after recognizing it. The original speech feature model output predicted value A as 0.4 and predicted value B as 0.6. After modification, it outputs predicted value A as 0.5 and predicted value B as 0.1. Of course, this embodiment does not restrict the output order; that is, after modification, it can also output predicted value A as 0.1 and predicted value B as 0.5. Specific settings can be configured based on requirements.

[0090] For example, the constrained predicted value C is 0.2, and the constrained predicted values ​​D and E are 0.35. The mean of the constrained predicted values ​​C, D, and E is 0.3, and the offset is 0.1. The original speech feature model output predicted value C as 0.2 and predicted values ​​D and E as 0.35. After modification, it outputs predicted value C as 0.5 and predicted values ​​D and E as 0.1. Similarly, this embodiment does not restrict the order of output, meaning that after modification, it can also output predicted value C as 0.1 and predicted values ​​D and E as 0.5. Specific settings can be configured based on requirements.

[0091] In the above method, changing the speech recognition model from predicted values ​​after prediction constraints to the mean and offset values ​​after prediction constraints allows the speech recognition model to more effectively control the offset after constraints within a preset offset when constraining the predicted values ​​based on the mean and offset, thereby further improving the accuracy of the output predicted values.

[0092] In one example of this application, the correlation includes mutual exclusivity, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0093] The predicted values ​​of mutually exclusive facial dimensions are constrained by their magnitude, and at most one of the predicted values ​​of the multiple mutually exclusive facial dimensions is not zero.

[0094] In this embodiment, when constraining the predicted value by its magnitude, only the largest predicted value is retained, and the other predicted values ​​are changed to zero.

[0095] For example, the facial dimensions corresponding to predicted values ​​F and G are mutually exclusive. Before constraints, predicted values ​​F and G are 0.6 and 0.8, respectively. Since two predicted values ​​are non-zero, constraints are applied to F and G based on their magnitudes, resulting in a constrained predicted value F of 0 and a constrained predicted value G of 0.8. Alternatively, the non-zero predicted value can be directly set to 1, meaning the constrained predicted value F is 0 and the constrained predicted value G is 1. The specific settings can be configured based on requirements.

[0096] For example, the facial dimensions corresponding to predicted values ​​H, J, and K are mutually exclusive. Before constraints, the predicted values ​​H, J, and K are 0.4, 0.5, and 0, respectively. Since two predicted values ​​are not zero, constraints are applied to H, J, and K based on their magnitudes, resulting in predicted values ​​H and K being 0, and predicted value J being 0.5. Similarly, the non-zero predicted value can be directly set to 1, meaning the constrained predicted value J is 1.

[0097] Mutually exclusive facial dimensions, such as pouting to the left and pouting to the right, cannot be simultaneously expressed in facial expressions. Therefore, when pouting, one can only pout to the left or only pout to the right. Existing methods do not consider the mutual exclusivity between facial dimensions, resulting in multiple non-zero values ​​in the predicted values ​​of mutually exclusive facial dimensions, and consequently, facial images generated based on these predicted values ​​lack vividness.

[0098] In the above scheme, the predicted values ​​are constrained by their magnitude. This ensures that at most one of the constrained predicted values ​​is not zero, guaranteeing the mutual exclusion relationship between these predicted values. Consequently, these predicted values ​​are more accurate, and the facial images generated based on these predicted values ​​have more vivid expressions.

[0099] In one example of this application, for the mutually exclusive facial dimension, the speech recognition model can be changed from the predicted value after prediction constraints to the valid value identifier after prediction constraints, and the valid value identifier indicates the validity of the predicted value of the first speech feature data.

[0100] For example, the constrained predicted value F is 0, the constrained predicted value G is 0.8, and the valid value identifier is 0 / 1, where the 0 on the left indicates that the predicted value F is zero, and the 1 on the right indicates that the predicted value G is not zero. Originally, the speech recognition model predicted the predicted value of the speech feature data after recognizing it. After modification, the speech recognition model predicts the valid value identifier of the speech feature data after recognizing it. The original speech feature model output predicted value F as 0 and predicted value G as 0.8. After modification, it outputs predicted value F as 0 / 1 and predicted value G as 0.8. Of course, this embodiment does not restrict the output order; that is, after modification, it can also output predicted value F as 0.8 and predicted value G as 0 / 1. Specific settings can be configured based on requirements.

[0101] For example, the constrained predicted values ​​H, J, and K are all 0, with valid values ​​indicated by 0 / 1 / 0. Here, a 0 on the left indicates that predicted value H is zero, a 1 in the middle indicates that predicted value J is not zero, and a 0 on the right indicates that predicted value K is zero. The original speech feature model output predicted values ​​H, J, and K as 0.5. After modification, it outputs predicted values ​​H as 0 / 1 / 0 and predicted values ​​J and K as 0.5. Similarly, this embodiment does not restrict the output order; that is, after modification, it can also output predicted values ​​H and J as 0.5 and predicted value K as 0 / 1 / 0. Specific settings can be configured based on requirements.

[0102] In the above method, changing the predicted value after prediction constraints to the effective value identifier after prediction constraints in the speech recognition model can ensure that when the speech recognition model constrains the predicted value based on the effective value identifier, it will necessarily retain only one of the multiple predicted values ​​that is not zero, thereby further improving the accuracy of the output predicted value.

[0103] In one example of this application, the correlation includes positive or negative correlation, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including:

[0104] The predicted values ​​of facial dimensions with positive or negative correlation are constrained by numerical comparison, and the values ​​of multiple predicted values ​​of facial dimensions with positive or negative correlation satisfy a specified proportional relationship.

[0105] In this embodiment, when constraining predicted values ​​through numerical comparison, for predicted values ​​of positively correlated facial dimensions, the smaller predicted value is constrained based on the larger predicted value. The ratio of the constrained smaller predicted value to the larger predicted value should be as close to 100% as possible; for example, the ratio should be greater than or equal to 50% and less than or equal to 100%. For predicted values ​​of negatively correlated facial dimensions, the larger predicted value is constrained based on the smaller predicted value. The ratio of the constrained larger predicted value to the smaller predicted value should be larger; for example, the ratio should be greater than or equal to 150%. In other embodiments, the specified ratio relationship can be set based on specific needs.

[0106] For example, there is a positive correlation between the facial dimensions corresponding to the predicted values ​​L and M. Before constraints, the predicted values ​​L and M were 0.2 and 0.8, respectively. By constraining the predicted values ​​L and M through numerical comparison, the constrained predicted value L is 0.4 and the predicted value M is 0.8. The constrained predicted value L is 50% of the predicted value M, satisfying the specified proportional relationship.

[0107] For example, the facial dimensions corresponding to predicted values ​​N, O, and P exhibit a negative correlation. Before constraints, the predicted values ​​N, O, and P were 0.2, 0.25, and 0.3, respectively. By constraining the predicted values ​​N, O, and P through numerical comparison, the constrained predicted values ​​are 0.2 for N, 0.4 for O, and 0.6 for P. The constrained predicted value O is 200% of the predicted value N, and the predicted value P is 300% of the predicted value N, both satisfying the specified proportional relationship.

[0108] Facial dimensions with positive or negative correlations, such as raising the left and right eyebrows, will affect facial expressions. Even if a person only intends to raise the left eyebrow during an expression, raising the left eyebrow will still cause the right eyebrow to raise slightly as well. Therefore, whether only the left or right eyebrow is raised, it will affect the other eyebrow. Existing methods do not consider the positive or negative correlations between facial dimensions, resulting in the predictions of facial dimensions with positive or negative correlations not exhibiting these correlations, and consequently, the facial expressions generated based on these predictions are not vivid enough.

[0109] In the above scheme, the predicted values ​​are constrained through numerical comparison. This ensures that multiple constrained predicted values ​​meet a specified proportional relationship, thereby making these predicted values ​​more accurate and the facial expressions generated based on these predicted values ​​more vivid.

[0110] In one example of this application, for facial dimensions that have positive or negative correlation, the speech recognition model can be changed from the predicted value after prediction constraints to the predicted proportion after prediction constraints.

[0111] For example, the constrained predicted value L is 0.4, the constrained predicted value M is 0.8, and the proportion is 50%. Originally, the speech recognition model predicted the predicted value of each speech feature data after recognizing it. After modification, the speech recognition model predicts the proportion of each speech feature data after recognizing it. The original speech feature model output predicted value L of 0.4 and predicted value M of 0.8. After modification, it outputs predicted value L of 50% and predicted value M of 0.8.

[0112] For example, after constraints, the predicted value N is 0.2, the predicted value O is 0.4, and the predicted value P is 0.6, with proportions of 200% and 300%. The original speech feature model output predicted value N of 0.2, predicted value O of 0.4, and predicted value P of 0.6. After modification, it outputs predicted value N of 0.2, predicted value O of 200%, and predicted value P of 300%.

[0113] In the above method, changing the speech recognition model from predicting the predicted value after the prediction constraint to predicting the proportion after the prediction constraint can improve the accuracy of the predicted proportion when the speech recognition model constrains the predicted value based on the proportion, thereby further improving the accuracy of the output predicted value.

[0114] This application also provides a method for generating a facial image, which further includes:

[0115] The multiple first speech feature data of the speech frame are classified according to multiple facial regions to obtain at least one first speech feature data for each facial region of the speech frame.

[0116] For example, there are first speech feature data A, first speech feature data B, first speech feature data C, and first speech feature data D. The facial dimension corresponding to first speech feature data A is left eye blinking, the facial dimension corresponding to first speech feature data B is right eye blinking, the facial dimension corresponding to first speech feature data C is left-turned-lip pouting, and the facial dimension corresponding to first speech feature data D is right-turned-lip pouting. These first speech feature data are classified according to the eye region and the mouth region. First speech feature data A and first speech feature data B are classified as first speech feature data of the eye region. First speech feature data C and first speech feature data D are classified as first speech feature data of the mouth region.

[0117] In one example of this application, such as Figure 3 As shown, generating the facial image corresponding to the speech frame based on multiple predicted values ​​includes:

[0118] Step 301: Generate a facial region image of the facial region of the speech frame based on the predicted value of at least one first speech feature data of each facial region of the speech frame.

[0119] For example, the eye region has first speech feature data A and first speech feature data B, and the mouth region has first speech feature data C and first speech feature data D. Then, an image of the eye region is generated based on the predicted values ​​corresponding to first speech feature data A and first speech feature data B, and an image of the mouth region is generated based on the predicted values ​​corresponding to first speech feature data C and first speech feature data D.

[0120] Step 302: Generate a facial image of the voice frame based on multiple facial region images of the voice frame.

[0121] In the above scheme, the facial image generation is changed from overall generation to generation based on facial regions by classifying the first speech feature data according to facial regions. Since only the facial dimensional features of a single facial region need to be considered when generating a single facial region image, the generated facial region image is more accurate, and the facial expression generated based on multiple facial region images is more vivid.

[0122] This application also provides a method for generating facial images, such as... Figure 4 As shown, the method also includes:

[0123] The correlation data is a first identifier, which indicates the symmetry between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including:

[0124] Based on the first identifier of the second speech feature data, a second speech feature data symmetrical to the second speech feature data is determined. Based on the label values ​​of the second speech feature data and the associated second speech feature data, the mean and offset of the label values ​​are determined. Based on the mean and offset, the label values ​​of the second speech feature data and the associated second speech feature data are corrected. The mean is the mean of the label values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the label value and the mean.

[0125] For example, the first identifier indicates that there is symmetry between the facial dimensions corresponding to the second voice feature data A and the second voice feature data B. The second voice feature data A has a corresponding label value A of 0.4, and the second voice feature data B has a corresponding label value B of 0.6. The mean of label values ​​A and B is determined to be 0.5, with an offset of 0.1. Therefore, label value A is corrected to 0.5, and label value B is corrected to 0.1. Of course, this embodiment does not limit the order of correction; label value A can also be corrected to 0.1, and label value B to 0.5.

[0126] For example, the second speech feature data C, the second speech feature data D, and the second speech feature data E exhibit symmetry. The second speech feature data C has a corresponding label value C = 0.2, the second speech feature data D has a corresponding label value D = 0.4, and the second speech feature data E has a corresponding label value E = 0.3. The mean of the label values ​​C, D, and E is determined to be 0.3. The offsets of the predicted values ​​C, D, and E from the mean are 0.1, 0.1, and 0, respectively. Since there are multiple offsets, in this embodiment, the maximum value among the multiple offsets can be selected, i.e., the final determined offset is 0.1. In other embodiments, when determining the offset, the maximum, minimum, average, or median value among multiple offsets can be selected, or other methods can be used to determine the final offset from multiple offsets. The label value C is corrected to 0.3, and the label values ​​D and E are corrected to 0.1. Similarly, the order of correction is not restricted; the label value C can also be corrected to 0.1, and the label values ​​D and E can be corrected to 0.5.

[0127] By determining the mean and offset of the label values ​​based on the second speech feature data and its associated label values, and then correcting the label values ​​corresponding to the symmetrical second speech feature data based on the mean and offset, the speech recognition model trained using these second speech feature data and their corresponding label values ​​can learn to predict the mean and offset of the symmetrical speech feature data. This improves the accuracy of the predicted mean and offset, resulting in more vivid facial expressions in the facial images generated based on the predicted values.

[0128] And / or,

[0129] The correlation data is a second identifier, which indicates the mutual exclusion between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including:

[0130] Based on the second identifier of the second speech feature data, a second speech feature data mutually exclusive with the second speech feature data is determined. Based on the label value of the second speech feature data and the associated second speech feature data, a valid value identifier is determined. Based on the valid value identifier, the label value of the second speech feature data and the associated second speech feature data is corrected. The valid value identifier indicates the validity of the label value of the second speech feature data.

[0131] For example, the second identifier indicates that there is symmetry between the facial dimensions corresponding to the second speech feature data F and the second speech feature data G. The second speech feature data F has a corresponding label value F of 0, and the second speech feature data G has a corresponding label value G of 1. Based on the label values ​​F and G, the valid value identifier is determined to be 0 / 1, where 0 on the left indicates that the label value F is zero, and 1 on the right indicates that the label value G is not zero. The predicted value F is corrected to 0 / 1, and the predicted value G is corrected to 1. The corrected value of the predicted value G is a non-zero predicted value (i.e., the original predicted value G). Similarly, this embodiment does not restrict the order of correction; the label value F can also be corrected to 1, and the label value G can be corrected to 0 / 1.

[0132] For example, the second speech feature data H, the second speech feature data J, and the second speech feature data K are mutually exclusive. The second speech feature data H has a corresponding label value H=0, the second speech feature data J has a corresponding label value J=1, and the second speech feature data K has a corresponding label value K=0. Based on the label values ​​H, J, and K, the valid value identifiers are determined as 0 / 1 / 0, where the left 0 indicates that the label value H is zero, the middle 1 indicates that the label value J is not zero, and the right 0 indicates that the label value K is zero. The predicted value H is corrected to 0 / 1 / 0, and the predicted values ​​J and K are corrected to 1. The corrected values ​​of the predicted values ​​J and K are non-zero predicted values ​​(i.e., the original predicted value J). Similarly, this embodiment does not restrict the order of correction; the label values ​​H and J can also be corrected to 1, and the label value K can be corrected to 0 / 1 / 0.

[0133] By determining valid value identifiers based on the second speech feature data and its associated label values, and then correcting the label values ​​corresponding to mutually exclusive second speech feature data based on these valid value identifiers, a speech recognition model trained using these second speech feature data and their corresponding label values ​​can learn to predict the valid value identifiers of mutually exclusive speech feature data. This improves the accuracy of the predicted valid value identifiers, resulting in more vivid facial expressions in the facial images generated based on the predicted values.

[0134] And / or,

[0135] The correlation data is a third identifier, which indicates the positive or negative correlation between facial dimensions. The correction of the label value of the second voice feature data based on the correlation data of the second voice feature data includes:

[0136] Based on the third identifier of the second speech feature data, determine the second speech feature data that is positively or negatively correlated with the second speech feature data. Based on the label values ​​of the second speech feature data and the associated second speech feature data, determine the ratio. Based on the ratio, correct the label values ​​of the second speech feature data and the associated second speech feature data.

[0137] For example, the third identifier indicates a positive correlation between the facial dimensions corresponding to the second speech feature data L and the second speech feature data M. The second speech feature data L has a corresponding label value L of 0.4, and the second speech feature data M has a corresponding label value M of 0.6. Based on the label values ​​L and M, the proportion is determined to be 80%. Therefore, based on this proportion, the label value L is adjusted to 0.48.

[0138] For example, there is a negative correlation between second speech feature data N, second speech feature data O, and second speech feature data P. Second speech feature data N has a corresponding label value N = 0.2, second speech feature data O has a corresponding label value O = 0.25, and second speech feature data P has a corresponding label value P = 0.3. Based on label values ​​N and P, a ratio of 200% is determined, and based on label values ​​O and P, a ratio of 300% is determined. Therefore, based on these ratios, label value N is adjusted to 0.4, and label value O is adjusted to 0.6.

[0139] By determining the ratio based on the second speech feature data and the label values ​​of the associated second speech feature data, and then correcting the label values ​​of the second speech feature data that have positive or negative correlation based on the ratio, the accuracy of the predicted values ​​predicted by the speech recognition model trained using these second speech feature data and the corresponding label values ​​can be improved, making the facial expressions generated based on the predicted values ​​more vivid.

[0140] This application also provides a method for generating facial images, such as... Figure 4 As shown, the speech sample contains multiple speech sample frames, and each speech sample frame contains multiple second speech feature data. The method further includes:

[0141] Step 401: Traverse all speech sample frames of the speech sample and determine the sub-loss value of the current speech sample frame based on the prediction value and label value of the current speech sample frame and the previous speech sample frame.

[0142] In this embodiment, the sub-loss value L is specifically calculated based on the following formula.S :

[0143]

[0144] Where N is the total number of second speech feature data, y it y is the predicted value of the i-th second speech feature data in the current speech sample frame. it-1 This is the predicted value of the i-th second speech feature data in the previous speech sample frame. This represents the label value of the i-th second speech feature data in the current speech sample frame. It is the label value of the i-th second speech feature data in the previous speech sample frame.

[0145] Step 402: After the traversal is completed, the sub-loss values ​​of all speech sample frames are summed to obtain the corrected loss value of the speech sample.

[0146] Step 403: Correct the loss value of the speech sample based on the corrected loss value.

[0147] In this embodiment, the corrected loss value L is calculated based on the following formula. ′ :

[0148] L ′ =+L S

[0149] In other implementations, other methods may be used to correct the loss value based on the corrected loss value.

[0150] Existing methods do not consider the relationship between adjacent speech sample frames when calculating the loss value, which leads to a large difference in the predicted value of the same facial dimension in two adjacent frames. As a result, the facial images generated based on these predicted values ​​will also have large differences in the same facial dimension, which in turn leads to jitter in the facial animation generated from multiple facial images.

[0151] In the aforementioned scheme, a sub-loss value for the current speech sample frame is determined based on the predicted and labeled values ​​of the current and previous speech sample frames. A corrected loss value is then determined based on this sub-loss value, and finally, the loss value of the speech sample is corrected based on the corrected loss value. Training the speech recognition model with the corrected loss value significantly reduces the difference between predicted values ​​for the same facial dimension in adjacent speech frames output after recognizing speech feature data. This, in turn, significantly reduces the differences between facial images generated based on these predicted values, and significantly reduces shakiness in facial animations generated from these facial images.

[0152] To implement the above-mentioned facial image generation method, such as Figure 5As shown, an example of this application provides a facial image generation apparatus, including:

[0153] Acquisition module 10 is used to acquire the speech to be tested and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information;

[0154] The deep learning module 20 is used to input multiple first speech feature data into the speech recognition model and output the predicted values ​​of each facial dimension corresponding to the speech frame; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0155] Processing module 30 is used to generate a facial image corresponding to the speech frame based on multiple predicted values.

[0156] The acquisition module 10 is further configured to acquire multiple speech samples, each speech sample containing multiple second speech feature data and label values ​​and correlation data for each second speech feature data, wherein the correlation data indicates the correlation between facial dimensions; wherein the multiple second speech feature data correspond to multiple facial dimension information.

[0157] The deep learning module 20 is further configured to input all the second speech feature data of the speech sample into the initial speech recognition model and output the predicted value of each second speech feature data; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions.

[0158] The processing module 30 is further configured to correct the label value of the second speech feature data based on the correlation data of the second speech feature data;

[0159] The processing module 30 is also used to calculate the loss value of the speech sample based on the predicted values ​​of all second speech feature data of the speech sample and the corrected label values;

[0160] The deep learning module 20 is further used to optimize the parameters of the initial speech recognition model based on the loss value to obtain a speech recognition model.

[0161] The deep learning module 20 is further configured to constrain the predicted values ​​of the symmetrical facial dimensions through a mean and an offset, wherein the mean is the average of the predicted values ​​of the multiple symmetrical facial dimensions, and the offset is the offset between the predicted value and the mean.

[0162] The deep learning module 20 is further used to constrain the predicted values ​​of mutually exclusive facial dimensions by their magnitude, wherein at most one of the predicted values ​​of the multiple mutually exclusive facial dimensions is not zero.

[0163] The deep learning module 20 is further used to constrain the predicted values ​​of facial dimensions with positive or negative correlation through numerical comparison, and the values ​​of multiple predicted values ​​of facial dimensions with positive or negative correlation satisfy a specified proportional relationship.

[0164] The processing module 30 is further configured to classify the multiple first speech feature data of the speech frame according to multiple facial regions, so as to obtain at least one first speech feature data of each facial region of the speech frame.

[0165] The processing module 30 is further configured to generate a facial region image of the facial region of the speech frame based on the predicted value of at least one first speech feature data of each facial region of the speech frame.

[0166] The processing module 30 is further configured to generate a facial image of the voice frame based on multiple facial region images of the voice frame.

[0167] The processing module 30 is further configured to use the correlation data as a first identifier, the first identifier indicating the symmetry between facial dimensions, and to correct the label value of the second voice feature data based on the correlation data of the second voice feature data, including:

[0168] The processing module 30 is further configured to determine second speech feature data symmetrical to the second speech feature data based on the first identifier of the second speech feature data, determine the mean and offset of the label values ​​based on the label values ​​of the second speech feature data and the associated second speech feature data, and correct the label values ​​of the second speech feature data and the associated second speech feature data based on the mean and offset, wherein the mean is the mean of the label values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the label value and the mean;

[0169] And / or,

[0170] The processing module 30 is further configured to use the correlation data as a second identifier, the second identifier indicating the mutual exclusion between facial dimensions, and to correct the label value of the second voice feature data based on the correlation data of the second voice feature data, including:

[0171] The processing module 30 is further configured to determine second speech feature data mutually exclusive with the second speech feature data based on the second identifier of the second speech feature data, determine a valid value identifier based on the label value of the second speech feature data and the associated second speech feature data, and correct the label value of the second speech feature data and the associated second speech feature data based on the valid value identifier, wherein the valid value identifier indicates the validity of the label value of the second speech feature data.

[0172] And / or,

[0173] The processing module 30 is further configured to use a third identifier for the correlation data, the third identifier indicating a positive or negative correlation between facial dimensions, and to correct the label value of the second voice feature data based on the correlation data of the second voice feature data, including:

[0174] The processing module 30 is further configured to determine, based on the third identifier of the second speech feature data, second speech feature data that is positively or negatively correlated with the second speech feature data, determine a ratio based on the label values ​​of the second speech feature data and the associated second speech feature data, and correct the label values ​​of the second speech feature data and the associated second speech feature data based on the ratio.

[0175] The processing module 30 is further configured to traverse all speech sample frames of the speech sample and determine the sub-loss value of the current speech sample frame based on the prediction value and label value of the current speech sample frame and the previous speech sample frame.

[0176] The processing module 30 is also used to sum the sub-loss values ​​of all speech sample frames after the traversal is completed, so as to obtain the corrected loss value of the speech sample.

[0177] The processing module 30 is further configured to correct the loss value of the speech sample based on the corrected loss value.

[0178] In the above scheme, multiple facial dimension-corresponding first speech feature data are extracted from each speech frame of the speech to be tested, and these first speech feature data are input into the speech recognition model to output the predicted values ​​corresponding to multiple facial dimensions for each speech frame. Among the facial dimensions, different facial dimensions are usually correlated (e.g., left and right eye blinks are symmetrical, left and right pursed lips are mutually exclusive, and left and right eyebrow raising are positively correlated). Therefore, constraining the predicted values ​​corresponding to correlated facial dimensions can improve the accuracy of the predicted values, thereby making the facial expressions generated based on these predicted values ​​more vivid.

[0179] In one example, this application embodiment also provides a mobile terminal, which includes at least one memory and a processor communicatively connected to the at least one memory; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the above-mentioned... Figures 1 to 4 The facial image generation method described in any one of the embodiments.

[0180] In addition, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which are used to perform the above-described... Figures 1 to 4 The facial image generation method flow described in any one of the embodiments.

[0181] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0182] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0183] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0184] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0185] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0187] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating a facial image, the method comprising: Obtain the speech to be tested, and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information; Multiple first speech feature data are input into a speech recognition model, and the predicted values ​​of each facial dimension corresponding to the speech frame are output; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions. A facial image corresponding to the speech frame is generated based on multiple predicted values.

2. The facial image generation method according to claim 1, further comprising: Multiple speech samples are acquired, each speech sample containing multiple second speech feature data and label values ​​and correlation data for each second speech feature data, wherein the correlation data indicates the correlation between facial dimensions; wherein, the multiple second speech feature data correspond to multiple facial dimension information; Input all the second speech feature data of the speech sample into the initial speech recognition model, and output the predicted value of each second speech feature data; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions; The label values ​​of the second speech feature data are corrected based on the correlation data of the second speech feature data. The loss value of the speech sample is calculated based on the predicted values ​​of all second speech feature data and the corrected label values ​​of the speech sample. The parameters of the initial speech recognition model are optimized based on the loss value to obtain a speech recognition model.

3. The facial image generation method according to claim 1 or 2, wherein the correlation includes symmetry, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including: The predicted values ​​of symmetrical facial dimensions are constrained by a mean and an offset, wherein the mean is the average of the predicted values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the predicted value and the mean.

4. The facial image generation method according to claim 1 or 2, wherein the correlation includes mutual exclusion, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including: The predicted values ​​of mutually exclusive facial dimensions are constrained by their magnitude, and at most one of the predicted values ​​of the multiple mutually exclusive facial dimensions is not zero.

5. The facial image generation method according to claim 1 or 2, wherein the correlation includes positive or negative correlation, and the predicted values ​​of the correlated facial dimensions satisfy constraints, including: The predicted values ​​of facial dimensions with positive or negative correlation are constrained by numerical comparison, and the values ​​of multiple predicted values ​​of facial dimensions with positive or negative correlation satisfy a specified proportional relationship.

6. The facial image generation method according to claim 1, further comprising: The multiple first speech feature data of the speech frame are classified according to multiple facial regions to obtain at least one first speech feature data for each facial region of the speech frame.

7. The facial image generation method according to claim 6, wherein generating the facial image corresponding to the speech frame based on a plurality of the predicted values ​​comprises: A facial region image of the facial region in the speech frame is generated based on the predicted value of at least one first speech feature data of each facial region in the speech frame. The facial image of the speech frame is generated based on multiple facial region images of the speech frame.

8. The facial image generation method according to claim 2, further comprising: The correlation data is a first identifier, which indicates the symmetry between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including: Based on the first identifier of the second speech feature data, second speech feature data symmetrical to the second speech feature data is determined. Based on the label values ​​of the second speech feature data and the associated second speech feature data, the mean and offset of the label values ​​are determined. Based on the mean and offset, the label values ​​of the second speech feature data and the associated second speech feature data are corrected. The mean is the mean of the label values ​​of multiple symmetrical facial dimensions, and the offset is the offset between the label value and the mean. And / or, The correlation data is a second identifier, which indicates the mutual exclusion between facial dimensions. The correlation data based on the second voice feature data corrects the label value of the second voice feature data, including: Based on the second identifier of the second speech feature data, a second speech feature data mutually exclusive with the second speech feature data is determined. Based on the label value of the second speech feature data and the associated second speech feature data, a valid value identifier is determined. Based on the valid value identifier, the label value of the second speech feature data and the associated second speech feature data is corrected. The valid value identifier indicates the validity of the label value of the second speech feature data. And / or, The correlation data is a third identifier, which indicates the positive or negative correlation between facial dimensions. The correction of the label value of the second voice feature data based on the correlation data of the second voice feature data includes: Based on the third identifier of the second speech feature data, determine the second speech feature data that is positively or negatively correlated with the second speech feature data. Based on the label values ​​of the second speech feature data and the associated second speech feature data, determine the ratio. Based on the ratio, correct the label values ​​of the second speech feature data and the associated second speech feature data.

9. The facial image generation method according to claim 2, wherein the speech sample includes multiple speech sample frames, each speech sample frame includes multiple second speech feature data, and the method further includes: Traverse all speech sample frames of the speech sample, and determine the sub-loss value of the current speech sample frame based on the predicted value and label value of the current speech sample frame and the previous speech sample frame. After the traversal is completed, the sub-loss values ​​of all speech sample frames are summed to obtain the corrected loss value of the speech sample. The loss value of the speech sample is corrected based on the corrected loss value.

10. A facial image generation apparatus, the apparatus comprising: The acquisition module is used to acquire the speech to be tested and extract at least one first speech feature data corresponding to multiple speech frames in the speech to be tested; wherein, the multiple first speech feature data correspond to multiple facial dimension information; The deep learning module is used to input multiple first speech feature data into the speech recognition model and output the predicted values ​​of each facial dimension corresponding to the speech frame; wherein the predicted values ​​of the related facial dimensions satisfy the constraint conditions. The processing module is used to generate a facial image corresponding to the speech frame based on multiple predicted values.

Images