Speaking face image generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of financial science and technology, and discloses a speaking face image generation method, device and equipment and a storage medium, and the method comprises the steps: obtaining a real face key point feature, a real head posture feature, a real sight direction vector and a real emotion category label in a real speaking face image; determining a first loss value based on the predicted face key point feature and the real face key point feature, determining a second loss value based on the predicted head posture feature and the real head posture feature, determining a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determining a third loss value based on the predicted emotion category tag and the real emotion category tag; determining a fourth loss value; determining a trained generation model; and obtaining a current speaking face image generated by the trained generation model based on the current voice data. According to the invention, the image quality of the generated current speaking face image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology. In particular, the present invention relates to a method, device, equipment and storage medium for generating a speaking face image. Background Art

[0002] The talking face generation method can generate realistic talking face images, add more visual elements to digital content, make digital content more vivid and interesting, enhance the attractiveness and appeal of digital content, and bring users a richer and more immersive interactive experience. It is an important technology in the current field of digital content creation and interaction.

[0003] However, the features learned by the existing speaking face generation methods are not comprehensive enough, which is not conducive to improving the image quality of the generated current speaking face image. The reason is that the existing speaking face generation methods mainly focus on the synchronization of lip movements and input audio, ignoring facial expressions, sight lines, head postures, and emotions. Therefore, the learned features are not comprehensive enough, resulting in the generated current speaking face image being unnatural, lacking in realism and vividness, and therefore not conducive to improving the image quality of the generated current speaking face image. Summary of the invention

[0004] The present invention provides a method, device, computer equipment and storage medium for generating a speaking face image, so as to solve the technical problem that the features learned by the existing speaking face generation method are not comprehensive enough, which is not conducive to improving the image quality of the generated current speaking face image.

[0005] In a first aspect, a method for generating a speaking face image is provided, comprising:

[0006] Acquire preset voice data and a real speaking face image corresponding to the preset voice data;

[0007] Based on a preset generation model, the preset voice data and a predefined acquisition method, obtaining a predicted speaking face image;

[0008] Acquire predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and acquire real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image;

[0009] Determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label;

[0010] Determine the generation model that has been trained based on the first loss value, the second loss value, the third loss value, the fourth loss value, and a predefined determination method;

[0011] Acquire a current speaking person's face image generated by the trained generation model based on current speech data.

[0012] Furthermore, the obtaining of the predicted speaking face image based on the preset generation model, the preset voice data and the predefined acquisition method includes:

[0013] Extracting features from the preset voice data using a voice encoder to obtain a first voice feature;

[0014] The first speech feature is input into the generative model to obtain a predicted speaking face image generated by the generative model based on the first speech feature.

[0015] Further, the step of obtaining predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and obtaining real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image includes:

[0016] Obtaining a first extraction instruction, executing the first extraction instruction, performing feature extraction on the predicted speaking face image, and obtaining predicted facial key point features, predicted head posture features, predicted sight direction vectors, and predicted emotion category labels in the predicted speaking face image;

[0017] A second extraction instruction is obtained, and the second extraction instruction is executed to perform feature extraction on the predicted speaking face image to obtain real facial key point features, real head posture features, real sight direction vectors, and real emotion category labels in the real speaking face image.

[0018] Further, the first loss value is determined based on the predicted facial key point features and the real facial key point features, the second loss value is determined based on the predicted head posture features and the real head posture features, the third loss value is determined based on the predicted sight line direction vector and the real sight line direction vector, and the fourth loss value is determined based on the predicted emotion category label and the real emotion category label, including:

[0019] Calculate the loss value between the predicted facial key point feature and the real facial key point feature through a loss function, and select the loss value between the predicted facial key point feature and the real facial key point feature as a first loss value;

[0020] Calculate the loss value between the predicted head posture feature and the real head posture feature by using the loss function, and select the loss value between the predicted head posture feature and the real head posture feature as the second loss value;

[0021] Calculating a loss value between the predicted sight line direction vector and the actual sight line direction vector by using the loss function, and selecting the loss value between the predicted sight line direction vector and the actual sight line direction vector as a third loss value;

[0022] The loss value between the predicted emotion category label and the true emotion category label is calculated by the loss function, and the loss value between the predicted emotion category label and the true emotion category label is selected as the fourth loss value.

[0023] Further, determining the trained generation model based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method includes:

[0024] Adding the first loss value, the second loss value, the third loss value, and the fourth loss value to generate the overall loss value, and training the generation model based on the overall loss value;

[0025] When the overall loss value is less than a preset loss value, the training of the generation model is stopped, and the generation model after training is obtained.

[0026] Furthermore, the step of obtaining the current speaker's face image generated by the trained generative model based on the current voice data includes:

[0027] Acquire current voice data, and perform feature extraction on the current voice data through a voice encoder to obtain a second voice feature;

[0028] The second speech feature is input into the trained generation model, and a face image of a current speaker generated by the trained generation model based on the second speech feature is obtained.

[0029] Furthermore, after acquiring the current speaker face image generated by the trained generation model based on the current voice data, the speaker face image generation method comprises:

[0030] A preset synthesis instruction is obtained, and the current speaking face image and the current voice data are synthesized by executing the synthesis instruction to obtain a speaking face video.

[0031] In a second aspect, a device for generating a speaking face image is provided, comprising:

[0032] A first acquisition module, used to acquire preset voice data and a real speaking face image corresponding to the preset voice data;

[0033] A second acquisition module, used to acquire a predicted speaking face image based on a preset generation model, the preset voice data and a predefined acquisition method;

[0034] A third acquisition module is used to acquire predicted facial key point features, predicted head posture features, predicted sight line direction vectors and predicted emotion category labels in the predicted speaker face image, and to acquire real facial key point features, real head posture features, real sight line direction vectors and real emotion category labels in the real speaker face image;

[0035] A first determination module is used to determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label;

[0036] A second determination module, configured to determine the generation model after training based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method;

[0037] The generation module is used to obtain the current speaking person's face image generated by the trained generation model based on the current voice data.

[0038] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating a speaking face image when executing the computer program.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for generating a speaking face image are implemented.

[0040] The present application provides a method, apparatus, computer equipment and storage medium for generating a speaking face image, which obtain preset voice data and a real speaking face image corresponding to the preset voice data; obtain a predicted speaking face image based on a preset generation model, the preset voice data and a predefined acquisition method; obtain predicted facial key point features, predicted head posture features, predicted sight line direction vector and predicted emotion category label in the predicted speaking face image, and obtain real facial key point features, real head posture features, real sight line direction vector and real emotion category label in the real speaking face image; determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, and determine a second loss value based on the predicted sight line direction vector and the real sight line direction vector. Determine a third loss value, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label; determine the trained generative model based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method; obtain the current speaker's face image generated by the trained generative model based on the current voice data. The beneficial effects are in two aspects. On the one hand, since the trained generative model can learn the real facial key point features, the real head posture features, the real line of sight direction vector and the real emotion category label, the problem of the learned features being not comprehensive enough is solved, which is conducive to improving the image quality of the generated current speaker's face image; on the other hand, since the trained generative model will not be affected by human intervention, it is conducive to improving the reliability of the acquired current speaker's face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0042] Figure 1 is a schematic diagram of an application environment of a method for generating a speaking face image in one embodiment of the present invention;

[0043] Figure 2 A schematic flow chart of a method for generating a speaking face image provided by an embodiment of the present invention;

[0044] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;

[0045] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;

[0046] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;

[0047] Figure 6 is a structural schematic diagram of a device for generating a speaking face image in one embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0049] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a method for generating a speaking face image according to an embodiment of the present invention. The method for generating a speaking face image according to an embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a client communicates with a server through a network.

[0052] The server obtains preset voice data and a real speaking face image corresponding to the preset voice data through the client;

[0053] Based on a preset generation model, the preset voice data and a predefined acquisition method, obtaining a predicted speaking face image;

[0054] Acquire predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and acquire real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image;

[0055] Determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label;

[0056] Determine the generation model that has been trained based on the first loss value, the second loss value, the third loss value, the fourth loss value, and a predefined determination method;

[0057] Acquire a current speaking person's face image generated by the trained generation model based on current speech data.

[0058] In the scheme implemented by the above-mentioned speaking face image generation method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, since the trained generation model can learn the real facial key point features, real head posture features, real line of sight direction vectors and real emotion category labels, the problem of the learned features being not comprehensive enough is solved, which is beneficial to improving the image quality of the generated current speaking face image; on the other hand, since the trained generation model will not be affected by human intervention, it is beneficial to improve the reliability of the acquired current speaking face image.

[0059] The device running the client is referred to as a client device.

[0060] The device running the server is referred to as the server device.

[0061] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0062] The server device can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.

[0063] See also Figure 2 , Figure 2 A flow chart of a method for generating a speaking face image provided by an embodiment of the present invention includes the following steps:

[0064] S21, obtaining preset voice data and a real speaking face image corresponding to the preset voice data;

[0065] Wherein, the target platform is connected, and preset voice data and a real speaking face image corresponding to the preset voice data are obtained from a storage device of the target platform.

[0066] Among them, the target platforms include but are not limited to financial platforms, technology platforms and insurance platforms.

[0067] Among them, real speaking face images refer to dynamic facial images captured and recorded in real time by camera equipment in a natural communication environment. These images contain the speaker's rich facial expressions, lip shape changes, head posture and other dynamic features.

[0068] S22, acquiring a predicted speaking face image based on a preset generation model, the preset voice data, and a predefined acquisition method;

[0069] Among them, the preset generative model uses a pre-trained model. Since the pre-trained model has been trained on a large-scale dataset for a long time and has accumulated rich feature representations and prior knowledge, the generative model can efficiently adapt to different downstream tasks through transfer learning.

[0070] Wherein, obtaining the predicted speaking face image based on the preset generation model, the preset voice data and the predefined acquisition method includes:

[0071] Extracting features from the preset voice data using a voice encoder to obtain a first voice feature;

[0072] The first speech feature is input into the generative model to obtain a predicted speaking face image generated by the generative model based on the first speech feature.

[0073] S23, obtaining predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and obtaining real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image;

[0074] The predicted facial key point features are predicted facial key point features.

[0075] The predicted head posture feature is the predicted head posture feature.

[0076] The predicted sight line direction vector is the predicted sight line direction vector.

[0077] Among them, the predicted emotion category label is the predicted emotion category label.

[0078] Among them, the real facial key point features are real facial key point features.

[0079] Among them, the real head posture feature is the real head posture feature.

[0080] The real sight direction vector is the real sight direction vector.

[0081] Among them, the true emotion category label is the true emotion category label.

[0082] Among them, facial key point features refer to a series of feature points with significant distinctiveness and stability that are automatically detected and located in facial images through algorithms. These features are usually distributed in key parts of the face, such as the corners of the eyes, corners of the mouth, tip of the nose, edge of the eyebrows, chin contour, etc.

[0083] Among them, the head posture feature refers to the position and direction information of an individual's head in space relative to the ground of a reference coordinate system.

[0084] Among them, the line of sight direction vector refers to a three-dimensional vector that can accurately represent the line of sight direction and orientation of an individual in the visual information obtained through image capture and analysis technology.

[0085] Among them, the emotion category label is a label that marks the emotion category. Emotion category labels include but are not limited to happiness, sadness, anger, surprise, fear, and disgust.

[0086] S24, determining a first loss value based on the predicted facial key point features and the real facial key point features, determining a second loss value based on the predicted head posture features and the real head posture features, determining a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determining a fourth loss value based on the predicted emotion category label and the real emotion category label;

[0087] The method of determining a first loss value based on the predicted facial key point features and the real facial key point features, determining a second loss value based on the predicted head posture features and the real head posture features, determining a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determining a fourth loss value based on the predicted emotion category label and the real emotion category label includes:

[0088] Calculate the loss value between the predicted facial key point feature and the real facial key point feature through the loss function, and select the loss value between the predicted facial key point feature and the real facial key point feature as the first loss value; calculate the loss value between the predicted head posture feature and the real head posture feature through the loss function, and select the loss value between the predicted head posture feature and the real head posture feature as the second loss value;

[0089] The loss value between the predicted sight line direction vector and the real sight line direction vector is calculated by the loss function, and the loss value between the predicted sight line direction vector and the real sight line direction vector is selected as the third loss value; the loss value between the predicted emotion category label and the real emotion category label is calculated by the loss function, and the loss value between the predicted emotion category label and the real emotion category label is selected as the fourth loss value.

[0090] The loss function includes a cross entropy loss function, an absolute loss function, or a combination thereof.

[0091] S25, determining the generation model that has been trained based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method;

[0092] Among them, the first loss value, the second loss value, the third loss value, and the fourth loss value are added to generate the overall loss value, the generation model is trained based on the overall loss value, and the generation model after the overall loss value converges is selected as the trained generation model.

[0093] The convergence of the overall loss value means that the generative model has found a relatively stable parameter configuration during the training process, so that the generative model can also show good generalization ability on unseen data. Therefore, selecting the generative model after the overall loss value converges as the trained generative model can not only improve the accuracy and stability of the trained generative model, but also ensure the reliability and effectiveness of the trained generative model in practical applications.

[0094] S26, obtaining the current speaker's face image generated by the trained generation model based on the current voice data.

[0095] In this embodiment, the current voice data includes voice data of the financial platform, voice data of the technology platform, and voice data of the insurance platform.

[0096] The voice data of the financial platform refers to the data records generated by the customer service through voice interaction on the financial service platform. The voice data of the financial platform is usually composed of the voice of the financial customer service personnel, including but not limited to the marketing voice and the response voice of the financial customer service personnel.

[0097] The voice data of the technology platform refers to the data set generated when the customer service uses the technology platform for voice interaction. The voice data of the technology platform is usually composed of the voice of the technology customer service personnel, including but not limited to the marketing voice and the answering voice of the technology customer service personnel.

[0098] In the context of the insurance platform, voice data refers to the audio information resources generated through voice interaction during the operation of insurance business. The voice data of the insurance platform is usually composed of the voices of insurance customer service personnel, including but not limited to the marketing voices and the response voices of insurance customer service personnel.

[0099] Wherein, after acquiring the current speaker face image generated by the trained generation model based on the current voice data, the speaker face image generation method comprises:

[0100] A preset synthesis instruction is obtained, and the current speaking face image and the current voice data are synthesized by executing the synthesis instruction to obtain a speaking face video.

[0101] Among them, since the image quality of the generated current speaking face image is improved, the current speaking face image and the current voice data are synthesized to obtain a speaking face video, which can improve the visual quality of the speaking face video.

[0102] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, since the generative model after training can learn the real facial key point features, real head posture features, real line of sight direction vectors and real emotion category labels, the problem of the learned features being not comprehensive enough is solved, which is beneficial to improving the image quality of the generated current speaking face image; on the other hand, since the generative model after training will not be affected by human intervention, it is beneficial to improve the reliability of the acquired current speaking face image.

[0103] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:

[0104] S31, obtaining a first extraction instruction, executing the first extraction instruction, performing feature extraction on the predicted speaking face image, and obtaining predicted facial key point features, predicted head posture features, predicted sight direction vector, and predicted emotion category label in the predicted speaking face image;

[0105] S32, obtaining a second extraction instruction, executing the second extraction instruction, performing feature extraction on the predicted speaking face image, and obtaining real facial key point features, real head posture features, real sight direction vector and real emotion category label in the real speaking face image.

[0106] In the embodiment of the present invention, the features learned by the generative model can be increased by using the real facial key point features, real head posture features, real sight direction vectors and real emotion category labels in the real speaking face image.

[0107] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:

[0108] S41, adding the first loss value, the second loss value, the third loss value, and the fourth loss value to generate the overall loss value, and training the generation model based on the overall loss value;

[0109] For the sake of illustration, an example is given below:

[0110] Based on a preset total loss value model, the first loss value, the second loss value, the third loss value, and the fourth loss value are added to generate a total loss value, and the total loss value model is:

[0111] Loss=Loss1+Loss2+Loss3+Loss4;

[0112] Among them, Loss represents the total loss value, Loss1 represents the first loss value, Loss2 represents the second loss value, Loss3 represents the third loss value, and Loss4 represents the fourth loss value.

[0113] S42: When the overall loss value is less than a preset loss value, stop training the generation model and obtain the generation model after training.

[0114] When the overall loss value is less than the preset loss value, the training of the generative model is stopped, and the generative model whose overall loss value is less than the preset loss value is selected as the generative model that has been trained. In an embodiment of the present invention, when the overall loss value is less than the preset loss value, the training of the generative model is stopped, and the generative model that has been trained is obtained. Since the generative model that has been trained is often verified by a large amount of data, it can provide more reliable prediction results, and thus can be directly applied to actual tasks.

[0115] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:

[0116] S51, obtaining current voice data, and performing feature extraction on the current voice data through a voice encoder to obtain a second voice feature;

[0117] S52: Input the second speech feature into the trained generation model, and obtain a face image of the current speaker generated by the trained generation model based on the second speech feature.

[0118] In an embodiment of the present invention, the second speech feature is input into the trained generative model, and the current speaker's face image generated by the trained generative model based on the second speech feature is obtained. Since the trained generative model will not be affected by human intervention, it is beneficial to improve the reliability of the obtained current speaker's face image.

[0119] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a device for generating a speaking face image according to an embodiment of the present invention. Figure 6 As shown, the speaker face image generation device includes a first acquisition module 101, a second acquisition module 102, a third acquisition module 103, a first determination module 104, a second determination module 105, and a generation module 106. The functional modules are described in detail as follows:

[0120] A first acquisition module 101 is used to acquire preset voice data and a real speaking face image corresponding to the preset voice data;

[0121] A second acquisition module 102, configured to acquire a predicted speaking face image based on a preset generation model, the preset voice data and a predefined acquisition method;

[0122] The third acquisition module 103 is used to acquire the predicted facial key point features, predicted head posture features, predicted sight line direction vector and predicted emotion category label in the predicted speaker face image, and acquire the real facial key point features, real head posture features, real sight line direction vector and real emotion category label in the real speaker face image;

[0123] A first determination module 104 is used to determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label;

[0124] A second determination module 105, configured to determine the generated model after training based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method;

[0125] The generation module 106 is used to obtain the current speaker's face image generated by the trained generation model based on the current speech data.

[0126] In one embodiment, the second acquisition module 102 includes:

[0127] A first extraction subunit is used to extract features from the preset voice data through a voice encoder to obtain a first voice feature;

[0128] The first acquisition subunit is used to input the first speech feature into the generation model to obtain the predicted speaking face image generated by the generation model based on the first speech feature.

[0129] In one embodiment, the third acquisition module 103 includes:

[0130] A second acquisition subunit is used to acquire a first extraction instruction, execute the first extraction instruction, perform feature extraction on the predicted speaker face image, and obtain predicted facial key point features, predicted head posture features, predicted sight direction vectors, and predicted emotion category labels in the predicted speaker face image;

[0131] The third acquisition subunit is used to acquire a second extraction instruction, execute the second extraction instruction, perform feature extraction on the predicted speaking face image, and obtain real facial key point features, real head posture features, real sight direction vector and real emotion category label in the real speaking face image.

[0132] In one embodiment, the first determining module 104 includes:

[0133] A first calculation subunit, configured to calculate a loss value between the predicted facial key point feature and the real facial key point feature through a loss function, and select the loss value between the predicted facial key point feature and the real facial key point feature as a first loss value;

[0134] A second calculation subunit is used to calculate the loss value between the predicted head posture feature and the real head posture feature through the loss function, and select the loss value between the predicted head posture feature and the real head posture feature as a second loss value;

[0135] A third calculation subunit is used to calculate the loss value between the predicted sight line direction vector and the real sight line direction vector by using the loss function, and select the loss value between the predicted sight line direction vector and the real sight line direction vector as a third loss value;

[0136] The fourth calculation subunit is used to calculate the loss value between the predicted emotion category label and the true emotion category label through the loss function, and select the loss value between the predicted emotion category label and the true emotion category label as the fourth loss value.

[0137] In one embodiment, the second determining module 105 includes:

[0138] an adding subunit, configured to add the first loss value, the second loss value, the third loss value, and the fourth loss value to generate the overall loss value, and train the generation model based on the overall loss value;

[0139] The fourth acquisition subunit is used to stop training the generation model and acquire the generation model after training when the overall loss value is less than a preset loss value.

[0140] In one embodiment, the generating module 106 includes:

[0141] A fifth acquisition subunit is used to acquire current voice data, and perform feature extraction on the current voice data through a voice encoder to obtain a second voice feature;

[0142] A generation subunit is used to input the second speech feature into the trained generation model, and obtain a current speaker's face image generated by the trained generation model based on the second speech feature.

[0143] In one embodiment, the apparatus for generating a speaking face image further includes:

[0144] The synthesis module is used to obtain a preset synthesis instruction, and synthesize the current speaker's face image and the current voice data by executing the synthesis instruction to obtain a speaker's face video.

[0145] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, since the generative model after training can learn the real facial key point features, real head posture features, real line of sight direction vectors and real emotion category labels, the problem of the learned features being not comprehensive enough is solved, which is beneficial to improving the image quality of the generated current speaking face image; on the other hand, since the generative model after training will not be affected by human intervention, it is beneficial to improve the reliability of the acquired current speaking face image.

[0146] The specific limitations of the speaker face image generating device can be found in the above limitations of the speaker face image generating method, which will not be elaborated here.

[0147] Each module in the above-mentioned speaker face image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0148] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.

[0149] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, the functions or steps of a method for generating a speaking face image can be implemented on the server device.

[0150] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database through a network connection. When the computer program is executed by the processor, the functions or steps of a method for generating a speaking face image can be implemented on a client device.

[0151] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.

[0152] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0154] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure.

Claims

1. A method for generating a speaking face image, characterized in that: include: Acquire preset voice data and a real speaking face image corresponding to the preset voice data; Based on a preset generation model, the preset voice data and a predefined acquisition method, obtaining a predicted speaking face image; Acquire predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and acquire real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image; Determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label; Determine the generation model that has been trained based on the first loss value, the second loss value, the third loss value, the fourth loss value, and a predefined determination method; The facial image of the current speaker generated by the trained generative model based on the current speech data is obtained.

2. The method for generating a speaking face image according to claim 1, characterized in that: The method of obtaining a predicted speaking face image based on a preset generation model, the preset voice data and a predefined acquisition method includes: Extracting features from the preset voice data using a voice encoder to obtain a first voice feature; The first speech feature is input into the generative model to obtain a predicted speaking face image generated by the generative model based on the first speech feature.

3. The method for generating a speaking face image according to claim 1, characterized in that: The step of obtaining predicted facial key point features, predicted head posture features, predicted sight line direction vectors, and predicted emotion category labels in the predicted speaker face image, and obtaining real facial key point features, real head posture features, real sight line direction vectors, and real emotion category labels in the real speaker face image includes: Obtaining a first extraction instruction, executing the first extraction instruction, performing feature extraction on the predicted speaking face image, and obtaining predicted facial key point features, predicted head posture features, predicted sight direction vectors, and predicted emotion category labels in the predicted speaking face image; A second extraction instruction is obtained, and the second extraction instruction is executed to perform feature extraction on the predicted speaking face image to obtain real facial key point features, real head posture features, real sight direction vectors, and real emotion category labels in the real speaking face image.

4. The method for generating a speaking face image according to claim 1, characterized in that: The method of determining a first loss value based on the predicted facial key point features and the real facial key point features, determining a second loss value based on the predicted head posture features and the real head posture features, determining a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determining a fourth loss value based on the predicted emotion category label and the real emotion category label includes: Calculate the loss value between the predicted facial key point feature and the real facial key point feature through a loss function, and select the loss value between the predicted facial key point feature and the real facial key point feature as a first loss value; Calculate the loss value between the predicted head posture feature and the real head posture feature by using the loss function, and select the loss value between the predicted head posture feature and the real head posture feature as the second loss value; Calculating a loss value between the predicted sight line direction vector and the actual sight line direction vector by using the loss function, and selecting the loss value between the predicted sight line direction vector and the actual sight line direction vector as a third loss value; The loss value between the predicted emotion category label and the true emotion category label is calculated by the loss function, and the loss value between the predicted emotion category label and the true emotion category label is selected as the fourth loss value.

5. The method for generating a speaking face image according to claim 1, characterized in that: The step of determining the training-completed generation model based on the first loss value, the second loss value, the third loss value, the fourth loss value, and a predefined determination method includes: Adding the first loss value, the second loss value, the third loss value, and the fourth loss value to generate the overall loss value, and training the generation model based on the overall loss value; When the overall loss value is less than a preset loss value, the training of the generation model is stopped, and the generation model after training is obtained.

6. The method for generating a speaking face image according to claim 1, characterized in that: The step of obtaining the current speaking face image generated by the trained generative model based on the current voice data comprises: Acquire current voice data, and perform feature extraction on the current voice data through a voice encoder to obtain a second voice feature; The second speech feature is input into the trained generation model, and a face image of a current speaker generated by the trained generation model based on the second speech feature is obtained.

7. The method for generating a speaking face image according to any one of claims 1 to 6, characterized in that: After acquiring the current speaker face image generated by the trained generation model based on the current voice data, the speaker face image generation method comprises: A preset synthesis instruction is obtained, and the current speaking face image and the current voice data are synthesized by executing the synthesis instruction to obtain a speaking face video.

8. A device for generating a speaking face image, characterized in that: include: A first acquisition module, used to acquire preset voice data and a real speaking face image corresponding to the preset voice data; A second acquisition module, used to acquire a predicted speaking face image based on a preset generation model, the preset voice data and a predefined acquisition method; A third acquisition module is used to acquire predicted facial key point features, predicted head posture features, predicted sight line direction vectors and predicted emotion category labels in the predicted speaker face image, and to acquire real facial key point features, real head posture features, real sight line direction vectors and real emotion category labels in the real speaker face image; A first determination module is used to determine a first loss value based on the predicted facial key point features and the real facial key point features, determine a second loss value based on the predicted head posture features and the real head posture features, determine a third loss value based on the predicted sight line direction vector and the real sight line direction vector, and determine a fourth loss value based on the predicted emotion category label and the real emotion category label; A second determination module, configured to determine the generation model after training based on the first loss value, the second loss value, the third loss value, the fourth loss value and a predefined determination method; The generation module is used to obtain the current speaking person's face image generated by the trained generation model based on the current voice data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a speaking face image according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a speaking face image according to any one of claims 1 to 7 are implemented.

Images