A method for generating a virtual image and related equipment

Abstract

The present application discloses a method for generating a virtual image and related equipment thereof, the method comprising: after acquiring voice data to be processed, firstly extracting voice features of the voice data to be processed to obtain voice features to be used, so that the voice features to be used can represent the voice information carried by the voice data to be processed; then determining virtual image display data corresponding to the voice data to be processed according to the voice features to be used, facial parameters to be used, and a pre-constructed virtual image generation model, so that the virtual image display data can represent a virtual image that matches the voice data to be processed, so that the virtual image can display the facial state presented by the character when the voice data to be processed is emitted, thereby achieving the purpose of automatically generating a virtual image.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for generating a virtual image and related equipment. Background Art

[0002] Virtual images are a new interactive display medium that has emerged with the rapid development of data processing technology (for example, speech synthesis technology, video generation technology, etc.); and this virtual image can greatly enhance the naturalness and experience of human-computer interaction, and has broad application and development prospects in anchor broadcasting, customer service, and interactive scenarios.

[0003] However, how to generate virtual images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a virtual image generation method and related equipment, which can realize automatic generation of a virtual image.

[0005] An embodiment of the present application provides a method for generating a virtual image, the method comprising: after acquiring voice data to be processed, performing voice feature extraction on the voice data to be processed to obtain voice features to be used; determining virtual image display data corresponding to the voice data to be processed based on the voice features to be used, facial parameters to be used, and a pre-constructed virtual image generation model.

[0006] An embodiment of the present application also provides a virtual image generation device, including: a feature extraction unit, which is used to extract voice features of the voice data to be processed after acquiring the voice data to be processed, so as to obtain voice features to be used; and an image generation unit, which is used to determine the virtual image display data corresponding to the voice data to be processed based on the voice features to be used, the facial parameters to be used, and a pre-constructed virtual image generation model.

[0007] An embodiment of the present application also provides a device, which includes: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute any implementation of the virtual image generation method provided in the embodiment of the application.

[0008] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any implementation of the virtual image generation method provided in the embodiment of the application.

[0009] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any implementation of the virtual image generation method provided in the embodiment of the present application.

[0010] Based on the above technical solution, this application has the following beneficial effects:

[0011] In the technical solution provided by the present application, after obtaining the voice data to be processed (for example, a frame of voice data in a voice stream, etc.), voice features are first extracted from the voice data to be processed to obtain the voice features to be used, so that the voice features to be used can represent the voice information (for example, voiceprints, phonemes, etc.) carried by the voice data to be processed; then, based on the voice features to be used, the facial parameters to be used, and a pre-constructed virtual image generation model, the virtual image display data corresponding to the voice data to be processed is determined, so that the virtual image display data can represent a virtual image that matches the voice data to be processed, so that the virtual image can display the facial state (for example, eye expressions, lip expressions, etc.) presented by the character when the voice data to be processed is issued, thereby achieving the purpose of automatically generating a virtual image.

[0012] In addition, because the virtual image generation model has good virtual image generation performance, the virtual image display data determined based on the virtual image generation model can better represent the virtual image that matches the voice data to be processed, so that the virtual image can better display the facial state (for example, eye expression, lip expression, etc.) of the character when the voice data to be processed is issued, which is conducive to improving the virtual image generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart of a method for generating a virtual image provided by an embodiment of the present application;

[0015] Figure 2 A schematic diagram of a virtual image generation process provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of another virtual image generation process provided in an embodiment of the present application;

[0017] Figure 4A schematic diagram of a facial parameter extraction process provided in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of the structure of a virtual image generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] Method Example 1

[0021] See also Figure 1 , which is a flow chart of a virtual image generation method provided in an embodiment of the present application.

[0022] The virtual image generation method provided in the embodiment of the present application includes S1-S2:

[0023] S1: After acquiring the speech data to be processed, extracting speech features from the speech data to be processed to obtain speech features to be used.

[0024] The above-mentioned "voice data to be processed" refers to the voice data that needs to be processed for virtual image generation; and the embodiment of the present application does not limit the "voice data to be processed", for example, the "voice data to be processed" can be a frame of voice data. For another example, the "voice data to be processed" can be a segment of voice data (that is, the "voice data to be processed" includes at least two frames of voice data).

[0025] In addition, the embodiment of the present application does not limit the acquisition process of the above-mentioned "voice data to be processed". For ease of understanding, the following is an explanation in combination with three situations.

[0026] Case 1: When a virtual image generation process is performed for a segment of acquired audio data, and the above-mentioned "voice data to be processed" includes a frame of voice data, the acquisition process of the above-mentioned "voice data to be processed" may specifically include: determining the mth frame of voice data in the audio data as the voice data to be processed, so that the virtual image display image corresponding to the mth frame of voice data can be determined with the help of the virtual image generation process for the voice data to be processed, so that the virtual image display image can represent the facial state of the character when the mth frame of voice data is sent. Wherein, m≤M, M is a positive integer, and M represents the number of frames of voice data in the above-mentioned "audio data".

[0027] Case 2: When a virtual image generation process is performed on a segment of acquired audio data, and the above-mentioned "voice data to be processed" includes at least two frames of voice data, the acquisition process of the above-mentioned "voice data to be processed" may specifically include: determining the audio data as voice data to be processed, so that subsequently, with the help of a virtual image generation process for each frame of voice data in the voice data to be processed, the virtual image display image corresponding to the audio data can be determined, so that the virtual image display image can represent the changes in the facial state of the person when the audio data is emitted.

[0028] Case 3, when a virtual image generation process is performed for a real-time voice stream, and the above-mentioned "voice data to be processed" includes a frame of voice data, the acquisition process of the above-mentioned "voice data to be processed" may specifically include: after acquiring the current frame of voice data, the current frame of voice data is determined as the voice data to be processed, so that the virtual image display image corresponding to the current frame of voice data can be determined with the help of the virtual image generation process for the voice data to be processed, so that the virtual image display image can show the facial state of the character when the current frame of voice data is issued, so that the real-time virtual image generation process can be realized for the voice stream, which is conducive to improving the real-time performance of the virtual image generation. Among them, the above-mentioned "current frame of voice data" is used to represent the voice data collected in real time from the voice stream.

[0029] The above-mentioned "speech feature extraction" is used to perform speech information extraction processing on a speech data (for example, Mel-scale Frequency Cepstral Coefficients (MFCC) extraction processing, phoneme extraction processing, etc.); and the embodiment of the present application is not limited to the "speech feature extraction", which can be implemented using any existing or future speech feature extraction method.

[0030] The above-mentioned “voice features to be used” are used to represent the voice information (for example, voiceprint, phonemes, etc.) carried by the voice data to be processed; and the embodiment of the present application does not limit the “voice features to be used”, for example, it may include the MFCC of the voice data to be processed, and at least one of the phoneme features of the voice data to be processed.

[0031] S2: Determine the virtual image display data corresponding to the voice data to be processed according to the voice features to be used, the facial parameters to be used, and the pre-built virtual image generation model.

[0032] The above-mentioned "facial parameters to be used" are used to represent the basic facial state required to be referenced when generating a virtual image for the voice data to be processed; and the embodiment of the present application does not limit the "facial parameters to be used", for example, it may include at least one of the facial expression parameters to be used, the facial posture parameters to be used, and the facial shape parameters to be used.

[0033] The above-mentioned "facial expression parameters to be used" are used to represent the basic facial expressions (e.g., opening the mouth, opening and closing the eyes, etc.) required to be referenced when generating the virtual image for the voice data to be processed; and the embodiments of the present application do not limit the "facial expression parameters to be used". For example, the "facial expression parameters to be used" may only include facial expression parameters unrelated to lip shape (that is, facial expression parameters related to other parts except the mouth shape). For another example, the "facial expression parameters to be used" may include facial expression parameters related to lip shape, and facial expression parameters unrelated to lip shape.

[0034] It should be noted that the above-mentioned "facial expression parameters related to lip shape" are used to control the lip shape on the human face (for example, opening the mouth, etc.); the above-mentioned "facial expression parameters not related to lip shape" are used to control other parts of the human face except the lip shape (for example, eyes, etc.).

[0035] The above-mentioned "facial posture parameters to be used" are used to represent the basic facial postures (for example, lowering the head, side face and other facial orientation postures) that need to be referred to when performing virtual image generation processing on the voice data to be processed.

[0036] The above-mentioned "face shape parameters to be used" are used to represent the basic face shape (for example, face length, fatness, thinness, etc.) that needs to be referred to when performing virtual image generation processing on the voice data to be processed.

[0037] In addition, the embodiments of the present application do not limit the process of obtaining the above-mentioned "facial parameters to be used". For example, it can be set by the user through human-computer interaction, or it can be extracted from a standard image of the virtual image provided by the user, or it can be implemented using the sample facial parameters involved in constructing the virtual image generation model.

[0038] It should be noted that the relevant content of the above-mentioned "sample facial parameters" can be found in the relevant content of step 43 below. In addition, the embodiment of the present application does not limit the implementation method of "extracting facial parameters to be used from the virtual image standard image", for example, it can be similar to the determination process of the "sample facial parameters" shown in step 43 below.

[0039] The above-mentioned "virtual image generation model" is used to perform virtual image generation processing on the input data of the virtual image generation model; and the "virtual image generation model" is a machine learning model. It should be noted that the relevant content of the above-mentioned "virtual image generation model" can be found below Method Example 2 .

[0040] The above-mentioned "virtual image display data corresponding to the voice data to be processed" is used to represent the virtual image that matches the voice data to be processed, so that the virtual image can display the facial state (for example, eye expression, lip expression, etc.) of the character when the voice data to be processed is issued.

[0041] In addition, the embodiments of the present application do not limit the above-mentioned "virtual image display data corresponding to the voice data to be processed". For example, if the above-mentioned "voice data to be processed" includes one frame of voice data, then the "virtual image display data corresponding to the voice data to be processed" is an image data. For another example, if the above-mentioned "voice data to be processed" includes at least two frames of voice data, then the "virtual image display data corresponding to the voice data to be processed" can be an image data sequence.

[0042] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned “image data sequence”. For example, when the above-mentioned “voice data to be processed” includes J frames of voice data, and the above-mentioned “voice features to be used” include voice features of J frames of voice data, the determination process of the “image data sequence” may specifically include steps 11-12:

[0043] Step 11: Determine the virtual image display image corresponding to the j-th frame of voice data based on the voice features of the j-th frame of voice data, the facial parameters to be used, and the pre-built virtual image generation model; wherein j is a positive integer, j≤J, J is a positive integer, and J≥2.

[0044] The above-mentioned “voice features of the j-th frame of voice data” is used to represent the voice information (for example, voiceprint, phoneme, etc.) carried by the j-th frame of voice data in the voice data to be processed; and the embodiment of the present application does not limit the “voice features of the j-th frame of voice data”, for example, it may include the MFCC of the j-th frame of voice data, and at least one of the phoneme features of the j-th frame of voice data.

[0045] The above-mentioned "virtual image display image corresponding to the j-th frame of voice data" is used to represent the virtual image that matches the j-th frame of voice data in the voice data to be processed, so that the virtual image can display the facial state (for example, eye expression, lip expression, etc.) of the character when the j-th frame of voice data is issued.

[0046] The present application embodiment does not limit the implementation of step 11. For example, it is similar to the following Method Example 2Any implementation method of obtaining "virtual image display data corresponding to the voice data to be processed" is shown.

[0047] Step 12: Determine the virtual image display data corresponding to the voice data to be processed based on the virtual image display image corresponding to the 1st frame of voice data to the virtual image display image corresponding to the Jth frame of voice data.

[0048] In an embodiment of the present application, after obtaining the virtual image display image corresponding to the first frame of voice data, the virtual image display image corresponding to the second frame of voice data, ..., and the virtual image display image corresponding to the J frame of voice data in the voice data to be processed, the virtual image display images corresponding to the J frames of voice data can be sorted and assembled according to the time sequence of the J frames of voice data to obtain the virtual image display data corresponding to the voice data to be processed, so that the virtual image display data includes J virtual image display images arranged in order, so that the virtual image display data can represent the changes in the facial state of the character when the voice data to be processed is issued in the form of an image data sequence, so that a video data can be constructed based on the virtual image display data and the voice data to be processed subsequently, so that the video data can synchronously play each frame of voice data and its corresponding virtual image display data.

[0049] Based on the relevant contents of the above steps 11 to 12, it can be known that for a voice segment including multiple frames of voice data, a virtual image generation process can be performed for each frame of voice data in the voice segment to obtain a virtual image display image corresponding to each frame of voice data; and then these virtual image display images are sorted and assembled in the order of the collection time of these voice data from early to late to obtain an image data sequence corresponding to the voice segment, so that the image data sequence can display the changes in facial state in accordance with the playback speed of the voice segment, so that the image data sequence can better express the changes in facial state of the character when the voice segment is uttered.

[0050] Based on the relevant contents of S1 to S2 above, it can be known that for the virtual image generation method provided in the embodiment of the present application, after obtaining the voice data to be processed, the voice features of the voice data to be processed are first extracted to obtain the voice features to be used, so that the voice features to be used can represent the voice information (for example, voiceprints, phonemes, etc.) carried by the voice data to be processed; then, according to the voice features to be used, the facial parameters to be used, and the pre-constructed virtual image generation model, the virtual image display data corresponding to the voice data to be processed is determined, so that the virtual image display data can represent a virtual image that matches the voice data to be processed, so that the virtual image can display the facial state (for example, eye expressions, lip expressions, etc.) presented by the character when the voice data to be processed is issued, thereby achieving the purpose of automatically generating a virtual image.

[0051] In addition, because the virtual image generation model has good virtual image generation performance, the virtual image display data determined based on the virtual image generation model can better represent the virtual image that matches the voice data to be processed, so that the virtual image can better display the facial state (for example, eye expression, lip expression, etc.) of the character when the voice data to be processed is issued, which is conducive to improving the virtual image generation effect.

[0052] In addition, the embodiments of the present application do not limit the execution subject of the virtual image generation method. For example, the virtual image generation method provided in the embodiments of the present application can be applied to data processing devices such as terminal devices or servers. The terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be an independent server, a cluster server or a cloud server.

[0053] Method Example 2

[0054] In order to improve the virtual image generation effect, the embodiment of the present application also provides a possible implementation method of the above-mentioned "virtual image generation model", which may specifically include: a lip expression parameter prediction module, a parameter fusion module, a textureless image rendering module and a texture image generation module; and the input data of the parameter fusion module includes the output data of the lip expression parameter prediction module, the input data of the textureless image rendering module includes the output data of the parameter fusion module, and the input data of the texture image generation module includes the output data of the textureless image rendering module.

[0055] In order to facilitate understanding of the working principle of the above-mentioned "virtual image generation model", the following is an explanation taking the determination process of the above-mentioned "virtual image display data corresponding to the voice data to be processed" as an example.

[0056] As an example, when the above “voice data to be processed” includes a frame of voice data, and the above “face parameters to be used” include face expression parameters to be used, face posture parameters to be used, and face shape parameters to be used, the determination process of the above “virtual image display data corresponding to the voice data to be processed” may specifically include steps 21 to 24:

[0057] Step 21: input the speech features to be used into the lip expression parameter prediction module to obtain the lip expression parameters to be used output by the lip expression parameter prediction module.

[0058] The above-mentioned "lip expression parameter prediction module" is used to perform lip expression parameter prediction processing on the input data of the lip expression parameter prediction module; and the embodiment of the present application does not limit the "lip expression parameter prediction module", for example, it may include: a multi-layer fully connected network (for example, a 3-layer fully connected network) for implementation.

[0059] It should be noted that the above-mentioned “lip shape expression parameters” refer to facial expression parameters related to lip shape.

[0060] The above-mentioned “lip expression parameters to be used” are used to represent the facial expressions related to the lip shape presented by the character when the voice data to be processed is uttered.

[0061] Step 22: Input the lip expression parameters to be used and the facial expression parameters to be used into a parameter fusion module to obtain the expression fusion parameters to be used output by the parameter fusion module.

[0062] The above-mentioned "parameter fusion module" is used to perform fusion processing on the input data of the parameter fusion module; and the embodiment of the present application does not limit the implementation method of the "parameter fusion module", for example, it can be implemented by any existing or future feature fusion method.

[0063] The above-mentioned "expression fusion parameters to be used" are used to represent the facial expressions presented by the character when the voice data to be processed is issued (for example, facial expressions related to the mouth shape, and facial expressions related to other parts except the mouth shape, etc.).

[0064] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned "expression fusion parameters to be used". For ease of understanding, two examples are used for explanation below.

[0065] Example 1: If the above-mentioned "facial expression parameters to be used" include facial expression parameters related to lip shape and facial expression parameters unrelated to lip shape, then the determination process of the above-mentioned "expression fusion parameters to be used" can be specifically: using the lip shape expression parameters to be used, directly replace the facial expression parameters related to lip shape in the facial expression parameters to be used, and obtain the expression fusion parameters to be used.

[0066] It should be noted that the embodiments of the present application do not limit the implementation method of the above-mentioned "replacement". For example, it can be specifically as follows: first, the facial expression parameters related to the lip shape in the facial expression parameters to be used are deleted to obtain the deleted expression parameters; then the deleted expression parameters are concatenated with the lip shape expression parameters to be used (as shown in formula (1)) to obtain the expression fusion parameters to be used.

[0067] β joint =concat(β other , β lip ) (1)

[0068] In the formula, β joint represents the expression fusion parameter to be used; β other represents the expression parameters after deletion (that is, the facial expression parameters that are not related to lip shape); β lip Indicates the lip expression parameters to be used.

[0069] Example 2: If the above-mentioned "facial expression parameters to be used" only include facial expression parameters unrelated to lip shape, the specific process of determining the above-mentioned "expression fusion parameters to be used" can be: the facial expression parameters to be used and the lip shape expression parameters to be used are spliced ​​(such as Figure 2 As shown), the expression fusion parameters to be used are obtained.

[0070] Based on the relevant content of step 22, it can be known that for the virtual image generation model, after obtaining the lip expression parameters to be used, the parameter fusion module in the virtual image generation model can fuse the lip expression parameters to be used with the facial expression parameters to be used, and obtain and output the expression fusion parameters to be used, so that the expression fusion parameters to be used can better represent the facial state of the character when the voice data to be processed is issued, so as to achieve the purpose of controllable editing of facial expression parameters related to the lip shape, thereby effectively avoiding the phenomenon of errors in the lip shape of the virtual image.

[0071] Step 23: Input the expression fusion parameters to be used, the facial posture parameters to be used, and the facial shape parameters to be used into the textureless image rendering module to obtain the textureless image to be used output by the textureless image rendering module.

[0072] The above-mentioned "untextured image to be used" refers to the untextured face image generated by the speech data to be processed (such as Figure 2 shown).

[0073] The above-mentioned "textureless image rendering module" is used to perform textureless image rendering processing (especially, textureless image micro-rendering processing) on ​​the input data of the textureless image rendering module; and the embodiment of the present application does not limit the working principle of the "textureless image rendering module". For ease of understanding, the following is an explanation taking the above-mentioned determination process of the "textureless image to be used" as an example.

[0074] As an example, the process of determining the above-mentioned “textureless image to be used” by using the textureless image rendering module may specifically include steps 31 to 33:

[0075] Step 31: Determine the transformation matrix to be used according to the facial pose parameters to be used.

[0076] The above-mentioned “transformation matrix to be used” refers to the transformation matrix obtained according to the facial posture parameters to be used; and the embodiment of the present application does not limit the determination process of the above-mentioned “transformation matrix to be used”.

[0077] Step 32: Determine the three-dimensional face model to be used according to the conversion matrix to be used, the expression fusion parameters to be used, and the face shape parameters to be used.

[0078] The above-mentioned "three-dimensional face model to be used" is used to represent the three-dimensional face of the person when the voice data to be processed is issued; and the embodiment of the present application does not limit the determination process of the "three-dimensional face model to be used", for example, it can be implemented using formula (2).

[0079]

[0080] Where V proj Indicates the 3D face model to be used, and the V proj Including the vertex coordinates of the 3D face model to be used; M represents the transformation matrix to be used (that is, the transformation matrix obtained according to the face pose parameters to be used); is the average face of the face model, and the Can be pre-set; SP ki Represents the face shape basis in the 3D face model, and the SP ki Can be preset; α represents the face shape parameter to be used; EP ki Represents the facial expression base in the three-dimensional face model, and the EP ki Can be pre-set; joint Indicates the expression fusion parameters to be used.

[0081] Step 33: Determine the texture-free image to be used according to the 3D face model to be used.

[0082] The embodiment of the present application does not limit the implementation method of step 33. For example, in order to better implement the end-to-end virtual image generation process, step 33 may specifically include: performing image rendering processing on the three-dimensional face model to be used according to the differentiable soft rasterization method to obtain the texture-free image to be used. In order to facilitate understanding of the working principle of the above-mentioned "soft rasterization method", the following is explained with examples.

[0083] As an example, step 33 may specifically include steps 331 and 332:

[0084] Step 331: Perform two-dimensional image projection processing on the three-dimensional face model to be used to obtain a two-dimensional probability density map.

[0085] The above-mentioned “two-dimensional probability density map” is used to represent the projection probability of each triangular surface in the three-dimensional face model to be used on each pixel point on the two-dimensional image.

[0086] In addition, the embodiment of the present application does not limit the implementation of step 331. For example, it can be implemented using formula (3).

[0087]

[0088] In the formula, represents the projection probability of the b-th triangular face in the 3D face model to be used to the i-th pixel point on the 2D image; b is a positive integer, b≤B, B is a positive integer, and B represents the number of triangular faces in the 3D face model to be used; represents the indicator factor, and the The determination process is as follows: if the i-th pixel point on the two-dimensional image is located inside the projection triangle of the b-th triangle face on the two-dimensional image, then However, if the i-th pixel point on the two-dimensional image is outside the projection triangle of the b-th triangle face on the two-dimensional image, then d 2 (i, b) represents the Euclidean distance between the i-th pixel point on the two-dimensional image and the nearest edge in the b-th triangle face; σ is a hyperparameter that can be set in advance, and σ is used to adjust the shape of the probability density map.

[0089] It should be noted that the meaning of formula (3) is: for the projection triangle of the b-th triangular face in the three-dimensional face model to be used on the two-dimensional image, the probability density range of the internal pixel points of the projection triangle is between [0.5, 1], and the external points are between [0, 0.5].

[0090] Step 332: Determine the texture-free image to be used based on the two-dimensional probability density map and the three-dimensional face model to be used (as shown in formulas (4)-(6)).

[0091]

[0092]

[0093]

[0094] In the formula, represents the pixel value (that is, depth value) of the i-th pixel in the texture-free image to be used; represents the depth value of the corresponding point of the i-th pixel in the b-th triangle of the 3D face model to be used; express The weighted weight of represents the depth value of the first vertex in the b-th triangle of the 3D face model to be used (that is, the z-axis coordinate value of the vertex coordinates of the first vertex); represents the depth value of the second vertex in the b-th triangle of the 3D face model to be used (that is, the z-axis coordinate value of the vertex coordinates of the second vertex); represents the depth value of the third vertex in the b-th triangle of the 3D face model to be used (that is, the z-axis coordinate value of the vertex coordinates of the third vertex); express The weighted weight of The determination may be made based on the distance between the above “the corresponding point of the i-th pixel point in the b-th triangular face of the 3D face model to be used” and the first vertex in the b-th triangular face; express The weighted weight of The determination may be made based on the distance between the above “the corresponding point of the i-th pixel point in the b-th triangular face of the 3D face model to be used” and the second vertex in the b-th triangular face; express The weighted weight of It can be determined based on the distance between the above “corresponding point of the ith pixel point in the bth triangular face of the three-dimensional face model to be used” and the third vertex in the bth triangular face; γ is a hyperparameter and can be set in advance.

[0095] It should be noted that the above It can refer to the depth value calculated by the barycentric coordinates; and in determining the The weights involved in the weighting can be determined based on the center of gravity coordinates. Depth values ​​that represent triangles that are closer to the pixel and closer to the surface will have a larger weight.

[0096] Based on the relevant contents of the above steps 331 to 332, it can be known that after obtaining the 3D face model to be used, the 3D face model to be used can be rendered onto the image according to the soft rasterization method to obtain the texture-free image to be used, so that pixel gradients can be returned, which is beneficial to improving the virtual image generation effect. In addition, because the soft rasterization method is a differentiable rendering method, the virtual image generation model implemented based on the soft rasterization method can perform gradient return, so that the end-to-end virtual image generation process can be better realized.

[0097] Based on the relevant content of the above step 23, it can be known that for the virtual image generation model, after obtaining the expression fusion parameters to be used, the textureless image rendering module in the virtual image generation model can perform textureless facial image generation processing on the expression fusion parameters to be used, the facial posture parameters to be used, and the facial shape parameters to be used to obtain the textureless image to be used, so that the virtual image display data corresponding to the voice data to be processed can be generated based on the textureless image to be used.

[0098] Step 24: Determine the virtual image display data corresponding to the voice data to be processed according to the texture-free image and texture image generation module to be used.

[0099] The above-mentioned “texture image generation module” is used to perform image generation processing on the input data of the texture image generation module; and the embodiment of the present application does not limit the “texture image generation module”, for example, it can be implemented using a generation network in a generative adversarial network (GAN) with a U-net structure.

[0100] The embodiment of the present application does not limit the implementation method of step 24. For example, it may specifically include: inputting the texture-free image to be used into a texture image generation module to obtain virtual image display data corresponding to the voice data to be processed output by the texture image generation module.

[0101] In addition, in order to improve the virtual image generation effect, the embodiment of the present application also provides another possible implementation of step 24, which may specifically include: determining the virtual image display data corresponding to the voice data to be processed based on the texture-free image to be used, at least one frame of virtual image reference image and the texture image generation module.

[0102] The above-mentioned “at least one frame of virtual image reference image” refers to the virtual image display image required to be referenced when performing virtual image generation processing on the voice data to be processed.

[0103] In addition, the present application embodiment does not limit the acquisition method of the above-mentioned "at least one frame of virtual image reference image", for example, it can be preset. For another example, in order to improve the continuity of the virtual image in time sequence, the above-mentioned "at least one frame of virtual image reference image" can refer to at least one virtual image historical image corresponding to the voice data to be processed (for example, Figure 3 The "image generation data corresponding to the first two frames of voice data" shown in the figure) and the "at least one virtual image historical image corresponding to the voice data to be processed" can be generated based on at least one frame of historical voice data corresponding to the voice data to be processed. The acquisition time of the "at least one frame of historical voice data corresponding to the voice data to be processed" is earlier than the acquisition time of the voice data to be processed.

[0104] As an example, assume that the above-mentioned "voice data to be processed" is the rth frame of voice data in a voice segment (or, a real-time voice stream); the reth virtual image historical image is determined by performing virtual image generation processing on the reth frame of voice data in the voice segment (or, the real-time voice stream); e is a positive integer, e≤E, E is a positive integer, and r is a positive integer.

[0105] Based on the above assumptions, it can be known that when the above-mentioned "at least one frame of virtual image reference image" includes E virtual image historical images, the above-mentioned "at least one frame of historical voice data corresponding to the voice data to be processed" can include the r-1th frame of voice data, the r-2th frame of voice data,..., and the rEth frame of voice data, and the "at least one frame of virtual image reference image" can include the r-1th virtual image historical image, the r-2th virtual image historical image,..., and the rEth virtual image historical image.

[0106] In addition, the embodiment of the present application does not limit the implementation method of the above step of "determining the virtual image display data corresponding to the voice data to be processed according to the texture-free image to be used, at least one frame of the virtual image reference image and the texture image generation module", for example, it may specifically include steps 241-242:

[0107] Step 241: Add the texture-free image to be used and at least one frame of the virtual image reference image to obtain the fused image to be used (as shown in formula (7)).

[0108]

[0109] In the formula, F Add Indicates the fused image to be used; f r represents the texture-free image to be used (for example, the virtual image display image corresponding to the “rth frame voice data” above); f r-eRepresents the e-th frame virtual image reference image (for example, the “re-th virtual image historical image” above); e is a positive integer, e≤E, E is a positive integer, and r is a positive integer.

[0110] Step 242: input the fused image to be used into the texture image generation module to obtain the virtual image display data corresponding to the voice data to be processed output by the texture image generation module.

[0111] In an embodiment of the present application, for the virtual image generation model, after obtaining the fused image to be used, the texture image generation module in the virtual image generation model performs image generation processing on the fused image to be used, and obtains and outputs virtual image display data corresponding to the voice data to be processed.

[0112] Based on the relevant contents of the above steps 21 to 24, it can be known that for the virtual image generation model including the lip expression parameter prediction module, the parameter fusion module, the textureless image rendering module and the texture image generation module, after obtaining the voice features to be used, the lip expression parameter prediction module can first perform lip expression parameter prediction processing on the voice features to be used to obtain the lip expression parameters to be used, so that the lip expression parameters to be used can more accurately represent the facial expressions related to the mouth shape presented by the character when the voice data to be processed is issued; and then the parameter fusion module can perform lip expression parameter prediction processing on the voice features to be used to obtain the lip expression parameters to be used. The facial expression parameters are fused with the facial expression parameters to be used to obtain expression fusion parameters, so that the expression fusion parameters can better represent the facial expression presented by the character when the voice data to be processed is issued; finally, the texture-free image rendering module and the texture image generation module refer to the expression fusion parameters, the facial posture parameters to be used, and the facial shape parameters to be used to generate virtual image display data corresponding to the voice data to be processed, so that the virtual image display data can more accurately represent the virtual image that matches the voice data to be processed, which is conducive to improving the virtual image generation effect.

[0113] Method Example 3

[0114] In order to improve the virtual image generation performance of the virtual image generation model, the embodiment of the present application also provides a possible implementation method of constructing the above-mentioned "virtual image generation model", which may specifically include steps 41-44:

[0115] Step 41: Get a sample video.

[0116] The above-mentioned "sample video" refers to the video data required to be used when constructing the virtual image generation model; and the embodiment of the present application does not limit the number of the "sample videos". It should be noted that, for the sake of convenience, steps 42 to 44 are all described by taking the use process of a sample video as an example.

[0117] Step 42: Determine at least one sample voice data and a video image corresponding to at least one sample voice data according to the sample video.

[0118] The above-mentioned "nth sample voice data" is used to represent the voice data corresponding to the nth frame of video image extracted from the sample video; and the playback time of the "nth sample voice data" in the sample video is consistent with the playback time of the nth frame of video image. Wherein, n is a positive integer, n≤N, N is a positive integer, and N represents the number of sample voice data.

[0119] “The video image corresponding to the nth sample voice data” refers to the video image corresponding to the nth sample voice data in the sample video (that is, the nth frame video image in the sample video). Where n is a positive integer, n≤N, N is a positive integer, and N represents the number of sample voice data.

[0120] The embodiment of the present application does not limit the implementation method of step 42. For example, when the sample video includes N frames of video images, step 42 may specifically include: determining the voice data corresponding to the n-th frame of video image in the sample video as the n-th sample voice data, and determining the n-th frame of video image in the sample video as the video image corresponding to the n-th sample voice data. Where n is a positive integer, n≤N.

[0121] Step 43: Determine the sample face parameters corresponding to each sample voice data according to the video image corresponding to each sample voice data.

[0122] The "sample face parameter corresponding to the nth sample voice data" is used to represent the face state carried by the video image corresponding to the nth sample voice data; and the embodiment of the present application does not limit the "sample face parameter corresponding to the nth sample voice data", for example, it may include: at least one of the sample face expression parameter corresponding to the nth sample voice data, the sample face posture parameter corresponding to the nth sample voice data, and the sample face shape parameter corresponding to the nth sample voice data. Where n is a positive integer, n≤N.

[0123] The above-mentioned "sample facial expression parameters corresponding to the n-th sample voice data" are used to represent the facial expression (e.g., open mouth, open and closed eyes, etc.) carried by the video image corresponding to the n-th sample voice data; and the embodiment of the present application does not limit the "sample facial expression parameters corresponding to the n-th sample voice data", for example, it may only include facial expression parameters unrelated to lip shape. For another example, the "sample facial expression parameters corresponding to the n-th sample voice data" may include facial expression parameters related to lip shape, and facial expression parameters unrelated to lip shape.

[0124] The above-mentioned “sample facial posture parameter corresponding to the nth sample voice data” is used to represent the facial posture (for example, lowering the head, sideways, and other facial orientation postures) carried by the video image corresponding to the nth sample voice data.

[0125] The above-mentioned "sample face shape parameter corresponding to the nth sample voice data" is used to represent the face shape (for example, face length, fatness, thinness, etc.) carried by the video image corresponding to the nth sample voice data.

[0126] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned "sample face parameters corresponding to the nth sample voice data", for example, it may specifically include steps 431-433:

[0127] Step 431: extracting facial key points from the video image corresponding to the nth sample voice data to obtain the actual coordinates of the facial key points corresponding to the nth sample voice data, where n is a positive integer, n≤N.

[0128] It should be noted that the embodiment of the present application does not limit the implementation method of step 431, and any existing or future facial key point extraction method can be used for implementation.

[0129] Step 432: Perform 3D face reconstruction processing on the actual coordinates of the facial key points corresponding to the nth sample voice data according to the open source 3D face model to obtain a 3D face reconstruction model corresponding to the nth sample voice data, where n is a positive integer, n≤N.

[0130] It should be noted that the embodiment of the present application does not limit the implementation method of step 432, and any existing or future three-dimensional face reconstruction method can be used for implementation.

[0131] Step 433: Determine the sample face parameters corresponding to the nth sample voice data according to the three-dimensional face reconstruction model corresponding to the nth sample voice data, where n is a positive integer, n≤N.

[0132] In the embodiment of the present application, after obtaining the three-dimensional face reconstruction model corresponding to the n-th sample voice data (for example, Figure 4After obtaining the three-dimensional face reconstruction model shown in the figure, facial shape parameters (controlling the length, fatness, etc. of the face), facial expression parameters (controlling facial expressions such as opening the mouth, opening and closing the eyes, etc.), and facial posture parameters (controlling facial orientations such as lowering the head and facing sideways, etc.) can be extracted from the reconstruction parameters of the three-dimensional face reconstruction model respectively; and then, based on the facial shape parameters, facial expression parameters and facial posture parameters, the sample facial parameters corresponding to the n-th sample voice data are determined, so that the "sample facial parameters corresponding to the n-th sample voice data" include the facial shape parameters, facial expression parameters and facial posture parameters.

[0133] Based on the relevant content of the above step 43, it can be known that after extracting the video image corresponding to each sample voice data from the sample video, the facial key point extraction process can be performed on the video image corresponding to each sample voice data to obtain the actual coordinates of the facial key points corresponding to each sample voice data; then, based on the actual coordinates of the facial key points corresponding to each sample voice data, the sample face parameters corresponding to each sample voice data can be determined.

[0134] Step 44: Construct a virtual image generation model using the voice features of at least one sample voice data, the sample face parameters corresponding to the at least one sample voice data, and the video image corresponding to the at least one sample voice data.

[0135] It should be noted that the embodiment of the present application does not limit the implementation of step 44. For example, step 44 may specifically include steps 441 to 444:

[0136] Step 441: Determine a virtual image generation image corresponding to the at least one sample voice data based on the voice features of the at least one sample voice data, the sample face parameters corresponding to the at least one sample voice data, and the model to be trained.

[0137] “The virtual image generated image corresponding to the nth sample voice data” refers to the virtual image prediction result of the nth sample voice data, where n is a positive integer, n≤N.

[0138] The above-mentioned "model to be trained" is used to perform virtual image generation processing for the input data of the model to be trained; and the embodiment of the present application does not limit the "model to be trained", for example, it may include a lip expression parameter prediction layer, a parameter fusion layer, a textureless image rendering layer and a texture image generation layer, and the input data of the texture image generation layer includes the output data of the textureless image rendering layer, the input data of the textureless image rendering layer includes the output data of the parameter fusion layer, and the input data of the parameter fusion layer includes the output data of the lip expression parameter prediction layer.

[0139] In addition, the embodiments of the present application do not limit the implementation method of the above-mentioned "texture image generation layer". For example, it can be implemented using a GAN network. It can be seen that the above-mentioned "texture image generation layer" can include a generation network and a judgment network. Among them, the above-mentioned "generation network" is used to perform image data generation processing; the above-mentioned "judgment network" is used to judge whether an image data is a true image.

[0140] In order to facilitate understanding of the working principle of the above-mentioned "model to be trained", the following is an explanation by taking the determination process of "generating an image of a virtual image corresponding to the nth sample voice data" as an example.

[0141] As an example, when the above-mentioned “sample face parameters” include sample face expression parameters, sample face posture parameters, and sample face shape parameters, and the above-mentioned “texture image generation layer” includes a generation network and a judgment network, the process of using the above-mentioned “model to be trained” to determine the “virtual image generation image corresponding to the nth sample voice data” may specifically include steps 51 to 55:

[0142] Step 51: input the speech features of the n-th sample speech data into the lip expression parameter prediction layer to obtain the n-th sample lip expression parameter output by the lip expression parameter prediction layer.

[0143] The above-mentioned "lip expression parameter prediction layer" is used to perform lip expression parameter prediction processing on the input data of the lip expression parameter prediction layer; and the embodiment of the present application does not limit the network structure of the "lip expression parameter prediction layer", for example, it can be similar to the network structure of the above-mentioned "lip expression parameter prediction module".

[0144] The above-mentioned “nth sample lip shape expression parameter” is used to represent the facial expression related to the lip shape that the person is predicted to present when the nth sample voice data is uttered.

[0145] Step 52: Input the nth sample lip expression parameter and the sample facial expression parameter corresponding to the nth sample voice data into the parameter fusion layer to obtain the nth expression fusion parameter output by the parameter fusion layer.

[0146] The above-mentioned “parameter fusion layer” is used to perform fusion processing on the input data of the parameter fusion layer; and the embodiment of the present application does not limit the network structure of the “parameter fusion layer”, for example, it can be similar to the network structure of the above-mentioned “parameter fusion module”.

[0147] The above-mentioned "nth expression fusion parameter" is used to represent the facial expression that the character is predicted to present when the voice data to be processed is emitted (for example, facial expressions related to the mouth shape, and facial expressions related to other parts except the mouth shape, etc.).

[0148] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned "nth expression fusion parameter", for example, it is similar to the determination process of the above-mentioned "expression fusion parameter".

[0149] Step 53: Input the nth expression fusion parameter, the sample face posture parameter corresponding to the nth sample voice data, and the sample face shape parameter corresponding to the nth sample voice data into the textureless image rendering layer to obtain the nth sample textureless image output by the textureless image rendering layer.

[0150] The above-mentioned "textureless image rendering layer" is used to perform textureless image rendering processing on the input data of the textureless image rendering layer; and the embodiment of the present application does not limit the working principle of the "textureless image rendering layer", for example, it is similar to the working principle of the above-mentioned "textureless image rendering module".

[0151] The above-mentioned "nth sample textureless image" refers to a textureless face image generated for the nth sample voice data; and the embodiment of the present application does not limit the determination process of the "nth sample textureless image", for example, the determination process of the "nth sample textureless image" is similar to the determination process of the "textureless image to be used" above. For ease of understanding, the following is explained in conjunction with an example.

[0152] As an example, the process of determining the “nth sample textureless image” may specifically include steps 531 to 533:

[0153] Step 531: Determine the transformation matrix corresponding to the nth sample speech data according to the sample face posture parameters corresponding to the nth sample speech data.

[0154] Step 532: Determine the three-dimensional face model corresponding to the nth sample voice data according to the conversion matrix corresponding to the nth sample voice data, the nth expression fusion parameter, and the sample face shape parameter corresponding to the nth sample voice data.

[0155] Step 533: Determine the nth sample texture-free image according to the three-dimensional face model corresponding to the nth sample speech data.

[0156] It should be noted that the implementation methods of step 531 to step 533 are similar to the implementation methods of step 31 to step 33 described above.

[0157] Based on the relevant content of the above step 53, it can be known that for the model to be trained, after obtaining the nth expression fusion parameter, the textureless image rendering layer in the model to be trained can refer to the nth expression fusion parameter, the sample face posture parameter corresponding to the nth sample voice data, and the sample face shape parameter corresponding to the nth sample voice data, and perform textureless image rendering processing to obtain and output the nth sample textureless image.

[0158] Step 54: Input the nth sample textureless image into the generation network to obtain the virtual image generation image corresponding to the nth sample voice data output by the generation network.

[0159] The above-mentioned “generation network” is used to perform image generation processing on the input data of the generation network; and the embodiment of the present application does not limit the “generation network”, for example, it can be implemented using the generation network in GAN.

[0160] Step 55: Input the virtual image generated image corresponding to the n-th sample voice data into the judgment network to obtain the n-th first judgment result output by the judgment network.

[0161] The above-mentioned "judgment network" is used to make a true or false judgment on the input data of the judgment network (that is, to determine whether it is a real image or an image generated by the generation network); and the embodiment of the present application does not limit the "judgment network", for example, it can be implemented using the judgment network in GAN.

[0162] The above-mentioned "nth first judgment result" refers to the truth or falsehood judgment result of the virtual image generated image corresponding to the nth sample voice data, so that the "nth first judgment result" is used to indicate whether the virtual image generated image corresponding to the nth sample voice data is a true image, thereby enabling the "nth first judgment result" to indicate the degree of authenticity (that is, the degree of realism) of the "virtual image generated image corresponding to the nth sample voice data".

[0163] Based on the relevant contents of the above steps 51 to 55, it can be known that for the model to be trained including the lip expression parameter prediction layer, the parameter fusion layer, the textureless image rendering layer and the texture image generation layer, the model to be trained can not only output a virtual image generation result for a voice data, but also output a true or false judgment result of the virtual image generation result, so that the prediction performance of the model to be trained can be determined based on the virtual image generation result and the true or false judgment result.

[0164] In addition, in order to further improve the virtual image generation effect, the present embodiment also provides another possible implementation of step 44 (eg Figure 3), which may specifically include: determining the virtual image generation image corresponding to the nth sample voice data according to the voice feature of the nth sample voice data, the sample face parameter corresponding to the nth sample voice data, the second virtual image reference data corresponding to the nth sample voice data, and the model to be trained. Wherein, n is a positive integer, n≤N, and N is a positive integer.

[0165] The above-mentioned “second virtual image reference data corresponding to the n-th sample voice data” refers to the virtual image display image required to be referenced when performing virtual image generation processing on the n-th sample voice data.

[0166] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned "second virtual image reference data corresponding to the nth sample voice data", for example, it may specifically include step 61-step 62:

[0167] Step 61: if n≥2, an image is generated according to the virtual image corresponding to the n-1th sample voice data, and second virtual image reference data corresponding to the nth sample voice data is determined.

[0168] Example 1: If the above-mentioned "second virtual image reference data" includes a frame of image data, step 61 may specifically be: generating an image of the virtual image corresponding to the n-1th sample voice data, and determining it as the second virtual image reference data corresponding to the nth sample voice data.

[0169] It can be seen that for the nth sample voice data, the virtual image generation image corresponding to the previous sample voice data of the nth sample voice data can be referred to, and the virtual image generation processing can be performed on the nth sample voice data to obtain the virtual image generation image corresponding to the nth sample voice data, so that the virtual image generation image can better represent the virtual image that matches the voice data to be processed.

[0170] It should be noted that the playback time of the above-mentioned "previous sample voice data of the nth sample voice data" in the above "sample video" is earlier than the playback time of the nth sample voice data, and the playback time of the "previous sample voice data of the nth sample voice data" in the above "sample video" is closest to the playback time of the nth sample voice data.

[0171] Example 2: When the above-mentioned “second virtual image reference data” includes D frames of image data, and D is a positive integer, and D>2, step 61 may specifically include steps 611-612:

[0172] Step 611: if n>D, determine the second virtual image reference data corresponding to the nth sample voice data according to the virtual image generation image corresponding to the nDth sample voice data to the virtual image generation image corresponding to the n-1th sample voice data.

[0173] It should be noted that the embodiment of the present application does not limit the implementation method of step 611. For example, it may specifically include: collectively processing the virtual image generation image corresponding to the nDth sample voice data to the virtual image generation image corresponding to the n-1th sample voice data to obtain the second virtual image reference data corresponding to the nth sample voice data.

[0174] Step 612: If 2≤n≤D, determine second virtual image reference data corresponding to the nth sample voice data based on at least one preset virtual image generation image and the virtual image generation images corresponding to the first n-1 sample voice data.

[0175] The above-mentioned “preset virtual image generation image” can be pre-set; and the embodiment of the present application does not limit the “preset virtual image generation image”; and the embodiment of the present application does not limit the number of the “preset virtual image generation image”, for example, it can be determined using formula (8).

[0176] Number Preset =D-(n-1)=D-n+1 (8)

[0177] In the formula, Number Preset Indicates the number of the above-mentioned "preset virtual image generation images".

[0178] In addition, the embodiment of the present application does not limit the implementation method of step 612. For example, it may specifically include: collectively processing at least one preset virtual image generation image and the virtual image generation images corresponding to the first n-1 sample voice data to obtain the second virtual image reference data corresponding to the nth sample voice data, so that the second virtual image reference data includes D image data.

[0179] Based on the relevant content of Example 2 above, it can be known that when the above-mentioned "second virtual image reference data" includes D frames of image data, for the nth sample voice data, if the number of image frames corresponding to the nth sample voice data in the above "sample video" is higher than D, then the virtual image generated images corresponding to the first D sample voice data of the nth sample voice data can be determined as the second virtual image reference data corresponding to the nth sample voice data; if the number of image frames corresponding to the nth sample voice data in the above sample video is not higher than D, then when determining the "second virtual image reference data corresponding to the nth sample voice data", it is necessary not only to use the virtual image generated images corresponding to all the sample voice data in the "sample video" whose playback time is earlier than the nth sample voice data, but also to use some preset virtual image generated images to complete the image data.

[0180] It should be noted that the playback time of the above-mentioned "the first D sample voice data of the nth sample voice data" in the above "sample video" is earlier than the playback time of the nth sample voice data, and the playback time of the "the first D sample voice data of the nth sample voice data" in the above "sample video" is relatively close to the playback time of the nth sample voice data.

[0181] Based on the relevant content of the above step 61, it can be known that for the nth sample voice data, if there is at least one sample voice data in the above "sample video" whose playback time is earlier than the nth sample voice data, then the image can be generated with reference to the virtual image corresponding to the at least one sample voice data to determine the second virtual image reference data corresponding to the nth sample voice data.

[0182] Step 62: If n=1, the preset virtual image display data is determined as the second virtual image reference data corresponding to the nth sample voice data.

[0183] The above-mentioned "preset virtual image display data" can be preset; and the embodiment of the present application does not limit the "preset virtual image display data". For example, if the above-mentioned "second virtual image reference data" includes a frame of image data, then the "preset virtual image display data" can be the above-mentioned "preset virtual image generated image". For another example, if the above-mentioned "second virtual image reference data" includes D image data, then the "preset virtual image display data" can include D preset virtual image generated images.

[0184] Based on the relevant contents of the above steps 61 to 62, it can be known that for the nth sample voice data, the virtual image generation images corresponding to the first few sample voice data of the nth sample voice data can be referred to to determine the second virtual image reference data corresponding to the nth sample voice data, so that the virtual image generation processing can be performed on the nth sample voice data based on the "second virtual image reference data corresponding to the nth sample voice data" to obtain the virtual image generation image corresponding to the nth sample voice data, which is conducive to improving the continuity of the virtual image in time sequence, thereby helping to improve the virtual image generation effect.

[0185] In addition, the embodiment of the present application does not limit the implementation method of the above step of "determining the virtual image generation image corresponding to the nth sample voice data according to the voice features of the nth sample voice data, the sample face parameters corresponding to the nth sample voice data, the second virtual image reference data corresponding to the nth sample voice data, and the model to be trained", for example, it may specifically include steps 71 to 76:

[0186] Step 71: input the speech features of the n-th sample speech data into the lip expression parameter prediction layer to obtain the n-th sample lip expression parameter output by the lip expression parameter prediction layer.

[0187] Step 72: Input the nth sample lip expression parameter and the sample facial expression parameter corresponding to the nth sample voice data into the parameter fusion layer to obtain the nth expression fusion parameter output by the parameter fusion layer.

[0188] Step 73: Input the nth expression fusion parameter, the sample face posture parameter corresponding to the nth sample voice data, and the sample face shape parameter corresponding to the nth sample voice data into the textureless image rendering layer to obtain the nth sample textureless image output by the textureless image rendering layer.

[0189] It should be noted that for the relevant contents of step 71 to step 73, please refer to the relevant contents of step 51 to step 53 respectively.

[0190] Step 74: Add the nth sample textureless image and the second virtual image reference data corresponding to the nth sample voice data to obtain the nth fused image.

[0191] It should be noted that the embodiment of the present application does not limit the implementation of step 74. For example, it may be similar to any implementation of step 241 above.

[0192] Step 75: Input the nth fused image into the generation network to obtain a virtual image generation image corresponding to the nth sample voice data output by the generation network.

[0193] It should be noted that the embodiment of the present application does not limit the implementation of step 75. For example, it can be similar to any implementation of step 54 above.

[0194] Step 76: Input the virtual image generated image corresponding to the n-th sample voice data into the judgment network to obtain the n-th first judgment result output by the judgment network.

[0195] It should be noted that for the relevant content of step 76, please refer to the relevant content of step 55.

[0196] Based on the relevant contents of steps 71 to 76 above, it can be known that for the model to be trained, the model to be trained can refer to the voice features of the nth sample voice data, the sample facial parameters corresponding to the nth sample voice data, and the second virtual image reference data corresponding to the nth sample voice data, to perform virtual image generation processing, and obtain and output the virtual image generated image corresponding to the nth sample voice data and its corresponding true or false judgment result, so that the prediction performance of the model to be trained can be determined based on the virtual image generated image and its corresponding true or false judgment result.

[0197] Based on the relevant content of the above step 441, it can be known that after obtaining the voice features of each sample voice data, the model to be trained can be used to perform virtual image generation processing on the voice features of each sample voice data, and obtain and output the virtual image generation image corresponding to each sample voice data and its corresponding true or false judgment result, so that the prediction performance of the model to be trained can be determined based on these virtual image generation images and their corresponding true or false judgment results.

[0198] Step 442: Determine whether the preset stop condition is reached, if so, execute step 444; if not, execute step 443.

[0199] The above-mentioned "preset stop condition" can be set in advance, and the embodiment of the present application does not limit the "preset stop condition". For example, it may specifically include: the model loss value of the model to be trained is lower than the preset loss threshold, the change rate of the model loss value of the model to be trained is lower than the preset change rate threshold, and the number of updates of the model to be trained reaches at least one of the preset number thresholds.

[0200] The above-mentioned “model loss value” is used to represent the prediction performance of the model to be trained; and the embodiment of the present application does not limit the determination process of the “model loss value”. For ease of understanding, it is explained below with examples.

[0201] As an example, assume that the training data set of the model to be trained includes (the first sample voice data, the sample face parameters corresponding to the first sample voice data, and the video image corresponding to the first sample voice data), (the second sample voice data, the sample face parameters corresponding to the second sample voice data, and the video image corresponding to the second sample voice data), ..., and (the Nth sample voice data, the sample face parameters corresponding to the Nth sample voice data, and the video image corresponding to the Nth sample voice data). Based on the above assumptions, it can be known that the determination process of the "model loss value" can specifically include steps 81-83:

[0202] Step 81: Determine an image reconstruction loss value based on an image generated by a virtual image corresponding to the N sample voice data and a video image corresponding to the N sample voice data.

[0203] The above-mentioned “image reconstruction loss value” is used to represent the image reconstruction performance of the model to be trained; and the embodiment of the present application does not limit the determination process of the “image reconstruction loss value”, for example, it may specifically include steps 811-812:

[0204] Step 811: Determine the reconstruction loss value corresponding to the nth sample voice data according to the virtual image generated image corresponding to the nth sample voice data and the video image corresponding to the nth sample voice data, where n is a positive integer, n≤N.

[0205] The above-mentioned “reconstruction loss value corresponding to the nth sample voice data” is used to indicate the difference between the virtual image generated image corresponding to the nth sample voice data and the video image corresponding to the nth sample voice data.

[0206] It should be noted that the embodiment of the present application does not limit the implementation method of step 811. For example, it can be implemented by using any existing or future image reconstruction loss calculation method.

[0207] Step 812: Determine the sum (or average) of the reconstruction loss value corresponding to the first sample speech data, the reconstruction loss value corresponding to the second sample speech data, ..., and the reconstruction loss value corresponding to the Nth sample speech data as the image reconstruction loss value.

[0208] Based on the relevant content of the above step 81, it can be known that for the model to be trained, the image reconstruction loss value of the model to be trained can be determined according to the difference between the image generated by the virtual image corresponding to the N sample voice data and the video image corresponding to the N sample voice data, so that the image reconstruction loss value can accurately represent the image reconstruction performance of the model to be trained.

[0209] Step 82: Determine a first judgment loss value according to the N first judgment results and the image label information corresponding to the N first judgment results.

[0210] The above-mentioned “first judgment loss value” is used for the true and false judgment performance of the model to be trained (especially, the judgment network in the model to be trained) for a single frame image; and the embodiment of the present application does not limit the implementation method of the “first judgment loss value”, for example, it may specifically include steps 821-822:

[0211] Step 821: Determine a first impairment value corresponding to the nth sample speech data according to the nth first judgment result and the image label information corresponding to the nth first judgment result, where n is a positive integer, n≤N.

[0212] The above-mentioned “image label information corresponding to the nth first judgment result” is used to indicate that the virtual image generated image corresponding to the nth sample voice data is fake.

[0213] The above-mentioned “first judgment loss value corresponding to the n-th sample speech data” is used to indicate the difference between the n-th first judgment result and the image label information corresponding to the n-th first judgment result.

[0214] It should be noted that the embodiment of the present application does not limit the implementation method of step 821. For example, it can be implemented by using any existing or future method that can calculate the difference between two data.

[0215] Step 822: Determine the sum (or average) of the first judgment loss value corresponding to the first sample voice data, the first judgment loss value corresponding to the second sample voice data, ..., and the first judgment loss value corresponding to the Nth sample voice data as the first judgment loss value.

[0216] Based on the relevant content of the above step 82, it can be known that for the model to be trained, the first judgment loss value of the model to be trained can be determined according to the difference between the N first judgment results and the image label information corresponding to the N first judgment results, so that the first judgment loss value can represent the true and false judgment performance of the model to be trained (especially the judgment network) for a single frame image.

[0217] Step 83: Determine the model loss value according to the image reconstruction loss value and the first judgment loss value.

[0218] It should be noted that the embodiment of the present application does not limit the implementation method of step 83. For example, it may specifically include: determining the sum of the image reconstruction loss value and the first judgment loss value as the model loss value.

[0219] Based on the relevant contents of steps 81 to 83 above, it can be known that after obtaining the virtual image generation images corresponding to each sample voice data and their corresponding first judgment results, the model loss value can be determined by referring to these virtual image generation images and their corresponding first judgment results, so that the model loss value can represent the prediction performance of the model to be trained.

[0220] In addition, in order to improve the accuracy of the model loss value, the embodiment of the present application also provides two possible implementation methods for determining the above-mentioned "model loss value", which are introduced below respectively.

[0221] In a first possible implementation manner, the process of determining the “model loss value” may further include steps 84 to 86 in addition to the above steps 81 to 82:

[0222] Step 84: Determine the predicted coordinates of the facial key points corresponding to the nth sample speech data, where n is a positive integer, n≤N.

[0223] The above-mentioned “prediction coordinates of facial key points corresponding to the n-th sample speech data” refer to the prediction results of facial key points determined for the n-th sample speech data.

[0224] In addition, the embodiment of the present application does not limit the implementation method of step 84. For example, it may specifically include: performing facial key point extraction processing on the nth sample textureless image to obtain the predicted coordinates of the facial key points corresponding to the nth sample voice data.

[0225] In addition, in order to more accurately obtain the predicted coordinates of facial key points, the embodiment of the present application also provides another possible implementation of step 85, which may specifically include steps 841 and 842:

[0226] Step 841: Determine the three-dimensional coordinates of the key points of the face corresponding to the n-th sample voice data according to the three-dimensional face model corresponding to the n-th sample voice data.

[0227] It should be noted that the above-mentioned “three-dimensional face model corresponding to the n-th sample voice data” can be found in step 532 above.

[0228] It should also be noted that the embodiment of the present application does not limit the implementation method of step 841. For example, it can be implemented using formula (9).

[0229]

[0230] In the formula, represents the three-dimensional coordinates of the key points of the face corresponding to the nth sample speech data; M n represents the transformation matrix corresponding to the nth sample speech data, and the M nis the transformation matrix determined according to the sample face posture parameters corresponding to the nth sample speech data; Indicates the average face of the face model; SP ki_lm The shape basis representing the vertex sequence of facial key points in the 3D face model; α n The sample face shape parameters corresponding to the nth sample speech data; EP ki_lm The expression basis representing the vertex sequence of facial key points in the 3D face model; β n Indicates the nth expression fusion parameter.

[0231] Step 842: Map the three-dimensional coordinates of the facial key points corresponding to the n-th sample voice data to the n-th sample texture-free image to obtain the predicted coordinates of the facial key points corresponding to the n-th sample voice data, so that the predicted coordinates of the facial key points are used to represent the mapping position of the “three-dimensional coordinates of the facial key points corresponding to the n-th sample voice data” in the n-th sample texture-free image.

[0232] It should be noted that the embodiment of the present application does not limit the implementation method of step 842. For example, it can be implemented using formula (10).

[0233]

[0234] Where Y n Indicates the predicted coordinates of the facial key points corresponding to the nth sample speech data; V represents the three-dimensional coordinates of the key points of the face corresponding to the nth sample speech data; 2D () represents a two-dimensional valued function, and Indicates from The coordinate values ​​under the first dimension and the coordinate values ​​under the second dimension are extracted so as to discard the coordinate values ​​under the third dimension.

[0235] Based on the relevant contents of the above step 84, it can be known that for the model to be trained, the nth sample textureless image involved in the textureless image rendering layer of the model to be trained, or the three-dimensional face model corresponding to the nth sample voice data, can be used to determine the predicted coordinates of the facial key points corresponding to the nth sample voice data, so that the facial key point prediction performance of the model to be trained can be determined based on the predicted coordinates of the facial key points corresponding to the nth sample voice data. Wherein, n is a positive integer, n≤N.

[0236] Step 85: Determine the key point prediction loss value according to the predicted coordinates of the facial key points corresponding to the N sample speech data and the actual coordinates of the facial key points corresponding to the N sample speech data.

[0237] The above-mentioned “key point prediction loss value” is used to indicate the face key point prediction performance of the model to be trained; and the embodiment of the present application does not limit the determination process of the “key point prediction loss value”, for example, it may specifically include steps 851-852:

[0238] Step 851: Determine the key point loss value corresponding to the nth sample speech data according to the predicted coordinates of the facial key points corresponding to the nth sample speech data and the actual coordinates of the facial key points corresponding to the nth sample speech data, where n is a positive integer, n≤N.

[0239] The above-mentioned “key point loss value corresponding to the nth sample voice data” is used to indicate the difference between the predicted coordinates of the facial key points corresponding to the nth sample voice data and the actual coordinates of the facial key points corresponding to the nth sample voice data.

[0240] It should be noted that the embodiment of the present application does not limit the implementation method of step 851. For example, it can be implemented by any existing or future method that can calculate the difference between two sets of facial key point coordinates. For another example, it can be implemented by formula (11).

[0241]

[0242] In the formula, Indicates the key point loss value corresponding to the nth sample speech data; Y n Indicates the predicted coordinates of the facial key points corresponding to the nth sample speech data; X n Indicates the actual coordinates of the facial key points corresponding to the nth sample speech data.

[0243] Step 852: Determine the sum (or average) of the key point loss value corresponding to the first sample speech data, the key point loss value corresponding to the second sample speech data, ..., and the key point loss value corresponding to the Nth sample speech data as the key point prediction loss value.

[0244] It should be noted that the embodiment of the present application does not limit the implementation method of step 852. For example, it can be described using formula (12).

[0245]

[0246] Based on the relevant content of step 85 above, it can be known that for the model to be trained, the key point prediction loss value of the model to be trained can be determined by utilizing the difference between the predicted coordinates of the facial key points corresponding to the N sample speech data and the actual coordinates of the facial key points corresponding to the N sample speech data, so that the key point prediction loss value can accurately represent the facial key point prediction performance of the model to be trained.

[0247] Step 86: Determine the model loss value according to the image reconstruction loss value, the first judgment loss value and the key point prediction loss value.

[0248] It should be noted that the embodiment of the present application does not limit the implementation method of step 86. For example, it may specifically include: determining the sum of the image reconstruction loss value, the first judgment loss value and the key point prediction loss value as the model loss value.

[0249] Based on the relevant contents of steps 84 to 86 above, it can be known that for the model loss value of the model to be trained, it not only refers to the virtual image generated image corresponding to each sample voice data and its corresponding first judgment result, but also can refer to the predicted coordinates of the facial key points corresponding to each sample voice data, so that the model loss value can better represent the prediction performance of the model to be trained, which is beneficial to improve the training effect of the model to be trained.

[0250] In a second possible implementation, the process of determining the “model loss value” may further include steps 87 to 90 in addition to the above steps (e.g., steps 81 to 82; or steps 81 to 82, and steps 84 to 85):

[0251] Step 87: splicing the virtual image generated image corresponding to the nth sample voice data and the first virtual image reference data corresponding to the nth sample voice data to obtain the image to be judged. Where n is a positive integer, n≤N.

[0252] The above-mentioned “first virtual image reference data corresponding to the nth sample voice data” refers to the virtual image display image required to measure the judgment performance of the judgment network for continuous frame image data in the model to be trained.

[0253] In addition, the embodiment of the present application does not limit the determination process of the above-mentioned "first virtual image reference data corresponding to the nth sample voice data", for example, it may specifically include steps 101 to 104:

[0254] Step 101: If n>P, the virtual image generated image corresponding to the nPth sample voice data to the virtual image generated image corresponding to the n-1th sample voice data are collectively processed to determine the first virtual image reference data corresponding to the nth sample voice data. Wherein, P is a positive integer, and P>2.

[0255] Step 102: If 2<n≤P, the virtual image generated image corresponding to the 1st sample voice data to the virtual image generated image corresponding to the n-1th sample voice data are collectively processed to determine as the first virtual image reference data corresponding to the nth sample voice data.

[0256] Step 103: If n=2, the virtual image corresponding to the first sample voice data is generated into an image and determined as the first virtual image reference data corresponding to the nth sample voice data.

[0257] Step 104: If n=1, an empty set is determined as the first virtual image reference data corresponding to the nth sample voice data.

[0258] Based on the relevant contents of the above steps 101 to 104, it can be known that for the nth sample voice data, the virtual image corresponding to the previous one or more sample voice data of the nth sample voice data can be generated into an image to determine the first virtual image reference data corresponding to the nth sample voice data.

[0259] In addition, the embodiment of the present application does not limit the implementation method of step 87. For example, any existing or future image stitching data method can be used for implementation.

[0260] Step 88: Input the image to be judged into the judgment network to obtain the nth second judgment result output by the judgment network, where n is a positive integer, n≤N.

[0261] The above-mentioned "nth second judgment result" refers to the true or false judgment result of the image to be judged, so that the "nth second judgment result" is used to indicate whether the image to be judged is a true image, thereby enabling the "nth second judgment result" to indicate the degree of authenticity (that is, the degree of real-worldness) of the above-mentioned "virtual image generated image corresponding to the nth sample voice data, and the first virtual image reference data corresponding to the nth sample voice data".

[0262] Step 89: Determine the second judgment loss value according to the N second judgment results and the image label information corresponding to the N second judgment results.

[0263] The above-mentioned “second judgment loss value” is used for the true and false judgment performance of the model to be trained (especially, the judgment network in the model to be trained) for continuous frame images; and the embodiment of the present application does not limit the implementation method of the “second judgment loss value”, for example, it may specifically include steps 891-892:

[0264] Step 891: Determine the second impairment value corresponding to the nth sample speech data according to the nth second judgment result and the image label information corresponding to the nth second judgment result, where n is a positive integer, n≤N.

[0265] The above-mentioned “image label information corresponding to the nth second judgment result” is used to indicate that the image to be judged is fake.

[0266] The above-mentioned “second judgment loss value corresponding to the n-th sample speech data” is used to indicate the difference between the n-th second judgment result and the image label information corresponding to the n-th second judgment result.

[0267] It should be noted that the embodiment of the present application does not limit the implementation method of step 891. For example, it can be implemented by using any existing or future method that can calculate the difference between two data.

[0268] Step 892: Determine the sum (or average) of the second judgment loss value corresponding to the first sample voice data, the second judgment loss value corresponding to the second sample voice data, ..., and the second judgment loss value corresponding to the Nth sample voice data as the second judgment loss value.

[0269] Based on the relevant content of the above step 89, it can be known that for the model to be trained, the second judgment loss value of the model to be trained can be determined according to the difference between the N second judgment results and the image label information corresponding to the N second judgment results, so that the second judgment loss value can represent the true and false judgment performance of the model to be trained (especially the judgment network) for continuous frame images.

[0270] Step 90: Determine a model loss value according to the image reconstruction loss value, the first judgment loss value and the second judgment loss value.

[0271] It should be noted that the embodiment of the present application does not limit the implementation of step 90. For example, it may specifically include: adding the image reconstruction loss value, the first judgment loss value, and the second judgment loss value to obtain the model loss value. For another example, step 90 may include: adding the image reconstruction loss value, the first judgment loss value, the key point prediction loss value, and the second judgment loss value to obtain the model loss value (as shown in formula (13)).

[0272]

[0273] Where, L model Represents the model loss value; Loss key-points Represents the key point prediction loss value; Loss reconstruction Represents the image reconstruction loss value; represents the first judgment loss value; Represents the second judgment loss value.

[0274] Based on the relevant content of the above step 442, it can be known that for the current round of the model to be trained, it can be determined whether the model to be trained has reached the preset stop condition. If it has reached it, it means that the model to be trained has good prediction performance, so the virtual image generation model can be directly constructed based on the model to be trained; however, if it has not reached it, it means that the prediction performance of the model to be trained is still relatively poor, so the model to be trained can be updated based on the model loss value of the model to be trained so that the updated model to be trained has better prediction performance.

[0275] Step 443: Generate an image based on the virtual image corresponding to at least one sample voice data and the video image corresponding to the at least one sample voice data, update the model to be trained, and return to execute step 441.

[0276] The embodiment of the present application does not limit the updating process of the above-mentioned “model to be trained”. For example, it may specifically include steps 111-112:

[0277] Step 111: Determine a model loss value based on a virtual image generated image corresponding to at least one sample voice data and a video image corresponding to the at least one sample voice data.

[0278] It should be noted that for the process of determining the above-mentioned "model loss value", please refer to the relevant content of step 442 above.

[0279] Step 112: Update the model to be trained according to the model loss value.

[0280] It should be noted that the embodiment of the present application does not limit the implementation method of step 112.

[0281] Based on the relevant content of the above step 443, it can be known that when it is determined that the current round of the model to be trained has not reached the preset stop condition, it can be determined that the prediction performance of the model to be trained is still relatively poor, so the model loss value of the model to be trained can be calculated first; then according to the model loss value, the model to be trained is updated so that the updated model to be trained has better prediction performance, so that step 442 and its subsequent steps can be executed based on the updated model to be trained to realize the next round of training process for the model to be trained.

[0282] Step 444: Determine a virtual image generation model based on the model to be trained.

[0283] As an example, when the above-mentioned "model to be trained" includes a lip expression parameter prediction layer, a parameter fusion layer, a textureless image rendering layer, a generation network, and a judgment network, step 444 may specifically include: determining the virtual image generation model based on the lip expression parameter prediction layer, parameter fusion layer, textureless image rendering layer and generation network in the model to be trained.

[0284] It can be seen that after obtaining the updated model to be trained, the lip expression parameter prediction layer in the model to be trained can be determined as the lip expression parameter prediction module in the virtual image generation model; the parameter fusion layer in the model to be trained can be determined as the parameter fusion module in the virtual image generation model; the textureless image rendering layer in the model to be trained can be determined as the textureless image rendering module in the virtual image generation model; the generation network in the model to be trained can be determined as the texture image generation module in the virtual image generation model.

[0285] Based on the relevant contents of the above steps 41 to 44, it can be known that for the virtual image generation model, the voice data and video images in some existing video data can be used to construct the training data of the virtual image generation model; and then the training data can be used to construct the virtual image generation model, so that the constructed virtual image generation model has better virtual image generation performance. Among them, because the virtual image generation model is implemented using an end-to-end framework, the training process of the virtual image generation model is relatively simple, which is conducive to improving the training effect of the virtual image generation model.

[0286] Method Example 4

[0287] In order to facilitate understanding of the above-mentioned "virtual image generation method", an explanation is given below in conjunction with a scenario.

[0288] As an example, when the virtual image generation method provided in the embodiment of the present application is used to perform virtual image generation processing on a real-time voice stream, the virtual image generation method may specifically include steps 121 to 123:

[0289] Step 121: After acquiring the current frame of voice data, the current frame of voice data is determined as the voice data to be processed.

[0290] It should be noted that for the relevant content of step 121, please refer to the relevant content of "Case 3" in S1 above.

[0291] Step 122: extract speech features from the speech data to be processed to obtain speech features to be used.

[0292] It should be noted that for the relevant content of step 122, please refer to S1 above.

[0293] Step 123: Determine the virtual image display data corresponding to the voice data to be processed based on the voice features to be used, the facial parameters to be used, the virtual image display data corresponding to at least one previous frame of voice data, and the virtual image generation model.

[0294] The acquisition time of the above-mentioned "at least one frame of voice data before" is earlier than the acquisition time of the current frame of voice data; and the time difference between the acquisition time of the "at least one frame of voice data before" and the acquisition time of the current frame of voice data is relatively small.

[0295] In addition, the embodiments of the present application do not limit the above-mentioned "at least one previous frame of voice data". For example, if the above-mentioned "current frame of voice data" is the rth frame of voice data in the real-time voice stream, then the "at least one previous frame of voice data" may include: the r-1th frame of voice data, the r-2th frame of voice data, ..., and the rEth frame of voice data in the real-time voice stream. Wherein, E represents the number of frames of voice data in the above-mentioned "at least one previous frame of voice data".

[0296] The "virtual image display data corresponding to the previous at least one frame of voice data" is obtained by performing a virtual image generation process on the previous at least one frame of voice data; and the "virtual image display data corresponding to the previous at least one frame of voice data" may include a virtual image display image corresponding to the r-1th frame of voice data, a virtual image display image corresponding to the r-2th frame of voice data, ..., and a virtual image display image corresponding to the rEth frame of voice data. Among them, the "virtual image display image corresponding to the reth frame of voice data" is obtained by performing a virtual image generation process on the reth frame of voice data. e is a positive integer, e≤E.

[0297] In addition, the embodiment of the present application does not limit the implementation method of step 123. For example, when the above-mentioned "virtual image generation model" includes a lip expression parameter prediction module, a parameter fusion module, a textureless image rendering module and a texture image generation module, step 123 may specifically include steps 1231-1235:

[0298] Step 1231: input the speech features to be used into the lip expression parameter prediction module to obtain the lip expression parameters to be used output by the lip expression parameter prediction module.

[0299] Step 1232: Input the lip expression parameters to be used and the facial expression parameters to be used into a parameter fusion module to obtain the expression fusion parameters to be used output by the parameter fusion module.

[0300] Step 1233: Input the expression fusion parameters to be used, the facial posture parameters to be used, and the facial shape parameters to be used into the textureless image rendering module to obtain the textureless image to be used output by the textureless image rendering module.

[0301] It should be noted that for the relevant contents of steps 1231 to 1233, please refer to the relevant contents of steps 21 to 23 above respectively.

[0302] Step 1234: Add the texture-free image to be used and the virtual image display data corresponding to at least one previous frame of voice data to obtain a fused image to be used.

[0303] It should be noted that the implementation of step 1234 is similar to the implementation of step 241 above.

[0304] Step 1235: input the fused image to be used into the texture image generation module to obtain the virtual image display data corresponding to the voice data to be processed output by the texture image generation module.

[0305] It should be noted that for the relevant content of step 1235, please refer to the relevant content of step 242 above.

[0306] Based on the relevant contents of the above steps 121 to 123, it can be known that for the current frame voice data in the real-time voice stream, after obtaining the current frame voice data, the virtual image generation model can be used to perform virtual image generation processing on the current frame voice data to obtain the virtual image display data corresponding to the current frame voice data, so that the virtual image display data can represent a virtual image that matches the current frame voice data, so that the virtual image can display the facial state (for example, eye expression, lip expression, etc.) presented by the character when the current frame voice data is issued, thereby achieving the purpose of generating a virtual image in real time.

[0307] Based on the virtual image generation method provided by the above method embodiment, the embodiment of the present application also provides a virtual image generation device, which is explained and illustrated below in conjunction with the accompanying drawings.

[0308] Device Embodiment

[0309] The device embodiment introduces the virtual image generation device, and related content can be found in the above method embodiment.

[0310] See also Figure 5 , which is a structural schematic diagram of a virtual image generating device provided in an embodiment of the present application.

[0311] The virtual image generation device 500 provided in the embodiment of the present application includes:

[0312] The feature extraction unit 501 is used to extract voice features from the voice data to be processed after acquiring the voice data to be processed, so as to obtain the voice features to be used;

[0313] The image generation unit 502 is used to determine the virtual image display data corresponding to the voice data to be processed according to the voice features to be used, the facial parameters to be used, and the pre-built virtual image generation model.

[0314] In a possible implementation manner, the facial parameters to be used include at least one of facial expression parameters to be used, facial posture parameters to be used, and facial shape parameters to be used.

[0315] In a possible implementation manner, the speech data to be processed includes a frame of speech data; the facial parameters to be used include facial expression parameters to be used, facial posture parameters to be used, and facial shape parameters to be used; the virtual image generation model includes a lip expression parameter prediction module, a parameter fusion module, a textureless image rendering module, and a texture image generation module;

[0316] The image generation unit 502 is specifically used to: input the voice features to be used into the lip expression parameter prediction module to obtain the lip expression parameters to be used output by the lip expression parameter prediction module; input the lip expression parameters to be used and the facial expression parameters to be used into the parameter fusion module to obtain the expression fusion parameters to be used output by the parameter fusion module; input the expression fusion parameters to be used, the facial posture parameters to be used, and the facial shape parameters to be used into the textureless image rendering module to obtain the textureless image to be used output by the textureless image rendering module; determine the virtual image display data corresponding to the voice data to be processed according to the textureless image to be used and the texture image generation module.

[0317] In one possible implementation, the process of determining the texture-free image to be used includes: determining the transformation matrix to be used based on the facial posture parameters to be used; determining the three-dimensional face model to be used based on the transformation matrix to be used, the expression fusion parameters to be used, and the face shape parameters to be used; and determining the texture-free image to be used based on the three-dimensional face model to be used.

[0318] In one possible implementation, the process of determining the virtual image display data corresponding to the voice data to be processed includes: determining the virtual image display data corresponding to the voice data to be processed based on the texture-free image to be used, at least one frame of virtual image reference image and the texture image generation module.

[0319] In one possible implementation, the process of determining the virtual image display data corresponding to the voice data to be processed includes: adding the texture-free image to be used and the at least one frame of the virtual image reference image to obtain the fused image to be used; inputting the fused image to be used into the texture image generation module to obtain the virtual image display data corresponding to the voice data to be processed output by the texture image generation module.

[0320] In one possible implementation, the at least one frame of virtual image reference image is generated based on at least one frame of historical voice data corresponding to the voice data to be processed; and the collection time of the at least one frame of historical voice data is earlier than the collection time of the voice data to be processed.

[0321] In a possible implementation manner, the speech data to be processed includes J frames of speech data; the speech features to be used include speech features of the J frames of speech data; wherein J is a positive integer, and J≥2;

[0322] The image generation unit 502 is specifically used to: determine the virtual image display image corresponding to the j-th frame of voice data based on the voice features of the j-th frame of voice data, the facial parameters to be used, and a pre-constructed virtual image generation model; wherein j is a positive integer, j≤J; and determine the virtual image display data corresponding to the voice data to be processed based on the virtual image display image corresponding to the 1st frame of voice data to the virtual image display image corresponding to the J-th frame of voice data.

[0323] In a possible implementation manner, the virtual image generating device 500 further includes:

[0324] The voice acquisition unit is used to determine the current frame voice data as the voice data to be processed after acquiring the current frame voice data.

[0325] In one possible implementation, the image generation unit 502 is specifically used to determine the virtual image display data corresponding to the voice data to be processed based on the voice features to be used, the facial parameters to be used, the virtual image display data corresponding to at least one previous frame of voice data, and the virtual image generation model; wherein the collection time of the at least one previous frame of voice data is earlier than the collection time of the current frame of voice data.

[0326] In a possible implementation manner, the virtual image generating device 500 further includes:

[0327] A sample acquisition unit, used for acquiring a sample video;

[0328] A first determining unit, configured to determine at least one sample voice data and a video image corresponding to the at least one sample voice data according to the sample video;

[0329] A second determining unit, configured to determine sample face parameters corresponding to each sample voice data according to the video image corresponding to each sample voice data;

[0330] The model building unit is used to build the virtual image generation model by using the voice features of the at least one sample voice data, the sample face parameters corresponding to the at least one sample voice data, and the video image corresponding to the at least one sample voice data.

[0331] In a possible implementation manner, the model building unit includes:

[0332] An image prediction unit, configured to determine, based on the speech features of the at least one sample speech data, sample face parameters corresponding to the at least one sample speech data, and a model to be trained, a virtual image generation image corresponding to the at least one sample speech data;

[0333] A model updating unit, configured to update the model to be trained according to the virtual image generation image corresponding to the at least one sample voice data and the video image corresponding to the at least one sample voice data, and return to the image prediction unit to continue to execute the step of determining the virtual image generation image corresponding to the at least one sample voice data according to the voice features of the at least one sample voice data, the sample face parameters corresponding to the at least one sample voice data, and the model to be trained;

[0334] The model determination unit is used to determine the virtual image generation model according to the model to be trained until a preset stop condition is reached.

[0335] In a possible implementation manner, the model to be trained includes a lip expression parameter prediction layer, a parameter fusion layer, a textureless image rendering layer, and a texture image generation layer; wherein the texture image generation layer includes a generation network and a judgment network;

[0336] The model determination unit is specifically used to determine the virtual image generation model based on the lip expression parameter prediction layer, parameter fusion layer, textureless image rendering layer and the generation network in the model to be trained.

[0337] In a possible implementation manner, the sample face parameters include sample face expression parameters, sample face posture parameters, and sample face shape parameters; the number of the sample voice data is N;

[0338] The image prediction unit is specifically used to: input the speech features of the nth sample speech data into the lip expression parameter prediction layer to obtain the nth sample lip expression parameters output by the lip expression parameter prediction layer; wherein n is a positive integer, n≤N, and N is a positive integer; input the nth sample lip expression parameters and the sample facial expression parameters corresponding to the nth sample speech data into the parameter fusion layer to obtain the nth expression fusion parameters output by the parameter fusion layer; input the nth expression fusion parameters, the sample facial posture parameters corresponding to the nth sample speech data, and the sample facial shape parameters corresponding to the nth sample speech data into the textureless image rendering layer to obtain the nth sample textureless image output by the textureless image rendering layer; input the nth sample textureless image into the generation network to obtain the virtual image generation image corresponding to the nth sample speech data output by the generation network; input the virtual image generation image corresponding to the nth sample speech data into the judgment network to obtain the nth first judgment result output by the judgment network.

[0339] In a possible implementation manner, the model updating unit includes:

[0340] A first determination subunit is used to determine an image reconstruction loss value according to a virtual image generated image corresponding to the N sample voice data and a video image corresponding to the N sample voice data;

[0341] A second determination subunit, configured to determine a first judgment loss value according to the N first judgment results and image label information corresponding to the N first judgment results;

[0342] A third determining subunit, configured to determine a model loss value according to the image reconstruction loss value and the first judgment loss value;

[0343] The model updating subunit is used to update the model to be trained according to the model loss value.

[0344] In a possible implementation manner, the model updating unit further includes:

[0345] The fourth determination subunit is used to determine the predicted coordinates of the key points of the face corresponding to the n-th sample voice data; wherein n is a positive integer, n≤N;

[0346] A fifth determining subunit, configured to determine a key point prediction loss value according to the predicted coordinates of the facial key points corresponding to the N sample voice data and the actual coordinates of the facial key points corresponding to the N sample voice data;

[0347] The third determination subunit is specifically used to determine the model loss value according to the image reconstruction loss value, the first judgment loss value and the key point prediction loss value.

[0348] In a possible implementation manner, the model updating unit further includes:

[0349] An image stitching subunit is used to stitch the virtual image generated image corresponding to the n-th sample voice data and the first virtual image reference data corresponding to the n-th sample voice data to obtain an image to be judged; wherein n is a positive integer, n≤N;

[0350] An image recognition subunit, used for inputting the image to be judged into the judgment network to obtain the nth second judgment result output by the judgment network; wherein n is a positive integer, n≤N;

[0351] a sixth determining subunit, configured to determine a second judgment loss value according to the N second judgment results and the image label information corresponding to the N second judgment results;

[0352] The third determination subunit is specifically used to determine the model loss value according to the image reconstruction loss value, the first judgment loss value and the second judgment loss value.

[0353] In one possible implementation, the number of sample voice data is N; the image prediction unit is specifically used to determine the virtual image generation image corresponding to the nth sample voice data based on the voice features of the nth sample voice data, the sample face parameters corresponding to the nth sample voice data, the second virtual image reference data corresponding to the nth sample voice data, and the model to be trained; wherein n is a positive integer, n≤N, and N is a positive integer.

[0354] In one possible implementation, the process of determining the second virtual image reference data corresponding to the nth sample voice data includes: if n≥2, generating an image based on the virtual image corresponding to the n-1th sample voice data, and determining the second virtual image reference data corresponding to the nth sample voice data; if n=1, determining the preset virtual image display data as the second virtual image reference data corresponding to the nth sample voice data.

[0355] In a possible implementation manner, the second virtual image reference data includes D image data; wherein D is a positive integer, and D>2;

[0356] The process of determining the second virtual image reference data corresponding to the nth sample voice data includes: if n>D, determining the second virtual image reference data corresponding to the nth sample voice data based on the virtual image generation images corresponding to the nDth sample voice data to the virtual image generation images corresponding to the n-1th sample voice data; if 2≤n≤D, determining the second virtual image reference data corresponding to the nth sample voice data based on at least one preset virtual image generation image and the virtual image generation images corresponding to the first n-1 sample voice data.

[0357] Furthermore, an embodiment of the present application also provides a device, including: a processor, a memory, and a system bus;

[0358] The processor and the memory are connected via the system bus;

[0359] The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute any implementation method of the above-mentioned virtual image generation method.

[0360] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned virtual image generation method.

[0361] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned virtual image generation method.

[0362] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0363] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0364] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0365] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a virtual image, characterized in that: The method comprises: After acquiring the voice data to be processed, performing voice feature extraction on the voice data to be processed to obtain the voice features to be used; Determining virtual image display data corresponding to the voice data to be processed according to the voice features to be used, the facial parameters to be used, and a pre-built virtual image generation model; The face parameters to be used include face expression parameters to be used, face posture parameters to be used, and face shape parameters to be used; The virtual image generation model includes a lip expression parameter prediction module, a parameter fusion module, a textureless image rendering module and a texture image generation module; The lip expression parameter prediction module is used to: determine the lip expression parameters to be used according to the speech features to be used; The parameter fusion module is used to: determine the expression fusion parameters to be used according to the lip expression parameters to be used and the facial expression parameters to be used; The textureless image rendering module is specifically used to: determine the transformation matrix to be used according to the facial posture parameters to be used; determine the three-dimensional facial model to be used according to the transformation matrix to be used, the expression fusion parameters to be used, and the facial shape parameters to be used; perform image rendering processing on the three-dimensional facial model to be used to obtain the textureless image to be used; perform pixel addition on the textureless image to be used and at least one frame of virtual image reference image to obtain the fused image to be used; input the fused image to be used into the texture image generation module to obtain the virtual image display data corresponding to the voice data to be processed output by the texture image generation module, and the texture image generation module is used to perform image generation processing based on the fused image to be used.

2. The method according to claim 1, characterized in that The at least one frame of virtual image reference image is generated based on at least one frame of historical voice data corresponding to the voice data to be processed; the collection time of the at least one frame of historical voice data is earlier than the collection time of the voice data to be processed.

3. The method according to claim 1, characterized in that The speech data to be processed includes J frames of speech data; the speech features to be used include speech features of the J frames of speech data; wherein J is a positive integer, and J≥2; The process of determining the virtual image display data includes: Determine the virtual image display image corresponding to the j-th frame of voice data according to the voice features of the j-th frame of voice data, the face parameters to be used, and the pre-built virtual image generation model; wherein j is a positive integer, j≤J; According to the virtual image display image corresponding to the 1st frame of voice data to the virtual image display image corresponding to the Jth frame of voice data, the virtual image display data corresponding to the voice data to be processed is determined.

4. The method according to claim 1, characterized in that: The process of acquiring the voice data to be processed includes: After the current frame voice data is acquired, the current frame voice data is determined as the voice data to be processed.

5. The method according to claim 4, characterized in that The step of determining the virtual image display data corresponding to the voice data to be processed according to the voice features to be used, the facial parameters to be used, and the pre-built virtual image generation model includes: The virtual image display data corresponding to the voice data to be processed is determined based on the voice features to be used, the facial parameters to be used, the virtual image display data corresponding to the previous at least one frame of voice data, and the virtual image generation model; wherein the collection time of the previous at least one frame of voice data is earlier than the collection time of the current frame of voice data.

6. The method according to any one of claims 1 to 5, characterized in that: The construction process of the virtual image generation model includes: Get sample video; Determining, according to the sample video, at least one sample voice data and a video image corresponding to the at least one sample voice data; Determine the sample face parameters corresponding to each sample voice data according to the video image corresponding to each sample voice data; The virtual image generation model is constructed using the voice features of the at least one sample voice data, the sample face parameters corresponding to the at least one sample voice data, and the video image corresponding to the at least one sample voice data.

7. A virtual image generating device, characterized in that: include: A feature extraction unit is used to extract voice features from the voice data to be processed after acquiring the voice data to be processed, so as to obtain the voice features to be used; The image generation unit is used to determine the virtual image display data corresponding to the voice data to be processed according to the voice features to be used, the facial parameters to be used, and the pre-constructed virtual image generation model; the facial parameters to be used include facial expression parameters to be used, facial posture parameters to be used, and facial shape parameters to be used; the virtual image generation model includes a lip expression parameter prediction module, a parameter fusion module, a textureless image rendering module and a texture image generation module; the lip expression parameter prediction module is used to determine the lip expression parameters to be used according to the voice features to be used; the parameter fusion module is used to determine the expression fusion parameters to be used according to the lip expression parameters to be used and the facial expression parameters to be used; the textureless image rendering module is specifically used to determine the conversion matrix to be used according to the facial posture parameters to be used; determine the three-dimensional facial model to be used according to the conversion matrix to be used, the expression fusion parameters to be used, and the facial shape parameters to be used; the three-dimensional facial model to be used is subjected to image rendering processing to obtain the textureless image to be used; The texture-free image to be used and at least one frame of virtual image reference image are added pixel by pixel to obtain the fused image to be used; the fused image to be used is input into the texture image generation module to obtain the virtual image display data corresponding to the voice data to be processed output by the texture image generation module, and the texture image generation module is used to perform image generation processing based on the fused image to be used.

8. A virtual image generation device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product is executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images