A method, device, equipment and storage medium for synthesizing virtual characters
By acquiring human driving parameters to construct a 3D model of a virtual character, and synthesizing facial expressions, head postures, and body movements step by step, the problem of insufficient realism in virtual characters is solved, and higher accuracy in virtual character synthesis is achieved.
Patent Information
- Application Number
- CN202110139542.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-02-01
AI Technical Summary
The expressions and movements of virtual characters in existing technologies have low realism and cannot meet user needs.
By acquiring human driving parameters, including image driving parameters and skeletal driving parameters, a human body model and a 3D mesh model of a pre-set image are constructed. The facial expressions, head postures and body movements of the virtual character are synthesized step by step and then fused to improve the accuracy of the synthesis results.
It improves the accuracy and realism of virtual character synthesis results, enhancing the user experience.
Smart Images

Figure CN112967212B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a method, device, equipment and storage medium for synthesizing a virtual character. Background Art
[0002] In the field of artificial intelligence, the demand and application of using virtual characters to generate videos are becoming more and more widespread. The expressions and movements of virtual characters synthesized by traditional methods are less realistic and cannot meet the needs of users.
[0003] Public Content
[0004] The embodiments of the present disclosure provide a method, apparatus, device and storage medium for synthesizing a virtual character, which can improve the accuracy of the synthesis result of the virtual character.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for synthesizing a virtual character, comprising:
[0006] Acquire human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters;
[0007] Acquire a first picture including key point detection results of the preset image according to the two-dimensional picture, and generate a second picture including image synthesis results according to the image driving parameters and the first picture;
[0008] Generating a human body model of the preset image according to the two-dimensional image, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image;
[0009] driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image;
[0010] The second image and the third image are fused to obtain a target image containing a synthesis result of the virtual character.
[0011] In a second aspect, the embodiments of the present disclosure further provide a device for synthesizing a virtual character, including:
[0012] An acquisition module, used for acquiring human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters;
[0013] An image synthesis result generating module, used for obtaining a first image including key point detection results of the preset image according to the two-dimensional image, and generating a second image including image synthesis results according to the image driving parameters and the first image;
[0014] A model generation module, used to generate a human body model of the preset image according to the two-dimensional image, and to generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image;
[0015] A driving module, used for driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image;
[0016] The fusion module is used to fuse the second image and the third image to obtain a target image containing a synthesis result of a virtual character.
[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including:
[0018] one or more processors;
[0019] A memory for storing one or more programs;
[0020] When the one or more programs are executed by the one or more processors, the method for synthesizing a virtual character as described in the first aspect is implemented.
[0021] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for synthesizing a virtual character as described in the first aspect.
[0022] The embodiments of the present disclosure provide a method, apparatus, device and storage medium for synthesizing a virtual character, which comprises obtaining human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; obtaining a first picture containing key point detection results of the preset image according to the two-dimensional picture, and generating a second picture containing image synthesis results according to the image driving parameters and the first picture; generating a human body model of the preset image according to the two-dimensional picture, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture, wherein the three-dimensional mesh model contains clothing information and texture features of the preset image; driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third picture; and fusing the second picture and the third picture to obtain a target picture containing the synthesis result of the virtual character. When constructing a three-dimensional model of a preset image, the above scheme first constructs a human body model of the preset image, and then obtains a three-dimensional mesh model on this basis, thereby improving the accuracy of the model. Moreover, the synthesis of a virtual character is divided into two parts, namely, the image of the virtual character is synthesized using image driving parameters and a first picture containing key point detection results, and the three-dimensional mesh model is driven by bone driving parameters at the same time, and the driven three-dimensional mesh model is rendered to obtain a third picture. Finally, the picture of the image synthesis result and the third picture are fused to obtain the synthesis result of the virtual character, thereby improving the accuracy of the synthesis result. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0024] Figure 1 A flowchart of a method for synthesizing a virtual character provided in the first embodiment of the present disclosure;
[0025] Figure 2 A flowchart of a method for synthesizing a virtual character provided in the second embodiment of the present disclosure;
[0026] Figure 3 A schematic diagram of a process for determining expression parameters and head posture parameters provided in the second embodiment of the present disclosure;
[0027] Figure 4 A flowchart of a method for synthesizing a virtual character provided in Embodiment 3 of the present disclosure;
[0028] Figure 5 A flowchart of a method for synthesizing a virtual character provided in Embodiment 4 of the present disclosure;
[0029] Figure 6 A schematic diagram of a synthesis process of a virtual character provided in the fourth embodiment of the present disclosure;
[0030] Figure 7 A structural diagram of a virtual character synthesis device provided in Embodiment 5 of the present disclosure;
[0031] Figure 8 This is a structural diagram of an electronic device provided in Example 6 of the present disclosure. DETAILED DESCRIPTION
[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0035] It should be noted that the concepts such as “first” and “second” mentioned in the present disclosure are only used to distinguish different objects, and are not used to limit the order or interdependence of the functions performed by these objects.
[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0038] Embodiment 1
[0039] Figure 1This is a flowchart of a method for synthesizing a virtual character provided in the first embodiment of the present disclosure. This embodiment is applicable to the case of synthesizing a virtual character. The method can be executed by a virtual character synthesis device, which can be implemented in software and / or hardware and can be configured in an electronic device with data processing function. Figure 1 As shown, the method may include the following steps:
[0040] S110, obtaining human body driving parameters and a two-dimensional picture containing a preset person.
[0041] Wherein, the human body driving parameters include image driving parameters and skeleton driving parameters. The human body driving parameters are used to drive the currently selected image so that the currently selected image generates expressions and actions corresponding to the human body driving parameters. In this embodiment, the currently selected image can be a preset image in a two-dimensional picture, and the preset image can be a real image or a virtual image. The real image can be an image existing in reality, and the virtual image can be an image that does not exist in reality, for example, it can be a cartoon image or an animation image used in works such as TV series, comics, and games. By driving the preset image in the two-dimensional picture by the human body driving parameters, the preset image in the two-dimensional picture can generate a variety of expressions and actions, which effectively solves the problem that the real image cannot imitate a certain expression and action, especially when the preset image is used to generate a video, a rich expression and action can be displayed to the user, thereby enhancing the user's experience. Optionally, a two-dimensional picture containing a preset image can be obtained from a local picture library, or a two-dimensional picture containing a preset image can be obtained online through a web page, or an image frame containing a preset image can be intercepted from a video to obtain a two-dimensional picture, or it can be obtained by shooting with a camera. The two-dimensional image may include one preset image or multiple preset images. When multiple preset images are included, one preset image may be selected as the preset image to be driven by the present solution according to the requirements.
[0042] The image driving parameters may include expression parameters and head posture parameters. The expression parameters are used to drive the preset image in the two-dimensional image so that it presents an expression corresponding to the expression parameters. For example, if the expression corresponding to the expression parameters is happy, the preset image can be driven to show a happy expression. Optionally, the expression parameters may include position information of key points corresponding to the expression, and the key points may include eyebrows, eyes, nose, mouth, etc. The head posture parameters are used to drive the head posture of the preset image so that it presents a head posture corresponding to the head posture parameters. For example, if the head posture corresponding to the head posture parameters is bowing, the head of the preset image can be driven to present a bowing posture. Optionally, the head posture parameters may include parameters of the rotation angle of the head, for example, parameters such as the pitch angle pitch, the yaw angle yaw, and the roll angle roll of the head. The embodiment does not limit the method for obtaining the image driving parameters, for example, it can be obtained through text or voice signals, through videos, or through expression templates.
[0043] The skeletal drive parameters are used to drive the preset image to present the body movements corresponding to the skeletal drive parameters. The skeletal drive parameters may include the position information of the key skeletal points of the human body, and the key skeletal points may include the key points such as the head, torso, and limbs of the human body. Optionally, the skeletal drive parameters may be determined by text, by video, or by template or preset action sequence.
[0044] S120, obtaining a first image including key point detection results of the preset image according to the two-dimensional image, and generating a second image including image synthesis results according to the image driving parameters and the first image.
[0045] Key point detection is also called key point positioning or image alignment, which refers to locating the positions of key areas of the face, which may include eyebrows, eyes, nose, mouth, facial contours, etc. The embodiment does not limit the specific method of key point detection. For example, key point detection may be performed using a method based on deep learning or a method based on cascaded pose regression (CPR). Taking into account that the head occupies a smaller proportion of the body of the preset image, in order to improve the accuracy of key point detection, the head of the preset image can be captured from the two-dimensional image to obtain a first image, thereby enlarging the head. Detecting the key points in the first image can improve the accuracy of the key point detection results. When performing expression synthesis later, the accuracy of the expression synthesis results can be improved, making the synthesized expressions more realistic.
[0046] The second image is an image containing the synthesis result of expression and head posture. In order to improve the accuracy of the synthesis result of the virtual character, this embodiment synthesizes the body movements of the preset image, and synthesizes the expression and head posture respectively, and then merges the synthesis results of the two to obtain a virtual character. The virtual character is a preset image under the action of human body driving parameters. Optionally, the image driving parameters and the first image can be input into the image synthesis network, and the image synthesis network outputs the second image containing the synthesis result of expression and head posture. The image synthesis network is an end-to-end neural network. This process does not require the construction of a three-dimensional model for the image in the first image, which simplifies the synthesis process. The embodiment does not limit the specific structure of the image synthesis network. For example, it may include an adversarial generation module, which is used to reconstruct the image in the first image based on the input image driving parameters and the first image to obtain the second image.
[0047] S130, generating a human body model of the preset image according to the two-dimensional image, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image.
[0048] Wherein, the three-dimensional mesh model includes the clothing information and texture features of the preset image. When synthesizing body movements, it is usually necessary to construct a three-dimensional model of the preset image, and use the skeleton driving parameters to drive the three-dimensional model to achieve the synthesis of body movements. When constructing the three-dimensional model of the preset image, optionally, the three-dimensional mesh model can be directly constructed based on the preset image, or a human body model of the preset image can be first constructed, and then a three-dimensional mesh model is generated based on the two-dimensional picture. This embodiment takes the latter as an example. The human body model is a three-dimensional model that does not include information such as clothing. Generating the human body model based on the two-dimensional picture can accurately reflect the current movement of the preset image. Generating a three-dimensional mesh model based on the human body model and combining the two-dimensional picture as the three-dimensional model of the preset image improves the accuracy of the three-dimensional model.
[0049] Optionally, the two-dimensional image can be segmented to separate the preset image from the background, and the human body model can be generated according to the body shape and posture of the portrait in the portrait segmentation result. Optionally, the human body model can be generated according to the body shape and posture of the portrait in combination with the Skinned Multi-Person Linear (SMPL model). Then, the human body model is warped (image affine) transformed using the contour of the preset image to generate a grid model containing clothing information. The contour of the preset image is the contour containing clothing information, and the clothing information contained in the grid model is clothing information without color. On this basis, the color of the corresponding grid in the grid model can be adjusted using the texture features of the preset image to obtain a three-dimensional model with the same color as the clothing information of the preset image in the two-dimensional image. The embodiment refers to the three-dimensional model as a three-dimensional grid model. The texture features can include the color of each pixel of the preset image. The clothing information and texture features can be obtained by identifying the preset image in the two-dimensional image, and the embodiment does not limit the specific process of identification.
[0050] S140: driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image.
[0051] After the three-dimensional mesh model is generated, the skeletal driving parameters can be used to drive the three-dimensional mesh model so that it presents the body movements corresponding to the skeletal driving parameters. The driven three-dimensional mesh model is rendered to obtain a two-dimensional image corresponding to the skeletal driving parameters. In the embodiment, the two-dimensional image is referred to as the third image, and the body movements of the preset image in the third image are synthesized body movements. The third image is used to merge with the second image to obtain the final synthesis result of the virtual character.
[0052] S150: Fusing the second picture and the third picture to obtain a target picture including a synthesis result of a virtual character.
[0053] Compared with the traditional method of adjusting the expression, head posture and body movement of the preset image at the same time, this embodiment separates the synthesis of expression and head posture from the synthesis of body movement, and uses the image driving parameters to drive the first picture containing the key point detection results, which can fully reflect the detailed features of the expression and head posture, thereby improving the accuracy of the expression and head posture synthesis results. At the same time, a human body model of the preset image is generated based on the two-dimensional picture, and a three-dimensional grid model of the preset image is generated in combination with the two-dimensional picture, which improves the accuracy of the model. When the three-dimensional grid model is driven by the skeleton driving parameters, the body movement corresponding to the skeleton driving parameters can be accurately reflected. On this basis, the synthesis results of expression and head posture and the synthesis results of body movement are fused as the final synthesis results of the virtual character, which improves the accuracy of the synthesis results and makes the synthesized virtual character more realistic. The embodiment refers to the picture containing the synthesis result of the virtual character as the target picture, and the target picture can be used to generate a video. For example, multiple frames of target pictures can be continuously output to obtain a video.
[0054] Embodiment 1 of the present disclosure provides a method for synthesizing a virtual character, by obtaining human body driving parameters and a two-dimensional picture containing a preset character, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; obtaining a first picture containing key point detection results of the preset image according to the two-dimensional picture, and generating a second picture containing image synthesis results according to the image driving parameters and the first picture; generating a human body model of the preset image according to the two-dimensional picture, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture, wherein the three-dimensional mesh model contains clothing information and texture features of the preset image; driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third picture; and fusing the second picture and the third picture to obtain a target picture containing the synthesis result of the virtual character. When constructing a three-dimensional model of a preset image, the above scheme first constructs a human body model of the preset image, and then obtains a three-dimensional mesh model on this basis, thereby improving the accuracy of the model. Moreover, the synthesis of a virtual character is divided into two parts, namely, using image driving parameters and a first picture containing key point detection results to synthesize the expression and head posture of the virtual character to obtain an image synthesis result, and at the same time using bone driving parameters to drive the three-dimensional mesh model, and rendering the driven three-dimensional mesh model to obtain a third picture, and finally the picture of the image synthesis result and the third picture are fused to obtain the synthesis result of the virtual character, thereby improving the accuracy of the synthesis result.
[0055] Embodiment 2
[0056] Figure 2This is a flow chart of a method for synthesizing a virtual character provided in the second embodiment of the present disclosure. This embodiment takes the generation of human body driving parameters by text as an example to describe the synthesis process of the virtual character. Figure 2 , the method may include the following steps:
[0057] S210, parsing the video text of the video to be generated, and obtaining the role information contained in the video text and the dialogue text corresponding to the role information.
[0058] The video copy is a descriptive text of the video to be generated. The video copy contains role information and the lines corresponding to the role information. The role information may include but is not limited to the number of roles, the actions corresponding to each role, and other information. The video copy can be obtained from the Internet or locally, or selected from the script. Text understanding of the video copy can determine the roles it contains, the lines corresponding to each role, and the actions corresponding to each role.
[0059] S220: For each piece of character information, parse the dialogue text corresponding to the character information to obtain the skeletal drive parameters corresponding to each time point.
[0060] For example, if the dialogue text contains "raise your left hand", the keyword "raise your left hand" can be obtained by parsing the dialogue text. Based on the action, the bone drive parameters corresponding to "raise your left hand" can be obtained.
[0061] S230: Convert the dialogue text corresponding to the character information into an audio signal, and generate first expression parameters and head posture parameters corresponding to each time point according to the voice features of the audio signal.
[0062] Considering that the text lacks expressive information and cannot accurately express the facial expressions and head postures of the characters, the embodiment converts the given text into an audio signal and generates face drive parameters based on the audio signal. Optionally, a TTS (Text to Speech) module can be used to convert the dialogue text into an audio signal. Specifically, the dialogue text can be converted into time-aligned features, such as Mel spectrum or fundamental frequency, and then the time-aligned features can be converted into audio signals using methods such as Wavenet, Waveglow or Transformer.
[0063] The dialogue text of this embodiment takes Chinese text, English text or text containing both Chinese and English as an example, and correspondingly, the audio signal is a Chinese audio signal, an English audio signal or an audio signal containing both Chinese and English. The accuracy of the speech feature directly affects the accuracy of the image driving parameters, and further affects the accuracy of the image synthesis result. Optionally, an end-to-end speech recognition network can be used to determine the speech features of the audio signal, and the end-to-end speech recognition network can include but is not limited to the DeepSpeech network. Taking the extraction of speech features of audio signals through the DeepSpeech network as an example, the dimension of the speech features output by the DeepSpeech network is FT*W*D, where F is the frame rate of the audio signal, for example, it can be 30fps, of course, it can also be other frame rates, T is the sampling duration of the audio signal, W is the size of the time domain window corresponding to the audio signal, and D is the dictionary corresponding to the audio signal. For example, when the audio signal contains English, the corresponding dictionary may include 26 English letters and 3 markers, and the markers may include silence, pause and short pause. When the audio signal contains Chinese, the corresponding dictionary may include 62 phonemes, 5 tone symbols and 3 markers, where the 62 phonemes may include 23 initials and 39 finals, and the 5 tone symbols may include yinping, yangping, shangsheng, qusheng and pingsheng.
[0064] In one example, reference Figure 3 , the audio signal can be input into the speech recognition network DeepSpeech to obtain speech features with dimensions of FT*W*D, and the speech features are input into the pre-trained parameter generation network, which outputs the expression parameters and head posture parameters corresponding to each time point. The structure of the parameter generation network can be set according to the actual situation, Figure 3 The parameter generation network shown takes a convolution module and a deconvolution module as an example. Considering that speech features are time-related features, the convolution module may include a convolution layer and a fully connected layer, wherein the convolution layer is used to perform time convolution on the input speech features to obtain a first feature; the fully connected layer is used to determine the first expression parameters and head posture parameters corresponding to each time point based on the first feature. For example, when the speech features of a single individual are input, the dimension of the speech features input to the time convolution layer is FT*W*D, then the dimension of the first feature output by the time convolution layer is W*D, eliminating the influence of the time factor. The embodiment does not limit the specific execution process of the time convolution layer and the fully connected layer. The deconvolution module corresponds to the convolution module, thereby obtaining the expression parameters and head posture parameters corresponding to each time point. Considering that the expression parameters output by the parameter generation network are the position information of a finite number of key points, in order to obtain the position information of other points and more intuitively obtain the expression and head posture corresponding to each time point, Figure 3The expression parameter and the head posture parameter are used to drive the neutral image, so that the neutral image displays the expression corresponding to the expression parameter and the head posture corresponding to the head posture parameter. Optionally, the driven neutral face can be called a driving mesh, which provides a basis for the synthesis of subsequent images. The neutral image can be an image model with a neutral expression and an undeflected head. Accordingly, the offset / motion amount of each vertex of the driving mesh relative to each vertex of the neutral image can be recorded as the first expression parameter.
[0065] S240. For each of the time points, check whether there is corresponding expression script information at the current time point. If yes, execute S250; otherwise, execute S270.
[0066] It is understandable that the audio signal may not accurately reflect the expression at that time point. In this case, the expression script information can be added to make up for the expression not included in the audio signal. For example, when the expression script information exists at a certain time point, it is considered that there is a special expression at that time point. The special expression can be happy, surprised or angry. The expression script information can be smiling at x seconds, etc.
[0067] S250, parsing the corresponding expression script information, acquiring an expression template corresponding to the corresponding expression script information, and obtaining a second expression parameter.
[0068] When there is expression script information at the current time point, the expression script information can be parsed to determine the expression words corresponding to the expression script information, and a special expression template can be obtained based on the expression words. Optionally, the expression script information can be processed by a natural language processing (NLP) module to identify the expression words contained therein, and a corresponding special expression template can be obtained based on the expression words. The second expression parameter can be the offset / movement of the vertices of the model corresponding to the special expression template relative to the vertices of the model corresponding to the neutral expression template.
[0069] S260: Fuse the first expression parameter and the second expression parameter to obtain an expression driving parameter.
[0070] When there is expression template information at the current time point, the first expression parameter and the second expression parameter can be fused, and the fusion result can be used as the expression driving parameter, which can improve the accuracy of the expression driving parameter, and thus improve the accuracy of the subsequent image synthesis result. Optionally, the time point when the expression corresponding to the expression script information appears and the duration of the corresponding expression can be determined; the weight of the second expression parameter at the current time point is determined according to the current time point, the time point when the expression corresponding to the expression script information appears and the duration of the corresponding expression; the second expression parameter is weighted according to the weight, and the sum of the weighted second expression parameter and the first expression parameter at the current time point is used as the expression driving parameter. Exemplarily, the expression driving parameter at the current time point can be expressed as: Vdisplacement=α*Vexpression*δ+β*Vtalking(t'), where Vdisplacement is the expression driving parameter at the current time point, α and β are coefficients, and the specific values can be set according to the actual situation, for example, they can be set to 0.5 and 0.5 respectively, δ is the weight, δ=1-2*|t'-t| / T, t' is the current time point, t is the time point when the expression corresponding to the expression script information appears, T is the duration of the expression corresponding to the expression script information, t' and t are both time points between the start and end of the duration, Vexpression is the second expression parameter, and Vtalking(t') is the first expression parameter at the current time point. Exemplarily, assuming T=2s, and starting from the first second to the third second, assuming t=2, t'=2, then δ=1, indicating that the expression corresponding to the expression script information changes the most when the time point is 2s.
[0071] S270: Use the first expression parameter corresponding to the current time point as an expression driving parameter.
[0072] When there is no expression script information at the current time point, the first expression parameter can be directly used as an expression driving parameter to drive the first picture.
[0073] S280: Use the expression driving parameters, head posture parameters and skeleton driving parameters as human body driving parameters corresponding to the character information.
[0074] S290, obtaining a two-dimensional picture including a preset image.
[0075] The embodiment does not limit the order of acquiring the human body driving parameters and the two-dimensional image, that is, the human body driving parameters and the two-dimensional image containing the preset image can be acquired at the same time, or the human body driving parameters can be acquired first and then the two-dimensional image containing the preset image, or the two-dimensional image containing the preset image can be acquired first and then the human body driving parameters.
[0076] S2100, obtaining a first image including key point detection results of the preset image according to the two-dimensional image.
[0077] S2110, inputting the image driving parameters and the first image into a pre-trained image synthesis network, and having the pre-trained image synthesis network output a second image containing an image synthesis result.
[0078] Among them, the pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module; the affine transformation module is used to adjust the position of the key point in the first picture according to the image driving parameters to obtain a first image picture; the motion estimation module is used to adjust the position of other pixel points in the first image picture except the key point according to the position change of the key point to obtain a second image picture, and the position change of the key point is the change of the position of the key point in the first image picture relative to the position of the key point in the first picture; the adversarial generation module is used to correct the second image picture to obtain a second picture.
[0079] The image synthesis network is used to drive the first image based on the image driving parameters, and output a picture containing the image synthesis result. The embodiment refers to the picture containing the image synthesis result as the second picture. The whole process does not need to construct a three-dimensional model for the image in the first picture, which simplifies the synthesis process. The structure of the image synthesis network can be set according to actual conditions. In order to improve the accuracy of the synthesis result, the image synthesis network set in this embodiment includes an affine transformation module, a motion estimation module and an adversarial generation module. The affine transformation module is used to adjust the position of the key point in the first picture according to the image driving parameters, so that the position change of the key point can be determined, that is, the change of the position of the key point after adjustment relative to the position of the key point before adjustment. For the convenience of description, the picture output by the affine transformation module can be recorded as the first face picture. The motion estimation module is used to adjust the position of other pixels in the first image picture according to the position change of the key point, obtain a rough expression synthesis result, and record the picture corresponding to the result as the second image picture. The adversarial generation module is used to correct the second image picture output by the motion estimation module to obtain a more accurate image synthesis result. When performing image synthesis, this embodiment first uses the affine transformation module to perform affine transformation on the key points in the first image to obtain the position change of the key points, and then uses the motion estimation module to perform the same transformation on other pixel points to obtain a rough result. The rough result is then corrected by the adversarial generation module, thereby improving the accuracy of the synthesis result and making the synthesized image more realistic, especially when using the synthesized image for live broadcasting or video generation, which can enhance the user experience.
[0080] In one example, the adversarial generation module may include a generator and a discriminator, the generator is used to generate a new image according to the image in the second image picture output by the motion estimation module, and obtain a third image picture, and the discriminator is used to judge the true and false images in the second image picture and the third image picture, that is, to determine which image in the two pictures is the original image and which is the image generated by the generator. The image synthesis network can be trained before application, and when the discriminator cannot determine whether the image generated by the generator is real, the training process ends. Thus, the image driving parameters and the first picture can be input into the trained image synthesis network, and the trained image synthesis network outputs a picture containing the image synthesis result. Subsequently, the synthesized image can be used to replace the real person for live broadcast or batch generation of videos, etc., so as to effectively solve the problem that the number of live content or the number of generated short videos is limited when using real people to broadcast or generate short videos in live broadcast scenes or short video scenes. The embodiment does not limit the type of video, for example, it can be an educational short video, a funny short video, or a talent short video.
[0081] S2120: Generate a human body model of the preset image according to the two-dimensional image, and generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image.
[0082] S2130: Drive the three-dimensional mesh model according to the skeleton driving parameters, and render the driven three-dimensional mesh model to obtain a third image.
[0083] S2140: Fuse the second image and the third image to obtain a target image including a synthesis result of a virtual character.
[0084] S2150: Add a synthesis mark to the target image.
[0085] The synthesis mark is used to indicate that the image in the target image is a synthetic image. The synthesis mark can be displayed in the target image, or it can be hidden by certain technical means. The embodiment does not limit the form of the synthesis mark.
[0086] Embodiment 2 of the present disclosure provides a method for synthesizing a virtual character, which obtains expression driving parameters, head posture parameters and bone driving parameters through text. In the process of obtaining the expression driving parameters, if there is no expression script information at the current time point, the expression parameters obtained through the text can be directly used as the expression driving parameters. If there is expression script information at the current time point, the expression template corresponding to the expression script information can be obtained, and the expression parameters obtained through the text and the expression parameters obtained through the expression template are fused to obtain the expression driving parameters, thereby improving the accuracy of the expression driving parameters and further improving the accuracy of the synthesis result.
[0087] Embodiment 3
[0088] Figure 4 This is a flow chart of a method for synthesizing a virtual character provided in the third embodiment of the present disclosure. This embodiment takes the generation of human body driving parameters by video as an example to describe the synthesis process of the virtual character. Figure 4 , the method may include the following steps:
[0089] S310, collecting a video stream containing a sample image, obtaining an image frame containing the sample image, and determining head posture parameters, key point position information, and skeleton drive parameters of the sample image in the image frame.
[0090] Optionally, a video stream containing a sample image can be captured by an image acquisition device such as a camera to obtain an image frame, and the sample image in the current image frame can be identified to determine the head posture parameters, key point location information, and bone drive parameters of the sample image.
[0091] S320, determining the mode of the sample image according to the position change of the key point within a set time.
[0092] The position change is the position change of the key point at each time point within the set time. The mode includes a speaking mode and an expression mode. Different modes correspond to different position changes of key points. For example, when in the expression mode, the position change of each key point is large, and when in the speaking mode, the position change of each key point is small. Optionally, the position change of each key point within a period of time can be counted, and the mode of the sample image can be determined based on the position change of each key point. For example, if the position change of each key point within the set time is large, it is considered that the sample image is in the expression mode, and if the position change of each key point within the set time is small, it is considered that the sample image is in the speaking mode. Optionally, when determining that the sample image is in the expression mode, the expression can also be classified based on the position of each key point in the expression mode. For example, when the difference between the position of each key point in the mode and the position of each key point of the angry expression is less than a set threshold, it is considered that the sample image is currently in the angry mode.
[0093] S330: Determine an expression base corresponding to the pattern and a weight of the expression base.
[0094] The expression base includes the position information of the key points corresponding to the expression. The expression base can be customized or directly called in advance. Different modes use different expression bases. For example, the expression base corresponding to the speaking mode can include a linear expression base, and the expression base corresponding to the expression mode can include a linear expression base and a non-linear expression base. The linear expression base is a linear decoupling expression base, and the non-linear expression base is an expression base that cannot be decoupled. For example, laughing is a non-linear expression base, which is manifested by opening the mouth, changing the corners of the mouth, and the eyes and eyebrows also changing accordingly.
[0095] Optionally, when the mode is a speaking mode, obtain a first expression base corresponding to the speaking mode; determine the weight of the first expression base based on the position of the key point in the first expression base and the position of the key point in the sample image; when the mode is an expression mode, obtain a second expression base corresponding to the expression mode; determine the weight of the second expression base based on the position of the key point in the second expression base and the position of the key point in the sample image.
[0096] The first expression base may be a linearly decoupled expression base, and the second expression base may be a nonlinearly non-decoupled expression base. Assume that the sample image in the image frame contains m key points, namely b1, b2, ..., bm, and the corresponding positions are B1, B2, ..., Bm. When in the speaking mode, there are n corresponding expression bases, namely A1, A2, ..., An, and the positions of the key points b1, b2, ..., bm in the expression base A1 are A11, A12, ..., A1m, and the positions of the key points b1, b2, ..., bm in the expression base A2 are A21, A22, ..., A2m, and so on. The positions of the key points b1, b2, ..., bm in the expression base An are An1, An2, ..., Anm. Then we can establish the mapping relationship between the expression base and the sample image in the image frame, that is, W1*b1+W2*b1+…+Wn*b1=B1, W1*b2+W2*b2+…+Wn*b2=B2, …, W1*bm+W2*bm+…+Wn*bm=Bm. Solving the above equations, we can get the weights corresponding to each expression base, where W1 is the weight corresponding to the expression base A1, W2 is the weight corresponding to the expression base A2, and so on. Wn is the weight corresponding to the expression base An. When in expression mode, the process of determining the weights of each expression base is similar.
[0097] S340: Determine expression driving parameters of the sample image in the mode according to the expression base and the weight of the expression base.
[0098] Still taking the speaking mode as an example, after the weights are determined, they are substituted into the left side of the above equation to obtain the positions of the key points in the sample image, thereby obtaining the expression driving parameters under this mode.
[0099] S350: Determine the head posture parameters, expression drive parameters and skeleton drive parameters as human body drive parameters.
[0100] S360: Obtain a two-dimensional picture containing a preset image.
[0101] S370, inputting the image driving parameters and the first image into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a second image containing an image synthesis result.
[0102] S380: Generate a human body model of the preset image according to the two-dimensional image, and generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image.
[0103] S390: Drive the three-dimensional mesh model according to the skeleton driving parameters, and render the driven three-dimensional mesh model to obtain a third image.
[0104] S3100: Fuse the second image and the third image to obtain a target image including a synthesis result of a virtual character.
[0105] S3110: Add a synthesis mark to the target image.
[0106] Embodiment 3 of the present disclosure provides a method for synthesizing a virtual character. On the basis of the above embodiments, head posture parameters, expression drive parameters and bone drive parameters can be obtained through video. Moreover, when generating expression drive parameters, the mode of the sample image is determined based on the position changes of key points in the sample image, and then the expression base and weight corresponding to the mode are obtained. The expression drive parameters are obtained based on the expression base and the corresponding weight, so that the generated expression drive parameters are smoother and more realistic, thereby improving the accuracy of the synthesis result.
[0107] Embodiment 4
[0108] Figure 5 This is a flowchart of a method for synthesizing a virtual character provided in the fourth embodiment of the present disclosure. This embodiment optimizes the construction process of the preset image three-dimensional model based on the above embodiment. Figure 5 , the method may include the following steps:
[0109] S410, obtaining human body driving parameters and a two-dimensional picture containing a preset image.
[0110] S420, obtaining a first image including key point detection results of the preset image according to the two-dimensional image, and generating a second image including image synthesis results according to the image driving parameters and the first image.
[0111] S430: Determine body shape parameters and posture parameters of the preset image in the two-dimensional image.
[0112] The body shape parameters are used to describe the morphological features of the preset image, for example, they may include the height, fatness, etc. of the preset image. The posture parameters are used to describe the current action of the preset image. Optionally, the position information of the bone points corresponding to the current action may be used as the posture parameters. Optionally, the preset image in the two-dimensional image may be segmented. For example, the two-dimensional image may be binarized, and the body shape parameters of the preset character may be determined based on the binarized portrait. This may reduce the impact of other factors on the preset image, improve the accuracy of the body shape parameters, and further improve the accuracy of subsequent modeling. Optionally, the preset image in the two-dimensional image may be detected for bone points, and the position information of its key bone points may be determined to obtain the posture parameters.
[0113] S440: Generate a human body model of the preset image according to the body shape parameters and posture parameters.
[0114] Optionally, a human body model of a preset image can be generated in combination with the SMPL model. The SMPL model is a statistical model and describes the human body through two types of statistical parameters. The two types of parameters include the body shape parameters and posture parameters described in this embodiment. The shape and movement of the human body model can be changed by adjusting the body shape parameters and posture parameters of the SMPL model. Using the SMPL model to construct a human body model can accurately reflect the current movement of the preset image.
[0115] S450, determining the outline and texture features of the preset image in the two-dimensional image, and adjusting the human body model based on the outline to obtain a first model including clothing information.
[0116] The preset image in the two-dimensional image includes clothing information. The texture feature may include the color value of each pixel of the preset image. It should be noted that the human body model constructed based on the SMPL model is a model that does not include non-human information, and non-human information may include clothing information. When constructing the human body model, although the body shape parameters of the human body model are taken into consideration, the human body model actually constructed will also have a certain deviation from the real human body model of the preset image. In order to improve the accuracy of the model, the human body model can be warped based on the contour of the preset image to obtain a mesh model that is closer to the preset image body shape and contains clothing information, that is, the first model described in the embodiment. The clothing information contained in the first model is clothing information without color.
[0117] S460: Project the texture features onto the first model to obtain a three-dimensional mesh model.
[0118] Optionally, the color of the corresponding grid in the first model can be updated based on the pixel value of each pixel point in the two-dimensional image, so that the color of the clothing in the updated first model is consistent with the color of the clothing in the two-dimensional image. In the embodiment, the updated first model is referred to as a three-dimensional grid model, or a three-dimensional mesh model.
[0119] S470: reposition key skeleton points in the three-dimensional mesh model according to the skeleton drive parameters, so that the positions of the key skeleton points change, and drive the three-dimensional mesh model to deform.
[0120] It is understandable that the bone points corresponding to the bone drive parameters do not necessarily match the bone points of the three-dimensional mesh model. For example, the leg bones corresponding to the bone drive parameters are longer, and the leg bones corresponding to the three-dimensional mesh model are shorter. In order to adapt the bone drive parameters to the three-dimensional mesh model, the bone drive parameters can be used to perform motion repositioning on the key bone points in the three-dimensional mesh model to adapt the bone drive parameters to the three-dimensional mesh model. Since there is a strong correlation between the bone points of the human body, when the position of the above-mentioned key bone points moves, other bone points will also move accordingly, thereby driving the three-dimensional mesh model to deform, presenting movements corresponding to the bone drive parameters, and completing posture migration.
[0121] S480: Render the driven three-dimensional mesh model to obtain a third image.
[0122] By rendering the three-dimensional mesh model that has completed the posture transfer, a corresponding two-dimensional image can be obtained. The embodiment does not limit the specific rendering process. Optionally, the third image can also be input into the adversarial generation module, and the adversarial generation module adjusts the third image to improve the accuracy and authenticity of the action synthesis result.
[0123] S490: Fuse the second image and the third image to obtain a target image including a synthesis result of a virtual character.
[0124] By fusing the second image containing the synthetic results of facial expressions and head postures and the third image containing the synthetic results of body movements, a synthesized virtual character can be obtained, which can fully reflect the detailed features of facial expressions and head postures, and present accurate movements, thereby improving the accuracy and authenticity of the virtual character synthesis results.
[0125] S4100: Add a synthesis mark to the target image.
[0126] Exemplary, reference Figure 6 , Figure 6A schematic diagram of a synthesis process of a virtual character provided for the fourth embodiment of the present disclosure. First, a two-dimensional image containing a preset image is obtained, and then the head of the preset image is intercepted and its key points are detected to obtain a first image containing the key point detection results. At the same time, the two-dimensional image is segmented to obtain a binary image, and the two-dimensional image is subjected to skeleton point detection to obtain a fourth image containing the skeleton point detection results. On this basis, the expression, head posture and body movement of the preset character are synthesized respectively. When performing the expression and head posture synthesis, Figure 3 The example shows that the driving mesh corresponding to the image driving parameters and the first picture containing the key point detection results are input into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a picture containing the expression and head posture synthesis results, that is, the second picture. The pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module. When performing body movement synthesis, the body shape parameters of the preset image are determined according to the binary image, and the posture parameters of the preset image are determined based on the bone point detection results of the preset image. Then, a human body model of the preset image is constructed in combination with the SMPL model. On this basis, the human body model is further optimized according to the contour and texture features of the preset image to obtain a three-dimensional mesh model. Then, the three-dimensional mesh model is driven by the pre-acquired bone driving parameters to complete the posture migration. The three-dimensional mesh model after the posture migration is rendered to obtain a third picture. The synthesized virtual character can be obtained by fusing the second picture and the third picture, thereby improving the accuracy of the virtual character synthesis result. In order to more intuitively present the position information of the bone points corresponding to the bone driving parameters, Figure 3 The method of using skeleton driving parameters to drive reference skeleton points is added to obtain drivingskeleton, and the 3D mesh model is driven based on the driving skeleton to complete the posture transfer.
[0127] Embodiment 4 of the present disclosure provides a method for synthesizing a virtual character. On the basis of the above embodiments, a human body model is constructed according to the body shape parameters and posture parameters of a preset image. On this basis, the human body model is optimized using the contour and texture features of the preset image to obtain a three-dimensional mesh model, which is used as the three-dimensional model of the preset image. This improves the accuracy of the model and makes the synthesized virtual character more realistic.
[0128] On the basis of the above embodiment, a video may also be generated based on the target image. Accordingly, the method may further include:
[0129] The target images are spliced in time sequence to generate a video, and subtitles corresponding to the preset image audio signal are displayed in the video.
[0130] Optionally, for a certain character, the target image generated for the character can be spliced in chronological order to obtain the character animation or video corresponding to the character. Optionally, similar operations can be performed for each character in the video copy to obtain the corresponding target image, generate the corresponding character animation, and splice the character animations corresponding to each character to obtain the corresponding video. Of course, the character animations corresponding to different characters can also be merged so that multiple characters appear in the same video at the same time. When generating a video, corresponding subtitles can also be generated based on the audio signal and superimposed on the corresponding video for user viewing convenience.
[0131] On the basis of the above embodiments, the skeleton driving parameters may also be generated based on a preset action sequence or action template.
[0132] Embodiment 5
[0133] Figure 7 This is a structural diagram of a virtual character synthesis device provided in Embodiment 5 of the present disclosure. The device can execute the virtual character synthesis method described in the above embodiments. Figure 7 , the device may include:
[0134] An acquisition module 51 is used to acquire human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters;
[0135] An image synthesis result generating module 52 is used to obtain a first image including the preset image key point detection result according to the two-dimensional image, and to generate a second image including the image synthesis result according to the image driving parameter and the first image;
[0136] A model generation module 53, used to generate a human body model of the preset image according to the two-dimensional image, and to generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image;
[0137] A driving module 54, configured to drive the three-dimensional mesh model according to the skeleton driving parameters, and render the driven three-dimensional mesh model to obtain a third image;
[0138] The fusion module 55 is used to fuse the second image and the third image to obtain a target image containing a synthesis result of the virtual character.
[0139] Embodiment 5 of the present disclosure provides a synthesis device for a virtual character, which obtains human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; obtains a first picture containing a key point detection result of the preset image according to the two-dimensional picture, and generates a second picture containing an image synthesis result according to the image driving parameters and the first picture; generates a human body model of the preset image according to the two-dimensional picture, and generates a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture, wherein the three-dimensional mesh model contains clothing information and texture features of the preset image; drives the three-dimensional mesh model according to the skeleton driving parameters, and renders the driven three-dimensional mesh model to obtain a third picture; and fuses the second picture and the third picture to obtain a target picture containing the synthesis result of the virtual character. When constructing a three-dimensional model of a preset image, the above scheme first constructs a human body model of the preset image, and then obtains a three-dimensional mesh model on this basis, thereby improving the accuracy of the model. Moreover, the synthesis of a virtual character is divided into two parts, namely, the image of the virtual character is synthesized using image driving parameters and a first picture containing key point detection results, and the three-dimensional mesh model is driven by bone driving parameters at the same time, and the driven three-dimensional mesh model is rendered to obtain a third picture. Finally, the picture of the image synthesis result and the third picture are fused to obtain the synthesis result of the virtual character, thereby improving the accuracy of the synthesis result.
[0140] Based on the above embodiment, the target image is used to generate a video;
[0141] The acquisition module 51 is used for:
[0142] Parsing the video text of the video to be generated to obtain the role information contained in the video text and the line text corresponding to the role information;
[0143] For each character information, parse the dialogue text corresponding to the character information to obtain the skeletal drive parameters corresponding to each time point;
[0144] Converting the dialogue text corresponding to the character information into an audio signal, and generating first expression parameters and head posture parameters corresponding to each time point according to the voice features of the audio signal;
[0145] For each of the time points, if the corresponding expression script information does not exist at the current time point, the first expression parameter corresponding to the current time point is used as the expression driving parameter; if the corresponding expression script information exists at the current time point, the corresponding expression script information is parsed, and the expression template corresponding to the corresponding expression script information is obtained to obtain the second expression parameter; the first expression parameter and the second expression parameter are fused to obtain the expression driving parameter;
[0146] The expression driving parameters, head posture parameters and skeleton driving parameters are used as human body driving parameters corresponding to the character information.
[0147] Based on the above embodiment, the step of fusing the first expression parameter and the second expression parameter to obtain the expression driving parameter includes:
[0148] Determine the time point when the expression corresponding to the expression script information appears and the duration of the corresponding expression;
[0149] Determine the weight of the second expression parameter at the current time point according to the current time point, the time point at which the expression corresponding to the expression script information appears, and the duration of the corresponding expression, wherein the current time point is the time point between the start and end of the duration;
[0150] The second expression parameter is weighted according to the weight, and the sum of the weighted second expression parameter and the first expression parameter at the current time point is used as the expression driving parameter.
[0151] Based on the above embodiment, the acquisition module 51 includes:
[0152] An image acquisition unit, used for acquiring a video stream containing a sample image, obtaining an image frame containing the sample image, and determining head posture parameters, position information of key points, and skeleton drive parameters of the sample image in the image frame;
[0153] A mode determination unit, used to determine the mode of the sample image according to the position change of the key point within a set time, wherein the position change is the position change of the key point at each time point within the set time;
[0154] A weight determination unit, used to determine an expression base corresponding to the pattern and a weight of the expression base, wherein the expression base includes position information of key points corresponding to the expression;
[0155] An expression driving parameter determination unit, used to determine the expression driving parameters of the sample image in the mode according to the expression base and the weight of the expression base;
[0156] The human body driving parameter determination unit is used to use the head posture parameters, expression driving parameters and skeleton driving parameters as human body driving parameters.
[0157] Based on the above embodiment, the mode includes a speech mode or an expression mode;
[0158] The weight determination unit is specifically used for:
[0159] When the mode is a speaking mode, obtaining a first expression base corresponding to the speaking mode; determining a weight of the first expression base according to the position of the key point in the first expression base and the position of the key point in the sample image;
[0160] When the mode is an expression mode, a second expression base corresponding to the expression mode is obtained; and a weight of the second expression base is determined according to the position of the key point in the second expression base and the position of the key point in the sample image.
[0161] On the basis of the above embodiment, the face synthesis result generating module 52 is specifically used for:
[0162] Inputting the image driving parameters and the first image into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a second image containing an image synthesis result;
[0163] Among them, the pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module; the affine transformation module is used to adjust the position of the key point in the first picture according to the image driving parameters to obtain a first image picture; the motion estimation module is used to adjust the position of other pixel points in the first image picture except the key point according to the position change of the key point to obtain a second image picture, and the position change of the key point is the change of the position of the key point in the first image picture relative to the position of the key point in the first picture; the adversarial generation module is used to correct the second image picture to obtain a second picture.
[0164] Based on the above embodiment, the model generation module 53 is specifically used for:
[0165] Determining body shape parameters and posture parameters of the preset image in the two-dimensional image;
[0166] A human body model of the preset image is generated according to the body shape parameters and posture parameters.
[0167] Based on the above embodiment, the model generation module 53 is specifically used for:
[0168] Determining the contour and texture features of the preset image in the two-dimensional picture, and adjusting the human body model based on the contour to obtain a first model including clothing information, wherein the preset image in the two-dimensional picture includes clothing information;
[0169] The texture features are projected onto the first model to obtain a three-dimensional mesh model.
[0170] Based on the above embodiment, the driving module 54 is specifically used for:
[0171] The key skeleton points in the three-dimensional mesh model are repositioned according to the skeleton driving parameters, so that the positions of the key skeleton points are changed, and the three-dimensional mesh model is deformed.
[0172] Based on the above embodiment, the device may further include:
[0173] The video generation module is used to merge the second image and the third image to obtain a target image containing the synthesis result of the virtual character, splice the target image in time sequence, generate a video, and display subtitles corresponding to the preset image audio signal in the video.
[0174] Based on the above embodiment, the device may further include:
[0175] A synthesis mark generation module is used to add a synthesis mark to the target image after fusing the second image and the third image to obtain a target image containing the synthesis result of the virtual character, wherein the synthesis mark is used to indicate that the image in the target image is a synthesized image.
[0176] The virtual character synthesis device provided in the embodiment of the present disclosure and the virtual character synthesis method provided in the above-mentioned embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above-mentioned embodiment, and this embodiment has the same beneficial effect as executing the virtual character synthesis method.
[0177] Embodiment 6
[0178] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0179] like Figure 8As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0180] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0181] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0182] Embodiment 7
[0183] The computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0184] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0185] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0186] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; obtains a first picture containing a key point detection result of the preset image according to the two-dimensional picture, and generates a second picture containing an image synthesis result according to the image driving parameters and the first picture; generates a human body model of the preset image according to the two-dimensional picture, and generates a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture, wherein the three-dimensional mesh model contains clothing information and texture features of the preset image; drives the three-dimensional mesh model according to the skeleton driving parameters, and renders the driven three-dimensional mesh model to obtain a third picture; and fuses the second picture and the third picture to obtain a target picture containing a virtual character synthesis result.
[0187] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0188] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0189] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the acquisition module may also be described as a "module for acquiring human body driving parameters and a two-dimensional picture containing a preset image".
[0190] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0191] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0192] According to one or more embodiments of the present disclosure, the present disclosure provides a method for synthesizing a virtual character, comprising:
[0193] Acquire human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters;
[0194] Acquire a first picture including key point detection results of the preset image according to the two-dimensional picture, and generate a second picture including image synthesis results according to the image driving parameters and the first picture;
[0195] Generating a human body model of the preset image according to the two-dimensional image, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image;
[0196] driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image;
[0197] The second image and the third image are fused to obtain a target image containing a synthesis result of the virtual character.
[0198] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, the target image is used to generate a video, and the obtaining of human body driving parameters includes:
[0199] Parsing the video text of the video to be generated to obtain the role information contained in the video text and the line text corresponding to the role information;
[0200] For each character information, parse the dialogue text corresponding to the character information to obtain the skeletal drive parameters corresponding to each time point;
[0201] Converting the dialogue text corresponding to the character information into an audio signal, and generating first expression parameters and head posture parameters corresponding to each time point according to the voice features of the audio signal;
[0202] For each of the time points, if the corresponding expression script information does not exist at the current time point, the first expression parameter corresponding to the current time point is used as the expression driving parameter; if the corresponding expression script information exists at the current time point, the corresponding expression script information is parsed, and the expression template corresponding to the corresponding expression script information is obtained to obtain the second expression parameter; the first expression parameter and the second expression parameter are fused to obtain the expression driving parameter;
[0203] The expression driving parameters, head posture parameters and skeleton driving parameters are used as human body driving parameters corresponding to the character information.
[0204] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, the fusing of the first expression parameter and the second expression parameter to obtain the expression driving parameter includes:
[0205] Determine the time point when the expression corresponding to the expression script information appears and the duration of the corresponding expression;
[0206] Determine the weight of the second expression parameter at the current time point according to the current time point, the time point at which the expression corresponding to the expression script information appears, and the duration of the corresponding expression, wherein the current time point is the time point between the start and end of the duration;
[0207] The second expression parameter is weighted according to the weight, and the sum of the weighted second expression parameter and the first expression parameter at the current time point is used as the expression driving parameter.
[0208] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, the step of obtaining human body driving parameters includes:
[0209] Collecting a video stream containing a sample image to obtain an image frame containing the sample image, and determining head posture parameters, key point position information, and skeleton drive parameters of the sample image in the image frame;
[0210] Determine the mode of the sample image according to the position change of the key point within the set time, wherein the position change is the position change of the key point at each time point within the set time;
[0211] Determine an expression base corresponding to the pattern and a weight of the expression base, wherein the expression base includes position information of key points corresponding to the expression;
[0212] Determining expression driving parameters of the sample image in the mode according to the expression base and the weight of the expression base;
[0213] The head posture parameters, expression driving parameters and skeleton driving parameters are used as human body driving parameters.
[0214] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, the mode includes a speech mode or an expression mode;
[0215] The determining of the expression base corresponding to the mode and the weight of the expression base includes:
[0216] When the mode is a speaking mode, obtaining a first expression base corresponding to the speaking mode; determining a weight of the first expression base according to the position of the key point in the first expression base and the position of the key point in the sample image;
[0217] When the mode is an expression mode, a second expression base corresponding to the expression mode is obtained; and a weight of the second expression base is determined according to the position of the key point in the second expression base and the position of the key point in the sample image.
[0218] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, generating a second picture including an image synthesis result according to the image driving parameter and the first picture includes:
[0219] Inputting the image driving parameters and the first image into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a second image containing an image synthesis result;
[0220] Among them, the pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module; the affine transformation module is used to adjust the position of the key point in the first picture according to the image driving parameters to obtain a first image picture; the motion estimation module is used to adjust the position of other pixel points in the first image picture except the key point according to the position change of the key point to obtain a second image picture, and the position change of the key point is the change of the position of the key point in the first image picture relative to the position of the key point in the first picture; the adversarial generation module is used to correct the second image picture to obtain a second picture.
[0221] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, generating a human body model of the preset image according to the two-dimensional image includes:
[0222] Determining body shape parameters and posture parameters of the preset image in the two-dimensional image;
[0223] A human body model of the preset image is generated according to the body shape parameters and posture parameters.
[0224] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture includes:
[0225] Determining the contour and texture features of the preset image in the two-dimensional picture, and adjusting the human body model based on the contour to obtain a first model including clothing information, wherein the preset image in the two-dimensional picture includes clothing information;
[0226] The texture features are projected onto the first model to obtain a three-dimensional mesh model.
[0227] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, the step of driving a three-dimensional mesh model according to the skeleton driving parameters includes:
[0228] The key skeleton points in the three-dimensional mesh model are repositioned according to the skeleton driving parameters, so that the positions of the key skeleton points are changed, and the three-dimensional mesh model is deformed.
[0229] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, after fusing the second picture and the third picture to obtain a target picture containing a synthesis result of the virtual character, the method further includes:
[0230] The target images are spliced in time sequence to generate a video, and subtitles corresponding to the preset image audio signal are displayed in the video.
[0231] According to one or more embodiments of the present disclosure, in the method for synthesizing a virtual character provided by the present disclosure, after fusing the second picture and the third picture to obtain a target picture containing a synthesis result of the virtual character, the method further includes:
[0232] A synthesis mark is added to the target image, and the synthesis mark is used to indicate that the image in the target image is a synthesized image.
[0233] According to one or more embodiments of the present disclosure, the present disclosure provides a synthesis device for a virtual character, including:
[0234] An acquisition module, used for acquiring human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters;
[0235] An image synthesis result generating module, used for obtaining a first image including key point detection results of the preset image according to the two-dimensional image, and generating a second image including image synthesis results according to the image driving parameters and the first image;
[0236] A model generation module, used to generate a human body model of the preset image according to the two-dimensional image, and to generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image;
[0237] A driving module, used for driving the three-dimensional mesh model according to the skeleton driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image;
[0238] The fusion module is used to fuse the second image and the third image to obtain a target image containing a synthesis result of a virtual character.
[0239] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:
[0240] one or more processors;
[0241] A memory for storing one or more programs;
[0242] When the one or more programs are executed by the one or more processors, a method for synthesizing a virtual character as described in any one of the present disclosures is implemented.
[0243] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for synthesizing a virtual character as described in any one of the present disclosures.
[0244] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0245] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0246] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A method for synthesizing a virtual character, It is characterized in that include: Acquire human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; Acquire a first picture including key point detection results of the preset image according to the two-dimensional picture, and generate a second picture including image synthesis results according to the image driving parameters and the first picture; Generating a human body model of the preset image according to the two-dimensional image, and generating a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image; driving the three-dimensional mesh model according to the skeletal driving parameters, and rendering the driven three-dimensional mesh model to obtain a third image, wherein the three-dimensional mesh model presents a body movement corresponding to the skeletal driving parameters; The second picture and the third picture are combined to obtain a target picture including a synthesis result of a virtual character; The step of generating a second image including an image synthesis result according to the image driving parameter and the first image comprises: Inputting the image driving parameters and the first image into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a second image containing an image synthesis result; Among them, the pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module; the affine transformation module is used to adjust the position of the key points in the first picture according to the image driving parameters to obtain a first image picture; the motion estimation module is used to adjust the positions of other pixel points in the first image picture except the key points according to the position change of the key points to obtain a second image picture, and the position change of the key points is the change of the position of the key points in the first image picture relative to the position of the key points in the first picture; the adversarial generation module is used to correct the second image picture to obtain a second picture; the image driving parameters include expression parameters and head posture parameters.
2. The method according to claim 1, It is characterized in that The target image is used to generate a video, and the obtaining of human body driving parameters includes: Parsing the video text of the video to be generated to obtain the role information contained in the video text and the line text corresponding to the role information; For each character information, parse the dialogue text corresponding to the character information to obtain the skeletal drive parameters corresponding to each time point; Converting the dialogue text corresponding to the character information into an audio signal, and generating first expression parameters and head posture parameters corresponding to each time point according to the voice features of the audio signal; For each of the time points, if the corresponding expression script information does not exist at the current time point, the first expression parameter corresponding to the current time point is used as the expression driving parameter; if the corresponding expression script information exists at the current time point, the corresponding expression script information is parsed, and the expression template corresponding to the corresponding expression script information is obtained to obtain the second expression parameter; the first expression parameter and the second expression parameter are fused to obtain the expression driving parameter; The expression driving parameters, head posture parameters and skeleton driving parameters are used as human body driving parameters corresponding to the character information.
3. The method according to claim 2, It is characterized in that The fusing of the first expression parameter and the second expression parameter to obtain the expression driving parameter comprises: Determine the time point when the expression corresponding to the expression script information appears and the duration of the corresponding expression; Determine the weight of the second expression parameter at the current time point according to the current time point, the time point at which the expression corresponding to the expression script information appears, and the duration of the corresponding expression, wherein the current time point is the time point between the start and end of the duration; The second expression parameter is weighted according to the weight, and the sum of the weighted second expression parameter and the first expression parameter at the current time point is used as the expression driving parameter.
4. The method according to claim 1, It is characterized in that The obtaining of human body driving parameters includes: Collecting a video stream containing a sample image to obtain an image frame containing the sample image, and determining head posture parameters, key point position information, and skeleton drive parameters of the sample image in the image frame; Determine the mode of the sample image according to the position change of the key point within the set time, wherein the position change is the position change of the key point at each time point within the set time; Determine an expression base corresponding to the pattern and a weight of the expression base, wherein the expression base includes position information of key points corresponding to the expression; Determining expression driving parameters of the sample image in the mode according to the expression base and the weight of the expression base; The head posture parameters, expression driving parameters and skeleton driving parameters are used as human body driving parameters.
5. The method according to claim 4, It is characterized in that The mode includes a speech mode or an expression mode; The determining of the expression base corresponding to the mode and the weight of the expression base includes: When the mode is a speaking mode, obtaining a first expression base corresponding to the speaking mode; determining a weight of the first expression base according to the position of the key point in the first expression base and the position of the key point in the sample image; When the mode is an expression mode, a second expression base corresponding to the expression mode is obtained; and a weight of the second expression base is determined according to the position of the key point in the second expression base and the position of the key point in the sample image.
6. The method according to claim 1, It is characterized in that The method of generating the human body model of the preset image according to the two-dimensional image comprises: Determining body shape parameters and posture parameters of the preset image in the two-dimensional image; A human body model of the preset image is generated according to the body shape parameters and posture parameters.
7. The method according to claim 1, It is characterized in that The step of generating the three-dimensional mesh model of the preset image according to the human body model and the two-dimensional picture comprises: Determining the contour and texture features of the preset image in the two-dimensional picture, and adjusting the human body model based on the contour to obtain a first model including clothing information, wherein the preset image in the two-dimensional picture includes clothing information; The texture features are projected onto the first model to obtain a three-dimensional mesh model.
8. The method according to claim 1, It is characterized in that The driving of the three-dimensional mesh model according to the skeleton driving parameters comprises: The key skeleton points in the three-dimensional mesh model are repositioned according to the skeleton driving parameters, so that the positions of the key skeleton points are changed, and the three-dimensional mesh model is deformed.
9. The method according to any one of claims 1 to 8, It is characterized in that After fusing the second picture and the third picture to obtain a target picture containing a synthetic result of a virtual character, the method further includes: The target images are spliced in time sequence to generate a video, and subtitles corresponding to the preset image audio signal are displayed in the video.
10. The method according to any one of claims 1 to 8, It is characterized in that After fusing the second picture and the third picture to obtain a target picture containing a synthetic result of a virtual character, the method further includes: A synthesis mark is added to the target image, and the synthesis mark is used to indicate that the image in the target image is a synthesized image.
11. A device for synthesizing a virtual character, It is characterized in that include: An acquisition module, used for acquiring human body driving parameters and a two-dimensional picture containing a preset image, wherein the human body driving parameters include image driving parameters and skeleton driving parameters; An image synthesis result generating module, used for obtaining a first image including key point detection results of the preset image according to the two-dimensional image, and generating a second image including image synthesis results according to the image driving parameters and the first image; A model generation module, used to generate a human body model of the preset image according to the two-dimensional image, and to generate a three-dimensional mesh model of the preset image according to the human body model and the two-dimensional image, wherein the three-dimensional mesh model includes clothing information and texture features of the preset image; A driving module, configured to drive the three-dimensional mesh model according to the skeleton driving parameters, and render the driven three-dimensional mesh model to obtain a third image, wherein the three-dimensional mesh model presents a body movement corresponding to the skeleton driving parameters; A fusion module, used for fusing the second picture and the third picture to obtain a target picture containing a synthesis result of a virtual character; The step of generating a second image including an image synthesis result according to the image driving parameter and the first image comprises: Inputting the image driving parameters and the first image into a pre-trained image synthesis network, and the pre-trained image synthesis network outputs a second image containing an image synthesis result; Among them, the pre-trained image synthesis network includes an affine transformation module, a motion estimation module and an adversarial generation module; the affine transformation module is used to adjust the position of the key points in the first picture according to the image driving parameters to obtain a first image picture; the motion estimation module is used to adjust the positions of other pixel points in the first image picture except the key points according to the position change of the key points to obtain a second image picture, and the position change of the key points is the change of the position of the key points in the first image picture relative to the position of the key points in the first picture; the adversarial generation module is used to correct the second image picture to obtain a second picture; the image driving parameters include expression parameters and head posture parameters.
12. An electronic device, It is characterized in that include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the method for synthesizing a virtual character as described in any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method for synthesizing a virtual character as described in any one of claims 1 to 10 is implemented.
