Virtual digital image driving method and device

By driving the action and lip of virtual digital images by audio data, the problem of expensive equipment and difficulty in synchronous driving in the existing technology is solved, and a simpler, lower-cost and privacy-protected virtual digital image driver is realized, improving the nature and reality Spend.

CN114972589BActive Publication Date: 2024-11-26BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210609632.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-31
Publication Date
2024-11-26
Estimated Expiration
2042-05-31

AI Technical Summary

Technical Problem

The existing virtual digital image driving technology requires expensive equipment and complex solution processes, which are difficult to achieve universal application, and cannot complete half-body movement driving simultaneously, resulting in low naturalness and authenticity.

Method used

By obtaining the audio data of the target object, converting it into stylized machine sounds, and obtaining the pronunciation order, posture coefficient and lip shape coefficient, synchronously driving the movements and lip shapes of the virtual digital image, realizing the movements and body of the virtual digital image by audio data driving the movements and body of the virtual digital image by audio data action.

Benefits of technology

Reduce costs, increase the use scenarios of virtual digital images, protect the privacy of target objects, and improve the naturalness and authenticity of virtual digital images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114972589B_ABST
    Figure CN114972589B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method and device for driving a virtual digital image, and relates to the field of artificial intelligence technology, and in particular to the fields of virtual images and intelligent media. The specific implementation scheme is: obtaining a virtual digital image corresponding to a target object; obtaining a stylized machine sound, and obtaining a target entity word and a pronunciation order of the target entity word in the stylized machine sound; obtaining a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; in the process of playing each target entity word in the pronunciation order, according to the first posture coefficient and the corresponding first lip shape coefficient corresponding to the currently played target entity word, synchronously driving the virtual digital image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of virtual images and intelligent media, and specifically to a method and device for driving a virtual digital image. Background Art

[0002] Virtual digital images have a wide range of industrial applications. The most common application areas include virtual anchors, virtual customer service, virtual assistants, virtual teachers, virtual idols, and other interactive games and entertainment. Taking virtual digital humans as an example, in related technologies, when establishing virtual digital humans, a real-life 3D digital high-precision facial scanning and reconstruction solution based on a camera array is generally adopted. This requires expensive acquisition equipment, high-end computer hardware, and complex and arduous solution processes, making it difficult to be universally applicable. In addition, since the overall process cannot synchronously complete the drive of half-body movements, the overall naturalness and authenticity of digital humans are low. Summary of the invention

[0003] The present disclosure provides a driving method, device, equipment and storage medium for a virtual digital image.

[0004] According to one aspect of the present disclosure, a method for driving a virtual digital image is provided, by obtaining a virtual digital image corresponding to a target object; obtaining a stylized machine sound, and obtaining a target entity word in the stylized machine sound and a pronunciation order corresponding to the target entity word; obtaining a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; and in the process of playing each target entity word in the pronunciation order, synchronously driving the virtual digital image according to the first posture coefficient and the corresponding first lip shape coefficient corresponding to the target entity word currently being played.

[0005] The present invention can drive a virtual digital image to simulate the actions of a target object based on the audio data of the target object, and synchronously complete the movements of the virtual digital image's body and mouth. Compared with related technical implementation methods, the implementation method is simpler and reduces costs. The audio data of the target object is converted into stylized machine sound, which protects the privacy of the target object and increases the usage scenarios of the virtual digital image.

[0006] According to another aspect of the present disclosure, a driving device for a virtual digital image is provided, comprising: a first acquisition module, used to acquire a virtual digital image corresponding to a target object; a second acquisition module, used to acquire a stylized machine sound, and acquire a target entity word in the stylized machine sound and a pronunciation order corresponding to the target entity word; a third acquisition module, used to acquire a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; and a synchronous driving module, used to synchronously drive the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to the target entity word currently being played during the process of playing each target entity word in the pronunciation order.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method of driving a virtual digital image.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method for driving a virtual digital image.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned driving method of the virtual digital image.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 It is a schematic diagram of a method for driving a virtual digital image according to an exemplary embodiment of the present disclosure.

[0013] Figure 2 It is a schematic diagram of a virtual digital image according to an exemplary embodiment of the present disclosure.

[0014] Figure 3 It is a schematic diagram of obtaining a target entity word in a stylized machine sound according to an exemplary embodiment of the present disclosure.

[0015] Figure 4 It is a schematic diagram of obtaining a first posture coefficient and a first lip shape coefficient corresponding to a target entity word according to an exemplary embodiment of the present disclosure.

[0016] FIG. 5( a ) is a schematic diagram of acquiring facial key points of a target object according to an exemplary embodiment of the present disclosure.

[0017] FIG5( b ) is a schematic diagram of acquiring body key points of a target object according to an exemplary embodiment of the present disclosure.

[0018] Figure 6 It is a schematic diagram of synchronously driving a virtual digital image according to a first posture coefficient and a first lip shape coefficient of a target entity word currently being played according to an exemplary embodiment of the present disclosure.

[0019] Figure 7 It is a schematic diagram of detecting the pronunciation duration of each word in the stylized machine sound according to an exemplary embodiment of the present disclosure.

[0020] Figure 8 It is a schematic diagram of adjusting the duration of a target entity word according to an exemplary embodiment of the present disclosure.

[0021] Fig. 9 is a schematic diagram of optimizing the second animation frame according to an exemplary embodiment of the present disclosure.

[0022] Fig.10 It is an overall flow chart of a method for driving a virtual digital image according to an exemplary embodiment of the present disclosure.

[0023] Fig.11 4 is a schematic diagram of a driving device for a virtual digital image according to an exemplary embodiment of the present disclosure.

[0024] Fig.12 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.

[0027] Machine vision technology is an interdisciplinary subject involving artificial intelligence, neurobiology, psychophysics, computer science, image processing, pattern recognition and many other fields. Machine vision mainly uses computers to simulate human visual functions, extract information from the images of objective objects, process and understand them, and finally use them for actual detection, measurement and control. The biggest features of machine vision technology are fast speed, large amount of information and multiple functions.

[0028] Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes, or character sequences. It is different from speaker recognition and speaker verification, which attempt to identify or verify the speaker of the speech rather than the vocabulary content contained in it.

[0029] Figure 1 is an exemplary embodiment of a method for driving a virtual digital image shown in the present disclosure, such as Figure 1 As shown, the driving method of the virtual digital image includes the following steps:

[0030] S101, obtaining a virtual digital image corresponding to a target object.

[0031] Virtual digital images do not exist in the real world, but in the non-physical world. They are created and used by various means such as motion capture, computer graphics, graphics rendering, deep learning, and speech synthesis. They have corresponding human-like appearance features, human performance capabilities, human interaction capabilities, and other human characteristics. They can also be called virtual images, virtual people, digital people, etc. Representative sub-applications include virtual assistants, virtual customer service, virtual idols / anchors, etc.

[0032] In the present disclosure, the target object refers to a user who uses a virtual digital image. For example, Figure 2 is a schematic diagram of a virtual digital image corresponding to the target object in the present disclosure, such as Figure 2 As shown, the virtual digital image proposed in the present disclosure models both the head and the upper body of the target object, so the virtual digital image corresponding to the target object includes the head and the upper body. It is not difficult to understand that Figure 2 This is just for illustration. In fact, the head and whole body of the target object can also be modeled to obtain a complete virtual digital image of the whole body.

[0033] Optionally, the virtual digital image of the target object may have been previously created and saved.

[0034] Optionally, if the target object does not have a corresponding virtual digital image, a video of the target object can be recorded, and based on the recorded video of the target object, a 3D human body parametric model and a 3D face parametric model can be used to reconstruct the target object's half-body posture and face respectively, so as to generate a virtual digital image of the target object.

[0035] S102, obtaining a stylized machine sound, and obtaining a target entity word in the stylized machine sound and a pronunciation order corresponding to the target entity word.

[0036] The audio data of the virtual digital image to be driven of the target object is obtained. In order to protect the privacy of the target object and increase the fun of the virtual digital image, the audio data of the target object can be converted into a stylized machine sound. Optionally, the stylized machine sound can be a lolita sound, an uncle sound, or a voice of a cartoon character.

[0037] After obtaining the stylized machine sound of the target object, a word segmentation operation may be performed on the text data corresponding to the stylized machine sound to obtain the target entity words in the stylized machine sound, and the pronunciation order of each target entity word may be obtained based on the stylized machine sound.

[0038] S103, obtaining a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word.

[0039] In each frame of animation in the animation sequence of the virtual digital image corresponding to the target object, the body posture and lip posture of the virtual digital image have corresponding coefficients. The coefficient corresponding to the body posture of the virtual digital image is called the posture coefficient, and the coefficient corresponding to the lip posture of the virtual digital image is called the blendshape coefficient.

[0040] According to the target entity words in the stylized machine sound obtained above, the first posture coefficient corresponding to each animation frame of the virtual digital image corresponding to each target entity word and the first lip shape coefficient corresponding to each animation frame of the virtual digital image corresponding to the target entity word can be determined. Optionally, when obtaining the first posture coefficient and the first lip shape coefficient corresponding to the target entity word, the preset mapping relationship between the entity word, the posture coefficient and the lip shape coefficient can be queried to obtain the first posture coefficient and the first lip shape coefficient corresponding to each animation frame of the virtual digital image corresponding to each target entity word.

[0041] S104, in the process of playing each target entity word in the pronunciation order, the virtual digital image is synchronously driven according to the first posture coefficient and the first lip shape coefficient corresponding to the target entity word currently being played.

[0042] According to the target entity words in the stylized machine sound obtained above and the pronunciation order of the target entity words, the target entity words are played in the pronunciation order, and based on the first posture coefficient and the first lip coefficient corresponding to each animation frame of the virtual digital image corresponding to the target entity word obtained above, each animation frame is generated, and each animation frame is arranged and played according to the pronunciation order of the target entity words, so that each target entity word in the stylized machine sound is played synchronously with each animation frame of the virtual digital image corresponding to the target entity word, so as to generate a virtual digital image audio and video animation with synchronized sound and picture.

[0043] Preferably, since the pronunciation duration of each target entity word after the audio data of the target object is converted into stylized machine sound is not necessarily exactly the same as the pronunciation duration of the corresponding target entity word in the audio data, the stylized machine sound may not be able to accurately achieve synchronous playback with each frame of animation of the virtual digital image corresponding to the target entity word. In order to achieve synchronization of sound and picture, the animation frames with asynchronous pronunciation duration in the virtual digital image corresponding to the target entity word can be stretched or compressed based on the pronunciation duration of each target entity word in the stylized machine sound, so that each target entity word in the stylized machine sound is played synchronously with each frame of animation of the virtual digital image corresponding to the target entity word, so as to generate a virtual digital image audio and video animation with synchronized sound and picture.

[0044] The disclosed embodiment proposes a method for driving a virtual digital image, by obtaining a virtual digital image corresponding to a target object; obtaining a stylized machine sound, and obtaining a target entity word in the stylized machine sound and the pronunciation order corresponding to the target entity word; obtaining a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; in the process of playing each target entity word in the pronunciation order, according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word, the virtual digital image is synchronously driven. The disclosed method can drive a virtual digital image to simulate the target object to perform actions based on the audio data of the target object, and synchronously complete the actions of the body and lip shape of the virtual digital image, which is simpler than the implementation method of the related technology and reduces the cost. The audio data of the target object is converted into a stylized machine sound, which protects the privacy of the target object and increases the usage scenarios of the virtual digital image.

[0045] Figure 3 is an exemplary embodiment of a method for driving a virtual digital image proposed in the present disclosure, such as Figure 3 As shown, based on the above embodiment, obtaining stylized machine sound and obtaining target entity words in the stylized machine sound includes the following steps:

[0046] S301, recording the audio of a target object to obtain audio data of the target object.

[0047] The audio data of the virtual digital image to be driven corresponding to the target object is obtained. Exemplarily, the audio data corresponding to the target object may be the real-time speech data of the target object or the pre-recorded audio data of the target object.

[0048] S302: Generate stylized machine sound based on the audio data.

[0049] Perform text recognition on the audio data of the target object to obtain text data corresponding to the audio data of the target object. Optionally, when performing text recognition on the candidate audio data, an automatic speech recognition technology (Automatic Speech Recognition, ASR) model can be used to perform text recognition on the audio data of the target object.

[0050] After obtaining the text data corresponding to the audio data of the target object, the text data can be converted into a stylized machine sound generated by a text to speech (TTS) model.

[0051] S303, obtaining text data of the stylized machine sound, and based on the text data, obtaining a target entity word in the stylized machine sound.

[0052] The text data of the stylized machine sound is obtained. Since the stylized machine sound is converted from the audio data of the target object, the text data corresponding to the audio data of the target object is obtained. After obtaining the text data of the stylized machine sound, a word segmentation model is used to perform a word segmentation operation on the text data to effectively segment the entity word range of the stylized machine sound, so as to obtain multiple target entity words obtained after the word segmentation operation. For example, if the text data corresponding to the audio data of the target object is "Welcome to visit my hometown", then after the word segmentation model is used to perform a word segmentation operation on the text data, the multiple target entity words that can be obtained are "welcome", "you", "visit", "me", "of", and "hometown".

[0053] The disclosed embodiment converts the audio data of the target object into stylized machine sound, thereby protecting the privacy of the target object and increasing the usage scenarios of the virtual digital image.

[0054] Figure 4 is an exemplary embodiment of a method for driving a virtual digital image proposed in the present disclosure, such as Figure 4 As shown, based on the above embodiment, obtaining the first posture coefficient corresponding to the target entity word and the first lip shape coefficient corresponding to the target entity word in step S102 includes the following steps:

[0055] S401, obtaining a mapping relationship between candidate entity words, candidate posture coefficients, and candidate lip shape coefficients.

[0056] Each entity word that has appeared in the historical audio and video data or the sampled audio and video data of the target object is taken as a candidate entity word, the posture coefficient of each frame of virtual digital image animation corresponding to each candidate entity word in the historical audio and video data of the target object or the sampled audio and video data is taken as a candidate posture coefficient, and the lip shape coefficient of each frame of virtual digital image animation corresponding to each candidate entity word in the historical audio and video data of the target object or the sampled audio and video data is taken as a candidate lip shape coefficient.

[0057] In order to realize that the movement of the virtual digital image can be driven based on the audio data of the target object, in the embodiment of the present disclosure, a mapping relationship between each candidate entity word and the candidate posture coefficient and the candidate lip shape coefficient is pre-established.

[0058] The process of establishing the mapping relationship is as follows:

[0059] Obtain candidate video data of the target object. The candidate video data may be sampled video data recorded specifically for the target object for building a mapping relationship, or may be historical video data of the target object. In the process of obtaining the candidate video data, the sound information of the target object is synchronously recorded and used as candidate audio data. The target object will be accompanied by the content or emotion of the candidate audio data corresponding to the candidate video data, and the body parts will make corresponding movements, for example, the target object's head will nod, shake its head, etc., or the target object's hands will wave their hands, punch, etc.

[0060] After obtaining the candidate video data of the target object, obtain the body key points, facial key points and candidate video frame sequence of the target object in each frame of the candidate video data. As shown in FIG5 , FIG5(a) is a schematic diagram of obtaining the facial key points of the target object, and FIG5(b) is a schematic diagram of obtaining the body key points of the target object. Based on the body key points and facial key points of the target object in each frame of the candidate video data, the target object is reconstructed in posture and face, and a virtual digital image corresponding to the target object is generated for subsequent calls, and candidate animation frames corresponding to each frame of the candidate video frame are generated. The posture coefficient corresponding to the virtual digital image in each frame of the candidate animation frame is used as the candidate posture coefficient, and the lip shape coefficient corresponding to the virtual digital image in each frame of the candidate animation frame is used as the candidate lip shape coefficient. Optionally, when reconstructing the posture of the target object, a 3D human body parameterized model can be used, and when reconstructing the face of the target object, a 3D human face parameterized model can be used.

[0061] After obtaining the candidate video frames corresponding to the candidate video data, all candidate animation frames are spliced ​​in order of the candidate video frames, and the obtained animation sequence of the target object is used as the candidate animation sequence to determine the candidate posture coefficients and candidate lip shape coefficients of the candidate animation frames in the candidate animation sequence based on the candidate animation sequence.

[0062] For the candidate audio data synchronously recorded in the process of obtaining the candidate video data of the target object, text recognition is performed on the candidate audio data to obtain candidate text data corresponding to the candidate audio data. Preferably, the candidate audio data may include multiple pronunciation syllables. Optionally, when performing text recognition on the candidate audio data, an automatic speech recognition technology (Automatic Speech Recognition, ASR) model may be used to perform text recognition on the candidate audio data.

[0063] After obtaining the candidate text data corresponding to the candidate audio data, the word segmentation model is used to perform word segmentation on the candidate text data to effectively segment the entity word range of the candidate audio data to obtain multiple candidate entity words obtained after the word segmentation operation. According to the candidate posture coefficients and candidate lip shape coefficients corresponding to the candidate entity words and the candidate animation frames after the adjustment of the length, a mapping relationship between the candidate entity words and the candidate posture coefficients and the candidate lip shape coefficients is established for subsequent calls.

[0064] S402, based on the target entity word, query the mapping relationship to obtain the first posture coefficient and the first lip shape coefficient corresponding to the target entity word.

[0065] According to each target entity word in the stylized machine sound of the target object obtained above, the mapping relationship between the candidate entity word and the candidate posture coefficient and the candidate lip shape coefficient is queried, and the candidate posture coefficient corresponding to the queried target entity word is used as the first posture coefficient, and the candidate lip shape coefficient corresponding to the queried target entity word is used as the first lip shape coefficient.

[0066] The disclosed embodiment pre-establishes a mapping relationship between the candidate entity words corresponding to the target object and the candidate posture coefficients and the candidate lip shape coefficients. After obtaining the audio data of the target person, the mapping relationship can be queried directly based on the target entity words in the audio data to obtain the first posture coefficient and the first lip shape coefficient corresponding to the target entity word, so as to obtain each frame of animation of the virtual digital image of the target object. Compared with the related technical implementation method, the implementation method is simpler and reduces the cost.

[0067] Figure 6 is an exemplary embodiment of a method for driving a virtual digital image proposed in the present disclosure, such as Figure 6 As shown, based on the above embodiment, in step S103, according to the first posture coefficient and the first lip shape coefficient corresponding to the target entity word currently being played, the virtual digital image is synchronously driven, including the following steps:

[0068] S601, generating a first animation frame of a virtual digital image according to a first posture coefficient and a first lip shape coefficient corresponding to each target entity word.

[0069] Based on the first body posture coefficient and the corresponding first mouth shape coefficient corresponding to each target entity word in the stylized machine voice of the target object determined above, in combination with the virtual digital image corresponding to the target object, generate multiple animation frames of the virtual digital image corresponding to each target entity word, denoted as the first animation frames.

[0070] S602. Obtain the audio duration of each target entity word in the stylized machine voice.

[0071] Detect the pronunciation duration of each word in the stylized machine voice, and based on the word segmentation result of the text data corresponding to the stylized machine voice obtained above, merge the pronunciation durations of the words belonging to the same target entity word to obtain the audio duration of each target entity word in the stylized machine voice. Figure 7 It is to detect the pronunciation duration of each word in the stylized machine voice. As Figure 7 shown, detect the pronunciation duration of individual words in the stylized machine voice "Hello" to obtain the pronunciation duration and pronunciation range of "Hello", "good", and "ah". Optionally, the vosk (vosk is an open-source ASR tool) tool can be used to detect the pronunciation duration of each word in the candidate audio data.

[0072] S603. For each target entity word, based on the corresponding audio duration, adjust the duration of the first animation frame corresponding to the target entity word to obtain the second animation frame with adjusted duration.

[0073] To prevent the stylized machine voice of the target object from being out of sync with the animation sequence of the generated virtual digital image, for each target entity word, stretch or compress the duration of the first animation frame corresponding to the target entity word with out-of-sync audio and video based on the audio duration of each target entity word, so that the stylized machine voice of the target object is in sync with the animation sequence of the generated virtual digital image. After traversing the duration of all the first animation frames, use the stretched or compressed first animation frames and the first animation frames that do not need to be adjusted as the second animation frames.

[0074] S604. During the process of playing each target entity word in the pronunciation order, splice the second animation frames corresponding to two adjacent target entity words in the pronunciation order to synchronously drive the virtual digital image.

[0075] Play the target entity words in the pronunciation order of the target entity words in the stylized machine voice of the target object, and play and splice the second animation frames corresponding to two adjacent target entity words in the pronunciation order to synchronously drive the virtual digital image.

[0076] In the disclosed embodiment, based on the audio duration of each target entity word in the stylized machine sound, the duration of the first animation frame corresponding to the target entity word is adjusted, thereby avoiding the situation where the virtual digital image is out of sync with the stylized machine sound and picture.

[0077] It is not difficult to understand that Figure 6 In order to achieve the synchronization of the stylized machine sound of the target object and the animation sequence sound and picture of the generated virtual digital image, the exemplary embodiment adopts a method of adjusting the duration of the first animation frame corresponding to the target entity word. In addition to this method, it is conceivable that the duration of each target entity word in the stylized machine sound of the target object can also be adjusted to achieve the synchronization of the stylized machine sound of the target object and the animation sequence sound and picture of the generated virtual digital image. Exemplary, Figure 8 is a schematic diagram of adjusting the duration of a target entity word shown in the present disclosure, such as Figure 8 As shown, the target entity words "you" and "good" are not synchronized with the animation sequence sound and picture of the virtual digital image, so the duration of the target entity words "you" and "good" is adjusted to achieve synchronization between the stylized machine sound of the target object and the animation sequence sound and picture of the generated virtual digital image.

[0078] Furthermore, Fig. 9 is an exemplary embodiment of a method for driving a virtual digital image proposed in the present disclosure, such as Fig. 9 As shown, in order to make the movement of the generated virtual digital image more natural and smooth, based on the above embodiment, the driving method of the virtual digital image further includes the following steps:

[0079] S901, generating a first animation frame of a virtual digital image according to a first posture coefficient and a first lip shape coefficient corresponding to each target entity word.

[0080] Regarding the implementation of step S901, reference may be made to the introduction of the relevant parts in the above embodiment, which will not be elaborated here.

[0081] S902, obtaining the audio duration of each target entity word in the stylized machine audio.

[0082] Regarding the implementation of step S902, reference may be made to the introduction of the relevant parts in the above embodiment, which will not be elaborated here.

[0083] S903: For each target entity word, based on the corresponding audio duration, adjust the duration of the first animation frame corresponding to the target entity word, and obtain a second animation frame after the duration adjustment.

[0084] Regarding the implementation of step S903, reference may be made to the introduction of the relevant parts in the above embodiment, which will not be elaborated here.

[0085] S904: Optimize the first posture coefficient and the first lip shape coefficient corresponding to the second animation frame to obtain an optimized second posture coefficient and an optimized second lip shape coefficient.

[0086] Obtain the adjacent animation frames of any second animation frame, and perform feature extraction on any second animation frame and the adjacent animation frames of the second animation frame to obtain feature information, and determine the second posture coefficient and the corresponding second lip shape coefficient corresponding to the second animation frame based on the feature information, so as to make the movement transition of the virtual digital image more natural and coordinated, and the generalization effect is better.

[0087] As a feasible method, the real posture coefficient and lip shape coefficient in the historical video or sampled video of the target object can be randomly perturbed to generate perturbation data, and the real posture coefficient and lip shape coefficient are used as supervision to train the deep neural network based on the long short-term memory network (LSTM) architecture to obtain the posture animation refinement network and the lip shape animation refinement network. The second animation frames corresponding to the virtual digital image are sequentially input into the posture animation refinement network. The posture animation refinement network can obtain the adjacent animation frames of any second animation frame based on the sequence of the second animation frames, and perform posture feature extraction on any second animation frame and the adjacent animation frame corresponding to the second animation frame to obtain posture feature information, and determine the second posture coefficient of the second animation frame based on the posture feature information; the second animation frames corresponding to the virtual digital image are sequentially input into the lip shape animation refinement network. The lip shape animation refinement network can obtain the adjacent animation frames of any second animation frame based on the sequence of the second animation frames, and perform lip shape feature extraction on any second animation frame and the adjacent animation frame corresponding to the second animation frame to obtain lip shape feature information, and determine the second lip shape coefficient of the second animation frame based on the lip shape feature information.

[0088] S905: Correct the second animation frame according to the second posture coefficient and the second lip coefficient to obtain a corrected second animation frame.

[0089] According to the second posture coefficient and the second lip shape coefficient obtained above, each corresponding second animation frame is corrected to obtain a second animation frame in which the movement of the corrected virtual digital image is more realistic and natural.

[0090] S906: splicing the corrected second animation frames corresponding to two adjacent target entity words according to the pronunciation order.

[0091] Based on the modified second animation frames obtained above, the modified second animation frames corresponding to two adjacent target entity words are spliced ​​according to the pronunciation order of the target entity words.

[0092] The disclosed embodiment optimizes the first posture coefficient and the first lip shape coefficient of the second animation frame corresponding to the virtual digital image. The optimized virtual digital image has more natural and coordinated motion transitions and better generalization effects.

[0093] Fig.10 is a general flow chart of a method for driving a virtual digital image shown in the present disclosure, such as Fig.10 As shown, the driving method of the virtual digital image includes the following steps:

[0094] S1001, obtaining a virtual digital image corresponding to a target object.

[0095] S1002: Record the audio of the target object to obtain audio data of the target object.

[0096] S1003, generating stylized machine sound based on the audio data.

[0097] S1004, obtaining text data of the stylized machine sound, and based on the text data, obtaining a target entity word in the stylized machine sound.

[0098] Regarding the implementation of steps S1002 to S1004, reference may be made to the introduction of the corresponding parts of the above embodiments, which will not be elaborated here.

[0099] S1005, obtaining a mapping relationship between candidate entity words, candidate posture coefficients, and candidate lip shape coefficients.

[0100] S1006, based on the target entity word, query the mapping relationship to obtain the first posture coefficient and the first lip shape coefficient corresponding to each target entity word.

[0101] Regarding the implementation of steps S1005 to S1006, reference may be made to the introduction of the corresponding parts of the above embodiments, which will not be described in detail here.

[0102] S1007, generating a first animation frame of the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to each target entity word.

[0103] S1008, obtaining the audio duration of each target entity word in the stylized machine audio.

[0104] S1009: For each target entity word, based on the corresponding audio duration, adjust the duration of the first animation frame corresponding to the target entity word to obtain a second animation frame after the duration adjustment.

[0105] Regarding the implementation of steps S1007 to S1009, reference may be made to the introduction of the corresponding parts of the above embodiments, which will not be described in detail here.

[0106] S1010: Obtain an adjacent animation frame of any second animation frame, and perform feature extraction on the second animation frame and the adjacent animation frames of the second animation frame to obtain feature information.

[0107] S1011, determining a second posture coefficient and a second lip shape coefficient corresponding to the second animation frame based on the feature information.

[0108] S1012: Correct the second animation frame according to the second posture coefficient and the second lip coefficient to obtain a corrected second animation frame.

[0109] S1013, splicing the corrected second animation frames corresponding to two adjacent target entity words according to the pronunciation order.

[0110] Regarding the implementation of steps S1010 to S1013, reference may be made to the introduction of the corresponding parts of the above embodiments, which will not be elaborated here.

[0111] The disclosed embodiment proposes a method for driving a virtual digital image, by obtaining a virtual digital image corresponding to a target object; obtaining a stylized machine sound, and obtaining a target entity word and the pronunciation order of the target entity word in the stylized machine sound; obtaining a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; in the process of playing each target entity word in the pronunciation order, according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word, the virtual digital image is synchronously driven. The disclosed method can drive a virtual digital image to simulate the target object to perform actions based on the audio data of the target object, which is simpler and reduces the cost compared to the related technical implementation method, and converts the audio data of the target object into a stylized machine sound, protects the privacy of the target object, and increases the use scenarios of the virtual digital image.

[0112] Fig.11 is a schematic diagram of a driving device for a virtual digital image shown in the present disclosure, such as Fig.11 As shown, the driving device 1100 of the virtual digital image includes a first acquisition module 1101, a second acquisition module 1102, a third acquisition module 1103 and a synchronous driving module 1104, wherein:

[0113] The first acquisition module 1101 is used to acquire a virtual digital image corresponding to a target object.

[0114] The second acquisition module 1102 is used to acquire the stylized machine sound, and acquire the target entity words in the stylized machine sound and the pronunciation order corresponding to the target entity words.

[0115] The third acquisition module 1103 is used to acquire a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word.

[0116] The synchronous driving module 1104 is used to synchronously drive the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word during the process of playing each target entity word in the pronunciation order.

[0117] The disclosed embodiment proposes a driving device for a virtual digital image, which obtains a virtual digital image corresponding to a target object; obtains a stylized machine sound, and obtains a target entity word and a pronunciation order of the target entity word in the stylized machine sound; obtains a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; in the process of playing each target entity word in the pronunciation order, the virtual digital image is synchronously driven according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word. The disclosed embodiment can drive a virtual digital image to simulate the target object to perform actions based on the audio data of the target object, and synchronously complete the actions of the body and lip shape of the virtual digital image, which is simpler than the implementation method of the related technology and reduces the cost. The audio data of the target object is converted into a stylized machine sound, which protects the privacy of the target object and increases the usage scenarios of the virtual digital image.

[0118] Furthermore, the third acquisition module 1103 is also used to: obtain the mapping relationship between the candidate entity words, the candidate posture coefficients and the candidate lip shape coefficients; based on the target entity word, query the mapping relationship to obtain the corresponding first posture coefficient and first lip shape coefficient.

[0119] Furthermore, the synchronous driving module 1104 is also used to: generate a first animation frame of the virtual digital image according to the first posture coefficient and the corresponding first lip shape coefficient corresponding to each target entity word; obtain the audio duration of each target entity word in the stylized machine sound; for each target entity word, based on the corresponding audio duration, adjust the duration of the first animation frame corresponding to the target entity word to obtain the second animation frame after the duration adjustment; in the process of playing each target entity word in the pronunciation order, splice the second animation frames corresponding to two adjacent target entity words in the pronunciation order to synchronously drive the virtual digital image.

[0120] Furthermore, the synchronous driving module 1104 is also used to: optimize the first posture coefficient and the corresponding first lip shape coefficient corresponding to the second animation frame to obtain an optimized second posture coefficient and an optimized second lip shape coefficient; correct the second animation frame according to the second posture coefficient and the second lip shape coefficient to obtain a corrected second animation frame; and splice the corrected second animation frames corresponding to two adjacent target entity words according to the pronunciation order.

[0121] Furthermore, the synchronous driving module 1104 is also used to: obtain the adjacent animation frames of any second animation frame, and perform feature extraction on the second animation frame and the adjacent animation frames of the second animation frame to obtain feature information; based on the feature information, determine the second posture coefficient and the corresponding second lip shape coefficient corresponding to the second animation frame.

[0122] Furthermore, the second acquisition module 1102 is also used to: record the audio of the target object to obtain audio data of the target object; generate stylized machine sound based on the audio data; obtain text data of the stylized machine sound, and obtain target entity words in the stylized machine sound based on the text data.

[0123] Furthermore, the driving device 1100 of the virtual digital image also includes a mapping relationship establishing module 1105, which is used to: obtain candidate video data of the target object, and generate a candidate animation sequence corresponding to the candidate video data based on the candidate video data; obtain candidate posture coefficients and corresponding candidate lip shape coefficients corresponding to the candidate animation frames in the candidate animation sequence; obtain candidate audio data of the target object, and based on the candidate audio data of the target object, obtain multiple candidate entity words contained in the candidate audio data; establish a mapping relationship between the candidate entity words and the candidate posture coefficients and the candidate lip shape coefficients based on the candidate posture coefficients and the candidate lip shape coefficients corresponding to the candidate entity words and the candidate animation frames.

[0124] Furthermore, the mapping relationship establishing module 1105 is also used to: obtain the body key points, facial key points and candidate video frame sequence of the target object in each frame of candidate video data based on the candidate video data; for each frame of candidate video data, reconstruct the body posture and face of the target object based on the body key points of the target object and the facial key points of the target object in the frame of candidate video data, and generate a candidate animation frame corresponding to the frame of candidate video frame; splice all candidate animation frames in the order of candidate video frames to obtain a candidate animation sequence of the target object.

[0125] Furthermore, the mapping relationship establishing module 1105 is also used to: perform text recognition on the candidate audio data to obtain candidate text data corresponding to the candidate audio data; perform word segmentation operation on the candidate text data to obtain multiple candidate entity words obtained after the word segmentation operation.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0127] Fig.12A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0129] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the driving method of the virtual digital image. For example, in some embodiments, the driving method of the virtual digital image may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the driving method of the virtual digital image described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the driving method of the virtual digital image in any other appropriate manner (eg, by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for driving a virtual digital image, comprising: Obtaining a virtual digital image corresponding to the target object; Obtaining a stylized machine sound, and obtaining a target entity word in the stylized machine sound and a pronunciation order corresponding to the target entity word; Obtaining the mapping relationship between candidate entity words, candidate posture coefficients, and candidate lip shape coefficients; Based on the target entity word, query the mapping relationship to obtain a first posture coefficient and a first lip shape coefficient; In the process of playing each target entity word in the pronunciation order, the virtual digital image is synchronously driven according to the first posture coefficient and the first lip shape coefficient corresponding to the target entity word currently being played; The process of determining the mapping relationship includes: Acquire candidate video data of the target object, and generate a candidate animation sequence corresponding to the candidate video data based on the candidate video data; Obtaining candidate posture coefficients and corresponding candidate lip shape coefficients corresponding to candidate animation frames in the candidate animation sequence; Acquire candidate audio data of the target object, and based on the candidate audio data of the target object, acquire multiple candidate entity words contained in the candidate audio data; Based on the candidate body posture coefficients and the corresponding candidate lip shape coefficients corresponding to the candidate entity words and the candidate animation frames, a mapping relationship among the candidate entity words, the candidate body posture coefficients and the candidate lip shape coefficients is established.

2. The method according to claim 1, wherein: The step of synchronously driving the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word comprises: Generate a first animation frame of the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to each target entity word; Obtaining the audio duration of each target entity word in the stylized machine audio; For each target entity word, based on the corresponding audio duration, the duration of the first animation frame corresponding to the target entity word is adjusted to obtain a second animation frame after the duration adjustment; In the process of playing each target entity word according to the pronunciation order, the second animation frames corresponding to two adjacent target entity words are spliced ​​according to the pronunciation order to synchronously drive the virtual digital image.

3. The method according to claim 1, wherein: Also includes: Optimizing the first posture coefficient and the first lip shape coefficient corresponding to the second animation frame to obtain an optimized second posture coefficient and an optimized second lip shape coefficient; Correcting the second animation frame according to the second posture coefficient and the second lip shape coefficient to obtain a corrected second animation frame; The step of splicing the second animation frames corresponding to two adjacent target entity words according to the pronunciation order includes: The modified second animation frames corresponding to two adjacent target entity words are spliced ​​according to the pronunciation order.

4. The method according to claim 3, wherein: The step of optimizing the first posture coefficient and the first lip shape coefficient corresponding to the second animation frame to obtain an optimized second posture coefficient and an optimized second lip shape coefficient includes: Obtaining adjacent animation frames of any second animation frame, and performing feature extraction on the second animation frame and the adjacent animation frames of the second animation frame to obtain feature information; Based on the feature information, a second posture coefficient and a second lip shape coefficient corresponding to the second animation frame are determined.

5. The method according to claim 1, wherein: The step of obtaining the stylized machine voice and obtaining the target entity word in the stylized machine voice includes: Recording the audio of the target object to obtain audio data of the target object; generating the stylized machine sound based on the audio data; The text data of the stylized machine sound is obtained, and based on the text data, the target entity word in the stylized machine sound is obtained.

6. The method according to claim 1, wherein: The step of generating a candidate animation sequence corresponding to the candidate video data based on the candidate video data includes: Based on the candidate video data, obtaining the body key points, facial key points and candidate video frame sequence of the target object in each frame of the candidate video data; For each frame of candidate video data, based on the body key points of the target object and the facial key points of the target object in the frame of candidate video data, the body posture and facial reconstruction of the target object are performed to generate a candidate animation frame corresponding to the frame of candidate video data; All candidate animation frames are spliced ​​according to the candidate video frame sequence to obtain the candidate animation sequence of the target object.

7. The method according to claim 1, wherein: The step of obtaining a plurality of candidate entity words contained in the candidate audio data based on the target object includes: Performing text recognition on the candidate audio data to obtain candidate text data corresponding to the candidate audio data; A word segmentation operation is performed on the candidate text data to obtain a plurality of candidate entity words obtained after the word segmentation operation.

8. A driving device for a virtual digital image, comprising: A first acquisition module is used to acquire a virtual digital image corresponding to a target object; A second acquisition module is used to acquire a stylized machine sound, and acquire a target entity word in the stylized machine sound and a pronunciation order corresponding to the target entity word; A third acquisition module is used to acquire a first posture coefficient corresponding to the target entity word and a first lip shape coefficient corresponding to the target entity word; A synchronous driving module, used for synchronously driving the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to the currently played target entity word during the process of playing each target entity word according to the pronunciation order; The third acquisition module is also used for: Obtaining the mapping relationship between candidate entity words, candidate posture coefficients, and candidate lip shape coefficients; Based on the target entity word, query the mapping relationship to obtain the first posture coefficient and the first lip shape coefficient; The device further includes a mapping relationship establishing module, wherein the mapping relationship establishing module is used to: Acquire candidate video data of the target object, and generate a candidate animation sequence corresponding to the candidate video data based on the candidate video data; Obtaining candidate posture coefficients and corresponding candidate lip shape coefficients corresponding to candidate animation frames in the candidate animation sequence; Acquire candidate audio data of the target object, and based on the candidate audio data of the target object, acquire multiple candidate entity words contained in the candidate audio data; Based on the candidate body posture coefficients and the candidate lip shape coefficients corresponding to the candidate entity words and the candidate animation frames, a mapping relationship among the candidate entity words, the candidate body posture coefficients and the candidate lip shape coefficients is established.

9. The device according to claim 8, wherein: The synchronous drive module is also used for: Generate a first animation frame of the virtual digital image according to the first posture coefficient and the first lip shape coefficient corresponding to each target entity word; Obtaining the audio duration of each target entity word in the stylized machine audio; For each target entity word, based on the corresponding audio duration, the duration of the first animation frame corresponding to the target entity word is adjusted to obtain a second animation frame after the duration adjustment; In the process of playing each target entity word according to the pronunciation order, the second animation frames corresponding to two adjacent target entity words are spliced ​​according to the pronunciation order to synchronously drive the virtual digital image.

10. The device according to claim 9, wherein: The synchronous drive module is also used for: Optimizing the first posture coefficient and the first lip shape coefficient corresponding to the second animation frame to obtain an optimized second posture coefficient and an optimized second lip shape coefficient; Correcting the second animation frame according to the second posture coefficient and the second lip shape coefficient to obtain a corrected second animation frame; The modified second animation frames corresponding to two adjacent target entity words are spliced ​​according to the pronunciation order.

11. The device according to claim 10, wherein: The synchronous drive module is also used for: Obtaining adjacent animation frames of any second animation frame, and performing feature extraction on the second animation frame and the adjacent animation frames of the second animation frame to obtain feature information; Based on the feature information, a second posture coefficient and a second lip shape coefficient corresponding to the second animation frame are determined.

12. The device according to claim 8, wherein: The second acquisition module is also used for: Recording the audio of the target object to obtain audio data of the target object; generating the stylized machine sound based on the audio data; The text data of the stylized machine sound is obtained, and based on the text data, the target entity word in the stylized machine sound is obtained.

13. The device according to claim 8, wherein: The mapping relationship establishing module is also used for: Based on the candidate video data, obtaining the body key points, facial key points and candidate video frame sequence of the target object in each frame of the candidate video data; For each frame of candidate video data, based on the body key points of the target object and the facial key points of the target object in the frame of candidate video data, the body posture and facial reconstruction of the target object are performed to generate a candidate animation frame corresponding to the frame of candidate video data; All candidate animation frames are spliced ​​according to the candidate video frame sequence to obtain the candidate animation sequence of the target object.

14. The device according to claim 8, wherein: The mapping relationship establishing module is also used for: Performing text recognition on the candidate audio data to obtain candidate text data corresponding to the candidate audio data; A word segmentation operation is performed on the candidate text data to obtain a plurality of candidate entity words obtained after the word segmentation operation.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.