Method for generating dynamic mouth shape of virtual digital human and related equipment

By performing audio recognition and phoneme frame alignment for virtual digital audio and video, combined with brightness tracking technology, the problem of incoordination between the virtual digital mouth shape and the body is solved, and the rapid and accurate generation of virtual digital mouth shape is achieved, which improves its Vivid and fun.

CN114996516BActive Publication Date: 2024-11-15SHANGHAI JITU SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210626179.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-02
Publication Date
2024-11-15
Estimated Expiration
2042-06-02

Smart Images

  • Figure CN114996516B_ABST
    Figure CN114996516B_ABST
Patent Text Reader

Abstract

The present invention provides a method for generating dynamic mouth shapes of virtual digital humans and related equipment, wherein the method comprises: extracting audio from audio and video of virtual digital humans; performing speech recognition on the audio to obtain a pinyin document; performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series; obtaining a mouth shape picture set corresponding to a phoneme set in a phoneme document from a mouth shape database; extracting a video from the audio and video of virtual digital humans; determining a video frame corresponding to each phoneme in the video based on the time series corresponding to each phoneme in the phoneme document in the phoneme set; obtaining a mouth shape picture corresponding to each phoneme from a mouth shape picture set; for each mouth shape picture, identifying the mouth position of the mouth shape picture on the video frame by brightness tracking, and pasting the mouth shape picture into the video frame according to the mouth position. The method of the present invention can quickly and accurately generate dynamic mouth shapes of virtual digital humans, and virtual digital humans are more vivid and interesting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of virtual image generation, and in particular to a method for generating a dynamic mouth shape of a virtual digital human and related equipment. Background Art

[0002] At present, AI synthesized virtual character videos are a new field of application of artificial intelligence technology, including speech synthesis, speech recognition, machine translation, expression recognition, human motion recognition, high-definition image processing and other advanced technologies, realizing customized virtual character solutions. Virtual characters generated by AI can be used in many scenarios of interaction with people, such as news broadcasting, classroom education, elderly care, human-computer interaction, etc. In the video industry, different character images can be customized to make the video interaction content more vivid and interesting. When making virtual digital people, the existing technology is prone to technical problems such as the virtual digital person's mouth shape and voice do not correspond, and the virtual digital person's mouth shape and the virtual digital person's body are not coordinated, which makes it difficult to meet the needs of users.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] The main purpose of the present invention is to provide a virtual digital human mouth shape generation scheme which generates a virtual digital human mouth shape quickly and accurately, and the virtual digital human mouth shape and the virtual digital human body are more coordinated.

[0005] A first aspect of the present invention provides a method for generating a dynamic mouth shape of a virtual digital human, the method comprising:

[0006] Extract audio from virtual digital human audio and video;

[0007] Performing speech recognition on the audio to obtain a pinyin document;

[0008] Performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series;

[0009] Acquire a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database;

[0010] Extracting video from the virtual digital human audio and video;

[0011] Determine the video frame corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document;

[0012] Acquire the mouth shape picture corresponding to each of the phonemes from the mouth shape picture set;

[0013] For each of the mouth shape pictures, the mouth position of the mouth shape picture on the video frame is identified by brightness tracking, and the mouth shape picture is pasted into the video frame according to the mouth position.

[0014] In an optional implementation of the first aspect of the present invention, the step of performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series includes:

[0015] Separate each pinyin in the pinyin document into initials and finals to obtain the phonemes of each pinyin;

[0016] Read the audio and obtain the audio frame corresponding to each of the phonemes in the audio;

[0017] Obtaining a time sequence of each of the phonemes based on a time stamp of the audio frame in the audio, wherein the time sequence includes a start time of the phoneme and an end time of the phoneme;

[0018] Each of the phonemes and the start time and the end time corresponding to each of the phonemes are written in the phoneme document according to the writing order of the start time, the end time, and the phoneme.

[0019] In an optional implementation of the first aspect of the present invention, determining the video frame corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document in the phoneme set includes:

[0020] For each of the phonemes, obtaining the start time and the end time of the phoneme;

[0021] Obtaining the frame rate of the video;

[0022] The video frame corresponding to the phoneme in the video is determined by a first value obtained by the start time*the frame rate and a second value obtained by the end time*the frame rate.

[0023] In an optional implementation of the first aspect of the present invention, the determining of the video frame corresponding to the phoneme in the video by a first value obtained by the start time*the frame rate and a second value obtained by the end time*the frame rate comprises:

[0024] Each of the phonemes and the video frame corresponding to each of the phonemes in the video are saved in a dictionary format to obtain a dictionary document of the phonemes and the video frames.

[0025] In an optional implementation of the first aspect of the present invention, for each of the mouth shape pictures, identifying a mouth position of the mouth shape picture on the video frame by brightness tracking, and pasting the mouth shape picture onto the video frame according to the mouth position comprises:

[0026] Creating a reference video with artificial markings of mouth positions based on the video;

[0027] Obtain the video frame corresponding to the mouth shape picture in the video;

[0028] Obtaining a reference video frame corresponding to the video frame in the reference video;

[0029] Subtracting the video frame from the reference video frame to remove the overlapping parts to obtain a binary image with a black background and an artificial mark of the mouth position;

[0030] Obtaining the coordinates of the artificial mark of the mouth position in the binary image by brightness tracking;

[0031] The mouth shape picture is pasted to a corresponding position of the video frame based on the coordinates.

[0032] In an optional implementation of the first aspect of the present invention, pasting the mouth shape picture to a corresponding position of the video frame based on the coordinates includes:

[0033] Obtaining image size parameters of the video frame;

[0034] Adjusting the size parameters of the mouth shape picture based on the image size parameters of the video frame;

[0035] Determine the swing angle of the mouth shape image based on the angle between the central axis of the virtual digital human in the video frame and the Y axis of the standard coordinate system;

[0036] The mouth shape picture is attached to the corresponding position of the video frame and the mouth shape picture is adjusted to the swing angle.

[0037] In an optional implementation of the first aspect of the present invention, the speech recognition of the audio to obtain the pinyin document is implemented through the wenet speech recognition model; the phoneme and audio frame alignment processing of the pinyin document and the audio to obtain the phoneme document with time series is implemented through the WAV2VEC2 acoustic model.

[0038] A second aspect of the present invention provides a device for generating a dynamic mouth shape of a virtual digital human. The device for generating a dynamic mouth shape of a virtual digital human comprises:

[0039] The first extraction module is used to extract the audio from the virtual digital human audio and video;

[0040] A speech recognition module, used for performing speech recognition on the audio to obtain a pinyin document;

[0041] An alignment module, used for performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series;

[0042] A first acquisition module is used to acquire a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database;

[0043] The second extraction module is used to extract the video in the virtual digital human audio and video;

[0044] A matching module, configured to determine a video frame corresponding to each phoneme in the video based on a time sequence corresponding to each phoneme in the phoneme set in the phoneme document;

[0045] A second acquisition module is used to acquire the mouth shape picture corresponding to each of the phonemes from the mouth shape picture set;

[0046] A fitting module is used to identify the mouth position of each mouth shape picture on the video frame by brightness tracking, and to fit the mouth shape picture to the video frame according to the mouth position.

[0047] The third aspect of the present invention provides a device for generating dynamic mouth shapes of virtual digital humans, wherein the device for updating client configuration data comprises: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line;

[0048] The at least one processor calls the instructions in the memory to enable the device for generating the dynamic mouth shape of a virtual digital human to execute the method for generating the dynamic mouth shape of a virtual digital human as described in any one of the above items.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a dynamic mouth shape of a virtual digital human as described in any one of the above items is implemented.

[0050] Beneficial effect: The present invention provides a method for generating dynamic mouth shapes of virtual digital humans and related equipment, wherein the method comprises: extracting audio from audio and video of virtual digital humans; performing speech recognition on the audio to obtain a pinyin document; performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series; obtaining a mouth shape picture set corresponding to a phoneme set in a phoneme document from a mouth shape database; extracting a video from the audio and video of virtual digital humans; determining a video frame corresponding to each phoneme in the video based on the time series corresponding to each phoneme in the phoneme document in the phoneme set; obtaining a mouth shape picture corresponding to each phoneme from a mouth shape picture set; for each mouth shape picture, identifying the mouth position of the mouth shape picture on the video frame by brightness tracking, and pasting the mouth shape picture into the video frame according to the mouth position. The method of the present invention can quickly and accurately generate dynamic mouth shapes of virtual digital humans, and virtual digital humans are more vivid and interesting. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of an embodiment of a method for generating a dynamic mouth shape of a virtual digital human according to the present invention;

[0052] Figure 2 A schematic diagram of an embodiment of a device for generating a dynamic mouth shape of a virtual digital human according to the present invention;

[0053] Figure 3 The present invention is a schematic diagram of an embodiment of a device for generating dynamic mouth shapes of a virtual digital human. DETAILED DESCRIPTION

[0054] The embodiment of the present invention provides a method for generating a dynamic mouth shape of a virtual digital human and related equipment. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] See also Figure 1 The first aspect of the present invention provides a method for generating a dynamic mouth shape of a virtual digital human, the method comprising:

[0056] S100, extracting audio (audio data) from the audio and video (audio and video data) of the virtual digital human; in the present invention, the virtual digital human dynamic mouth shape is generated by a virtual digital human audio and video without a mouth shape, and before the virtual digital human mouth shape is generated, the audio needs to be extracted from the virtual digital human audio and video;

[0057] S200, performing speech recognition on the audio to obtain a Pinyin document; the performing speech recognition on the audio to obtain a Pinyin document is implemented through a wenet speech recognition model; specifically, the input of the neural reasoning network is a piece of audio, and the output is a Pinyin document. The output obtained in this step does not contain a time series, so a neural network is required next to extract the phonemes from the Pinyin and align them (alignment refers to the time series corresponding to the phonemes); the wenet speech recognition model has different neural network reasoning modes, and the attention_rescoring mode is selected here. The attention_rescoring mode is a network structure based on ctc-loss and beam search. Its advantage is that it allows the length of the Pinyin we output to be non-fixed (the output length of most neural networks is fixed), while taking into account both time and accuracy.

[0058] S300, aligning the phonemes and audio frames of the pinyin document and the audio to obtain a phoneme document with a time series; the aligning the phonemes and audio frames of the pinyin document and the audio to obtain a phoneme document with a time series is implemented by the WAV2VEC2 acoustic model. More specifically, the WAV2VEC2_ASR_BASE_960H model of the torchaudio neural network library based on the classic neural network framework pytorch is used here. Compared with other speech alignments, its advantages are fast speed and easy use (just import and call it in the python library). The input of the WAV2VEC2_ASR_BASE_960H model is a piece of audio and a piece of pinyin, and the output is a phoneme with a timestamp. There are many examples of torchaudio's model use on the Internet, which will not be introduced in detail here.

[0059] In an optional implementation of step S300, performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series includes:

[0060] Each pinyin in the pinyin document is split into initials and finals to obtain the phonemes of each pinyin; for example, in step S200, a pinyin document is obtained, which records "ju bao dao Zhongfan yi jing bei si shi liu wai guo ren shi jia ru hei ming dan"; in this step, the pinyin document will be split into "jub ao d ao Zh ong fan yiij ing b ei si shi il iuw ai g uo r en sh ij ia ruh ei ming dan";

[0061] The audio is read to obtain audio frames corresponding to each of the phonemes in the audio. In this embodiment, the audio frame corresponding to each phoneme is obtained mainly by acoustic feature matching.

[0062] Obtaining a time sequence of each of the phonemes based on a timestamp of the audio frame in the audio, wherein the time sequence includes a start time and an end time of the phoneme; in this embodiment, after each phoneme is matched with the audio frame, obtaining a start time and an end time corresponding to the phoneme through a timestamp of the audio frame in the audio;

[0063] Each of the phonemes and the start time and the end time corresponding to each of the phonemes are written in the phoneme document according to the writing order of the start time, the end time, and the phoneme. After obtaining the start time and the end time of each phoneme, the start time, the end time, and the phoneme are stored in the phoneme document. For example, taking "jub ao dao Zh ong f an yi ing b ei si sh il iu w ai g uo r en sh ij ia ruh eim ing d an" as an example, in the phoneme document, the storage format is:

[0064] "0.000 0.085j

[0065] 0.085 0.150u

[0066] 0.150 0.220b

[0067] …………

[0068] 0.790 0.130an".

[0069] S400, obtaining a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database; in this embodiment, a mouth shape picture corresponding to each phoneme is stored in advance in the mouth shape database on the cloud. After all the phonemes are obtained, the corresponding mouth shape pictures can be downloaded from the mouth shape database on the cloud, and finally a mouth shape picture set of all the phonemes is obtained;

[0070] S500, extracting the video (video data) in the audio and video of the virtual digital human; in this embodiment, after obtaining the mouth shape pictures of all phonemes, it is necessary to paste the meanings of all the mouth shape pictures into the video frames of the video in the audio and video of the virtual digital human, so before pasting the mouth shape pictures, it is necessary to extract the video from the audio and video of the virtual digital human;

[0071] S600, determining the video frames corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document in the phoneme set; specifically, determining the video frames corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document in the phoneme set includes:

[0072] For each of the phonemes, obtain the start time and the end time of the phoneme; obtain the frame rate of the video; determine the video frame corresponding to the phoneme in the video by the first value obtained by the start time * the frame rate and the second value obtained by the end time * the frame rate. Simply put, this step can use Python's opencv to read the video frames of the video, and find the corresponding relationship list between each phoneme and the video frame according to the time flow of the video. For example, the phoneme corresponding to 0.000 to 0.085s is j. Assuming that the frame rate of the long video is 25, the video frame corresponding to 0.000s to 0.085s is (0.000*25, 0.085*25), which is rounded to 1 frame to 2 frames. Then, the video frame corresponding to the phoneme 'D' is the 1st frame to the 2nd frame.

[0073] After determining the video frame corresponding to the phoneme in the video by the first value obtained by the start time * the frame rate and the second value obtained by the end time * the frame rate, the method includes: saving each of the phonemes and the video frame corresponding to each of the phonemes in the video in a dictionary format to obtain a dictionary document of the phonemes and the video frames. In this embodiment, after obtaining the correspondence between the phonemes and the video frames, the present invention maintains the mapping relationship between the phonemes and the video frames through a dictionary document. Of course, the mapping relationship between the phonemes, mouth shape pictures and video frames can also be stored in this dictionary document.

[0074] S700, obtaining a mouth shape picture corresponding to each of the phonemes from the mouth shape picture set; in this embodiment, the mouth shape picture corresponding to each phoneme can be obtained according to the mapping relationship of downloading the mouth shape picture from the cloud according to the phoneme in step S400;

[0075] S800. For each of the mouth shape pictures, identify the mouth position of the mouth shape picture on the video frame by brightness tracking, and paste the mouth shape picture into the video frame according to the mouth position. In this embodiment, if the phoneme corresponding to 0.000s to 0.085s is j. Assuming that the frame rate of the video is 25, then the video frame corresponding to 0.000s to 0.085s is (0.000*25, 0.085*25), which is rounded to 1 frame to 2 frames, then put the mouth shape corresponding to the phoneme 'j' obtained in the previous step into the position of the first frame and the second frame. If the number of frames of the mouth shape of the phoneme 'j' obtained in the previous step is only 1, then copy this mouth shape twice. If the number of mouth frames of the phoneme 'j' obtained in the previous step is greater than 2, then the extra frames are removed on average to ensure that the number of mouth frames of the new combination can completely match the time stream of the generated phoneme. Arrange them according to the same rules to obtain a series of mouth shape pictures arranged in the order of phonemes. The number of these pictures is the duration of the audio multiplied by the frame rate.

[0076] In step S800, for each of the mouth shape pictures, identifying the mouth position of the mouth shape picture on the video frame by brightness tracking, and pasting the mouth shape picture onto the video frame according to the mouth position includes:

[0077] A reference video with a mouth position artificially marked is produced based on the video; in this embodiment, the video is first copied, and then the copied video is input into a video editing software, and the video editing software actually marks the mouth position in each frame of the copied video. In this step, the mouth position is artificially marked as a rectangular black box, and the central axis of the black box is used as the central axis of the virtual character in the video frame;

[0078] Obtain the video frame corresponding to the mouth shape picture in the video; obtain the reference video frame corresponding to the video frame in the reference video; subtract the video frame from the reference video frame to remove the overlapping parts to obtain a binary image with a black background and an artificial mark on the mouth position; obtain the coordinates of the artificial mark on the mouth position in the binary image by brightness tracking; in this step, the subtract() function of opencv in python can be used to subtract the two images to obtain the position of the mouth. Since the reference video is almost the same as the video except for the mouth mark point, after the reference video is subtracted from the video, except for the mouth, the rest is all black, and the white mark point is the position of the mouth, and the white mark point is the black rectangle in the reference video. For the position of the mouth, we only need to track the position of the white rectangle to obtain the position of the mouth in each video frame. To obtain the position of the mouth in the video frame, we can use Python's opencv's cv2.findContours() function to track the white rectangle in the binary image, and use cv2.boundingRect() to find the position of the rectangle and store the position result as an array (ie numpy format). For example, the last saved position data of this white rectangle is (200, 200, 25, 15), where (200, 200) is the position of the upper left vertex of the white rectangle, and (25, 15) is the width and height of the white rectangle. Finally, based on the coordinates, the mouth image is pasted to the corresponding position of the video frame.

[0079] In an optional implementation of the first aspect of the present invention, pasting the mouth shape picture to a corresponding position of the video frame based on the coordinates includes:

[0080] Obtain the image size parameters of the video frame; adjust the size parameters of the mouth shape picture based on the image size parameters of the video frame; determine the swing angle of the mouth shape picture based on the angle between the central axis of the virtual digital person in the video frame and the Y axis of the standard coordinate system; fit the mouth shape picture to the corresponding position of the video frame and adjust the mouth shape picture to the swing angle. In this embodiment, since the size of the mouth may be disproportionate to the body, the mouth needs to be scaled so that it will not be inconsistent after being integrated with the body. Similarly, since the body of the virtual character in the video frame is dynamic, the mouth needs to rotate after the body of the virtual character rotates, which requires determining the swing angle that the mouth needs to change. The swing angle of the mouth can be determined based on the swing angle of the small black rectangular box during the previous manual marking. After obtaining the coordinates of the four corners of the black rectangular box (i.e., the white rectangle obtained in the binary image), the trigonometric function calculation method is used to calculate the swing angle based on the width and height of the rectangle. This set of calculations can also be done using Python's opencv library. After the swing angle is obtained, the swing angle of the mouth shape picture (the whole is a square picture) is adjusted to adjust the central axis of the mouth shape picture to be parallel to the central axis of the black box. In the present invention, the rotation angle of the mouth shape picture is manually calibrated using a rectangular black box according to the state of the virtual digital human in the actual video frame, so that the mouth shape and the virtual digital human can be adapted more naturally; finally, after the video frame with the dynamic mouth shape is completed, the present invention needs to use the cv2.VideoWriter() function of opencv of python to convert the video frame with the mouth shape picture back into a video, and use the ffmpeg command to merge the audio and the video with the mouth shape into the audio and video of the virtual digital human.

[0081] See also Figure 2 The second aspect of the present invention provides a device for generating a dynamic mouth shape of a virtual digital human, the device for generating a dynamic mouth shape of a virtual digital human comprising:

[0082] The first extraction module 10 is used to extract the audio from the virtual digital human audio and video;

[0083] A speech recognition module 20, used for performing speech recognition on the audio to obtain a pinyin document;

[0084] An alignment module 30, for performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series;

[0085] A first acquisition module 40 is used to acquire a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database;

[0086] The second extraction module 50 is used to extract the video in the virtual digital human audio and video;

[0087] A matching module 60, configured to determine a video frame corresponding to each phoneme in the video based on a time sequence corresponding to each phoneme in the phoneme set in the phoneme document;

[0088] A second acquisition module 70 is used to acquire a mouth shape picture corresponding to each of the phonemes from the mouth shape picture set;

[0089] The fitting module 80 is used to identify the mouth position of each mouth shape picture on the video frame by brightness tracking, and paste the mouth shape picture into the video frame according to the mouth position.

[0090] In an optional implementation of the second aspect of the present invention, the alignment module 30 includes:

[0091] A splitting unit, used for splitting each pinyin in the pinyin document into initials and finals to obtain the phonemes of each pinyin;

[0092] A reading unit, used for reading the audio and obtaining an audio frame corresponding to each of the phonemes in the audio;

[0093] an alignment unit, configured to obtain a time sequence of each of the phonemes based on a time stamp of the audio frame in the audio, wherein the time sequence includes a start time and an end time of the phoneme;

[0094] The storage unit is used to write each of the phonemes and the start time and the end time corresponding to each of the phonemes in the phoneme document according to the writing order of the start time, the end time, and the phoneme.

[0095] In an optional implementation of the second aspect of the present invention, the matching module 60 includes:

[0096] A first acquisition unit, configured to acquire, for each of the phonemes, the start time and the end time of the phoneme;

[0097] A second acquisition unit, used to acquire a frame rate of the video;

[0098] A matching unit is used to determine the video frame corresponding to the phoneme in the video through a first value obtained by the start time*the frame rate and a second value obtained by the end time*the frame rate.

[0099] In an optional implementation of the second aspect of the present invention, the matching module 60 further includes:

[0100] The storage unit is used to store each of the phonemes and the video frames corresponding to each of the phonemes in the video in a dictionary format to obtain a dictionary document of the phonemes and the video frames.

[0101] In an optional implementation of the second aspect of the present invention, the laminating module 80 includes:

[0102] A production unit, used for producing a reference video with an artificial mark of a mouth position based on the video;

[0103] A third acquisition unit is used to acquire the video frame corresponding to the mouth shape picture in the video;

[0104] A fourth acquisition unit, configured to acquire a reference video frame corresponding to the video frame in the reference video;

[0105] A cutout unit, used for subtracting the video frame from the reference video frame to remove the overlapping parts to obtain a binary image with a black background and an artificial mark of the mouth position;

[0106] A tracking unit, used for obtaining the coordinates of the artificial mark of the mouth position in the binary image by brightness tracking;

[0107] A pasting unit is used to paste the mouth shape picture to a corresponding position of the video frame based on the coordinates.

[0108] In an optional embodiment of the second aspect of the present invention, the laminating unit includes:

[0109] An acquisition subunit, used to acquire an image size parameter of the video frame;

[0110] An adjusting subunit, configured to adjust a size parameter of the mouth shape picture based on an image size parameter of the video frame;

[0111] A determination subunit, used to determine the swing angle of the mouth shape image based on the angle between the central axis of the virtual digital human in the video frame and the Y axis of the standard coordinate system;

[0112] The fitting subunit is used to fit the mouth shape picture to the corresponding position of the video frame and adjust the mouth shape picture to the swing angle.

[0113] In an optional implementation of the second aspect of the present invention, the speech recognition of the audio to obtain the pinyin document is implemented through the wenet speech recognition model; the phoneme and audio frame alignment processing of the pinyin document and the audio to obtain the phoneme document with time series is implemented through the WAV2VEC2 acoustic model.

[0114] Figure 31 is a schematic diagram of the structure of a device for generating a dynamic mouth shape of a virtual digital human provided by an embodiment of the present invention. The device for generating a dynamic mouth shape of a virtual digital human may have relatively large differences due to different configurations or performances, and may include one or more processors 90 (central processing units, CPU) (for example, one or more processors) and a memory 100, and one or more storage media 110 for storing application programs or data (for example, one or more mass storage devices). Among them, the memory and the storage medium may be temporary storage or permanent storage. The program stored in the storage medium may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device for generating a dynamic mouth shape of a virtual digital human. Furthermore, the processor may be configured to communicate with the storage medium to execute a series of instruction operations in the storage medium on the device for generating a dynamic mouth shape of a virtual digital human.

[0115] The device for generating dynamic mouth shapes of virtual digital human of the present invention may also include one or more power supplies 120, one or more wired or wireless network interfaces 130, one or more input and output interfaces 140, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It can be understood by those skilled in the art that Figure 3 The structure of the device for generating dynamic mouth shapes of virtual digital humans shown does not constitute a specific limitation on the device for generating dynamic mouth shapes of virtual digital humans of the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0116] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for generating a dynamic mouth shape of a virtual digital human.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0119] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a dynamic mouth shape of a virtual digital human, characterized in that: The method for generating the dynamic mouth shape of a virtual digital human comprises: Extract audio from virtual digital human audio and video; Performing speech recognition on the audio to obtain a pinyin document; Performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series; Acquire a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database; Extracting video from the virtual digital human audio and video; Determine the video frame corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document; Acquire the mouth shape picture corresponding to each of the phonemes from the mouth shape picture set; For each of the mouth shape pictures, a mouth position of the mouth shape picture on the video frame is identified by brightness tracking, and the mouth shape picture is pasted into the video frame according to the mouth position; For each of the mouth shape pictures, identifying the mouth position of the mouth shape picture on the video frame by brightness tracking, and pasting the mouth shape picture onto the video frame according to the mouth position includes: Creating a reference video with artificial markings of mouth positions based on the video; Obtain the video frame corresponding to the mouth shape picture in the video; Obtaining a reference video frame corresponding to the video frame in the reference video; Subtracting the video frame from the reference video frame to remove the overlapping parts to obtain a binary image with a black background and an artificial mark of the mouth position; Obtaining the coordinates of the artificial mark of the mouth position in the binary image by brightness tracking; The mouth shape picture is pasted to a corresponding position of the video frame based on the coordinates.

2. The method for generating a dynamic mouth shape of a virtual digital human according to claim 1, characterized in that: The step of performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series includes: Separate each pinyin in the pinyin document into initials and finals to obtain the phonemes of each pinyin; Read the audio and obtain the audio frame corresponding to each of the phonemes in the audio; Obtaining a time sequence of each of the phonemes based on a time stamp of the audio frame in the audio, wherein the time sequence includes a start time of the phoneme and an end time of the phoneme; Each of the phonemes and the start time and the end time corresponding to each of the phonemes are written in the phoneme document according to the writing order of the start time, the end time, and the phoneme.

3. The method for generating a dynamic mouth shape of a virtual digital human according to claim 2, characterized in that: Determining the video frames corresponding to each phoneme in the video based on the time sequence corresponding to each phoneme in the phoneme document in the phoneme set includes: For each of the phonemes, obtaining the start time and the end time of the phoneme; Obtaining the frame rate of the video; The video frame corresponding to the phoneme in the video is determined by a first value obtained by the start time*the frame rate and a second value obtained by the end time*the frame rate.

4. The method for generating a dynamic mouth shape of a virtual digital human according to claim 3, characterized in that: The determining of the video frame corresponding to the phoneme in the video by a first value obtained by the start time*the frame rate and a second value obtained by the end time*the frame rate comprises: Each of the phonemes and the video frame corresponding to each of the phonemes in the video are saved in a dictionary format to obtain a dictionary document of the phonemes and the video frames.

5. The method for generating a dynamic mouth shape of a virtual digital human according to claim 1, characterized in that: The step of pasting the mouth shape picture to the corresponding position of the video frame based on the coordinates includes: Obtaining image size parameters of the video frame; Adjusting the size parameters of the mouth shape picture based on the image size parameters of the video frame; Determine the swing angle of the mouth shape image based on the angle between the central axis of the virtual digital human in the video frame and the Y axis of the standard coordinate system; The mouth shape picture is attached to the corresponding position of the video frame and the mouth shape picture is adjusted to the swing angle.

6. The method for generating a dynamic mouth shape of a virtual digital human according to claim 1, characterized in that: The speech recognition of the audio to obtain the pinyin document is achieved through the wenet speech recognition model; the phoneme and audio frame alignment of the pinyin document and the audio to obtain the phoneme document with time series is achieved through the WAV2VEC2 acoustic model.

7. A device for generating dynamic mouth shapes of virtual digital humans, characterized in that: The device for generating the dynamic mouth shape of the virtual digital human comprises: The first extraction module is used to extract the audio from the virtual digital human audio and video; A speech recognition module, used for performing speech recognition on the audio to obtain a pinyin document; An alignment module, used for performing phoneme and audio frame alignment processing on the pinyin document and the audio to obtain a phoneme document with a time series; A first acquisition module is used to acquire a mouth shape picture set corresponding to the phoneme set in the phoneme document from a mouth shape database; The second extraction module is used to extract the video in the virtual digital human audio and video; A matching module, configured to determine a video frame corresponding to each phoneme in the video based on a time sequence corresponding to each phoneme in the phoneme set in the phoneme document; A second acquisition module is used to acquire the mouth shape picture corresponding to each of the phonemes from the mouth shape picture set; A fitting module, for identifying, for each of the mouth shape images, a mouth position of the mouth shape image on the video frame by brightness tracking, and pasting the mouth shape image onto the video frame according to the mouth position; The bonding module comprises: A production unit, used for producing a reference video with an artificial mark of a mouth position based on the video; A third acquisition unit is used to acquire the video frame corresponding to the mouth shape picture in the video; A fourth acquisition unit, configured to acquire a reference video frame corresponding to the video frame in the reference video; A cutout unit, used for subtracting the video frame from the reference video frame to remove the overlapping parts to obtain a binary image with a black background and an artificial mark of the mouth position; A tracking unit, used for obtaining the coordinates of the artificial mark of the mouth position in the binary image by brightness tracking; A pasting unit is used to paste the mouth shape picture to a corresponding position of the video frame based on the coordinates.

8. A device for generating dynamic mouth shapes of virtual digital humans, characterized in that: The generating device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the device for generating the dynamic mouth shape of a virtual digital human to execute the method for generating the dynamic mouth shape of a virtual digital human as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a dynamic mouth shape of a virtual digital human as described in any one of claims 1 to 6 is implemented.