A method and device for generating a mouth shape animation, an electronic device, and a storage medium

Abstract

The application discloses a method and device for generating mouth shape animation, electronic equipment and storage medium. The method comprises the following steps: obtaining target speech data for generating mouth shape animation; analyzing the target speech data to obtain a phoneme sequence corresponding to the target speech data; obtaining a target duration corresponding to each phoneme unit in the phoneme sequence, and obtaining a target animation unit corresponding to the phoneme unit; and generating a mouth shape animation corresponding to the target speech data based on the target duration corresponding to each phoneme unit in the phoneme sequence and the target animation unit. The application detects phoneme units in speech data, determines the duration of each phoneme unit in the speech data and the animation unit corresponding to the phoneme unit, and finally generates a mouth shape animation using the duration of the phoneme unit and the animation unit. In this way, the pronunciation action of the mouth shape animation is accurately matched with the pronunciation order of the characters in the text, so that the finally generated mouth shape animation can be more consistent with the real scene.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for generating lip-sync animation. Background Technology

[0002] Voice, as a natural form of communication, has demonstrated a significant influence in the field of human-computer interaction, as evidenced by the increasing prevalence of social media in our daily lives. However, generating realistic lip-sync animation during human-computer interaction is extremely complex. Existing technologies rely solely on limited, single keyframe animations to produce lip-sync animation frame data that conforms to a Gaussian distribution. This approach struggles to accurately represent the mouth shape and facial muscle movements of a normal person speaking, resulting in lip-sync animation effects that do not accurately reflect real-world scenarios.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a method, apparatus, electronic device and storage medium for generating lip-sync animation.

[0005] According to one aspect of the embodiments of this application, a method for generating lip-sync animation is provided, comprising:

[0006] Acquire the target speech data used to generate lip-sync animation;

[0007] The target speech data is parsed to obtain the phoneme sequence corresponding to the target speech data, wherein the phoneme sequence includes phoneme units arranged in chronological order;

[0008] Obtain the target duration corresponding to each phoneme unit in the phoneme sequence, and obtain the target animation unit corresponding to the phoneme unit, wherein the target animation unit includes at least one animation keyframe carrying the lip movement corresponding to the phoneme unit;

[0009] Based on the target duration corresponding to each phoneme unit in the phoneme sequence and the target animation unit, the lip-sync animation corresponding to the target speech data is generated.

[0010] Furthermore, the step of parsing the target speech data to obtain the phoneme sequence corresponding to the target speech data includes:

[0011] Identify the target speech data to obtain the text content corresponding to the target speech data;

[0012] Obtain the characters included in the text content, and the pinyin information corresponding to each character;

[0013] The phoneme unit corresponding to each character is obtained from the pinyin information, and the phoneme units are arranged based on the character playback data to generate the phoneme sequence.

[0014] Furthermore, obtaining the target duration corresponding to each phoneme unit in the phoneme sequence includes:

[0015] Obtain the start and end timestamps corresponding to the text in the target speech data, and determine the time length corresponding to the text based on the start and end timestamps;

[0016] Obtain the weight value of each phoneme unit corresponding to the character, wherein the weight value is used to represent the pronunciation intensity corresponding to the phoneme unit, and the pronunciation intensity is directly proportional to the duration;

[0017] The target duration corresponding to each phoneme unit is calculated using the weight and the time length.

[0018] Furthermore, obtaining the weight value of each phoneme unit corresponding to the character includes:

[0019] The target type of the phoneme unit corresponding to the character is determined based on the pinyin information, wherein the target type includes initial consonant type and final vowel type;

[0020] Obtain the weight list corresponding to the target type, and obtain the weight value corresponding to the phoneme unit from the weight list.

[0021] Furthermore, obtaining the target animation unit corresponding to the phoneme unit includes:

[0022] Obtain an animation database, wherein the animation database includes at least one animation unit;

[0023] Obtain an original animation unit that matches the phoneme unit from at least one animation unit in the animation database, wherein the original animation unit includes animation keyframes of a first frame number;

[0024] Obtain the mapping relationship between the number of animation keyframes for the actual lip-syncing action and the number of animation keyframes for the standard lip-syncing action;

[0025] Based on the mapping relationship, the second frame number corresponding to the first frame number included in the original animation unit is determined, and the target animation unit is generated based on the second frame number and the animation keyframes.

[0026] Furthermore, generating the lip-sync animation corresponding to the target speech data based on the target duration corresponding to each phoneme unit in the phoneme sequence and the target animation unit includes:

[0027] The target duration is determined as the target playback time corresponding to the target animation unit;

[0028] The lip-sync animation is generated by arranging the target animation units according to the time sequence corresponding to the phoneme units and the target playback time corresponding to each target animation unit.

[0029] Furthermore, the method also includes:

[0030] Based on the lip-sync animation, obtain the animation keyframe set corresponding to each phoneme unit, and determine the original start animation keyframe and the original end animation keyframe corresponding to the phoneme unit based on the animation keyframe set.

[0031] Obtain the Nth animation keyframe before the original starting animation keyframe and the Mth animation keyframe after the original ending animation keyframe, where M and N are both integers greater than 0;

[0032] The Nth animation keyframe is determined as the target start animation keyframe, and the Mth animation keyframe is determined as the target end animation keyframe;

[0033] The set of animation keyframes corresponding to each phoneme unit is updated using the target start animation keyframe and the target end animation keyframe to obtain the updated lip-sync animation.

[0034] According to another aspect of the embodiments of this application, a lip-syncing animation generation apparatus is also provided, comprising:

[0035] The first acquisition module is used to acquire the target speech data for generating lip-sync animation;

[0036] The parsing module is used to parse the target speech data to obtain the phoneme sequence corresponding to the target speech data, wherein the phoneme sequence includes phoneme units arranged in chronological order;

[0037] The second acquisition module is used to acquire the target duration corresponding to each phoneme unit in the phoneme sequence, and the animation unit of the phoneme unit pair, wherein the animation unit includes the lip movement corresponding to the phoneme unit;

[0038] The generation module is used to generate lip-sync animation corresponding to the target speech data based on the target duration and animation unit corresponding to each phoneme in the phoneme sequence.

[0039] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0040] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0041] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0042] Compared with the prior art, the above-mentioned technical solution provided in this application has the following advantages: The method provided in this application detects phoneme units in speech data, determines the duration of each phoneme unit in the speech data and the animation unit corresponding to the phoneme unit, and finally uses the duration of the phoneme unit and the animation unit to generate lip-sync animation, thereby accurately matching the pronunciation action of the lip-sync animation with the pronunciation order of the text, so that the final generated lip-sync animation can better match the real scene. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a method for generating lip-sync animation provided in an embodiment of this application;

[0046] Figure 2 A block diagram of a lip-sync animation generation device provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another similar entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] This application provides a method, apparatus, electronic device, and storage medium for generating lip-sync animation. The method provided by this invention can be applied to any electronic device as needed, such as a server, terminal, or other electronic device. No specific limitation is made here, and for ease of description, it will be referred to as an electronic device below.

[0051] According to one aspect of the embodiments of this application, a method embodiment for generating lip-sync animation is provided. Figure 1 A flowchart illustrating a method for generating lip-sync animation provided in this application embodiment is shown below. Figure 1 As shown, the method includes:

[0052] Step S11: Obtain the target speech data used to generate lip-sync animation.

[0053] The method provided in the embodiments of the present application is applied to intelligent terminals capable of audio and video processing, and the intelligent terminals may be computers, smartphones, iPads, and the like. In the embodiments of the present application, the process for the intelligent terminal to acquire target voice data may be: acquiring through an audio file uploaded by a user, or acquiring from a video in a video library. Specifically, query information input by a user is acquired, a target video matching the query information is acquired from the video library, a user requirement for segmenting the target video is acquired, a target video segment matching the user requirement is acquired from the to-be-processed video, and target voice data for generating a lip animation is extracted from the target video segment. Alternatively, query information input by a user is acquired, a target video matching the query information is acquired from the video library, a candidate video segment with a popularity value greater than a preset popularity value is acquired from the target video, the candidate video segment is displayed, a selection operation triggered by the user is detected, and a target video segment is determined based on the selection operation. Target voice data for generating a lip animation is extracted from the target video segment.

[0054] Step S12: Parse the target voice data to obtain a phoneme sequence corresponding to the target voice data, wherein the phoneme sequence includes phoneme units arranged in chronological order.

[0055] In the embodiments of the present application, step S12 of parsing the target voice data to obtain a phoneme sequence corresponding to the target voice data includes the following steps A1 to A3:

[0056] Step A1: Recognize the target voice data to obtain text content corresponding to the target voice data.

[0057] Step A2: Acquire characters included in the text content, and pinyin information corresponding to each character.

[0058] Step A3: Acquire a phoneme unit corresponding to each character from the pinyin information, and arrange the phoneme units based on the character playback sequence to generate a phoneme sequence.

[0059] In the embodiments of the present application, a speech recognition model is invoked, and the target voice data is input into the speech recognition model, so that the speech recognition model extracts speech features corresponding to the target voice data and determines text content corresponding to the speech features. Then characters included in the text content are extracted, and pinyin information corresponding to each character is determined. The phoneme unit corresponding to each character is acquired from the pinyin information, for example, the pinyin corresponding to the characters in the text "波动" is "bo" and "dong", wherein the phoneme units are b, o, d, o, n, g. Then the phoneme units are arranged according to the character playback sequence to generate the phoneme sequence "b-o-d-o-n-g".

[0060] Step S13: Obtain the target duration corresponding to each phoneme unit in the phoneme sequence, and obtain the target animation unit corresponding to the phoneme unit, wherein the target animation unit includes at least one animation keyframe carrying the lip movement corresponding to the phoneme unit.

[0061] In this embodiment of the application, step S13, obtaining the target duration corresponding to each phoneme unit in the phoneme sequence, includes the following steps B1-B3:

[0062] Step B1: Obtain the start and end timestamps corresponding to the text in the target speech data, and determine the time length corresponding to the text based on the start and end timestamps.

[0063] In this embodiment of the application, the start timestamp T1 and end timestamp T2 corresponding to the text in the target speech data are obtained, and the time length corresponding to the text is calculated based on the start timestamp T1 and end timestamp T2, and the time length is T2-T1.

[0064] Step B2: Obtain the weight value of each phoneme unit corresponding to the text. The weight value is used to represent the pronunciation intensity of the phoneme unit. The pronunciation intensity is directly proportional to the duration.

[0065] In this embodiment of the application, obtaining the weight value of each phoneme unit corresponding to the text includes: determining the target type of the phoneme unit corresponding to the text based on the pinyin information, wherein the target type includes the initial consonant type and the final vowel type; obtaining the weight list corresponding to the target type, and obtaining the weight value corresponding to the phoneme unit from the weight list.

[0066] It should be noted that, in order to obtain the precise duration corresponding to each phoneme unit, the target type corresponding to the phoneme unit is first determined based on the pinyin information. The target type includes initial consonant type and final vowel type. Initial consonant type includes: plosive consonant type, retroflex consonant type, and alveolar consonant type, etc. Final vowel type includes: simple final vowel type and compound final vowel type.

[0067] As an example: among the initials b, p, and m, they belong to the plosive type, and their pronunciation intensity is greater than that of the finals. Therefore, in a certain phoneme sequence of pinyin, the weight of the initials of the plosive type is less than 0.5, while the weight of the finals is greater than 0.5.

[0068] Step B3: Calculate the target duration corresponding to each phoneme unit using weights and time length.

[0069] In this embodiment, the process of calculating the target duration corresponding to each phoneme unit using weights and time length is as follows: the time length is T2-T1, the phoneme units include b and o, where the weight corresponding to phoneme unit b is w1, and the weight corresponding to phoneme unit o is 2. The target duration corresponding to phoneme unit b is: (T2-T1)×w1. The target duration corresponding to phoneme unit o is: (T2-T1)×w2.

[0070] In this embodiment of the application, obtaining the target animation unit corresponding to the phoneme unit includes: obtaining an animation database, wherein the animation database includes at least one animation unit; and obtaining the target animation unit that matches the phoneme unit from the at least one animation unit in the animation database.

[0071] In this embodiment, an animation database is obtained, which includes at least one animation unit. Each animation unit is pre-selected animation data that conforms to the lip movements of a real human speaker. However, this animation data has issues with varying lengths and inconsistent durations of any phoneme during pronunciation, making direct calculation impossible. Therefore, this embodiment performs polynomial fitting on the animation data, using a polynomial function curve to describe each animation data point, resulting in an animation unit. Each animation unit includes at least one animation keyframe corresponding to the lip movement of a phoneme unit.

[0072] In this embodiment, since there is a certain difference between the number of frames required for the mouth movements of the phoneme unit in actual pronunciation and the number of frames required for the mouth movements in standard pronunciation (i.e., the number of frames included in the animation unit), after obtaining the animation unit corresponding to the phoneme unit from the animation database, the first number of frames included in the original animation unit corresponding to the phoneme unit is first determined, and the mapping relationship between the number of animation keyframes of the actual mouth movements and the number of animation keyframes of the standard mouth movements is obtained. Based on the mapping relationship, the second number of frames corresponding to the first number of frames is determined, and finally the target animation unit corresponding to the phoneme unit is generated using the second number of frames and the animation keyframes.

[0073] It should be noted that, in this embodiment of the application, the target animation unit representing the actual lip-syncing action is obtained by mapping the number of animation keyframes of the actual lip-syncing action to the number of animation keyframes of the standard lip-syncing action, thereby ensuring a higher degree of adaptation between the actual lip-syncing action and the text.

[0074] Step S14: Based on the target duration and target animation unit corresponding to each phoneme unit in the phoneme sequence, generate the lip-sync animation corresponding to the target speech data.

[0075] In this embodiment of the application, step S14, generating lip-sync animation corresponding to the target speech data based on the target duration and target animation unit corresponding to each phoneme unit in the phoneme sequence, includes the following steps C1-C2:

[0076] Step C1: Determine the target duration as the target playback time corresponding to the target animation unit.

[0077] Step C2: Arrange the target animation units according to the time sequence corresponding to the phoneme units, and generate lip-sync animations according to the target playback time corresponding to each target animation unit.

[0078] In this embodiment, when generating lip-sync animation, the target duration is first determined as the target playback time corresponding to the target animation unit. For example, the phoneme units include R1, R2, R3, and R4. The target duration corresponding to each phoneme unit is T1, T2, T3, and T4. The target animation units corresponding to each phoneme unit are P1, P2, P3, and P4. Then, T1 is set as the target playback time of P1, T2 is set as the target playback time of P2, T3 is set as the target playback time of P3, and T4 is set as the target playback time of P4. Finally, the target animation units are arranged in the time order corresponding to the phoneme units, and the target playback time corresponding to each target animation unit is used to generate the lip-sync animation.

[0079] The method provided in this application detects phoneme units in speech data, determines the duration of each phoneme unit in the speech data and the corresponding animation unit, and finally uses the duration of the phoneme unit and the animation unit to generate lip-sync animation, thereby accurately matching the pronunciation action of the lip-sync animation with the pronunciation order of the text, so that the final generated lip-sync animation can better match the real scene.

[0080] In this embodiment of the application, the method further includes the following steps D1-D4:

[0081] Step D1: Obtain the set of animation keyframes corresponding to each phoneme unit based on the lip-sync animation, and determine the original start animation keyframe and the original end animation keyframe corresponding to the phoneme unit based on the set of animation keyframes.

[0082] Step D2: Obtain the Nth animation keyframe before the original start animation keyframe and the Mth animation keyframe after the original end animation keyframe, where M and N are both integers greater than 0.

[0083] Step D3: Determine the Nth animation keyframe as the target start animation keyframe, and determine the Mth animation keyframe as the target end animation keyframe.

[0084] Step D4: Update the set of animation keyframes corresponding to each phoneme unit using the target start animation keyframe and the target end animation keyframe to obtain the updated lip-sync animation.

[0085] In this embodiment, for any speech segment, after segmenting it into a phoneme sequence and the timestamp information corresponding to each phoneme unit, the lip-sync animation of this speech segment can be obtained by directly combining the animation units corresponding to the phoneme units. However, the effect of the lip-sync animation obtained by this combination, when displayed by animation software, still does not conform to the lip-sync movements of a normal person speaking. Therefore, this embodiment expands the animation data between adjacent phonemes. Specifically, it obtains the Nth animation keyframe before the original start animation keyframe and the Mth animation keyframe after the original end animation keyframe. The Nth animation keyframe is determined as the target start animation keyframe, and the Mth animation keyframe is determined as the target end animation keyframe to ensure that the animation data of adjacent phonemes overlap.

[0086] As an example: the keyframes for the phoneme "b" in a speech segment are frames n to n+5, and the keyframes for the phoneme "o" are frames n+6 to n+12. In processing the animation data, the range of phoneme "b" can be set to frames n-1 to n+6, and the range of phoneme "o" can be set to frames n+5 to n+13. This ensures that the two phonemes overlap in frames n+5 and n+6. Furthermore, the overlapping data is weighted and fused using the mean, and this data is used as the data for these two frames, thus ensuring good pronunciation connection between the phonemes.

[0087] Figure 2 This is a block diagram of a lip-sync animation generation device provided in an embodiment of this application. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 2 As shown, the device includes:

[0088] The first acquisition module 21 is used to acquire target speech data for generating lip-sync animation;

[0089] The parsing module 22 is used to parse the target speech data to obtain the phoneme sequence corresponding to the target speech data, wherein the phoneme sequence includes phoneme units arranged in time order;

[0090] The second acquisition module 23 is used to acquire the target duration corresponding to each phoneme unit in the phoneme sequence, and the animation unit of the phoneme unit pair, wherein the animation unit includes the lip movement corresponding to the phoneme unit.

[0091] The generation module 24 is used to generate lip-sync animations corresponding to the target speech data based on the target duration and animation unit corresponding to each phoneme in the phoneme sequence.

[0092] In this embodiment of the application, the parsing module 22 is used to identify the target speech data and obtain the text content corresponding to the target speech data; obtain the characters included in the text content and the pinyin information corresponding to each character; obtain the phoneme unit corresponding to each character from the pinyin information, and arrange the phoneme units based on the text playback data to generate a phoneme sequence.

[0093] In this embodiment of the application, the second acquisition module 23 is used to acquire the start and end timestamps corresponding to the text in the target speech data, and determine the time length corresponding to the text based on the start and end timestamps; acquire the weight value of each phoneme unit corresponding to the text, wherein the weight value is used to represent the pronunciation intensity corresponding to the phoneme unit, and the pronunciation intensity is directly proportional to the duration; and calculate the target duration corresponding to each phoneme unit using the weight and the time length.

[0094] In this embodiment of the application, the second acquisition module 23 is used to determine the target type of the phoneme unit corresponding to the text based on the pinyin information, wherein the target type includes the initial consonant type and the final vowel type; acquire the weight list corresponding to the target type, and acquire the weight value corresponding to the phoneme unit from the weight list.

[0095] In this embodiment of the application, the second acquisition module 23 is used to acquire an animation database, wherein the animation database includes at least one animation unit; acquire an original animation unit that matches the phoneme unit from the at least one animation unit in the animation database, wherein the original animation unit includes animation keyframes of a first frame number; acquire a mapping relationship between the number of animation keyframes of the actual lip-sync action and the number of animation keyframes of the standard lip-sync action; determine the second frame number corresponding to the first frame number included in the original animation unit based on the mapping relationship; and generate a target animation unit based on the second frame number and the animation keyframes.

[0096] In this embodiment of the application, the generation module 24 is used to determine the target duration as the target playback time corresponding to the target animation unit; and to generate lip-sync animation based on the time sequence corresponding to the phoneme unit and the target playback time corresponding to each target animation unit.

[0097] In this embodiment of the application, the generation module 24 is specifically used to obtain the set of animation keyframes corresponding to each phoneme unit based on the lip-sync animation, and determine the original start animation keyframe and the original end animation keyframe corresponding to the phoneme unit based on the set of animation keyframes; obtain the Nth animation keyframe before the original start animation keyframe and the Mth animation keyframe after the original end animation keyframe, where M and N are both integers greater than 0; determine the Nth animation keyframe as the target start animation keyframe and the Mth animation keyframe as the target end animation keyframe; update the set of animation keyframes corresponding to each phoneme unit using the target start animation keyframe and the target end animation keyframe to obtain the updated lip-sync animation.

[0098] This application also provides an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0099] Memory 1503 is used to store computer programs;

[0100] When the processor 1501 executes the computer program stored in the memory 1503, it implements the steps of the above embodiments.

[0101] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0102] The communication interface is used for communication between the aforementioned terminal and other devices.

[0103] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0104] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the lip-sync animation generation method described in any of the above embodiments.

[0106] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the lip-sync animation generation method described in any of the above embodiments.

[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive).

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

[0109] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating lip-sync animation, characterized in that, include: Acquire the target speech data used to generate lip-sync animation; The target speech data is parsed to obtain the phoneme sequence corresponding to the target speech data, wherein the phoneme sequence includes phoneme units arranged in chronological order; Obtain the target duration corresponding to each phoneme unit in the phoneme sequence, and obtain the target animation unit corresponding to the phoneme unit, wherein the target animation unit includes at least one animation keyframe carrying the lip movement corresponding to the phoneme unit; Based on the target duration corresponding to each phoneme unit in the phoneme sequence and the target animation unit, generate the lip-sync animation corresponding to the target speech data; The method further includes: obtaining an animation keyframe set corresponding to each phoneme unit based on the lip-sync animation, and determining the original start animation keyframe and the original end animation keyframe corresponding to the phoneme unit based on the animation keyframe set; obtaining the Nth animation keyframe before the original start animation keyframe and the Mth animation keyframe after the original end animation keyframe, where M and N are both integers greater than 0; determining the Nth animation keyframe as the target start animation keyframe and the Mth animation keyframe as the target end animation keyframe; updating the animation keyframe set corresponding to each phoneme unit using the target start animation keyframe and the target end animation keyframe to obtain the updated lip-sync animation.

2. The method according to claim 1, characterized in that, The step of parsing the target speech data to obtain the phoneme sequence corresponding to the target speech data includes: Identify the target speech data to obtain the text content corresponding to the target speech data; Obtain the characters included in the text content, and the pinyin information corresponding to each character; The phoneme unit corresponding to each character is obtained from the pinyin information, and the phoneme units are arranged according to the character playback order to generate the phoneme sequence.

3. The method according to claim 2, characterized in that, The step of obtaining the target duration corresponding to each phoneme unit in the phoneme sequence includes: Obtain the start and end timestamps corresponding to the text in the target speech data, and determine the time length corresponding to the text based on the start and end timestamps; Obtain the weight value of each phoneme unit corresponding to the character, wherein the weight value is used to represent the pronunciation intensity corresponding to the phoneme unit, and the pronunciation intensity is directly proportional to the duration; The target duration corresponding to each phoneme unit is calculated using the weight and the time length.

4. The method according to claim 3, characterized in that, The step of obtaining the weight value of each phoneme unit corresponding to the character includes: The target type of the phoneme unit corresponding to the character is determined based on the pinyin information, wherein the target type includes initial consonant type and final vowel type; Obtain the weight list corresponding to the target type, and obtain the weight value corresponding to the phoneme unit from the weight list.

5. The method according to claim 1, characterized in that, The step of obtaining the target animation unit corresponding to the phoneme unit includes: Obtain an animation database, wherein the animation database includes at least one animation unit; Obtain an original animation unit that matches the phoneme unit from at least one animation unit in the animation database, wherein the original animation unit includes animation keyframes of a first frame number; Obtain the mapping relationship between the number of animation keyframes for the actual lip-syncing action and the number of animation keyframes for the standard lip-syncing action; Based on the mapping relationship, the second frame number corresponding to the first frame number included in the original animation unit is determined, and the target animation unit is generated based on the second frame number and the animation keyframes.

6. The method according to claim 1, characterized in that, The step of generating lip-sync animation corresponding to the target speech data based on the target duration corresponding to each phoneme unit in the phoneme sequence and the target animation unit includes: The target duration is determined as the target playback time corresponding to the target animation unit; The lip-sync animation is generated by arranging the target animation units according to the time sequence corresponding to the phoneme units and the target playback time corresponding to each target animation unit.

7. A device for generating lip-sync animation, characterized in that, include: The first acquisition module is used to acquire the target speech data for generating lip-sync animation; The parsing module is used to parse the target speech data to obtain the phoneme sequence corresponding to the target speech data, wherein the phoneme sequence includes phoneme units arranged in chronological order; The second acquisition module is used to acquire the target duration corresponding to each phoneme unit in the phoneme sequence, and the animation unit of the phoneme unit pair, wherein the animation unit includes the lip movement corresponding to the phoneme unit; The generation module is used to generate lip-sync animation corresponding to the target speech data based on the target duration and animation unit corresponding to each phoneme in the phoneme sequence. The generation module is further configured to: obtain an animation keyframe set corresponding to each phoneme unit based on the lip-sync animation; determine the original start animation keyframe and the original end animation keyframe corresponding to the phoneme unit based on the animation keyframe set; obtain the Nth animation keyframe before the original start animation keyframe and the Mth animation keyframe after the original end animation keyframe, where M and N are both integers greater than 0; determine the Nth animation keyframe as the target start animation keyframe and the Mth animation keyframe as the target end animation keyframe; update the animation keyframe set corresponding to each phoneme unit using the target start animation keyframe and the target end animation keyframe to obtain the updated lip-sync animation.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program, when executed, performs the method steps of any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other through the communication bus; wherein: Memory, used to store computer programs; A processor for executing the method steps of any one of claims 1 to 6 by running a program stored in memory.

Images