Two-dimensional character interaction mouth shape animation generation method and related equipment

Abstract

The invention discloses a two-dimensional character interaction mouth shape animation generation method and related equipment. The method comprises the steps that a two-dimensional role photo and input information of a target user are acquired, and the input information comprises character input information and / or voice input information; generating a role mouth shape resource picture based on the two-dimensional role picture and predefined mouth shape video data; performing resource optimization operation based on the role mouth shape resource picture to adaptively obtain a mouth shape area picture; obtaining pronunciation phonemes based on the input information and performing phoneme analysis operation to obtain a mouth shape animation corresponding to the pronunciation phonemes; and performing synchronous operation on the mouth shape area picture and the mouth shape animation to generate a two-dimensional character interaction mouth shape animation.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method for generating a two-dimensional character interactive lip-sync animation and related equipment. Background Art

[0002] At present, there are some existing technologies and products in the industry for the generation of lip animation for virtual characters. For example, DeepBrainAI provides 3D and 2D virtual character services that enable real-time conversational interaction. However, this product must be based on pre-made characters and animation content, which lacks flexibility. Users cannot flexibly generate interactive characters based on any 2D images, which leads to long production cycles and high costs, and is not suitable for personalized or low-cost needs.

[0003] Another related technology proposes a technique for generating a character speech video from a single character picture and text. After the user inputs a character photo and text, the system generates a video that narrates the content. However, this solution has the problem of taking a long time to generate lip animation. For example, it may take 1 to 2 minutes to generate a 10-second animation, which is difficult to meet real-time requirements. In addition, the output is a fixed video file that cannot dynamically respond to voice input and lacks interactivity.

[0004] Therefore, it is necessary to propose a method for generating interactive lip-syncing animation of two characters and related equipment to at least solve the problems of poor flexibility and long generation time of interactive lip-syncing animation generation. Summary of the invention

[0005] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0006] In a first aspect, the present invention provides a method for generating a two-dimensional character interactive lip animation, the method comprising:

[0007] Obtaining a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information;

[0008] Generate a character lip-sync resource image based on the two-dimensional character photo and predefined lip-sync video data;

[0009] Based on the above-mentioned character lip shape resource images, resource optimization operations are performed to adaptively obtain lip shape area images;

[0010] Acquire pronunciation phonemes based on the input information and perform phoneme analysis operations to acquire lip animations corresponding to the pronunciation phonemes;

[0011] The lip area image and the lip animation are synchronized to generate a two-dimensional character interactive lip animation.

[0012] In a feasible implementation manner, the above-mentioned generating a character lip shape resource picture based on the above-mentioned two-dimensional character photo and predefined lip shape video data includes:

[0013] The above-mentioned two-dimensional character photo and the above-mentioned predefined lip shape video data are input into the generative AI model to generate the above-mentioned character lip shape resource picture, wherein the above-mentioned generative AI model is generated based on implicit key points of the face, and the above-mentioned character lip shape resource picture includes basic lip shape resource picture and transition lip shape resource picture.

[0014] In a feasible implementation manner, a smooth transition is performed between the basic lip shape resource image and its corresponding transition lip shape resource image based on a FILM method.

[0015] In a feasible implementation, the resource optimization operation is performed based on the above-mentioned character lip shape resource image to adaptively obtain the lip shape area image, including:

[0016] Based on the above-mentioned character lip shape resource image, facial key point detection is performed to extract the mouth area for resource optimization operations to obtain the lip shape area image.

[0017] In a feasible implementation manner, the above-mentioned obtaining the pronunciation phonemes based on the above-mentioned input information and performing the phoneme analysis operation to obtain the lip animation corresponding to the above-mentioned pronunciation phonemes includes:

[0018] Analyze the voice information based on the input information according to the Azure voice analysis method to obtain pronunciation phonemes, wherein the pronunciation phonemes include phoneme categories and pronunciation durations;

[0019] The corresponding lip animation is determined based on the phoneme category and the pronunciation duration.

[0020] In a feasible implementation manner, the synchronizing operation of the lip area image and the lip animation to generate a two-dimensional character interactive lip animation includes:

[0021] A synchronization operation is performed based on the time information according to the lip area image and the lip animation to generate a two-dimensional character interactive lip animation.

[0022] In a feasible implementation, it also includes:

[0023] Interpolation frames are generated for the above-mentioned two-dimensional character interactive lip-syncing animation based on the pronunciation duration and the time difference between the lip-syncing resources, so as to make the above-mentioned two-dimensional character interactive lip-syncing animation smoother.

[0024] In a second aspect, the present invention further provides a two-dimensional character interactive lip animation generation device, comprising:

[0025] A first acquisition unit, configured to acquire a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information;

[0026] A first generating unit, configured to generate a character lip shape resource image based on the two-dimensional character photo and the predefined lip shape video data;

[0027] A second acquisition unit is used to perform a resource optimization operation based on the above-mentioned character lip shape resource image to adaptively acquire a lip shape area image;

[0028] A third acquisition unit, configured to acquire pronunciation phonemes based on the input information and perform phoneme analysis operations to acquire lip animations corresponding to the pronunciation phonemes;

[0029] The second generating unit is used to synchronize the lip-shaped area image and the lip-shaped animation to generate a two-dimensional character interactive lip-shaped animation.

[0030] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the method for generating a two-dimensional character interactive lip-syncing animation as described in any one of the above-mentioned first aspects when executing the computer program stored in the memory.

[0031] In a fourth aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a two-dimensional character interactive lip animation according to any one of the above items in the first aspect is implemented.

[0032] In summary, this application can generate interactive characters based on any two-dimensional character photos without pre-making characters or animation content. By inputting the character photos and standard lip-sync videos uploaded by the user, the generative model is used to extract mouth features and generate lip-sync resource pictures consistent with the character style. Compared with the limitation of DeepBrainAI relying on pre-made characters, this application greatly improves flexibility, and users can quickly create exclusive virtual character animations, which are suitable for personalized scene requirements. This application combines FILM technology with pre-trained generative models to achieve rapid generation of lip-sync animations and support instant dialogue and interaction. By automatically generating lip-sync resources and using optimization algorithms to crop the mouth area, the computational burden of the generation process is effectively reduced. At the same time, by dynamically extracting voice phonemes (Viseme) to generate a timeline, it is played synchronously with the lip-sync animation to ensure that the animation matches the input content in real time. Compared with SadTalker's fixed video generation method, this application can respond to user input in real time, generate highly interactive lip-sync animations, and significantly shorten the generation time. This application supports two interactive modes: text input and voice input. The pronunciation phonemes are extracted through voice analysis technology (such as Azure voice service) and synchronized with the lip-sync animation in real time. The output animation is not a fixed video file, but an animation that dynamically responds to input information, supporting instant dialogue and personalized interaction. This application introduces FILM technology in animation generation, generates smooth transition frames between key frames, and makes the lip shape change more natural. By analyzing the duration and order of phonemes, the timeline of the lip shape animation is dynamically generated, and the corresponding lip shape area picture is selected and played according to the timeline to ensure that the voice content is accurately synchronized with the animation rhythm. After generating the lip shape resource, this application crops the mouth area through an optimization algorithm based on facial key point detection, retaining only the necessary mouth feature pictures, significantly reducing the storage occupancy. At the same time, the dynamically generated lip shape animation avoids the high storage cost of static video files. Compared with the high storage requirements of video generation (such as the static video file output of SadTalker), this application is more lightweight and suitable for application scenarios that require large-scale resource generation and transmission. This application is based on a predefined multilingual phoneme (Viseme) library, supports the generation of lip shape resources corresponding to common pronunciations (such as a, e, u, etc.), and can adapt to voice input in multiple languages. This enables the application to have cross-language and cross-cultural application capabilities, further expanding its scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0034] Figure 1A schematic diagram of the process of generating a two-dimensional character interactive lip animation method provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the principle of a method for generating a two-dimensional character interactive lip-sync animation provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the principle of automatic generation of lip shapes of a two-dimensional character provided in an embodiment of the present application;

[0037] Figure 4 A two-dimensional character interactive lip animation generation device provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of an electronic device for generating interactive lip-sync animation of a two-dimensional character provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0040] See also Figure 1 Schematic diagram of the process of generating a two-dimensional character interactive lip animation in an embodiment of the present application. Figure 2 A schematic diagram of the principle of a method for generating a two-dimensional character interactive lip-sync animation provided in an embodiment of the present application is provided. Figure 3 A schematic diagram of the principle of automatic generation of lip shape of a two-dimensional character provided in an embodiment of the present application, the method includes:

[0041] S110, obtaining a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information;

[0042] Exemplarily, the two-dimensional character photo is a character picture provided by the user, such as a virtual image or a cartoon avatar, which is used to generate the basic image of the lip-sync animation. The input information can be the user's text input (such as chat text) or voice input (such as the user's voice). The two-dimensional character photo and the input information provide two sources of information for the generation of the lip-sync animation. The two-dimensional character photo is used to determine the appearance characteristics of the character, and the voice input information is used to drive the generation of the lip-sync animation.

[0043] S120, generating a character lip shape resource picture based on the two-dimensional character photo and the predefined lip shape video data;

[0044] For example, use the character photo uploaded by the user and the predefined standard lip shape video (including different lip shape changes when pronouncing). Use the generative model to extract the character's mouth features and generate lip shape resource images consistent with the character's style. Each common pronunciation (phoneme) corresponds to a set of lip shape images (such as "a", "e", "u", etc.). By automatically generating the character's lip shape resources, ensure that the lip shape is consistent with the character's photo style, while also having a variety of lip shape changes.

[0045] S130, performing a resource optimization operation based on the above-mentioned character lip shape resource image to adaptively obtain a lip shape area image;

[0046] Exemplarily, the generated character lip-shaped resource image is automatically cropped through an optimization algorithm (such as based on facial key point detection) to retain only the mouth area. The cropping range is adaptively adjusted according to the size and shape of the character's mouth features. This step can reduce the storage usage of the generated lip-shaped image resources while ensuring that the retained mouth area can accurately present the animation effect.

[0047] S140, obtaining pronunciation phonemes based on the input information and performing phoneme analysis operations to obtain lip animations corresponding to the pronunciation phonemes;

[0048] Exemplarily, text input is used to convert text into phonetic phonemes, and the pronunciation duration and order are analyzed through a language model (such as a TTS system). Voice input directly extracts phoneme sequences and their corresponding pronunciation times through voice analysis technology (such as Azure Voice Service). The correspondence between phonemes and mouth shapes (Viseme) is obtained to generate a timeline for the lip animation. The language features in the input information are extracted, and the pronunciation details are analyzed to provide an accurate basis for the subsequent generation of lip animation.

[0049] S150: Synchronize the lip area image and the lip animation to generate a two-dimensional character interactive lip animation.

[0050] Exemplarily, according to the phoneme timeline generated in step S140, a lip area image corresponding to the phoneme is selected. The selected lip image is played synchronously in the order of the timeline to generate a lip animation that matches the speech content. A transition frame (such as through FILM technology) can be inserted to make the animation more natural and smooth. A two-dimensional character lip animation driven by text or voice input is implemented to ensure that the lip animation is accurately synchronized with the content and rhythm of the speech pronunciation.

[0051] In summary, this application can generate interactive characters based on any two-dimensional character photos without pre-making characters or animation content. By inputting the character photos and standard lip-sync videos uploaded by the user, the generative model is used to extract mouth features and generate lip-sync resource pictures consistent with the character style. Compared with the limitation of DeepBrainAI relying on pre-made characters, this application greatly improves flexibility, and users can quickly create exclusive virtual character animations, which are suitable for personalized scene requirements. This application combines FILM technology with pre-trained generative models to achieve rapid generation of lip-sync animations and support instant dialogue and interaction. By automatically generating lip-sync resources and using optimization algorithms to crop the mouth area, the computational burden of the generation process is effectively reduced. At the same time, by dynamically extracting voice phonemes (Viseme) to generate a timeline, it is played synchronously with the lip-sync animation to ensure that the animation matches the input content in real time. Compared with SadTalker's fixed video generation method, this application can respond to user input in real time, generate highly interactive lip-sync animations, and significantly shorten the generation time. This application supports two interactive modes: text input and voice input. The pronunciation phonemes are extracted through voice analysis technology (such as Azure voice service) and synchronized with the lip-sync animation in real time. The output animation is not a fixed video file, but an animation that dynamically responds to input information, supporting instant dialogue and personalized interaction. This application introduces FILM technology in animation generation, generates smooth transition frames between key frames, and makes the lip shape change more natural. By analyzing the duration and order of phonemes, the timeline of the lip shape animation is dynamically generated, and the corresponding lip shape area picture is selected and played according to the timeline to ensure that the voice content is accurately synchronized with the animation rhythm. After generating the lip shape resource, this application crops the mouth area through an optimization algorithm based on facial key point detection, retaining only the necessary mouth feature pictures, significantly reducing the storage occupancy. At the same time, the dynamically generated lip shape animation avoids the high storage cost of static video files. Compared with the high storage requirements of video generation (such as the static video file output of SadTalker), this application is more lightweight and suitable for application scenarios that require large-scale resource generation and transmission. This application is based on a predefined multilingual phoneme (Viseme) library, supports the generation of lip shape resources corresponding to common pronunciations (such as a, e, u, etc.), and can adapt to voice input in multiple languages. This enables the application to have cross-language and cross-cultural application capabilities, further expanding its scope of application.

[0052] In some examples, generating a character lip shape resource image based on the two-dimensional character photo and the predefined lip shape video data includes:

[0053] The above-mentioned two-dimensional character photo and the above-mentioned predefined lip shape video data are input into the generative AI model to generate the above-mentioned character lip shape resource picture, wherein the above-mentioned generative AI model is generated based on implicit key points of the face, and the above-mentioned character lip shape resource picture includes basic lip shape resource picture and transition lip shape resource picture.

[0054] Exemplarily, the two-dimensional character photo is a character image (such as a virtual image or cartoon avatar) provided by the user, which is used to determine the appearance characteristics of the character. This photo provides a visual basis for generating the character's lip shape resource image. At the same time, the predefined lip shape video data contains standardized lip shape changes, covering the pronunciation corresponding lip shapes of a variety of common phonemes (such as a, e, u, etc.). By combining these two types of input, the solution can generate diversified lip shape resources for different characters that match their styles.

[0055] The core of this solution is a generative AI model based on implicit facial key points. The pre-training process of this model has learned the distribution and change rules of a large number of key points of human faces, so it can extract features from the input two-dimensional character photos and combine them with predefined lip-sync video data to generate lip-sync resource images consistent with the input character style.

[0056] The generation process includes:

[0057] S210, facial implicit key point analysis: The generative AI model first analyzes the facial features of the two-dimensional character photo and extracts implicit key points including the mouth area.

[0058] S220, key point and lip shape matching: matching the facial features of the character with the lip shape corresponding to each phoneme in the predefined lip shape video data.

[0059] S230, generating resource images: generating basic lip shape resource images and transition lip shape resource images of the character according to the matching results.

[0060] The character lip shape resource images generated by this solution include two parts: basic lip shape resource images and transition lip shape resource images.

[0061] The basic lip-sync resource images correspond to the key pronunciation lip shapes in the predefined lip-sync video data (such as "a", "e", "u", etc.). Each image accurately matches the pronunciation characteristics of a phoneme and is consistent with the character's mouth shape to ensure visual consistency when the animation is generated.

[0062] Transition lip-sync resource images are interpolated images generated by the generative AI model based on the conversion logic between basic lip-sync resource images. These transition lip-sync resource images make the lip-sync changes in the animation smoother and more natural, avoiding the harsh effects caused by direct switching.

[0063] Implicit key points can accurately express the morphology and dynamic features of the face, ensuring that the generated lip-sync resource images are highly consistent with the characteristics of the character. Compared with traditional explicit key points, this method is more suitable for diverse character designs and can automatically adapt to two-dimensional characters of different styles. This method not only considers the key frames of pronunciation (basic lip-sync resource images), but also generates transition frames between pronunciations (transition lip-sync resource images), thereby meeting the animation requirements of different pronunciation combinations.

[0064] Through this process, the automatic generation of character lip-sync resource images from 2D character photos and predefined lip-sync video data is achieved. These resource images provide a high-quality foundation for subsequent lip-sync animation generation, which is not only consistent in style and natural and smooth, but also greatly improves the efficiency of animation generation.

[0065] In some examples, a smooth transition is performed between the basic lip shape resource image and its corresponding transition lip shape resource image based on a FILM method.

[0066] Exemplarily, the basic lip resource image and the corresponding transition lip resource image are smoothly transitioned through the FILM method (Frame Interpolation for Large Motion) to generate a natural and smooth lip animation. The basic lip resource image is generated by a generative AI model based on two-dimensional character photos and predefined lip video data, representing the character's lip shape under specific pronunciation phonemes (such as "a", "e", "u", etc.). Each basic lip resource image accurately reflects the static lip shape characteristics of a single phoneme.

[0067] The transition lip-sync resource image represents the intermediate state in the transition process from one basic lip-sync resource image to another basic lip-sync resource image. It is generated by the generative AI model based on the dynamic characteristics between phonemes to ensure that the lip-sync changes in the animation are more natural.

[0068] The FILM method (Frame Interpolation for Large Motion) is a technique for interpolating frames in large motion scenes. In this solution, the FILM method is used to generate an intermediate frame (transition frame) between a basic lip-sync resource image and its corresponding transition lip-sync resource image.

[0069] The generation process of the FILM method includes:

[0070] S310, input two pictures: input a basic lip shape resource picture and a target transition lip shape resource picture.

[0071] S320, Motion vector analysis: The FILM method uses deep learning technology to analyze the motion vector between two pictures and capture the changing trend of the key points of the mouth.

[0072] S330, interpolating to generate transition frames: Calculating the details of the intermediate frames according to the motion vectors, generating multiple smooth transition frames, so that the lip animation is naturally connected between two key frames.

[0073] The transition between the basic lip resource images of the transition frames generated by the FILM method is no longer abrupt, but is gradually completed through multiple delicate transition frames. For example, from the basic lip resource image of "a" to the basic lip resource image of "e", the opening and closing of the mouth and the curve changes are gradually presented. The FILM method can accurately simulate the movement trajectory of the mouth, making the animation smooth and realistic. The transition frames generated by interpolation are visually consistent with the style of the two-dimensional character photos, ensuring the unity of the overall animation effect.

[0074] In summary, this solution uses the FILM method to generate smooth transition frames, providing an efficient and natural solution for the animation transition between basic lip-sync resource images and transition lip-sync resource images, ensuring that the visual effect of the two-dimensional character interactive lip-sync animation reaches the best state.

[0075] In some examples, the resource optimization operation based on the above-mentioned character lip shape resource image is performed to adaptively obtain the lip shape area image, including:

[0076] Based on the above-mentioned character lip shape resource image, facial key point detection is performed to extract the mouth area for resource optimization operations to obtain the lip shape area image.

[0077] Exemplarily, the process of adaptively acquiring lip area images by performing resource optimization operations based on the above-mentioned character lip shape resource images specifically includes performing facial key point detection based on the above-mentioned character lip shape resource images to extract the mouth area, thereby optimizing the character lip shape resource images.

[0078] The character lip-sync resource images are generated by the generative AI model based on 2D character photos and predefined lip-sync video data. They include basic lip-sync resource images and transitional lip-sync resource images, which are used to build the basis for lip-sync animation. The character lip-sync resource images are optimized to extract the mouth area and generate lighter lip-sync area images, reducing storage usage and transmission costs while retaining the key parts of the animation effect.

[0079] Use the facial key point detection algorithm to analyze the facial structure in the character's lip shape resource image. The algorithm accurately locates the boundaries of the mouth area, including the positions of specific points such as the upper edge of the lips, the lower edge of the lips, and the corners of the mouth. Facial key point detection can generate a rectangular or polygonal bounding box of the character's mouth area and mark the lip shape area to be extracted. Based on the results of facial key point detection, the character's lip shape resource image is adaptively cropped to retain only the mouth area. The extraction range is dynamically adjusted according to the character's mouth shape, size, and key point distribution to ensure that the cropped lip shape area image can accurately reflect the changes in the mouth. The optimized lip shape area image only contains mouth-related content, such as lips, mouth opening degree, mouth shape, etc., and removes unnecessary background or other facial areas.

[0080] The size of the optimized lip area image is significantly reduced, and only the lip-related content is retained, thereby greatly reducing the storage capacity occupied. The optimized image still maintains the same style as the original character, ensuring that the animation is natural and realistic. By detecting facial key points and extracting the mouth area based on the character's lip resource image, the solution realizes the resource optimization operation of the character's lip resource image. The optimized lip area image is not only lightweight and efficient to store, but also can accurately reflect the dynamic characteristics of the mouth, laying a high-quality foundation for the subsequent generation of 2D character interactive lip animation.

[0081] In some examples, the above-mentioned obtaining the pronunciation phonemes based on the above-mentioned input information and performing the phoneme analysis operation to obtain the lip animation corresponding to the above-mentioned pronunciation phonemes includes:

[0082] Analyze the voice information based on the input information according to the Azure voice analysis method to obtain pronunciation phonemes, wherein the pronunciation phonemes include phoneme categories and pronunciation durations;

[0083] The corresponding lip animation is determined based on the phoneme category and the pronunciation duration.

[0084] Exemplarily, the input information can be a voice input provided by the user, such as a natural language voice segment. The voice input information is the direct driving data source for generating lip animation. The input voice information is parsed using the Azure voice analysis method. The category (such as a, e, u, k, etc.) and the corresponding pronunciation duration of each pronunciation phoneme (Phoneme) in the voice signal are analyzed. Through the Azure voice analysis method, the phonemes in the voice signal are classified into predefined phoneme categories. For example, the vowels "a" and consonants "p" detected in the voice input are classified into corresponding phoneme categories. The Azure voice analysis method simultaneously identifies the pronunciation duration (in milliseconds) of each phoneme. Example: A phoneme "a" in the input voice lasts for 300 milliseconds, and the consonant "p" lasts for 150 milliseconds. Each phoneme category corresponds to one or more specific lip resource images (such as the mouth shape of "a" is a wide open mouth, and the mouth shape of "p" is closed lips). The system selects the corresponding basic lip resource image from the character lip resource image according to the phoneme category. Combined with the pronunciation duration of the phonemes, the selected basic lip resource images and the possible transitional lip resource images are sorted according to the timeline. The system combines the character's lip resource images and transition frames into a complete lip animation sequence based on the timeline generated by the phoneme category and pronunciation duration.

[0085] Through synchronization operations based on time information, this solution achieves accurate matching of lip area images and lip animations. The generated 2D character interaction lip animations can accurately reflect the speech content and rhythm, and the animation is natural and smooth. It is suitable for scenes such as virtual character interaction, intelligent customer service, and virtual anchors, significantly improving the user experience.

[0086] In summary, this scheme generates a two-dimensional character interactive lip animation with high real-time performance and strong naturalness by performing synchronization operations on the lip area image and the lip animation based on time information, which has strong practical value and technical advantages.

[0087] In some examples, the synchronizing operation of the lip area image and the lip animation to generate a two-dimensional character interactive lip animation includes:

[0088] A synchronization operation is performed based on the above-mentioned lip area image and the above-mentioned lip animation based on time information to generate a two-dimensional character interactive lip animation.

[0089] In some examples, this also includes:

[0090] Interpolation frames are generated for the above-mentioned two-dimensional character interactive lip-syncing animation based on the pronunciation duration and the time difference between the lip-syncing resources, so as to make the above-mentioned two-dimensional character interactive lip-syncing animation smoother.

[0091] Exemplarily, the lip area image is obtained from the character lip resource image through resource optimization operation, and only contains the mouth area, ensuring that the animation data is lightweight and efficient. The lip animation is generated by phoneme analysis operation, including phoneme category, pronunciation duration and timeline information. The system selects the corresponding image from the lip area image according to the phoneme category. For example: the phoneme "a" corresponds to the lip area image of the open mouth. The phoneme "p" corresponds to the lip area image of the closed mouth.

[0092] Control the display based on the pronunciation duration: According to the time information of the lip animation, control the display duration of each lip area image. For example: if the phoneme "a" lasts for 300 milliseconds, the lip area image of "a" will be displayed for 300 milliseconds. If the phoneme "p" lasts for 150 milliseconds, the lip area image of "p" will be displayed for 150 milliseconds.

[0093] Arrange the lip area images in order according to the phoneme sequence to form a basic two-dimensional character interactive lip animation.

[0094] There may be significant differences in the lip area images of different phoneme categories, and direct switching may result in stiff or unnatural animation. The role of the interpolation frame is to fill the time difference between the two lip area images and make a smooth transition. For the lip resource images of continuous phonemes, the time range that the interpolation frame needs to cover is calculated based on their pronunciation duration and switching time. The number and display time of the interpolation frames are dynamically adjusted according to the phoneme pronunciation duration and the amplitude of the change between the lip resources. For example: a small mouth change (such as "m" to "n") may only require one frame of interpolation. Large mouth changes (such as "a" to "p") require the generation of multiple frames of interpolation.

[0095] The lip-sync area images are displayed in the order of the lip-sync animation's timeline. The generated interpolation frames are inserted between different lip-sync area images to ensure a natural and smooth animation. Each frame of animation (including the base image frame and the interpolation frame) strictly follows the timing information of the lip-sync animation to achieve precise matching with the rhythm of the speech content.

[0096] The synchronization operation based on the time information ensures that the lip animation strictly matches the speech rhythm.

[0097] In summary, this scheme successfully solves the problem of insufficient fluency in traditional lip animation generation by synchronizing the time information of the lip area image and the lip animation, and generating interpolated frames based on the pronunciation duration and the time difference between the lip resources. It provides an efficient, natural and real-time lip animation generation method with broad application value.

[0098] See also Figure 4 An embodiment of the two-dimensional character interactive lip animation generation device 20 in the embodiment of the present application may include:

[0099] A first acquisition unit 21 is used to acquire a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information;

[0100] A first generating unit 22, configured to generate a character lip shape resource image based on the two-dimensional character photo and the predefined lip shape video data;

[0101] A second acquisition unit 23 is used to perform a resource optimization operation based on the above-mentioned character lip shape resource image to adaptively acquire a lip shape area image;

[0102] A third acquisition unit 24 is used to acquire pronunciation phonemes based on the input information and perform phoneme analysis operations to acquire lip animations corresponding to the pronunciation phonemes;

[0103] The second generating unit 25 is used to synchronize the lip-shaped area image and the lip-shaped animation to generate a two-dimensional character interactive lip-shaped animation.

[0104] like Figure 5 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 511 stored in the memory 320 and executable on the processor, and when the processor 320 executes the computer program 311, the steps of any method for generating the above-mentioned two-dimensional character interactive lip animation are implemented.

[0105] Since the electronic device introduced in this embodiment is a device used to implement a two-dimensional character interactive lip animation generation device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation method of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0106] In the specific implementation process, when the computer program 311 is executed by the processor, it can achieve Figure 1 Any implementation manner in the corresponding embodiments.

[0107] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] The present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device is caused to execute the following Figure 1 The process of the method for generating interactive lip-sync animation of a two-dimensional character in the corresponding embodiment.

[0113] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0119] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a two-dimensional character interactive lip animation, characterized in that: include: Acquire a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information; Generate a character lip shape resource picture based on the two-dimensional character photo and the predefined lip shape video data; Performing resource optimization operations based on the character's lip shape resource images to adaptively obtain lip shape area images; Acquire pronunciation phonemes based on the input information and perform phoneme analysis operations to acquire lip animations corresponding to the pronunciation phonemes; The lip-shaped area image and the lip-shaped animation are synchronized to generate a two-dimensional character interactive lip-shaped animation.

2. The method for generating 2D character interactive lip animation according to claim 1, characterized in that: The generating of the character lip shape resource picture based on the two-dimensional character photo and the predefined lip shape video data comprises: The two-dimensional character photo and the predefined lip shape video data are input into a generative AI model to generate the character lip shape resource image, wherein the generative AI model is generated based on implicit facial key points, and the character lip shape resource image includes a basic lip shape resource image and a transition lip shape resource image.

3. The method for generating 2D character interactive lip-sync animation according to claim 2, characterized in that: The basic lip shape resource image and its corresponding transitional lip shape resource image are smoothly transitioned based on the FILM method.

4. The method for generating 2D character interactive lip-sync animation according to claim 1, characterized in that: The performing of resource optimization operation based on the character's lip shape resource image to adaptively obtain the lip shape area image includes: Based on the character's lip shape resource image, facial key point detection is performed to extract the mouth area for resource optimization operations to obtain a lip shape area image.

5. The method for generating 2D character interactive lip animation according to claim 1, characterized in that: The step of acquiring the pronunciation phonemes based on the input information and performing a phoneme analysis operation to acquire the lip animation corresponding to the pronunciation phonemes includes: Analyzing the voice information according to the Azure voice analysis method based on the input information to obtain pronunciation phonemes, wherein the pronunciation phonemes include phoneme categories and pronunciation durations; The corresponding lip animation is determined based on the phoneme category and the pronunciation duration.

6. The method for generating 2D character interactive lip animation according to claim 1, characterized in that: The synchronizing operation of the lip area image and the lip animation to generate a two-dimensional character interactive lip animation includes: A synchronization operation is performed based on the lip area image and the lip animation based on time information to generate a two-dimensional character interactive lip animation.

7. The method for generating 2D character interactive lip animation according to claim 6, characterized in that: Also includes: An interpolation frame is generated for the two-dimensional character interactive lip animation based on the pronunciation duration and the time difference between the lip resources, so as to make the two-dimensional character interactive lip animation smoother.

8. A two-dimensional character interactive lip animation generation device, characterized in that: include: A first acquisition unit, configured to acquire a two-dimensional character photo and input information of a target user, wherein the input information includes text input information and / or voice input information; A first generating unit, configured to generate a character lip shape resource picture based on the two-dimensional character photo and predefined lip shape video data; A second acquisition unit is used to perform a resource optimization operation based on the character's lip shape resource image to adaptively acquire a lip shape area image; A third acquisition unit, configured to acquire pronunciation phonemes based on the input information and perform a phoneme analysis operation to acquire a lip animation corresponding to the pronunciation phonemes; The second generating unit is used to synchronize the lip-shaped area image and the lip-shaped animation to generate a two-dimensional character interactive lip-shaped animation.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the two-dimensional character interactive lip animation generation method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a two-dimensional character interactive lip-sync animation are implemented as described in any one of claims 1 to 7.

Images