Method and system for automatic generation of AI animation synchronized with lip sync based on voice dubbing driving

Abstract

The application provides an AI animation automatic generation and lip synchronization method and system based on voice driving, and relates to the technical field of AI animation automatic generation. The application collects audio data and video data, and when the audio duration is greater than the video duration, encodes the tail frame of the video as a latent space condition seed. Then, based on the condition seed, the character features and the picture dynamics, an extended frame sequence is generated by using a diffusion model for prediction. Subsequently, the video frame sequence is obtained by calculating the adjacent frame movement vector, and the binary mask sequence is generated by recognizing the face. Then, the audio is converted into phoneme features to determine the target lip shape parameters, and the audio energy is mapped into a displacement vector sequence. Then, based on the two, the mouth area marked by the binary mask is updated frame by frame to obtain a synchronous video sequence. Finally, the mouth area is matched and recombined for feature reconstruction by using a VQ-GAN repair network, and then fused to obtain a target synchronous video, so that the AI animation automatic generation with accurate audio-visual synchronization can be realized.

Description

Technical Field

[0001] This application relates to the technical field of AI animation automatic generation, and in particular to a method and system for AI animation automatic generation and lip-syncing based on voice-over driven technology. Background Technology

[0002] With the rapid development of artificial intelligence technology, audio-driven facial animation generation technology has been widely used in many fields such as education, entertainment, publicity, and news broadcasting. Its core demand is to quickly and efficiently use audio to create virtual human videos, reduce manual recording costs, and improve content production efficiency.

[0003] Currently, the existing audio-driven video generation methods are mainly as follows: video dubbing technology achieves synchronization between lip shape and new audio by editing the mouth area. For example, audio-to-lip shape technology achieves lip shape driving by constructing an end-to-end audio-visual mapping model.

[0004] However, traditional lip-syncing technology only performs local editing on the mouth area, which easily leads to lip-sync drift and pixel tearing when the character turns their head or moves over a large range. In addition, the generated mouth area often appears blurry or has color differences at the edges. Furthermore, existing generation models are mostly short drafts of fixed 5-10 seconds. When the dubbing is longer, the scene can only be looped abruptly, which can cause interruption of action or content gaps. AI-generated long videos usually have a low frame rate, and using them directly for lip-syncing synthesis will result in a sense of audio-visual tearing, with smooth sound but choppy visuals. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for automatic generation and lip-syncing of AI animation based on voice-over driven technology, so as to solve the problem that it is difficult to balance the naturalness of the picture and the accuracy of audio-visual synchronization in the animation generation process in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for automatic generation and lip-syncing of AI animation based on voice-over driven technology, comprising: Collect audio and video data, wherein the audio data includes voice duration data and audio energy data, and the video data includes human characteristic information and dynamic image data; If the difference between the duration of the audio data and the duration of the video data is greater than zero, the last frame image of the video data is encoded as a condition seed for the latent space. Based on the conditional seed, the character feature information, and the dynamic data of the scene, a pre-trained diffusion model is used to make predictions and generate an extended frame sequence. By calculating the motion vector between two adjacent frames in the extended frame sequence, a video frame sequence is obtained. From the video frame sequence, the face region is identified and a binary mask sequence is generated. The audio data is converted into phoneme features, and the target lip shape parameters are determined according to the preset mapping relationship between the phoneme features and lip shape parameters. The audio energy data is then mapped into a displacement vector sequence. Based on the target lip shape parameters and the displacement vector sequence, the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence are updated frame by frame to obtain a synchronized video sequence. Using a trained repair network based on the VQ-GAN architecture (Vector Quantized-Generative Adversarial Network, VQ-GAN), the mouth region in the synchronized video sequence is first encoded and matched, then its features are reconstructed. The reconstructed mouth region is then fused with the synchronized video sequence to obtain the target synchronized video.

[0007] Optionally, the method further includes: Speech recognition is performed on the audio data to obtain action semantic information corresponding to the audio data; Based on the action semantic information, the target action tag is determined using a preset mapping relationship between action semantics and action tags; The step of generating an extended frame sequence by using a pre-trained diffusion model to predict based on the conditional seed, the character feature information, and the dynamic data of the scene includes: The conditional seed, the character feature information, the dynamic data of the scene, and the target action label are concatenated into vectors to obtain the diffusion input vector; The diffusion input vector is input into a trained diffusion model based on the Transformer architecture, and non-repetitive content prediction is performed in the latent space to obtain an extended frame sequence. The diffusion model is trained using samples including historical diffusion input vectors and corresponding real image frame sequences based on the Transformer architecture.

[0008] Optionally, when the difference between the duration of the speech data and the duration of the video data is greater than zero, encoding the last frame image of the video data as a conditional seed for the latent space includes: Calculate the difference between the duration of the audio data and the duration of the video data. If the difference is greater than zero, determine the last frame of the video data as the tail frame. According to the preset grid division rules, the pixel matrix of the tail frame image is divided into multiple image blocks; Using a preset feature index table, the reference image block most similar to each of the image blocks is found to obtain the corresponding index value. The feature index table includes multiple reference image blocks and the index value corresponding to each reference image block. The index value corresponds to a pre-generated feature vector in the latent space. All the index values ​​are combined according to the order in which each image block is arranged in the pixel matrix to obtain an index value sequence, and the index value sequence is used as a condition seed.

[0009] Optionally, the step of obtaining a video frame sequence by calculating the motion vector between two adjacent frames in the extended frame sequence, identifying face regions from the video frame sequence, and generating a binary mask sequence includes: The displacement direction and displacement distance of each pixel between two adjacent frames in the extended frame sequence are calculated to obtain the motion vector, and the number of intermediate images to be inserted is determined according to the ratio of the preset target frame rate to the original frame rate of the extended frame sequence. Based on the time ratio coefficient corresponding to the motion vector and the number of intermediate images, the mapping position of each pixel in the intermediate image in two adjacent frames is calculated, and the intermediate image is generated by weighted averaging of the pixel values ​​at the mapping positions. The intermediate image is then inserted between two adjacent frames in the extended frame sequence to obtain a video frame sequence. Face detection is performed on the video frame sequence to obtain at least one face identifier, and voiceprint recognition is performed on the audio data to obtain a role identifier; Based on the role identifier, a target face identifier is determined from at least one of the face identifiers. Pixels within the mouth boundary contour of the face region corresponding to the target face identifier are marked as a first value, and pixels outside the mouth boundary contour are marked as a second value, thus obtaining a binary mask corresponding to each frame image. All the binary masks are combined in frame order to obtain a binary mask sequence.

[0010] Optionally, the step of converting the audio data into phoneme features, determining the target lip shape parameters according to the preset mapping relationship between phoneme features and lip shape parameters, and mapping the audio energy data into a displacement vector sequence includes: The audio data is divided into frames according to a preset time window length to obtain an audio frame sequence. The audio frame sequence is then subjected to a Fourier transform to obtain the amplitude of each audio frame at each frequency point. The amplitudes are then grouped according to a preset frequency band division rule to form the phoneme feature vector of each audio frame. The phoneme feature vector with the smallest Euclidean distance from the preset reference feature vector is selected as the target phoneme. The degree of mouth opening and the roundness of the lips of the target phoneme are determined from the preset lookup table. The degree of mouth opening and the roundness of the lips are arranged in chronological order to obtain the target lip shape parameters. Calculate the average value of the instantaneous energy data of the audio energy data at a preset number of time points before and after each audio frame's corresponding time point to obtain the smoothed energy value of each audio frame; The smoothing energy value is multiplied by a preset scaling factor to obtain the displacement amplitude value of each audio frame. All the displacement amplitude values ​​are then arranged in the time order of the audio frames to obtain a displacement vector sequence.

[0011] Optionally, the step of updating the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame based on the target lip shape parameters and the displacement vector sequence to obtain a synchronized video sequence includes: Mouth key point detection is performed on the video frame sequence to obtain an initial coordinate set, which includes the pixel coordinates of the upper lip control point, lower lip control point, left corner of mouth control point, and right corner of mouth control point; The product of the degree of mouth opening of the target lip shape parameter and the first preset coefficient is added to the vertical coordinate of the upper lip control point and the lower lip control point. The product of the lip roundness value of the target lip shape parameter and the second preset coefficient is added to the horizontal coordinate of the left corner of the mouth control point and the right corner of the mouth control point to obtain the intermediate coordinate set. Using the displacement amplitude value as the disturbance intensity, a random offset is added to each control point in the intermediate coordinate set to obtain the target coordinate set; Based on the correspondence between the initial coordinate set and the target coordinate set, coordinate mapping is performed on each pixel in the mouth region marked by the binary mask sequence in the video frame sequence to obtain the updated current frame image. All updated current frame images are then arranged in the order in the video frame sequence to obtain a synchronized video sequence.

[0012] Optionally, the step of using a trained VQ-GAN-based repair network to perform prior encoding matching on the mouth region in the synchronized video sequence, followed by feature reconstruction, and then fusing the reconstructed mouth region with the synchronized video sequence to obtain the target synchronized video, includes: Based on the mouth region marked by the binary mask in the binary mask sequence, mouth image blocks are cropped from each frame of the synchronized video sequence. The encoder of the repair network based on the VQ-GAN architecture is used to map the mouth image blocks into input feature vectors in the latent space. In the latent feature space, the similarity between the input feature vector and each reference feature vector in the pre-stored prior coding library is calculated using the Swin-Transformer operator. The top K reference feature vectors with the highest similarity are then weighted and summed according to their inner product values ​​to obtain the recombined feature vector, where K is a positive integer. Using the decoder of the repair network, the reconstructed feature vector is mapped back to the image space to obtain the repaired mouth image block; The repaired mouth image block is overlaid onto the mouth region of the corresponding frame image in the synchronized video sequence, and the boundary pixels of the mouth region are weighted and averaged to obtain the target synchronized video.

[0013] Secondly, this application provides a voice-driven AI animation automatic generation and lip-syncing system, including: The acquisition module is used to acquire audio data and video data. The audio data includes voice duration data and audio energy data, and the video data includes human feature information and dynamic image data. The encoding module is used to encode the last frame image of the video data as a conditional seed in the latent space when the difference between the duration of the speech data and the duration of the video data is greater than zero. The prediction module is used to predict based on the conditional seed, the character feature information and the dynamic data of the scene using a pre-trained diffusion model, generate an extended frame sequence, obtain a video frame sequence by calculating the motion vector between two adjacent frames in the extended frame sequence, identify the face region from the video frame sequence and generate a binary mask sequence. The determination module is used to convert the audio data into phoneme features, determine the target lip shape parameters according to the preset mapping relationship between phoneme features and lip shape parameters, and map the audio energy data into a displacement vector sequence; The update module is used to update the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame based on the target lip shape parameters and the displacement vector sequence, so as to obtain a synchronized video sequence; The fusion module is used to perform prior encoding matching on the mouth region in the synchronized video sequence using a trained VQ-GAN-based repair network, then perform feature reconstruction, and fuse the reconstructed mouth region with the synchronized video sequence to obtain the target synchronized video.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the voice-driven AI animation automatic generation and lip-syncing method described in the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the voice-driven AI animation automatic generation and lip-syncing method described in the first aspect above.

[0016] The AI ​​animation automatic generation and lip-syncing method based on voice-over driven provided in this application has the following beneficial effects: First, it collects audio and video data to provide basic information for subsequent processing; then, it automatically triggers video content completion based on the audio duration and generates a coherent extended video to improve the frame rate, while generating a mask to distinguish the mouth areas of multiple characters; then, it extracts audio features to obtain lip shape control parameters and physiological micro-motion parameters, then performs precise synchronization between lip shape and audio, and incorporates natural micro-vibrations; finally, it repairs the image quality of the mouth area to eliminate compositing traces, thereby making the video clear and natural as a whole.

[0017] Furthermore, this application crops a mouth image patch from the synchronized video using a mask and maps it to a latent feature vector via an encoder. Then, the similarity between this vector and reference features in a priori coding library is calculated using the Swin-Transformer operator, and the top K features with the highest similarity are weighted and summed to achieve recombination. The decoder then restores the repaired mouth image patch, which is finally overlaid onto the original image and fused using a weighted average at the boundaries to obtain the target video. This method restores high-frequency details of the mouth through prior matching and feature recombination, and eliminates seams by combining boundary fusion, achieving high-fidelity restoration of the mouth region image quality. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for automatic generation and lip-syncing of AI animation based on voice-over driven technology, provided in an embodiment of this application; Figure 2 A schematic diagram illustrating a specific implementation of a voice-driven AI animation automatic generation and lip-syncing method provided in this application embodiment; Figure 3 A schematic diagram of the structure of an AI animation automatic generation and lip-syncing system based on voice-over driving provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] When dealing with a mismatch between the duration of voice-over and video footage, existing audio-driven animation technology typically has to fill the time gap with awkward looping shots, resulting in disjointed action. At the same time, issues such as low frame rate of generated videos, lip-sync interference when multiple characters are in the same frame, and blurry image quality in locally redrawn areas make it difficult for the final product to meet the commercial standards of smooth, natural, and consistent image quality.

[0021] To address the aforementioned issues, this application provides a method for automatic AI animation generation and lip-syncing based on voice-over driven technology. This method uses audio duration as a baseline, automatically fills in missing frames and increases the frame rate. It also generates masks to differentiate between characters, enabling independent lip-syncing for multiple characters. Furthermore, it combines lip shape parameters and audio energy to achieve lip-syncing and physiological micro-vibrations. Finally, it utilizes a priori coding library to repair mouth image quality. This approach ensures accurate audio-visual synchronization while solving challenges such as duration adaptation, smooth animation, multi-character decoupling, and consistent image quality.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The core of this application is to provide a method for automatic generation and lip-syncing of AI animation based on voice-over driven technology. A flowchart of one specific implementation is shown below. Figure 1 As shown in the diagram, its specific implementation is illustrated below. Figure 2 As shown, the method includes: S101. Collect audio data and video data, wherein the audio data includes voice duration data and audio energy data, and the video data includes human characteristic information and dynamic image data; In this sub-step, the speech duration data in the audio data refers to the physical length of the entire audio file from beginning to end. Audio energy data refers to the amplitude and intensity information of the audio waveform at various time points, which can include changes in loudness and accent positions. The character feature information in the video data refers to the appearance attributes of the characters in the video frame, which can include the character's facial features, skin texture, and clothing style. The motion data refers to the motion information in the video that changes over time, which can include the flow of background light and shadow, the camera's zoom and panning trajectories, and the natural undulations of the characters' limbs.

[0024] In this embodiment, the user-uploaded original audio file is first obtained, and its physical duration is analyzed through waveform scanning as speech duration data. Simultaneously, the amplitude value at each time point in the audio stream is extracted to generate audio energy data. Next, the user-uploaded original video clip is obtained as video data, and static features such as facial structure and skin texture are analyzed as character feature information. Dynamic features such as background lighting changes and camera movement trajectory are extracted as dynamic image data. For example, if a user uploads a dubbing file, the analysis shows a speech duration of 15 seconds, and the audio energy data exhibits an energy curve that is flat for the first 5 seconds, increases in the middle 8 seconds, and decreases in the last 2 seconds. Simultaneously, if a user uploads a character video clip, the extracted character feature information includes the facial contours and skin texture of character A, and the dynamic image data includes the character's movement trajectory from looking down to slightly looking up, as well as the natural flow of background lighting.

[0025] This application, by collecting audio and video data, provides a duration alignment benchmark, lip movement energy source, and visual style basis for subsequent processing, thus laying the foundation for automated generation.

[0026] S102, if the difference between the duration of the voice data and the duration of the video data is greater than zero, the last frame image of the video data is encoded as a condition seed for the latent space. In one specific implementation, S102 includes: Step 1021: Calculate the difference between the duration of the voice data and the duration of the video data. If the difference is greater than zero, determine the last frame of the video data as the tail frame. In this embodiment of the application, the voice duration data obtained in step S101 is first acquired. Next, obtain the duration of the video data. Then calculate the difference in duration between the audio and video data. Then judge If the value > 0 is true, then the last frame of the video data arranged in chronological order is extracted and determined as the tail frame image.

[0027] Step 1022: Divide the pixel matrix of the tail frame image into multiple image blocks according to the preset grid division rules; In this sub-step, the grid partitioning rule refers to the method used to divide the image into equal-sized blocks, which may include the size of each image block and the number of rows and columns. An image block refers to each small region obtained after segmenting the last frame image according to the grid partitioning rule, and each image block contains a set of pixels. The pixel matrix is ​​a two-dimensional array that arranges all pixels in the last frame image according to their row and column positions, with each element in the array corresponding to the color value of a pixel.

[0028] In this embodiment, the last frame image is converted into a pixel matrix, with a width of W pixels and a height of H pixels. Then, according to a preset grid division rule, the size of each image block is set to B×B pixels. Next, the entire pixel matrix is ​​divided into rows and columns at intervals of B pixels to obtain the horizontal... Image blocks, vertical direction Image patches, of which This indicates rounding up, with a total of M×V image blocks, and each image block contains... The color value information of each pixel, these image patches are arranged in order from top to bottom and left to right, denoted as . , where i=1,2,...,V, j=1,2,...,M.

[0029] Step 1023: Using a preset feature index table, find the reference image block that is most similar to each of the image blocks and obtain the corresponding index value. The feature index table includes multiple reference image blocks and the index value corresponding to each reference image block. The index value corresponds to a pre-generated feature vector in the latent space. In this sub-step, the feature index table refers to a pre-constructed lookup table that stores multiple reference image patches and their corresponding index values. Each reference image patch represents a typical type of local image texture feature. A reference image patch is a representative small image patch extracted from a large amount of image data, and each reference image patch corresponds to a specific visual pattern. An index value is a unique number corresponding to a reference image patch, and each index value is associated with a pre-generated feature vector in the latent space. A feature vector is a data representation in vector form consisting of multiple numerical values, used to describe the core visual features of an image patch in the latent space.

[0030] In this embodiment of the application, for each image block Extract the color values ​​of all pixels within the image patch to form the feature description of the image patch, denoted as . Next, The similarity is compared with the feature descriptions of each reference image patch stored in the feature index table. That is, the degree of difference between the two is calculated by Euclidean distance. The feature index table pre-constructs the correspondence between reference image patches and index values. Each index value is associated with a pre-generated feature vector in the latent space. The specific correspondence is shown in Table 1.

[0031] Table 1: Feature Index Table

[0032] Suppose there are L reference image patches in the feature index table, and their feature descriptions are as follows: The corresponding index value is o, where o = 1, 2, ..., L; then calculate the Euclidean distance. And select to make The smallest 'o' is used as the index value of the image patch, and each index value is associated with a pre-generated feature vector in the latent space.

[0033] Step 1024: Combine all the index values ​​according to the arrangement order of each image block in the pixel matrix to obtain an index value sequence, and use the index value sequence as a condition seed.

[0034] In this embodiment, the index values ​​obtained for each image block in step 1023 are combined according to the original arrangement order of the image blocks in the pixel matrix. That is, the index values ​​of the first row and first column image blocks are placed first, followed by the index values ​​of the first row and second column image blocks. After processing all rows and columns in sequence, a sequence of data containing M×V index values ​​is finally obtained. Then, a preset nonlinear projection operator is used to map the sequence of index values ​​into a fixed-dimensional continuous vector space feature. Specifically, a pre-trained linear embedding layer or multilayer perceptron network is used to convert the discrete index value sequence into a dense feature vector of dimension 128, which is then used as a condition seed.

[0035] This application uses duration difference judgment to automatically trigger a video growth mechanism, encodes the last frame image as a condition seed, and provides a precise starting benchmark and texture feature basis for the subsequent generation of extended content with the same visual style as the original video.

[0036] S103. Based on the conditional seed, the character feature information and the dynamic data of the scene, a pre-trained diffusion model is used to make a prediction to generate an extended frame sequence. A video frame sequence is obtained by calculating the motion vector between two adjacent frames in the extended frame sequence. The face region is identified from the video frame sequence and a binary mask sequence is generated. In one specific implementation, step S103 further includes: Step 1031: Perform speech recognition on the audio data to obtain action semantic information corresponding to the audio data; In this sub-step, action semantic information refers to the meaning content related to body movements identified from audio data, which may include the action intention expressed by keywords, phrases or sentences describing the action in the text.

[0037] In this embodiment of the application, the audio data obtained in step S101 is processed by speech recognition, that is, the audio signal is converted into text content in text form, and then keywords related to the action are extracted from the converted text content, such as words describing body movements such as running, jumping, waving, and nodding. Finally, these text contents related to the action are used as action semantic information. Specifically, if the text recognized from the audio data is: He waved his hand and walked forward, then the extracted action semantic information includes waving and walking.

[0038] Step 1032: Based on the action semantic information, determine the target action tag using the preset mapping relationship between action semantics and action tags; In this sub-step, the target action label refers to the standardized action category identifier obtained by matching the action semantic information, and each target action label corresponds to a preset limb action pattern.

[0039] In this embodiment of the application, a mapping table between action semantics and action tags is pre-constructed. Each row in the table records an action description word and its corresponding standardized action tag. The specific mapping relationship is shown in Table 2.

[0040] Table 2: Mapping Relationship between Action Semantics and Action Tags

[0041] Then, the action semantic information obtained in step 1031 is matched with the action description words in the mapping table. That is, when the action semantic information contains a word from the table, the action label corresponding to that word is selected as the target action label. For example, if the action semantic information contains "waving", the matched target action label is "wave".

[0042] Step S103 includes: Step 1033: Concatenate the condition seed, the character feature information, the dynamic data of the scene, and the target action label into vectors to obtain the diffusion input vector; In this embodiment of the application, the condition seed obtained in step S102 is denoted as c, the character feature information obtained in step S101 is denoted as p, the dynamic data of the screen obtained in step S101 is denoted as d, and the target action label obtained in step S1032 is denoted as l; then the four vectors c, p, d, and l are concatenated according to a preset dimension order to obtain the diffusion input vector xin=[c;p;d;l]. For example, if the conditional seed is a 128-dimensional vector, the character feature information is a 64-dimensional vector, the dynamic data of the scene is a 64-dimensional vector, and the target action label is converted into a 32-dimensional vector through the embedding layer, then the concatenated diffusion input vector is a 288-dimensional vector. The embedding layer is a one-hot encoding mapping matrix pre-trained on a large-scale corpus. Its function is to map discrete semantic labels to a continuous Euclidean space. The specific processing is as follows: after converting the action label into a one-hot vector, matrix multiplication is performed with the weight matrix to obtain a dense feature vector that can express the semantic similarity of actions. This vector can quantify the semantic distance between different action labels and provide clear dynamic direction guidance for the diffusion model when generating the scene.

[0043] Step 1034: Input the diffusion input vector into the trained diffusion model based on the Transformer architecture, perform non-repetitive content prediction in the latent space, and obtain the extended frame sequence. The diffusion model is trained using samples including historical diffusion input vectors and corresponding real image frame sequences based on the Transformer architecture.

[0044] In this sub-step, the historical diffusion input vector refers to the sample input data used during the training phase. The real image frame sequence refers to the actual image frames that correspond to the historical diffusion input vector and serve as the training target.

[0045] In this embodiment, the diffusion input vector obtained in step 1033 is input into a pre-trained diffusion model based on the Transformer architecture. The training process of the diffusion model uses supervised learning with sample pairs, and each training sample includes a historical diffusion input vector and a corresponding real image frame sequence, i.e. ,in, Let i be the historical diffusion input vector of the i-th sample. For the corresponding real image frame sequence; during the inference phase, the diffusion model performs multi-step iterative denoising on the input diffusion input vector, gradually restoring clear image features in the latent space. Finally, the latent space features are decoded into multiple frames in the image space, and these images are arranged in the generation order to obtain the extended frame sequence, denoted as . , where T is the number of frames generated.

[0046] Step 1035: Calculate the displacement direction and displacement distance of each pixel between two adjacent frames in the extended frame sequence to obtain the motion vector, and determine the number of intermediate images to be inserted based on the ratio of the preset target frame rate to the original frame rate of the extended frame sequence. In this sub-step, the original frame rate refers to the frame rate of the extended frame sequence itself. The target frame rate refers to the final desired video frame rate.

[0047] In this embodiment of the application, for the extended frame sequence obtained in step 1034 For each adjacent two frames and Perform optical flow calculations to obtain the optical flow of each pixel from arrive The direction and distance of displacement, where, The preceding frame in two adjacent frames. This refers to the next frame in a pair of adjacent frames, forming the motion vector field corresponding to those two adjacent frames. Specifically, for an image... The pixel located at coordinates (x, y) in the image Search for the position that is most similar to the grayscale distribution of its surrounding pixels. The offset at that position ( ) is the movement vector of that pixel, where the displacement direction is determined by ( The direction of ) determines the displacement distance. Then obtain the original frame rate of the extended frame sequence. Preset target frame rate Calculate the number of intermediate images that need to be inserted. ,in, The number of intermediate images that need to be inserted between two adjacent frames to extend the frame sequence. To extend the original frame rate of the frame sequence, The preset target frame rate.

[0048] Step 1036: Calculate the mapping position of each pixel in the intermediate image in two adjacent frames according to the time ratio coefficient corresponding to the number of intermediate images and the motion vector, and generate the intermediate image by weighted averaging of the pixel values ​​at the mapping positions. Insert the intermediate image into the extended frame sequence between two adjacent frames to obtain a video frame sequence. In this sub-step, the intermediate image refers to an additional image frame generated through interpolation between two adjacent frames. The temporal scaling factor refers to the temporal position ratio of the intermediate image between two adjacent frames, and its value ranges from 0 to 1. The mapped position refers to the coordinate position of a pixel on the intermediate image in the two adjacent frames.

[0049] In this embodiment of the application, for each pair of adjacent frames, a corresponding time scaling factor sequence is generated based on the number of intermediate images determined in step 1035. Where k=1,2,..., , The number of intermediate images to be inserted between each pair of adjacent frames is determined; then, using the motion vector field obtained in step 1035, the position (x, y) of each pixel in the intermediate image is calculated. The mapping position in is ,exist The mapping position in is Where (q, v) are the displacement components of the moving vector field at (x, y), q is the horizontal displacement, and v is the vertical displacement. Indicates the current intermediate image is in and The normalized time offset within the time interval, with a value range of (0, 1); then, pixel values ​​are read from the two mapped positions respectively. and Then, using the linear weighted average formula: Get the pixel value of that pixel in the intermediate image. Finally, all intermediate images are inserted into the intervals between adjacent frames of the extended frame sequence according to the time ratio to obtain the video frame sequence.

[0050] Step 1037: Perform face detection on the video frame sequence to obtain at least one face identifier, and perform voiceprint recognition on the audio data to obtain a role identifier; In this sub-step, the face identifier refers to the unique identification number assigned to each face region detected in the video frame sequence. The role identifier refers to the unique identification number of the current speaker role obtained from the audio data through voiceprint recognition.

[0051] In this embodiment, face detection processing is performed on each frame of the video frame sequence obtained in step 1036. That is, a face detection algorithm is used to locate the position of all face regions in the image, and a unique face identifier is assigned to each detected face region. Where j is the index of the face region; simultaneously, the audio data obtained in step S101 is processed for voiceprint recognition, that is, the voiceprint features in the audio are extracted and matched with the pre-stored role voiceprint templates, and the speaker identity corresponding to each time point is determined to obtain the role identifier. , where l is the frame index.

[0052] Step 1038: Determine the target face identifier from at least one of the face identifiers based on the role identifier, mark the pixels within the mouth boundary contour of the face region corresponding to the target face identifier as the first value, and mark the pixels outside the mouth boundary contour as the second value to obtain the binary mask corresponding to each frame image, and combine all the binary masks in frame order to obtain a binary mask sequence.

[0053] In this sub-step, the target face identifier refers to the identity number corresponding to the face region that matches the current speaker's role. A binary mask is a labeled image that divides image pixels into two categories of regions, where the first value is usually 1 indicating a valid region and the second value is usually 0 indicating an invalid region. A binary mask sequence is a sequence composed of the binary masks corresponding to all frames of images arranged in frame order.

[0054] In this embodiment, for each frame in the video frame sequence, the set of face identifiers and the set of character identifiers corresponding to that frame are obtained. Then, cross-modal association logic between face identifiers and character identifiers is established. Specifically, a correspondence table between the voiceprint features and face features of each character is pre-stored, and modal alignment is achieved by retrieving this table. During real-time processing, if there is no pre-set correspondence, dynamic binding is performed through spatiotemporal consistency detection logic: the mouth opening and closing frequency of each face region in the video in historical frames is calculated, and correlation analysis is performed between this frequency and the fluctuation frequency of the synchronously extracted audio energy envelope. Then, the face identifier with the highest correlation coefficient is associated and bound with the character identifier of the current audio. Subsequently, a target face identifier that matches the character identifier is selected from the face identifier set, and the corresponding vocal subject area is locked based on the target face identifier to determine the boundary contour of the mouth area. Then, a binary mask image with the same size as the frame image is created, and all pixels within the mouth boundary contour are marked with a first value, such as 1, and all pixels outside the mouth boundary contour are marked with a second value, such as 0. Then, the binary masks corresponding to each frame image are combined in frame order to obtain a binary mask sequence.

[0055] For example, in this application, the training samples of the diffusion model consist of historical diffusion input vectors and corresponding real image frame sequences. For each training sample, each frame of the real image frame sequence is first input into a variational autoencoder to obtain a latent tensor, and its spatial size is recorded as h×w. Then, according to a preset noise scheduling strategy, Gaussian noise is gradually added to the latent tensor to obtain a noisy latent tensor zt, where t is the diffusion time step and its value ranges from 1 to 1000. At the same time, the historical diffusion input vector is used as conditional information and converted into a conditional feature vector through an embedding layer.

[0056] Subsequently, a diffusion model based on the Transformer architecture was constructed. This model adopts an encoder-decoder structure, in which the encoder consists of 12 stacked Transformer blocks, each containing a multi-head self-attention mechanism with 12 attention heads and a model dimension of 768. Each Transformer block also contains two multilayer perceptrons and uses Gaussian error linear unit activation function. The model also introduces a time-step encoding module, which converts the time step t into a 256-dimensional vector through sinusoidal position encoding, and then maps it into a time-step embedding vector through two multilayer perceptrons. At the same time, a cross-attention mechanism is introduced, which uses the conditional feature vector as a key and value to interact with the latent features of the image to achieve conditional control over the generated content.

[0057] During training, the noisy latent tensor, time-step embedding vector, and conditional feature vector are input into the model, and the model predicts the noise added at the current time step. Then, the L2 loss function is used to calculate the prediction noise. With real noise The error between the time steps is calculated, and different weights are assigned to the loss at different time steps. The loss is adaptively adjusted according to the signal-to-noise ratio (SNR) (t), that is, the time steps with low SNR are assigned smaller weights to avoid gradient conflicts and accelerate model convergence. Then, the model parameters are iteratively optimized through the backpropagation algorithm until the loss function converges, thus obtaining a trained diffusion model based on the Transformer architecture. This model can perform multi-step denoising in the latent space based on the input conditional seed, character feature information, dynamic data of the scene, and other conditional information, and gradually restore clear image features. Finally, it is restored to the extended frame sequence in the image space by the variational autodecoder.

[0058] This application guides video growth through action semantics and obtains a smooth video frame sequence by combining optical flow frame interpolation. At the same time, it uses voiceprint and visual matching to generate an accurate binary mask sequence, which can lay a high-quality dynamic base and regional control foundation for subsequent lip-syncing.

[0059] S104. The audio data is converted into phoneme features, the target lip shape parameters are determined according to the preset mapping relationship between phoneme features and lip shape parameters, and the audio energy data is mapped into a displacement vector sequence. In one specific implementation, step S104 includes: Step 1041: Divide the audio data into frames according to the preset time window length to obtain an audio frame sequence. Perform Fourier transform on the audio frame sequence to obtain the amplitude of each audio frame at each frequency point. Group the amplitudes according to the preset frequency band division rules to form the phoneme feature vector of each audio frame. In this embodiment, the audio data obtained in step S101 is first acquired, and the time window length L milliseconds is set, typically 25 milliseconds, with a window movement step of 10 milliseconds. Then, the audio data is segmented into frames according to the time window length to obtain an audio frame sequence, denoted as... Where N is the total number of audio frames, a fast Fourier transform is performed on each audio frame, that is, its time domain signal is converted into a frequency domain signal, and the amplitude of the frame at each frequency point is obtained. Then, the frequency range is divided into Z consecutive frequency bands according to the preset frequency band division rules, where the value of Z ranges from 13 to 40. After obtaining the amplitude of each frequency band, logarithmic compression and discrete cosine transform processing are further performed to extract the cepstral coefficient information reflecting the shape characteristics of the vocal tract, and these are used to form the phoneme feature vector of each audio frame. Similarly, all audio frames are processed in sequence to obtain the phoneme feature vector sequence.

[0060] Step 1042: Select the phoneme feature vector with the smallest Euclidean distance from the preset reference feature vector as the target phoneme, determine the degree of mouth opening and the roundness value of the lips of the target phoneme from the preset lookup table, and arrange the degree value and the roundness value in chronological order to obtain the target lip shape parameters. In this sub-step, the degree of mouth opening is used to describe the vertical opening of the upper and lower lips, and the lip roundness value is used to describe the degree of horizontal deformation of the lip edges as they converge towards the center. Together, they define the spatial morphological features of the target mouth shape.

[0061] In this embodiment of the application, a reference feature vector library is pre-constructed, which contains reference feature vectors corresponding to P standard phonemes. Each reference feature vector corresponds to a phoneme label. For each phoneme feature vector obtained in step 1041, the Euclidean distance between it and each reference feature vector is calculated, and the phoneme label corresponding to the reference feature vector that minimizes the Euclidean distance is selected as the target phoneme of the frame. A reference table for phonemes and lip shape parameters is established in advance, as shown in Table 3. Each row in the table records a phoneme label and its corresponding mouth opening degree value and lip roundness value.

[0062] Table 3: Comparison Table of Phonemes and Lip Shape Parameters

[0063] Finally, the mouth opening degree and lip roundness values ​​of all audio frames are arranged in chronological order to obtain the target lip shape parameters.

[0064] Step 1043: Calculate the average value of the instantaneous energy values ​​of the audio energy data at a preset number of time points before and after each audio frame's corresponding time point to obtain the smoothed energy value of each audio frame; In this embodiment of the application, the audio energy data obtained in step S101 is acquired. This data includes the instantaneous energy value at each time point, denoted as... Where T is the total number of time points; then set the sliding window radius R, which represents averaging the R time points before and after the current time point, that is, for the time point corresponding to the i-th audio frame. Its smooth energy value The calculation formula is ,in, The time point corresponding to the i-th audio frame The instantaneous energy value after offset j, when When the value is outside the valid range, only the points within the range are averaged, and this process is repeated for all audio frames to obtain a smooth energy value sequence. Where N is the total number of audio frames.

[0065] Step 1044: Multiply each smoothing energy value by a preset scaling factor to obtain the displacement amplitude value of each audio frame, and arrange all the displacement amplitude values ​​in the time order of the audio frames to obtain a displacement vector sequence.

[0066] In this embodiment, a preset scaling factor b is used to map the energy value to a suitable displacement range based on empirical values; then, for each smoothed energy value obtained in step 1043... Calculate the corresponding displacement amplitude value. Then, the displacement amplitude values ​​of all audio frames are arranged in chronological order to obtain the displacement vector sequence. .

[0067] This application obtains target lip shape parameters by extracting phoneme feature mapping, which can ensure that the mouth shape and the pronunciation content correspond accurately. At the same time, it converts audio energy into displacement vector sequence, which can introduce natural physiological micro-vibration into the mouth, thereby improving the realism and expressiveness of lip-sync.

[0068] S105. Based on the target lip shape parameters and the displacement vector sequence, update the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame to obtain a synchronized video sequence. In one specific implementation, step S105 includes: Step 1051: Detect key points of the mouth in the video frame sequence to obtain an initial coordinate set, which includes the pixel coordinates of the upper lip control point, lower lip control point, left corner of mouth control point and right corner of mouth control point; In this sub-step, the upper lip control point is a key point located at the center of the upper lip contour, used to control the vertical movement of the upper lip. The lower lip control point is a key point located at the center of the lower lip contour, used to control the vertical movement of the lower lip. The left corner of the mouth control point is a key point located at the left corner of the mouth, used to control the horizontal movement of the corner of the mouth. The right corner of the mouth control point is a key point located at the right corner of the mouth, used to control the horizontal movement of the corner of the mouth.

[0069] In this embodiment, the video frame sequence obtained in step S103 is acquired. This sequence contains multiple frames of images. For each frame, a regression-based cascaded convolutional network is used to locate multiple key points in the mouth region. The key point at the center of the upper lip is selected as the upper lip control point, the key point at the center of the lower lip as the lower lip control point, the key point at the left corner of the mouth as the left corner of the mouth control point, and the key point at the right corner of the mouth as the right corner of the mouth control point. Then, the pixel coordinates of these four control points in the image coordinate system are recorded, where the horizontal coordinate represents the horizontal position and the vertical coordinate represents the vertical position. Finally, all control point coordinates are combined into an initial coordinate set. These correspond to the upper lip, lower lip, left corner of the mouth, and right corner of the mouth, respectively.

[0070] Step 1052: Add the product of the degree of mouth opening of the target lip shape parameter and the first preset coefficient to the vertical coordinate of the upper lip control point and the lower lip control point, and add the product of the lip roundness value of the target lip shape parameter and the second preset coefficient to the horizontal coordinate of the left corner of the mouth control point and the right corner of the mouth control point to obtain the intermediate coordinate set. In this sub-step, the first preset coefficient refers to the scaling factor used to convert the mouth opening value into a change in the vertical axis. The second preset coefficient refers to the scaling factor used to convert the lip roundness value into a change in the horizontal axis.

[0071] In this embodiment, the target lip shape parameters obtained in step S104 are acquired. These parameters include the degree of mouth opening and the roundness value of the lips for each frame. Simultaneously, a preset first coefficient is acquired. Second coefficient These two coefficients are set through empirical values, for example... =10, =8; For the current frame, read the degree of mouth opening (g) and lip roundness (s) from the target lip shape parameters, and then adjust the ordinate of the upper lip control point to 8. ′= ×g indicates that the upper lip moves upward; the ordinate of the lower lip control point is adjusted to... ′= + ×g indicates that the lower lip moves downwards; adjust the x-coordinate of the control point at the left corner of the mouth to... ′= ×s indicates that the left corner of the mouth moves to the left; the x-coordinate of the control point for the right corner of the mouth is adjusted to... ′= + ×s represents the right corner of the mouth moving to the right, while other coordinates remain unchanged, resulting in the intermediate coordinate set. .

[0072] Step 1053: Using the displacement amplitude value as the disturbance intensity, add random offsets to each control point in the intermediate coordinate set to obtain the target coordinate set; In this embodiment of the application, the displacement vector sequence obtained in step S104 is acquired, and the sequence contains the displacement amplitude value corresponding to each frame. The displacement amplitude of the current frame is used as the disturbance intensity; for each control point in the intermediate coordinate set, a two-dimensional random offset vector is generated. , ),in and In [ The coordinates of each control point are randomly selected uniformly within the range [w, w], where w represents the displacement amplitude of the current frame; then, the coordinates of each control point are added with the corresponding random offset to obtain the target coordinate set. .

[0073] Step 1054: Based on the correspondence between the initial coordinate set and the target coordinate set, perform coordinate mapping on each pixel in the mouth region marked by the binary mask sequence in the video frame sequence to obtain the updated current frame image, and arrange all the updated current frame images in the order in the video frame sequence to obtain the synchronized video sequence.

[0074] In this embodiment of the application, for the current frame, its initial coordinate set is obtained. and target coordinate set The image is processed by first generating a binary mask for the corresponding frame, which marks the pixel positions of the mouth region. Then, an image deformation algorithm based on control points, such as moving least squares deformation or thin plate spline interpolation, is used to establish a mapping function f(x,y) from the original image space to the target image space based on the correspondence of the four control points. For each pixel (x,y) marked by the binary mask, its corresponding position (x,y) = f(x,y) in the target image is calculated. Since the mapped coordinates may be non-integer, bilinear interpolation is used to sample the pixel value from the original image and fill the same position (x,y) in the target image with this pixel value. At the same time, for non-mouth regions, the original pixel values ​​are directly retained. This results in the updated current frame image. All frames are processed sequentially, and the updated frames are arranged in the original order to obtain a synchronized video sequence.

[0075] This application achieves precise synchronization between lip shape and audio by applying lip shape parameters and displacement vectors to the mouth area through control point transformation, while also giving the mouth a natural physiological micro-vibration, generating a synchronized video sequence with realistic lip shape and smooth movements.

[0076] S106. Using a trained repair network based on the VQ-GAN architecture, the mouth region in the synchronized video sequence is subjected to prior coding matching and feature reconstruction. The reconstructed mouth region is then fused with the synchronized video sequence to obtain the target synchronized video.

[0077] In one specific implementation, step S106 includes: Step 1061: Based on the mouth region marked by the binary mask in the binary mask sequence, crop out the mouth image patch from each frame of the synchronized video sequence, and use the encoder of the repair network based on the VQ-GAN architecture to map the mouth image patch into an input feature vector in the latent space. In this embodiment, the synchronized video sequence obtained in step S105 is acquired. This sequence contains multiple frames of images. Simultaneously, the binary mask sequence obtained in step S103 is acquired. Each frame in this sequence corresponds to a binary mask, marking the pixel positions of the mouth region. For each frame in the synchronized video sequence, the minimum bounding rectangle of the mouth region is determined based on the corresponding binary mask, and this rectangular region is cropped from the image as a mouth image patch. Then, the cropped mouth image patch is input into a trained VQ-GAN-based inpainting network. The encoder part of this network consists of multiple convolutional layers and downsampling layers. The mouth image patch undergoes layer-by-layer convolution and downsampling operations by the encoder, gradually reducing the spatial resolution and increasing the number of channels, ultimately mapping it to a fixed-dimensional vector in the latent space. This vector is used as the input feature vector. For example, the mouth image patch cropped from a certain frame is 64×64 pixels in size, and after encoder processing, an input feature vector with a dimension of 512 is obtained.

[0078] Step 1062: In the latent feature space, the similarity between the input feature vector and each reference feature vector in the pre-stored prior coding library is calculated using the Swin-Transformer operator, and the top K reference feature vectors with the highest similarity are weighted and summed according to the size of their inner product values ​​to obtain the recombined feature vector, where K is a positive integer; In this sub-step, the prior coding library is a semantic feature set constructed based on a multi-scale residual quantization mechanism. In order to maintain the consistency of the character's identity ID during the restoration process, the reference feature vector is associated with identity attribute labels during storage. Before performing similarity matching, the prior coding library is first screened using the character's feature information to remove features that are too different from the current character's skin tone and bone proportion. Subsequently, feature recombination is performed in the screened sub-library. Through this retrieval mechanism with identity prior constraints, it is ensured that the restored mouth image block has both high-frequency teeth and lip line details and is completely aligned with the inherent physiological features of the original video character, thereby avoiding the visual unfamiliarity caused by local redrawing.

[0079] In this embodiment, the input feature vector obtained in step 1061 is acquired in the latent feature space, and its dimension is denoted as D. J reference feature vectors are pre-stored in the prior coding library, each with a dimension of D. The Swin-Transformer operator is then used to calculate the similarity between the input feature vector and each reference feature vector. In its specific implementation, the sequence of input and reference feature vectors is combined into serialized data, and the sliding window self-attention mechanism of the Swin-Transformer is used to calculate the cross-correlation between feature points within a locally non-overlapping window. This hierarchical attention calculation not only compares the global similarity of vectors but also captures subtle points of convergence between the input features and codebook features in the local texture distribution. This allows for the selection of the semantically closest high-fidelity prior coding from the candidate feature library, resulting in a similarity value sequence. ; Next, the K most similar reference feature vectors are selected, where K is a positive integer, typically 3 or 5. These K reference feature vectors are then weighted and summed according to their similarity values, with higher similarity vectors assigned higher weights. The weight calculation formula is as follows: Then, a weighted sum is performed to obtain the recombined feature vector. ,in, Let j be the selected reference feature vector.

[0080] Step 1063: Using the decoder of the repair network, the recombined feature vector is mapped back to the image space to obtain the repaired mouth image block; In this embodiment, the reconstructed feature vector obtained in step 1062 is input into the decoder of the repair network. The decoder consists of multiple upsampling layers and convolutional layers and is symmetrical with the encoder structure. Then, the reconstructed feature vector undergoes layer-by-layer upsampling and convolution operations by the decoder to gradually restore the spatial resolution and reduce the number of channels. Finally, it is mapped back to the image space to generate a high-quality image region with the same size as the original mouth image block, which serves as the repaired mouth image block. The mouth texture details in this image block are clearer, and high-frequency information such as teeth and lip wrinkles are restored, while blurring and artifacts are eliminated.

[0081] Step 1064: Overlay the repaired mouth image block onto the mouth region of the corresponding frame image in the synchronized video sequence, and perform weighted averaging on the boundary pixels of the mouth region to obtain the target synchronized video.

[0082] In this embodiment, for each frame of the synchronized video sequence, the repaired mouth image block obtained in step 1063 and the corresponding binary mask are obtained, and the repaired mouth image block is overlaid on the mouth region of the original frame image. During overlay, the pixel value of the region is directly replaced. In order to ensure a smooth transition between the repaired region and the surrounding background, the boundary pixels of the overlaid region are processed by weighted averaging. Specifically, at the boundary of the mouth region, a strip region of G pixels inside and outside the boundary is taken. For each pixel in this region, its final pixel value is obtained by weighted averaging of the pixel value of the repaired image block and the pixel value of the original image according to the weight of the distance from the boundary. The closer to the boundary, the greater the weight of the repaired image block. The farther from the boundary, the greater the weight of the original image. After processing all frames in sequence, the target synchronized video is obtained.

[0083] For example, in this application, the training of the repair network based on the VQ-GAN architecture adopts a two-stage training strategy: first, a training dataset is constructed by selecting image samples from a large-scale, high-quality face image dataset, where each sample image contains a complete facial region and clear mouth texture details; then, for each training sample image, it is first mapped to a feature map in the latent space by an encoder, and then the continuous feature map is discretized into index values ​​corresponding to the pre-trained codebook by a vector quantization layer, and finally the image is reconstructed from the discrete index by a decoder.

[0084] The first phase of training focuses on self-supervised reconstruction learning, employing a reconstruction loss function. Measuring the input image x and the reconstructed image The pixel-level differences between them, combined with codebook learning loss and promised losses ,in, For encoder output features, The quantized feature vector is sg, which indicates stopping the gradient operation and optimizing the encoder output to align with the codebook vector.

[0085] The second stage introduces an adversarial training mechanism to enhance the realism of the reconstructed image. This involves adding a discriminator loss to the original reconstruction loss. The discriminator employs a block-based generative adversarial network architecture to assess the realism of local regions in the reconstructed image. The generator loss function is expanded to... ,in, This represents the total loss value of the generator. To reconstruct the loss value, The weighting coefficients are the weights of the codebook learning loss. Weighting coefficients for committed losses The weighting coefficients of the generator against the loss. = The generator's adversarial loss value is [value], and the discriminator's loss value is [value]. ,in, To score the realism of the original real image x by the discriminator, For the discriminator to reconstruct the image The model is scored for realism. Then, by alternately optimizing the parameters of the generator and discriminator, the repair network can accurately reconstruct the mouth structure while generating clearer and more natural texture details. During training, an exponential moving average strategy is used to stabilize the model parameters, and the reconstruction quality is evaluated on the validation set at regular intervals until the model converges and a well-trained repair network based on the VQ-GAN architecture is obtained.

[0086] This application repairs the mouth area through prior coding matching and feature recombination, which can eliminate blur artifacts and restore high-frequency details such as teeth and lip lines. Then, boundary fusion is used to achieve seamless connection between the repaired area and the background, thereby improving video quality and realism.

[0087] Figure 3 This application provides a schematic diagram of a specific implementation of a voice-driven AI animation automatic generation and lip-syncing system, as illustrated in the embodiments below. Figure 3 The system may include: Acquisition module 31 is used to acquire audio data and video data. The audio data includes voice duration data and audio energy data, and the video data includes human feature information and dynamic image data. Encoding module 32 is used to encode the last frame image of the video data as a condition seed for the latent space when the difference between the duration of the voice data and the duration of the video data is greater than zero. Prediction module 33 is used to predict based on the condition seed, the character feature information and the dynamic data of the scene using a pre-trained diffusion model, generate an extended frame sequence, obtain a video frame sequence by calculating the motion vector between two adjacent frames in the extended frame sequence, identify the face region from the video frame sequence and generate a binary mask sequence. The determining module 34 is used to convert the audio data into phoneme features, determine the target lip shape parameters according to the preset mapping relationship between phoneme features and lip shape parameters, and map the audio energy data into a displacement vector sequence. Update module 35 is used to update the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame based on the target lip shape parameters and the displacement vector sequence, so as to obtain a synchronized video sequence; The fusion module 36 is used to perform prior coding matching on the mouth region in the synchronized video sequence using a trained repair network based on the VQ-GAN architecture, and then perform feature reconstruction. The reconstructed mouth region is then fused with the synchronized video sequence to obtain the target synchronized video.

[0088] The AI ​​animation automatic generation and lip-syncing system based on voice-over driving in this application embodiment is used to implement the aforementioned AI animation automatic generation and lip-syncing method based on voice-over driving. Therefore, the specific implementation of the AI ​​animation automatic generation and lip-syncing system based on voice-over driving can be found in the embodiment section of the AI ​​animation automatic generation and lip-syncing method based on voice-over driving above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0089] like Figure 4 As shown, this application also provides an electronic device, including: a memory 41 for storing a computer program; and a processor 42 for executing the computer program to implement the steps of any of the above-described methods for automatic generation and lip-syncing of AI animation based on voice-driven technology.

[0090] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for automatic generation and lip-syncing of AI animation based on voice-driven technology.

[0091] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0092] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the AI ​​animation automatic generation and lip-syncing method based on voice-driven processing.

[0093] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] The above provides a detailed description of the AI ​​animation automatic generation and lip-syncing method and system based on voice-over driven technology provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for automatically generating AI animation synchronized with lip movement based on voice dubbing driving, characterized in that, include: Collect audio and video data, wherein the audio data includes voice duration data and audio energy data, and the video data includes human characteristic information and dynamic image data; If the difference between the duration of the audio data and the duration of the video data is greater than zero, the last frame image of the video data is encoded as a condition seed for the latent space. Based on the conditional seed, the character feature information, and the dynamic data of the scene, a pre-trained diffusion model is used to make predictions and generate an extended frame sequence. By calculating the motion vector between two adjacent frames in the extended frame sequence, a video frame sequence is obtained. From the video frame sequence, the face region is identified and a binary mask sequence is generated. The audio data is converted into phoneme features, and the target lip shape parameters are determined according to the preset mapping relationship between the phoneme features and lip shape parameters. The audio energy data is then mapped into a displacement vector sequence. Based on the target lip shape parameters and the displacement vector sequence, the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence are updated frame by frame to obtain a synchronized video sequence. Using a trained repair network based on the VQ-GAN architecture, the mouth region in the synchronized video sequence is first encoded and matched with prior codes, and then its features are reconstructed. The reconstructed mouth region is then fused with the synchronized video sequence to obtain the target synchronized video.

2. The method of claim 1, wherein, The method further includes: Speech recognition is performed on the audio data to obtain action semantic information corresponding to the audio data; Based on the action semantic information, the target action tag is determined using a preset mapping relationship between action semantics and action tags; The step of generating an extended frame sequence by using a pre-trained diffusion model to predict based on the conditional seed, the character feature information, and the dynamic data of the scene includes: The conditional seed, the character feature information, the dynamic data of the scene, and the target action label are concatenated into vectors to obtain the diffusion input vector; The diffusion input vector is input into a trained diffusion model based on the Transformer architecture, and non-repetitive content prediction is performed in the latent space to obtain an extended frame sequence. The diffusion model is trained using samples including historical diffusion input vectors and corresponding real image frame sequences based on the Transformer architecture.

3. The method of claim 1, wherein, When the difference between the duration of the speech data and the duration of the video data is greater than zero, encoding the last frame image of the video data as a conditional seed for the latent space includes: Calculate the difference between the duration of the audio data and the duration of the video data. If the difference is greater than zero, determine the last frame of the video data as the tail frame. According to the preset grid division rules, the pixel matrix of the tail frame image is divided into multiple image blocks; Using a preset feature index table, the reference image block most similar to each of the image blocks is found to obtain the corresponding index value. The feature index table includes multiple reference image blocks and the index value corresponding to each reference image block. The index value corresponds to a pre-generated feature vector in the latent space. All the index values ​​are combined according to the order in which each image block is arranged in the pixel matrix to obtain an index value sequence, and the index value sequence is used as a condition seed.

4. The method of claim 1, wherein, The step of obtaining a video frame sequence by calculating the motion vector between two adjacent frames in the extended frame sequence, identifying face regions from the video frame sequence, and generating a binary mask sequence includes: The displacement direction and displacement distance of each pixel between two adjacent frames in the extended frame sequence are calculated to obtain the motion vector, and the number of intermediate images to be inserted is determined according to the ratio of the preset target frame rate to the original frame rate of the extended frame sequence. Based on the time ratio coefficient corresponding to the motion vector and the number of intermediate images, the mapping position of each pixel in the intermediate image in two adjacent frames is calculated, and the intermediate image is generated by weighted averaging of the pixel values ​​at the mapping positions. The intermediate image is then inserted between two adjacent frames in the extended frame sequence to obtain a video frame sequence. Face detection is performed on the video frame sequence to obtain at least one face identifier, and voiceprint recognition is performed on the audio data to obtain a role identifier; Based on the role identifier, a target face identifier is determined from at least one of the face identifiers. Pixels within the mouth boundary contour of the face region corresponding to the target face identifier are marked as a first value, and pixels outside the mouth boundary contour are marked as a second value, thus obtaining a binary mask corresponding to each frame image. All the binary masks are combined in frame order to obtain a binary mask sequence.

5. The method of claim 1, wherein, The step of converting the audio data into phoneme features, determining the target lip shape parameters based on the preset mapping relationship between phoneme features and lip shape parameters, and mapping the audio energy data into a displacement vector sequence includes: The audio data is divided into frames according to a preset time window length to obtain an audio frame sequence. The audio frame sequence is then subjected to a Fourier transform to obtain the amplitude of each audio frame at each frequency point. The amplitudes are then grouped according to a preset frequency band division rule to form the phoneme feature vector of each audio frame. The phoneme feature vector with the smallest Euclidean distance from the preset reference feature vector is selected as the target phoneme. The degree of mouth opening and the roundness of the lips of the target phoneme are determined from the preset lookup table. The degree of mouth opening and the roundness of the lips are arranged in chronological order to obtain the target lip shape parameters. Calculate the average value of the instantaneous energy values ​​of the audio energy data at a preset number of time points before and after each audio frame's corresponding time point to obtain the smoothed energy value of each audio frame; The smoothing energy value is multiplied by a preset scaling factor to obtain the displacement amplitude value of each audio frame. All the displacement amplitude values ​​are then arranged in the time order of the audio frames to obtain a displacement vector sequence.

6. The method of claim 1, wherein, The step of updating the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame based on the target lip shape parameters and the displacement vector sequence to obtain a synchronized video sequence includes: Mouth key point detection is performed on the video frame sequence to obtain an initial coordinate set, which includes the pixel coordinates of the upper lip control point, lower lip control point, left corner of mouth control point, and right corner of mouth control point; The product of the degree of mouth opening of the target lip shape parameter and the first preset coefficient is added to the vertical coordinate of the upper lip control point and the lower lip control point. The product of the lip roundness value of the target lip shape parameter and the second preset coefficient is added to the horizontal coordinate of the left corner of the mouth control point and the right corner of the mouth control point to obtain the intermediate coordinate set. Using the displacement amplitude value as the disturbance intensity, a random offset is added to each control point in the intermediate coordinate set to obtain the target coordinate set; Based on the correspondence between the initial coordinate set and the target coordinate set, coordinate mapping is performed on each pixel in the mouth region marked by the binary mask sequence in the video frame sequence to obtain the updated current frame image. All updated current frame images are then arranged in the order in the video frame sequence to obtain a synchronized video sequence.

7. The method of claim 1, wherein, The process involves using a pre-trained VQ-GAN-based repair network to perform prior encoding matching on the mouth region in the synchronized video sequence, followed by feature reconstruction. The reconstructed mouth region is then fused with the synchronized video sequence to obtain the target synchronized video, including: Based on the mouth region marked by the binary mask in the binary mask sequence, mouth image blocks are cropped from each frame of the synchronized video sequence. The encoder of the repair network based on the VQ-GAN architecture is used to map the mouth image blocks into input feature vectors in the latent space. In the latent feature space, the similarity between the input feature vector and each reference feature vector in the pre-stored prior coding library is calculated using the Swin-Transformer operator. The top K reference feature vectors with the highest similarity are then weighted and summed according to their inner product values ​​to obtain the recombined feature vector, where K is a positive integer. Using the decoder of the repair network, the reconstructed feature vector is mapped back to the image space to obtain the repaired mouth image block; The repaired mouth image block is overlaid onto the mouth region of the corresponding frame image in the synchronized video sequence, and the boundary pixels of the mouth region are weighted and averaged to obtain the target synchronized video.

8. A lip-sync system for automatically generating AI animation based on voice driving, characterized in that, include: The acquisition module is used to acquire audio data and video data. The audio data includes voice duration data and audio energy data, and the video data includes human feature information and dynamic image data. The encoding module is used to encode the last frame image of the video data as a conditional seed in the latent space when the difference between the duration of the speech data and the duration of the video data is greater than zero. The prediction module is used to predict based on the conditional seed, the character feature information and the dynamic data of the scene using a pre-trained diffusion model, generate an extended frame sequence, obtain a video frame sequence by calculating the motion vector between two adjacent frames in the extended frame sequence, identify the face region from the video frame sequence and generate a binary mask sequence. The determination module is used to convert the audio data into phoneme features, determine the target lip shape parameters according to the preset mapping relationship between phoneme features and lip shape parameters, and map the audio energy data into a displacement vector sequence; The update module is used to update the pixel values ​​of the mouth region marked by the binary mask sequence in the video frame sequence frame by frame based on the target lip shape parameters and the displacement vector sequence, so as to obtain a synchronized video sequence; The fusion module is used to perform prior encoding matching on the mouth region in the synchronized video sequence using a trained VQ-GAN-based repair network, then perform feature reconstruction, and fuse the reconstructed mouth region with the synchronized video sequence to obtain the target synchronized video.

9. An electronic device, comprising: include: Memory, used to store computer programs; A processor, configured to implement the steps of the voice-driven AI animation automatic generation and lip-syncing method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the AI ​​animation automatic generation and lip-syncing method based on voice-driven technology as described in any one of claims 1 to 7.

Images