Lip-shaped driving method and related device, electronic equipment and storage medium

Abstract

The invention discloses a lip-shaped driving method, a related device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the framing based on a target video, obtaining a plurality of video frames, extracting an audio frame group which is aligned with the time of each video frame from a target audio, and taking the audio frame group as a target frame group of the video frames; performing analysis based on the image features of the video frame to obtain a face posture of the target object in the video frame; selecting a first decoder or a second decoder to decode the image features and the acoustic features of the target frame group on the basis that the facial posture of the target object in the video frame is represented as a first posture type or a second posture type, and obtaining a decoded image of a mask area in the video frame; and fusing the decoded image of the mask region in the video frame and the target frame group of the video frame based on the video frame to obtain sub-multimedia data of the video frame for realizing lip-shaped driving. According to the scheme, the lip-shaped driving efficiency and robustness can be improved.

Description

Technical Field

[0001] This application relates to the field of multimedia interactive technology, and in particular to a lip-syncing method and related devices, electronic devices and storage media. Background Technology

[0002] With the development of artificial intelligence and multimedia interaction technology, lip-syncing technology enables virtual characters or smart devices to generate smooth lip movements by simulating speaking actions, thereby enhancing the immersiveness and realism of human-computer interaction.

[0003] In existing technologies, a unified lip-syncing model with massive parameters and complex structures is generally used to lip-sync relevant objects (such as virtual avatars) in each video frame. This leads to stuttering and latency issues when the lip-syncing model is running continuously, affecting the efficiency of lip-syncing. Therefore, improving the efficiency and robustness of lip-syncing has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a lip-driving method and related devices, electronic devices, and storage media that can improve the efficiency and robustness of lip-driving.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a lip-syncing method, comprising: segmenting a target video into several video frames, and extracting audio frame groups that are time-aligned with each video frame from the target audio, as target frame groups for the video frames; wherein the lip region of the target object in the video frame is a mask region; analyzing the image features of the video frame to obtain the facial pose of the target object in the video frame; wherein the facial pose belongs to a first pose type or a second pose type; based on the facial pose of the target object in the video frame being characterized as a first pose type or a second pose type, selecting a first decoder or a second decoder to decode the image features and the acoustic features of the target frame group to obtain a decoded image of the mask region in the video frame; wherein the number of parameters of the first decoder is lower than the number of parameters of the second decoder; fusing the video frame, the decoded image of the mask region in the video frame, and the target frame group of the video frame to obtain sub-multimedia data for lip-syncing of the video frame; wherein the target multimedia data of the target video after lip-syncing includes the sub-multimedia data of each video frame.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a lip-syncing device, comprising: an extraction module, an analysis module, a selection module, and a fusion module. The extraction module is used to segment a target video into several video frames and extract audio frame groups that are time-aligned with each video frame from the target audio, serving as the target frame group of the video frames; wherein the lip region of the target object in the video frame is a mask region. The analysis module is used to analyze the image features of the video frame to obtain the facial pose of the target object in the video frame; wherein the facial pose belongs to a first pose type or a second pose type. The selection module is used to select a first decoder or a second decoder to decode the image features and acoustic features of the target frame group based on the facial pose representation of the target object in the video frame as either the first pose type or the second pose type, obtaining a decoded image of the mask region in the video frame; wherein the number of parameters of the first decoder is lower than that of the second decoder. The fusion module is used to fuse the video frame, the decoded image of the mask region in the video frame, and the target frame group of the video frame to obtain sub-multimedia data of the video frame implementing lip-syncing; wherein the target multimedia data of the target video after lip-syncing includes the sub-multimedia data of each video frame.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the lip-sync driving method of the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the lip-sync driving method of the first aspect described above.

[0009] The above scheme involves segmenting the target video into frames to obtain several video frames, and extracting audio frame groups that are time-aligned with each video frame from the target audio, which serve as the target frame groups for the video frames. The lip region of the target object in the video frames is used as a mask region. Based on the image features of the video frames, the facial pose of the target object in the video frames is determined to be of a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frames as either the first or second pose type, a first decoder or a second decoder is selected to decode the image features and acoustic features of the target frame groups to obtain the decoded image of the mask region in the video frames. The number of parameters of the first decoder is lower than that of the second decoder. The video frames, the decoded image of the mask region in the video frames, and the target frame groups of the video frames are fused to obtain the sub-multimedia data of the video frames that implement lip-syncing. Based on the sub-multimedia data of each video frame, the target multimedia data of the target video with lip-syncing is obtained. Therefore, by selecting either a first decoder or a second decoder with different parameter values ​​based on the different facial poses of the target object, and by dynamically selecting the decoder, compared to applying the same model to all video frames indiscriminately for lip-syncing, the overall computational cost of lip-syncing for the target video can be reduced. This avoids the performance loss caused by indiscriminately using the same model while minimizing impact on the decoder's decoding performance, thus improving the efficiency of lip-syncing. Furthermore, because the decoding strategy can be adaptively adjusted according to the target object's facial pose, it has stronger adaptability to different scenarios. Therefore, it improves the efficiency and robustness of lip-syncing. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the lip-driving method of this application; Figure 2 This is a schematic diagram of an embodiment of the lip-sync driving method for acquiring video frames in this application; Figure 3 This is a schematic diagram of an embodiment of the lip-sync driving method for extracting audio frame groups in this application; Figure 4 This is a schematic diagram of the frame of an embodiment of the lip-shaped driving device of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0013] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0014] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0015] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the lip-driving method of this application. Specifically, it may include the following steps: Step S11: Segment the target video into several video frames, and extract audio frame groups that are time-aligned with each video frame from the target audio, as the target frame groups of the video frames.

[0016] In this embodiment, the target video is the original video material for which lip-syncing is required. Its format can be common video formats such as MP4 or AVI. Specifically, the target video can be a video of a digital human speaking, an animated video, etc. The target audio is the audio material corresponding to the target video. Its format can be common audio formats such as MP3 or WAV. Specifically, the target video can be a news broadcast audio, a song audio, etc.

[0017] It should be noted that the source of the target video and target audio is not limited in this application. For example, the target video can be a video generated by modeling a virtual avatar, or it can be a real video; the target audio can be real recorded audio, or it can be audio converted from the text to be broadcast. As mentioned above, when the target video and target audio are obtained by recording video and capturing sound from a real person, the usage rights of the target video and target audio can be obtained in advance by issuing an authorization request. The authorization request can, for example, clearly specify the scope of use, time limit, and other authorization conditions of the target video and target audio to meet compliance requirements.

[0018] In one implementation scenario, the target video is a non-streaming video. After the target video is acquired, it is split into several video frames through frame segmentation. The frame segmentation frequency can be set according to actual needs, such as 24 frames per second, 30 frames per second, etc.

[0019] In another implementation scenario, the target video is a streaming video. After acquiring the streaming video, it can also be split into several video frames through frame segmentation. The frame segmentation process can be dynamically adjusted according to the real-time characteristics of the video stream.

[0020] In one implementation scenario, the lip region of the target object in the video frame is the masked region. This can be achieved through masking technology. The masked region refers to marking or covering the lip region of the target object in the video frame so that lip-shaped driving operations can be performed on that region later.

[0021] In a specific implementation scenario, the lips of the target object in the video frame may be in an initial state that does not match the target audio at all, or may be in a state that matches the target audio with low accuracy, or may be in a missing state. This application does not limit these possibilities.

[0022] It should be noted that the mask area in the video frame must at least cover the lip area of ​​the target object. The specific area can be set according to actual needs. For example, it can also cover part of the facial area around the lips of the target object. This application does not limit this.

[0023] In a specific implementation scenario, the lip region is located using image recognition technology, and then a corresponding mask image is generated. This mask image has the same size as the video frame and is only masked in the lip region.

[0024] In another specific implementation scenario, based on a preset mask region, several video frames in the target video are masked. The preset mask region can be a fixed rectangular area that completely covers the lips of the target object in the video frame and the surrounding areas that may involve lip shape changes. Its position and size can be preset according to the average position of the target object in the video and the size of the lips. After acquiring each video frame of the target video, the preset mask region is directly applied to each video frame to achieve masking processing of the lip region.

[0025] In one implementation scenario, after framing to obtain several video frames, audio frame groups that are time-aligned with each video frame are extracted from the target audio and used as the target frame groups for the video frames. Specifically, timestamp matching and other methods can be used to ensure that each video frame has a time-corresponding audio frame group, thereby guaranteeing the synchronization and accuracy of subsequent lip-syncing. For example, for each video frame, its start and end times are found, and then audio segments matching these time ranges are found in the target audio and used as the target frame groups for that video frame.

[0026] Please refer to the following: Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of an embodiment of the lip-sync driving method for acquiring video frames in this application. Figure 3 This is a schematic diagram of an embodiment of the lip-sync driving method for extracting audio frame groups according to this application. Figure 2 and Figure 3 As shown, after acquiring the target video, frame segmentation is first performed, breaking the continuous video stream into a series of discrete video frames. This can be achieved through video processing algorithms. The frame segmentation frequency can be set according to actual needs, such as 24 or 30 frames per second, to ensure the continuity and smoothness of the video frames. Simultaneously, for each video frame, the lip region of the target object is identified and masked. Specifically, masking can be achieved using image recognition technology, such as using a deep learning model to locate the lip region, and then generating the final video frame. After completing the video frame segmentation and masking, audio frame groups aligned with the time of each video frame are extracted from the target audio.

[0027] In a specific implementation scenario, before extracting audio frame groups that are time-aligned with each video frame from the target audio, the consistency of parameters such as sampling rate and encoding format between the target audio and target video is checked to ensure accurate time alignment. If the parameters of the target audio and target video are inconsistent, appropriate conversion or adjustment is required to meet the time alignment requirements.

[0028] Step S12: Analyze the image features of the video frame to obtain the facial pose of the target object in the video frame.

[0029] In one implementation scenario, the facial pose belongs to either the first pose type or the second pose type. The first pose type presents more facial features than the second pose type.

[0030] In a specific implementation scenario, the first posture type and the second posture type represent facial postures under different facial deflection angles. Specifically, the facial deflection angle is the deflection angle of the target object's face relative to the direction of the face-viewing video acquisition device. The facial deflection angle under the first posture type is not greater than a preset threshold, while the facial deflection angle under the second posture type is greater than the preset threshold.

[0031] In a specific implementation scenario, the first pose type is the frontal face pose, and the second pose type is the non-frontal face pose, such as the upper side face pose, the lower side face pose, the left side face pose, the right side face pose, etc.

[0032] In a specific implementation scenario, a pre-trained facial pose classifier is used to analyze the facial pose of the target object. Specifically, the facial pose classifier calculates the mean of all encoded feature vectors in the frontal face sample set based on several sample images labeled as frontal faces, and obtains the frontal face encoded feature center. When analyzing the facial pose of the target object in the video frame of the target video, the distance between the video frame and the frontal face encoded feature center is calculated based on the image features of the video frame. If the distance is less than a preset threshold, the facial pose of the target object is determined to be of the first pose type; if the distance is greater than or equal to the preset threshold, the facial pose of the target object is determined to be of the second pose type.

[0033] In another specific implementation scenario, deep learning models, such as convolutional neural networks (CNNs), are used to extract and analyze the image features of video frames. This model is trained on a large amount of image data labeled with facial poses (front view, upper side view, lower side view, left side view, right side view, etc.) and can identify the facial poses of the target object in the video frame.

[0034] In one implementation scenario, before analyzing the image features of a video frame to obtain the facial pose of the target object, a first sub-feature of the facial region of the target object is extracted, and a second sub-feature of the region outside the facial region is extracted. Specifically, the region outside the facial region includes areas such as the neck and shoulders. Based on the first target weight of the facial region and the second target weight of the region outside the facial region, the first and second sub-features are weighted and fused respectively to obtain the image features of the video frame. The first target weight represents the degree of influence of the facial region on the determination of facial pose, and the second target weight represents the degree of influence of the region outside the facial pose determination. This approach, by extracting sub-features from the facial region and the region outside the facial region separately and then weighting and fusing them, can more comprehensively utilize the information in the video frame, helping to improve the accuracy of facial pose analysis. For example, in some special cases, it may be difficult to accurately determine facial pose based solely on the facial region, while combining information from the neck, shoulders, and other regions can provide more clues, thereby more accurately determining whether the target object belongs to the first or second pose type.

[0035] In a specific implementation scenario, the target object parts corresponding to the regions other than the face region are further assigned corresponding sub-weights. For example, the target object's neck region corresponds to the neck sub-weight, the shoulder region corresponds to the shoulder sub-weight, and the target object's limbs region corresponds to the limbs sub-weight, etc.

[0036] In a specific implementation scenario, based on a first target weight for the facial region and a second target weight for the surrounding regions, weighted fusion is performed on the first and second sub-features, respectively. Before obtaining the image features of the video frame, the weight allocation ratio is determined based on the area ratio between the facial region and the surrounding regions in the video frame. The weight allocation ratio is positively correlated with the area ratio. For example, if the facial region is larger, the first target weight is relatively higher, and the second target weight for the surrounding regions is relatively lower; conversely, if the surrounding regions are larger, the second target weight is relatively higher, and the first target weight is relatively lower. In this way, the weight allocation ratio can be dynamically adjusted according to the actual area of ​​different regions in the video frame, further improving the accuracy and adaptability of facial pose analysis. Based on the weight allocation ratio, the first preset weight of the facial region and the second preset weight of the surrounding regions are adjusted to obtain the first target weight and the second target weight. Specifically, the first preset weight and the second preset weight represent the initial weight values ​​preset before the weight allocation ratio is adjusted. These preset weights can be set based on experience or preliminary experiments. The above scheme, by dynamically adjusting the weight allocation ratio based on the area ratio and further adjusting the preset weights to obtain the target weights, can more flexibly adapt to the differences in area in different video frames, making the facial pose analysis process more accurate and reliable.

[0037] In one implementation scenario, before analyzing image features from video frames to obtain the facial pose of the target object, sound source localization is performed based on the target frame group to obtain a first candidate pose and a first confidence level of the target object's face. Then, prediction is performed based on image features from the video frames to obtain a second candidate pose and a second confidence level of the target object's face. Finally, the first candidate pose and its first confidence level, along with the second candidate pose and its second confidence level, are fused to obtain the final facial pose. This scheme, by combining facial pose information obtained from sound source localization and facial pose information predicted from image features, and comprehensively considering the confidence levels of both, fully utilizes information from both audio and video dimensions, improving the accuracy and robustness of facial pose determination.

[0038] In a specific implementation scenario, the physical representations of the first and second candidate poses are angle data. Specifically, by performing sound source localization based on target frame groups, the angle data of the target audio pronunciation position relative to the center of the target object can be obtained. That is, the angle data in the first candidate pose reflects the position of the audio pronunciation, while the angle data in the second candidate pose, obtained by predicting based on the image features of the video frame, reflects the angular state of the target object's face in the video frame. During the fusion process, the first and second candidate poses can be weighted and averaged according to the magnitude of the first and second confidence levels. For example, if the first confidence level is higher, the first candidate pose is given a larger weight; if the second confidence level is higher, the second candidate pose is given a larger weight, ultimately resulting in the fused facial pose.

[0039] In another specific implementation scenario, the physical representations of the first and second candidate poses are category results. Specifically, sound source localization is performed based on the target frame group, and the possible facial pose categories corresponding to the audio pronunciation positions are determined to obtain the category result of the first candidate pose, such as frontal face, side face, etc. The category result of the second candidate pose, obtained by predicting the image features of the video frame, is also a determination of the category to which the target object's facial pose belongs in the video frame. During the fusion process, the first and second candidate poses can be voted on based on the magnitude of the first and second confidence scores. For example, if the first confidence score is significantly higher than the second confidence score, the category result of the first candidate pose is selected as the final facial pose category; if the two confidence scores are similar, the category results of both are considered comprehensively, such as selecting the more common category or determining the final category through other rules, ultimately obtaining the fused facial pose.

[0040] It should be noted that the descriptions of the first and second candidate postures above are only possible implementation methods, and the specific physical representations are not limited in this application.

[0041] In a specific implementation scenario, sound source localization can be achieved through various technical means, such as sound source localization technology based on microphone arrays.

[0042] In a specific implementation scenario, the first confidence level of the first candidate pose represents the reliability of the sound source localization result. Its numerical range can be set according to actual needs, for example, between 0 and 1, with the closer the value is to 1, the higher the confidence level. Similarly, the second confidence level of the second candidate pose represents the reliability of the prediction result based on image features, and is also set between 0 and 1.

[0043] In one implementation scenario, when processing video frames sequentially based on time sequence, before analyzing the image features of the current video frame to obtain the current facial pose of the target object in the current video frame, the historical facial pose of the target object in the previous video frame is obtained. Based on the historical facial pose and the image features of the current video frame, the current facial pose of the target object in the current video frame is obtained through analysis. This scheme, by introducing historical facial pose information, leverages the temporal correlation between video frames, making the analysis of the current facial pose more accurate and stable.

[0044] In a specific implementation scenario, a temporal analysis model is used to capture the dynamic changes between video frames based on historical facial poses and image features of the current video frame. Specifically, the temporal analysis model receives historical facial pose information from the previous video frame and image features of the current video frame as input. Through an internal state transition mechanism or memory unit, it comprehensively considers historical information and current features to output the current facial pose of the target object in the current video frame. For example, a temporal analysis model with a network structure such as a recurrent neural network (RNN) or its variant, a long short-term memory network (LSTM), can be employed. This approach, when processing videos with continuous movements or facial expression changes, can more accurately track the changing trends of the target object's facial pose, improving the continuity and stability of facial pose analysis.

[0045] In a specific implementation scenario, the decoded image of the previous video frame can also be obtained. Based on the decoded image of the previous video frame, historical facial poses, and image features of the current video frame, the current facial pose of the target object in the current video frame can be obtained.

[0046] In one implementation scenario, the image features of a video frame can be extracted by an encoder. Specifically, the encoder consists of three residual convolutional modules and a downsampled convolutional layer. The input video frame has a size of H×W×3, and after passing through the encoder, the output image features have a size of (H / 16)×(W / 16)×192.

[0047] Step S13: Based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, select a first decoder or a second decoder to decode the image features and the acoustic features of the target frame group to obtain the decoded image of the masked region in the video frame.

[0048] In this embodiment of the disclosure, the number of parameters of the first decoder is lower than the number of parameters of the second decoder. Specifically, the first decoder can be a lightweight version of the second decoder.

[0049] It should be noted that the structure of the first decoder and the second decoder is not limited in this application.

[0050] In one implementation scenario, the acoustic features of the target frame group include the audio's frequency, energy, and spectral characteristics. Specifically, acoustic features of the target frame group can be extracted using algorithms such as Short Time Fourier Transform (STFT), converting the time-domain audio signal into a frequency-domain spectral representation to obtain rich acoustic feature vectors. Alternatively, features such as Mel-frequency cepstral coefficients (MFCCs) of the target frame group can be extracted to better characterize the audio's timbre, pitch, and other information, providing a more comprehensive acoustic basis for the subsequent decoding process.

[0051] In a specific implementation scenario, the extracted acoustic features can be further normalized to map them to a specific numerical range, such as between 0 and 1, in order to eliminate the differences between different audio samples caused by factors such as volume and recording environment, improve the stability and consistency of acoustic features, and thus improve the accuracy of decoding.

[0052] In one implementation scenario, in response to the facial pose representation of the target object in a video frame being classified as a first pose type, a first decoder is selected to decode the image features and the acoustic features of the target frame group to obtain a decoded image. In this scheme, since the facial features and lip movements of the target object under the first pose type can vary only slightly, using a first decoder with relatively few parameters can improve processing efficiency and reduce computational resource consumption while ensuring decoding quality as much as possible.

[0053] In a specific implementation scenario, the first decoder can adopt a simple network structure, such as a shallow convolutional neural network structure, to reduce the number of network layers and parameters. At the same time, by optimizing training strategies, such as using appropriate data augmentation methods and loss functions, its performance in the first pose type decoding task can be ensured.

[0054] In a specific implementation scenario, when the facial pose of the target object in a video frame is represented as a first pose type, facial reference features of the non-masked region under the first pose type of the target object are extracted based on the image features of the video frame. The facial reference features represent the contour data and key point positions of the face under the first pose type. Specifically, when the first pose type is a frontal face pose, the key point positions of the frontal face, such as the center coordinates of key parts like the eyes, nose, and mouth, can be detected and obtained through the first decoder. These key point positions can describe the geometric structure of the frontal face, and the contour data is used to represent the outer edge shape of the frontal face. Based on the facial reference features, the quantized data of lip feature changes under the first pose type of the target object is predicted. The quantized data of lip feature changes represents the inter-frame displacement of lip key points and the amplitude of contour deformation. Based on the quantized data of lip feature changes, image features, and acoustic features of the target frame group, decoding is performed to obtain the decoded image. Even if the first decoder adopts a lighter network structure, the above scheme can still provide more accurate auxiliary information for the decoding process by extracting facial reference features under the first pose type and predicting lip feature changes to quantify the data. This can compensate for the information loss that may be caused by the lightweight network structure and ensure the accuracy and naturalness of the lip region in the decoded image.

[0055] In another implementation scenario, in response to the facial pose representation of the target object in the video frame being classified as a second pose type, a second decoder is selected to decode the image features and the acoustic features of the target frame group to obtain a decoded image. In the above scheme, because the facial features and lip movements of the target object change significantly under the second pose type, and because facial features and lip movements in different offset directions also exhibit significant differences under the second pose type, using a second decoder with relatively higher parameters and a more complex structure can better capture these detailed changes, improving the accuracy and quality of the decoded image.

[0056] In a specific implementation scenario, the second decoder can employ a deep convolutional neural network structure, increasing the number of network layers and parameters to enhance its ability to model complex facial poses and lip movements. Simultaneously, by training on large-scale image data labeled with second pose types and lip movements, the second decoder can learn rich facial features and lip movement patterns under the second pose type. During training, various loss functions can be combined, such as using the mean squared error loss function to measure the pixel difference between the decoded image and the real image, and using a structural similarity loss function to focus on the structural similarity between the decoded image and the real image, thereby improving the quality of the decoded image.

[0057] In a specific implementation scenario, the second decoder can be further optimized for different second pose types, such as upper, lower, left, and right profiles. Specifically, for the upper second pose type, the second decoder can enhance its ability to capture forehead and eyebrow movements. By adding samples of different expressions and lip movements under the upper second pose type to the training data, the decoder can more accurately reproduce facial details under the upper second pose type. For the lower profile, the focus is on optimizing the modeling of the chin and neck regions. Considering the impact of chin contour changes and neck muscle movements on the overall facial posture when the lower profile is viewed, the feature extraction capability for these regions is enhanced by adjusting the size and number of relevant convolutional kernels in the network structure. For the left and right profiles, the second decoder can focus on the extraction and analysis of features on the left or right side of the face, respectively. For example, different channel attention mechanisms can be set in the convolutional neural network, enabling the network to automatically focus on key feature regions of the corresponding profile. For the left profile, the focus is on the left eye and left cheek, while for the right profile, the focus is on the right eye and right cheek, thereby more accurately decoding images under different second pose types. The above scheme enables the second decoder to achieve better decoding results when processing second pose types in different directions, thereby improving the accuracy and naturalness of the overall decoded image.

[0058] Step S14: Based on the video frame, the decoded image of the masked region in the video frame, and the target frame group of the video frame, fuse them to obtain the sub-multimedia data of the video frame that implements lip-sync driving.

[0059] In this embodiment of the disclosure, the target multimedia data of the target video driven by lip-sync includes sub-multimedia data of each video frame.

[0060] In one implementation scenario, during the fusion process, video frames are used as the base images. The decoded images of the masked regions in the video frames are superimposed onto the corresponding masked positions of the video frames. Then, the target frame groups of the video frames are superimposed accordingly. Based on the temporal relationship of each video frame, the fused sub-multimedia data are combined sequentially to obtain smooth target multimedia to achieve lip-syncing.

[0061] The above scheme involves segmenting the target video into frames to obtain several video frames, and extracting audio frame groups that are time-aligned with each video frame from the target audio, which serve as the target frame groups for the video frames. The lip region of the target object in the video frames is used as a mask region. Based on the image features of the video frames, the facial pose of the target object in the video frames is determined to be of a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frames as either the first or second pose type, a first decoder or a second decoder is selected to decode the image features and acoustic features of the target frame groups to obtain the decoded image of the mask region in the video frames. The number of parameters of the first decoder is lower than that of the second decoder. The video frames, the decoded image of the mask region in the video frames, and the target frame groups of the video frames are fused to obtain the sub-multimedia data of the video frames that implement lip-syncing. Based on the sub-multimedia data of each video frame, the target multimedia data of the target video with lip-syncing is obtained. Therefore, by selecting either a first decoder or a second decoder with different parameter values ​​based on the different facial poses of the target object, and by dynamically selecting the decoder, compared to applying the same model to all video frames indiscriminately for lip-syncing, the overall computational cost of lip-syncing for the target video can be reduced. This avoids the performance loss caused by indiscriminately using the same model while minimizing impact on the decoder's decoding performance, thus improving the efficiency of lip-syncing. Furthermore, because the decoding strategy can be adaptively adjusted according to the target object's facial pose, it has stronger adaptability to different scenarios. Therefore, it improves the efficiency and robustness of lip-syncing.

[0062] Please see Figure 4 , Figure 4This is a schematic diagram of the framework of an embodiment of the lip-shaped driving device of this application. The lip-sync driving device 40 includes an extraction module 41, an analysis module 42, a selection module 43, and a fusion module 44. The extraction module 41 is used to segment the target video into several video frames and extract audio frame groups that are time-aligned with each video frame from the target audio, which are used as the target frame groups of the video frames. The lip region of the target object in the video frame is a mask region. The analysis module 42 is used to analyze the image features of the video frame to obtain the facial pose of the target object in the video frame. The facial pose belongs to a first pose type or a second pose type. The selection module 43 is used to select a first decoder or a second decoder to decode the image features and acoustic features of the target frame group based on whether the facial pose of the target object in the video frame is a first pose type or a second pose type, so as to obtain the decoded image of the mask region in the video frame. The parameter number of the first decoder is lower than that of the second decoder. The fusion module 44 is used to fuse the video frame, the decoded image of the mask region in the video frame, and the target frame group of the video frame to obtain the sub-multimedia data of the video frame that implements lip-sync driving. The target multimedia data of the target video after lip-sync driving includes the sub-multimedia data of each video frame.

[0063] In the above scheme, the lip-syncing device 40 performs frame segmentation based on the target video to obtain several video frames, and extracts audio frame groups that are time-aligned with each video frame from the target audio as the target frame group of the video frame, with the lip region of the target object in the video frame being the mask region; based on the image features of the video frame, it is determined that the facial pose of the target object in the video frame belongs to a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, a first decoder or a second decoder is selected to decode the image features and acoustic features of the target frame group to obtain the decoded image of the mask region in the video frame, with the number of parameters of the first decoder being lower than that of the second decoder. Based on the video frame, the decoded image of the mask region in the video frame, and the target frame group of the video frame, the sub-multimedia data of the video frame implementing lip-syncing is obtained, and the target multimedia data of the target video after lip-syncing is obtained based on the sub-multimedia data of each video frame. Therefore, by selecting either a first decoder or a second decoder with different parameter values ​​based on the different facial poses of the target object, and by dynamically selecting the decoder, compared to applying the same model to all video frames indiscriminately for lip-syncing, the overall computational cost of lip-syncing for the target video can be reduced. This avoids the performance loss caused by indiscriminately using the same model while minimizing impact on the decoder's decoding performance, thus improving the efficiency of lip-syncing. Furthermore, because the decoding strategy can be adaptively adjusted according to the target object's facial pose, it has stronger adaptability to different scenarios. Therefore, it improves the efficiency and robustness of lip-syncing.

[0064] In some disclosed embodiments, the selection module 43 further includes a first response module (not shown), used to select a first decoder to decode the image features and the acoustic features of the target frame group in response to the facial pose representation of the target object in the video frame being a first pose type, thereby obtaining a decoded image; the selection module 43 further includes a second response module (not shown), used to select a second decoder to decode the image features and the acoustic features of the target frame group in response to the facial pose representation of the target object in the video frame being a second pose type, thereby obtaining a decoded image.

[0065] In some disclosed embodiments, the first response module (not shown) further includes a reference feature extraction module (not shown), used to extract facial reference features of the non-masked region of the target object under the first pose type based on the image features of the video frame; wherein, the facial reference features represent the facial contour data and key point positions under the first pose type; the first response module (not shown) further includes a lip change prediction module (not shown), used to predict the quantized data of lip feature changes under the first pose type of the target object based on the facial reference features; wherein, the quantized data of lip feature changes represents the inter-frame displacement of lip key points and the amplitude of contour deformation; the first response module (not shown) further includes a decoding submodule (not shown), used to decode based on the quantized data of lip feature changes, image features and acoustic features of the target frame group to obtain a decoded image.

[0066] In some disclosed embodiments, before analyzing the image features based on the video frame to obtain the facial pose of the target object in the video frame, the lip-shaped driving device 40 further includes a sub-feature extraction module (not shown) for extracting a first sub-feature of the facial region of the target object in the video frame and extracting a second sub-feature of the region outside the facial region of the target object in the video frame; the lip-shaped driving device 40 also includes a first weighting module (not shown) for weighting and fusing the first sub-feature and the second sub-feature based on a first target weight of the facial region and a second target weight of the region outside the facial region to obtain the image features of the video frame; wherein, the first target weight represents the degree of influence of the facial region on the determination of facial pose, and the second target weight represents the degree of influence of the region outside the facial pose determination.

[0067] In some disclosed embodiments, the first sub-feature and the second sub-feature are weighted and fused based on the first target weight of the facial region and the second target weight of the outer region, respectively. Before obtaining the image features of the video frame, the lip-shaped driving device 40 further includes a ratio determination module (not shown), which is used to determine the weight allocation ratio based on the area ratio between the facial region and the outer region in the video frame; wherein the weight allocation ratio is positively correlated with the area ratio; the lip-shaped driving device 40 also includes a weight adjustment module (not shown), which is used to adjust the first preset weight of the facial region and the second preset weight of the outer region based on the weight allocation ratio, respectively, to obtain the first target weight and the second target weight.

[0068] In some disclosed embodiments, before analyzing the image features of the video frame to obtain the facial pose of the target object in the video frame, the lip-syncing device 40 further includes a sound source localization module (not shown) for performing sound source localization based on the target frame group to obtain a first candidate pose of the target object's face and a first confidence level of the first candidate pose; the analysis module 42 further includes a pose prediction module (not shown) for predicting based on the image features of the video frame to obtain a second candidate pose of the target object's face and a second confidence level of the second candidate pose; the analysis module 42 further includes a pose fusion module (not shown) for fusing the first candidate pose and the first confidence level of the first candidate pose, the second candidate pose and the second confidence level of the second candidate pose to obtain the facial pose.

[0069] In some disclosed embodiments, when processing video frames sequentially based on time sequence, before analyzing the image features of the current video frame to obtain the current facial pose of the target object in the current video frame, the lip-syncing device 40 further includes a historical pose acquisition module (not shown) for acquiring the historical facial pose of the target object in the previous video frame; the analysis module 42 further includes an analysis submodule (not shown) for analyzing the historical facial pose and the image features of the current video frame to obtain the current facial pose of the target object in the current video frame.

[0070] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes at least a memory 51 and a processor 52 coupled to each other. The memory 51 stores at least program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-described lip-driving method embodiments.

[0071] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described lip-driving method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0072] In the above scheme, the electronic device 50 performs frame segmentation based on the target video to obtain several video frames, and extracts audio frame groups that are time-aligned with each video frame from the target audio as the target frame group of the video frame, and the lip region of the target object in the video frame is the mask region; based on the image features of the video frame, it is determined that the facial pose of the target object in the video frame belongs to a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, a first decoder or a second decoder is selected to decode the image features and acoustic features of the target frame group to obtain the decoded image of the mask region in the video frame, and the number of parameters of the first decoder is lower than that of the second decoder. Based on the video frame, the decoded image of the mask region in the video frame and the target frame group of the video frame, the sub-multimedia data of the video frame implemented with lip-sync is obtained, and the target multimedia data of the target video with lip-sync is obtained based on the sub-multimedia data of each video frame. Therefore, by selecting either a first decoder or a second decoder with different parameter values ​​based on the different facial poses of the target object, and by dynamically selecting the decoder, compared to applying the same model to all video frames indiscriminately for lip-syncing, the overall computational cost of lip-syncing for the target video can be reduced. This avoids the performance loss caused by indiscriminately using the same model while minimizing impact on the decoder's decoding performance, thus improving the efficiency of lip-syncing. Furthermore, because the decoding strategy can be adaptively adjusted according to the target object's facial pose, it has stronger adaptability to different scenarios. Therefore, it improves the efficiency and robustness of lip-syncing.

[0073] Please see Figure 6 , Figure 6This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described embodiments of the lip-driving method.

[0074] In the above scheme, the computer-readable storage medium 60 performs frame segmentation based on the target video to obtain several video frames, and extracts audio frame groups that are time-aligned with each video frame from the target audio as the target frame group of the video frame, with the lip region of the target object in the video frame being the mask region; based on the image features of the video frame, it is determined that the facial pose of the target object in the video frame belongs to a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, a first decoder or a second decoder is selected to decode the image features and acoustic features of the target frame group to obtain the decoded image of the mask region in the video frame, with the number of parameters of the first decoder being lower than that of the second decoder. Based on the video frame, the decoded image of the mask region in the video frame, and the target frame group of the video frame, the sub-multimedia data of the video frame implementing lip-sync is obtained, and the target multimedia data of the target video with lip-sync is obtained based on the sub-multimedia data of each video frame. Therefore, by selecting either a first decoder or a second decoder with different parameter values ​​based on the different facial poses of the target object, and by dynamically selecting the decoder, compared to applying the same model to all video frames indiscriminately for lip-syncing, the overall computational cost of lip-syncing for the target video can be reduced. This avoids the performance loss caused by indiscriminately using the same model while minimizing impact on the decoder's decoding performance, thus improving the efficiency of lip-syncing. Furthermore, because the decoding strategy can be adaptively adjusted according to the target object's facial pose, it has stronger adaptability to different scenarios. Therefore, it improves the efficiency and robustness of lip-syncing.

[0075] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0076] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0079] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A lip-shaped actuation method, characterized in that, include: The target video is segmented into frames to obtain several video frames. Audio frame groups that are time-aligned with each of the video frames are extracted from the target audio and used as the target frame groups of the video frames. The lip region of the target object in the video frame is a mask region. Based on the image features of the video frame, the facial pose of the target object in the video frame is obtained; wherein the facial pose belongs to a first pose type or a second pose type. Based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, a first decoder or a second decoder is selected to decode the image features and the acoustic features of the target frame group to obtain the decoded image of the masked region in the video frame; wherein, the number of parameters of the first decoder is lower than the number of parameters of the second decoder. Based on the video frame, the decoded image of the masked region in the video frame, and the target frame group of the video frame, the sub-multimedia data of the video frame implemented by lip-sync is obtained; wherein, the target multimedia data of the target video implemented by lip-sync includes the sub-multimedia data of each video frame.

2. The method according to claim 1, characterized in that, The first pose type presents more facial features than the second pose type. The step of representing the facial pose of the target object in the video frame as either the first or second pose type, selecting a first decoder or a second decoder to decode the image features and the acoustic features of the target frame group, and obtaining the decoded image of the masked region in the video frame includes: In response to the facial pose of the target object in the video frame being characterized as the first pose type, the first decoder is selected to decode the image features and the acoustic features of the target frame group to obtain the decoded image; In response to the facial pose representation of the target object in the video frame being of the second pose type, the second decoder is selected to decode the image features and the acoustic features of the target frame group to obtain the decoded image.

3. The method according to claim 2, characterized in that, The response is that the facial pose of the target object in the video frame is characterized as the first pose type, and the first decoder is selected to decode the image features and the acoustic features of the target frame group to obtain the decoded image, including: Based on the image features of the video frame, facial reference features of the non-masked region of the target object under the first pose type are extracted; wherein, the facial reference features represent the facial contour data and key point positions under the first pose type; Based on the facial reference features, predict the quantitative data of lip feature changes under the first pose type of the target object; wherein, the quantitative data of lip feature changes characterizes the inter-frame displacement and contour deformation amplitude of lip key points; The decoded image is obtained by decoding based on the quantized data of lip feature changes, the image features, and the acoustic features of the target frame group.

4. The method according to claim 1, characterized in that, Before analyzing the image features based on the video frame to obtain the facial pose of the target object in the video frame, the method further includes: Extract the first sub-feature of the facial region of the target object in the video frame, and extract the second sub-feature of the region other than the facial region of the target object in the video frame; Based on the first target weight of the facial region and the second target weight of the external region, the first sub-feature and the second sub-feature are weighted and fused respectively to obtain the image features of the video frame; wherein, the first target weight represents the degree of influence of the facial region on the determination of facial posture, and the second target weight represents the degree of influence of the external region on the determination of facial posture.

5. The method according to claim 4, characterized in that, Before weighted fusing the first sub-feature and the second sub-feature based on the first target weight of the facial region and the second target weight of the other regions to obtain the image features of the video frame, the method further includes: A weight allocation ratio is determined based on the area ratio between the facial region and the other regions in the video frame; wherein the weight allocation ratio is positively correlated with the area ratio. Based on the weight allocation ratio, the first preset weight of the facial region and the second preset weight of the other regions are adjusted to obtain the first target weight and the second target weight.

6. The method according to claim 1, characterized in that, Before analyzing the image features based on the video frame to obtain the facial pose of the target object in the video frame, the method further includes: Based on the target frame group, sound source localization is performed to obtain the first candidate pose of the target object's face and the first confidence level of the first candidate pose. The step of analyzing the image features based on the video frame to obtain the facial pose of the target object in the video frame includes: Based on the image features of the video frame, a second candidate pose of the target object's face and a second confidence level of the second candidate pose are obtained; The facial pose is obtained by fusing the first candidate pose and its first confidence level, the second candidate pose and its second confidence level.

7. The method according to claim 1, characterized in that, In the case of processing the video frames sequentially based on time sequence, before analyzing the image features of the current video frame to obtain the current facial pose of the target object in the current video frame, the method further includes: Obtain the historical facial pose of the target object in the previous video frame; The step of analyzing image features based on the current video frame to obtain the current facial pose of the target object in the current video frame includes: Based on the analysis of the historical facial pose and the image features of the current video frame, the current facial pose of the target object in the current video frame is obtained.

8. A lip-shaped driving device, characterized in that, include: The extraction module is used to divide the target video into frames to obtain several video frames, and extract audio frame groups that are time-aligned with each of the video frames from the target audio, as the target frame group of the video frames; wherein, the lip region of the target object in the video frame is a mask region; The analysis module is used to analyze the image features of the video frame to obtain the facial pose of the target object in the video frame; wherein the facial pose belongs to a first pose type or a second pose type. The selection module is used to select a first decoder or a second decoder to decode the image features and the acoustic features of the target frame group based on the facial pose representation of the target object in the video frame as a first pose type or a second pose type, so as to obtain the decoded image of the masked region in the video frame; wherein, the number of parameters of the first decoder is lower than the number of parameters of the second decoder. The fusion module is used to fuse the video frame, the decoded image of the masked region in the video frame, and the target frame group of the video frame to obtain the lip-sync driven sub-multimedia data of the video frame; wherein, the lip-sync driven target multimedia data of the target video includes the sub-multimedia data of each video frame.

9. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the lip-driving method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the lip-driving method according to any one of claims 1 to 7.

Images