Audio and video synchronization method and device, storage medium and program product

Abstract

The invention discloses an audio and video synchronization method and device, a storage medium and a program product. The method comprises the following steps: acquiring multimedia data, decomposing audio data and video data corresponding to a first time length from the multimedia data, and caching multiple frames of images contained in the video data; extracting a mouth region in the multiple frames of images to obtain multiple first pictures containing the mouth region; a standard face model is driven based on the audio data, and at least one second picture containing mouth features of the standard face model is obtained; and comparing the plurality of first pictures with the at least one second picture, and performing synchronization processing on the audio data and the video data according to a comparison result, thereby improving the accuracy of judging audio and video synchronization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an audio-visual synchronization method, device, storage medium, and program product. Background Technology

[0002] With the widespread use of video applications, maintaining audio and video synchronization during playback has become crucial for improving the user viewing experience. Currently, the main causes of audio-video desynchronization include misaligned arrival times of audio and video packets, uncorrected cumulative latency, and timestamp discrepancies.

[0003] The current audio and video synchronization methods include the following two: one is to use the timestamps of audio and video packets to match the system clock to achieve synchronization. However, when the timestamps of the generated audio and video packets are out of sync, it is impossible to achieve the effect of audio and video synchronization by aligning the system time.

[0004] Another approach is to use audio data to drive the lip movements of the person in the image, replacing the lip movements in the video data with the generated lip movements, thus achieving audio-video synchronization. However, this method results in the user seeing a lip-driven image that is then pasted back onto the original image, rather than the original image. Furthermore, current digital human simulation technology is immature, producing poor-quality digital humans that cannot truly solve the problem of audio-video asynchrony. Summary of the Invention

[0005] To address the existing technical problems, embodiments of the present invention provide an audio-video synchronization method, device, storage medium, and program product to improve the accuracy of determining audio-video synchronization.

[0006] To achieve the above objectives, the technical solution of this invention is implemented as follows: In a first aspect, embodiments of the present invention provide an audio-video synchronization method, the method comprising: Acquire multimedia data, extract audio and video data corresponding to a first time length from the multimedia data, and cache multiple frames of images contained in the video data; Extract the mouth region from the multiple frames of images to obtain multiple first images containing the mouth region; Based on the audio data, a standard face model is driven to obtain at least one second image containing the mouth features of the standard face model; The audio data and the video data are synchronized based on the comparison results by comparing the multiple first images and the at least one second image.

[0007] In the above scheme, the step of driving a standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model includes: extracting features from a frame of image to obtain head pose information in the image; rotating the standard face model according to the head pose information to obtain a rotated standard face model; and driving the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features.

[0008] In the above scheme, the step of extracting features from a frame of image to obtain head pose information in the image includes: performing face detection on the frame of image to obtain face regions in the image; performing head pose detection on the face regions to obtain head pose information; the head pose information includes one or more of the head pitch angle, yaw angle and roll angle.

[0009] In the above scheme, the step of driving the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model includes: driving the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model; and / or, performing speech recognition processing on the audio data to obtain text information corresponding to the audio data, and driving the rotated standard face model based on the text information to obtain at least one second image containing the mouth features of the standard face model; wherein, the first audio data corresponding to each second image corresponds to the same time parameter as one frame of the multi-frame images.

[0010] In the above scheme, comparing the plurality of first images and the at least one second image, and synchronizing the audio data and the video data based on the comparison results, includes: calculating the similarity of the mouth region between the nth second image and each of the plurality of first images; determining the mth first image to be synchronized with the nth second image based on the similarity; and synchronizing the audio data and the video data based on the nth second image and the mth first image.

[0011] In the above scheme, determining the m-th first image synchronized with the n-th second image based on the similarity includes: determining a first similarity among multiple similarities that meets a preset condition, and using the first image corresponding to the first similarity as the m-th first image synchronized with the n-th second image; wherein, the preset condition includes: the similarity is greater than or equal to a preset threshold, or the similarity is the maximum.

[0012] In the above scheme, the synchronization processing of the audio data and the video data based on the nth second image and the mth first image includes: performing time alignment processing on the audio data and the video data according to the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image, so that the first audio data and the first image correspond to the same time parameter; or, determining the time difference according to the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image, and adjusting the output strategy of the audio data and / or the video data based on the time difference to perform synchronization processing on the audio data and the video data.

[0013] Secondly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the embodiments of the present invention.

[0014] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0015] Fourthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the embodiments of the present invention.

[0016] This invention provides an audio-video synchronization method, device, storage medium, and program product. By acquiring multimedia data, including audio and video data, corresponding to a multimedia video, multiple first images containing the mouth region are extracted from multiple frames of cached video data. Based on the audio data, a standard face model is driven to obtain at least one second image containing the mouth features of the standard face model. Synchronization processing of the audio and video data in the multimedia data is achieved based on the comparison result obtained by comparing the first and second images. By using the original audio data to drive the standard face model, the mouth (or lip shape) features corresponding to the original audio data are accurately simulated. The second image corresponding to the standard face model and the first image obtained based on the actual video data are used to determine whether the audio and video are synchronized. Synchronization processing of the audio and video data is then performed to improve the accuracy of audio-video synchronization judgment. This avoids the synchronization method of using a digitally generated human with poor facial and mouth effects to replace the human face in the actual video data. Without modifying the original audio and video data, this solves the problem of audio-video asynchrony caused by time delays or uncorrected time delays between the audio and video streams, or timestamp discrepancies between the audio and video streams. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating existing audio-video synchronization methods. Figure 2 This is a flowchart illustrating the audio and video synchronization method according to an embodiment of the present invention. Figure 1 ; Figure 3 This is a flowchart illustrating the audio and video synchronization method according to an embodiment of the present invention. Figure 2 ; Figure 4 This is a flowchart illustrating an example of an audio-video synchronization method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the composition structure of the audio and video synchronization device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0019] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0020] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] With the widespread use of video applications, maintaining audio and video synchronization during playback has become crucial for improving the user viewing experience. Currently, the main causes of audio-video desynchronization include misaligned arrival times of audio and video packets, uncorrected cumulative latency, and timestamp discrepancies.

[0022] Figure 1This is a flowchart illustrating existing audio-video synchronization methods; such as... Figure 1 As shown, the existing process for achieving audio-video synchronization is as follows: Step 1: Acquire audio and video streams, and decompose the acquired audio and video streams to obtain the corresponding audio and video streams; Step 2: Use an audio decoder to decode the audio stream to obtain the corresponding audio data, and cache the decoded audio data in the audio buffer; Step 3: Use a video decoder to decode the video stream to obtain the corresponding image sequence, and cache the decoded image sequence in the image buffer; Step 4: Use audio-video synchronization methods to achieve synchronized processing of audio data and image sequences; Step 5: Display the image to be synchronized and play the corresponding audio data to be synchronized at the same time to complete the audio and video synchronization.

[0023] Based on the above Figure 1 The existing audio and video synchronization processing flow includes the following two methods: one is to use the timestamps of audio packets and video packets to match the system clock to achieve synchronization. However, when the timestamps of the generated audio packets and video packets are out of sync, it is impossible to achieve the effect of audio and video synchronization by aligning the system time.

[0024] Another approach is to use audio data to drive the lip movements of the person in the image, replacing the lip movements in the video data with the generated lip movements, thus achieving audio-video synchronization. However, this method results in the user seeing a lip-driven image that is no longer the original image, but rather an image that has been pasted back onto the original image. Furthermore, current digital human simulation technology is not mature, and the simulated digital human effect is poor, failing to truly solve the problem of audio-video asynchrony.

[0025] Based on this, embodiments of the present invention provide an audio-video synchronization method. Figure 2 This is a flowchart illustrating the audio and video synchronization method according to an embodiment of the present invention. Figure 1 ;like Figure 2 As shown, the method includes: Step 101: Acquire multimedia data, extract audio data and video data corresponding to a first time length from the multimedia data, and cache the multiple frames of images contained in the video data; Step 102: Extract the mouth region from the multi-frame images to obtain multiple first images containing the mouth region; Step 103: Based on the audio data, drive the standard face model to obtain at least one second image containing the mouth features of the standard face model; Step 104: Compare the plurality of first images with the at least one second image, and perform synchronous processing on the audio data and the video data based on the comparison results.

[0026] In this embodiment, multimedia data is acquired, and audio and video data within a first time period are extracted from the multimedia data.

[0027] Multimedia data refers to data involving multiple media types such as sound and images. This embodiment of the invention does not impose specific restrictions on the multimedia data to be synchronized, such as live video, communication video, and video files downloaded to the local machine.

[0028] The first time length refers to the duration of the multimedia video to be synchronized. The first time length can be determined based on synchronization requirements or preset based on different scenarios.

[0029] For example, in a live streaming scenario, the duration T corresponding to the first time length can be set to 0.5 seconds; in a video-on-demand scenario, the duration T corresponding to the first time length can be set to 2 seconds; for a pre-acquired video file, the duration T corresponding to the first time length can be set to 2 groups. In other words, the first time length can be related to the latency requirements in different scenarios; that is, the higher the latency requirement, the smaller the corresponding first time length; the lower the latency requirement, the larger the corresponding first time length.

[0030] Audio data refers to digitized sound data extracted from multimedia data, i.e., an audio stream; video data refers to image data composed of continuous static images extracted from multimedia data, i.e., a video stream. This invention does not impose specific limitations on the storage method of audio data; for example, audio data can be stored using Pulse Code Modulation (PCM).

[0031] It should be noted that the methods used in the embodiments of the present invention to decompose audio and video data from multimedia video are not specifically limited, such as audio and video editing software, traditional encoding and decoding and streaming technologies (such as Moving Pictures Experts Group-Transport Stream (MPEG-TS) separation technology), time-frequency domain deep learning models, etc.

[0032] It should be noted that this embodiment of the invention does not limit whether the currently acquired multimedia data is multimedia data to be synchronized. Therefore, after acquiring the video and audio data corresponding to the multimedia data, the method further includes: a preliminary judgment on whether the acquired multimedia data requires audio-video synchronization processing, that is, a preliminary judgment on whether the acquired multimedia data has audio and video discrepancies. This embodiment of the invention does not impose specific limitations on the method for judging whether multimedia data has audio-video asynchrony. For example, it can determine whether the current multimedia data needs synchronization processing by judging whether there is a time delay between the audio data and the video data.

[0033] In this embodiment, after decomposing video data from multimedia data, multiple frames of images are decomposed from the video data, and the multiple frames of images are cached according to the time order in the video data.

[0034] It should be noted that the embodiments of the present invention do not impose specific limitations on the method of acquiring multi-frame images contained in video data. For example, professional video decomposition tools (such as FastForward MPEG) can be used to extract the images frame by frame, or computer vision segmentation algorithms (such as Segment Anything Model 2.1) can be used.

[0035] It should be noted that the embodiments of the present invention do not impose specific restrictions on the caching method of the decomposed multi-frame images. For example, a sliding buffer can be configured according to the first time length, and the decomposed multi-frame images can be stored in the sliding buffer to obtain the image buffer sequence On corresponding to the video data, which includes multi-frame images.

[0036] In this embodiment, after decomposing the corresponding multi-frame images from the video data, the mouth region of each frame in the multi-frame images is extracted to obtain multiple first images including the mouth region. Each first image corresponds to each frame in the stored multi-frame images.

[0037] The first image is an image of the mouth region of a person in each frame, and it only includes the mouth feature information of the person in the corresponding image. Multiple first images are stored in chronological order, corresponding to the cached multi-frame images.

[0038] It should be noted that the embodiments of the present invention do not impose specific limitations on the method of obtaining the first image. For example, traditional segmentation methods (such as threshold-based segmentation, edge-based segmentation, region-based segmentation, etc.) can be used, as well as deep learning methods (such as fully convolutional networks (FCN), U-shaped neural networks, etc.) to achieve image segmentation.

[0039] It should be noted that during or after obtaining multiple first images, feature extraction can be performed on the mouth information in the first images. This embodiment of the invention does not impose specific restrictions on the method of mouth feature extraction. For example, traditional local feature extraction methods (such as Scale Invariant Feature Transform (SIFT)) can be used, or machine learning algorithms such as Convolutional Neural Network (CNN) can be used.

[0040] In this embodiment, after extracting audio data from multimedia data, a pre-acquired standard face model is driven by the audio data to obtain at least one second image containing mouth features.

[0041] The standard face model is a pre-prepared model used to simulate the head state of a human face in video data. This model can simulate corresponding head movements based on different input head posture information, such as turning the head, turning the face to the side, tilting the head back, and tilting the head down. It can also simulate corresponding mouth movements based on input mouth feature information, such as opening the mouth, pursing the lips, and pursing the lips. Furthermore, it can simulate lip movements during human pronunciation based on audio data. This embodiment of the invention does not impose specific limitations on the method of obtaining the standard face model; for example, professional standard face generation tools (such as Blender modeling, Artbreeder, etc.) can be used.

[0042] In this embodiment, after obtaining multiple first images containing the mouth region extracted from multiple cached frames of images, and at least one second image containing mouth features simulated by driving a standard face model, the comparison results are obtained by comparing multiple first images and one or more second images. Based on the comparison results, it is determined whether the audio data and video data are synchronized and the synchronization processing of audio data and video data is achieved.

[0043] In this embodiment of the invention, multimedia data, including audio and video data, corresponding to a multimedia video is acquired. Multiple first images containing the mouth region are extracted from multiple frames of images corresponding to the cached video data. A standard face model is driven by the audio data to obtain at least one second image containing the mouth features of the standard face model. Synchronization processing of the audio and video data in the multimedia data is achieved based on the comparison result obtained by comparing the first and second images. By using the original audio data to drive the standard face model, the mouth (or lip shape) features of a human corresponding to the original audio data are accurately simulated. The second image corresponding to the standard face model and the first image obtained based on the actual video data are used to determine whether the audio and video are synchronized. Synchronization processing of the audio and video data improves the accuracy of audio-video synchronization judgment and avoids the use of digitally generated humans with poor facial and mouth effects to replace the human face in the actual video data. Without modifying the original audio and video data, this solves the problem of audio-video asynchrony caused by latency or uncorrected latency between the audio and video streams, or asynchronous timestamps between the audio and video streams.

[0044] Figure 3 This is a flowchart illustrating the audio and video synchronization method according to an embodiment of the present invention. Figure 2 ;like Figure 3 As shown, step 103 includes: Step 1031: Perform feature extraction on a frame of image to obtain head pose information in the image.

[0045] In this embodiment, head pose information in the image corresponding to a cached multi-frame image is obtained by extracting head features from one frame of the cached image.

[0046] It should be noted that the embodiments of the present invention do not impose specific restrictions on the selection method of images from cached multi-frame images. For example, when the multi-frame image cache is an image buffer sequence On, a frame image Pi can be selected for feature extraction according to the time order of each cached frame image.

[0047] In some optional embodiments, step 1031 includes: performing face detection on the frame image to obtain a face region in the image; performing head pose detection on the face region to obtain head pose information; the head pose information includes one or more of the head pitch angle, yaw angle and roll angle.

[0048] In this embodiment, the specific process of feature extraction for one frame of the multi-frame images corresponding to the cached video data is as follows: face detection is performed on the person in the image to obtain the face region in the image, and head pose detection is performed on the face region in the three-dimensional image to obtain the head pose information corresponding to the face region in the image.

[0049] Among them, the face region refers to the facial position of a person in the image, that is, the head region of the person. The head pose information corresponding to the face region is the three-dimensional spatial information of the person's head, specifically including one or more of the head's pitch angle, yaw angle, and roll angle.

[0050] Pitch angle refers to the rotation angle about the horizontal axis (x-axis) in the three-dimensional space of the physical world where the person in the image is located. The effect of pitch angle on the person's head is specifically manifested as the up and down rotation of the person's head. Yaw angle refers to the rotation angle about the vertical axis (y-axis) in the three-dimensional space of the physical world where the person in the image is located. The effect of yw angle on the person's head is specifically manifested as the left and right rotation of the person's head. Roll angle refers to the rotation angle about the vertical axis (z-axis) in the three-dimensional space of the physical world where the person in the image is located. The effect of roll angle on the person's head is manifested as the left and right tilt of the person's head.

[0051] It should be noted that the embodiments of the present invention do not impose specific limitations on the face detection method. For example, a ResNet algorithm based on a CNN architecture can be used to extract multi-level features, thereby detecting the face region in the image.

[0052] It should be noted that the embodiments of the present invention do not impose specific limitations on the head pose extraction method for three-dimensional images, such as the orthogonal projection iterative transformation algorithm (POSIT, Pose from Orthography and Scaling with Iterations), the n-point perspective pose estimation algorithm (PnP, Perspective-n-Point), etc.

[0053] Step 1032: Rotate the standard face model according to the head posture information to obtain the rotated standard face model.

[0054] In this embodiment, after obtaining the head pose information corresponding to the face region in a frame image, the standard face model is rotated based on one or more of the head pitch angle, yaw angle and roll angle included in the head pose information to obtain the rotated standard face model, which corresponds to the head pose of the face region in the frame image.

[0055] For example, when the head pose information includes the head's pitch angle, yaw angle, and roll angle, the standard face model is rotated up and down based on the pitch angle in the head pose information, rotated left and right based on the yaw angle in the head pose information, and tilted left and right based on the roll angle in the head pose information to obtain the rotated standard face model.

[0056] Step 1033: Drive the rotated standard face model based on the audio data to obtain at least one second image containing mouth features.

[0057] In this embodiment, after rotating the standard face model according to the head posture information to obtain the rotated standard face model, the rotated standard face model is driven by the audio data decomposed from the multimedia data to obtain at least one simulated second image containing mouth features. The simulated at least one second image corresponds to the first image containing actual mouth features corresponding to the frame image.

[0058] It should be noted that the embodiments of the present invention do not impose specific limitations on the method for driving the standard face model. For example, it can use speech-to-animation (Sta) or other speech-to-image generation methods, or text generation methods based on generative adversarial networks (GANs) such as the Stack GAN model or text generation methods based on diffusion models such as the Stable Diffusion model.

[0059] Understandably, in order to improve the accuracy of simulating human faces based on standard face models, during the process of driving the rotation of the standard face model based on audio data, one or more simulation results are obtained to obtain at least one second image corresponding to the frame image.

[0060] For example, for each frame of a multi-frame image, after driving the rotated standard face model based on the audio data corresponding to each frame of the image to obtain at least one corresponding second image, a lip-shape image sequence Sn including the second image corresponding to each frame of the image can be obtained, and the second images corresponding to each frame of the image in the lip-shape image sequence Sn are arranged in chronological order.

[0061] It should be noted that after obtaining at least one third image from the standard face model after rotation based on the audio data, image segmentation of the mouth region in the at least one third image is required to obtain one or more second images including mouth features.

[0062] In some optional embodiments, step 1033 includes: driving the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model; and / or, performing speech recognition processing on the audio data to obtain text information corresponding to the audio data, and driving the rotated standard face model based on the text information to obtain at least one second image containing the mouth features of the standard face model; wherein the first audio data corresponding to each second image corresponds to the same time parameter as one frame of the multi-frame images.

[0063] In this embodiment, there are two specific ways to obtain the second image based on the standard face model after rotation driven by audio data. One way is to drive the standard face model after rotation based on the first audio data corresponding to each second image, thereby obtaining at least one corresponding second image.

[0064] In this context, the first audio data corresponding to each second image corresponds to the same time parameter as one frame in the multi-frame image set. That is, the first audio data currently used to drive the rotated standard face model corresponds to the frame image selected for head pose information extraction. For example, the video data includes N frames, and the audio data includes first audio data corresponding to N sampling points. The first frame corresponds to the first audio data corresponding to the first sampling point, the second frame corresponds to the first audio data corresponding to the second sampling point, and so on.

[0065] Another approach is to perform speech recognition processing on the audio data corresponding to the multimedia data to obtain the text information corresponding to the audio data, and then drive the rotated standard face model based on the text information corresponding to the first audio data in the audio data to obtain at least one corresponding second image.

[0066] In this context, the text information corresponding to the first audio data has the same time parameter as one of the multiple frames of images. That is, the text information currently used to drive the rotated standard face model is the text information corresponding to the currently selected frame image for head pose extraction from the text information corresponding to the audio data. For example, the text information corresponding to the first frame image.

[0067] It should be noted that the embodiments of the present invention do not impose specific limitations on the Automatic Speech Recognition (ASR) method, such as Recurrent Neural Network Transducer (RNN-T), Deep Neural Network-Hidden Markov Model (DNN-HMM), etc., to achieve speech-to-text conversion processing.

[0068] It should be noted that, in this embodiment of the invention, by performing speech recognition on the audio data, the presence of human voices in the current audio data can be determined based on whether text information is recognized, thereby determining whether to drive the standard face model. Specifically, if text information is recognized, the rotated standard face model is driven; if no text information is recognized, the synchronization processing of the current video data and audio data is terminated. Based on this, in some optional embodiments, the frame image selected for head pose information extraction for rotating the standard face model can be the frame image corresponding to the first audio data containing human voices.

[0069] In some alternative implementations, step 104 includes: calculating the similarity of the mouth region between the nth second image and each of the plurality of first images; determining the mth first image synchronized with the nth second image based on the similarity; and performing synchronization processing on the audio data and the video data based on the nth second image and the mth first image.

[0070] In this embodiment, the specific process of obtaining the comparison result by comparing the first image and the second image is as follows: The similarity of the mouth region between the nth second image and each of the multiple first images is calculated; based on the similarity scores, the mth first image synchronized with the nth second image is determined. After determining the mth first image synchronized with the nth second image, the audio data and the video data are synchronized based on the nth second image and the mth first image.

[0071] For example, for each second image S in the lip-shape image sequence Sn, which includes the second image corresponding to each frame image, the similarity of the lip features to the first image P corresponding to each frame image Pi in the image buffer sequence On corresponding to multiple frames images is calculated. By calculating the similarity between the nth second image and all the first images, the mth first image synchronized with the nth second image is determined.

[0072] It should be noted that the embodiments of the present invention do not impose specific limitations on the calculation method of the mouth region similarity. For example, pixel-level comparison methods such as mean square error or structural similarity index can be used to quantify similarity, or image similarity calculation methods such as hash algorithm and feature vector analysis can be used.

[0073] In some optional implementations, determining the m-th first image synchronized with the n-th second image based on the similarity includes: determining a first similarity among multiple similarities that meets a preset condition, and using the first image corresponding to the first similarity as the m-th first image synchronized with the n-th second image; wherein the preset condition includes: the similarity is greater than or equal to a preset threshold, or the similarity is the maximum.

[0074] In this embodiment, after calculating the similarity between the nth second image and each first image to obtain multiple similarities, the first similarity that meets the preset conditions is selected from the multiple similarities, and the mth first image corresponding to the first similarity is used as the first image synchronized with the nth second image.

[0075] In this invention, considering that the number of images to be synchronized varies depending on the different audio and video asynchrony situations, the preset condition can be set as: similarity greater than or equal to a preset threshold, or maximum similarity.

[0076] Specifically, for scenarios where there are multiple images to be synchronized, under the preset condition that the similarity is greater than or equal to a preset threshold, in the process of calculating the similarity between each first image and the nth second image in chronological order, when the first similarity corresponding to the mth first image is greater than or equal to the preset threshold, the mth first image is used as the first image to be synchronized with the nth second image.

[0077] Given the preset condition of maximum similarity, among the multiple similarities obtained for the nth second image, the first similarity corresponding to the mth first image with the highest similarity is determined, and the mth first image is used as the first image to be synchronized with the nth second image.

[0078] For scenarios where there is only one image to be synchronized, under the preset condition that the similarity is greater than or equal to a preset threshold, in the process of calculating the similarity between each second image and each first image in chronological order, when it is determined that the first similarity between the nth second image and the mth first image is greater than or equal to the first threshold, the nth second image and the mth first image corresponding to the first similarity are used as the synchronized images for multimedia data.

[0079] Under the preset condition of maximum similarity, from the similarity set obtained including multiple similarities corresponding to each second image, the nth second image and the mth first image corresponding to the first similarity with the highest similarity in the similarity set are selected as the synchronization images of the multimedia data.

[0080] In some optional embodiments, before synchronizing the audio data and the video data according to the comparison result, the method further includes: determining whether the audio data and the video data are synchronized according to the comparison result.

[0081] In this embodiment, when determining the m-th first image synchronized with the n-th second image based on the similarity of the mouth region between the n-th second image and each of the plurality of first images, if This indicates that the audio data and the video data are synchronized; if This indicates that the audio data and the video data are out of sync.

[0082] In some optional implementations, the synchronization processing of the audio data and the video data based on the nth second image and the mth first image includes: performing time alignment processing on the audio data and the video data according to a first time parameter of the first audio data corresponding to the nth second image and a second time parameter corresponding to the first image corresponding to the mth first image, so that the first audio data and the first image correspond to the same time parameter; or, determining a time difference according to the first time parameter of the first audio data corresponding to the nth second image and the second time parameter corresponding to the first image corresponding to the mth first image, and adjusting the output strategy of the audio data and / or the video data based on the time difference to perform synchronization processing on the audio data and the video data.

[0083] In this embodiment, after determining the nth second image and the mth first image used for synchronization processing, the specific process of synchronizing audio and video data based on the nth second image and the mth first image is as follows: determining the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image; matching the first time parameter of the first audio data and the second time parameter of the first image to the same time parameter; and performing time alignment of the audio and video data based on the same time parameter, thereby achieving synchronized processing of audio and video data corresponding to multimedia data.

[0084] Alternatively, after determining the nth second image and the mth first image for synchronization processing, the time difference between the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image is calculated. Based on the time difference, an output strategy for the audio data and / or video data corresponding to the multimedia data is determined to perform synchronization processing on the audio data and video data.

[0085] The output strategy for audio data corresponding to multimedia data may include: based on the second time parameter of the first image corresponding to the m-th first image, delaying or shortening the first time parameter of the first audio data corresponding to the n-th second image, so as to adjust the first time parameter to the second time parameter.

[0086] For example, if the second time parameter is greater than the first time parameter, empty audio frames or repeated audio frames are added to delay the first time parameter corresponding to the first audio data; or, if the second time parameter is less than the first time parameter, some audio frames are discarded to shorten the first time parameter corresponding to the first audio data. After adjusting the first time parameter to the second time parameter, the video data and audio data corresponding to the multimedia data are synchronized by displaying the m-th first image at the second time parameter and simultaneously playing the first audio data corresponding to the n-th second image.

[0087] The output strategy for video data corresponding to multimedia data may include: based on the first time parameter of the first audio data corresponding to the nth second image, delaying or shortening the second time parameter of the first image corresponding to the mth first image, so as to adjust the second time parameter to the first time parameter.

[0088] For example, if the first time parameter is greater than the second time parameter, duplicate video frames or multiple frames are added to delay the second time parameter corresponding to the first image; or, if the first time parameter is less than the second time parameter, some video frames or multiple frames are discarded to shorten the second time parameter corresponding to the first image. After adjusting the second time parameter to the first time parameter, the first image m is displayed at the first time parameter, and the first audio data corresponding to the second image n is played simultaneously to achieve synchronous processing of video data and audio data corresponding to multimedia data.

[0089] It should be noted that in scenarios where there are multiple time parameters to be adjusted, the video and audio data corresponding to the multimedia data can be synchronously processed based on the adjusted time parameters after all the time parameters have been adjusted; or, the video and audio data corresponding to each time parameter can be synchronously processed while adjusting each time parameter in chronological order.

[0090] As an example, Figure 4 This is a flowchart illustrating an example of an audio-video synchronization method according to an embodiment of the present invention; as shown below. Figure 4 As shown, the flow of the audio-video synchronization method is as follows: Step 201: Obtain the audio and video data corresponding to the multimedia data.

[0091] Specifically, the multimedia data corresponding to the media video is obtained, and the audio and video data within the first time period are extracted from the multimedia data.

[0092] Step 202: Determine whether the multimedia data is data to be synchronized.

[0093] Specifically, the process determines whether the current multimedia data needs to be processed synchronously by judging whether there is a time delay between the audio data and the video data. If it is determined that the current multimedia data has audio and video asynchrony, step 203 is executed; if it is determined that the current multimedia data does not have audio and video asynchrony, the current process ends.

[0094] Step 203: Obtain multiple first images based on video data.

[0095] Specifically, multiple frames of images are decomposed from the video data and cached according to the time sequence in the video data. After decomposing the corresponding multiple frames of images from the video data, the mouth region of each frame of the multiple frames of images is extracted to obtain multiple first images including the mouth region. Each first image corresponds to each frame of the stored multiple frames of images.

[0096] Step 204: Obtain head pose information based on video data.

[0097] Specifically, by extracting head features from one frame of a cached multi-frame image, the head pose information of the corresponding image is obtained.

[0098] Step 205: Rotate the standard face model according to the head pose information to obtain the rotated standard face model.

[0099] Specifically, after obtaining the head pose information corresponding to the face region in a frame of an image, the standard face model is rotated based on one or more of the head pitch angle, yaw angle and roll angle included in the head pose information to obtain the rotated standard face model, which corresponds to the head pose of the face region in the frame of the image.

[0100] Step 206: Perform speech recognition on the audio data to obtain the recognition result.

[0101] Step 207: Determine whether the audio data contains human voices based on whether the recognition result includes text information.

[0102] Specifically, based on the speech recognition results of the audio data, if the speech recognition results include text information, step 208 is executed; if the speech recognition results do not contain text information, the current process ends.

[0103] Step 208: Based on the text information, drive the rotated standard face model to obtain at least one second image.

[0104] Specifically, after rotating the standard face model based on the head pose information to obtain the rotated standard face model, the rotated standard face model is driven by the text information recognized from the audio data to obtain at least one simulated second image containing mouth features. The simulated second image corresponds to the first image containing the actual mouth features of the frame image.

[0105] Step 209: Compare multiple first images with at least one second image.

[0106] Specifically, the similarity between the nth second image and the mouth region of each of the multiple first images is calculated. The first similarity that meets the preset conditions is selected from the multiple similarities, and the mth first image corresponding to the first similarity is used as the first image synchronized with the nth second image.

[0107] Step 210: Synchronize the audio and video data based on the comparison results.

[0108] Specifically, after determining the nth second image and the mth first image used for synchronization processing, the specific process of synchronizing audio and video data based on the nth second image and the mth first image is as follows: determine the first time parameter of the first audio data corresponding to the nth second image, and the second time parameter of the first image corresponding to the mth first image; match the first time parameter of the first audio data and the second time parameter of the first image to the same time parameter; and perform time alignment of the audio and video data based on the same time parameter, thereby achieving synchronized processing of audio and video data corresponding to multimedia data.

[0109] Step 211: Output and display the synchronized multimedia data.

[0110] Specifically, the m-th first image is displayed at the same time parameter determined based on the first time parameter and the second time parameter, and the first audio data corresponding to the n-th second image is played simultaneously to complete the synchronization of video data and audio data corresponding to multimedia data.

[0111] Based on the above embodiments, this invention also provides an audio-video synchronization device. Figure 5 This is a schematic diagram of the composition structure of the audio and video synchronization device provided in an embodiment of the present invention; as shown below. Figure 5 As shown, the device includes: a data decoding module 31, a standard face driving module 32, a comparison module 33, and a synchronization processing module 34, wherein... The data decoding module 31 is used to acquire multimedia data, decompose audio data and video data corresponding to a first time length from the multimedia data, cache multiple frames of images contained in the video data; and is also used to extract the mouth region from the multiple frames of images to obtain multiple first images containing the mouth region. The standard face driving module 32 is used to drive the standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model. The comparison module 33 is used to compare the plurality of first images and the at least one second image; The synchronization processing module 34 is used to perform synchronization processing on the audio data and the video data according to the comparison result.

[0112] In an optional embodiment of the present invention, the standard face driving module 32 is used to extract features from a frame of image to obtain head pose information in the image; rotate the standard face model according to the head pose information to obtain a rotated standard face model; and drive the rotated standard face model based on the audio data to obtain at least one second image containing mouth features.

[0113] In an optional embodiment of the present invention, the standard face driving module 32 shown is used to perform face detection on the frame image to obtain the face region in the image; and to perform head pose detection on the face region to obtain the head pose information; the head pose information includes one or more of the head pitch angle, yaw angle and roll angle.

[0114] In an optional embodiment of the present invention, the standard face driving module 32 is used to drive the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model; and / or, The device further includes an audio recognition module 35, used to perform speech recognition processing on the audio data to obtain text information corresponding to the audio data; the standard face driving module 32 is used to drive the rotated standard face model based on the text information to obtain at least one second image containing the mouth features of the standard face model; wherein, the first audio data corresponding to each second image corresponds to the same time parameter as one frame of the multi-frame images.

[0115] In an optional embodiment of the present invention, the comparison module 33 is used to calculate the similarity between the nth second image and the mouth region of each of the plurality of first images; and to determine the mth first image synchronized with the nth second image based on the similarity. The synchronization processing module 34 is used to perform synchronization processing on the audio data and the video data based on the nth second image and the mth first image.

[0116] In an optional embodiment of the present invention, the comparison module 33 is used to determine a first similarity among multiple similarities that meets a preset condition, and to use the first image corresponding to the first similarity as the m-th first image synchronized with the n-th second image; wherein, the preset condition includes: the similarity is greater than or equal to a preset threshold, or the similarity is the maximum.

[0117] In an optional embodiment of the present invention, the synchronization processing module 34 is configured to perform time alignment processing on the audio data and the video data according to the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image, so that the first audio data and the first image correspond to the same time parameter; or, determine a time difference according to the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image, and adjust the output strategy of the audio data and / or the video data based on the time difference to perform synchronization processing on the audio data and the video data.

[0118] In the implementation of this invention, the data decoding module 31, standard face driving module 32, comparison module 33, synchronization processing module 34, and audio recognition module 35 can all be implemented by the central processing unit (CPU), digital signal processor (DSP), microcontroller unit (MCU), or field-programmable gate array (FPGA) in the device in practical applications.

[0119] It should be noted that the above-described audio-video synchronization device is only illustrated by the division of the above-described program modules when performing audio-video synchronization processing. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the audio-video synchronization device and the audio-video synchronization method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0120] This invention also provides an electronic device. Figure 6 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of the present invention; as shown below. Figure 6 As shown, the electronic device includes a memory 42, a processor 41, and a computer program stored in the memory 42 and executable on the processor 41.

[0121] Optionally, when the processor 41 executes the program, it implements the steps of the audio-video synchronization method of the present invention.

[0122] Optionally, the various components in the electronic device can be coupled together via a bus system 43. It is understood that the bus system 43 is used to implement communication between these components. In addition to a data bus, the bus system 43 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general labeled all buses as Bus System 43.

[0123] It is understood that memory 42 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 42 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0124] The methods disclosed in the above embodiments of the present invention can be applied to processor 41, or implemented by processor 41. Processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 41 or by instructions in the form of software. The processor 41 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 41 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 42. Processor 41 reads the information in memory 42 and completes the steps of the aforementioned method in combination with its hardware.

[0125] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0126] This invention also provides a computer-readable storage medium having a computer program stored thereon.

[0127] Optionally, the computer-readable storage medium can be applied to the audio and video synchronization device of the present invention; then, when the program is executed by the processor, it implements the steps of the audio and video synchronization method of the present invention.

[0128] This invention also provides a computer program product, including a computer program that can be executed by a processor 41 of an electronic device to complete the steps of the audio-video synchronization method described in this invention.

[0129] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0130] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0131] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0133] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0135] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An audio-video synchronization method, characterized in that, The method includes: Acquire multimedia data, extract audio and video data corresponding to a first time length from the multimedia data, and cache multiple frames of images contained in the video data; Extract the mouth region from the multiple frames of images to obtain multiple first images containing the mouth region; Based on the audio data, a standard face model is driven to obtain at least one second image containing the mouth features of the standard face model; The audio data and the video data are synchronized based on the comparison results by comparing the multiple first images and the at least one second image.

2. The method according to claim 1, characterized in that, The process of obtaining at least one second image containing mouth features of the standard face model based on the audio data includes: Feature extraction is performed on a frame of image to obtain head pose information in the image; The standard face model is rotated based on the head posture information to obtain the rotated standard face model; Based on the audio data, the rotated standard face model is driven to obtain at least one second image containing mouth features.

3. The method according to claim 2, characterized in that, The step of extracting features from a frame of image to obtain head pose information in the image includes: Perform face detection on the image frame to obtain the face region in the image; The face region is subjected to head posture detection to obtain head posture information; the head posture information includes one or more of the head pitch angle, yaw angle and roll angle.

4. The method according to claim 2, characterized in that, The process of driving the rotated standard face model based on the audio data to obtain at least one second image containing the mouth features of the standard face model includes: Based on the audio data, drive the rotated standard face model to obtain at least one second image containing the mouth features of the standard face model; and / or, The audio data is processed by speech recognition to obtain the text information corresponding to the audio data. Based on the text information, the rotated standard face model is driven to obtain at least one second image containing the mouth features of the standard face model. In this context, the first audio data corresponding to each second image corresponds to the same time parameter as one of the multiple frames of images.

5. The method according to any one of claims 1 to 4, characterized in that, The comparison of the plurality of first images and the at least one second image, and the synchronization processing of the audio data and the video data based on the comparison result, includes: Calculate the similarity between the nth second image and the mouth region of each of the plurality of first images; Based on the similarity, a first image m is determined to be synchronized with the nth second image. The audio data and the video data are then synchronized based on the nth second image and the mth first image.

6. The method according to claim 5, characterized in that, The step of determining the m-th first image synchronized with the n-th second image based on the similarity includes: Determine the first similarity among multiple similarities that meets the preset conditions, and use the first image corresponding to the first similarity as the m-th first image synchronized with the n-th second image; The preset conditions include: similarity greater than or equal to a preset threshold, or maximum similarity.

7. The method according to claim 5, characterized in that, The synchronization processing of the audio data and the video data based on the nth second image and the mth first image includes: Based on the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image, time alignment processing is performed on the audio data and the video data so that the first audio data and the first image correspond to the same time parameter; or... The time difference is determined based on the first time parameter of the first audio data corresponding to the nth second image and the second time parameter of the first image corresponding to the mth first image. The output strategy of the audio data and / or the video data is adjusted based on the time difference to synchronize the audio data and the video data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images