Video data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a video data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises the steps of obtaining a target audio slice and a target image picture corresponding to a timestamp of the target audio slice based on video data; determining whether the target audio slice and the target image picture meet a sound picture synchronization condition and a sound picture identity consistency condition; wherein the sound and picture synchronization condition comprises that the synchronism of lip motion of the target audio slice and the target image in the target image picture is greater than or equal to a first threshold value; the sound picture identity consistency condition comprises that the matching degree of the target audio slice and a target image in the target image picture is greater than or equal to a second threshold value; and associating the target audio slice and the target image picture into a data pair under the condition of determining that the sound picture synchronization condition and the sound picture identity consistency condition are met. According to the method, high-quality data pairs can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Speech synthesis technology is a technique that converts input text into natural, human-understandable speech. However, existing speech customization solutions have shortcomings in ensuring the quality of data sources. Summary of the Invention

[0003] This disclosure provides a video data processing method, apparatus, device, storage medium, and program product.

[0004] In a first aspect, embodiments of this disclosure propose a video data processing method, comprising: acquiring a target audio slice and a target image frame corresponding to a timestamp of the target audio slice based on video data; determining whether the target audio slice and the target image frame satisfy an audio-visual synchronization condition and an audio-visual identity consistency condition; wherein, the audio-visual synchronization condition includes: the synchronization of the lip movements of the target audio slice and the target image frame in the target image frame is greater than or equal to a first threshold; the audio-visual identity consistency condition includes: the matching degree between the target audio slice and the target image frame in the target image frame is greater than or equal to a second threshold; and, if it is determined that the audio-visual synchronization condition and the audio-visual identity consistency condition are satisfied, associating the target audio slice and the target image frame as a data pair.

[0005] In one possible implementation, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition may include: determining whether the target audio slice and the target image meet the audio-visual synchronization condition; and if the audio-visual synchronization condition is met, determining whether the target audio slice and the target image meet the audio-visual identity consistency condition.

[0006] In one possible implementation, determining whether the target audio slice and the target image meet the audio-visual identity consistency condition may include: generating a timbre feature vector based on the target audio slice; generating a target image feature vector based on the target image; and determining whether the matching degree between the timbre feature vector and the target image feature vector is greater than or equal to a second threshold.

[0007] In one possible implementation, the method provided in this disclosure may further include: discarding the target audio slice and the target image image when it is determined that the target audio slice and the target image image do not meet the audio-visual synchronization conditions, and generating a first prompt message; wherein the first prompt message may include: a first reason information and / or a first suggestion information for indicating that the target audio slice and the target image image do not meet the audio-visual synchronization conditions.

[0008] In one possible implementation, the method provided in this disclosure may further include: generating a second prompt message when it is determined that the target audio slice and the target image do not meet the audio-visual identity consistency condition; wherein the second prompt message may include: a second reason information and / or a second suggestion information for indicating that the target audio slice and the target image do not meet the audio-visual identity consistency condition.

[0009] In one possible implementation, the above-mentioned acquisition of a target audio slice and a target image corresponding to a timestamp of the target audio slice based on video data may include: determining audio track data in the video data; obtaining a timestamp log based on the audio track data; wherein the timestamp log may include: a timestamp and one or more image identifiers associated with the timestamp, the timestamp being used to identify the start and end times of an audio segment in the audio track data; determining the target audio slice corresponding to each image identifier associated with the timestamp based on the timestamp log and the audio track data; and acquiring the target image corresponding to the timestamp of the target audio slice based on the target audio slice and the video data.

[0010] In one possible implementation, the above-mentioned determination of the target audio slices corresponding to each image identifier associated with a timestamp based on timestamp logs and audio track data may include: when there is only one image identifier associated with a timestamp, obtaining the target audio slice corresponding to the timestamp and image identifier based on the timestamp and audio track data; when there are multiple image identifiers associated with timestamps, obtaining the mixed audio slice corresponding to the timestamp based on the timestamp and audio track data, and obtaining the target audio slices corresponding to multiple image identifiers based on speech separation of the mixed audio slices.

[0011] In one possible implementation, before determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition, the method provided in this disclosure embodiment may further include: determining whether the target audio slice and the target image meet quality conditions; wherein, the quality conditions may include: the target image includes only one image, the target audio slice includes only the audio of a single image, and / or the noise of the target audio slice is less than or equal to a third threshold; wherein, the above determination of whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition may include: if it is determined that the target audio slice and the target image meet the quality conditions, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition.

[0012] In one possible implementation, the method provided in this disclosure may further include: discarding the target audio slice and the target image image when it is determined that the quality conditions are not met, and generating a third prompt message; wherein the third prompt message may include: a third reason message and / or a third suggestion message; wherein the third reason message is used to indicate the reason why the target audio slice and the target image image do not meet the quality conditions, and the third suggestion message is used to indicate the acquisition of new video data.

[0013] In one possible implementation, the method provided in this disclosure may further include: training a mapping model using data pairs, wherein the mapping model is used to generate a predicted timbre feature vector corresponding to the image feature vector based on the image feature vector, and the image feature vector is generated based on the image image.

[0014] In one possible implementation, the above-mentioned acquisition of a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data may include: determining whether the video data meets quality conditions; wherein, the quality conditions may include: the number of images included in the video data is one, the audio of the video data includes only the audio of a single image, and / or the noise of the video data is less than or equal to a third threshold; if it is determined that the video data does not meet the quality conditions, discarding the video data and generating a third prompt message; wherein, the third prompt message may include: third reason information indicating that the video data does not meet the quality conditions and third suggestion information indicating that new video data is provided; if it is determined that the video data meets the quality conditions, acquiring a target audio slice and a target image corresponding to the timestamp of the target audio slice based on the video data.

[0015] In one possible implementation, the method provided in this disclosure may further include: generating a timbre feature vector based on a target audio slice; and generating a speech stream based on the timbre feature vector and text information of the speech to be generated.

[0016] Secondly, embodiments of this disclosure propose a video data processing apparatus, including: a first acquisition module, a first determination module, and a first association module. The first acquisition module is configured to acquire a target audio slice and a target image image corresponding to a timestamp of the target audio slice based on video data. The first determination module is configured to determine whether the target audio slice and the target image image satisfy an audio-visual synchronization condition and an audio-visual identity consistency condition; wherein the audio-visual synchronization condition includes: the synchronization of lip movements of the target audio slice and the target image image in the target image image is greater than or equal to a first threshold; the audio-visual identity consistency condition includes: the matching degree between the target audio slice and the target image image in the target image image is greater than or equal to a second threshold. The first association module is configured to associate the target audio slice and the target image image as a data pair if the audio-visual synchronization condition and the audio-visual identity consistency condition are satisfied.

[0017] In one possible implementation, the first determining module is further configured to: determine whether the target audio slice and the target image meet the audio-visual synchronization condition; if the audio-visual synchronization condition is met, determine whether the target audio slice and the target image meet the audio-visual identity consistency condition.

[0018] In one possible implementation, the first determining module is further configured to: generate a timbre feature vector based on the target audio slice; generate a target image feature vector based on the target image; and determine whether the matching degree between the timbre feature vector and the target image feature vector is greater than or equal to a second threshold.

[0019] In one possible implementation, the video data processing apparatus may further include a first prompting module. The first prompting module is configured to discard the target audio slice and the target image slice, and generate first prompt information, when it is determined that the target audio slice and the target image slice do not meet the audio-visual synchronization conditions. The first prompt information may include: first reason information and / or first suggestion information indicating that the target audio slice and the target image slice do not meet the audio-visual synchronization conditions.

[0020] In one possible implementation, the video data processing apparatus may further include a second prompting module. The second prompting module is configured to generate second prompting information when it is determined that the target audio slice and the target image do not meet the audio-visual identity consistency condition. The second prompting information may include second reason information and / or second suggestion information indicating that the target audio slice and the target image do not meet the audio-visual identity consistency condition.

[0021] In one possible implementation, the first acquisition module is further configured to: determine audio track data in the video data; obtain a timestamp log based on the audio track data; wherein the timestamp log may include: a timestamp and one or more image identifiers associated with the timestamp, the timestamp being used to identify the start and end times of audio segments in the audio track data; determine the target audio slices corresponding to each image identifier associated with the timestamp based on the timestamp log and the audio track data; and obtain the target image image corresponding to the timestamp of the target audio slice based on the target audio slice and the video data.

[0022] In one possible implementation, the first acquisition module is further configured to: when there is only one image identifier associated with a timestamp, obtain a target audio slice corresponding to the timestamp and the image identifier based on the timestamp and audio track data; when there are multiple image identifiers associated with timestamps, obtain a mixed audio slice corresponding to the timestamp based on the timestamp and audio track data, and obtain target audio slices corresponding to multiple image identifiers based on speech separation of the mixed audio slices.

[0023] In one possible implementation, the video data processing apparatus may further include a second determining module. The second determining module is configured to determine whether the target audio slice and the target image frame meet quality conditions; wherein the quality conditions may include: the target image frame includes only one image, the target audio slice includes only the audio of a single image, and / or the noise of the target audio slice is less than or equal to a third threshold. The first determining module is further configured to, if it is determined that the target audio slice and the target image frame meet the quality conditions, determine whether the target audio slice and the target image frame meet audio-visual synchronization conditions and audio-visual identity consistency conditions.

[0024] In one possible implementation, the video data processing apparatus may further include a third prompt module. The third prompt module is configured to: discard the target audio slice and the target image when it is determined that the quality conditions are not met, and generate third prompt information. The third prompt information may include: third reason information and / or third suggestion information; wherein the third reason information indicates the reason why the target audio slice and the target image do not meet the quality conditions, and the third suggestion information indicates the acquisition of new video data.

[0025] In one possible implementation, the video data processing apparatus may further include a model training module. The model training module is configured to train a mapping model using data pairs, wherein the mapping model is used to generate predicted timbre feature vectors corresponding to image feature vectors, the image feature vectors being generated based on image images.

[0026] In one possible implementation, the first acquisition module is further configured to: determine whether the video data meets quality conditions; wherein the quality conditions may include: the number of images included in the video data is one, the audio of the video data includes only the audio of a single image, and / or the noise of the video data is less than or equal to a third threshold; if it is determined that the video data does not meet the quality conditions, discard the video data and generate a third prompt message; wherein the third prompt message may include: third reason information indicating that the video data does not meet the quality conditions and third suggestion information indicating that new video data is provided; if it is determined that the video data meets the quality conditions, based on the video data, acquire a target audio slice and a target image image corresponding to the timestamp of the target audio slice.

[0027] In one possible implementation, the video data processing apparatus may further include a first generation module and a second generation module. The first generation module is configured to generate a timbre feature vector based on a target audio slice. The second generation module is configured to generate a speech stream based on the timbre feature vector and text information of the speech to be generated.

[0028] According to a third aspect of this disclosure, a chip is provided that includes a processor, which can be used to implement the methods of the first aspect and any possible implementation thereof.

[0029] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions.

[0030] The processor is configured to implement the method of the first aspect and any possible implementation thereof when executing instructions stored in memory.

[0031] According to a fifth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the methods of the first aspect and any possible implementation thereof.

[0032] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect and any possible implementation thereof.

[0033] The video data processing method, apparatus, chip, electronic device, computer-readable storage medium, and computer program product provided in this disclosure can obtain high-quality data pairs and ensure the quality of the data source.

[0034] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0035] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart of a video data processing method provided in this embodiment of the disclosure; Figure 3 A flowchart of another video data processing method provided in this disclosure embodiment; Figure 4 A flowchart illustrating yet another video data processing method provided in this disclosure embodiment; Figure 5 A structural block diagram of a video data processing apparatus provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device suitable for performing a video data processing method, provided as an embodiment of the present disclosure. Detailed Implementation

[0036] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0037] It should be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0038] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this application are not limited to the order in which they are described, but can be performed in any order or in parallel. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the video data processing methods, apparatus, chips, electronic devices, computer-readable storage media, and computer program products of the present disclosure can be applied.

[0041] like Figure 1 As shown, system architecture 100 may include a first device 101 and a second device 102, which can be connected via a network. The network serves as the medium for providing a communication link between the first device 101 and the second device 102. The network may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0042] The first device 101 and the second device 102 can be either hardware or software. When the first device 101 and the second device 102 are hardware, the first device 101 can be various electronic devices, including but not limited to mobile phones, computers, tablets, smart robots, smart home terminals, smart terminals, in-vehicle devices, components on various devices, etc. The second device 102 can be a cloud server, a server, etc., where the server can also be a server for a distributed system or a server combined with blockchain. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability. When the first device 101 and the second device 102 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitations are made here.

[0043] It should be understood that Figure 1The number of first and second devices shown is merely illustrative. Any number of first and second devices can be used depending on implementation requirements.

[0044] The method provided in this disclosure is applicable to scenarios requiring customized voice timbres, such as 2D / 3D digital humans, virtual anchors, non-player characters (NPCs) in games, and animated characters. The specific technical implementation or deployment method of the method provided in this disclosure is not limited; for example, it can be deployed via a web platform, integrated with a software development kit (SDK), deployed privately, deployed as a plug-in edge, or deployed on a hardware platform.

[0045] The speech synthesis system supports both generative and replicative timbre techniques, catering to the customized timbre needs of different users. The technical solutions provided in this disclosure are applicable to both generative and replicative timbre techniques.

[0046] For example, the Generative Timbre technique primarily serves scenarios where users only provide static images or silent videos, such as creating a matching timbre "out of thin air" for virtual digital humans or game character designs. It relies on a pre-trained cross-modal mapping model to learn the mapping relationship from visual features (such as faces) to sound features (timbre).

[0047] For example, the replicative timbre technology mode primarily serves scenarios where users want to replicate their own voice. Users typically need to upload a short video of themselves appearing and speaking. The system analyzes the audio track in the video to extract the user's voiceprint features (speaker embedding), achieving timbre cloning with few or no samples.

[0048] Some customized voice solutions have deficiencies in ensuring the quality of data sources.

[0049] For example, regarding generative timbre technology: the low quality of training data for offline training mapping models leads to insufficient accuracy of the mapping models. For instance, they cannot identify common audio-visual mismatches and audio impurities in real-world videos, resulting in severely contaminated training data.

[0050] Sound-image mismatch: For example, dubbing in movies and TV dramas, narration in documentaries, and camera switching in multi-person interviews where the image is of the listener but the voice is of the speaker can all cause the model to learn a large number of incorrect associations of "face A-voice B".

[0051] Impure audio: For example, scenarios with multiple people speaking simultaneously (the cocktail party effect), background noise in noisy environments, and echo reverberation from indoor recordings can severely contaminate the extraction of voiceprint features. This data directly leads to insufficient accuracy in the trained mapping model, poor matching between the generated timbre and the face, and low fidelity.

[0052] For example, regarding the cloned voice technology model: the lack of verification in online services leads to poor system robustness. In cloned voice scenarios, the system requires users to upload their own videos. However, there is a lack of effective verification mechanisms for user input. When a user uploads a low-quality video, such as one recorded in a noisy environment, one where someone else is speaking, one where lip movements do not match the voice, or one where multiple people are speaking, the system cannot identify these anomalies and will continue to extract voiceprints based on these input videos. Ultimately, this results in poor-quality cloned voices, or even completely incorrect ones, greatly damaging the user experience and system reliability.

[0053] The method provided in this disclosure acquires a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data. By matching the audio slice and the image based on the timestamp, it can ensure that the time dimension of the associated objects is unbiased. The target audio slice and the target image that meet the conditions of audio-visual synchronization and audio-visual identity consistency are associated as data pairs. Data that is out of sync with audio and visuals is excluded by lip movement synchronization, and data that does not match the subject of audio and visuals is excluded by identity consistency matching. The double verification can further improve the accuracy of the data pairs.

[0054] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information, video data, and facial recognition-related information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] refer to Figure 2 , Figure 2 This is a flowchart of a video data processing method provided in an embodiment of the present disclosure. The executing entity is a first device 101 or a second device 102, wherein process 200 includes the following steps.

[0056] Step 201: Based on the video data, obtain the target audio slice and the target image corresponding to the timestamp of the target audio slice.

[0057] For example, the target image can include an image, which can include a face, a real person, a human figure, an animal figure, a virtual person, a character illustration, and / or a cartoon character, etc.

[0058] For example, video data could be a short video file uploaded or recorded by a user through a front-end application interface, featuring the user speaking. Optionally, this video file could be sent as an input data stream to the system backend. Alternatively, video data could be video files from object storage, a media library, or a scraped source, such as speeches or interview videos.

[0059] Step 202: Determine whether the target audio slice and target image meet the conditions of audio-visual synchronization and audio-visual identity consistency.

[0060] For example, the audio-visual synchronization condition may include: the synchronization between the target audio slice and the lip movements of the target image in the target image frame is greater than or equal to a first threshold. This allows determination of whether the image and sound in the video are precisely synchronized in time.

[0061] For example, the temporal fit between lip movements and the pronunciation syllables in the target audio slice is greater than or equal to the first threshold, such as 90%.

[0062] Optionally, the first threshold can be adjusted according to the shot size. For example, in close-up shots, where lip details are clear, the first threshold is larger, such as 95%. In medium to long shots, where the visual focus is scattered, the first threshold is smaller, such as 85%.

[0063] For example, the audio-visual identity consistency condition may include: the matching degree between the target audio slice and the target image in the target image picture is greater than or equal to a second threshold.

[0064] For example, the matching degree between the target audio slice and the target image in the target image image may include, but is not limited to, one or more of the following: identity matching, scene context matching, and emotional trait matching.

[0065] For example, if the voiceprint matching degree between the target audio slice and the target image in the target image is greater than or equal to the second threshold, such as 95%, then the target audio slice is compared with the benchmark voiceprint library of the target image to ensure that the audio speaker and the image in the image are the same subject.

[0066] For example, the matching degree between audio ambient sound and visual scene features is greater than or equal to the second threshold, such as 90%: background sounds in the audio (such as office keyboard sounds, outdoor wind sounds, and restaurant conversation sounds) must completely correspond to the visual scene (office, outdoors, restaurant) without any cross-scene ambient sound conflicts.

[0067] For example, the matching degree between the audio emotion and the visual emotion is greater than or equal to the second threshold, such as 80%: the emotional tone of the audio (happy, angry, sad, calm) must be consistent with the facial expressions and body movements of the target image in the image (e.g., if the audio is crying, the image in the image must show sadness such as tears or frowning).

[0068] For example, the first threshold and the second threshold can be preset.

[0069] Step 203: If the conditions of audio-visual synchronization and audio-visual identity consistency are met, associate the target audio slice and the target image with data pairs.

[0070] For example, in response to determining that the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition, the target audio slice and the target image are stored for subsequent speech synthesis.

[0071] The method provided in this disclosure acquires a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data. By matching the audio slice and the image based on the timestamp, it can ensure that the time dimension of the associated objects is unbiased. The target audio slice and the target image that meet the conditions of audio-visual synchronization and audio-visual identity consistency are associated as data pairs. Data that is out of sync with audio and visuals is excluded by lip movement synchronization, and data that does not match the subject of audio and visuals is excluded by identity consistency matching. The double verification can further improve the accuracy of the data pairs.

[0072] When constructing training data for the mapping model in the generative timbre technology mode, the technical solution provided in the embodiments of this disclosure can be used to automatically and multidimensionally clean and verify massive amounts of original videos, which can provide the mapping model with an extremely pure and high-quality dataset from the source, thereby significantly improving the accuracy of the mapping model.

[0073] In the sound customization service of the replica sound technology mode, the technical solution provided in the embodiments of this disclosure is used to perform real-time input quality inspection on the video uploaded by the user, intercept unqualified input, ensure the robustness of the service and the output quality, and optimize user interaction through intelligent feedback.

[0074] The technical solutions provided in this disclosure are applicable to both generative and replicative timbre technology modes. They can verify audio-visual quality. In online application scenarios, they can perform real-time quality checks on user-uploaded videos for timbre replication. In offline application scenarios, they can automatically construct a high-purity training dataset that can be used to train models from massive amounts of unlabeled raw videos. The technical solutions are reusable and have high data accuracy.

[0075] In some embodiments, step 202 above, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition, may include steps A1-A2.

[0076] Step A1: Determine whether the target audio slice and target visual image meet the audio-visual synchronization conditions.

[0077] Step A2: If the audio-visual synchronization condition is met, determine whether the target audio slice and the target image meet the condition of audio-visual identity consistency.

[0078] For example, audio-visual synchronization verification can be performed first, followed by audio-visual identity consistency verification. By filtering out out-of-sync data first, the amount of invalid data that does not need to enter identity verification can be reduced, avoiding the repeated consumption of high computing resources on invalid data and improving processing speed.

[0079] For example, a neural network model can be used to determine whether the target audio slice and the target visual image meet the audio-visual synchronization condition. For instance, this neural network model can be a self-supervised learning model, such as the discriminator model of SyncNet or Wav2Lip.

[0080] For example, by employing a self-supervised learning model, the model learns from the differences between "normal synchronization" and "artificial misalignment" samples on a large number of unlabeled videos, allowing the model to grasp the intrinsic physical relationship between sound and lip movements, thereby accurately identifying asynchronous phenomena such as dubbing, narration, and audio-visual delay.

[0081] The audio-visual synchronization verification stage can determine whether the target audio slice is precisely synchronized with the video frame at the corresponding timestamp in time, and can filter out dubbing, narration, and other situations.

[0082] For example, the target audio segment and its temporally corresponding video frame, particularly the face and lip areas, are input into a pre-trained audio-visual synchronization verification model. This model analyzes the correlation between the sound signal and lip movements to output a synchronization score. If the synchronization score is below a preset threshold, the {video, audio} data pair is considered out of sync, and the segment is deemed to be dubbed or narrated, thus discarding the segment. If the synchronization score is greater than or equal to the preset threshold, the {video, audio} data pair is saved.

[0083] For example, given that the audio-visual synchronization condition is met, a neural network model can be used to determine whether the target audio slice and the target visual image satisfy the audio-visual identity consistency condition. This neural network model can be pre-trained on a dataset with identity labels through metric learning (such as triplet loss) or binary classification.

[0084] The audio-visual identity consistency verification stage can further confirm the identity of data pairs that have passed the synchronization verification, ensuring that the person in the picture and the speaker are the same person, and filtering out misidentification in multi-person scenarios.

[0085] In some embodiments, step A2, determining whether the target audio slice and the target image meet the audio-visual identity consistency condition, may include steps A21-A23.

[0086] Step A21: Generate timbre feature vectors based on the target audio slices.

[0087] Step A22: Generate a target image feature vector based on the target image image.

[0088] Step A23: Determine whether the matching degree between the timbre feature vector and the target image feature vector is greater than or equal to the second threshold.

[0089] For example, for a {image, audio} data pair, two different encoders are invoked to extract its identity features. Taking a face as an example, a face recognition model is invoked to extract the target image feature vector, such as facial identity features, from the target image image. A speaker recognition model is invoked to extract the speaker identity features, also known as timbre feature vectors, from the target audio segment. The similarity between the facial identity features and the speaker identity features is compared using a pre-trained cross-modal identity verification model. If the similarity is lower than a preset threshold, the data pair is determined to be inconsistent and discarded. If the similarity is greater than or equal to the preset threshold, the {image, audio} data pair is saved. The face recognition model can also be called an image encoding model.

[0090] In other embodiments, step 202 above, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition, may include: determining whether the target audio slice and the target image meet the audio-visual identity consistency condition; and if the audio-visual identity consistency condition is met, determining whether the target audio slice and the target image meet the audio-visual synchronization condition.

[0091] For example, audio-visual identity verification can be performed first, followed by audio-visual synchronization verification. The specific implementation methods for audio-visual identity verification and audio-visual synchronization verification can refer to the corresponding implementation methods in the above-mentioned method of performing audio-visual synchronization verification first and then audio-visual identity verification, which will not be repeated here.

[0092] In some embodiments, the method provided in this disclosure may further include step B1.

[0093] Step B1: If it is determined that the target audio slice and the target image do not meet the audio-visual synchronization conditions, discard the target audio slice and the target image, and generate the first prompt message.

[0094] For example, the first prompt information may include: first reason information and / or first suggestion information for indicating that the target audio slice and the target image do not meet the audio-visual synchronization conditions.

[0095] For example, the first prompt information may include: a first cause information and / or a first suggestion information. The first cause information indicates why the target audio slice and the target visual image do not meet the audio-visual synchronization conditions. The first suggestion information includes suggestions for resolving the audio-visual desynchronization problem.

[0096] For example, if it is determined that the target audio slice and the target visual image do not meet the audio-visual synchronization conditions, the following message will be returned: "Lip movements do not match the sound. Please ensure that you speak normally in the video."

[0097] Thus, by establishing an intelligent feedback mechanism to guide users to provide high-quality input and improve processing efficiency, the interactive experience can be enhanced.

[0098] In some embodiments, the method provided in this disclosure may further include step C1.

[0099] Step C1: If it is determined that the target audio slice and the target image do not meet the condition of audio-visual identity consistency, generate a second prompt message.

[0100] For example, the second prompt information may include: a second reason information and / or a second suggestion information for indicating that the target audio slice and the target image do not meet the audio-visual identity consistency condition.

[0101] For example, the second prompt information may include: a second reason information and / or a second suggestion information. The second reason information is used to indicate the reason why the target audio slice and the target image do not meet the audio-visual identity consistency condition, and the second suggestion information includes suggestions for resolving the audio-visual identity inconsistency problem.

[0102] For example, if it is determined that the target audio clip and the target visual do not meet the requirement of audio-visual identity consistency, a prompt message is returned: "The person in the picture does not appear to be the same person as the speaker. Please ensure that the person in the picture is the actual person speaking." In this way, by establishing an intelligent feedback mechanism, guiding users to provide high-quality input, and improving processing efficiency, the interactive experience can be enhanced.

[0103] In some embodiments, step 201 above, which involves obtaining a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data, may include steps D1-D4.

[0104] Step D1: Determine the audio track data in the video data.

[0105] Step D2: Obtain timestamp logs based on audio track data.

[0106] Step D3: Based on the timestamp logs and audio track data, determine the target audio slices corresponding to each image identifier associated with the timestamp.

[0107] Step D4: Based on the target audio slice and video data, obtain the target image corresponding to the timestamp of the target audio slice.

[0108] Steps D1-D4 are applicable to both generative timbre technology mode scenarios and replica timbre technology mode scenarios.

[0109] For example, a timestamp log may include a timestamp and one or more visual identifiers associated with the timestamp, which can be used to identify the start and end times of audio segments in the audio track data.

[0110] For example, an audio segment associated with a timestamp can be called a single-person audio segment. Similarly, an audio segment associated with multiple timestamps can be called a multi-person audio segment or a mixed audio segment.

[0111] For example, based on video data and the target image corresponding to the timestamp of the target audio slice, the target image corresponding to the timestamp of the target audio slice is obtained.

[0112] Steps D1-D4 can process the original mixed audio track, which may contain multiple voices or noise, into a series of clean, single-speaker audio tracks that are tied to the speaker's identity.

[0113] For example, the raw audio track of the video data is input into a pre-trained SpeakerDiarization model, which analyzes the audio track and outputs a detailed timestamp log. The timestamp log includes "who spoke and when", that is, each detected speech segment is labeled with a temporary speaker ID (such as SPEAKER_00, SPEAKER_01) and the start and end times of the audio segment.

[0114] The speaker log model in this disclosure is not limited to a specific implementation and can employ any advanced deep learning-based model, such as the pre-trained model provided in the pyannote.audio open-source toolkit.

[0115] In some embodiments, step D3 above, which determines the target audio slice corresponding to each image identifier associated with the timestamp based on timestamp logs and audio track data, may include steps D31-D32.

[0116] Step D31: If there is only one image identifier associated with the timestamp, obtain the target audio slice corresponding to the timestamp and image identifier based on the timestamp and audio track data.

[0117] Step D32: When there are multiple image identifiers associated with timestamps, obtain the mixed audio slices corresponding to the timestamps based on the timestamps and audio track data, and perform speech separation on the mixed audio slices to obtain the target audio slices corresponding to the multiple image identifiers respectively.

[0118] For example, based on timestamp logs and audio track data, for audio segments explicitly marked as single-person speech in the timestamp logs, they can be directly segmented according to the timestamp to obtain the target audio segment corresponding to a single image identifier associated with the timestamp. For segments with detected speech overlap, such as overlapping audio of multiple people speaking simultaneously, a pre-trained speech separation model can be invoked to separate the audio signals of each person, resulting in multiple independent single-person audio tracks, i.e., target audio slices corresponding to multiple image identifiers associated with the timestamps.

[0119] Optionally, for all separated or segmented solo audio tracks, a speech enhancement model can be further invoked to perform noise reduction and dérarization.

[0120] The speech separation model and speech enhancement model in this embodiment are not limited to a specific implementation and can adopt temporal audio separation networks such as Conv-TasNet.

[0121] In some embodiments, before determining whether the target audio slice and target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition in step 202 above, the method provided in this disclosure embodiment may further include step E1.

[0122] Step E1: Determine whether the target audio slice and target image meet the quality requirements.

[0123] For example, quality conditions may include at least one of the following: the target image frame includes only one image, the target audio slice includes only the audio of a single image, and the noise of the target audio slice is less than or equal to a third threshold.

[0124] For example, the number of images included in the target image is one, which means that there is only one subject in the target image, such as only one person in the picture, without other irrelevant people interfering, which makes it easy to accurately locate lip movements.

[0125] For example, a target audio slice that includes only the audio of a single image can mean that the target audio slice contains only the voice of this one subject, such as only the voice of this person speaking, without the voices of other people superimposed.

[0126] For example, the noise of the target audio slice being less than or equal to the third threshold means that the noise of the target audio slice (such as background noise, current noise, etc.) is small enough not to exceed the set third threshold, such as the noise intensity being less than 10% of the main audio sound, to ensure that the audio is clear and distinguishable.

[0127] Optionally, step 202 above, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition, may include: if it is determined that the target audio slice and the target image meet the quality conditions, determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition.

[0128] In this way, by setting quality standards, it can be ensured that the materials used for audio-visual verification are clear and free of interference, and the accuracy of verification can be avoided by multiple subjects or poor audio quality.

[0129] In some embodiments, the method provided in this disclosure may further include: discarding the target audio slice and the target image when it is determined that the quality conditions are not met, and generating a third prompt message.

[0130] For example, the third prompt information may include: third reason information and / or third suggestion information. The third reason information can be used to indicate why the target audio slice and the target image do not meet the quality conditions, and the third suggestion information is used to indicate the acquisition of new video data.

[0131] For example, materials that do not meet the quality requirements will cause distortion in the audio-visual synchronization verification results and the identity consistency verification results. Discarding target audio slices and target image slices that do not meet the quality requirements can ensure that the final associated data pairs are accurate and reliable, and avoid inferior data interfering with the quality of the audio stream.

[0132] In some embodiments, the method provided in this disclosure may further include: training a mapping model using data pairs.

[0133] For example, a mapping model can be used to generate a predicted timbre feature vector corresponding to an image feature vector, where the image feature vector is generated based on an image image. For instance, a mapping model can generate a predicted timbre feature vector corresponding to a target image feature vector. For example, given a person's facial feature vector as input, the mapping model can mimic the output timbre vector of that person, thus binding the image and timbre together.

[0134] The data pairs obtained in this embodiment can be used to construct a high-purity training set for the mapping model. This training dataset includes data pairs used to train the mapping model. It can provide extremely pure training data for the mapping model, improving model accuracy from the source.

[0135] Based on the constructed dataset, a mapping model is trained. This mapping model learns the mapping relationship between visual features (face_embedding) extracted from face images and voiceprint features (voice_embedding) extracted from audio clips.

[0136] In the application phase, a user uploads a facial image, which is then used by a mapping model to generate a voiceprint feature that can be used in a TTS system. This voiceprint feature can also be called a timbre feature vector. The TTS system generates a speech stream based on the timbre feature vector; for example, it generates a speech stream based on the timbre feature vector and the text information of the speech to be generated.

[0137] The method provided in this disclosure can be applied to the generative timbre technology model. For example, it takes a large amount of unlabeled raw video as input. The method provided in this disclosure is automatically executed on the raw video, outputting high-quality data pairs. All verified {image, audio slice} data pairs are collected to form the final training dataset. Using this high-quality dataset, a mapping model is trained. Since the training dataset at least satisfies the audio-visual synchronization condition and the audio-visual identity consistency condition, the trained mapping model can achieve a significant improvement in the matching degree and fidelity of timbre generation.

[0138] In some embodiments, step 201 above, which involves obtaining a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data, may include steps F1-F3.

[0139] Step F1: Determine if the video data meets the quality requirements.

[0140] Step F2: If it is determined that the video data does not meet the quality requirements, discard the video data and generate a third prompt message.

[0141] Step F3: If the video data meets the quality requirements, based on the video data, obtain the target audio slice and the target image corresponding to the timestamp of the target audio slice.

[0142] For example, quality conditions may include at least one of the following: the number of images in the video data is one, the audio in the video data includes only the audio of a single image, and the noise in the video data is less than or equal to a third threshold.

[0143] For example, the third prompt information may include: a third reason information indicating that the video data does not meet the quality conditions and a third suggestion information indicating that new video data should be provided.

[0144] Steps F1-F3 are applicable to scenarios using the replica timbre technology mode.

[0145] In some embodiments, the method provided by the embodiments of the present disclosure may further include: step G1 - step G2.

[0146] Step G1: Generate a timbre feature vector based on the target audio slice.

[0147] Step G2: Generate a speech stream based on the timbre feature vector and the text information of the speech to be generated.

[0148] Exemplarily, a timbre feature vector is obtained based on the target audio slice in the data pair, and a speech stream is generated based on the timbre feature vector and the text information of the speech to be generated.

[0149] Exemplarily, the text information of the speech to be generated may include the text content of the speech to be generated, such as "The weather is nice today", or may also be a preprocessed phoneme sequence or syllable sequence, such as "j n ti n ti n qì bù cuò" corresponding phonemes.

[0150] Optionally, the text information may be preprocessed. Exemplarily, preprocessing the text information may include: normalizing the input text information, for example, including the expansion of numbers, dates, abbreviations, such as expanding "2025" to "two thousand and twenty-five", eliminating text format ambiguities, and may also include: performing grapheme-to-phoneme conversion (G2P) or pinyinization according to the target language, and finally outputting a phoneme sequence or syllable sequence recognizable by the speech synthesis model, without the need to develop an additional front-end module.

[0151] Optionally, the text information may further include style parameters, and the style parameters may include: speech rate adjustment, emotion setting, fundamental frequency / energy control, or prompt word guidance (if supported by the speech synthesis model).

[0152] For example, the text information may be compatible with multi-language input to meet the speech synthesis requirements of different scenarios.

[0153] Exemplarily, by combining the predicted timbre feature vector and the text information, a speech stream of a specified timbre audible to the human ear is output through the operation of the speech synthesis model, completing voice generation by "looking at the face".

[0154] Optionally, the speech synthesis model may also be referred to as a text-to-speech (TTS) model.

[0155] When the method provided by the embodiments of the present disclosure is applied to an online scenario (replicated timbre technology mode), real-time input quality inspection can be provided to ensure the robustness and output quality of the service.

[0156] Reference Figure 3 , Figure 3 The flowchart of another video data processing method provided in this embodiment of the disclosure is shown. The execution subject is a first device 101 or a second device 102, wherein process 300 includes the following steps. Figure 3 This is a flowchart illustrating the application of the method provided in this embodiment to an online scenario.

[0157] Step 301: Obtain video data.

[0158] For example, user video uploads, where users upload or record a video of themselves appearing on camera and speaking, for use in voice cloning.

[0159] Step 302: Determine whether the video data meets the quality requirements.

[0160] Step 303: If it is determined that the video data does not meet the quality requirements, generate a third prompt message.

[0161] For example, if it is determined that the video data does not meet the quality conditions, the video data is discarded, a third prompt message is generated, and the third prompt message is output to the user.

[0162] Step 304: If the video data meets the quality requirements, determine whether the audio-visual synchronization requirements are met.

[0163] Step 305: If it is determined that the audio-visual synchronization conditions are not met, generate the first prompt message.

[0164] For example, in response to determining that the audio-visual synchronization condition is not met, video data is discarded, a first prompt message is generated, and the first prompt message is output to the user.

[0165] For example, if multiple voices are detected, the message will be: "Multiple voices detected. Please ensure that only you are speaking and try again." If loud noise is detected, the message will be: "The environment is somewhat noisy. Please re-record in a quieter place."

[0166] Step 306: If the audio-visual synchronization condition is met, determine whether the audio-visual identity consistency condition is met.

[0167] Step 307: If it is determined that the audio-visual identity does not meet the condition, generate a second prompt message.

[0168] For example, in response to determining that the audio-visual identity consistency condition is not met, the video data is discarded, a second prompt message is generated, and the second prompt message is output to the user.

[0169] For example, if the video data fails to meet any of the quality conditions, audio-visual synchronization conditions, or audio-visual identity consistency conditions, the detection fails. Based on the specific stage and reason for the failure, specific and actionable repair suggestions are returned to the user, and the process terminates. If the quality inspection is successful, it means the user-uploaded video is of acceptable quality, and the process proceeds to the next step. Optionally, for any of the quality conditions, audio-visual synchronization conditions, or audio-visual identity consistency conditions, the detection result can be output, for example, a JSON object containing multiple boolean flags and scores.

[0170] Step 308: If the condition of audio-visual identity consistency is met, obtain the timbre feature vector based on the target audio slice.

[0171] For example, a speaker coding model can be used to obtain timbre feature vectors based on target audio slices.

[0172] Step 309: Generate a speech stream based on the timbre feature vector.

[0173] For example, a speech stream is generated based on timbre feature vectors and text information of the speech to be generated.

[0174] For example, for videos that pass quality inspection, pure timbre feature vectors are extracted and the TTS system is invoked to complete a high-quality timbre remix for the user.

[0175] Step 310: Return the audio stream.

[0176] For example, the generated audio stream can be provided to the user, and a notification message indicating that the audio stream was successfully generated can also be sent to the user.

[0177] In this way, multi-stage audio-visual verification is applied to online services, and a mapping mechanism from verification results to user feedback is established. This transforms complex backend technical judgments into simple instructions that users can understand and execute, thereby improving user experience and interactive intelligence.

[0178] Furthermore, employing a cascaded fail-fast strategy can improve the response speed of online verification. That is, the three stages of verifying quality conditions, audio-visual synchronization conditions, and audio-visual identity consistency conditions are executed sequentially. If any stage fails, the subsequent stages will not be executed, and the system will immediately return the corresponding failure feedback to the user. For example, if significant ambient noise is detected during the quality condition detection stage, a "noisy environment" message will be returned directly, eliminating the need for subsequent synchronization and identity consistency verifications, thus saving computational resources and accelerating response time.

[0179] refer to Figure 4 , Figure 4The flowchart of another video data processing method provided in this disclosure embodiment is shown. The execution subject is a first device 101 or a second device 102, wherein process 400 includes the following steps. Figure 4 This is a flowchart illustrating the application of the method provided in this embodiment to an offline scenario.

[0180] Step 401: Obtain video data.

[0181] For example, video data could be a video of an interview program containing conversations between multiple people.

[0182] Step 402: Clean up the audio track data of the video data.

[0183] For example, the audio track data in the video data is determined. Using a speaker log model, the audio track data is processed to obtain a timestamp log, outputting a structured timestamp log, such as [('SPEAKER_00', 0.5, 15.2), ('SPEAKER_01', 15.8, 35.5), ('SPEAKER_00', 36.1, 58.9)], recording the start and end times of each speaker's speech, such as 0.5 and 15.2. Next, based on this timeline data, overlapping parts of the audio are processed using a speech separation model (e.g., Conv-TasNet). All speech segments can also be denoised and dedevered using a speech enhancement model. The output is a series of clean single-speaker audio slices bound to speaker IDs (e.g., SPEAKER_00, SPEAKER_01).

[0184] For example, a speaker log model can include a pre-trained model based on the pyannote.audio open-source library.

[0185] Step 403: Perform audio-visual synchronization verification.

[0186] For example, the clean audio of the image identifier SPEAKER_00 between 0.5 and 15.2 seconds is compared with the video footage (especially the face and lip area in the footage) between 0.5 and 15.2 seconds and analyzed as a data pair.

[0187] For example, a pre-trained audio-visual synchronization verification model is used to calculate the synchronization score of the data pair. If the synchronization score is lower than a preset threshold, the segment is considered to be dubbed or narrated and is discarded. If the synchronization score is greater than or equal to the preset threshold, audio-visual identity consistency verification is performed.

[0188] Thus, by applying audio-visual synchronization verification to the data construction process of timbre generation, and by analyzing the physical relationship between sound and lip movement, the data can be filtered for first-level hard mismatches, thereby improving the accuracy of the generated timbre.

[0189] Step 404: Perform audio-visual identity verification.

[0190] For example, the clearest image is detected and extracted from the video segment corresponding to the audio slice, and an image feature vector is generated based on the image using an image encoding model. A voiceprint recognition model (e.g., ECAPA-TDNN) is used to extract timbre feature vectors based on the audio slice; this voiceprint recognition model can also be called a speaker encoding model.

[0191] A pre-trained cross-modal identity verification model projects timbre feature vectors and image feature vectors onto a unified feature space and calculates their similarity scores. If the similarity score is below a preset threshold, for example, 0.9, the audio and image identities are considered inconsistent; for example, the person in the image and the speaker are not the same person, and the data pair is discarded. If the similarity score is greater than or equal to the preset threshold, the data pair, including the audio slice and the image, is stored in the training database.

[0192] Step 405: Obtain the dataset used to train the mapping model.

[0193] For example, the mapping model can be used to generate a predicted timbre feature vector corresponding to the image feature vector, which is generated based on the image image.

[0194] For example, the validated data pairs are stored in a training database, and the dataset in the training database can be directly used for any model training task that requires high-quality audio-visual matching data, such as a mapping model.

[0195] For example, after obtaining the dataset, the data in the dataset can be used to train the mapping model.

[0196] In the application phase, a user uploads a facial image, and the mapping model can be used to obtain a timbre feature vector. Then, a speech stream is generated based on the timbre feature vector through TTS.

[0197] The method for obtaining data pairs provided in this disclosure can be flexibly applied to two different business scenarios: offline data construction and online real-time verification, ensuring high-quality output from the system in both modes.

[0198] The technical solution provided in this disclosure can significantly improve the accuracy and fidelity of the mapping model in generative timbre technology. It can automatically filter and process high-quality training data with perfect audio-visual matching from massive, noisy, and complex raw videos. By eliminating sample contamination caused by "audio-visual mismatch" and "audio impurity" at the source, the downstream mapping model can learn more accurate and essential cross-modal associations, thereby significantly improving the timbre it generates in terms of matching degree with human faces, purity of sound quality, and personalization of timbre.

[0199] The technical solution provided in this disclosure greatly enhances the robustness and user experience of online services using the replicated sound technology mode, solving the problem of uncontrollable online input quality. In online sound replication scenarios, real-time quality checks are performed on user-uploaded videos, and an intelligent feedback mechanism is established. This effectively blocks unqualified user input, such as multiple voices, loud noise, and non-user speech, ensuring the stability and high-quality output of the sound cloning service. Furthermore, when quality checks fail, the system provides users with specific and actionable repair suggestions, transforming a failed interaction into effective user guidance, which greatly improves the product's professionalism, usability, and user trust.

[0200] The technical solution provided in this disclosure enables system-level technology reuse, reduces R&D and maintenance costs, and proposes a novel system architecture. Its core is a unified audio-visual quality verification pipeline that can be invoked by both offline and online processes. This "one pipeline, two applications" architecture solves two originally independent problems—offline data processing and online service verification—through a common basic technology module. This avoids redundant development for different scenarios, simplifies system complexity, reduces overall R&D and maintenance costs, and improves the efficiency of technology iteration.

[0201] The technical solution provided in this disclosure can improve the utilization rate of raw data and expand the scale and diversity of the training set. In the audio purification stage, a speaker log and speech separation scheme is adopted. It can transform complex videos (such as interviews, debates, and multi-person dialogue scenes) that are directly discarded due to multiple voices or faces into multiple effective, high-quality single-person audio-visual training samples. This ability to turn waste into treasure greatly expands the source, scale, and diversity of effective training data, providing a solid data foundation for training a mapping model with greater generalization ability.

[0202] The technical solution provided in this disclosure applies a unified audio-visual quality verification process to two different stages of a dual-mode (generative / replicative) speech synthesis system: the offline training data construction stage and the online input quality inspection stage. Offline, the audio-visual verification process is used to clean massive amounts of video data to train a generative timbre model, also known as a mapping model. Online, the same audio-visual verification process is used to perform real-time quality inspection on user-uploaded videos, ensuring the input quality of the replicative timbre service.

[0203] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video data processing device, which corresponds to the above method embodiment and can be specifically applied to various electronic devices.

[0204] like Figure 5 As shown, the video data processing apparatus 500 of this embodiment includes: a first acquisition module 501, a first determination module 502, and a first association module 503. The first acquisition module 501 is configured to acquire a target audio slice and a target image image corresponding to the timestamp of the target audio slice based on video data. The first determination module 502 is configured to determine whether the target audio slice and the target image image satisfy the audio-visual synchronization condition and the audio-visual identity consistency condition; wherein the audio-visual synchronization condition includes: the synchronization of the lip movements of the target audio slice and the target image image in the target image image is greater than or equal to a first threshold; the audio-visual identity consistency condition includes: the matching degree between the target audio slice and the target image image in the target image image is greater than or equal to a second threshold. The first association module 503 is configured to associate the target audio slice and the target image image as a data pair if the audio-visual synchronization condition and the audio-visual identity consistency condition are satisfied.

[0205] In this embodiment, the specific processing of the first acquisition module 501, the first determination module 502, and the first association module 503 in the video data processing device 500 and the resulting technical effects can be referred to the relevant descriptions of the above method embodiments, and will not be repeated here.

[0206] In one possible implementation, the first determining module 502 is further configured to: determine whether the target audio slice and the target image meet the audio-visual synchronization condition; if the audio-visual synchronization condition is met, determine whether the target audio slice and the target image meet the audio-visual identity consistency condition.

[0207] In one possible implementation, the first determining module 502 is further configured to: generate a timbre feature vector based on the target audio slice; generate a target image feature vector based on the target image; and determine whether the matching degree between the timbre feature vector and the target image feature vector is greater than or equal to a second threshold.

[0208] In one possible implementation, the video data processing apparatus 500 may further include a first prompting module. The first prompting module is configured to discard the target audio slice and the target image slice, and generate first prompt information, when it is determined that the target audio slice and the target image slice do not meet the audio-visual synchronization conditions. The first prompt information may include: first reason information and / or first suggestion information indicating that the target audio slice and the target image slice do not meet the audio-visual synchronization conditions.

[0209] In one possible implementation, the video data processing apparatus 500 may further include a second prompting module. The second prompting module is configured to generate second prompting information when it is determined that the target audio slice and the target image do not meet the audio-visual identity consistency condition. The second prompting information may include second reason information and / or second suggestion information indicating that the target audio slice and the target image do not meet the audio-visual identity consistency condition.

[0210] In one possible implementation, the first acquisition module 501 is further configured to: determine audio track data in the video data; obtain a timestamp log based on the audio track data; wherein the timestamp log may include: a timestamp and one or more image identifiers associated with the timestamp, the timestamp being used to identify the start and end times of audio segments in the audio track data; determine the target audio slices corresponding to each image identifier associated with the timestamp based on the timestamp log and the audio track data; and obtain the target image image corresponding to the timestamp of the target audio slice based on the target audio slice and the video data.

[0211] In one possible implementation, the first acquisition module 501 is further configured to: when the number of image identifiers associated with a timestamp is one, obtain a target audio slice corresponding to the timestamp and the image identifier based on the timestamp and audio track data; when the number of image identifiers associated with a timestamp is multiple, obtain a mixed audio slice corresponding to the timestamp based on the timestamp and audio track data, and obtain target audio slices corresponding to multiple image identifiers based on speech separation of the mixed audio slices.

[0212] In one possible implementation, the video data processing apparatus 500 may further include a second determining module. The second determining module is configured to determine whether the target audio slice and the target image frame meet quality conditions; wherein the quality conditions may include: the target image frame includes only one image, the target audio slice includes only the audio of a single image, and / or the noise of the target audio slice is less than or equal to a third threshold. The first determining module 502 is further configured to, if it is determined that the target audio slice and the target image frame meet the quality conditions, determine whether the target audio slice and the target image frame meet audio-visual synchronization conditions and audio-visual identity consistency conditions.

[0213] In one possible implementation, the video data processing apparatus 500 may further include a third prompt module. The third prompt module is configured to: discard the target audio slice and the target image when it is determined that the quality conditions are not met, and generate third prompt information. The third prompt information may include: third reason information and / or third suggestion information; wherein the third reason information indicates the reason why the target audio slice and the target image do not meet the quality conditions, and the third suggestion information indicates the acquisition of new video data.

[0214] In one possible implementation, the video data processing apparatus 500 may further include a model training module. The model training module is configured to train a mapping model using data pairs, wherein the mapping model is used to generate predicted timbre feature vectors corresponding to image feature vectors, the image feature vectors being generated based on image images.

[0215] In one possible implementation, the first acquisition module 501 is further configured to: determine whether the video data meets quality conditions; wherein the quality conditions may include: the number of images included in the video data is one, the audio of the video data includes only the audio of a single image, and / or the noise of the video data is less than or equal to a third threshold; if it is determined that the video data does not meet the quality conditions, discard the video data and generate a third prompt message; wherein the third prompt message may include: a third reason message indicating that the video data does not meet the quality conditions and a third suggestion message indicating that new video data should be provided; if it is determined that the video data meets the quality conditions, acquire a target audio slice and a target image image corresponding to the timestamp of the target audio slice based on the video data.

[0216] In one possible implementation, the video data processing apparatus 500 may further include: a first generation module and a second generation module. The first generation module is configured to generate a timbre feature vector based on a target audio slice. The second generation module is configured to generate a speech stream based on the timbre feature vector and text information of the speech to be generated.

[0217] This embodiment exists as a device embodiment corresponding to the above method embodiment. The technical effects brought about by the video data processing device provided in this embodiment can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0218] This disclosure also provides a chip including a processor that can be used to implement any of the possible implementations of the methods provided in this disclosure.

[0219] This disclosure also proposes an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing instructions stored in the memory.

[0220] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the described method. The computer-readable storage medium can be volatile or non-volatile.

[0221] This disclosure also provides a computer program product that, when executed by a processor, can implement the methods described in any of the above embodiments.

[0222] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0223] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 600 of this embodiment includes a processor 601 and may further include a memory 602; wherein, the memory 602 is used to store computer execution instructions; the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0224] Alternatively, the memory 602 can be either standalone or integrated with the processor 601.

[0225] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0226] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0227] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0228] Bus 603 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0229] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0230] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0231] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0232] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0233] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0234] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0235] According to the technical solution of this disclosure, a target audio slice and a target image corresponding to the timestamp of the target audio slice are obtained based on video data. The audio slice and the image are matched based on the timestamp, which can ensure that the time dimension of the associated object is without deviation. The target audio slice and the target image that meet the conditions of audio-visual synchronization and audio-visual identity consistency are associated as data pairs. Data that is not synchronized with the audio-visual is excluded by lip movement synchronization, and data that does not match the subject of the audio-visual is excluded by identity consistency matching degree. The double verification can further improve the accuracy of the data pairs.

[0236] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0237] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video data processing method, comprising: Based on video data, obtain the target audio slice and the target image image corresponding to the timestamp of the target audio slice; Determine whether the target audio slice and the target image meet the conditions of audio-visual synchronization and audio-visual identity consistency; wherein, the audio-visual synchronization condition includes: the synchronization of the lip movements of the target audio slice and the target image in the target image is greater than or equal to a first threshold; the audio-visual identity consistency condition includes: the matching degree between the target audio slice and the target image in the target image is greater than or equal to a second threshold. Once it is determined that the audio-visual synchronization condition and the audio-visual identity consistency condition are met, the target audio slice and the target image are associated as a data pair.

2. The method according to claim 1, wherein, The determination of whether the target audio slice and the target image meet the conditions of audio-visual synchronization and audio-visual identity consistency includes: Determine whether the target audio slice and the target image satisfy the audio-visual synchronization condition; If the audio-visual synchronization condition is met, determine whether the target audio slice and the target image meet the audio-visual identity consistency condition.

3. The method according to claim 2, wherein, Determining whether the target audio slice and the target image satisfy the audio-visual identity consistency condition includes: Based on the target audio slice, generate a timbre feature vector; Based on the target image, a target image feature vector is generated; Determine whether the matching degree between the timbre feature vector and the target image feature vector is greater than or equal to the second threshold.

4. The method according to claim 2, further comprising: If it is determined that the target audio slice and the target image do not meet the audio-visual synchronization conditions, the target audio slice and the target image are discarded, and a first prompt message is generated; wherein, the first prompt message includes: a first reason information and / or a first suggestion information for indicating that the target audio slice and the target image do not meet the audio-visual synchronization conditions.

5. The method according to claim 2 or 3, further comprising: If it is determined that the target audio slice and the target image do not meet the audio-visual identity consistency condition, a second prompt message is generated; wherein, the second prompt message includes: a second reason information and / or a second suggestion information for indicating that the target audio slice and the target image do not meet the audio-visual identity consistency condition.

6. The method according to any one of claims 1-3, wherein, The step of obtaining a target audio slice and a target image corresponding to the timestamp of the target audio slice based on video data includes: Determine the audio track data in the video data; Based on the audio track data, a timestamp log is obtained; wherein, the timestamp log includes: a timestamp and one or more graphic identifiers associated with the timestamp, the timestamp being used to identify the start and end times of audio segments in the audio track data; Based on the timestamp logs and the audio track data, the target audio slices corresponding to each of the image identifiers associated with the timestamps are determined. Based on the target audio slice and the video data, obtain the target image image corresponding to the timestamp of the target audio slice.

7. The method according to claim 6, wherein, The step of determining the target audio slice corresponding to each of the image identifiers associated with the timestamp based on the timestamp log and the audio track data includes: When the number of image identifiers associated with the timestamp is one, the target audio slice corresponding to the timestamp and the image identifier is obtained based on the timestamp and the audio track data; When there are multiple image identifiers associated with the timestamp, a mixed audio slice corresponding to the timestamp is obtained based on the timestamp and the audio track data, and speech separation is performed on the mixed audio slice to obtain target audio slices corresponding to multiple image identifiers respectively.

8. The method according to claim 6, wherein, Before determining whether the target audio slice and the target image satisfy the audio-visual synchronization condition and the audio-visual identity consistency condition, the method further includes: Determine whether the target audio slice and the target image meet quality conditions; wherein, the quality conditions include at least one of the following: the target image includes only one image, the target audio slice includes only the audio of a single image, and the noise of the target audio slice is less than or equal to a third threshold. The step of determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition includes: if it is determined that the target audio slice and the target image meet the quality conditions, then determining whether the target audio slice and the target image meet the audio-visual synchronization condition and the audio-visual identity consistency condition.

9. The method according to claim 8, further comprising: If the quality conditions are not met, the target audio slice and the target image are discarded, and a third prompt message is generated; wherein the third prompt message includes: a third reason message and / or a third suggestion message; wherein the third reason message is used to indicate the reason why the target audio slice and the target image do not meet the quality conditions, and the third suggestion message is used to indicate the acquisition of new video data.

10. The method according to any one of claims 1-3, further comprising: The data is used to train a mapping model, wherein the mapping model is used to generate a predicted timbre feature vector corresponding to the image feature vector based on the image feature vector, and the image feature vector is generated based on the image image.

11. The method according to any one of claims 1-3, wherein obtaining a target audio slice and a target image corresponding to a timestamp of the target audio slice based on video data comprises: Determine whether the video data meets the quality conditions; wherein the quality conditions include at least one of the following: the number of images in the video data is one, the audio of the video data includes only the audio of a single image, and the noise of the video data is less than or equal to a third threshold. If it is determined that the video data does not meet the quality conditions, the video data is discarded and a third prompt message is generated; wherein, the third prompt message includes: a third reason for indicating that the video data does not meet the quality conditions and a third suggestion message for indicating that new video data is to be provided; If the video data is determined to meet the quality requirements, a target audio slice and a target image corresponding to the timestamp of the target audio slice are obtained based on the video data.

12. The method according to any one of claims 1-3, further comprising: Based on the target audio slice, generate a timbre feature vector; Based on the timbre feature vector and the text information of the speech to be generated, a speech stream is generated.

13. A video data processing apparatus, comprising: The first acquisition module is configured to acquire a target audio slice and a target image image corresponding to the timestamp of the target audio slice based on video data. The first determining module is configured to determine whether the target audio slice and the target image image satisfy the audio-visual synchronization condition and the audio-visual identity consistency condition; wherein, the audio-visual synchronization condition includes: the synchronization of the lip movements of the target audio slice and the target image image in the target image image is greater than or equal to a first threshold; the audio-visual identity consistency condition includes: the matching degree between the target audio slice and the target image image in the target image image is greater than or equal to a second threshold. The first association module is configured to associate the target audio slice and the target image as a data pair when it is determined that the audio-visual synchronization condition and the audio-visual identity consistency condition are met.

14. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 12 when executing instructions stored in the memory.

15. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images