Method, apparatus, device and storage medium for data processing

Abstract

Embodiments of this disclosure relate to methods, apparatus, devices, and storage media for data processing. The proposed method includes: determining a set of candidate video segments from a plurality of video segments based on scene information of sample video segments; determining a set of candidate audio segments by removing speech content from a plurality of audio segments of sample audio; determining correlation information between the set of candidate video segments and the set of candidate audio segments; determining at least one sample pair based on the correlation information, each sample pair including a video segment and a corresponding audio segment; and constructing a training dataset for training an audio generation model based on the at least one sample pair, the audio generation model being configured to generate audio content corresponding to an input video. Thus, embodiments of this disclosure can improve the quality of the constructed training dataset. Furthermore, embodiments of this disclosure can subsequently improve the quality of the audio generation model based on such a training dataset.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for data processing. Background Technology

[0002] With the rapid development of computing power, audio generation technologies based on machine learning models (e.g., audio generation models) can be used to synthesize higher-quality audio more efficiently for applications in film and television production, game development, advertising creation, virtual reality, and augmented reality. Improving the quality of the training samples used to train these audio generation models can enhance their training performance, thereby improving the quality of the audio content generated. Summary of the Invention

[0003] In a first aspect of this disclosure, a data processing method is provided. The method includes: determining a set of candidate video segments from multiple video segments based on scene information of these video segments; determining a set of candidate audio segments by removing speech content from multiple audio segments of sample audio; determining correlation information between the set of candidate video segments and the set of candidate audio segments; determining at least one sample pair based on the correlation information, each sample pair including a video segment and a corresponding audio segment; and constructing a training dataset for training an audio generation model based on the at least one sample pair, the audio generation model being configured to generate audio content corresponding to an input video.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: a first determining module configured to determine a set of candidate video segments from a plurality of video segments based on scene information of these video segments; a second determining module configured to determine a set of candidate audio segments by removing speech content from a plurality of audio segments of sample audio; a third determining module configured to determine correlation information between the set of candidate video segments and the set of candidate audio segments; a fourth determining module configured to determine at least one sample pair based on the correlation information, each sample pair including a video segment and a corresponding audio segment; and a construction module configured to construct a training dataset for training an audio generation model based on the at least one sample pair, the audio generation model being configured to generate audio content corresponding to an input video.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A schematic diagram illustrating an example sample construction process according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart illustrating an example process of data processing according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for data processing according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As briefly mentioned above, audio generation technologies based on machine learning models (e.g., audio generation models) can be applied to the synthesis of audio such as speech, music, and sound effects for use in film and television production, game development, advertising creation, virtual reality, and augmented reality. By improving the quality of the training sample set used to train these audio generation models, the training effect of the audio generation models can be improved, thereby enhancing the quality of the audio content generated by the audio generation models.

[0020] To ensure the quality of audio content generated by such audio generation models, traditional techniques typically train suitable audio generation models separately for specific scenarios. Examples include audio generation models for video background music, video dubbing, and generating sound effects for input videos. Therefore, traditional techniques construct training sample sets using sample data collected from the specified scenarios. However, this approach often results in training sample sets with insufficient data size and inconsistent sample quality, thus affecting the training effectiveness and overall quality of the trained audio generation model.

[0021] Embodiments of this disclosure propose a data processing scheme. According to this scheme, a set of candidate video segments can be determined from multiple video segments based on scene information of these video segments. Further, a set of candidate audio segments can be determined by removing speech content from multiple audio segments of sample audio. Further, correlation information between the set of candidate video segments and the set of candidate audio segments can be determined. Further, based on the correlation information, at least one sample pair can be determined, each sample pair including a video segment and a corresponding audio segment. Additionally, based on at least one sample pair, a training dataset for training an audio generation model can be constructed, the audio generation model being configured to generate audio content corresponding to the input video.

[0022] Based on this approach, embodiments of this disclosure can determine a set of candidate video segments based on scene information from multiple video segments of a sample video, reducing the impact of scenes that interfere with the training of the audio generation model on the subsequent training process. Furthermore, embodiments of this disclosure can avoid interference from speech content in the training process of the audio generation model by removing speech content from multiple audio segments of the sample audio. Furthermore, embodiments of this disclosure can improve the matching degree between the video and audio segments included in the sample pair by determining the correlation information between a set of candidate video segments and a set of candidate audio segments, thereby improving the quality of the sample pair. Thus, based on such a higher-quality set of candidate audio segments and the more relevant sample pairs determined by such a higher-quality set of candidate audio segments, embodiments of this disclosure can utilize a larger amount of sample video and sample audio data to construct a higher-quality training dataset for a specified scene, thereby improving the training effect of such an audio generation model and the quality of the audio content corresponding to the input video subsequently generated based on such an audio generation model.

[0023] Therefore, embodiments of this disclosure can improve the quality of a set of candidate video segments by utilizing scene information from video clips, improve the quality of a set of candidate audio clips by removing speech content from audio clips, and improve the quality of sample pairs by ensuring the correlation between video and audio clips in the sample pairs. This, in turn, can improve the quality of the training dataset constructed based on such sample pairs. Furthermore, embodiments of this disclosure can also improve the training effect and model quality of subsequently trained audio generation models based on such training datasets.

[0024] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0025] Example Environment

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, electronic device 110 can perform data processing on sample video 120 and sample audio 130 to construct a training sample set 140 for training an audio generation model. Such electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, and / or server devices, etc.

[0027] The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device may also support any type of user-facing interface (such as "wearable" circuitry).

[0028] Server equipment can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server equipment may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments, etc.

[0029] A communication connection can be established between the server device and the terminal device. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections, and the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the server device and the terminal device can achieve signaling interaction through the communication connection between them.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example sample construction process

[0033] The following will be referenced Figure 2 This describes some example sample construction processes according to embodiments of the present disclosure. Figure 2 The diagram illustrates an example sample construction process according to some embodiments of the present disclosure. Process 200 may be, for example, constructed by... Figure 1 The electronic device 110 shown is provided.

[0034] In some embodiments, the electronic device 110 can directly construct a training sample set for an audio generation model using sample data. However, the quality of such a training sample set is low, resulting in poor training performance of the audio generation model. Therefore, in some embodiments, the electronic device 110 can process the sample data to determine at least one sample pair containing a video clip and an audio clip, and then construct a training sample set for training the audio generation model based on such at least one sample pair. For example, such sample data may include at least one sample video and at least one sample audio, thus embodiments of this disclosure can process at least such at least one sample video and at least one sample audio. For example, such at least one sample video and at least one sample audio may be extracted from the same video content or may correspond to different media content sources. Therefore, embodiments of this disclosure can guarantee the amount of data collected from the sample data.

[0035] The following embodiments of this disclosure use the example of electronic device 110 processing sample video 120 and sample audio 130 to construct a corresponding training sample set. The specific implementation of electronic device 110 processing multiple sample videos and multiple sample audios is achieved by reusing such data processing process. The embodiments of this disclosure will not elaborate on such reusing process.

[0036] As an example, such sample video 120 may include scene content from at least one scenario, examples of which include, but are not limited to, indoor scenes, outdoor scenes, static images, dynamic images, etc. Such sample audio 130 may include at least one type of audio sound, examples of which include, but are not limited to, speech, music, etc. To ensure the application quality of the subsequently trained audio generation model in the specified scenario, the electronic device 110 may perform data cleaning on such sample video 120 and such sample audio 130. The following uses such a specified scenario as an example of an audio effect generation scenario, that is, such an audio generation model is used to generate the audio effect content corresponding to the input video. It should be understood that such a specified scenario is only used to clearly describe the embodiments of this disclosure, and this disclosure is not intended to limit the specific application scenarios of such an audio generation model.

[0037] In the description of embodiments of this disclosure, the term "sound effect" and similar terms should be understood as effects created by sound. Sound effects (also known as sound content) are widely used in movies, video games, music, and other media. Artificially created or enhanced sound effects can enhance the artistic effect of media content. Typically, sound effects are used in the processing of media content, rather than through dialogue or music inherent in the media content itself. For example, sound effects may include thunder, the sound of horses' hooves, birdsong, applause, car horns, and so on.

[0038] In some embodiments, the electronic device 110 determines a set of candidate video segments from multiple video segments of the sample video 120 based on scene information. This allows the electronic device 110 to reduce the interference of video content from other scenes unrelated to the specified scene on the subsequent training process of the audio generation model. For example, such scene information includes, but is not limited to, the image content corresponding to the video segment (e.g., static image, dynamic image, etc.), environmental information (e.g., indoor environment, outdoor environment), image elements (e.g., musical instrument performance scene, non-musical instrument performance scene), etc. Thus, the electronic device 110 can determine a set of candidate video segments with less interference from multiple video segments of the sample video 120, thereby improving the quality of such a set of candidate video segments.

[0039] In some embodiments, such a sample video 120 may be associated with multiple shots (also known as storyboards). To ensure the accuracy of scene information for the multiple video segments of the sample video, the electronic device 110 may divide the sample video into multiple video segments corresponding to different storyboards. As an example, such a shot detection tool may include any suitable shot detection algorithm, etc., and this disclosure is not intended to limit the specific implementation of shot detection.

[0040] Furthermore, the electronic device 110 can perform scene recognition for each video segment under each shot, thereby improving the accuracy of scene recognition. As an example, such scene recognition can be achieved by any suitable model, such as a pre-trained scene classification model 202. Such a scene classification model 202 can classify the input video segments into scenes to obtain the corresponding scenes.

[0041] In some embodiments, the electronic device 110 can determine whether multiple video segments of the sample video 120 include video segments corresponding to a preset scene based on scene information. As an example, such a preset scene could be a scene that interferes with the training effect of such an audio generation model. For example, such a specified scene could indicate a video scene with a high probability of sound effects, such as a moving scene, an outdoor scene, etc. Conversely, such a preset scene could indicate a video scene with a low probability of sound effects. For example, such a preset scene could indicate that the video segment corresponds to a static scene. Additionally or alternatively, such a preset scene could indicate that the video segment corresponds to an indoor environment.

[0042] Furthermore, if the multiple video segments include a video segment corresponding to a preset scene, the electronic device 110 can remove at least one video segment corresponding to the preset scene from the multiple video segments to obtain a set of video segments (labeled as group A for easy distinction). As an example, the electronic device 110 can use such group A video segments as such a set of candidate video segments.

[0043] In some embodiments, a musical instrument performance scene can be associated with musical instrument performance sound effects. To improve the training effect of the audio generation model, the electronic device 110 can provide the audio generation model with samples corresponding to such musical instrument performance scenes for learning. For example, if one of the multiple video clips includes a video clip corresponding to musical instrument performance scene 206, the electronic device 110 can add the video clip corresponding to musical instrument performance scene 206 to such a set of candidate video clips. However, for video clips among these multiple video clips that correspond to non-musical instrument performance scene 204 but also to a preset scene, the electronic device 110 can remove such video clips from the multiple video clips. Thus, the electronic device 110 can retain video clips related to musical instrument performance scenes, thereby improving the diversity of the audio generated by the audio generation model.

[0044] In some embodiments, if such a set of candidate video clips contains subtitle content, this subtitle content will affect the subsequent training process of the audio generation model. Therefore, the electronic device 110 can also crop the portion of the screen corresponding to the subtitle content from such a set of candidate video clips to obtain a cropped set of candidate video clips (e.g., labeled as group B video clips). As an example, the electronic device 110 can utilize a subtitle cropping tool 208 to perform this cropping process. Such a subtitle cropping tool can include any model capable of cropping specified content from a video or a media editing tool containing any appropriate subtitle detection algorithm, etc.

[0045] Based on this video processing procedure, the electronic device 110 can obtain a set of candidate video segments of high quality. Further, the electronic device 110 can process the sample audio 130 to improve the quality of the determined set of candidate audio segments. Specifically, the electronic device 110 determines a set of candidate audio segments by removing speech content from multiple audio segments of such sample audio 130. Thus, embodiments of this disclosure can avoid the influence of speech content on the training process of the audio generation model.

[0046] To ensure the accuracy of subsequent matching with a set of candidate video segments, the electronic device 110 can segment the sample audio 130 to obtain a set of audio segments corresponding to a preset length. As an example, this preset length can be consistent with the length of multiple video segments of the sample video 120 described above. Further, the electronic device 110 can utilize the sound event detection model 210 to determine multiple audio segments containing sound events from the audio segment set. As an example, such sound events can correspond to events in the audio content that the audio generation model needs to output, such as sound effect events. Therefore, embodiments of this disclosure can avoid audio segments unrelated to such sound events interfering with the subsequent training process of the audio generation model, thereby improving the training effect of the subsequent audio generation model.

[0047] In some embodiments, the electronic device 110 may retain audio segments related to instrument performance while removing music segments unrelated to instrument performance from the audio segments, thereby improving the diversity of audio content generated by the subsequent audio generation model. Specifically, taking the electronic device 110 performing audio processing on a first audio segment as an example, if such a first audio segment contains music content (e.g., such a first audio segment is music segment 212), the electronic device 110 can determine the target video segment corresponding to such a first audio segment.

[0048] Furthermore, if the target video segment is related to a musical instrument performance scene, the electronic device 110 can retain the music content in the first audio segment and determine the candidate audio segment corresponding to the first audio segment by removing the speech content. If the target video segment is not related to a musical instrument performance scene, for example, if the first audio segment is speech segment 214, the electronic device 110 can determine the candidate audio segment corresponding to the first audio segment by removing both the music and speech content. In this way, embodiments of this disclosure can improve the diversity of audio content generated by the subsequent audio generation model while avoiding interference from speech content, music content from non-musical instrument performance scenes, etc., in the subsequent training process for the audio generation model.

[0049] In some embodiments, the higher the energy of the speech content removed by the electronic device 110, the more severely the sound quality of the resulting audio segment is degraded. Therefore, if the energy of the removed speech content in an audio segment exceeds a first threshold, the electronic device 110 can remove such an audio segment to ensure the sound quality of a set of candidate audio segments. For example, taking a second audio segment, the electronic device 110 can determine the energy corresponding to the removed speech content in the second audio segment. For example, such a second audio segment and a first audio segment can be the same audio segment or different audio segments; for example, such a second audio segment can be obtained by processing the first audio segment, or such a first audio segment can be obtained by processing the second audio segment.

[0050] In some embodiments, the electronic device 110 may use a sound source separation tool 216 (e.g., any tool or model capable of separating speech and audio) to remove speech content from an audio segment, or may trim the portion of the audio segment containing speech content. This disclosure is not intended to limit the specific implementation of removing speech content. Further, the electronic device 110 may use a sound quality detection tool 222 (e.g., any tool or model capable of detecting the energy of speech content) to detect the energy corresponding to the removed speech content in the second audio segment. If the energy corresponding to the removed speech content in such a second audio segment is less than a first threshold, the electronic device 110 may add the second audio segment after removing the speech content to such a set of candidate audio segments.

[0051] Based on this audio processing procedure, embodiments of this disclosure can obtain a set of candidate audio segments of high quality. Further, the electronic device 110 can determine the correlation information between a set of candidate video segments and a set of candidate audio segments, and based on this correlation information, pair such a set of candidate video segments (Group A candidate video segments, or Group B candidate video segments, etc.) with such a set of candidate audio segments, thereby improving the matching degree between the video segment 226 and the corresponding audio segment 228 in the determined sample pair 224, and thus improving the quality of such sample pairs.

[0052] Specifically, the electronic device 110 can determine video and audio segments with a matching degree greater than a second threshold based on the correlation information between a set of candidate video segments and a set of candidate audio segments, and determine such sample pairs 224 based on such video and audio segments with a matching degree greater than the second threshold. Furthermore, the electronic device 110 can construct a training sample set for training an audio generation model based on such sample pairs 224, thereby improving the quality of such training sample set.

[0053] In some embodiments, the electronic device 110 may utilize the audio-video matching unit 220 to determine the relevance information of a candidate video segment and a candidate audio segment by characterization similarity. Specifically, the electronic device 110 may determine multiple image feature representations by encoding multiple image frames in the candidate video segment. Further, the electronic device 110 may determine the video feature representation corresponding to the candidate video segment based on the multiple image feature representations. Further, the electronic device 110 may determine the similarity between the video feature representation and the audio feature representation of the candidate audio segment. Further, the electronic device 110 may determine the relevance information of the candidate video segment and the candidate audio segment based on the similarity. As an example, the similarity between the video feature representation and the audio feature representation of the candidate audio segment can be determined by various feature similarity calculation methods, such as vector distance, cosine similarity, etc., and this disclosure is not intended to limit such feature similarity calculation methods.

[0054] In some embodiments, the electronic device 110 can also perform joint classification of such a set of candidate video segments and a set of candidate audio segments by the audio-video matching unit 220, and then use a joint feature classification model to classify the joint features to determine the correlation information between the set of candidate video segments and the set of candidate audio segments. As an example, such joint classification can include various implementation methods, such as random pairing, pairing based on preset rules, etc., and such joint feature classification models can also be implemented by any suitable model. This disclosure is not intended to limit such joint classification methods and the classification methods of correlation information. Thus, the embodiments of this disclosure can comprehensively improve the quality of the constructed training sample set from multiple aspects, such as improving audio quality, improving video quality, and improving the degree of audio-video matching.

[0055] In this way, embodiments of this disclosure can determine a set of candidate video segments based on scene information from multiple video segments of a sample video, reducing the impact of scenes that interfere with the training of the audio generation model on the subsequent training process of the audio generation model. Furthermore, embodiments of this disclosure can avoid interference from speech content in the training process of the audio generation model by removing speech content from multiple audio segments of the sample audio. Furthermore, embodiments of this disclosure can improve the matching degree between the video segments and audio segments included in the sample pair by determining the correlation information between a set of candidate video segments and a set of candidate audio segments, thereby improving the quality of the sample pair. Therefore, based on such a higher-quality set of candidate audio segments and the more relevant sample pairs determined by such a higher-quality set of candidate audio segments, embodiments of this disclosure can construct a higher-quality training dataset for a specified scene, thereby improving the training effect of such an audio generation model and the quality of the audio content corresponding to the input video subsequently generated based on such an audio generation model.

[0056] Therefore, embodiments of this disclosure can improve the quality of a set of candidate video segments by utilizing scene information from video clips, improve the quality of a set of candidate audio clips by removing speech content from audio clips, and improve the quality of sample pairs by ensuring the correlation between video and audio clips in the sample pairs. This, in turn, can improve the quality of the training dataset constructed based on such sample pairs. Furthermore, embodiments of this disclosure can also improve the training effect and model quality of subsequently trained audio generation models based on such training datasets.

[0057] Example process

[0058] Figure 3 A flowchart of an example process 300 for data processing according to some embodiments of the present disclosure is shown. Process 300 can be implemented at electronic device 110. Reference is made below. Figure 1Describe the process 300.

[0059] like Figure 3 As shown in box 310, electronic device 110 determines a set of candidate video segments from multiple video segments based on scene information of multiple video segments of sample video.

[0060] In box 320, electronic device 110 determines a set of candidate audio segments by removing speech content from multiple audio segments of sample audio.

[0061] In box 330, electronic device 110 determines the correlation information between a set of candidate video segments and a set of candidate audio segments.

[0062] In box 340, electronic device 110 determines at least one sample pair based on correlation information, each sample pair including a video clip and a corresponding audio clip.

[0063] In box 350, electronic device 110 constructs a training dataset for training an audio generation model based on at least one sample pair. The audio generation model is configured to generate audio content corresponding to the input video.

[0064] In some embodiments, process 300 further includes: dividing the sample video into multiple video segments corresponding to different scene cuts.

[0065] In some embodiments, determining a set of candidate video segments from multiple video segments based on scene information of multiple video segments of a sample video includes: determining whether the multiple video segments include a video segment corresponding to a preset scene based on scene information of the multiple video segments of the sample video; and removing at least one video segment from the multiple video segments in response to the multiple video segments including at least one video segment corresponding to the preset scene, thereby determining a set of candidate video segments.

[0066] In some embodiments, the preset scene indicates at least one of the following: the video clip corresponds to a static image; the video clip corresponds to an indoor environment.

[0067] In some embodiments, determining the correlation information between a set of candidate video segments and a set of candidate audio segments includes: cropping the screen portion corresponding to the subtitle content from a set of candidate video segments; and determining the correlation information between the cropped set of candidate video segments and the set of candidate audio segments.

[0068] In some embodiments, process 300 further includes: segmenting the sample audio into a set of audio segments corresponding to a preset length; and using a sound event detection model to determine multiple audio segments containing sound events from the set of audio segments.

[0069] In some embodiments, determining a set of candidate audio segments by removing speech content from multiple audio segments of sample audio includes: determining a target video segment corresponding to a first audio segment; and in response to determining that the video content of the target video segment is not related to the instrument performance, removing music content and speech content from the first audio segment to determine the corresponding candidate audio segment.

[0070] In some embodiments, determining a set of candidate audio segments by removing speech content from multiple audio segments of sample audio includes: determining the energy corresponding to the removed speech content in a second audio segment; and adding the second audio segment after removing the speech content to the set of candidate audio segments in response to the energy being less than a first threshold.

[0071] In some embodiments, the degree of matching between the video segment and the corresponding audio segment in at least one sample pair is greater than a second threshold.

[0072] In some embodiments, determining the correlation information between a set of candidate video segments and a set of candidate audio segments includes: determining multiple image feature representations by encoding multiple image frames in the candidate video segments; determining video feature representations corresponding to the candidate video segments based on the multiple image feature representations; determining the similarity between the video feature representations and the audio feature representations of the candidate audio segments; and determining the correlation information between the candidate video segments and the candidate audio segments based on the similarity.

[0073] Example devices and equipment

[0074] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for data processing according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in electronic device 110. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0075] like Figure 4As shown, the apparatus 400 includes a first determining module 410 configured to determine a set of candidate video segments from multiple video segments based on scene information of multiple video segments of sample videos; a second determining module 420 configured to determine a set of candidate audio segments by removing speech content from multiple audio segments of sample audio; a third determining module 430 configured to determine correlation information between a set of candidate video segments and a set of candidate audio segments; a fourth determining module 440 configured to determine at least one sample pair based on the correlation information, each sample pair including a video segment and a corresponding audio segment; and a construction module 450 configured to construct a training dataset for training an audio generation model based on at least one sample pair, the audio generation model being configured to generate audio content corresponding to the input video.

[0076] In some embodiments, process 300 further includes: dividing the sample video into multiple video segments corresponding to different scene cuts.

[0077] In some embodiments, the first determining module 410 is further configured to: determine whether the multiple video segments include a video segment corresponding to a preset scene based on scene information of multiple video segments of the sample video; and in response to the multiple video segments including at least one video segment corresponding to the preset scene, remove at least one video segment from the multiple video segments to determine a set of candidate video segments.

[0078] In some embodiments, the preset scene indicates at least one of the following: the video clip corresponds to a static image; the video clip corresponds to an indoor environment.

[0079] In some embodiments, the third determining module 430 is further configured to: crop the portion of the screen corresponding to the subtitle content from a set of candidate video segments; and determine the correlation information between the cropped set of candidate video segments and a set of candidate audio segments.

[0080] In some embodiments, the apparatus 400 further includes a fifth determining module configured to: segment the sample audio into a set of audio segments corresponding to a preset length; and use a sound event detection model to determine multiple audio segments containing sound events from the set of audio segments.

[0081] In some embodiments, the second determining module 420 is further configured to: determine a target video segment corresponding to the first audio segment; and in response to determining that the picture content of the target video segment is not related to the instrument performance, remove the music content and speech content from the first audio segment to determine the corresponding candidate audio segment.

[0082] In some embodiments, the second determining module 420 is further configured to: determine the energy corresponding to the removed speech content in the second audio segment; and in response to the energy being less than a first threshold, add the second audio segment after removing the speech content to a set of candidate audio segments.

[0083] In some embodiments, the degree of matching between the video segment and the corresponding audio segment in at least one sample pair is greater than a second threshold.

[0084] In some embodiments, the third determining module 430 is further configured to: determine multiple image feature representations by encoding multiple image frames in a candidate video segment; determine a video feature representation corresponding to the candidate video segment based on the multiple image feature representations; determine the similarity between the video feature representation and the audio feature representation of the candidate audio segment; and determine the correlation information between the candidate video segment and the candidate audio segment based on the similarity.

[0085] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0086] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 4 Device 400.

[0087] like Figure 5As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0088] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0089] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0090] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0091] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: Based on scene information from multiple video segments of the sample video, a set of candidate video segments is determined from the multiple video segments; A set of candidate audio segments is determined by removing speech content from multiple audio segments of the sample audio. Determine the correlation information between the set of candidate video segments and the set of candidate audio segments; Based on the correlation information, at least one sample pair is determined, and each sample pair includes a video segment and a corresponding audio segment. as well as Based on the at least one sample pair, a training dataset is constructed for training an audio generation model, which is configured to generate audio content corresponding to the input video.

2. The method according to claim 1, further comprising: The sample video is divided into multiple video segments corresponding to different scene sequences.

3. The method according to claim 1, wherein determining a set of candidate video segments from the plurality of video segments based on scene information of the sample video comprises: Based on scene information from multiple video segments of a sample video, determine whether the multiple video segments include video segments corresponding to a preset scene; as well as In response to the plurality of video segments including at least one video segment corresponding to the preset scene, the at least one video segment is removed from the plurality of video segments to determine the set of candidate video segments.

4. The method of claim 3, wherein the preset scenario indicates at least one of the following: Video clips correspond to still images; The video clip corresponds to an indoor environment.

5. The method according to claim 1, wherein determining the correlation information between the set of candidate video segments and the set of candidate audio segments includes: From the set of candidate video clips, crop the portion of the image corresponding to the subtitle content; as well as Determine the correlation information between the cropped set of candidate video segments and the set of candidate audio segments.

6. The method according to claim 1, further comprising: The sample audio is divided into a set of audio segments corresponding to a preset length; as well as Using a sound event detection model, the plurality of audio segments containing sound events are identified from a set of audio segments.

7. The method of claim 1, wherein determining a set of candidate audio segments by removing speech content from multiple audio segments of the sample audio comprises: Identify the target video segment corresponding to the first audio segment; as well as In response to determining that the content of the target video segment is unrelated to the instrumental performance, the music content and the speech content are removed from the first audio segment to determine the corresponding candidate audio segment.

8. The method of claim 1, wherein determining the set of candidate audio segments by removing speech content from a plurality of audio segments of the sample audio comprises: Determine the energy corresponding to the removed speech content in the second audio segment; as well as In response to the energy being less than a first threshold, the second audio segment after removing the speech content is added to the set of candidate audio segments.

9. The method according to claim 1, wherein the degree of matching between the video segment and the corresponding audio segment in the at least one sample pair is greater than a second threshold.

10. The method of claim 1, wherein determining the correlation information between the set of candidate video segments and the set of candidate audio segments comprises: Multiple image feature representations are determined by encoding multiple image frames in a candidate video segment; Based on the multiple image feature representations, determine the video feature representation corresponding to the candidate video segment; Determine the similarity between the video feature representation and the audio feature representation of the candidate audio segment; as well as Based on the similarity, the relevance information of the candidate video segment and the candidate audio segment is determined.

11. An apparatus for data processing, comprising: The first determining module is configured to determine a set of candidate video segments from multiple video segments based on scene information of the sample video. The second determining module is configured to determine a set of candidate audio segments by removing speech content from multiple audio segments of the sample audio. The third determining module is configured to determine the correlation information between the set of candidate video segments and the set of candidate audio segments; The fourth determining module is configured to determine at least one sample pair based on the correlation information, each sample pair including a video segment and a corresponding audio segment; as well as A building module is configured to build a training dataset for training an audio generation model based on the at least one sample pair, the audio generation model being configured to generate audio content corresponding to the input video.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images