Method for generating background audio model, method for generating background audio and related devices

Abstract

The present disclosure relates to a background audio model generation method, a background audio generation method and related devices. The generation method comprises: obtaining a training sample set, wherein each training sample in the training sample set comprises a sample synthesized video, and the sample synthesized video comprises a sample video and a sample background audio configured for the sample video; obtaining image frame features of the sample video, video semantic features of the sample synthesized video, and audio features of the sample background audio respectively; inputting the image frame features, the video semantic features and the audio features as first input parameters of an audio model, and inputting the image frame features and the audio features as second input parameters of the audio model, into the audio model to obtain a first background audio; and training the audio model according to at least the first background audio and the sample background audio to obtain a background audio model. In this way, the background audio generated by the trained background audio model has a high matching degree with the video, and the generalization ability and robustness of the model are improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for generating a background audio model, a background audio generation method, and related apparatus. Background Technology

[0002] Social media platforms have spawned a new form of entertainment, where diverse visuals are often paired with carefully chosen background music. While advancements in mobile device technology have made shooting high-quality videos easier, finding suitable background music that perfectly complements the video content remains a challenging task, and such music is often protected by copyright.

[0003] Currently, background sound effects for videos are mostly generated using neural network models. For example, the Capability Maturity Model (CMT) is used to generate background audio from videos. However, the CMT model generates background audio based on pre-defined rules and three key features, including video frame extraction, the temporal relationship between frames, and the image similarity between frames. Another example is V2Meo, a visually modulated music generation system capable of generating high-fidelity music audio from silent videos. V2Meo uses pre-trained visual features extracted from silent video clips to generate music audio waveforms. Furthermore, it supports text prompts and video modulation to control the music style. However, V2Meo uses audio waveforms as training input and output, rather than symbolic music data (e.g., not MIDI (Musical Instrument Digital Interface) data), making the musical attributes ambiguous. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a method for generating a background audio model, a background audio generation method, and related apparatus.

[0005] According to a first aspect of the present disclosure, a method for generating a background audio model is provided, the background audio model including an audio model, the generation method comprising: Obtain a training sample set, wherein the training samples in the training sample set include sample synthesized videos, and the sample synthesized videos include sample videos and sample background audio configured for the sample videos; The image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio are obtained respectively. The image frame features, the video semantic features, and the audio features are used as the first input parameters of the audio model, and the image frame features and the audio features are used as the second input parameters of the audio model, which are then input into the audio model to obtain the first background audio. The audio model is trained based on at least the first background audio and the sample background audio to obtain a background audio model.

[0006] According to a second aspect of the present disclosure, a background audio generation method is provided, the background audio generation method comprising: Get the video to be processed and initialize the audio; The video semantic features, image features, and audio features of the initial audio of the video to be processed are obtained respectively; The video semantic features, image frame features, and audio features of the initial audio of the video to be processed are used as the first input parameters of the audio model in the background audio model, and the image frame features and audio features of the initial audio of the video to be processed are used as the second input parameters of the audio model. The audio model is then input to obtain the third background audio. The background audio model is generated according to the background audio model generation method described in the first aspect of the present disclosure.

[0007] A third aspect of the present disclosure provides an apparatus for generating a background audio model, the background audio model including an audio model, the generating apparatus comprising: The first acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample synthesized video, and the sample synthesized video includes a sample video and sample background audio configured for the sample video; The first acquisition module is configured to acquire the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio, respectively. The third acquisition module is configured to use the image frame features, the video semantic features, and the audio features as the first input parameters of the audio model, and use the image frame features and the audio features as the second input parameters of the audio model to input the audio model to obtain the first background audio. The training module is configured to train the audio model based on at least the first background audio and the sample background audio to obtain a background audio model.

[0008] According to a fourth aspect of the present disclosure, a background audio generation apparatus is provided, the background audio generation apparatus comprising: The fourth acquisition module is configured to acquire the video to be processed and initialize the audio; The fifth acquisition module is configured to acquire the video semantic features, image features, and audio features of the initial audio of the video to be processed, respectively. The sixth acquisition module is configured to use the video semantic features, image features, and audio features of the initial audio of the video to be processed as the first input parameters of the audio model in the background audio model, and use the image features of the video to be processed and the audio features of the initial audio of the video to be processed as the second input parameters of the audio model, and input them into the audio model to obtain the third background audio; The background audio model is generated according to the background audio model generation method described in the first aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, the processor is configured to, when executing the instructions, implement the steps of the method for generating a background audio model as described in any one of the first aspects of the embodiments of this disclosure, and / or implement the steps of the method for generating a background audio model as described in any one of the first aspects of the embodiments of this disclosure.

[0010] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method for generating a background audio model as described in any one of the first aspects of the present disclosure, and / or implement the steps of the method for generating a background audio model as described in any one of the first aspects of the present disclosure.

[0011] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method for generating a background audio model as described in any of the first aspects of the present disclosure, and / or implements the steps of the method for generating a background audio model as described in any of the first aspects of the present disclosure.

[0012] By employing the above technical solution, image frame features, video semantic features, and audio features are jointly used as the first input parameter of the model, while image frame features and audio features are used as the second input parameter of the audio model. This means that during training, the audio features of the sample background audio are also input into the model as guiding features for training. Thus, the background audio generated by the trained background audio model has a high degree of matching with the video, improving the model's generalization ability and robustness. Furthermore, since video semantic features are considered during training, the resulting background audio model can generate matching background audio based on video semantics, further improving the matching degree between the generated background audio and the video, thereby enhancing the user's entertainment experience.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0015] Figure 1 This is a flowchart illustrating a method for generating a background audio model according to an exemplary embodiment.

[0016] Figure 2 This is a schematic diagram illustrating the generation of a background audio model according to an exemplary embodiment.

[0017] Figure 3 This is a flowchart illustrating a background audio generation method according to an exemplary embodiment.

[0018] Figure 4 This is a schematic diagram illustrating a background audio generation method according to an exemplary embodiment.

[0019] Figure 5 This is a block diagram illustrating a background audio model generation apparatus according to an exemplary embodiment.

[0020] Figure 6 This is a block diagram illustrating a background audio generation apparatus according to an exemplary embodiment.

[0021] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0023] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0024] In related technologies, the technology for generating background audio for videos is a multi-stage synthesis technology, which includes, for example, a video processing stage, a video understanding stage, and a background audio generation stage. It is composed of single modules stacked together, which is cumbersome and has significant shortcomings in the correlation between audio effects and video. That is, the existing models cannot generate background audio with a high degree of correlation for videos, resulting in a low matching degree between the generated background audio and the video, which affects the user's entertainment experience.

[0025] In view of this, this disclosure provides a method for generating a background audio model, a background audio generation method, and related apparatus. Image frame features, video semantic features, and audio features are jointly used as input parameters for an audio model, and sample background audio is used as the output parameter for the audio model. The audio model is then trained to obtain the background audio model. During training, the audio features of the sample background audio are also input into the model as guiding features for model training. This results in a high degree of matching between the background audio generated by the trained background audio model and the video, improving the model's generalization ability and robustness. Furthermore, video semantic features are considered during training; therefore, the obtained background audio model can generate background audio that matches the video semantics, further improving the matching degree between the generated background audio and the video, thereby enhancing the user's entertainment experience.

[0026] Figure 1 This is a flowchart illustrating a method for generating a background audio model according to an exemplary embodiment. Figure 1 As shown, the generation method may include the following steps.

[0027] In step S11, the training sample set is obtained.

[0028] The training samples in the training sample set include sample-synthesized videos, which consist of sample videos and sample background audio configured for the sample videos.

[0029] In this disclosure, one sample background audio can be pre-configured for each sample video, or multiple sample background audios can be pre-configured for each sample video to enrich the training samples. For example, for sample video A, sample background audio a can be configured for sample video A to obtain sample synthesized video 1, sample background audio b can be configured for sample video A to obtain sample synthesized video 2, sample background audio c can be configured for sample video A to obtain sample synthesized video 3, and so on.

[0030] The background audio can be background music or background sound effects.

[0031] In step S12, the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio are obtained respectively.

[0032] For example, other modules besides the background audio model can be used to obtain the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio, respectively. This disclosure does not specifically limit this.

[0033] The image frame features of the sample video refer to the features obtained by extracting features from multiple sample images after dividing the sample video into multiple frames, which are used to represent the image frames. Image frame features include at least one of the following: text description features, emotion features, scene change features, motion state features, and video acoustic event features. The audio features of the sample background audio include at least one of the following: note density features, loudness features, chord features, pitch features, and audio acoustic event features. The video semantic features of the synthesized sample video can be token sequence encoded features.

[0034] The audio features can be of two types. One type involves transcribing chords and MIDI files from music tracks and extracting features from these MIDI files. These features are used in model training to enable the trained model to generate music with different rhythms and volume levels. The second type is traditional audio waveform data, which provides time-frequency domain-based audio features and can also provide useful information for modeling acoustic events.

[0035] Of the five categories of features—note density, loudness, chord, pitch, and audio acoustic event features—the first four align with music, while the last aligns with sound effects. These features play a crucial role in capturing the musical characteristics and composition of audio. In other words, when the background audio is background sound effects, the audio features are audio acoustic event features; when the background audio is background music, the audio features include at least one of the following: note density, loudness, chord, pitch, and audio acoustic event features.

[0036] The following describes the methods for extracting audio features.

[0037] (1) Note density characteristics: In order to calculate the audio density, the audio files in the music video are transcribed into MIDI files. For each transcribed MIDI file, the total number of notes in each second interval is calculated and used as the note density of the audio.

[0038] (2) Loudness characteristics: To accurately estimate the loudness per second of the audio files, an additional standard library was used to calculate the root mean square loudness. The root mean square loudness values ​​were then converted to decibels and then to characteristic values ​​on a 0-1 scale. Since decibels are closer to human perception of loudness, this conversion process ensures accurate and consistent loudness measurement, and mapping these values ​​to a 0-1 scale further enhances the interpretability of the loudness values.

[0039] (3) Chord Features: Chord sequences were extracted from audio files using an open-source Transformer-based chord recognition model. One chord was detected in each 1-1.5 second window, generating chord sequences of 16 different chord types, including major chords, diminished chords, suspended chords, minor sevenths, minor chords, augmented chords, diminished sevenths, major sixths, half-diminished sevenths, sevenths, minor sixths, and major sevenths, etc. The chord sequences of each file were normalized based on the detected key. Depending on whether the detected key was major or minor, the song's chords were transposed to C major or A minor, respectively.

[0040] (4) Pitch characteristics: After extracting the chord sequence, the chord sequence is converted into a MIDI file using simple music theory. The duration of each chord is precisely mapped to the notes in the MIDI file using the start and end times of each chord. Then, the MIDI file is used to determine the key of each song, aiming to capture a wide range of audio features that can help with pitch recognition.

[0041] (5) Audio acoustic event features: For audio features, the acoustic event categories and timestamps included are identified by audio detection technology. The timestamps are filtered by a median filter window to ensure that the sound effects have a smooth transition time. For single or superimposed audio acoustic events, their Mel features are extracted, where the frame shift is 10 milliseconds, the frame length is 20 milliseconds, the FFT window is 1024, and the Mel feature dimension is extracted to 128 dimensions.

[0042] Compared to existing technologies that only use MIDI files for audio generation, using the aforementioned audio features avoids the problem that existing technologies can only generate single chords instead of polyphonic notes, and improves the matching degree between rhythm, speed, emotion and video.

[0043] In related technologies, raw video frames are typically used directly as conditional inputs to background audio models, leading to difficulties in effectively learning the correspondences between different modalities. Therefore, in this disclosure, meaningful features are extracted from raw video frames as intermediate representations to effectively learn the correspondences between different modalities, thus simplifying the learning process. For example, the image frame features of the video may include, but are not limited to: textual description features, emotion features, scene change features, motion state features, and video acoustic event features.

[0044] (6) Textual Description Features: Using CLIP (Contrastive Language-Image Pre-training, a pre-trained model based on contrastive text-image pairs) as a feature extractor, the original video frames are encoded into textual description feature tags without fine-tuning. Latent features are extracted from each video frame using CLIP. These extracted latent features cover a wide range of video textual descriptions, including, for example, tranquil beach scenes, adventurous outdoor activities, and bustling city streets.

[0045] (7) Emotional Features: To estimate the emotions expressed in each second of video, the CLIP emotion detection probabilities are included across eight emotion categories. By leveraging pre-trained knowledge gained from exposure to various image-text pairs during training, CLIP can provide a probability distribution for different emotion categories for each frame in the video. For example, emotion categories may include excitement, passion, fear, tension, sadness, relaxation, soothing, and neutral. The emotion probability for each second of video is obtained, and a smooth window of size x is applied to each of the emotion probability time series corresponding to the eight emotion categories. Since one frame is extracted per second, this means that x seconds of review are required when detecting emotions, allowing for more stable detection of emotional content.

[0046] (8) Scene Change Features: An open-source C++ tool library was used to accurately detect shot changes in the video. Instead of directly using the scene ID as a feature to integrate scene change information, scene change was calculated based on the detected scene ID. By introducing a scene change value that starts from 0 and gradually increases until the next scene change, the relative position of each frame in the scene was effectively captured. This approach allows scene change information to be implicitly encoded by considering the temporal distance between frames within and between scenes.

[0047] (9) Motion State Features: To estimate the motion state or changes of the video visual content, the RGB difference between the current frame and the previous frame is first calculated within each second interval. This process involves independently calculating the absolute difference in color values ​​of the corresponding pixels on the red, green, and blue channels. Subsequently, the average value of all pixel values ​​in the generated RGB difference image is determined. The calculated average value is used as the motion state feature, thus effectively characterizing the overall difference between corresponding pixels in two frames within a specified one-second interval.

[0048] (10) Video acoustic event features: The video content is identified using a multimodal large language model, and the audio signal is identified using acoustic event detection technology. The identification results are aligned at the text description level, and the audio features of the identified acoustic events are converted into frequency domain Mel features, and the text descriptions are converted into text features. For example, acoustic events can be "car engine sound", "bird call", "dog bark", etc.

[0049] In step S13, image frame features, video semantic features, and audio features are used as the first input parameters of the audio model, and image frame features and audio features are used as the second input parameters of the audio model to obtain the first background audio.

[0050] In this disclosure, image frame features, video semantic features, and audio features are jointly used as the first input parameters of the audio model, and image frame features and audio features are used as the second input parameters to train the audio model. That is, during training, labeled ground truth values ​​(i.e., audio features of the sample background audio) are input into the model as guiding features for training. This results in a high degree of matching between the background audio generated by the trained audio model and the video, improving the model's generalization ability and robustness. Furthermore, since video semantic features are considered during training, the resulting background audio model can generate matching background audio based on video semantics, further improving the matching degree between the generated background audio and the video.

[0051] In step S14, the audio model is trained based on at least the first background audio and the sample background audio to obtain the background audio model.

[0052] In this disclosure, the error between the first background audio and the sample background audio can be determined, and an audio model can be trained based on this error to obtain a background audio model upon completion of training. For example, the MSE loss function between the first background audio and the sample background audio can be calculated as the error between the two. Training is considered complete when this error is less than an error threshold, at which point the background audio model is obtained.

[0053] By employing the above technical solution, image frame features, video semantic features, and audio features are jointly used as the first input parameter of the model, while image frame features and audio features are used as the second input parameter of the audio model. This means that during training, the audio features of the sample background audio are also input into the model as guiding features for training. Thus, the background audio generated by the trained background audio model has a high degree of matching with the video, improving the model's generalization ability and robustness. Furthermore, since video semantic features are considered during training, the resulting background audio model can generate matching background audio based on video semantics, further improving the matching degree between the generated background audio and the video, thereby enhancing the user's entertainment experience.

[0054] In one embodiment, the background audio model may further include a video content parsing model. Accordingly, obtaining the video semantic features of the sample synthesized video may include: inputting the sample synthesized video into the video content parsing model to obtain the video semantic features of the sample synthesized video.

[0055] The video content parsing model can be a multimodal large language model (Multi-LLM). This model can be pre-trained or untrained. When the video content parsing model is pre-trained, it does not need to be trained again during the background audio model generation process. When the video content parsing model is untrained, it can be trained during the background audio model generation process. To simplify the background audio model generation process, in this disclosure, the video content parsing model can be pre-trained, thus eliminating the need for further training during the background audio model generation process.

[0056] For example, a specific implementation method for obtaining the video semantic features of a sample synthesized video by inputting the sample synthesized video into a video content parsing model can be as follows: input the sample synthesized video into the video content parsing model to obtain the text encoding features output by the video content parsing model; and obtain the video semantic features of the sample synthesized video based on the text encoding features and a preset semantic feature extractor.

[0057] Figure 2 This is a schematic diagram illustrating the generation of a background audio model according to an exemplary embodiment. For example... Figure 2As shown, the background audio model can include not only an audio model but also a video content parsing model. First, the sample synthesized video is input into the video content parsing model to obtain the text-encoded features output by the model. These text-encoded features can be token sequence encodings. Then, the text-encoded features are input into a pre-defined semantic feature extractor to obtain the video semantic features of the sample video.

[0058] For example, the preset semantic feature extractor can be a CLIP model, in which encoding is performed using CLIP text encoding. Since the later feature layers are more detailed, the output features of the last two CLIP layers can be retained, and these output features can be concatenated between channels to obtain the video semantic features of the sample synthesized video.

[0059] Then, the image frame features, video semantic features, and audio features are used as the first input parameters of the audio model, and the image frame features and audio features are used as the second input parameters of the audio model. The first background audio is obtained by inputting these into the audio model.

[0060] Finally, the audio model can be trained based on the first background audio and the sample background audio to obtain the background audio model.

[0061] In one implementation, the audio model can be trained based on the error between the first background audio and the sample background audio to obtain the background audio model.

[0062] In another implementation, training the audio model based at least on the first background audio and sample background audio to obtain a background audio model may include: generating a first synthesized video based on the first background audio and sample video; and training the audio model based on the first background audio and sample background audio, the first synthesized video and sample synthesized video to obtain the background audio model.

[0063] In this embodiment, the audio model can be trained based on the first background audio and sample video, the first synthesized video and sample synthesized video. In this way, the model can be trained with reference to more parameters, which improves the robustness and reliability of the model.

[0064] In one possible approach, training an audio model based on a first background audio and sample background audio, a first synthesized video and sample synthesized videos to obtain a background audio model includes: inputting the first synthesized video into a video content parsing model to obtain the video semantic features of the first synthesized video; and training the audio model based on the first background audio and sample background audio, the video semantic features of the first synthesized video and the video semantic features of the sample synthesized videos to obtain the background audio model.

[0065] It should be understood that the video semantic features of the first synthesized video can be obtained by referring to the specific implementation method for obtaining the video semantic features of the sample synthesized video described above, and this disclosure will not repeat it here.

[0066] For example, a background audio model is obtained by training an audio model based on the error between the first background audio and the sample background audio, and the error between the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video. For instance, the model error can be determined based on the error between the first background audio and the sample background audio, the error between the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video, and their respective error weights. The parameters of the audio model are then adjusted based on this model error, and the background audio model is obtained when the model error is less than a preset error.

[0067] It should be understood that, with reference Figure 2 Alternatively, the first synthesized video can be input into a video content parsing model to obtain the text encoding features of the first synthesized video. Based on the first background audio, sample background audio, the text encoding features of the first synthesized video, and the text encoding features of the sample synthesized video, an audio model can be trained to obtain a background audio model. This disclosure does not impose specific limitations on this.

[0068] By employing the aforementioned technical solution, during model training, in addition to considering the error between the first background audio output by the audio model and the sample background audio, the error between the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video is also taken into account. That is, the background audio model is generated by comprehensively considering both audio loss and video content loss. Thus, generating the background audio model based on audio loss and video content loss ensures that the semantic content of the synthesized video generated using this background audio model is more closely matched to the semantic content of the sample synthesized video, particularly in terms of harmony, rhythm, and loudness matching. Furthermore, the background audio model not only successfully generates music that matches the emotional tone of the video but also maintains high music quality, improving the ability to coordinate music and video content.

[0069] In another embodiment, training the audio model based at least on the first background audio and the sample background audio to obtain the background audio model may include: inputting the first background audio into a preset emotion encoder to obtain a first result, inputting the sample background audio into the emotion encoder to obtain a second result; training the audio model based on the first background audio, the sample background audio, the first result, and the second result to obtain the background audio model; or, training the audio model based on the first result and the second result to obtain the background audio model.

[0070] For example, the first background audio and the sample background audio can be input into a preset Emotion Encoder to obtain embedding vectors of the same output dimension, which are denoted as the first result and the second result, respectively.

[0071] In one implementation, the audio model can be trained based solely on the error between the first result and the second result to obtain the background audio model. In another implementation, the audio model can be trained based on the error between the first background audio and the sample background audio, as well as the error between the first result and the second result, to obtain the background audio model.

[0072] By adopting the above technical solution, the loss of audio emotion is taken into account when generating the background audio model, which improves the accuracy of the generated background audio model in predicting background audio.

[0073] In another embodiment, in conjunction with the above embodiments, the audio model can be trained based on the error between the first background audio and the sample background audio, the error between the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video, and the error between the first result and the second result, to obtain a background audio model. Thus, by comprehensively generating background audio from the aspects of sentiment loss, content loss, and audio loss, the accuracy of the generated background audio model in predicting background audio is further improved.

[0074] In this disclosure, the background audio model may further include a multimodal model; for example, the multimodal model may be a multimodal Transformer model. Accordingly, the generation method may further include: inputting image frame features and audio features into the multimodal model to obtain a first sample fusion feature output by the multimodal model; obtaining a second sample fusion feature based on the first sample fusion feature and video semantic features; and inputting the second sample fusion feature as a first input parameter of the audio model, and the first sample fusion feature as a second input parameter of the audio model, to obtain the first background audio.

[0075] Reference Figure 2The background audio model can also include a multimodal Transformer model. First, image frame features are extracted from the sample video to obtain image frame features, and audio features are extracted from the sample background audio to obtain audio features. Specifically, after extracting image frame features from the sample video, the image frame features are concatenated into a two-dimensional sequence, and a fully connected layer is applied to create the final image frame embedding vector (denoted as the image frame feature), which is subsequently input into the encoder of the multimodal Transformer model. After extracting audio features, a comprehensive audio embedding vector (denoted as the audio feature) is formed, which is subsequently input into the decoder of the multimodal Transformer model.

[0076] For example, such as Figure 2 As shown, the background audio model can also include an image frame feature extraction model and an audio feature extraction model. The image frame feature extraction model is used to extract image frame features from the sample video, while the audio feature extraction model is used to extract audio features from the sample background audio.

[0077] Both the image frame feature extraction model and the audio feature extraction model are connected to the multimodal Transformer model. After obtaining the image frame features and audio features, the extracted features are input into the multimodal Transformer model. The multimodal Transformer model generates adaptive fusion features based on the image frame features and audio features. This part consists of two basic components: an encoder that uses the image frame features as conditional factors; and a decoder that extracts input features related to chords, pitch, spectrum, acoustic events, etc., from the audio features during training. Under the adjustment of the encoder, these features are ultimately fused together, that is, the image frame features and audio features are fused together to obtain the first sample fusion feature that matches the input sample video. This achieves the goal of embedded fusion of image frame features and audio features.

[0078] For example, audio features can be denoted as Where En represents the dimension operation of converting Linear to BCWH. Characterizing chord features, Characterizing the density features of musical notes, Characterizing tonal features, Characterizing loudness features, Characterizing the features of audio acoustic events. Image frame features can be denoted as... ,in, Characterizes the features of scene changes. Characterizing the features of motion state, Representing emotional characteristics, Characterize textual descriptive features, Characterize the acoustic event features of a video.

[0079] Audio features and image frame features are input into a multimodal Transformer model, which then fuses them to obtain the first sample fused features.

[0080] Reference Figure 2 The background audio model also includes a feature fusion unit, which fuses the first sample fused features and video semantic features to obtain the second sample fused features. For example, the feature fusion unit concatenates the video semantic features and the channel dimensions of the first sample fused features output by the multimodal Transformer model, embeds and fuses the concatenated features using an MLP, and then downsamples the fused features using a down-sampling module (not shown) to obtain the second sample fused features.

[0081] For example, the second sample fusion feature can be denoted as ,in, Characterize the semantic features of the video.

[0082] After obtaining the first sample fusion feature and the second sample fusion feature in the manner described above, the first sample fusion feature and the second sample fusion feature are used as the first input parameter and the second input parameter of the audio model, respectively. For example, as shown... Figure 2 As shown, the audio model can be an audio diffusion model. The second sample fusion feature can be the initial feature of the audio model, with feature dimensions (B, C, M, N), where B is the number of features, C is the fusion channel, and M and N represent the length and width of the feature.

[0083] Accordingly, when the background audio model also includes a multimodal model, in addition to training the audio model, the multimodal model can also be trained when generating the background audio model to improve the efficiency of feature fusion. For example, the audio model and the multimodal model can be trained based on the first background audio and the sample background audio to obtain the background audio model.

[0084] Similarly, as described above, the audio model and the multimodal model can be trained based on the first background audio and the sample background audio, the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video, the first result and the second result to obtain the background audio model. For example, the audio model and the multimodal model can be trained based on the error between the first background audio and the sample background audio, the error between the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video, and the error between the first result and the second result to obtain the background audio model.

[0085] By adopting the above technical solution, in the process of generating the background audio model, in addition to training the audio model included in the background audio model, a multimodal model is also trained, thereby improving the reliability and accuracy of the generated background audio model.

[0086] For the audio diffusion model, the structure of the open-source technology solution Stable Audio 1.0 was referenced, and the number of cross-attention mechanisms in the audio diffusion model was expanded. The calculation method of the cross-attention mechanism is shown in formula (1): (1) Where Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The representation decodes the query matrix.

[0087] For the diffusion decoder part, the text feature introduction method of stable diffusion (SD) is modified to be the input of fused features. That is, the first sample fused feature of the image frame features and audio features is input into the diffusion decoder part of the audio diffusion model.

[0088] For example, the fused features of the first sample can be directly input into the diffusion decoding part of the audio diffusion model. As another example, such as... Figure 2 As shown, the background audio model may further include a feature enhancement unit. Before inputting the first sample fused features into the diffusion decoding part of the audio diffusion model, the feature enhancement unit enhances the first sample fused features to obtain the fused features of the feature dimensions required by each network layer in the diffusion decoding part. The feature enhancement unit can be a ResNet structure, thus effectively preserving features without causing loss or distortion of features in the network's deepening.

[0089] In this disclosure, an audio diffusion model is used to model audio. To output audio features, the audio diffusion model takes a second sample fusion feature as input, a first sample fusion feature as guidance feature, and outputs a single-frame audio chroma map with features (B, 1, K, F), where 1 represents a channel, K represents the number of training samples in each training round, and F represents the bin in the frequency domain. During training, the first sample fusion is used as the guidance feature of the model. In this way, the background audio generated by the trained audio model has a high degree of matching with the video, improving the model's generalization ability and robustness, thereby enhancing the user's entertainment experience.

[0090] It should be understood that when the audio model is an audio diffusion model, this model progressively denoises the fused features of the second sample. Furthermore, the audio diffusion model outputs an audio chroma map. To convert this audio chroma map into the first background audio, such as... Figure 2 As shown, the background audio model may also include a decoder connected to the audio model to decode the audio chroma map output by the audio model to obtain the first background audio.

[0091] In this disclosure, the training sample set may include training samples from multiple training rounds. The training samples for each training round are typically determined by the computational performance of the audio model. For example, if the audio model can process features from 5 frames of video simultaneously each time, then the training samples for each training round may include a synthesized video from five frames of samples.

[0092] Using image frame features, video semantic features, and audio features as the first input parameters of the audio model, and using image frame features and audio features as the second input parameters of the audio model, to obtain the first background audio, may include: using the image frame features, video semantic features, audio features of the training samples in the current training round, and the first background audio obtained in the previous training round as the first input parameters of the audio model in the current training round, and using the image frame features and audio features of the training samples in the current training round as the second input parameters of the audio model in the current training round, to obtain the first background audio of the current training round.

[0093] In this disclosure, the audio model's role is to achieve rapid denoising and generation using fused features. It generates the audio chroma map of the training samples for the current training round through a single-step generation method, and obtains the first background audio after decoding by the decoder. For example, assuming each training round's training samples include five frames of synthesized video, the first background audio from the first to fifth frames of the synthesized video can be obtained in the first training round. In the second training round, the image frame features, video semantic features, and audio features of the sixth to tenth frames of the synthesized video, along with the first background audio from the first to fifth frames, can be fused and used as the first input parameter for the second training round of the audio model. The image frame features and audio features of the sixth to tenth frames of the synthesized video are then used as the second input parameter for the second training round of the audio model to obtain the first background audio for the second training round. This process continues until the output first background audio meets the required audio length.

[0094] For example, the calculation formula for the audio diffusion model is shown in formula (2): (2) in, Characterizing the first i The first background audio of the training round. Characterizing the first i-1 The first background audio of the training round. i The value range is [2, N], where N is the total number of training rounds.

[0095] In one embodiment, the audio features may include note density features and / or loudness features, wherein the specific methods for obtaining the note density features and / or loudness features have been described above and will not be repeated here. Figure 2 As shown, the background audio model may further include an audio feature prediction model and a processing unit. Accordingly, training the audio model based at least on the first background audio and the sample background audio to obtain the background audio model may include: obtaining a pre-trained audio feature prediction model, which is trained by using the sample background audio as input parameters and the note density features and / or loudness features of the sample background audio as output parameters; inputting the first background audio into the pre-trained audio feature prediction model to obtain predicted note density features and / or loudness features; processing the first background audio based on the predicted note density features and / or loudness features through the processing unit to obtain a second background audio; and training the audio model and the audio feature prediction model based at least on the second background audio and the sample background audio to obtain the background audio model.

[0096] To ensure better alignment and interplay between the generated background audio and video, the first background audio output by the audio model can be post-processed or fine-tuned to obtain more expressive music output that better matches the video.

[0097] In this embodiment, an LSTM regression model can be pre-trained by using sample background audio as input parameters and the note density and / or loudness features of the sample background audio as output parameters. Then, the first background audio is input into the pre-trained audio feature prediction model to obtain predicted note density and / or loudness features. These predicted note density and / or loudness features are subsequently used to determine the tempo adjustment for chord arpeggio patterns. The tempo adjustment for chord arpeggio patterns produces music with subtle rhythmic variations and dynamic intensity to synchronize the background audio with the mood and rhythm of the video.

[0098] In this embodiment, to make the generated background audio rhythmically interesting, the generated chords are arpeggiated. Arpeggiations expand the notes of the chords over time, and their patterns may repeat, typically in a progressively upward or downward sequence. This step adds rhythm to the audio, aligning it with the rhythm of the video.

[0099] Finally, to synchronize the audio volume with the emotional intensity and visual dynamics of the video, a velocity parameter can be obtained based on the predefined correspondence between note density features and / or loudness features and the velocity parameter (finding areas with higher frequency energy and incrementing the velocity by 0.2). This velocity parameter controls the perceived loudness of notes in the audio. By establishing a link between loudness levels and video features, the music can respond by becoming louder as the video becomes more intense, and by adopting a softer style as the video becomes calmer.

[0100] In this embodiment, training the audio model and the audio feature prediction model based at least on the second background audio and the sample background audio to obtain the background audio model may further include: Based on the second background audio and the sample video, a second synthesized video is obtained, and the second synthesized video is input into a video content parsing model to obtain the video semantic features of the second synthesized video. A second error is determined between the video semantic features of the second synthesized video and the video semantic features of the sample synthesized video; and / or The second background audio is input into the preset emotion encoder to obtain the third result, the sample background audio is input into the emotion encoder to obtain the second result, and the third error between the second result and the third result is determined; Determine the first error between the sample background audio and the second background audio; The audio model and the audio feature prediction model are trained based on the first error and the target error to obtain the background audio model. The target error includes the second error and / or the third error.

[0101] Reference Figure 2 In one implementation, a second synthesized video is obtained based on the second background audio and the sample video. This second synthesized video is then input into a video content parsing model to obtain the video semantic features of the second synthesized video. A second error, Loss2, is then determined between the video semantic features of the second synthesized video and the video semantic features of the sample synthesized video. The second error, Loss2, represents the content loss of the synthesized video.

[0102] Accordingly, the audio model and the audio feature prediction model can be trained based on the first error Loss1 and the second error Loss2 to obtain the background audio model.

[0103] In another embodiment, a second background audio is input into a preset emotion encoder to obtain a third result, and a sample background audio is input into the emotion encoder to obtain a second result. A third error, Loss3, is then determined between the second and third results. Here, the third error, Loss3, characterizes the content loss of the synthesized video.

[0104] It should be understood that an emotion encoder is needed in the process of generating the background audio model, but it is not needed in the inference process of the background audio model.

[0105] Accordingly, the audio model and the audio feature prediction model can be trained based on the first error Loss1 and the third error Loss3 to obtain the background audio model.

[0106] In another embodiment, the second error Loss2 and the third error Loss3 can be determined respectively in the manner described above. Accordingly, the audio model and the audio feature prediction model are trained based on the first error Loss1, the second error Loss2, and the third error Loss3 to obtain the background audio model.

[0107] It should be understood that, such as Figure 2 As shown, when the background audio model also includes a multimodal model, the audio model, multimodal model, and audio feature prediction model can be trained based on the first error Loss1, the second error Loss2, and the third error Loss3 to obtain the background audio model.

[0108] For example, assuming the error weights of the first error Loss1, the second error Loss2, and the third error Loss3 are 0.5, 0.2, and 0.3 respectively, the error function of the background audio model can be expressed as follows: .

[0109] The above technical solution was used to generate a background audio model, realizing the joint modeling of video and background audio.

[0110] Figure 3 This is a flowchart illustrating a background audio generation method according to an exemplary embodiment. Figure 3 As shown, the background audio generation method may include the following steps.

[0111] In step S31, the video to be processed and the audio are acquired.

[0112] Since the video to be processed does not include background audio in the background audio generation method, it is necessary to determine the initial audio. For example, the user could select an audio from a set of preset audio files as the initial audio. Alternatively, the user could not input an initial audio, and it could be randomly generated based on the video to be processed.

[0113] In step S32, the video semantic features, image frame features, and audio features of the initial audio of the video to be processed are obtained respectively.

[0114] The specific methods for obtaining the video semantic features of the video to be processed, the image features to initialize the audio features of the audio, and the methods for obtaining the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio described in the background audio model generation method are the same and will not be repeated here.

[0115] In step S33, the video semantic features, image frame features, and audio features of the initial audio of the video to be processed are used as the first input parameters of the audio model in the background audio model, and the image features of the video to be processed and the audio features of the initial audio are used as the second input parameters of the audio model to obtain the third background audio.

[0116] The background audio model is generated according to the background audio model generation method provided in this disclosure.

[0117] By adopting the above technical solution, background audio that matches the emotional tone of the video to be processed can be generated, thereby enhancing the user's entertainment experience.

[0118] In addition, the background audio generation method may further include: synthesizing the third background audio and the video to be processed to obtain a third synthesized video configured with the third background audio.

[0119] In this way, the goal of configuring a third background audio for the video to be processed is achieved, satisfying the user's entertainment needs.

[0120] Figure 4This is a schematic diagram illustrating a background audio generation method according to an exemplary embodiment. Figure 4 As shown, the video to be processed and the initial audio are input into the background audio model.

[0121] In the background audio model, video semantic features, image frame features, and audio features of the initial audio are obtained from the video to be processed. For example, the video to be processed is input into a video content parsing model to obtain text-encoded features output by the model. These text-encoded features are then processed to obtain video semantic features. Furthermore, image frame features are extracted from the video to be processed, and audio features are extracted from the initial audio. The audio features and image frame features are then input into a multimodal model to obtain a first fused feature. The first fused feature and video semantic features are input into a feature fusion unit to obtain a second fused feature. This second fused feature serves as the first input parameter of the audio model. The first fused feature is then input into a feature enhancement unit to obtain an enhanced first fused feature. This enhanced first fused feature serves as the second input parameter of the audio model to obtain an audio chroma map output by the audio model. The audio chroma map is then input into a decoder to obtain the third background audio. Finally, the third background audio is input into the audio feature prediction model to obtain predicted note density features and / or loudness features. These predicted features are then input into the processing unit to obtain velocity parameters. Based on these velocity parameters, the third background audio is processed to obtain the fourth background audio. Finally, the fourth background audio is synthesized with the video to be processed to obtain a third composite video configured with the fourth background audio.

[0122] Based on the same inventive concept, this disclosure also provides a background audio model generation apparatus. Figure 5 This is a block diagram illustrating a background audio model generation apparatus according to an exemplary embodiment. Figure 5 As shown, the background audio model generation device 500 may include: The first acquisition module 501 is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample synthesized video, and the sample synthesized video includes a sample video and sample background audio configured for the sample video. The second acquisition module 502 is configured to acquire the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio, respectively. The third acquisition module 503 is configured to use the image frame features, the video semantic features and the audio features as the first input parameters of the audio model, and use the image frame features and the audio features as the second input parameters of the audio model to input the audio model to obtain the first background audio. Training module 504 is configured to train the audio model based on at least the first background audio and the sample background audio to obtain a background audio model.

[0123] Optionally, the background audio model further includes a video content parsing model, and the second acquisition module 502 is configured to obtain the video semantic features of the sample synthesized video by inputting the sample synthesized video into the video content parsing model.

[0124] Optionally, the training module 504 may include: The first generation submodule is configured to generate a first synthesized video based on the first background audio and the sample video; The first training submodule is configured to train the audio model based on the first background audio and the sample background audio, the first synthesized video and the sample synthesized video, to obtain a background audio model.

[0125] Optionally, the first training submodule is configured to: input the first synthesized video into the video content parsing model to obtain the video semantic features of the first synthesized video; and train the audio model based on the first background audio and the sample background audio, the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video to obtain a background audio model.

[0126] Optionally, the training module 504 may include: The first acquisition submodule is configured to input the first background audio into a preset emotion encoder to obtain a first result, and input the sample background audio into the emotion encoder to obtain a second result; The second training submodule is configured to train the audio model based on the first background audio and the sample background audio, the first result and the second result to obtain a background audio model; or, to train the audio model based on the first result and the second result to obtain a background audio model.

[0127] Optionally, the audio features include note density features and / or loudness features, and the background audio model further includes an audio feature prediction model and a processing unit; the training module 504 may include: The second acquisition submodule is configured to acquire a pre-trained audio feature prediction model, which is trained by using the sample background audio as the input parameters of the audio feature prediction model and the note density features and / or loudness features of the sample background audio as the output parameters of the audio feature prediction model. The third acquisition submodule is configured to input the first background audio into a pre-trained audio feature prediction model to obtain predicted note density features and / or loudness features. The fourth acquisition submodule is configured to process the first background audio according to the predicted note density features and / or loudness features through the processing unit to obtain the second background audio; The third training submodule is configured to train the audio model and the audio feature prediction model based at least on the second background audio and the sample background audio to obtain a background audio model.

[0128] Optionally, the third training submodule is configured as follows: Based on the second background audio and the sample video, a second synthesized video is obtained, and the second synthesized video is input into the video content parsing model to obtain the video semantic features of the second synthesized video, and a second error between the video semantic features of the second synthesized video and the video semantic features of the sample synthesized video is determined; and / or The second background audio is input into a preset emotion encoder to obtain a third result, the sample background audio is input into the emotion encoder to obtain a second result, and a third error between the second result and the third result is determined; Determine the first error between the sample background audio and the second background audio; The audio model and the audio feature prediction model are trained based on the first error and the target error to obtain a background audio model, wherein the target error includes the second error and / or the third error.

[0129] Optionally, the background audio model further includes a multimodal model, and the background audio model generation device 500 may include: The fourth acquisition module is configured to input the image frame features and the audio features into the multimodal model to obtain the first sample fusion features output by the multimodal model; The fifth acquisition module is configured to obtain the second sample fusion feature based on the first sample fusion feature and the video semantic feature; The third acquisition module 503 is configured to: use the second sample fusion feature as the first input parameter of the audio model, use the first sample fusion feature as the second input parameter of the audio model, and input the audio model to obtain the first background audio.

[0130] Optionally, the training module 504 includes: The fourth training submodule is configured to train the audio model and the multimodal model based at least on the first background audio and the sample background audio to obtain a background audio model.

[0131] Optionally, the training sample set includes training samples from multiple training rounds. The third acquisition module 503 is configured to: use the image frame features, video semantic features, audio features, and the first background audio obtained from the previous training round of the training samples as the first input parameters of the current training round of the audio model; use the image frame features and audio features of the training samples from the current training round as the second input parameters of the current training round of the audio model; and input these into the audio model to obtain the first background audio of the current training round.

[0132] Optionally, the second acquisition module 502 is configured to: input the sample synthesized video into a video content parsing model to obtain the text encoding features output by the video content parsing model; Based on the text encoding features and the preset semantic feature extractor, the video semantic features of the sample synthesized video are obtained.

[0133] Optionally, the image frame features include at least one of the following: text description features, emotion features, scene change features, motion state features, and video acoustic event features; The audio features include at least one of the following: note density features, loudness features, chord features, pitch features, and audio acoustic event features.

[0134] Figure 6 This is a block diagram illustrating a background audio generation apparatus according to an exemplary embodiment. Figure 6 As shown, the background audio generation device 600 may include: The sixth acquisition module 601 is configured to acquire the video to be processed and initialize the audio; The seventh acquisition module 602 is configured to acquire the video semantic features, image features, and audio features of the initial audio of the video to be processed, respectively; The eighth acquisition module 603 is configured to use the video semantic features, image frame features, and audio features of the initial audio of the video to be processed as the first input parameters of the audio model in the background audio model, and use the image frame features of the video to be processed and the audio features of the initial audio of the video to be processed as the second input parameters of the audio model to input the audio model to obtain the third background audio. The background audio model is generated according to the background audio model generation method provided in this disclosure, and the background audio model includes the audio model.

[0135] Optionally, the background audio generation device 600 may include: The compositing module is configured to combine the third background audio and the video to be processed to obtain a third composite video containing the third background audio.

[0136] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0137] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the background audio model generation method provided in this disclosure, and / or implement the steps of the background audio generation method provided in this disclosure.

[0138] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0139] Reference Figure 7 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812, sensor component 814, and communication component 816.

[0140] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0141] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0142] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0143] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0144] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0145] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0146] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0147] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0148] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0150] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the background audio model generation method and / or background audio generation method described above when executed by the programmable device.

[0151] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.

[0152] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0153] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0154] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0155] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0156] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating a background audio model, the method comprising: The background audio model includes an audio model, and the generation method includes: Obtain a training sample set, wherein the training samples in the training sample set include sample synthesized videos, and the sample synthesized videos include sample videos and sample background audio configured for the sample videos; The image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio are obtained respectively. The image frame features, the video semantic features, and the audio features are used as the first input parameters of the audio model, and the image frame features and the audio features are used as the second input parameters of the audio model, which are then input into the audio model to obtain the first background audio. The audio model is trained based on at least the first background audio and the sample background audio to obtain a background audio model.

2. The generation method of claim 1, wherein, The background audio model also includes a video content parsing model; obtaining the video semantic features of the sample synthesized video includes: The video semantic features of the sample synthesized video are obtained by inputting the sample synthesized video into the video content parsing model.

3. The generation method of claim 2, wherein, The step of training the audio model based at least on the first background audio and the sample background audio to obtain a background audio model includes: A first synthesized video is generated based on the first background audio and the sample video; The audio model is trained based on the first background audio and the sample background audio, the first synthesized video and the sample synthesized video to obtain the background audio model.

4. The generation method of claim 3, wherein, The step of training the audio model based on the first background audio and the sample background audio, the first synthesized video and the sample synthesized video to obtain a background audio model includes: The first synthesized video is input into the video content parsing model to obtain the video semantic features of the first synthesized video. The audio model is trained based on the first background audio and the sample background audio, the video semantic features of the first synthesized video and the video semantic features of the sample synthesized video to obtain the background audio model.

5. The generation method of claim 1, wherein, The step of training the audio model based at least on the first background audio and the sample background audio to obtain a background audio model includes: The first background audio is input into a preset emotion encoder to obtain a first result, and the sample background audio is input into the emotion encoder to obtain a second result; Based on the first background audio and the sample background audio, the first result and the second result, the audio model is trained to obtain a background audio model; or Based on the first result and the second result, the audio model is trained to obtain a background audio model.

6. The generation method of claim 1, wherein, The audio features include note density features and / or loudness features, and the background audio model further includes an audio feature prediction model and a processing unit; the step of training the audio model based at least on the first background audio and the sample background audio to obtain the background audio model includes: A pre-trained audio feature prediction model is obtained by training the audio feature prediction model by using the sample background audio as the input parameters of the audio feature prediction model and the note density features and / or loudness features of the sample background audio as the output parameters of the audio feature prediction model. The first background audio is input into a pre-trained audio feature prediction model to obtain predicted note density features and / or loudness features. The processing unit processes the first background audio based on the predicted note density and / or loudness characteristics to obtain the second background audio. The audio model and the audio feature prediction model are trained based on at least the second background audio and the sample background audio to obtain the background audio model.

7. The generation method according to claim 6, characterized in that, The step of training the audio model and the audio feature prediction model based at least on the second background audio and the sample background audio to obtain the background audio model includes: Based on the second background audio and the sample video, a second synthesized video is obtained, and the second synthesized video is input into the video content parsing model to obtain the video semantic features of the second synthesized video, and a second error between the video semantic features of the second synthesized video and the video semantic features of the sample synthesized video is determined; and / or The second background audio is input into a preset emotion encoder to obtain a third result, the sample background audio is input into the emotion encoder to obtain a second result, and a third error between the second result and the third result is determined; Determine the first error between the sample background audio and the second background audio; The audio model and the audio feature prediction model are trained based on the first error and the target error to obtain a background audio model, wherein the target error includes the second error and / or the third error.

8. The generation method according to claim 1, characterized in that, The background audio model further includes a multimodal model, and the generation method further includes: The image frame features and the audio features are input into the multimodal model to obtain the first sample fusion features output by the multimodal model; The second sample fusion feature is obtained based on the first sample fusion feature and the video semantic feature; The step of using the image frame features, the video semantic features, and the audio features as the first input parameters of the audio model, and using the image frame features and the audio features as the second input parameters of the audio model, to input the audio model to obtain the first background audio includes: The second sample fusion feature is used as the first input parameter of the audio model, and the first sample fusion feature is used as the second input parameter of the audio model. The audio model is then input to obtain the first background audio.

9. The generation method of claim 8, wherein, The step of training the audio model based at least on the first background audio and the sample background audio to obtain a background audio model includes: The audio model and the multimodal model are trained based on at least the first background audio and the sample background audio to obtain the background audio model.

10. The method of generating according to any one of claims 1-9, wherein, The training sample set includes training samples from multiple training rounds. The first input parameter of the audio model is the image frame feature, the video semantic feature, and the audio feature; the second input parameter of the audio model is the image frame feature and the audio feature. This input is then used to obtain the first background audio, including: The image frame features of the training samples in the current training round, the video semantic features, the audio features, and the first background audio obtained in the previous training round are used as the first input parameters of the audio model in the current training round. The image frame features and the audio features of the training samples in the current training round are used as the second input parameters of the audio model in the current training round. These are then input into the audio model to obtain the first background audio of the current training round.

11. The method of generating of claim 2, wherein, The step of inputting the sample synthesized video into the video content parsing model to obtain the video semantic features of the sample synthesized video includes: The sample-synthesized video is input into the video content parsing model to obtain the text encoding features output by the video content parsing model; Based on the text encoding features and the preset semantic feature extractor, the video semantic features of the sample synthesized video are obtained.

12. The method of generating according to any one of claims 1-9, wherein, The image frame features include at least one of the following: text description features, emotion features, scene change features, motion state features, and video acoustic event features; The audio features include at least one of the following: note density features, loudness features, chord features, pitch features, and audio acoustic event features.

13. A background audio generation method, characterized by, The background audio generation method includes: Get the video to be processed and initialize the audio; The video semantic features, image features, and audio features of the initial audio of the video to be processed are obtained respectively; The video semantic features, image frame features, and audio features of the initial audio of the video to be processed are used as the first input parameters of the audio model in the background audio model, and the image frame features and audio features of the initial audio of the video to be processed are used as the second input parameters of the audio model. The audio model is then input to obtain the third background audio. The background audio model is generated according to the background audio model generation method as described in any one of claims 1-12.

14. The background audio generation method of claim 13, wherein, The background audio generation method further includes: The third background audio and the video to be processed are combined to obtain a third composite video configured with the third background audio.

15. An apparatus for generating a background audio model, the apparatus comprising: The background audio model includes an audio model, and the generation device includes: The first acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample synthesized video, and the sample synthesized video includes a sample video and sample background audio configured for the sample video; The first acquisition module is configured to acquire the image frame features of the sample video, the video semantic features of the sample synthesized video, and the audio features of the sample background audio, respectively. The third acquisition module is configured to use the image frame features, the video semantic features, and the audio features as the first input parameters of the audio model, and use the image frame features and the audio features as the second input parameters of the audio model to input the audio model to obtain the first background audio. The training module is configured to train the audio model based on at least the first background audio and the sample background audio to obtain a background audio model.

16. A background audio generation apparatus, characterized by comprising: The background audio generation device includes: The fourth acquisition module is configured to acquire the video to be processed and initialize the audio; The fifth acquisition module is configured to acquire the video semantic features, image features, and audio features of the initial audio of the video to be processed, respectively. The sixth acquisition module is configured to use the video semantic features, image features, and audio features of the initial audio of the video to be processed as the first input parameters of the audio model in the background audio model, and use the image features of the video to be processed and the audio features of the initial audio of the video to be processed as the second input parameters of the audio model, and input them into the audio model to obtain the third background audio; The background audio model is generated according to the background audio model generation method as described in any one of claims 1-12.

17. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the background audio model generation method according to any one of claims 1-12 when executing the instructions, and / or to implement the steps of the background audio generation method according to claim 13 or 14.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method for generating a background audio model according to any one of claims 1-12, and / or implements the steps of the method for generating a background audio model according to claim 13 or 14.

19. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the steps of the method for generating a background audio model according to any one of claims 1-12, and / or implements the steps of the method for generating a background audio model according to claim 13 or 14.

Images