Video processing method, apparatus, device, computer readable storage medium and computer program product

Abstract

The application provides a video processing method, device and equipment, computer readable storage medium and computer program product; the method comprises: carrying out first encoding processing on each video frame in a to-be-processed video through a video encoder to obtain a frame embedding feature vector of each video frame; carrying out embedding level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector used for representing dynamic change information of the to-be-processed video; carrying out second encoding processing on each video frame in the to-be-processed video through the video encoder to obtain a static semantic condition vector; carrying out iterative denoising generation based on the static semantic condition vector and the dynamic change vector through a diffusion generation model to obtain an audio latent representation vector; and carrying out decoding processing on the audio latent representation vector to obtain generated audio corresponding to the to-be-processed video. Through the application, the matching accuracy and dynamic response capability of the generated audio in time sequence with the to-be-processed video can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, a video processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of multimodal generative models, video-to-audio (V2A) generation has become an important research direction for realizing the collaborative expression of audiovisual content. This technology aims to automatically generate audio signals that are semantically and temporally consistent with the visual content of the input video, and is widely used in film and television production, virtual reality, and accessibility assistance.

[0003] In related technologies, cross-modal pre-trained models such as Contrastive Language–Image Pre-training (CLIP) are commonly used to extract high-level semantic information from video frames, which is then used as a condition to guide audio generation. Furthermore, to enhance temporal alignment, some methods introduce additional audio prediction features or temporal detection modules, such as CAVP or Synchformer, to extract more refined temporal information.

[0004] However, these solutions typically rely on additional network structures or independent training processes, increasing system complexity and computational overhead. Furthermore, these techniques struggle to effectively utilize the video's own temporal dynamics to improve the temporal synchronization of audio generation without adding extra networks or training phases. This results in discrepancies between the generated audio and the rhythm of the video actions, affecting overall audiovisual consistency. Summary of the Invention

[0005] This application provides a video processing method, apparatus, device, computer-readable storage medium, and computer program product. By extracting the temporal dynamic change information of the video through embedded-level differential processing, and introducing the embedded-level dynamic change information as a dynamic auxiliary condition, it can more accurately reflect the temporal dimension information of the video, thereby significantly improving the timing matching accuracy and dynamic response capability of the generated audio with the video to be processed, and ensuring the overall audio-visual consistency between the generated audio and the video to be processed.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a video processing method, comprising: performing a first encoding process on each video frame in a video to be processed using a video encoder to obtain a frame embedding feature vector for each video frame; the video to be processed is a silent video without audio information; performing embedding-level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector representing the dynamic change information of the video to be processed; performing a second encoding process on each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; performing iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and decoding the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0008] This application provides a video processing apparatus, comprising: an encoding module, configured to perform a first encoding process on each video frame in a video to be processed using a video encoder to obtain a frame embedding feature vector for each video frame; the video to be processed is a silent video without audio information; a differential processing module, configured to perform embedding-level differential processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector representing the dynamic change information of the video to be processed; the encoding module is further configured to perform a second encoding process on each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; a generation module, configured to perform iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and a decoding module, configured to perform decoding processing on the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0009] This application provides an electronic device, including: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-described video processing method.

[0010] This application provides a computer program product that stores computer-executable instructions or a computer program, which are used to cause a processor to execute the computer-executable instructions or the computer program to implement the above-described video processing method.

[0011] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which are used to cause a processor to execute the computer-executable instructions or computer programs to implement the above-described video processing method.

[0012] The above scheme has the following beneficial effects: In this embodiment, video frames are first encoded using a video encoder to obtain the frame embedding feature vector of each video frame. Then, the dynamic change information of the video to be processed is extracted through embedding-level differential processing to form a dynamic change vector. Simultaneously, static semantic information is extracted through another encoding path as semantic condition input. The dynamic change vector and the static semantic condition vector are jointly input into the diffusion generation model to guide the generated audio to maintain consistency in both time and semantics. This embodiment effectively utilizes the temporal change characteristics of the video to be processed itself, without the need for additional training modules or the introduction of external prediction networks, thereby reducing system complexity, improving the temporal alignment accuracy and rhythm matching ability of audio generation, and enhancing the naturalness and detail richness of the generated audio. Thus, it significantly improves the temporal matching accuracy and dynamic response capability of the generated audio with the video to be processed, ensuring the overall audio-visual consistency between the generated audio and the video to be processed. Attached Figure Description

[0013] Figure 1 This is an optional flowchart illustrating the video processing method provided in an embodiment of this application;

[0014] Figure 2 This is a schematic diagram of the implementation process of embedded level differential processing provided in the embodiments of this application;

[0015] Figure 3 This is a schematic diagram illustrating the implementation process of embedding-level differential processing of frame embedding feature vectors provided in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram illustrating the implementation flow of the training method for the diffusion generation model provided in this application embodiment;

[0017] Figure 5 This is a schematic diagram illustrating the implementation process of fine-tuning the text-based audio model provided in this application embodiment;

[0018] Figure 6 This is a system architecture diagram of a video-to-audio generation system based on visual change assistance provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the structure of the V2A inference module provided in the embodiments of this application;

[0020] Figure 8 This is a structural block diagram of the video processing apparatus provided in the embodiments of this application;

[0021] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application. In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal and can be implemented, in whole or in part, using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of that module or unit.

[0024] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the application. When applying these embodiments, the data collection and processing should strictly comply with the requirements of relevant laws and regulations, obtain informed consent or separate consent from the data subject, and conduct subsequent data use and processing within the scope of laws, regulations, and the authorization of the data subject.

[0025] Before describing the simulator-based data processing method provided in the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0026] 1) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0027] 2) Frame embedding feature vector: refers to the high-dimensional vector obtained after encoding video frames by a video encoder (such as the CLIP model). The frame embedding feature vector can represent the semantic content of the video frame and is an important conditional input for subsequent audio generation.

[0028] 3) Embedding-level differential processing: This refers to performing element-wise differential calculations between the frame embedding feature vectors of adjacent video frames to extract dynamic change information between video frames. This processing method can capture the temporal change features in the video to be processed, thereby assisting in the generation of audio with time synchronization.

[0029] 4) Static Semantic Condition Vector: This refers to the vector extracted after encoding video frames using a video encoder, which represents the overall semantic content of the video. This vector serves as one of the conditional inputs to the diffusion generation model, ensuring semantic consistency between the generated audio and the video to be processed.

[0030] 5) Dynamic Change Vector: Composed of a vector sequence obtained from embedding-level differential processing, this vector represents short-term dynamic information such as actions and event changes in the video to be processed, or fine-grained temporal information such as action rhythm and event triggering time in the video to be processed. This vector serves as another conditional input to the diffusion generation model, used to enhance the matching accuracy of the generated audio in terms of rhythm and time.

[0031] 6) Diffusion Generative Model: A deep learning-based generative model whose core idea is to progressively remove noise in the latent space to generate target data. In the embodiments of this application, the diffusion generative model generates audio latent representation vectors that conform to the semantic and temporal characteristics of video content by iteratively denoising the process and combining static semantic condition vectors with dynamically changing vectors.

[0032] 7) Audio Latent Representation Vector: An intermediate representation generated after iterative denoising using the diffusion generation model. It contains the main feature information of the audio signal and is used to describe the key features of the generated audio. This vector can be decoded into the final audio signal and serves as a bridge connecting the video content and the generated audio.

[0033] 8) Audio and video synchronization: This refers to aligning the generated audio with the video frames of the original silent video on the timeline after the audio is generated, ensuring that the start time of the audio is synchronized with the first frame of the video, and then packaging them into the same container file using an audio and video packaging tool to form an audio and video file that is consistent with the audio.

[0034] 9) Video-to-Audio (V2A): This is short for video-to-audio generation, referring to the process of generating an audio signal that is semantically and temporally consistent with video content. This task is widely used in film and television post-production, game sound effect generation, virtual reality, and other fields.

[0035] 10) CLIP: A language-image contrastive pre-trained model capable of extracting semantically meaningful feature vectors from images. In this embodiment, CLIP is used as a video encoder to extract static semantic information from video frames, which serves as semantic guidance for generating audio.

[0036] In related technologies, video-to-audio generation methods mainly rely on language-image contrastive pre-trained models (such as CLIP) to extract high-level semantic features from video frames. While these methods achieve good results in terms of semantic consistency, they struggle to capture the fine-grained temporal dynamics of the original video, leading to asynchrony between the generated audio and the original video's actions in terms of rhythm and event triggering. Furthermore, to enhance temporal synchronization, these methods typically introduce additional prediction networks or temporal coding modules, increasing system complexity and computational cost, and also resulting in the accumulation of prediction errors.

[0037] To address the aforementioned problems in related technologies, this application provides a video processing method that addresses the difficulty in capturing fine-grained temporal dynamic changes in the original video, leading to asynchronous rhythm and event triggering of the generated audio with the original video's actions. This method is a video-to-audio generation method based on visual change information. Specifically, a video encoder extracts the frame embedding feature vector for each video frame, and then performs embedding-level difference processing on the frame embedding feature vector to obtain a dynamic change vector representing the video's dynamic changes. Simultaneously, a video encoder extracts a static semantic condition vector representing the overall semantics of the video. Subsequently, these two vectors are used as conditional inputs, and iterative denoising is performed through a diffusion generation model to obtain the audio latent representation vector. Finally, decoding is used to generate the final generated audio. This method effectively improves the performance of the generated audio in terms of temporal alignment accuracy, rhythm consistency, and dynamic response capability without introducing additional network structures or training steps, achieving a high degree of synergy between video and audio in both semantic and temporal dimensions.

[0038] The video processing method provided in this application embodiment can be implemented using an electronic device, which can be a video processing device. The video processing device (i.e., electronic device) provided in this application embodiment can be implemented as a terminal or as a server. In one implementation, the video processing device provided in this application embodiment can be implemented as any terminal with video processing capabilities, such as a laptop, tablet, desktop computer, or intelligent robot. In another implementation, the video processing device provided in this application embodiment can also be implemented as a server, wherein the server can be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application embodiment does not impose any restrictions.

[0039] The video processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0040] Figure 1 This is an optional flowchart illustrating a video processing method provided in an embodiment of this application. This method can be applied to an electronic device, which can be a server. That is, the video processing methods in the embodiments of this application can be executed by a server, or they can be executed through interaction between the server and a terminal. The following will use an electronic device as a server as an example for illustrative explanation. Figure 1 As shown, the method includes the following steps S101 to S105:

[0041] Step S101: Perform a first encoding process on each video frame in the video to be processed by a video encoder to obtain the frame embedding feature vector of each video frame; the video to be processed is a silent video that does not contain audio information.

[0042] In this embodiment, the video encoder can employ the CLIP model or other visual encoders with strong semantic representation capabilities. The video encoder can extract high-dimensional embedding features from each input video frame, forming a frame embedding feature vector. The frame embedding feature vector reflects the semantic content of each video frame, such as scene content, object category, and action state. The frame embedding feature vector serves as the foundational data source for subsequently constructing static semantic conditional vectors and dynamic change vectors.

[0043] The process of obtaining the frame embedding feature vector is as follows: For the input silent video, each frame is fed into the video encoder. The video encoder extracts features from each video frame and outputs a fixed-dimensional vector representation. The frame embedding feature vector can be 2048-dimensional or higher, depending on the model structure of the video encoder. Since the silent video is continuous temporal data, the frame embedding feature vector constitutes a time series. In practice, the extraction operation of the frame embedding feature vector can preserve the high-level semantic information of each frame in the silent video, providing a foundation for subsequent semantic consistency and temporal synchronization.

[0044] Step S102: Perform embedding-level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector used to characterize the dynamic change information of the video to be processed.

[0045] Embedding-level differential processing refers to performing element-wise difference calculations on the frame embedding feature vectors of adjacent video frames to capture dynamic changes between video frames. For example, if the frame embedding feature vector of the current frame is v... t The frame embedding feature vector of the previous frame is v t-1 The difference result of the embedded-level difference processing is v t -v t-1 By performing element-wise difference calculations on all adjacent frames, a series of dynamic change vectors can be obtained. These vectors reflect the changing trends of object movement, scene transitions, or action rhythm in the silent video. The dynamic change vectors not only include the temporal differences between video frames but also implicitly contain the frequency and intensity of short-term events in the silent video. Embedded-level difference methods are more semantically expressive and can more accurately capture the dynamic changes of visual content.

[0046] In practical applications, embedded-level differential processing can extract high-quality visual dynamic signals without adding extra network structures, thereby effectively improving the timing accuracy and rhythm response of audio generation. Compared to methods based on optical flow or other complex motion estimation, embedded-level differential processing has higher computational efficiency and is suitable for real-time applications.

[0047] In some embodiments, see Figure 2 , Figure 2 The embedding-level difference processing of the frame embedding feature vector for each video frame in step S102 is shown, which can be achieved through the following steps S1021 and S1022:

[0048] Step S1021: Sample the frame embedding feature vector to obtain a frame embedding feature vector sequence; the frame embedding feature vector sequence is a vector sequence composed of the frame embedding feature vectors of multiple video frames in the video to be processed.

[0049] Sampling refers to selecting frame embedding feature vectors from a subset of video frames in the original video frame sequence according to certain rules, in order to form a more compact and representative feature sequence. Sampling can be implemented through time-interval sampling (e.g., sampling every n frames), keyframe detection, or attention mechanisms. Sampling reduces redundant information, lowers computational complexity, and preserves the main dynamic features of the video to be processed.

[0050] A frame embedding feature vector sequence is a set of vectors composed of multiple sampled frame embedding feature vectors arranged in chronological order. This sequence not only contains the static semantic information of each frame but also implicitly reveals the temporal relationships between frames, providing the foundation for subsequent difference calculations. Sampling effectively controls the scale of data input to subsequent processing modules, improving computational efficiency while preserving the overall dynamic information of the video. Furthermore, a reasonable sampling strategy can enhance sensitivity to important actions or scene transitions, thereby improving the temporal alignment accuracy of the generated audio and video content.

[0051] Here, the frame embedding feature vector is sampled. First, the sampling interval (frames) is determined. Then, the frame embedding feature vector is sampled according to the sampling interval to obtain a sequence of frame embedding feature vectors. The sampling interval (frames) refers to selecting the frame embedding feature vector of one video frame every certain number of frames in the video frame sequence as a sample to construct the temporal condition signal required for subsequent audio generation. The sampling interval (frames) determines the temporal resolution of the final generated sequence of frame embedding feature vectors. A smaller sampling interval (frames) retains more temporal details but increases computational burden; a larger sampling interval (frames) may lose key dynamic change information. Therefore, in practical applications, the sampling interval (frames) is usually set according to the dynamic complexity of the video content and the rhythm requirements of the generated audio. For example, a smaller sampling interval (frames) is used in fast-moving scenes, and a larger sampling interval (frames) is used in static or slowly changing scenes. By reasonably setting the sampling interval (frames), the computational efficiency and generation quality of the method for sampling the frame embedding feature vector can be effectively balanced, ensuring that the visual change features extracted by reasonably setting the sampling interval (frames) accurately reflect the key dynamic changes of the video content, thereby providing more accurate temporal guidance for audio generation.

[0052] Sampling processing refers to extracting feature points from consecutive frame embedding feature vectors at fixed intervals based on a predetermined sampling interval, forming a temporally ordered sequence of frame embedding feature vectors. Sampling processing can be implemented using an indexing method, where a frame embedding feature vector is taken every specified number of frames, and these vectors are sequentially concatenated to form the frame embedding feature vector sequence. Sampling processing can extract representative dynamic semantic segments from the video without altering the original embedding space structure, providing clear temporal clues for subsequent diffusion generation models.

[0053] In this embodiment, by sampling the frame embedding feature vector, important dynamic change information in the video to be processed can be extracted efficiently, and this dynamic change information can be transformed into temporal condition input suitable for use by the diffusion generation model, thereby improving the matching accuracy of the generated audio in terms of rhythm, event triggering time, etc.

[0054] Step S1022: Perform embedding level difference processing on every two adjacent frame embedding feature vectors in the frame embedding feature vector sequence to obtain a dynamic change vector used to characterize the dynamic change information of the video to be processed.

[0055] Embedding-level differential processing refers to the mathematical calculation of the difference between the frame embedding feature vectors of two adjacent frames. This can be achieved through vector subtraction or element-wise difference calculation to obtain a new vector representing the degree of visual change between the two frames. Embedding-level differential processing is performed in the semantic embedding space, not the pixel space, thus capturing changes at a higher semantic level, such as object movement, actions, and scene transitions. Dynamic change vectors are a series of vectors obtained from embedding-level differential processing, which collectively describe the visual change patterns that evolve over time in the video. Dynamic change vectors not only reflect local motion information between video frames but also serve as conditional signals for the diffusion generation model, guiding the generated audio to maintain synchronization with the video visuals in terms of rhythm, pitch, and event triggering. Embedding-level differential processing better preserves semantic consistency and avoids interference from background noise or irrelevant details. Dynamic change vectors provide a clear temporal guidance signal for subsequent audio generation, enabling the audio output to more naturally follow changes in the video content, thereby significantly improving audiovisual synchronization and immersion.

[0056] In this embodiment, by introducing a method for constructing frame embedding feature vector sequences and an embedding level difference processing method, dynamic change information in videos can be efficiently extracted without additional networks or training phases. This enhances the diffusion generation model's responsiveness to changes in video rhythm and motion, thereby improving the temporal matching accuracy between generated audio and video content, and ultimately enhancing the overall audiovisual consistency experience.

[0057] In some embodiments, see Figure 3 , Figure 3 The embedding-level difference processing of every two adjacent frame embedding feature vectors in the frame embedding feature vector sequence in step S1022 is shown, which can be achieved through the following steps S10221 and S10223:

[0058] Step S10221: For any two adjacent frame embedding feature vectors in the frame embedding feature vector sequence, perform difference calculation on the element values ​​at the same element positions in the two frame embedding feature vectors to obtain the difference embedding feature vector corresponding to the two frame embedding feature vectors.

[0059] Differential embedding feature vectors extract the differences between two video frames by subtracting the values ​​at the same element position from the embedding feature vectors of adjacent frames. This subtraction operation captures dynamic features in the temporal dimension of the video, such as object motion, scene changes, or action rhythm. For example, during a character's jump, changes in body posture may occur between consecutive video frames. These changes are represented as numerical fluctuations in the frame embedding feature vectors. Differential calculations can extract these posture changes and use them as temporal input for audio generation. Since the frame embedding feature vector is a high-dimensional vector output from encoding single video frames, the difference operation between adjacent frame embedding feature vectors not only focuses on numerical changes but also implicitly contains the temporal evolution trend of the video content. Therefore, differential embedding feature vectors exhibit strong temporal sensitivity.

[0060] Step S10222: Sort the differential embedding feature vectors corresponding to each pair of adjacent frame embedding feature vectors in the video to be processed according to the order in which the two adjacent frame embedding feature vectors correspond to each other, to form a differential embedding feature vector sequence.

[0061] The differential embedding feature vector sequence is a sequence of vectors formed by arranging all differential embedding feature vectors in the temporal order of their occurrence in the video to be processed. This ordered arrangement ensures the temporal continuity of the differential information and preserves the temporal relationships of changes between video frames. For example, in a video containing multiple shot transitions, each shot transition causes significant changes in the differential embedding feature vectors, and these changes are reflected in a specific order in the differential embedding feature vector sequence, thus helping the diffusion generative model understand visual events occurring at different points in time and their evolutionary trends.

[0062] By constructing a differential embedding feature vector sequence, it is possible to maintain sensitivity to dynamic changes in the video during the audio generation process, thereby better matching the rhythm of actions in the video to be processed and achieving a high degree of synchronization between audio and video content in the time dimension.

[0063] Step S10223: Determine the dynamic change vector based on the differential embedding feature vector sequence.

[0064] The dynamic change vector is a comprehensive representation further abstracted from the differential embedding feature vector sequence, used to describe the overall dynamic change pattern of the video to be processed. The dynamic change vector can be the average or maximum value of the differential embedding feature vector sequence, the projection result of principal component analysis, or other forms of aggregated features. The dynamic change vector not only reflects the intensity and direction of changes between individual local frames in the video to be processed, but also captures the overall temporal evolution trend, such as acceleration, deceleration, and periodic motion.

[0065] By constructing dynamically changing vectors, complex sequences of differentially embedded feature vectors can be compressed into a compact vector representation, facilitating subsequent use by the diffusion generation model. Based on this vector representation, the diffusion generation model can adjust its generation strategy according to the dynamic changes of the video being processed, thereby achieving more precise audio rhythm control and event response, and further improving the semantic and temporal consistency between audio and video.

[0066] In this embodiment of the application, determining the dynamically changing vector based on the differential embedding feature vector sequence can be achieved in the following two ways:

[0067] Method 1 involves concatenating the differential embedding feature vectors in the differential embedding feature vector sequence to obtain a dynamically changing vector.

[0068] The concatenation operation joins the differential embedding feature vectors sequentially into a single long vector, constructing a holistic representation that includes visual changes across multiple time points. This operation preserves the temporal order and dimensional structure of the differential embedding feature vectors, comprehensively reflecting the dynamic changes of the video content at various points in time. In practical applications, the concatenated dynamic change vector can serve as a conditional input to a diffusion generation model, guiding the audio generation process to remain synchronized with the events occurring in the video. For example, in a video of a character running, the concatenated vector would contain the visual changes resulting from each step, thus giving the audio generated by the diffusion generation model rhythm and coherence.

[0069] Method 2 involves taking the mean of all differential embedding feature vectors in the differential embedding feature vector sequence according to the position of each element, and then determining the mean embedding feature vector as a dynamically changing vector.

[0070] The mean embedding feature vector is a vector formed by averaging the values ​​at each position in the difference embedding feature vector sequence. The method of averaging the difference embedding feature vectors to obtain the mean embedding feature vector aims to extract the overall visual change trend in the video being processed, while ignoring short-term fluctuations or noise. For example, in a continuous video, if the visual changes of multiple adjacent video frames are similar, the difference embedding feature vectors corresponding to these multiple adjacent video frames will also tend to be consistent. Averaging these difference embedding feature vectors corresponding to multiple adjacent video frames yields a stable feature representing the overall change.

[0071] Mean embedding feature vector construction is suitable for scenarios that emphasize overall trends rather than subtle changes. For example, in videos where people move slowly or the background changes gradually, mean embedding feature vectors can more effectively express long-term dynamic changes without being affected by instantaneous interference. Furthermore, compared to concatenation operations, mean embedding feature vector construction is more computationally efficient and consumes fewer system resources, making it suitable for deployment on edge devices or mobile platforms. By using mean embedding feature vector construction, feature representation can be simplified, thereby reducing computational overhead and enabling efficient audio generation in resource-constrained environments. Ultimately, this helps expand the application scope and practicality of techniques based on differentially embedded feature vector sequences to determine dynamic change vectors.

[0072] In this embodiment, by providing two different methods for constructing dynamic change vectors, the system can adapt to the needs of different scenarios. By providing two different methods for constructing dynamic change vectors, both temporal details and overall trends can be taken into account, thereby flexibly adapting to various video content types and improving the robustness and generalization ability of the video-to-audio generation system.

[0073] Step S103: Perform a second encoding process on each video frame in the video to be processed by the video encoder to obtain a static semantic condition vector that represents the static semantic information of the video to be processed.

[0074] The second encoding process uses the same video encoder as the first, but with different purposes. The first encoding process aims to obtain the embedded feature vector for each video frame, while the second encoding process aims to extract the static semantic information of the video as a whole. The static semantic conditional vector is a general description of the entire video content, reflecting the overall theme, background environment, and main objects of the video. The static semantic conditional vector provides a description of the global semantic background of the video content, ensuring that the generated audio remains semantically consistent with the video content.

[0075] Static semantic conditional vectors can be generated by embedding feature vectors from all frames and then performing average pooling, max pooling, or weighted summation to obtain a unified high-dimensional vector. This unified high-dimensional vector serves as one of the conditional inputs to the diffusion generation model, ensuring that the generated audio is semantically consistent with the video content. In practice, the introduction of static semantic conditional vectors ensures that the generated audio is not only temporally synchronized with the video but also semantically accurately matches the video content. For example, if the video content is a scene of a car driving, the static semantic conditional vector can assist in generating environmental sound effects such as engine sounds and wind sounds associated with that conditional vector, thereby enhancing audiovisual consistency.

[0076] In this embodiment, the process of performing a second encoding process on each video frame of the video to be processed by the video encoder can be implemented as follows: The second encoding process is performed on each video frame of the video to be processed by the parameter-frozen video encoder in the audio variational autoencoder, resulting in a static semantic condition vector representing the static semantic information of the video to be processed. In this embodiment, the parameter-frozen video encoder in the audio variational autoencoder operates in a parameter-freezing manner to ensure that the semantic features output by the parameter-frozen video encoder are not affected by the current task. Since the parameter-frozen video encoder in the audio variational autoencoder has been sufficiently pre-trained, the semantic features output by the parameter-frozen video encoder have high generalization ability and robustness, and can provide stable semantic descriptions in different scenarios.

[0077] Step S104: Using a diffusion generation model, iterative denoising generation is performed based on static semantic condition vectors and dynamic change vectors to obtain audio latent representation vectors.

[0078] The diffusion generation model is a generative model based on an iterative denoising mechanism. Its core idea is to generate the target signal by progressively removing noise. In this embodiment, the diffusion generation model accepts a static semantic condition vector and a dynamic change vector as conditional inputs, using these inputs to guide the audio signal generation process. Specifically, in each iteration, the diffusion generation model adjusts its generation direction based on its current state and conditional information such as the static semantic condition vector and the dynamic change vector. Ultimately, the diffusion generation model generates an audio latent representation vector that is highly consistent with the video content.

[0079] The workflow of the diffusion generation model is as follows: First, the model injects initial random noise into the latent space; then, it performs multiple iterative denoising processes to gradually reconstruct audio feature representations that match the video content. In each iteration, the diffusion generation model relies on the current state and guiding information from the static semantic condition vector and the dynamic change vector. In practice, the diffusion generation model fully leverages the advantages of the static semantic condition vector and the dynamic change vector to achieve accurate audio generation in both semantic and temporal dimensions. The diffusion generation model achieves efficient and stable audio generation without requiring additional prediction networks or training modules.

[0080] In some embodiments, the iterative denoising generation process based on static semantic condition vectors and dynamic change vectors using a diffusion generation model can be implemented as follows: First, a noise tensor conforming to a normal distribution is randomly generated. The noise tensor is a set of random values ​​composed of multiple dimensions, used to simulate uncertainty and randomness in the real world. Then, according to a preset total number of diffusion steps, the diffusion generation model iteratively denoises and generates audio latent representation vectors in the audio latent space based on the noise tensor, static semantic condition vectors, and dynamic change vectors. The audio latent space is a low-dimensional latent space mapped to audio data after processing by a compression encoder (such as a variational autoencoder (VAE)). In the audio latent space, the temporal structure and frequency characteristics of the audio are preserved, facilitating cross-modal fusion and generation control. In actual implementation, the server jointly inputs the above three inputs (noise tensor, static semantic condition vector, and dynamic change vector) into the diffusion generation model and iteratively performs denoising according to a set total number of diffusion steps (such as 100 or 500 steps). In each step, the diffusion generation model adjusts its output based on the current latent state and conditional signals, ultimately generating an audio latent representation vector that highly matches the video content to be processed. By integrating the static semantic conditional vector and the dynamically changing vector into the conditional input of the diffusion generation model, the generated audio is not only semantically consistent with the video but also precisely aligned with the video actions in time. This mechanism significantly improves the consistency and immersiveness of audio and video, while avoiding the complexity and error accumulation problems caused by relying on additional networks in traditional methods.

[0081] In some embodiments, during each iteration of denoising, the static semantic condition vector and the dynamic change vector corresponding to the current time step are concatenated to obtain the current concatenated vector. Then, the current noisy first latent representation vector, the current time step, and the current concatenated vector are input into the diffusion generation model to obtain the predicted noise. Specifically, in the first iteration of denoising, the first latent representation vector is a noise tensor; in the i-th iteration of denoising, the first latent representation vector is the denoised vector obtained after the (i-1)-th iteration of denoising; i is an integer greater than 1. Next, the predicted noise is used to determine the second latent representation vector after removing some noise, and this second latent representation vector is determined as the denoised vector at the current time step. If the current time step equals the total number of diffusion steps, then the denoised vector at the current time step is determined as the audio latent representation vector.

[0082] Step S105: Decode the audio latent representation vector to obtain the generated audio corresponding to the video to be processed.

[0083] Decoding can be performed using a VAE (Video Audio Array). In this embodiment, the VAE is used to convert the audio latent representation vector into an audible audio waveform. During decoding, the temporal characteristics of the audio signal, such as frequency, amplitude, and rhythm, are gradually recovered, ultimately outputting complete generated audio corresponding to the input silent video. In practice, VAE decoding can convert the generated audio latent representation vector into a high-quality audio signal, ensuring that the audio signal is highly consistent with the input silent video in both semantics and time.

[0084] The video-to-audio generation method provided in this application extracts the static semantic condition vector and dynamic change vector of video frames through a video encoder, and passes them as conditional inputs to a diffusion generation model, thereby achieving cross-modal audio generation. The video-to-audio generation method provided in this application significantly improves the performance of audio generation in terms of temporal synchronization, rhythmic consistency, and dynamic response capabilities without introducing additional network structures or training stages, achieving a high degree of synergy between video and audio in both semantic and temporal dimensions.

[0085] In some embodiments, after decoding the audio latent representation vector in step S105 to obtain the generated audio corresponding to the video to be processed, the generated audio can also be post-processed by volume normalization. The volume normalization post-processing can be implemented in the following way: First, traverse the waveform sequence corresponding to the generated audio to obtain the absolute value of the amplitude of the generated audio at each moment; then, determine the scaling factor based on the preset target absolute value of amplitude and the maximum absolute value of amplitude in the waveform sequence; finally, use the scaling factor to scale the amplitude of the generated audio at each moment to obtain the target generated audio.

[0086] Here, a waveform sequence refers to time-series data composed of the amplitudes of the audio signals generated at consecutive time points. The amplitude value at each time point reflects the energy intensity of the audio signal at that time point and is a fundamental parameter describing changes in sound intensity. In this embodiment, the waveform sequence of the generated audio is traversed frame by frame or sample by sample point to obtain the absolute value of the amplitude of the generated audio at each time point. The absolute value of amplitude is defined as the magnitude of the amplitude regardless of its sign, and is used to uniformly measure the energy distribution of the audio signal. In practical applications, the waveform sequence can be represented as a one-dimensional array, where each element of the array corresponds to the amplitude value at a time point.

[0087] The target amplitude absolute value is a desired amplitude upper limit set according to the application scenario, used to limit the maximum output energy of the final generated audio. For example, in speech synthesis scenarios, to avoid popping or distortion, the target amplitude absolute value can be set to a smaller value; while in music generation scenarios, because a larger dynamic range is desired, the target amplitude absolute value is higher. The maximum amplitude absolute value is the maximum amplitude absolute value extracted from the waveform sequence, representing the maximum energy level of the currently generated audio. The scaling factor is calculated based on the ratio between the target amplitude absolute value and the maximum amplitude absolute value. The scaling factor linearly scales the amplitude of the generated audio to the target range, ensuring that the application of the scaling factor prevents overload or insufficient energy issues during audio playback. When the maximum amplitude absolute value is less than the target amplitude absolute value, the scaling factor is greater than 1, in which case the audio signal of the generated audio is amplified; when the maximum amplitude absolute value is greater than the target amplitude absolute value, the scaling factor is less than 1, in which case the audio signal of the generated audio is compressed. Using this scaling factor to scale the amplitude of the generated audio achieves adaptive adjustment of the overall loudness of the generated audio.

[0088] Scaling refers to multiplying the amplitude value of the generated audio by a scaling factor to adjust the overall loudness level of the generated audio. Scaling can be performed point-by-point in the time domain or as a whole in the frequency domain. Because audio signals have continuous and time-varying characteristics, it is necessary to maintain the smoothness of amplitude changes during scaling to avoid audio distortion or abruptness due to improper scaling. In practical applications, the audio generated after scaling is the target generated audio, and the amplitude distribution of the target generated audio has been adjusted to meet the expected range. For example, in a virtual reality environment, the loudness of the target generated audio can be dynamically adjusted according to the user's distance from the sound source, thereby enhancing the user's immersion and realism.

[0089] In some embodiments, the target amplitude absolute value can be adapted to the playback capabilities of different devices. For example, on mobile devices, the target amplitude absolute value can be set to a lower value to prevent speaker damage; while in professional audio systems, the target amplitude absolute value can be set to a higher value to retain more detail.

[0090] In summary, in this embodiment of the application, by traversing the waveform sequence of the generated audio and extracting the absolute value of the amplitude, and combining it with the preset target absolute value of the amplitude to calculate the scaling factor, the amplitude of the generated audio is scaled accordingly. This enables precise control of the loudness of the generated audio, ensuring the stability and consistency of the generated audio under different playback environments, thereby improving the user's auditory experience and audiovisual synchronization effect.

[0091] In some embodiments, after decoding the audio latent representation vector in step S105 to obtain the generated audio corresponding to the video to be processed, the obtained generated audio can also be post-processed by audio-video synchronization encapsulation. The audio-video synchronization encapsulation post-processing can be implemented in the following way: First, determine the frame rate of the silent video and the sampling rate of the generated audio; and based on the frame rate and sampling rate, construct a timeline that is used for both the silent video and the generated audio; then, align the starting audio waveform data of the generated audio with the first video frame of the silent video on the timeline; finally, call the audio-video encapsulation tool to package the aligned video frame sequence of the silent video and the audio waveform data stream of the generated audio into the same container file to obtain an audio-video synchronized file.

[0092] Frame rate refers to the number of video frames displayed per second, measured in FPS (Frames Per Second). Frame rate determines the smoothness of video playback. Sampling rate refers to the number of audio samples captured per second, measured in Hz. Sampling rate affects the temporal resolution and reproduction accuracy of the audio. A timeline is a shared temporal reference frame that unifies the video frame sequence to be processed and the audio waveform data stream to be generated onto the same time scale. The key to constructing a timeline is converting video frame numbers into timestamps and establishing a one-to-one correspondence with the index of the generated audio. For example, the first video frame corresponds to time 0 seconds, the nth video frame corresponds to time (n-1) / frame rate seconds, and the generated audio is time-converted according to the sampling rate. Constructing a unified timeline enables a clear correspondence between the video to be processed and the generated audio in the time dimension, avoiding audio-visual asynchrony problems caused by differences in time bases between different modalities. This is of great significance for achieving accurate matching of audio events and video actions, especially when generating ambient sounds, action sounds, or emotional sounds, significantly improving overall audio-visual consistency.

[0093] Alignment is the process of precisely matching audio waveform data with a sequence of video frames on the timeline. Specifically, electronic devices align the starting position of the generated audio (usually the first audio sample) with the timestamp of the first frame of the video to be processed, ensuring that the audio signal begins synchronously with the video content. Alignment can be achieved by adjusting the audio playback start offset or by truncating audio segments of a specific length.

[0094] Audio and video encapsulation tools are software components used to integrate and store video and audio data into standard format files. Examples include FFmpeg and MP4Box. In this embodiment, the audio and video encapsulation tool is invoked to encapsulate the aligned video frame sequence and audio waveform data stream according to standard container formats (such as MP4, AVI, MKV, etc.) to generate the final audio and video file. By performing the encapsulation operation, unified storage and transmission of video and audio are achieved, ensuring that the video and audio maintain time synchronization during playback. This encapsulation method is of great significance for practical applications such as video dubbing, film and television post-production, and game sound effect synthesis. Furthermore, using standard container formats in multimedia development improves the compatibility of multimedia files, facilitating playback and distribution on different platforms and devices.

[0095] In this embodiment of the application, by constructing a unified timeline and achieving audio-visual alignment, the problem of audio-visual asynchrony caused by inconsistent time bases in traditional methods can be effectively solved. This enables precise matching of audio and video in terms of rhythm, event triggering time, etc., further improving the overall quality of multimedia content and the user's perception of realism.

[0096] The video processing method provided in this application has wide application value in various video-to-audio generation fields, such as automatic dubbing after AI video generation, silent film restoration and soundtrack creation, automatic soundtrack creation for educational videos, short video content creation, and automatic soundtrack creation for game videos. The video processing method provided in this application can generate audio corresponding to silent videos. The following are examples illustrating the application scenarios of the video processing method provided in this application.

[0097] Application Scenario 1: Automatic Dubbing After AI Video Generation. Current AI video generation models produce silent videos, requiring manual addition of sound effects and background music in post-production. The method in this application embodiment can automatically add music to AI-generated videos with a single click, significantly improving creation efficiency. In the implementation process, the AI-generated silent video can be input into a CLIP visual encoder to extract the frame embedding feature vector for each frame. For example, for a 10-second 30FPS video, feature vectors from 300 frames are extracted. Then, the CLIP embedding difference between adjacent frames (e.g., k=2) is calculated to obtain a dynamic change vector. For example, the inter-frame difference in a person's waving gesture in the video is captured as a high-modulus change vector. The video is then further encoded using the CLIP encoder to obtain a static semantic condition vector, representing the overall content of the video (e.g., "a person is walking in a forest"). Next, the static semantic condition vector and the dynamic change vector are concatenated and input into a diffusion generation model for iterative denoising to generate a latent audio representation. The diffusion generation model adjusts the rhythm and intensity of the sound effects based on the dynamic change vector. Finally, the audio latent representation vector is decoded to obtain sound effects (such as footsteps, wind sounds, birdsong, etc.) synchronized with the video actions. Compared with traditional manual music accompaniment, the method provided in this application embodiment achieves automatic audio-visual synchronization, avoiding the problem of audio effects being out of sync with actions, and shortening the production cycle from several hours to seconds.

[0098] Application Scenario 2: Silent Film Restoration and Soundtrack Production. This involves automatically generating sound effects and background music that match the historical context of historical silent films (such as early documentaries), enhancing the viewing experience. The process involves digitizing the black-and-white silent film, extracting the video frame sequence, and using a CLIP encoder to extract the semantic features of each frame, capturing scene content (e.g., "20th-century city street" or "train station scene"). Then, inter-frame embedding differences are calculated to identify motion rhythms (e.g., walking frequency of people, speed of vehicles). Static semantic conditions (historical context, scene type) are combined with dynamic change vectors to generate background music and sound effects that match the historical style (e.g., steam train sounds, horse-drawn carriage sounds). Finally, stereo audio is decoded and output synchronously with the video.

[0099] Application Scenario 3: Automatic Background Music for Educational Videos. This feature automatically generates narration and sound effects for instructional videos (such as programming tutorials and experimental demonstrations), enhancing teaching effectiveness. In its implementation, the CLIP encoder analyzes the static semantics of the instructional video (e.g., "Python programming interface," "chemical experiment operation"). Then, through embedded differential processing, it identifies the temporal changes in operational actions (e.g., keyboard clicks, mouse clicks, experimental equipment operation). Based on the static semantic conditions, it generates narration (e.g., "Now let's write a function"), while simultaneously generating operational sound effects (e.g., keyboard clicks, mouse clicks) based on dynamic change vectors. The diffusion generation model precisely controls the triggering timing of the sound effects based on the timestamps of inter-frame changes, ensuring complete synchronization between the sound effects and the operational actions. This achieves "one-click generation of instructional videos with narration," significantly lowering the barrier to educational content creation. Teachers only need to record the operation video to automatically obtain a complete instructional video.

[0100] Application Scenario 4: Short Video Content Creation. This feature automatically generates background music and sound effects for UGC content on short video platforms, enhancing content appeal. The implementation uses a CLIP encoder to identify video styles (e.g., "funny," "touching," "thriller"). Through embedded differential processing, it analyzes the video's editing rhythm and action frequency. Then, it selects a music style (e.g., upbeat, lyrical, tense) based on static semantic conditions and adjusts the music rhythm and intensity based on dynamic change vectors. It also identifies key actions in the video (e.g., transitions, special effects) and generates corresponding sound effects (e.g., transition sounds, special effect sounds). This solves the pain points of short video creators, such as difficulty finding background music and audio-visual asynchrony, improving content creation efficiency and work quality.

[0101] Application Scenario 5: Automatic Music Addition to Game Videos. This feature automatically generates background music and combat sound effects that match the game's scenes in screen recordings, enhancing the immersive experience. The implementation uses a CLIP encoder to identify game scene types (e.g., "battle scenes," "exploration scenes," "story scenes"). Then, through embedded differential processing, it analyzes the intensity and rhythm changes of combat actions. Based on dynamic change vectors, it adjusts the rhythm, volume, and instrument configuration of the background music in real time, making the music faster and louder during intense battles and more soothing during calmer times. Finally, based on the timestamps of frame changes, it accurately generates skill release sound effects and attack sound effects, ensuring complete synchronization with the game visuals. This achieves intelligent music addition to game videos, eliminating the need for manual sound effect editing and significantly improving the efficiency and quality of game video production.

[0102] This application embodiment further provides a training method for a diffusion generation model, see [link to relevant documentation]. Figure 4 , Figure 4This is a schematic diagram illustrating the implementation flow of the training method for the diffusion generation model provided in this application embodiment. This training method can be executed by an electronic device, which can be the same device as the electronic device used to implement the video processing method, or it can be a different device. The electronic device can be implemented as a terminal or as a server. It should be noted that the training method for the diffusion generation model is executed before the iterative denoising generation step based on the static semantic condition vector and the dynamic change vector using the diffusion generation model. The training method for the diffusion generation model includes the following steps S201 and S202:

[0103] Step S201: The initial diffusion model is pre-trained using the first sample dataset to obtain the Wensheng audio model.

[0104] The first sample data in the first sample dataset consists of sample data pairs composed of sample text and sample audio.

[0105] Here, we first use a first sample dataset consisting of sample text and its corresponding sample audio to pre-train the initial diffusion model. The goal of the pre-training process is to enable the initial diffusion model to convert text descriptions into high-quality audio. For example, given the description "birdsong in the morning," the text-based audio model should be able to generate a clear and natural birdsong. The pre-training stage establishes the basic generation capability of the text-based audio model and provides a strong prior for its subsequent fine-tuning.

[0106] Step S202: Using the second sample dataset, fine-tune the Wensheng audio model to obtain the diffusion generation model.

[0107] The second sample data in the second sample dataset consists of sample data pairs composed of sample videos and sample audio.

[0108] After initial pre-training, the Wensheng audio model was further fine-tuned on a second sample dataset. This second dataset consists of sample videos and their corresponding audio samples, used to adapt the Wensheng audio model to the video-to-audio task. By incorporating sample video information, the Wensheng audio model can learn the correlation between the visual content of the sample videos and the sample audio, thereby improving the temporal matching accuracy of the generated audio.

[0109] In this embodiment, a two-stage training process is introduced, including pre-training of the Wensheng audio model and fine-tuning based on sample videos. This training process can improve the quality of audio generation and time alignment accuracy, and can achieve more natural audio output that fits the sample video content better.

[0110] In some embodiments, see Figure 5 , Figure 5 The fine-tuning of the Wensheng audio model using the second sample dataset in step S202 can be achieved through the following steps S2021 to S2025:

[0111] Step S2021: The parameter-frozen encoder in the audio variational autoencoder performs the first encoding process on each sample video frame of the sample video to obtain the sample frame embedding feature vector of each sample video frame; the sample video is a silent video that does not contain audio information.

[0112] The parameter-frozen encoder in an audio variational autoencoder is a deep learning model used to compress audio signals into low-dimensional latent representations. The structure of the parameter-frozen encoder in an audio variational autoencoder can include an encoder and a decoder. In this embodiment, the encoder part of the audio variational autoencoder with frozen parameters is used, and the parameters of the encoder are set to a frozen state, meaning that the encoder parameters are not updated during training. This is because the encoder has already been pre-trained and possesses good audio representation capabilities, thus eliminating the need for retraining to extract effective semantic information.

[0113] In an audio variational autoencoder, the parameter-frozen encoder maps the input video frames to a feature vector in a low-dimensional embedding space; this feature vector is called the sample frame embedding feature vector. Since the sample video is a silent video that does not contain audio information, visual information must be relied upon to construct the conditional signal. In this embodiment, after performing the first encoding process, the static visual features of each video frame can be extracted and used as the basis for subsequent dynamic change calculations.

[0114] In this embodiment, by using a parameter-frozen encoder in the audio variational autoencoder, the sample frame embedding feature vector corresponding to each sample video frame can be extracted efficiently, effectively avoiding the complexity and computational overhead caused by additional training. Furthermore, the parameter-frozen encoder in the audio variational autoencoder ensures the high quality and stability of the extracted features. Therefore, it provides a reliable input foundation for subsequent dynamic change modeling.

[0115] Step S2022: Perform embedding level difference processing on the sample frame embedding feature vector of each sample video frame to obtain the sample dynamic change vector used to characterize the dynamic change information of the sample video.

[0116] Embedding-level differential processing refers to calculating the differences between adjacent video frames in the embedding space to capture the visual dynamic information that changes over time in the sample video. Specifically, for the sample frame embedding feature vector of each video frame, the difference between this video frame's sample frame embedding feature vector and the sample frame embedding feature vector of the previous or next video frame is calculated to form a new vector, which represents the degree of change of the current video frame relative to the previous or next video frame. This processing method can extract time-sensitive dynamic features without relying on pixel-level operations.

[0117] Step S2023: The parameter-frozen encoder in the audio variational autoencoder performs a second encoding process on each sample video frame in the sample video to obtain a sample static semantic condition vector used to characterize the static semantic information of the sample video.

[0118] Here, the parameter-frozen encoder from the audio variational autoencoder is used again, but for a different purpose. The first encoding process is to extract the embedded feature vectors of the sample frames for difference calculation, while the second encoding process is to obtain more stable static semantic features. The sample static semantic condition vector is used to describe the overall theme, scene, object category, and other invariant information of the sample video content, and is the core conditional signal guiding the generation of audio content. Since the parameter-frozen encoder in the audio variational autoencoder is frozen, the audio variational autoencoder does not introduce additional training costs during use. The sample static semantic condition vector and the sample dynamic change vector complement each other. The sample static semantic condition vector provides global semantic guidance, while the sample dynamic change vector provides local temporal constraints. Together, the sample static semantic condition vector and the sample dynamic change vector constitute a complete conditional input system.

[0119] Step S2024: Using the Wensheng audio model, iterative denoising is performed based on the static semantic condition vector and the dynamic change vector of the sample to generate the potential representation vector of the sample audio.

[0120] The Wensheng audio model is a generative model based on a diffusion mechanism. Its core idea is to progressively remove noise from the latent space to ultimately generate a suitable audio representation. In this embodiment, the Wensheng audio model accepts two conditional inputs: a static semantic condition vector and a dynamic change vector. The static semantic condition vector and the dynamic change vector guide the content direction and temporal rhythm of the audio, respectively.

[0121] The iterative denoising generation process is divided into multiple time steps, with each step adjusting the denoising strategy based on the current noise level. As the iteration progresses, the text-based audio model gradually recovers a clear latent audio representation, making it highly compatible with the input sample video in both semantic and temporal dimensions. By jointly using static semantic conditions and dynamic change vectors of the samples, the text-based audio model can achieve a multi-dimensional understanding of the sample video content during the generation process. This ensures both the consistency of audio semantics and enhances the temporal alignment capability of the text-based audio model, thereby improving generation quality and user experience.

[0122] Step S2025: Based on the latent representation vector of the sample audio and the latent representation vector corresponding to the sample audio, fine-tune the parameters of the text-generated audio model to obtain the diffusion generation model.

[0123] After generating the latent representation vectors of the sample audio, these vectors are compared with the corresponding latent representation vectors of the sample audio, and a loss function is calculated accordingly. Through backpropagation of the loss function, the trainable parameters of the Wensheng audio model are fine-tuned to optimize its generation performance. This fine-tuning process aims to better adapt the Wensheng audio model to the data distribution of a specific task, improving its generalization ability and generation accuracy. The key to this fine-tuning process lies in selecting a suitable loss function and optimization strategy. The loss function can include mean squared error (MSE), adversarial loss, and perceptual loss. In this embodiment, a combined loss can also be used to balance audio quality and time synchronization.

[0124] By using parameter fine-tuning methods based on latent representations, the performance of the Wensheng audio model in specific tasks can be effectively improved. The audio generated by the Wensheng audio model is superior to the audio generated by the un-fine-tuned model in terms of naturalness, detail richness, and temporal alignment, thus meeting the needs of practical applications.

[0125] In summary, in practical implementation, the fine-tuning steps in the training method of the diffusion generation model exhibit clear logical relationships and collaborative mechanisms. First, the parameter-frozen encoder in the audio variational autoencoder extracts the sample frame embedding feature vector for each sample video frame, providing a foundation for subsequent dynamic change analysis. Next, embedding-level difference processing extracts dynamic change information from the samples, capturing temporal features in the sample videos. Then, the same encoder is used again to extract static semantic information from the samples, constructing a complete conditional input system. Based on this, the Wensheng audio model iteratively denoises and generates audio latent representations that initially meet the requirements, according to the static and dynamic conditions of the samples. Finally, by comparing the initially generated sample audio latent representation vector with the latent representation vector of real audio, the Wensheng audio model is fine-tuned, continuously optimizing the generation effect. These steps work together to achieve a high-quality, highly consistent generation process from sample videos to audio.

[0126] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0127] This application provides a video-to-audio generation system based on visual change assistance (i.e., the video processing system described above), which can be applied to... Figure 6 In the system architecture shown. Figure 6 This demonstrates the data interaction and processing flow between the user terminal and the backend inference engine in the video-to-audio generation system. It is applicable to general audio application scenarios such as automatic dubbing of video content, virtual scene sound effect generation, automatic sound effect production in film and television post-production, and immersive interactive experiences. In this system architecture, the video input interface 501 receives the silent video to be dubbed and sends the video frame sequence of the silent video (i.e., video pre-trained features) to the mobile device 502. The mobile device can be a server or a locally deployed V2A inference module (i.e., the video-to-audio generation model). The mobile device 502 generates the audio signal 503 (i.e., the final generated audio) and outputs the audio signal through the audio output interface 504. Figure 7 This is a schematic diagram of the structure of the V2A inference module provided in the embodiments of this application, as shown below. Figure 7As shown, the V2A inference module includes: a video encoder 701 (CLIP) for extracting semantic information from video frames of the input silent video to obtain a static semantic condition vector, and extracting frame embedding feature vectors of the video frames; a visual change calculation module 702 for performing embedding-level differential processing on the frame embedding feature vectors of each video frame to obtain an embedding-level dynamic change vector for representing the dynamic change information of the video to be processed; a diffusion generation model 703 for iterative denoising generation by combining conditional signals (i.e., static semantic condition vectors and dynamic change vectors) to obtain an audio latent representation vector; and a compression model 704 (i.e., a VAE decoder) for mapping the audio latent representation vector to a low-dimensional latent space to obtain the final generated audio.

[0128] It should be noted that the main technical solutions of this application are as follows: First, a text-to-audio pre-training and visual change joint conditionalization method is adopted. Specifically, a large-scale text-to-audio pre-training model (T2A) is used to initialize audio generation capabilities, enhancing the naturalness and semantic matching ability of the audio; visual conditional signals and visual change information are introduced in the V2A fine-tuning stage to achieve cross-modal adaptation from silent video to audio and improve semantic and temporal alignment; by freezing the VAE compression model and CLIP video encoder, dynamic changes are efficiently captured while maintaining semantic stability; the stability and naturalness of the generated audio in terms of rhythm, scene consistency, and sound effect details are improved. Second, a visual change-assisted method is adopted to design an embedding-level (CLIP-Level) visual change extraction method to capture short-term dynamic information; the visual change vector and the static semantic conditional vector are integrated as the conditional input of the diffusion generation model, without the need for additional training modules or loss functions; the diffusion generation model is guided in the latent space to strengthen the generation of audio signals corresponding to video dynamic events; the detail richness, dynamic response capability, and video-audio temporal alignment accuracy of the generated audio are improved.

[0129] The implementation process of the video-to-audio generation method based on visual change assistance provided in the embodiments of this application is described below.

[0130] First, a T2A generative model is pre-trained. Text is then embedded... With target audio The latent representation corresponding to (L is the audio length) ( and Using latent space temporal sequence and feature dimension as inputs, a stable audio generation system is pre-trained. This T2A pre-training establishes a strong generative prior, reducing the difficulty of V2A training and improving audio reproduction capabilities.

[0131] Then, visual change information is constructed from the input silent video. (in (where represents the i-th frame of a silent video, and N is the total number of frames in the video), extracting CLIP-Level Visual Variation. Specifically, the CLV is calculated using the following formula (1):

[0132] (1).

[0133] Where k represents the number of frames between which visual changes are calculated (i.e., the number of sampling interval frames). Based on the ablation experiment results, k=2 is optimal; too small a value is insufficient to capture motion, while too large a value easily introduces noise. Visual changes provide motion-sensitive features, guiding the diffusion generation model to capture dynamic audio events.

[0134] Next, conditional diffusion generation from video to audio is performed. The pre-trained T2A model is fine-tuned to V2A by using text conditional... Replace with video conditions With visual change information Optimize diffusion model parameters Generate potential audio representations The optimization objective is as follows (2):

[0135] (2).

[0136] in, The noise is added during the forward process, and t is the noise addition time step. for Latent variables after adding noise. Semantic guidance is provided through CLIP embedding, and temporal dynamic constraints are provided through visual change information. After model training, the generated audio and video content are highly consistent in semantics and time.

[0137] Finally, audio decoding and output are performed. The generated latent audio representation is then processed. The final audio waveform is decoded using VAE. The output is the audio sequence corresponding to the input silent video.

[0138] In some embodiments, result evaluation can also be performed. Specifically, in the VGGSound benchmark test, the embodiments of this application achieved better results than models trained solely on CLIP features across multiple evaluation metrics, including FAD, KL, IS, FD, ImageBind Score, and AA, achieving the effect of introducing additional features (CAVP). Furthermore, by introducing a T2A pre-training mechanism, the embodiments of this application also demonstrate significant advantages in V2A tasks, with overall performance surpassing current mainstream models, showcasing stronger cross-modal understanding and generation capabilities.

[0139] This application's embodiments introduce a visual change-assisted mechanism to calculate embedded-level visual difference features in the video frame sequence and use them as dynamic conditional inputs to the diffusion generation model. This enables the generated audio signal to accurately respond to subtle motion changes and event triggering moments in the video. Compared to traditional video-to-audio generation methods based solely on static semantic conditions, this application's embodiments significantly improve temporal alignment accuracy, rhythmic consistency, and dynamic sound effect response capabilities, achieving adaptive matching of the audio generation process to the video's temporal dynamics. Furthermore, by combining visual semantic features extracted from a language-image contrast pre-trained model, this application's embodiments achieve joint conditionalization of video semantics and visual change features without introducing additional network structures or training stages. This significantly improves the naturalness, detail richness, and semantic consistency of the generated audio while maintaining system simplicity and efficiency. This mechanism effectively avoids the error accumulation problem caused by traditional methods relying on external feature prediction, enhancing the model's generalization ability and stability across scenes and action types.

[0140] It should be noted that current video-to-audio generation methods generally rely on language-image contrast pre-trained models to extract static semantic features to achieve semantic matching between video content and generated audio. However, these models mainly focus on image-level content understanding and lack sensitivity to dynamic changes in the temporal dimension, making it difficult to accurately reflect the rhythm of actions and the timing of event triggers. In contrast, the "visual change information-assisted guidance mechanism" proposed in this application directly calculates the differences between adjacent frames and extracts motion-related features, enhancing temporal alignment capabilities without introducing additional networks or training stages. While maintaining low computational overhead, it significantly improves the matching effect between dynamic events and audio rhythms.

[0141] In addition to the inter-frame difference features in the embodiments of this application, optical flow field estimation, sparse keypoint tracking, and temporal gradient accumulation can also be used to replace frame difference calculation. These methods can capture the object's movement direction, velocity, and regional dynamics more precisely. If deployed in scenarios where system resources permit, such solutions can also achieve visually dynamic-driven audio generation, which is equivalent to the principle of the embodiments of this application and belongs to an equivalent implementation.

[0142] The video-to-audio generation method provided in this application can be applied to at least the following scenarios: 1) Film and game sound effect generation scenarios: In film and television production or game engines, it is often necessary to automatically generate ambient sound, action sound, or emotional sound for silent videos. The visual change guidance mechanism of this application can accurately capture dynamic information such as character movements, camera transitions, and object collisions, thereby generating sound that is highly consistent with the rhythm of the picture, realizing automated onomatopoeia and post-production sound effect generation. 2) Virtual reality and immersive interaction scenarios: In virtual reality (VR), augmented reality (AR), or digital twin systems, the spatial consistency and temporal synchronization of ambient sound are crucial to the sense of immersion. Through the method provided in this application, the audio output can be automatically adjusted according to changes in the user's perspective or scene dynamics, realizing three-dimensional sound field rendering synchronized with visual changes. 3) Intelligent monitoring and event detection scenarios: In intelligent security or traffic monitoring, this application can be used to generate or predict potential sound events from silent videos, such as collisions, alarms, and mechanical operations, to achieve sound completion and event recognition, providing supplementary information for multimodal early warning systems. 4) Digital Human and Virtual Character Performance Scenarios: In systems such as virtual humans, AI anchors, and intelligent customer service, this application embodiment can automatically drive speech or onomatopoeia generation through facial movements or body changes, achieving consistency between visual behavior and audio performance, and enhancing the expressiveness and naturalness of digital characters. 5) Content Generation and Creative Synthesis Scenarios: In short video, animation production, and multimodal creation platforms, this application embodiment can serve as a "silent video dubbing engine," automatically generating corresponding sound effects or background music based on the rhythm of the scene and changes in movement, lowering the creative threshold and expanding multimedia content production methods. 6) Scientific Research and Educational Assistance Scenarios: In scientific research visualization or teaching demonstrations, this application embodiment can be used to generate sound simulations based on experimental recordings, such as chemical reactions and mechanical movements, helping users perceive dynamic processes through hearing, improving teaching interactivity and depth of understanding. 7) Mobile and Edge Computing Deployment Scenarios: This application embodiment requires no additional training stage; the model is lightweight and inference is efficient, allowing direct embedding into mobile or edge devices to achieve real-time video and audio completion, intelligent speech generation, or scene sound feedback, providing an audiovisual collaborative experience for portable multimedia devices.

[0143] Based on the video processing method described in the above embodiments, Figure 8This paper illustrates a structural block diagram of a video processing apparatus according to an embodiment of this application. The video processing apparatus 100 can be a device in an electronic device (e.g., a server). It can be implemented in software, such as programs and plugins, and includes the following software modules: an encoding module 101, used to perform a first encoding process on each video frame in the video to be processed using a video encoder to obtain a frame embedding feature vector for each video frame; the video to be processed is a silent video without audio information; a differential processing module 102, used to perform embedding-level differential processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector representing the dynamic change information of the video to be processed; the encoding module 101 is further used to perform a second encoding process on each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; a generation module 103, used to perform iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and a decoding module 104, used to decode the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0144] In some embodiments, the differential processing module 102 is further configured to: perform sampling processing on the frame embedding feature vectors to obtain a frame embedding feature vector sequence; the frame embedding feature vector sequence is a vector sequence composed of the frame embedding feature vectors of multiple video frames in the video to be processed; and perform embedding-level differential processing on each two adjacent frame embedding feature vectors in the frame embedding feature vector sequence to obtain a dynamic change vector used to characterize the dynamic change information of the video to be processed.

[0145] In some embodiments, the differential processing module 102 is further configured to: determine the number of sampling interval frames; and perform sampling processing on the frame embedding feature vector according to the number of sampling interval frames to obtain a sequence of frame embedding feature vectors.

[0146] In some embodiments, the differential processing module 102 is further configured to: perform differential calculation on the element values ​​at the same element positions in any two adjacent frame embedding feature vectors in the frame embedding feature vector sequence to obtain differential embedding feature vectors corresponding to the two frame embedding feature vectors; sort the differential embedding feature vectors corresponding to each pair of adjacent frame embedding feature vectors according to the order in which they correspond in the video to be processed to form a differential embedding feature vector sequence; and determine the dynamic change vector based on the differential embedding feature vector sequence.

[0147] In some embodiments, the difference processing module 102 is further configured to: concatenate the difference embedding feature vectors in the difference embedding feature vector sequence to obtain a dynamically changing vector; or, calculate the mean value embedding feature vector according to the position of each element of all the difference embedding feature vectors in the difference embedding feature vector sequence; and determine the mean value embedding feature vector as the dynamically changing vector.

[0148] In some embodiments, the apparatus further includes: a post-processing module, configured to, after decoding the audio latent representation vector to obtain the generated audio corresponding to the video to be processed, traverse the waveform sequence corresponding to the generated audio to obtain the absolute value of the amplitude of the generated audio at each moment; determine a scaling factor based on a preset target absolute value of amplitude and the maximum absolute value of amplitude in the waveform sequence; and scale the amplitude of the generated audio at each moment using the scaling factor to obtain the target generated audio.

[0149] In some embodiments, the post-processing module is further configured to, after decoding the audio latent representation vector to obtain the generated audio corresponding to the video to be processed, determine the frame rate of the silent video and the sampling rate of the generated audio; construct a timeline for both the silent video and the generated audio based on the frame rate and the sampling rate; align the starting audio waveform data of the generated audio with the first video frame of the silent video on the timeline; and call an audio-video encapsulation tool to package the aligned video frame sequence of the silent video and the audio waveform data stream of the generated audio into the same container file to obtain an audio-video synchronized file.

[0150] In some embodiments, the apparatus further includes: a model training module, configured to train the diffusion generation model through the following steps before iteratively denoising and generating the model based on static semantic condition vectors and dynamic change vectors: pre-training the initial diffusion model using a first sample dataset to obtain a text-based audio model; the first sample data in the first sample dataset consists of sample data pairs composed of sample text and sample audio; fine-tuning the text-based audio model using a second sample dataset to obtain the diffusion generation model; the second sample data in the second sample dataset consists of sample data pairs composed of sample video and sample audio.

[0151] In some embodiments, the model training module is further configured to: perform a first encoding process on each sample video frame of the sample video using the parameter-frozen encoder in the audio variational autoencoder to obtain a sample frame embedding feature vector for each sample video frame; the sample video is a silent video without audio information; perform embedding level difference processing on the sample frame embedding feature vector of each sample video frame to obtain a sample dynamic change vector for representing the dynamic change information of the sample video; perform a second encoding process on each sample video frame in the sample video using the parameter-frozen encoder in the audio variational autoencoder to obtain a sample static semantic condition vector for representing the static semantic information of the sample video; perform iterative denoising generation based on the sample static semantic condition vector and the sample dynamic change vector using the text-generated audio model to obtain a sample audio latent representation vector; and fine-tune the parameters of the text-generated audio model based on the sample audio latent representation vector and the latent representation vector corresponding to the sample audio to obtain a diffusion generation model.

[0152] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0153] This application provides an electronic device. Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 9 (Only one is shown in the diagram), memory 132, and computer-executable instructions 133 stored in memory 132 and executable on at least one processor 131, which, when executed by processor 131, implement the steps in any of the above-described audio generation method embodiments.

[0154] The electronic device may include, but is not limited to, a processor 131 and a memory 132. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 130 and does not constitute a limitation on electronic device 130. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0155] Processor 131 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0156] In some embodiments, memory 132 may be an internal storage unit of electronic device 130, such as a hard disk or memory of electronic device 130. In other embodiments, memory 132 may be an external storage device of electronic device 130, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on electronic device 130. Furthermore, memory 132 may include both internal and external storage units of electronic device 130. Memory 132 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 132 may also be used to temporarily store data that has been output or will be output.

[0157] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video processing method described in this application.

[0158] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the audio generation method provided in this application. For example, ... Figure 1 The video processing method shown.

[0159] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0160] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0161] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0162] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0163] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video processing, the method comprising: The method comprises: performing first encoding processing on each video frame in a to-be-processed video by a video encoder to obtain a frame embedding feature vector of each video frame; the to-be-processed video is a silent video without audio information; performing embedding-level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector for representing dynamic change information of the to-be-processed video; performing second encoding processing on each video frame in the to-be-processed video by the video encoder to obtain a static semantic condition vector for representing static semantic information of the to-be-processed video; generating an audio latent representation vector through an iterative denoising generation based on the static semantic condition vector and the dynamic change vector by a diffusion generative model; performing decoding processing on the audio latent representation vector to obtain generated audio corresponding to the to-be-processed video.

2. The method of claim 1, wherein, The embedding-level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector for representing dynamic change information of the to-be-processed video comprises: performing sampling processing on the frame embedding feature vector to obtain a frame embedding feature vector sequence; the frame embedding feature vector sequence is a vector sequence composed of frame embedding feature vectors of multiple video frames in the to-be-processed video; performing embedding-level difference processing on each adjacent two frame embedding feature vectors in the frame embedding feature vector sequence to obtain a dynamic change vector for representing dynamic change information of the to-be-processed video.

3. The method of claim 2, wherein, The sampling processing on the frame embedding feature vector to obtain a frame embedding feature vector sequence comprises: determining a sampling interval frame number; performing sampling processing on the frame embedding feature vector according to the sampling interval frame number to obtain the frame embedding feature vector sequence.

4. The method of claim 2, wherein, The embedding-level difference processing on each adjacent two frame embedding feature vectors in the frame embedding feature vector sequence to obtain a dynamic change vector for representing dynamic change information of the to-be-processed video comprises: for any adjacent two frame embedding feature vectors in the frame embedding feature vector sequence, performing difference calculation on element values at the same element positions in the two frame embedding feature vectors to obtain a difference embedding feature vector corresponding to the two frame embedding feature vectors; sequencing the difference embedding feature vectors corresponding to each adjacent two frame embedding feature vectors according to the sequence of the two frame embedding feature vectors in the to-be-processed video to form a difference embedding feature vector sequence; determining the dynamic change vector based on the difference embedding feature vector sequence.

5. The method of claim 4, wherein, The determining the dynamic change vector based on the difference embedding feature vector sequence comprises: splicing the difference embedding feature vectors in the difference embedding feature vector sequence to obtain the dynamic change vector; or, averaging all the difference embedding feature vectors in the difference embedding feature vector sequence according to each element position to obtain an average embedding feature vector; and determining the average embedding feature vector as the dynamic change vector.

6. The method according to any one of claims 1 to 5, characterized in that, After the decoding processing on the audio latent representation vector to obtain generated audio corresponding to the to-be-processed video, the method further comprises: traversing the waveform sequence corresponding to the generated audio to obtain an amplitude absolute value of the generated audio at each time instant; determining a scaling factor based on a preset target amplitude absolute value and a maximum amplitude absolute value in the waveform sequence; scaling the amplitude of the generated audio at each time instant using the scaling factor to obtain target generated audio.

7. The method according to any one of claims 1 to 5, characterized in that, After decoding the audio latent representation vector to obtain the generated audio corresponding to the to-be-processed video, the method further comprises: determining a frame rate of the silent video and a sampling rate of the generated audio; based on the frame rate and the sampling rate, constructing a time axis for the silent video and the generated audio at the same time; aligning the starting audio waveform data of the generated audio with the first frame video frame of the silent video on the time axis; calling an audio and video packaging tool to package the aligned video frame sequence of the silent video and the audio waveform data stream of the generated audio into the same container file to obtain an audio and video synchronized audio and video file.

8. The method according to any one of claims 1 to 5, characterized in that, Before generating the static semantic condition vector and the dynamic change vector based on the diffusion generation model, the method further comprises: training the diffusion generation model by the following steps: pre-training an initial diffusion model by a first sample data set to obtain a text-generated audio model; the first sample data in the first sample data set is a sample data pair composed of a sample text and a sample audio; fine-tuning the text-generated audio model by a second sample data set to obtain the diffusion generation model; the second sample data in the second sample data set is a sample data pair composed of a sample video and the sample audio.

9. The method of claim 8, wherein, The fine-tuning of the text-generated audio model by the second sample data set to obtain the diffusion generation model comprises: performing first encoding processing on each sample video frame of the sample video by an encoder with frozen parameters in an audio variational autoencoder to obtain a sample frame embedding feature vector of each sample video frame; the sample video is a silent video without audio information; performing embedding differential processing on the sample frame embedding feature vector of each sample video frame to obtain a sample dynamic change vector for representing dynamic change information of the sample video; performing second encoding processing on each sample video frame in the sample video by the encoder with frozen parameters in the audio variational autoencoder to obtain a sample static semantic condition vector for representing static semantic information of the sample video; generating a sample audio latent representation vector based on the sample static semantic condition vector and the sample dynamic change vector by the text-generated audio model; fine-tuning the parameters of the text-generated audio model based on the sample audio latent representation vector and the latent representation vector corresponding to the sample audio to obtain the diffusion generation model.

10. A video processing device, comprising: The device comprises: an encoding module configured to perform first encoding processing on each video frame in a to-be-processed video by a video encoder to obtain a frame embedding feature vector of each video frame; the to-be-processed video is a silent video without audio information; The difference processing module is configured to perform embedding-level difference processing on the frame embedding feature vector of each video frame to obtain a dynamic change vector used to represent dynamic change information of the to-be-processed video. The encoding module is further configured to perform second encoding processing on each video frame in the to-be-processed video by using a video encoder to obtain a static semantic condition vector used to represent static semantic information of the to-be-processed video. The generating module is configured to perform iterative denoising generation on the static semantic condition vector and the dynamic change vector by using a diffusion generative model to obtain an audio latent representation vector. The decoding module is configured to perform decoding processing on the audio latent representation vector to obtain generated audio corresponding to the to-be-processed video.

11. An electronic device, comprising: The electronic device comprises: The memory is configured to store computer executable instructions or computer programs; and the processor is configured to execute the computer executable instructions or computer programs stored in the memory to implement the video processing method in any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the video processing method in any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the video processing method in any one of claims 1 to 9.

Images