Video processing method, apparatus, device, computer readable storage medium and computer program product

Abstract

The application provides a video processing method, device and equipment, computer readable storage medium and computer program product; the method comprises: performing frame-level difference processing on video frames in a to-be-processed video to obtain a visual change frame sequence; performing encoding processing on each visual change frame in the visual change frame sequence through a video encoder to obtain a dynamic change vector; performing encoding processing on each video frame in the to-be-processed video through the video encoder to obtain a static semantic condition vector used for representing static semantic information of the to-be-processed video; performing iterative denoising generation based on the static semantic condition vector and the dynamic change vector through a diffusion generative model to obtain an audio latent representation vector; and performing decoding processing on the audio latent representation vector to obtain generated audio corresponding to the to-be-processed video. Through the application, the matching accuracy and dynamic response capability of the generated audio in time sequence with the to-be-processed video can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, a video processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of multimodal generative models, the intelligent generation and fusion of video and audio has become a research hotspot. Among them, video-to-audio (V2A) generation aims to automatically generate audio signals that are consistent with the semantics and temporal sequence of the input video content, and is widely used in virtual reality, film and television production, and accessibility assistance.

[0003] In related technologies, deep learning models are typically used to extract visual features from video frames and generate corresponding audio based on these features. Mainstream methods rely on cross-modal pre-trained models, such as Contrastive Language–Image Pre-training (CLIP), to extract high-level semantic information from video frames and combine this with diffusion models to generate audio. However, CLIP models primarily focus on static semantics and struggle to effectively capture dynamic changes and temporal information in videos, resulting in a time-synchronization issue between the generated audio and the video actions. Summary of the Invention

[0004] This application provides a video processing method, apparatus, device, computer-readable storage medium, and computer program product. By performing frame-level differential processing on video frames in the video to be processed and encoding each visual change frame in the visual change frame sequence, frame-level visual change information is introduced as a dynamic auxiliary condition. This can more accurately reflect the temporal dimension information of the video, thereby significantly improving the timing matching accuracy and dynamic response capability of the generated audio with the silent video, and thus ensuring that the generated audio is synchronized with the action of the silent video in time.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video processing method, comprising: performing frame-level differential processing on video frames in a video to be processed to obtain a visual change frame sequence; the video to be processed is a silent video without audio information; encoding each visual change frame in the visual change frame sequence using a video encoder to obtain a dynamic change vector representing the dynamic change information of the video to be processed; encoding each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; performing iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and decoding the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0007] This application provides a video processing apparatus, comprising: a differential processing module, configured to perform frame-level differential processing on video frames in a video to be processed to obtain a visual change frame sequence; the video to be processed is a silent video without audio information; an encoding module, configured to encode each visual change frame in the visual change frame sequence using a video encoder to obtain a dynamic change vector representing the dynamic change information of the video to be processed; the encoding module is further configured to encode each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; a generation module, configured to perform iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and a decoding module, configured to decode the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0008] This application provides an electronic device, including: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-described video processing method.

[0009] This application provides a computer program product that stores computer-executable instructions or a computer program, which are used to cause a processor to execute the computer-executable instructions or the computer program to implement the above-described video processing method.

[0010] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which are used to cause a processor to execute the computer-executable instructions or computer programs to implement the above-described video processing method.

[0011] The above scheme has the following beneficial effects: In this embodiment, firstly, the visual difference features between adjacent video frames in the video to be processed are extracted by frame-level differential processing to construct a visual change frame sequence, thereby capturing short-term dynamic change information in the video; then, the static semantic condition vector and dynamic change vector are extracted by the video encoder respectively, and the two are input into the diffusion generation model as joint conditions to realize audio generation under the dual constraints of semantics and time. Thus, on the one hand, compared to traditional V2A methods that rely solely on static semantic features, this embodiment introduces dynamic change vectors that can characterize frame-level visual changes by performing frame-level differential processing and encoding on the video frames of the video to be processed. Using frame-level visual change information as dynamic auxiliary conditions, it can more accurately reflect the temporal dimension information such as the rhythm of video actions and the timing of event triggers in the video to be processed, thereby significantly improving the temporal matching accuracy and dynamic response capability between the generated audio and the video to be processed. On the other hand, this embodiment utilizes shared parameters of a unified video encoder to extract static semantic condition vectors and dynamic change vectors, thereby obtaining the static and dynamic features of the video to be processed. This avoids the complexity and error accumulation problems caused by additional networks or training phases, improving the stability and generalization capability of cross-modal generation while maintaining the simplicity of the video processing system. Attached Figure Description

[0012] Figure 1 This is an optional flowchart illustrating the video processing method provided in an embodiment of this application;

[0013] Figure 2 This is a schematic diagram of the implementation process of frame-level differential processing provided in the embodiments of this application;

[0014] Figure 3 This is a schematic diagram of the implementation process of iterative denoising generation provided in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram illustrating the implementation process of iterative denoising generation in the audio latent space provided in the embodiments of this application;

[0016] Figure 5 This is a system architecture diagram of a video-to-audio generation system based on visual change assistance provided in an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of the V2A inference module provided in the embodiments of this application;

[0018] Figure 7 This is a structural block diagram of the video processing apparatus provided in the embodiments of this application;

[0019] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application. In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal and can be implemented, in whole or in part, using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of that module or unit.

[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the application. When applying these embodiments, the data collection and processing should strictly comply with the requirements of relevant laws and regulations, obtain informed consent or separate consent from the data subject, and conduct subsequent data use and processing within the scope of laws, regulations, and the authorization of the data subject.

[0023] Before describing the simulator-based data processing method provided in the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0024] 1) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0025] 2) Frame-level differential processing: A method for extracting visual change information between video frames. This method generates visual change frames representing the differences between frames by calculating the difference between pixels in two adjacent video frames. This processing method can capture dynamic change features such as object movement and scene switching in the video to be processed, providing a temporal synchronization basis for subsequent audio generation.

[0026] 3) Static semantic condition vector: This vector consists of high-level semantic features extracted from each video frame by the video encoder. This static semantic condition vector characterizes the overall content semantics of the video to be processed, such as scene category and main object types, and is an important conditional input for generating audio that is consistent with the semantics of the video content.

[0027] 4) Dynamic Change Vector: This vector, obtained by encoding the visual change frame sequence through a video encoder, represents the dynamic change information between video frames in the video to be processed. This dynamic change vector reflects fine-grained temporal information such as action rhythm and event triggering times in the video to be processed. As an auxiliary input to the diffusion generation model, it can enhance the matching accuracy between the generated audio and the video to be processed in the temporal dimension.

[0028] 5) Diffusion Generative Model: A deep learning-based generative model whose core idea is to progressively remove noise in the latent space to generate target data. In the embodiments of this application, the diffusion generative model generates an audio latent representation vector that conforms to the semantic and temporal characteristics of the video content by iteratively denoising the process and combining static semantic condition vectors with dynamic change vectors.

[0029] 6) Audio Latent Representation Vector: An intermediate representation generated after iterative denoising using the diffusion generation model, used to describe the key features of the generated audio. This vector can be decoded into the final audio signal, serving as a bridge between the video content and the generated audio.

[0030] 7) Parameter-frozen encoder / decoder: In this embodiment, a pre-trained audio variational autoencoder (VAE) is used, in which the encoder and decoder maintain unchanged parameters during the fine-tuning phase. This design ensures that the model retains its original pre-training capabilities while avoiding semantic drift or performance degradation caused by retraining.

[0031] 8) Silent video: refers to video that does not contain audio information, i.e., video material that only contains visual content. The objective of this application's embodiments is to automatically generate audio that matches the content of such silent videos, achieving audiovisual consistency.

[0032] 9) Video-to-Audio (V2A): This is short for video-to-audio generation, referring to the process of generating an audio signal that is semantically and temporally consistent with video content. This task is widely used in film and television post-production, game sound effect generation, virtual reality, and other fields.

[0033] 10) CLIP: A language-image contrastive pre-trained model capable of extracting semantically meaningful feature vectors from images. In this embodiment, CLIP is used as a video encoder to extract static semantic information from video frames, which serves as semantic guidance for generating audio.

[0034] 11) Text-to-Audio (T2A): This is a generative model that generates corresponding audio signals based on text descriptions. In this embodiment, the T2A model is used as a pre-trained model to provide generative priors for the V2A task, improving the naturalness and semantic matching ability of audio generation.

[0035] In related technologies, video-to-audio generation methods typically rely on deep learning models to extract visual features from video frames, using these as conditional signals to guide the audio generation process. Among these, the CLIP model, due to its powerful cross-modal semantic representation capabilities, is widely used to extract high-level visual semantic features from video frames, thus achieving good results in terms of semantic consistency. However, traditional CLIP-based V2A methods still have significant shortcomings in terms of time synchronization. The CLIP model mainly focuses on frame-level semantic abstraction information, making it difficult to capture fine-grained temporal dynamic changes in the video, such as object movement, scene transitions, or action rhythms. This can lead to the generated audio being out of sync with the video action in terms of timing, thus affecting the overall audiovisual consistency and realism.

[0036] To improve the temporal matching accuracy of audio generation, related technologies have explored introducing auxiliary conditional features in addition to CLIP features. These features can be divided into two categories: The first category consists of auxiliary features based on the audio target, such as predicting the downsampled Mel spectrogram, the start position of the sound event, or the audio energy curve of the target audio, to guide the diffusion model in generating temporally aligned audio signals. This type of method typically adds a feature prediction network to the existing video-to-audio diffusion model to generate auxiliary representations related to the temporal distribution of audio events. Specifically, the main process is as follows: First, the input video is encoded using a video frame extractor to obtain visual semantic features. Then, the additionally designed feature prediction network infers the temporal feature representation of the target audio based on the visual features, such as the downsampled Mel spectrogram, the start time of the sound event, or the audio energy curve. After obtaining these auxiliary features, the diffusion model simultaneously accepts the video encoded features and the predicted temporal feature representation as conditional inputs during the audio generation stage, thereby enhancing the structural constraints of the generated audio in the temporal domain and achieving temporal alignment with the video action. This scheme improves the temporal matching degree and rhythmic consistency of the audio generation process by explicitly introducing signals related to the audio temporal structure. However, such methods typically require additional neural networks and independent training phases, significantly increasing model complexity and inference costs. Furthermore, since auxiliary features are derived from the target audio rather than the video itself, the inference phase must first predict audio features before audio generation, increasing latency and reducing generation efficiency. Simultaneously, the accuracy of feature prediction directly impacts audio generation quality; large prediction errors can lead to a mismatch between audio rhythm and video content. The model also fails to fully utilize the inherent temporal variations in the video, resulting in insufficient modeling of the intrinsic relationship between visual motion and sound events.

[0037] The second category is based on auxiliary features from video input: for example, using Contrastive Audio-Visual Pre-training (CAVP) models or dedicated synchronization detection modules (such as Synchformer) to extract temporal information. This involves directly extracting temporal change information from the input video to enhance the temporal alignment of the generated audio. These methods typically add an additional video coding module or cross-modal synchronization detection network outside the CLIP model to capture dynamic changes by modeling the frame sequence of the video. Specific implementations include using a CAVP model to extract temporal features from the video, or employing a dedicated synchronization detection module to identify the correspondence between actions and sounds in the video. During the training phase, these models need to learn the correspondence between the video temporal structure and audio events through additional pre-training steps; during the generation phase, the model jointly inputs the language-image contrastive pre-trained features and additional temporal coding features into the diffusion generation network to enhance the synchronization between the generated audio and video image changes. However, these methods still rely on additional pre-training steps or specific network structures, leading to complex system implementation and high computational resource consumption. They require the introduction of additional video temporal coding networks or synchronization detection modules, increasing system complexity and parameter scale. Furthermore, because such models require independent pre-training or multi-stage training, their development cost is high, making them difficult to apply quickly in general video scenarios. The lack of a unified feature space between the video coding module and the language-image contrast pre-trained features leads to redundancy or conflicting information during conditional fusion, affecting generation stability. Simultaneously, this method does not fully utilize the direct visual differences between video frames and ignores dynamic change signals that can be easily obtained through frame-level differencing, resulting in low computational efficiency and insufficient temporal capture.

[0038] In summary, while significant progress has been made in achieving semantic consistency between video and audio generation, how to efficiently utilize the temporal variation information of the video itself to enhance the temporal alignment capability of audio generation without additional networks and training phases remains a pressing issue in the current technological field.

[0039] Based on at least one of the aforementioned problems in related technologies, this application provides a video processing method to address the issue of the inability to efficiently utilize the temporal variation information of the video itself to enhance the temporal alignment capability of audio generation. This video processing method aims to solve the following technical problem: While video-to-audio generation methods based on language-image contrast pre-trained models in related technologies can effectively ensure the semantic consistency between the generated audio and video content, they struggle to fully capture the fine-grained temporal dynamic changes in the video, leading to asynchrony between the generated audio and video actions in terms of rhythm and event triggering moments. Therefore, a feature extraction and conditional generation method is needed that can effectively utilize the temporal variation information of the video itself without adding additional network structures or training stages, in order to improve the temporal alignment accuracy and dynamic response capability of audio generation. Related technologies generally rely on external predictive features (such as audio energy curves, Mel spectrograms, or action start markers) or additional temporal coding networks to enhance temporal synchronization. This not only increases system complexity and computational overhead but also introduces the problem of prediction error accumulation. Therefore, a concise and computationally efficient conditional enhancement mechanism is needed that can directly extract representative dynamic change features of sound events from video frame sequences as auxiliary conditional inputs, thereby improving generation quality and temporal matching while keeping the model lightweight.

[0040] The video processing method provided in this application embodiment can be implemented using an electronic device, which can be a video processing device. The video processing device (i.e., electronic device) provided in this application embodiment can be implemented as a terminal or as a server. In one implementation, the video processing device provided in this application embodiment can be implemented as any terminal with video processing capabilities, such as a laptop, tablet, desktop computer, or intelligent robot. In another implementation, the video processing device provided in this application embodiment can also be implemented as a server, wherein the server can be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application embodiment does not impose any restrictions.

[0041] The video processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] Figure 1 This is an optional flowchart illustrating a video processing method provided in an embodiment of this application. This method can be applied to an electronic device, which can be a server. That is, the video processing methods in the embodiments of this application can be executed by a server, or they can be executed through interaction between the server and a terminal. The following will use an electronic device as a server as an example for illustrative explanation. Figure 1As shown, the method includes the following steps S101 to S105:

[0043] Step S101: Perform frame-level differential processing on the video frames in the video to be processed to obtain a visual change frame sequence; the video to be processed is a silent video that does not contain audio information.

[0044] Frame-level differential processing refers to calculating the differences in pixel values ​​between two adjacent video frames to extract dynamic features such as object movement and scene transitions in the video. For example, in an action video, when a character starts walking from a stationary state, a clear movement area appears between adjacent video frames. This movement area is highlighted after frame-level differential processing, forming a visual change frame. Frame-level differential processing can extract short-term visual changes in the video, providing a time synchronization basis for subsequent audio generation. The result of frame-level differential processing is a sequence of visual change frames, recording the visual differences between each frame and the previous frame in the video. Frame-level differential processing effectively captures fine-grained temporal dynamic changes in the video and provides conditional input for the audio generation system to maintain consistency with the video content in terms of rhythm and event triggering moments.

[0045] In some embodiments, see Figure 2 , Figure 2 The step S101, which involves performing frame-level differential processing on video frames in the video to be processed to obtain a sequence of visually changing frames, can be achieved through the following steps S1011 and S1012:

[0046] Step S1011: Perform video frame sampling on the video to be processed to obtain a video frame sequence composed of multiple video frames.

[0047] Video frame sampling refers to extracting keyframes from the original video to be processed according to certain sampling intervals or rules to form a continuous video frame sequence. The video frame sampling process can be implemented through time steps (e.g., sampling one frame every n frames) or adaptive sampling methods based on content changes. The purpose of video frame sampling is to reduce computational burden while retaining sufficient temporal information to support subsequent frame-level differential processing. For example, in a 60 frames per second video, video frame sampling can be performed every 5 frames to generate a 12 frames per second video frame sequence for subsequent processing.

[0048] In this embodiment, video frame sampling of the video to be processed can be achieved as follows: First, determine the sampling interval frame number; then, sample the video to be processed according to the sampling interval frame number to obtain a video frame sequence composed of multiple video frames. The sampling interval frame number refers to how many frames in the video frame sequence are extracted as the sampling object. For example, if the sampling interval frame number is set to 5, then one frame is extracted every 5 frames for subsequent processing. This parameter determines the density and temporal resolution of the final sampled video frame sequence. The choice of this parameter directly affects the temporal alignment accuracy and computational efficiency of the generated audio. A smaller sampling interval frame number can capture dynamic changes in the video more precisely, but it increases the data volume and processing burden; a larger sampling interval frame number may lead to audio generation temporal distortion, but it can reduce computational resource consumption. In practical applications, the value of the sampling interval frame number can be adjusted according to specific task requirements (such as whether to focus on fast actions, scene switching frequency, etc.). By reasonably setting the sampling interval frame number and performing video frame sampling according to this setting, the model complexity can be controlled while ensuring audio generation quality, and video processing efficiency can be improved.

[0049] Video frame sampling refers to selecting video frames that conform to a predetermined sampling interval from the original video to be processed, generating a sparse but representative sequence of video frames. The sampled video frames serve as the basis for subsequent visual feature extraction and conditional input. There is a direct data dependency between video frame sampling and the sampling interval. Video frame sampling is the result of extracting a frame sequence from the original video based on the interval rules set by the sampling interval. This mechanism of sampling video frames according to the interval rules ensures that the frame sequence obtained by this method reflects both the overall semantic content of the video to be processed and retains key temporal dynamic information. In practice, the video frame sampling process can be implemented through a video decoder. The video decoder reads the frame information of the video frames at specified intervals and caches this frame information as image data in a uniform format for use by subsequent modules. For example, in a 60 frames per second video, if the sampling interval is 5 frames, only 12 frames per second are extracted for processing.

[0050] In this embodiment, by reasonably setting the sampling interval frame number and performing video frame sampling according to the reasonably set sampling interval frame number, the preservation of video dynamic information and the consumption of computing resources can be effectively balanced. Furthermore, by reasonably setting the sampling interval frame number and performing video frame sampling, it can be ensured that the generated audio is synchronized with the video frame in time, thereby accurately matching the rhythm of actions and the timing of event triggers in the video.

[0051] Step S1012: Perform frame-level differential processing on every two adjacent video frames in the video frame sequence to obtain a visual change frame sequence.

[0052] Frame-level differencing is a method for detecting pixel-level differences between video frames. It highlights moving areas and dynamic changes by subtracting pixel values ​​from adjacent frames. Frame-level differencing can quickly capture short-term dynamic information such as motion points and scene transitions in the video being processed, and convert the results into a time-synchronization signal that can be used for audio generation. For example, in a video where a person is moving, the frame-level differencing result will show motion trajectories and speed changes, which can provide a basis for generating audio rhythms that match these motion trajectories and speed changes.

[0053] In this embodiment, by sampling video frames and performing frame-level differential processing on the video to be processed, a sequence of visually changing frames reflecting short-term dynamic changes in the video can be extracted without introducing additional network structures. Through the aforementioned video frame sampling and frame-level differential processing, precise temporal constraints can be provided for the diffusion generation model, thereby improving the synchronization accuracy in the audio generation process and enhancing the overall audiovisual consistency and immersive experience.

[0054] In this embodiment, frame-level differential processing of every two adjacent video frames in the video frame sequence can be achieved in the following way: First, for any two adjacent video frames in the video frame sequence, obtain the pixel value of each pixel in each video frame of the two video frames; then, perform differential calculation on the pixel values ​​of pixels at the same pixel position in the two video frames to obtain the visual change frames corresponding to the two video frames; finally, sort the visual change frames corresponding to every two adjacent video frames according to their order in the video frame sequence to form a visual change frame sequence.

[0055] A video frame sequence is a temporal sequence composed of continuously captured or generated image frames, used to represent dynamic visual content. Each frame represents a static image at a specific point in time, while the entire video frame sequence together constitutes the moving image. In this embodiment, the video frame sequence serves as input data, and its quality and coherence directly affect the time synchronization accuracy of subsequent audio generation.

[0056] A pixel is the basic unit that makes up a digital image. Each pixel has a specific coordinate position and carries color or grayscale information. A pixel value is a set of numerical values ​​describing the color intensity of a pixel. It can be stored in RGB (red, green, blue) three-channel format or as a single-channel grayscale value. In this embodiment, the server can quantify the visual changes within the region formed by pixels at the same position in two adjacent video frames by extracting the pixel values ​​of pixels at the same position in those two frames.

[0057] Differential calculation involves subtracting the pixel values ​​at the same location in two adjacent video frames to obtain the change in pixel values ​​at that location between adjacent frames. Differential calculation can effectively capture dynamic changes within local regions, such as object movement, lighting changes, or background transitions. The results of differential calculation reflect the intensity of visual differences between frames and can be used to construct frames showing visual changes.

[0058] Visual change frames are new image frames generated through inter-frame differencing. The pixel values ​​of a visual change frame represent the degree of visual change between adjacent video frames. Visual change frames highlight areas of action or change in the video being processed, helping to identify key events or motion trajectories. As dynamic conditional signals, visual change frames guide the diffusion generation model to adjust the timing and duration of sound events according to the rhythm of visual changes. A visual change frame sequence is a sequence of all visual change frames arranged chronologically, reflecting the frame-by-frame visual change process throughout the video. By recombining visual change frames according to the playback order of the original video frames, temporal continuity and logical consistency can be maintained, ensuring a high degree of temporal matching between the generated audio and video content.

[0059] In practical applications, the video processing method of this application embodiment can be used for automatic sound effect generation in virtual reality scenes. For example, in a video of a character running, the server extracts the changes in the character's position between each frame, generates corresponding visual change frames, and generates environmental sound effects such as footsteps and wind sounds based on the visual change frames. Furthermore, the video processing method of this application embodiment is also applicable to anomaly detection and sound prediction in intelligent monitoring systems. When events such as collisions or falls occur in the video, the visual change frame sequence can capture the visual features of these sudden events, and drive a diffusion generation model to generate and ultimately output corresponding alarm sounds or prompts, thereby achieving multimodal early warning functionality.

[0060] In practical implementation, there is a close logical relationship between the various steps of the entire frame-level differential processing. First, the pixel values ​​of each pixel in each video frame are acquired to provide basic data for subsequent differential calculations. Second, differential calculations are performed based on these pixel values ​​to generate visual change frames that reflect visual changes. Finally, all visual change frames are reorganized in chronological order to form a visual change frame sequence, which serves as the input for the subsequent video encoder. The entire process ensures accurate capture of visual changes and precise control of time synchronization. In summary, in this embodiment, by performing pixel-level differential calculations on adjacent video frames and constructing a visual change frame sequence, dynamic change features in the video can be accurately captured. By constructing a visual change frame sequence, this embodiment can enhance the sensitivity of the diffusion generation model to video rhythm and event triggering time, thereby improving the matching accuracy between the generated audio and video content in the semantic and temporal dimensions, and ultimately achieving high-quality video-to-audio generation effects.

[0061] In some embodiments, after performing frame-level differential processing on every two adjacent video frames in the video frame sequence to obtain a visual change frame sequence, post-processing can be performed on each visual change frame in the visual change frame sequence to obtain a processed visual change frame sequence; here, post-processing includes at least one of the following: absolute value processing, thresholding processing, and normalization processing.

[0062] Here, post-processing of each visual change frame in the visual change frame sequence refers to optimizing and standardizing the original visual change frame sequence to improve its usability and stability in subsequent encoding and generation processes. Absolute value processing involves calculating the absolute value of the difference between each pixel in the visual change frame to eliminate interference from positive and negative differences. Absolute value processing can enhance the contrast of moving areas in the visual change frame. For example, in a visual change frame, if two adjacent pixels have brightness values ​​of 120 and 115 respectively, the difference between them is 5. After absolute value processing, the result remains 5, which helps to highlight dynamic change information. Thresholding processing involves setting pixels in the visual change frame below a set threshold to zero (indicating no significant change), while pixels above the set threshold retain their original values ​​(indicating significant change). The above threshold-based processing can filter out weak noise and retain meaningful motion information. For example, when the threshold is set to 10, the server can consider all pixel changes in visual change frames less than 10 as invalid, retaining only pixel changes greater than or equal to 10 for subsequent processing. Normalization refers to mapping the numerical range of visual change frames to a fixed interval (such as [0,1] or [-1,1]) to ensure the comparability of data between different visual change frames and to avoid affecting the training and generation quality of the model due to excessively large or small values. Normalization can be performed using methods such as min-max normalization or Z-score normalization (standard deviation normalization).

[0063] Post-processing each visual change frame in a visual change frame sequence can effectively improve the quality of the sequence and make it more suitable for subsequent encoding and generation tasks. For example, in practical applications, when the original visual change frame sequence contains a large amount of tiny random noise, this noise may cause computer vision models to misjudge motion events. Thresholding can remove these invalid signals, ensuring that only genuine motion changes are preserved. Furthermore, post-processing each visual change frame in the sequence can improve the consistency of visual change frame sequences acquired under different devices or shooting conditions. For example, visual change frames captured in low-light environments may have lower contrast, but absolute value processing and normalization can enhance the dynamic range of the visual change frames, making the visual change information clearer and more discernible.

[0064] A video encoder is a module used to extract semantic information from each visual change frame in a sequence of visual change frames. The video encoder can be implemented based on the CLIP model. In this embodiment, the video encoder receives a sequence of visual change frames after post-processing each visual change frame as input, and encodes the received sequence of visual change frames into a high-dimensional vector, which is used as the conditional input to the diffusion generation model. Because post-processing each visual change frame in the sequence improves the accuracy and stability of the visual change frames, the video encoder can more effectively capture dynamic information related to audio events, thereby guiding the diffusion generation model to output more accurate time-aligned audio.

[0065] Specifically, during the encoding process, the video encoder performs an embedding operation on each visual change frame in the sequence, converting the visual change information into a feature representation that the diffusion generation model can understand. This feature representation is then fed into the diffusion generation model to guide the audio generation process. After post-processing each visual change frame in the sequence, irrelevant information is removed and key dynamic features are enhanced. Therefore, the video encoder can more efficiently extract semantic and temporal information related to audio events, thereby improving the naturalness and synchronization of the final generated audio.

[0066] In summary, in this embodiment, the visual change frame sequence is optimized by post-processing each visual change frame. The processed data is then input into the video encoder. This optimization method improves the accuracy and stability of the visual change information. Because the accuracy of the visual change information is improved, the diffusion generation model can be guided more effectively, ultimately achieving higher precision audio-visual time synchronization and dynamic response capabilities.

[0067] Step S102: Each visual change frame in the visual change frame sequence is encoded by a video encoder to obtain a dynamic change vector that represents the dynamic change information of the video to be processed.

[0068] In practice, a video encoder can be a feature extractor built on the CLIP model. This feature extractor extracts high-level semantic features from visually changing frames and maps these features into vector forms that can be used as input to the diffusion generation model. For example, for a given visually changing frame, the video encoder can identify dynamic behaviors such as a person running forward or a vehicle accelerating, and convert these behaviors into corresponding numerical representations. The dynamic change vector is an abstract description of the dynamic events in the video being processed. It contains not only physical features such as the direction and speed of motion, but also semantic information such as action type and behavior pattern. The information contained in the dynamic change vector can serve as auxiliary input to the diffusion generation model, guiding it to generate audio signals corresponding to the dynamic changes in the video within the latent space. In practice, the dynamic change vector provides temporal information corresponding to the rhythm of video actions and the timing of event triggers. This temporal information is used to ensure that the generated audio is highly consistent with the video footage in time, thus significantly improving audiovisual consistency and naturalness.

[0069] In some embodiments, since after performing frame-level differential processing on every two adjacent video frames in the video frame sequence to obtain a visual change frame sequence, each visual change frame in the visual change frame sequence can also be post-processed to obtain a processed visual change frame sequence. Therefore, when encoding each visual change frame in the visual change frame sequence by a video encoder, the video encoder can be used to encode each visual change frame in the processed visual change frame sequence.

[0070] In this embodiment of the application, each visual change frame in the visual change frame sequence is encoded by a video encoder, which can be achieved in the following way: each visual change frame in the visual change frame sequence is encoded by a video encoder with parameters frozen in the audio variational autoencoder, to obtain a dynamic change vector that represents the dynamic change information of the video to be processed.

[0071] An Audio Variational Autoencoder (AVAE) is a deep generative model for audio signal modeling, capable of compressing audio signals into low-dimensional latent representations and reconstructing the original audio during decoding. The AVAE used in this embodiment has been pre-trained. The parameter-frozen video encoder portion within the AVAE is set to a parameter-frozen state, meaning its weight parameters are no longer updated. This parameter-frozen video encoder can be built based on a large-scale pre-trained CLIP model, possessing the ability to extract high-level semantic features from images or video frames.

[0072] Parameter freezing refers to fixing the parameters of certain network layers during the model inference phase to prevent these parameters from participating in gradient updates. In this embodiment, parameter freezing is used in the audio variational autoencoder, setting the parameters of the video encoder to a frozen state. This means that the video encoder will not undergo structural changes or output format adjustments due to the data used in the current task. This design ensures that the video encoder in the audio variational autoencoder, in its parameter-frozen state, can continuously and stably extract features consistent with the training data distribution, thereby avoiding performance degradation or overfitting problems caused by fine-tuning operations.

[0073] By using a parameter-frozen video encoder within a parameter-frozen audio variational autoencoder, the diffusion generation model's ability to perceive video temporal dynamics can be improved, thereby enhancing the temporal consistency between the generated audio and video actions and improving the overall audio-video coordination. Furthermore, video processing systems can maintain model structural stability, reduce computational overhead, and facilitate deployment on resource-constrained devices by using a parameter-frozen video encoder within the parameter-frozen audio variational autoencoder.

[0074] Step S103: Each video frame in the video to be processed is encoded by a video encoder to obtain a static semantic condition vector that represents the static semantic information of the video to be processed.

[0075] The static semantic conditional vector (SCD) consists of high-level semantic features extracted from each video frame by the video encoder. The SCD characterizes the overall semantic content of the video to be processed, including scene category, main object types, and environmental atmosphere. For example, in a video depicting a forest scene, the video encoder might extract semantic information such as trees, animals, and natural light, and convert this information into numerical vector representations. The SCD provides a description of the global semantic background of the video content, ensuring that the generated audio remains semantically consistent with the video content. For instance, if the video depicts a city street scene, the generated audio might include traffic sounds and crowd noise; if the video depicts a forest scene, the generated audio might include birdsong and wind sounds. In practice, the SCD ensures that the generated audio remains semantically consistent with the video content, thereby improving the overall generation quality and naturalness.

[0076] In this embodiment of the application, the process of encoding each video frame in the video to be processed by a video encoder to obtain a static semantic condition vector that represents the static semantic information of the video to be processed can also be implemented in the following way: encoding each video frame in the video to be processed by a video encoder with parameters frozen in an audio variational autoencoder to obtain a static semantic condition vector that represents the static semantic information of the video to be processed; wherein, the video encoder that encodes each visual change frame can share the same encoder parameters with the video encoder that encodes each video frame.

[0077] Here, the static semantic conditional vector can be a set of feature vectors obtained by encoding each video frame in the input video to be processed by the parameter-frozen video encoder in the audio variational autoencoder, used to describe the semantic information of the video content. In this embodiment, the parameter-frozen video encoder in the audio variational autoencoder also operates in a parameter-freezing manner to ensure that the semantic features output by the parameter-frozen video encoder in the audio variational autoencoder are not affected by the current task. Since the parameter-frozen video encoder in the audio variational autoencoder has been fully pre-trained, the semantic features output by the parameter-frozen video encoder in the audio variational autoencoder have high generalization ability and robustness, and can provide stable semantic descriptions in different scenarios.

[0078] It's important to note that sharing the same encoder parameters means using the same parameter-frozen video encoder within the audio variational autoencoder, regardless of whether visually changing frames or ordinary video frames are being encoded. This implies that both encoding processes share the same network structure and parameter configuration, eliminating the need to train different encoders for different types of video frames. This design offers several advantages: First, it simplifies the model structure, reduces the number of parameters and computational complexity, and helps lower latency and reduce hardware resource consumption in video processing systems. Second, because the video encoder parameters remain consistent, the two types of feature vectors can be fused in a unified feature space, avoiding potential feature conflicts or redundancy issues between different video encoders.

[0079] By efficiently integrating video semantic information with dynamic change information, the diffusion generative model can better understand video content. Using parameter-frozen video encoder parameters in the audio variational autoencoder further reduces the cost of model deployment and maintenance, and improves the overall efficiency and stability of the system. In practice, there is a complementary relationship between the dynamic change vector and the static semantic condition vector: the dynamic change vector emphasizes the temporal changes in the video being processed, while the static semantic condition vector focuses on the spatial semantic content. The combination of the dynamic change vector and the static semantic condition vector enables the diffusion generative model to simultaneously perceive what happened and how it happened, thereby generating more natural and highly synchronized audio signals. Furthermore, the design of sharing parameter-frozen video encoder parameters in the audio variational autoencoder ensures semantic consistency between the dynamic change vector and the static semantic condition vector, avoiding representational bias caused by using different encoders.

[0080] In this embodiment, the video encoder with parameters frozen in the audio variational autoencoder is used to extract dynamic change vectors and static semantic condition vectors. Feature fusion is achieved by sharing the parameters of the video encoder with parameters frozen in the audio variational autoencoder. This operation mode enables the video processing system to effectively combine the semantic information and temporal dynamic information of the video content, thereby improving the cross-modal understanding and generation capabilities of the diffusion generation model.

[0081] Step S104: Using a diffusion generation model, iterative denoising generation is performed based on static semantic condition vectors and dynamic change vectors to obtain audio latent representation vectors.

[0082] Diffusion generative models are a type of generative model based on deep learning. The core idea of ​​diffusion generative models is to progressively remove noise from the latent space to generate target data. In the embodiments of this application, the diffusion generative model generates audio latent representation vectors that conform to the semantic and temporal characteristics of video content by iteratively denoising the model and combining static semantic conditional vectors with dynamically changing vectors.

[0083] Specifically, the diffusion generation model first inputs initial noise into the latent space and determines the overall semantic direction of the audio based on the static semantic condition vector. Then, it adjusts the audio's structure and rhythm in the temporal dimension based on the dynamic change vector. After multiple rounds of denoising iterations, the final generated audio latent representation vector accurately reflects the semantic and temporal characteristics of the video content. In practical implementation, the diffusion generation model can fuse static semantics and dynamic change information in a unified feature space, thereby significantly improving the semantic consistency and temporal alignment accuracy of audio generated by the diffusion generation model without introducing additional networks or training stages.

[0084] In some embodiments, see Figure 3 , Figure 3 The process of iterative denoising generation based on static semantic condition vectors and dynamic change vectors using the diffusion generation model in step S104 is shown and can be implemented through the following steps S1041 and S1042:

[0085] Step S1041: Randomly generate a noise tensor that conforms to a normal distribution.

[0086] A noise tensor is a set of random numerical values ​​composed of multiple dimensions, used to simulate uncertainty and randomness in the real world. In this embodiment, the noise tensor is used to initialize the audio generation process to ensure that the diffusion generation model can perform effective denoising operations in the latent space. The normal distribution is a probability distribution function with symmetry and centrality. In this embodiment, the normal distribution is used to generate initial noise to ensure the uniform distribution of the noise tensor in the latent space, thereby improving the stability and controllability of the subsequent denoising process. In practical applications, the server generates a random tensor based on preset noise dimensions (such as time length, bandwidth, etc.) and sends this random tensor as the initial input to the diffusion generation model. For example, when generating a 5-second audio clip, the server may generate a time-series tensor of length 512, where each element follows a normal distribution with a mean of 0 and a standard deviation of 1. By introducing a noise tensor that conforms to a normal distribution, a stable and controllable starting point can be provided for the diffusion generation model. Introducing this noise tensor helps improve the naturalness and continuity of the generated audio. It should be noted that the introduction of this noise tensor avoids the problem of generation quality fluctuations caused by inconsistent initial states.

[0087] Step S1042: According to the preset total number of diffusion steps, the audio latent representation vector is obtained by iteratively denoising and generating the audio latent representation vector in the audio latent space based on the noise tensor, static semantic condition vector and dynamic change vector through the diffusion generation model.

[0088] In this embodiment, the diffusion generation model can be a deep learning model based on Markov chains. It transforms the original data into noise by progressively adding noise, and then removes the noise through a reverse process to generate new data. In this embodiment, the diffusion model is used for audio generation tasks. The core of the diffusion model lies in utilizing visual information (static semantic condition vectors and dynamic change vectors) as conditional signals, with the visual information guiding the generation process.

[0089] The audio latent space refers to a low-dimensional latent space to which audio data is mapped after processing by a compression encoder (such as a VAE). In the audio latent space, the temporal structure and frequency characteristics of the audio are preserved, thus facilitating cross-modal fusion and generative control.

[0090] The static semantic condition vector (SMT) is a high-level semantic feature extracted from video frames by the CLIP model. It reflects the overall content and object categories of the scene and guides the generated audio to maintain semantic consistency with the video. The dynamic change vector (DVC) is obtained by calculating the visual differences between adjacent frames. It can capture temporal dynamic information such as the rhythm of actions and the movement of objects in the video, thereby enhancing the temporal synchronization and detail richness of the generated audio. In practice, the server jointly inputs the above three inputs (noise tensor, static semantic condition vector, and dynamic change vector) into the diffusion generation model and iterates the denoising step by step according to the set total number of diffusion steps (e.g., 100 or 500 steps). In each step, the diffusion generation model adjusts its output according to the current latent state and conditional signals, ultimately generating an audio latent representation vector that highly matches the content of the video to be processed. By integrating the static semantic condition vector and the dynamic change vector into the conditional input of the diffusion generation model, the generated audio is not only semantically consistent with the video but also precisely aligned with the video actions in time. This mechanism significantly improves the consistency and immersion of audio and video and avoids the complexity and error accumulation problems caused by relying on additional networks in traditional methods.

[0091] In this embodiment, by introducing a noise tensor conforming to a normal distribution and using static semantic condition vectors and dynamic change vectors as conditional inputs to the diffusion generation model, a more stable and natural audio generation process can be achieved. Furthermore, it can effectively improve the semantic and temporal matching accuracy between the generated audio and video content, thereby enhancing the overall consistency and immersive experience of the audio and video.

[0092] In some embodiments, see Figure 4 , Figure 4 The process of iterative denoising generation in step S1042, which involves using a diffusion generation model to generate noise in the audio latent space based on noise tensors, static semantic condition vectors, and dynamic change vectors according to a preset total number of diffusion steps, can be achieved through steps S10421 to S10424:

[0093] In step S10421, during each iteration of denoising, the static semantic condition vector and the dynamic change vector corresponding to the current time step are concatenated to obtain the current concatenated vector.

[0094] The static semantic condition vector is high-level visual semantic information extracted from the input video frames, used to describe the overall scene, object category, action type, and other semantic aspects of the video to be processed. The dynamic change vector is calculated by analyzing the differences between adjacent video frames, capturing short-term motion, object movement, scene transitions, and other temporal information in the video. The concatenation operation connects the static semantic condition vector and the dynamic change vector along the feature dimension, forming a comprehensive condition vector that integrates semantic and temporal dynamic information. The purpose of the concatenation operation is to enable the diffusion generation model to understand the semantic content of the video and perceive its temporal trends during audio generation, thereby generating more natural and synchronized audio signals. The concatenation process can be performed by the server using a simple linear concatenation method, or it can incorporate attention mechanisms or other non-linear fusion methods. The server can adjust the specific implementation of the concatenation process according to the actual application scenario. For example, in some cases, the dynamic change vector can be used only for specific time steps, while relying solely on the static semantic condition vector at other time steps, thereby reducing computational overhead and maintaining generation stability.

[0095] Step S10422: Input the current noisy first latent representation vector, the current time step, and the current concatenation vector into the diffusion generation model to obtain the predicted noise.

[0096] In the first iteration of denoising, the first latent representation vector is the noise tensor; in the i-th iteration of denoising, the first latent representation vector is the denoised vector obtained after the (i-1)-th iteration of denoising; i is an integer greater than 1.

[0097] The diffusion generation model in this embodiment is a deep learning model based on iterative denoising. The core idea of ​​this model is to progressively remove noise from the latent space to ultimately recover a clear output signal. In each iteration, the diffusion generation model receives three main inputs: the noisy latent representation vector at the current time step, the identifier of the current time step, and the current concatenation vector. Based on the noisy latent representation vector, the identifier of the current time step, and the current concatenation vector, the diffusion generation model predicts the noise component at the current time step and uses this prediction result to generate the denoised latent representation vector.

[0098] In the initial stage, i.e., the first iteration, the first latent representation vector is a randomly generated noise tensor, representing a completely unstructured audio signal. During each iteration, the diffusion generation model generates the latent representation vector for the current time step based on the denoising result of the previous time step. This process allows the diffusion generation model to progressively approximate the latent representation of the target audio. This iterative mechanism allows the diffusion generation model to be updated with the latest semantic and temporal information at each step, thereby continuously optimizing the temporal alignment and semantic consistency of the audio during the generation process. Furthermore, because the diffusion generation model considers the information of the current time step in each iteration, it can better respond to instantaneous events and rhythmic changes in the video being processed.

[0099] Step S10423: Use the predicted noise to determine the second potential representation vector after removing part of the noise, and determine the second potential representation vector as the denoising vector at the current time step.

[0100] After obtaining the predicted noise, the diffusion generation model subtracts the predicted noise from the current noisy latent representation vector to obtain a denoised second latent representation vector. The process of subtracting the predicted noise from the current noisy latent representation vector is essentially a correction to the current latent state, making it closer to the true latent representation of the target audio.

[0101] The denoising vector reflects the best estimate of the latent audio representation by the diffusion generation model at the current time step. It serves as the basis for subsequent iterations and is a crucial component of the final generated audio. By continuously updating the denoising vector, the diffusion generation model can progressively approach the ideal audio output throughout the diffusion process. It's important to note that the denoising process is not completed in one step but through multiple iterations. Each iteration brings some improvement but is also affected by the noise level. Therefore, the diffusion generation model needs to balance denoising intensity with preserving the integrity of the original signal at each step to ensure that the final generated audio is both high-quality and retains its original semantic and temporal characteristics.

[0102] Step S10424: In response to the current time step being equal to the total number of diffusion steps, the denoising vector at the current time step is determined as the audio latent representation vector.

[0103] After the diffusion generation model completes all preset iterations, the denoised vector obtained in the last iteration is considered the final audio latent representation vector. This denoised second latent representation vector can be directly used in subsequent decoding operations to reconstruct an audible audio waveform. The total number of diffusion steps is a pre-defined parameter that determines the number of iterations and accuracy of the diffusion generation model. More total diffusion steps result in higher audio quality, but also increase computational cost. Therefore, in practical applications, an appropriate total number of diffusion steps can be selected based on specific hardware resources and performance requirements.

[0104] In this embodiment of the application, by using the final denoised vector as the audio latent representation vector, it can be ensured that the generated audio is not only semantically consistent with the content of the video to be processed, but also highly synchronized with the actions and rhythm of the video to be processed in time.

[0105] In this embodiment, the static semantic condition vector and the dynamically changing vector are concatenated in each iterative denoising step to form the current concatenated vector. This concatenated vector is then input into the diffusion generation model along with the current time step and the noisy latent representation vector to obtain the predicted noise and generate the denoised vector. Finally, after the total number of diffusion steps is completed, the audio latent representation vector is output. By concatenating the static semantic condition vector and the dynamically changing vector in each iterative denoising step to form the current concatenated vector, and then inputting it into the diffusion generation model along with the current time step and the noisy latent representation vector, both the static semantic information and the dynamic temporal information of the video can be utilized simultaneously. This significantly improves the semantic matching degree and temporal alignment accuracy of the audio generation, thereby achieving a more natural and realistic audio-visual synchronization effect.

[0106] Step S105: Decode the audio latent representation vector to obtain the generated audio corresponding to the video to be processed.

[0107] In this embodiment, the audio latent representation vector is restored to an audible audio waveform by a decoder. The decoder can use a parameter-frozen audio variational autoencoder (VAE), where the encoder and decoder of the audio variational autoencoder (VAE) maintain unchanged parameters during the fine-tuning phase. This design ensures that the model retains its original pre-training capabilities while avoiding semantic drift or performance degradation caused by retraining the audio variational autoencoder (VAE).

[0108] The decoded audio output can take various forms, such as speech, music, and ambient sound effects. The specific form of the audio output depends on the semantic and dynamic characteristics of the video content being processed. For example, if the video shows a person running, the generated audio might be a mixture of footsteps and breathing sounds; if the video shows a car driving through an intersection, the generated audio might be a combination of engine sounds and horn sounds.

[0109] In practical implementation, the audio decoding process can restore the audio representation in the latent space into high-quality audible audio, thus achieving an audio generation effect that is highly consistent with the input video in both semantics and time. During implementation, the audio latent representation vector can be decoded using a parameter-frozen decoder in the audio variational autoencoder to obtain the generated audio corresponding to the video to be processed. In this embodiment, the decoder restores the latent representation vector to the original audio waveform; that is, the audio variational autoencoder is used to decode the latent audio representation generated by the diffusion generation model into audible audio output. The parameter-frozen decoder design means that the parameters of the decoder are fixed during training and are no longer updated.

[0110] In practical applications, after the diffusion model completes iterative denoising and generates the final latent audio representation vector, this latent audio representation is input into the decoder of the audio VAE. Since the parameters of the audio VAE decoder are already frozen, high-quality audio waveforms can be directly output without additional training. Using the audio VAE decoder for audio generation improves the inference efficiency of audio generation and reduces the computational resources consumed during the audio generation process, making it suitable for resource-constrained devices and real-time applications. Furthermore, the audio variational autoencoder can also serve as part of audio quality assessment. By performing spectral analysis and time-domain waveform comparison on the audio signal output by the parameter-frozen decoder, the generation strategy can be further optimized, thereby improving the realism and naturalness of the audio signal.

[0111] In this embodiment, the audio latent representation vector is decoded using a parameter-frozen decoder in an audio variational autoencoder. This decoding process ensures the stability and consistency of audio generation. After decoding the audio latent representation vector using the parameter-frozen decoder, an audio output that highly matches the video content semantically and temporally can be efficiently reconstructed. Reconstructing an audio output that highly matches the video content semantically and temporally significantly improves the overall audio-visual collaborative experience.

[0112] The video processing method provided in this application is a video-to-audio generation method. This method introduces frame-level differential processing to extract dynamic change features (i.e., dynamic change vectors), and then jointly inputs these dynamic change features with static semantic features (i.e., static semantic conditional vectors) into a diffusion generation model. This significantly improves the performance of the generated audio in terms of time synchronization, rhythmic consistency, and dynamic response capabilities without requiring additional network structures or training steps. The video processing method provided in this application not only improves the consistency of audio and video content but also reduces system complexity and computational overhead, possessing good practical value and promising prospects for widespread application.

[0113] The video processing method provided in this application has wide application value in various video-to-audio generation fields, such as post-production audio and automatic sound effect generation for short videos, immersive sound field rendering in virtual reality, audio early warning for security anomalies, synchronized lip-syncing and voice-over for digital humans and virtual anchors, and audio diagnosis for industrial equipment status monitoring and fault diagnosis. The video processing method provided in this application can generate audio corresponding to silent videos. The following examples illustrate the application scenarios of the video processing method provided in this application.

[0114] Application Scenario 1: Automatic Sound Effect Generation in Film and Television Post-Production and Short Videos. In the post-production of movies, TV series, animations, or short videos, this involves automatically generating matching ambient sounds and action sound effects (such as footsteps, door closing sounds, and weapon collision sounds) for silent shots, significantly reducing the cost and time of manual sound effects creation and search. In this process, silent video clips requiring sound effects can be input into the video processing system. The system performs frame-level differential processing on consecutive video frames. For example, when a person's foot touches the ground, the pixel values ​​in adjacent frames will change significantly in the foot area. This sequence of visually changing frames accurately captures the instantaneous point and trajectory of the action. The video encoder encodes these differential frames into dynamic change vectors, which essentially encode the dynamic event of "the instant the foot touches the ground." Simultaneously, the video encoder (such as CLIP) encodes each original frame to obtain a static semantic condition vector, identifying the scene as an "indoor marble corridor" and the person as "wearing leather shoes." Then, the diffusion generation model receives the above two vectors as conditions. The static semantic condition vector ensures that the generated sound has the texture of "leather shoes walking on a hard surface," rather than "sneakers stepping on mud." The dynamic change vector ensures that the rhythm and weight of the footsteps are precisely synchronized with the moment when the person in the video lifts and lands their foot. The diffusion generation model generates an audio latent representation vector that integrates material properties and rhythm information through iterative denoising. Finally, the decoder converts this latent vector into a realistic leather shoe footstep waveform and automatically aligns it with the video for output.

[0115] Application Scenario 2: Immersive Sound Field Rendering in Virtual Reality. In VR games or metaverse social platforms, audio with spatial awareness and synchronization is generated in real time based on the changes in perspective caused by the user's head rotation and the movement of objects in the virtual environment, enhancing immersion. In implementation, the rendered screen within the user's current field of view (a series of consecutive silent frames) can be used as the video to be processed. When the user's head rotates, a global pixel shift occurs between consecutive frames, and frame-level differential processing can sensitively capture the speed and direction of this perspective change. Simultaneously, a rolling sphere in the virtual world also generates local pixel motion between frames. This information is encoded into a dynamic change vector, representing the combined motion of "viewpoint turning left" and "sphere rolling right." Furthermore, the video encoder analyzes each frame to generate a static semantic condition vector, identifying the environment as an "empty metal hall" and the sphere as "wooden." Then, the diffusion generation model combines two types of vectors for generation. The static semantic condition vector determines the basic properties of the sound: the sound of a rolling sphere resonates in wood and produces reverberation in the metal hall. The dynamic change vector provides fine control: based on the speed and direction of the viewpoint change, corresponding binaural audio is generated through an acoustic model (which can be part of the decoder or post-processing) to simulate the change in sound position in space; simultaneously, based on the pixel displacement speed of the rolling sphere, a rolling sound with gradually increasing intensity is generated from slow to fast. The final output is an audio latent representation vector that is completely synchronized with the visual changes. After decoding this audio latent representation vector, the user hears the sound of a wooden ball rolling on a metal floor through headphones, and the sound's location dynamically changes as the user turns their head, creating an immersive experience.

[0116] Application Scenario 3: Audio Early Warning for Security Anomalies. This system allows for real-time analysis of critical silent surveillance videos, generating corresponding audio to aid in determining if an anomaly (such as broken glass, violent impact, or explosion) has occurred, thus achieving multimodal early warning. In implementation, the silent video stream transmitted in real-time from the surveillance camera is used as the video to be processed, and the video processing system continuously performs frame-level differential processing. Under normal circumstances, the changes in differential frames are small (e.g., slow changes in light). Once an anomaly occurs, such as a window's pixel area exhibiting drastic, high-brightness flickering and splashing changes within consecutive frames, the visual change frame sequence immediately captures this abnormal dynamic. The video encoder encodes this into a dynamic change vector, which contains the pattern of "drastic changes in local pixel areas within a short period." Simultaneously, the video encoder generates a static semantic conditional vector, identifying the monitored scene as "the window area of ​​a warehouse." Then, the diffusion generation model combines the "drastic change" pattern in the dynamic vector with the "warehouse window" semantic in the static vector. The diffusion generation model is conditionally guided to generate an audio latent representation vector highly correlated with "broken glass," rather than other sounds. Finally, the crisp sound of breaking glass is decoded, triggering an alarm system to notify security personnel. This method provides an additional dimension of perception besides image analysis, improving the reliability of early warnings.

[0117] Application Scenario 4: Synchronized Lip-Sync and Action-Based Dubbing for Digital Humans and Virtual Anchors. This involves automatically generating highly synchronized speech and non-verbal sounds such as breathing for the animations of digital humans or virtual anchors (including lip movements, facial expressions, and body movements), making their performance more lifelike. In this process, a silent animated video of a digital human speaking can be used as the video to be processed. Then, high-precision frame-level differential processing is performed on the digital human's facial region, which can capture extremely subtle changes in lip opening and closing and facial muscle movements. These changes are encoded into dynamic change vectors, precisely corresponding to the phoneme switching points and the timing of tone changes in speech. Simultaneously, the video encoder analyzes the image to obtain a static semantic condition vector, identifying the digital human's identity, gender, age, and other basic attributes. These attributes determine the timbre and intonation style of the voice. Here, the diffusion-generative model is the core of this scenario. The static semantic conditional vector determines that the anchor should have a "mature and intellectual female voice," while the dynamic change vector strictly controls the rhythm of the speech: vowels are generated when the mouth is open, plosives are generated when it is closed, and a slightly furrowed eyebrow may generate a tone with a questioning tone. The diffusion-generative model generates an audio latent representation vector that perfectly matches the lip movements and facial expressions through iterative denoising. After decoding this audio latent representation vector, a natural, fluent speech that is precisely synchronized with every frame of the digital human animation is obtained, achieving a realistic lip-sync effect.

[0118] Application Scenario 5: Industrial Equipment Status Monitoring and Fault Audio Diagnosis. This involves analyzing monitoring videos of industrial equipment (such as motors and pumps) and generating their expected normal sound. This sound is then compared with the actual captured sound, or anomalies in the generated sound are directly analyzed to help determine if the equipment is faulty. In this process, monitoring videos of the operating equipment (assuming they are silent or have their original audio removed) can be used as the video to be processed. Frame-level differential processing is performed on the rotating parts of the equipment in the video (such as fan blades and drive shafts) to calculate their rotational speed. The visual change frame sequence reflects the periodic movement of the blades, and the encoded dynamic change vector directly corresponds to the rotational speed frequency of the equipment. Simultaneously, the video encoder generates a static semantic condition vector, identifying the equipment type as a "large industrial centrifugal fan." When the equipment is operating normally, the diffusion generation model generates a stable and regular audio latent representation vector of a "whooshing" sound based on the static vector (fan type) and the dynamic vector (current rotational speed), with its fundamental frequency proportional to the rotational speed. Furthermore, the generated "normal" sound waveform is compared with the actual sound waveform of the equipment. If the actual sound contains sharp noises or periodic impact sounds not present in the generated sound, it indicates a potential fault in the equipment, such as bearing wear or blade imbalance. This method links visual operating status with auditory characteristics, providing a novel means of fault prediction.

[0119] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0120] This application provides a video-to-audio generation system based on visual change assistance (i.e., the video processing system described above), which can be applied to... Figure 5 In the system architecture shown. Figure 5 This demonstrates the data interaction and processing flow between the user terminal and the backend inference engine in the video-to-audio generation system. It is applicable to general audio application scenarios such as automatic dubbing of video content, virtual scene sound effect generation, automatic sound effect production in film and television post-production, and immersive interactive experiences. In this system architecture, the video input interface 501 receives the silent video to be dubbed and sends the video frame sequence of the silent video (i.e., video pre-trained features) to the mobile device 502. The mobile device can be a server or a locally deployed V2A inference module (i.e., the video-to-audio generation model). The mobile device 502 generates the audio signal 503 (i.e., the final generated audio) and outputs the audio signal through the audio output interface 504. Figure 6 This is a schematic diagram of the structure of the V2A inference module provided in the embodiments of this application, as shown below. Figure 6As shown, the V2A inference module includes: a visual change calculation module 601, used to extract dynamic change information between silent video frames to obtain frame-level visual change FLV (i.e., the aforementioned visual change frame sequence); a video encoder 602 (CLIP), used to extract semantic information of video frames to obtain static semantic condition vectors, and to extract dynamic change information of frame-level visual change FLVs to obtain dynamic change vectors used to represent the dynamic change information of silent videos; a diffusion generation model 603, used to combine conditional signals for iterative denoising generation to obtain audio latent representation vectors; and a compression model 604 (i.e., VAE decoder), used to map the audio latent representation vectors to a low-dimensional latent space to obtain the final generated audio.

[0121] It should be noted that the main technical solutions of this application are as follows: First, a text-to-audio pre-training and visual change joint conditionalization method is adopted. Specifically, a large-scale text-to-audio pre-training model (T2A) is used to initialize audio generation capabilities, enhancing the naturalness and semantic matching ability of the audio; visual conditional signals and visual change information are introduced in the V2A fine-tuning stage to achieve cross-modal adaptation from silent video to audio and improve semantic and temporal alignment; by freezing the VAE compression model and CLIP video encoder, dynamic changes are efficiently captured while maintaining semantic stability; the stability and naturalness of the generated audio in terms of rhythm, scene consistency, and sound effect details are improved. Second, a visual change-assisted method is adopted to design a frame-level visual change extraction method to capture short-term dynamic information; the visual change vector and the static semantic conditional vector are integrated as the conditional input of the diffusion generation model, without the need for additional training modules or loss functions; the diffusion generation model is guided in the latent space to strengthen the generation of audio signals corresponding to video dynamic events; the detail richness, dynamic response capability, and video-audio temporal alignment accuracy of the generated audio are improved.

[0122] The implementation process of the video-to-audio generation method based on visual change assistance provided in the embodiments of this application is described below.

[0123] First, a T2A generative model is pre-trained. Text is then embedded... With target audio The latent representation corresponding to (L is the audio length) ( and Using latent space temporal sequence and feature dimension as inputs, a stable audio generation system is pre-trained. This T2A pre-training establishes a strong generative prior, reducing the difficulty of V2A training and improving audio reproduction capabilities.

[0124] Then, visual change information is constructed from the input silent video. (in (where i represents the i-th frame of a silent video, and N is the total number of frames in the video), extracting frame-level visual variation (FLV). Specifically, the FLV is calculated using the following formula (1):

[0125] (1).

[0126] Where k represents the number of frames between which visual changes are calculated (i.e., the number of sampling interval frames). Based on the ablation experiment results, k=2 is optimal; too small a value is insufficient to capture motion, while too large a value easily introduces noise. Visual changes provide motion-sensitive features, guiding the diffusion generation model to capture dynamic audio events.

[0127] Next, conditional diffusion generation from video to audio is performed. The pre-trained T2A model is fine-tuned to V2A by using text conditional... Replace with video conditions With visual change information Optimize diffusion model parameters Generate potential audio representations The optimization objective is as follows (2):

[0128] (2).

[0129] in, The noise is added during the forward process, and t is the noise addition time step. for Latent variables after adding noise. Semantic guidance is provided through CLIP embedding, and temporal dynamic constraints are provided through visual change information. After model training, the generated audio and video content are highly consistent in semantics and time.

[0130] Finally, audio decoding and output are performed. The generated latent audio representation is then processed. The final audio waveform is decoded using VAE. The output is the audio sequence corresponding to the input silent video.

[0131] In some embodiments, result evaluation can also be performed. In the VGGSound benchmark test, the embodiments of this application achieved better results than models trained solely on CLIP features across multiple evaluation metrics, including FAD, KL, IS, FD, ImageBind Score, and AA, effectively introducing additional features. Furthermore, by introducing a T2A pre-training mechanism, the embodiments of this application also demonstrate significant advantages in the V2A task, outperforming current mainstream models and exhibiting stronger cross-modal understanding and generation capabilities. In addition, experimental results show that the scheme introducing frame-level visual changes can more accurately capture the dynamic features of video content than embedding-level visual changes, thereby further improving generation consistency.

[0132] This application's embodiments introduce a visual change-assisted mechanism to calculate frame-level visual difference features in a video frame sequence and use them as dynamic conditional inputs to a diffusion generation model. This enables the generated audio signal to accurately respond to subtle motion changes and event triggering moments in the video. Compared to traditional video-to-audio generation methods based solely on static semantic conditions, this application's embodiments significantly improve temporal alignment accuracy, rhythmic consistency, and dynamic sound effect response capabilities, achieving adaptive matching of the audio generation process to the video's temporal dynamics. Furthermore, by combining visual semantic features extracted from a language-image contrast pre-trained model, this application's embodiments achieve joint conditionalization of video semantics and visual change features without introducing additional network structures or training stages. This significantly improves the naturalness, detail richness, and semantic consistency of the generated audio while maintaining system simplicity and efficiency. This mechanism effectively avoids the error accumulation problem caused by traditional methods relying on external feature prediction, enhancing the model's generalization ability and stability across scenes and action types.

[0133] It should be noted that current video-to-audio generation methods generally rely on language-image contrast pre-trained models to extract static semantic features to achieve semantic matching between video content and generated audio. However, these models mainly focus on image-level content understanding and lack sensitivity to dynamic changes in the temporal dimension, making it difficult to accurately reflect the rhythm of actions and the timing of event triggers. In contrast, the "visual change information-assisted guidance mechanism" proposed in this application directly calculates the differences between adjacent frames and extracts motion-related features, enhancing temporal alignment capabilities without introducing additional networks or training stages. While maintaining low computational overhead, it significantly improves the matching effect between dynamic events and audio rhythms.

[0134] In addition to the inter-frame difference features in the embodiments of this application, optical flow field estimation, sparse keypoint tracking, and temporal gradient accumulation can also be used to replace frame difference calculation. These methods can capture the object's movement direction, velocity, and regional dynamics more precisely. If deployed in scenarios where system resources permit, such solutions can also achieve visually dynamic-driven audio generation, which is equivalent to the principle of the embodiments of this application and belongs to an equivalent implementation.

[0135] The video-to-audio generation method provided in this application can be applied to at least the following scenarios: 1) Film and game sound effect generation scenarios: In film and television production or game engines, it is often necessary to automatically generate ambient sound, action sound, or emotional sound for silent videos. The visual change guidance mechanism of this application can accurately capture dynamic information such as character movements, camera transitions, and object collisions, thereby generating sound that is highly consistent with the rhythm of the picture, realizing automated onomatopoeia and post-production sound effect generation. 2) Virtual reality and immersive interaction scenarios: In virtual reality (VR), augmented reality (AR), or digital twin systems, the spatial consistency and temporal synchronization of ambient sound are crucial to the sense of immersion. Through the method provided in this application, the audio output can be automatically adjusted according to changes in the user's perspective or scene dynamics, realizing three-dimensional sound field rendering synchronized with visual changes. 3) Intelligent monitoring and event detection scenarios: In intelligent security or traffic monitoring, this application can be used to generate or predict potential sound events from silent videos, such as collisions, alarms, and mechanical operations, to achieve sound completion and event recognition, providing supplementary information for multimodal early warning systems. 4) Digital Human and Virtual Character Performance Scenarios: In systems such as virtual humans, AI anchors, and intelligent customer service, this application embodiment can automatically drive speech or onomatopoeia generation through facial movements or body changes, achieving consistency between visual behavior and audio performance, and enhancing the expressiveness and naturalness of digital characters. 5) Content Generation and Creative Synthesis Scenarios: In short video, animation production, and multimodal creation platforms, this application embodiment can serve as a "silent video dubbing engine," automatically generating corresponding sound effects or background music based on the rhythm of the scene and changes in movement, lowering the creative threshold and expanding multimedia content production methods. 6) Scientific Research and Educational Assistance Scenarios: In scientific research visualization or teaching demonstrations, this application embodiment can be used to generate sound simulations based on experimental recordings, such as chemical reactions and mechanical movements, helping users perceive dynamic processes through hearing, improving teaching interactivity and depth of understanding. 7) Mobile and Edge Computing Deployment Scenarios: This application embodiment requires no additional training stage; the model is lightweight and inference is efficient, allowing direct embedding into mobile or edge devices to achieve real-time video and audio completion, intelligent speech generation, or scene sound feedback, providing an audiovisual collaborative experience for portable multimedia devices.

[0136] Based on the video processing method described in the above embodiments, Figure 7This paper illustrates a structural block diagram of a video processing apparatus according to an embodiment of this application. The video processing apparatus 100 can be a device in an electronic device (e.g., a server). It can be implemented in software, such as programs and plugins, and includes the following software modules: a differential processing module 101, used to perform frame-level differential processing on video frames in the video to be processed to obtain a sequence of visual change frames; the video to be processed is a silent video without audio information; an encoding module 102, used to encode each visual change frame in the sequence of visual change frames using a video encoder to obtain a dynamic change vector representing the dynamic change information of the video to be processed; the encoding module 102 is also used to encode each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; a generation module 103, used to perform iterative denoising generation based on the static semantic condition vector and the dynamic change vector using a diffusion generation model to obtain an audio latent representation vector; and a decoding module 104, used to decode the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

[0137] In some embodiments, the differential processing module 101 is further configured to: sample video frames of the video to be processed to obtain a video frame sequence composed of multiple video frames; and perform frame-level differential processing on each two adjacent video frames in the video frame sequence to obtain a visual change frame sequence.

[0138] In some embodiments, the differential processing module 101 is further configured to: determine the number of sampling interval frames; and sample the video to be processed according to the number of sampling interval frames to obtain a video frame sequence composed of multiple video frames.

[0139] In some embodiments, the differential processing module 101 is further configured to: for any two adjacent video frames in the video frame sequence, obtain the pixel value of each pixel in each video frame of the two video frames; perform differential calculation on the pixel values ​​of pixels at the same pixel position in the two video frames to obtain the visual change frame corresponding to the two video frames; and sort the visual change frames corresponding to each pair of adjacent video frames according to the order of each pair of adjacent video frames in the video frame sequence to form a visual change frame sequence.

[0140] In some embodiments, the apparatus further includes: a post-processing model for post-processing each visual change frame in the visual change frame sequence to obtain a processed visual change frame sequence; the post-processing includes at least one of the following: absolute value processing, thresholding processing, and normalization processing; correspondingly, the encoding module 102 is further configured to: encode each visual change frame in the processed visual change frame sequence using a video encoder.

[0141] In some embodiments, the generation module 103 is further configured to: randomly generate a noise tensor that conforms to a normal distribution; and, according to a preset total number of diffusion steps, perform iterative denoising generation in the audio latent space based on the noise tensor, static semantic condition vector, and dynamic change vector to obtain an audio latent representation vector.

[0142] In some embodiments, the generation module 103 is further configured to: during each iteration of denoising, concatenate the static semantic condition vector and the dynamic change vector corresponding to the current time step to obtain the current concatenated vector; input the current noisy first latent representation vector, the current time step, and the current concatenated vector into the diffusion generation model to obtain predicted noise; wherein, during the first iteration of denoising, the first latent representation vector is the noise tensor; during the i-th iteration of denoising, the first latent representation vector is the denoised vector obtained after the (i-1)-th iteration of denoising; i is an integer greater than 1; use the predicted noise to determine the second latent representation vector after removing part of the noise, and determine the second latent representation vector as the denoised vector at the current time step; in response to the current time step being equal to the total number of diffusion steps, determine the denoised vector at the current time step as the audio latent representation vector.

[0143] In some embodiments, the encoding module 102 is further configured to: encode each visual change frame in the visual change frame sequence using a video encoder with parameters frozen in the audio variational autoencoder to obtain a dynamic change vector representing the dynamic change information of the video to be processed; and encode each video frame in the video to be processed using a video encoder with parameters frozen in the audio variational autoencoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed; wherein the video encoder that encodes each visual change frame shares the same encoder parameters as the video encoder that encodes each video frame.

[0144] In some embodiments, the decoding module 104 is further configured to: decode the audio latent representation vector using a decoder with parameters frozen in the audio variational autoencoder to obtain the generated audio corresponding to the video to be processed.

[0145] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0146] This application provides an electronic device. Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 8 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 8(Only one is shown in the diagram), memory 132, and computer-executable instructions 133 stored in memory 132 and executable on at least one processor 131, which, when executed by processor 131, implement the steps in any of the above-described audio generation method embodiments.

[0147] The electronic device may include, but is not limited to, a processor 131 and a memory 132. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 130 and does not constitute a limitation on electronic device 130. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0148] Processor 131 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0149] In some embodiments, memory 132 may be an internal storage unit of electronic device 130, such as a hard disk or memory of electronic device 130. In other embodiments, memory 132 may be an external storage device of electronic device 130, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on electronic device 130. Furthermore, memory 132 may include both internal and external storage units of electronic device 130. Memory 132 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 132 may also be used to temporarily store data that has been output or will be output.

[0150] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video processing method described in this application.

[0151] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the audio generation method provided in this application. For example, ... Figure 1 The video processing method shown.

[0152] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0153] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0154] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0155] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0156] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video processing, the method comprising: The method comprises: frame-level difference processing is performed on video frames in a to-be-processed video to obtain a visual change frame sequence; the to-be-processed video is a silent video that does not contain audio information; each visual change frame in the visual change frame sequence is encoded by a video encoder to obtain a dynamic change vector for representing dynamic change information of the to-be-processed video; each video frame in the to-be-processed video is encoded by a video encoder to obtain a static semantic condition vector for representing static semantic information of the to-be-processed video; an audio latent representation vector is generated by an iterative denoising generation based on the static semantic condition vector and the dynamic change vector through a diffusion generative model; the audio latent representation vector is decoded to obtain generated audio corresponding to the to-be-processed video.

2. The method of claim 1, wherein, The frame-level difference processing on the video frames in the to-be-processed video to obtain the visual change frame sequence comprises: video frame sampling is performed on the to-be-processed video to obtain a video frame sequence composed of multiple video frames; frame-level difference processing is performed on each pair of adjacent video frames in the video frame sequence to obtain the visual change frame sequence.

3. The method of claim 2, wherein, The video frame sampling on the to-be-processed video to obtain the video frame sequence composed of multiple video frames comprises: a sampling interval frame number is determined; video frame sampling is performed on the to-be-processed video according to the sampling interval frame number to obtain the video frame sequence composed of multiple video frames.

4. The method of claim 2, wherein, The frame-level difference processing on each pair of adjacent video frames in the video frame sequence to obtain the visual change frame sequence comprises: for any pair of adjacent video frames in the video frame sequence, pixel values of each pixel point in each video frame of the pair of video frames are obtained; pixel values of pixel points at the same pixel point positions in the pair of video frames are calculated by difference to obtain a visual change frame corresponding to the pair of video frames; the visual change frames corresponding to each pair of adjacent video frames are sorted according to the order of each pair of adjacent video frames in the video frame sequence to form the visual change frame sequence.

5. The method of claim 2, wherein, After the frame-level difference processing on each pair of adjacent video frames in the video frame sequence to obtain the visual change frame sequence, the method further comprises: each visual change frame in the visual change frame sequence is post-processed to obtain a processed visual change frame sequence; the post-processing comprises at least one of the following: absolute value processing, thresholding processing, and normalization processing; correspondingly, the encoding processing on each visual change frame in the visual change frame sequence by the video encoder comprises: the encoding processing on each visual change frame in the processed visual change frame sequence by the video encoder.

6. The method of claim 1, wherein, The iterative denoising generation based on the static semantic condition vector and the dynamic change vector through the diffusion generative model to obtain the audio latent representation vector comprises: a noise tensor conforming to a normal distribution is randomly generated; According to the preset total number of diffusion steps, the diffusion generation model iteratively denoises and generates audio latent representation vectors in the audio latent space based on the noise tensor, the static semantic condition vector, and the dynamic change vector.

7. The method of claim 6, wherein, The process involves iteratively denoising and generating an audio latent representation vector in the audio latent space based on the noise tensor, the static semantic condition vector, and the dynamic change vector, according to a preset total number of diffusion steps and the diffusion generation model. This process includes: In each iteration of denoising, the static semantic condition vector and the dynamic change vector corresponding to the current time step are concatenated to obtain the current concatenated vector; The current noisy first latent representation vector, the current time step, and the current concatenated vector are input into the diffusion generation model to obtain the predicted noise; wherein, in the first step of denoising, the first latent representation vector is the noise tensor; in the i-th step of denoising, the first latent representation vector is the denoised vector obtained after the (i-1)-th step of denoising; i is an integer greater than 1; The predicted noise is used to determine a second potential representation vector after removing some noise, and the second potential representation vector is determined as the denoised vector at the current time step. In response to the current time step being equal to the total number of diffusion steps, the denoising vector at the current time step is determined as the audio latent representation vector.

8. The method according to any one of claims 1 to 7, characterized in that, The step of encoding each visual change frame in the visual change frame sequence using a video encoder to obtain a dynamic change vector representing the dynamic change information of the video to be processed includes: The video encoder with parameters frozen in the audio variational autoencoder encodes each visual change frame in the visual change frame sequence to obtain a dynamic change vector that represents the dynamic change information of the video to be processed. The step of encoding each video frame in the video to be processed using a video encoder to obtain a static semantic condition vector representing the static semantic information of the video to be processed includes: The video encoder with frozen parameters in the audio variational autoencoder encodes each video frame in the video to be processed to obtain a static semantic condition vector that represents the static semantic information of the video to be processed. The video encoder that encodes each visual change frame shares the same encoder parameters as the video encoder that encodes each video frame.

9. The method according to any one of claims 1 to 7, characterized in that, The step of decoding the audio latent representation vector to obtain the generated audio corresponding to the video to be processed includes: The audio latent representation vector is decoded using a parameter-frozen decoder in an audio variational autoencoder to obtain the generated audio corresponding to the video to be processed.

10. A video processing device, comprising: The device includes: The differential processing module is used to perform frame-level differential processing on video frames in the video to be processed to obtain a sequence of visually changing frames; the video to be processed is a silent video that does not contain audio information. The encoding module is configured to perform encoding processing on each visual change frame in the sequence of visual change frames by using a video encoder to obtain a dynamic change vector representing dynamic change information of the video to be processed; The encoding module is further configured to perform encoding processing on each video frame in the video to be processed by using a video encoder to obtain a static semantic condition vector representing static semantic information of the video to be processed; The generation module is configured to perform iterative denoising generation based on the static semantic condition vector and the dynamic change vector by using a diffusion generative model to obtain an audio latent representation vector; The decoding module is configured to perform decoding processing on the audio latent representation vector to obtain generated audio corresponding to the video to be processed.

11. An electronic device, comprising: The electronic device comprises: a memory configured to store computer executable instructions or computer programs; a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the video processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the video processing method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the video processing method according to any one of claims 1 to 9. The computer executable instructions or computer programs are executed by the processor to implement the video processing method according to any one of claims 1 to 9.

Images