Video compression method and system, electronic equipment and storage medium

Abstract

The invention provides a video compression method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring to-be-compressed video data and audio data corresponding to the to-be-compressed video data; inputting the to-be-compressed video data and audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; wherein the pre-trained video compression model is used for carrying out joint processing on the audio data and the video data, so that the video data utilizes implicit semantics of the audio data to enhance video compression perception; the pre-trained video compression model is obtained based on training of a video training set. According to the video data, key information in the audio information can be fully utilized, the compression efficiency and the video quality are guaranteed, the flexibility of video compression is enhanced, and diversified user requirements are met.

Description

Technical Field

[0001] The present invention relates to the technical field of video compression, and in particular to a video compression method, system, electronic equipment and storage medium. Background Art

[0002] As digital media develops rapidly, video content can carry more information than other media such as text and pictures. In today's society, video content is increasingly disseminated and plays an increasingly critical role in people's daily lives. With the popularization of high-definition video formats such as 4K and 8K, the amount of video data has expanded dramatically, placing higher requirements on the storage and transmission of video data. Video compression technology, as a core means to reduce the amount of video data and improve transmission efficiency, has attracted widespread attention and in-depth research in the industry.

[0003] However, traditional video compression methods mainly rely on removing spatial and temporal redundancy, which may affect video quality to a certain extent. Especially when pursuing high compression rates, the effect of video compression is limited and it is difficult to fully meet the viewing needs of users. Therefore, how to enhance the flexibility of video compression while ensuring compression efficiency and video quality to adapt to the diverse needs of users is a technical problem that needs to be overcome urgently. Summary of the invention

[0004] The present invention provides a video compression method, system, electronic device and storage medium to solve the defect that the video quality cannot be guaranteed while compressing the video in the prior art. The video data of the present invention can make full use of the key information in the audio information, ensure the compression efficiency and video quality, enhance the flexibility of video compression, and adapt to the diverse user needs.

[0005] The present invention provides a video compression method, comprising: obtaining video data to be compressed and audio data corresponding to the video data to be compressed; inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance video compression perception; the pre-trained video compression model is obtained by training based on a video training set.

[0006] According to a video compression method provided by the present invention, the video compression model includes a video encoder, an audio encoder, an audio-video attention channel integrator and a conditional decoder; the video encoder is used to encode the video data to be compressed; the audio encoder is used to encode the audio data; the audio-video attention channel integrator is used to synchronize the encoded video data and the encoded audio data by using channel fusion, and uses an external attention mechanism to capture features to obtain audio-video fusion features; the conditional decoder is used to decode the audio-video fusion features to obtain the compressed video data.

[0007] According to a video compression method provided by the present invention, the audio and video attention channel integrator includes a projection layer, a channel fusion layer, an external attention module and a residual convolution layer; the projection layer is used to project the encoded audio data into an audio space so that the dimension of the projected audio data is the same as that of the encoded video data; the channel fusion layer is used to spatially fuse the projected audio data with the encoded video data; the external attention module is used to identify and filter relevant information in the spatially fused audio and video data to obtain the audio-video fusion feature; the residual convolution layer is used to enhance the visual information of the audio-video fusion feature.

[0008] According to a video compression method provided by the present invention, the conditional decoder includes a time embedding module, a SNeRV block, a TAT residual block and a header layer; the time embedding module is used to convert the frame index corresponding to the audio-video fusion feature into a time embedding vector; the SNeRV block is used to convert the audio-video fusion feature into an intermediate representation feature; the TAT residual block is used to align the intermediate representation feature with the time embedding vector; the header layer is used to convert the features output by the TAT residual block into the compressed video data.

[0009] According to a video compression method provided by the present invention, the video compression model also includes a conditional decoder optimization module; the conditional decoder optimization module uses a pre-trained diffusion model and combines a probability density distillation loss function to perform gradient descent optimization on the conditional decoder; wherein the pre-trained diffusion model uses linear and exponential annealing strategies to dynamically adjust the time step in the diffusion process.

[0010] According to a video compression method provided by the present invention, the video data to be compressed and the audio data are input into a pre-trained video compression model to obtain the compressed video data output by the video compression model, including: inputting the video data to be compressed into the video encoder to obtain the encoded video data output by the video encoder; inputting the audio data into the audio encoder to obtain the encoded audio data output by the audio encoder; inputting the encoded video data and the encoded audio data into the audio-video attention channel integrator to obtain the audio-video fusion features output by the audio-video attention channel integrator; inputting the audio-video fusion features into the conditional decoder to obtain the compressed video data output by the conditional decoder.

[0011] According to a video compression method provided by the present invention, the loss function of the pre-trained video compression model is: , in, is the loss function of the pre-trained video compression model, is the L1 loss function, is the frequency domain L1 loss function, is the multi-scale structural similarity index loss function, is the probability density distillation loss function, For expectations, is noise, which follows the standard normal distribution , For data distribution x The input data sampled in are the parameters of the diffusion model, To predict the noise, x t For potential representation, t is the time step.

[0012] The present invention also provides a video compression system, comprising: an acquisition module, used to acquire video data to be compressed and audio data corresponding to the video data to be compressed; a compression module, used to input the video data to be compressed and the audio data into a pre-trained video compression model, and acquire the compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance video compression perception; the pre-trained video compression model is trained based on a video training set.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-mentioned video compression methods when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video compression method described above is implemented.

[0015] The present invention provides a video compression method, system, electronic device and storage medium, the method comprising: obtaining video data to be compressed and audio data corresponding to the video data to be compressed; inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is obtained by training based on a video training set. The video data of the present invention can make full use of the key information in the audio information, ensure the compression efficiency and video quality, enhance the flexibility of video compression, and adapt to the diverse user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a video compression method provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the principle of the video compression model provided by the present invention.

[0019] Figure 3 It is a principle schematic diagram of the audio and video attention channel integrator provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the principle of the conditional decoder provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of a video compression system provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] With the rapid development of digital media, the demand for video data storage and transmission is increasing. As a key technology to reduce the amount of video data and improve transmission efficiency, video compression technology has received extensive research and application attention. Traditional video compression methods mainly rely on removing spatial and temporal redundancy, but cannot guarantee video quality while compressing the video.

[0025] Please refer to Figure 1 , Figure 1 A schematic flow chart of a video compression method provided by the present invention.

[0026] The present invention provides a video compression method, comprising: 101: Obtain video data to be compressed and audio data corresponding to the video data to be compressed; 102: Input the video data and audio data to be compressed into a pre-trained video compression model, and obtain the compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data uses the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is trained based on a video training set.

[0027] In order to solve the technical problems existing in the prior art, the present invention provides a video compression method, which aims to improve the overall effect of video compression and generation through the deep fusion of multimodal information. In the field of video compression, the semantic information contained in audio information is crucial for understanding the video content. This method uses a pre-trained video compression model to jointly process the video data and audio data to be compressed, so that the video data can make full use of the implicit semantics of the audio data, thereby more effectively enhancing the expression ability of video features during the compression process and significantly improving the compression rate (bpp, bit perpoint). Video compression perception can be significantly improved through audio, ensuring that the high fidelity of video quality is maintained while compressing, enhancing the flexibility of video compression, and adapting to diverse user needs.

[0028] It needs to be explained that the compression rate is the average number of bits occupied by each point after the video is compressed, which reflects the compression ratio.

[0029] Please refer to Figure 2 , Figure 2 A schematic diagram of the principles of the video compression model provided by the present invention.

[0030] As a preferred embodiment, the video compression model includes a video encoder, an audio encoder, an audio-video attention channel integrator and a conditional decoder; the video encoder is used to encode the video data to be compressed; the audio encoder is used to encode the audio data; the audio-video attention channel integrator is used to synchronize the encoded video data and the encoded audio data by using channel fusion, and uses an external attention mechanism to capture features to obtain audio-video fusion features; the conditional decoder is used to decode the audio-video fusion features to obtain compressed video data.

[0031] As a preferred embodiment, the video data and audio data to be compressed are input into a pre-trained video compression model to obtain compressed video data output by the video compression model, including: inputting the video data to be compressed into a video encoder to obtain encoded video data output by the video encoder; inputting the audio data into an audio encoder to obtain encoded audio data output by the audio encoder; inputting the encoded video data and the encoded audio data into an audio and video attention channel integrator to obtain audio-video fusion features output by the audio and video attention channel integrator; and inputting the audio-video fusion features into a conditional decoder to obtain compressed video data output by the conditional decoder.

[0032] In this embodiment, the video compression model is a model that utilizes implicit speech semantics to enhance the video compression effect in an action data set. The video compression model includes a video encoder, an audio encoder, an audio and video attention channel integrator, and a conditional decoder.

[0033] Among them, the video encoder can use ConvNeXt. ConvNeXt is an efficient convolutional neural network that uses deep convolution, MLP and interpolation layers to build a deep and wide network architecture with high capacity. This design makes the video data to be compressed It can be effectively processed by ConvNeXt to extract rich features, provide high-quality image representation for video compression, and generate encoded video data .

[0034] In order to enhance the understanding and processing of video content through implicit semantic guidance of audio, the audio encoder can adopt a lightweight pre-trained HuBERT-soft model. HuBERT-soft is a feature extractor based on HuBERT-Base. Its advantage is that for a given 1×T , which can efficiently extract rich speech features and generate 256-dimensional features . These features not only contain explicit information of speech, but also implicitly encode semantic information related to the action video content. This implicit semantics-guided video compression model can more accurately capture the motion features in the video, thereby achieving higher efficiency and better quality in the video compression and encoding process. The video compression model of the present invention is a compression model designed for action video datasets, which means that it can better adapt to the characteristics of action videos, such as fast motion and changes in complex scenes. By integrating HuBERT-soft, the video compression model not only improves the compression efficiency, but also performs well in maintaining the continuity and details of video motion.

[0035] In order to enhance the video compression performance of action datasets through implicit semantic guidance of audio, the audio-video attention channel integrator uses channel fusion to synchronize the encoded video data. and the encoded audio data , and uses an external attention mechanism to capture features and obtain audio-video fusion features.

[0036] The conditional decoder can adopt a time-aware conditional decoder, which uses the normalization-free temporal affine transform and its associated residual block to unlock the potential of feature alignment and optimize the representation of video data.

[0037] Please refer to Figure 3 , Figure 3 A schematic diagram of the principles of the audio and video attention channel integrator provided by the present invention.

[0038] As a preferred embodiment, the audio and video attention channel integrator includes a projection layer, a channel fusion layer, an external attention module and a residual convolution layer; the projection layer is used to project the encoded audio data into the audio space so that the dimension of the projected audio data is the same as that of the encoded video data; the channel fusion layer is used to spatially fuse the projected audio data with the encoded video data; the external attention module is used to identify and filter relevant information in the spatially fused audio and video data to obtain audio-video fusion features; the residual convolution layer is used to enhance the visual information of the audio-video fusion features.

[0039] In this embodiment, the audio and video attention channel integrator includes a projection layer, a channel fusion layer, an external attention module and a residual convolution layer. In order to synchronize audio data and video data at the encoding level, the encoded audio data Project it into the audio space, resize it to the same dimensions as the video data, and then compare the resulting projected audio data to the encoded video data. They are input into the channel fusion layer for integration. Channel fusion is used to synchronize and integrate audio-video information, solving the problem of audio and video asynchrony. Subsequently, an external attention module is used to identify and filter relevant and redundant information in the spatially fused audio and video data to obtain audio-video fusion features. The external attention module can use two linear layers and one normalization layer, has linear complexity, and implicitly considers the relationship between different feature maps. In order to prevent the fused information from forgetting the underlying visual information, residual module fusion and jump connection are added. Finally, the integrated audio-video fusion features are used to enhance the visual information through the residual convolution layer, and the output generates compressed video data. .

[0040] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the principle of the conditional decoder provided by the present invention.

[0041] As a preferred embodiment, the conditional decoder includes a time embedding module, a SNeRV block, a TAT residual block and a header layer; the time embedding module is used to convert the frame index corresponding to the audio-video fusion feature into a time embedding vector; the SNeRV block is used to convert the audio-video fusion feature into an intermediate representation feature; the TAT residual block is used to align the intermediate representation feature with the time embedding vector; the header layer is used to convert the features output by the TAT residual block into compressed video data.

[0042] In this embodiment, the conditional decoder includes a time embedding module, a SNeRV block, a TAT residual block and a header layer. and frame indices normalized to the interval [0, 1] t As input, it outputs the reconstructed image. The temporal embedding module is used to convert the frame index corresponding to the audio-video fusion feature into a temporal embedding vector. Specifically, the frame index t First processed by a positional encoding, then converted to a temporal embedding vector by an MLP At the same time, audio-video fusion features Transformed into intermediate representation features through SNeRV blocks Intermediate representation features and the time embedding vector After six stages of SNeRV blocks and TAT residual blocks, a header layer is used to obtain compressed video data. .

[0043] As a preferred embodiment, the video compression model also includes a conditional decoder optimization module; the conditional decoder optimization module uses a pre-trained diffusion model and combines the probability density distillation loss function to perform gradient descent optimization on the conditional decoder; wherein the pre-trained diffusion model uses linear and exponential annealing strategies to dynamically adjust the time step in the diffusion process.

[0044] In this embodiment, the conditional decoder is further optimized using a conditional decoder optimization module. The conditional decoder optimization module can utilize a pre-trained diffusion model. Score Distillation Sampling (SDS) is used to guide 3D reconstruction using a pre-trained 2D diffusion model, and high-quality generation from text to 3D content is achieved through SDS. The core advantage of SDS is that it can transfer powerful 2D image prior knowledge to 3D reconstruction tasks, providing a valuable reference for optimizing video compression models.

[0045] A loss function based on probability density distillation is introduced. This loss function allows the use of a pre-trained diffusion model as a prior to optimize the above conditional decoder, thereby guiding the conditional decoder to learn to generate samples consistent with the diffusion model prior during training. The diffusion model generates high-quality images through forward diffusion and reverse denoising processes. Forward diffusion converts the generated image into a latent space representation by gradually adding noise, and introduces different types of noise (such as Gaussian noise and Poisson noise). In this process, the model output image is first normalized to the interval [0, 1], and in order to promote stable convergence of the model and control the impact of noise, linear and exponential annealing scheduling strategies are used to dynamically adjust the time step in the diffusion process to gradually reduce the noise. (This strategy provides a larger step size in the early stages of training so that the model can quickly adapt to the data distribution; the step size is reduced in the later stages to fine-tune the model output and improve the final convergence effect.) The latent diffusion model samples from Gaussian noise and converts it into a latent space image x t In the reverse diffusion process, the denoising model (U-Net structure) predicts the noise at each step in the low-dimensional space and restores the clear image. The loss function based on probability density distillation measures the difference between the predicted noise and the prior noise distribution.

[0046] The latent diffusion model, as a prior, provides a strong guiding signal for the conditional decoder, helping the conditional decoder to learn to reconstruct high-quality pixel-level features from random noise. The key to this method is to introduce a loss function based on probability density distillation during the training process to measure the difference between the samples generated by the conditional decoder and the diffusion model prior. By minimizing this loss, the conditional decoder is guided to generate samples consistent with the diffusion model prior, which means that after being processed by the diffusion model, the image generated by the conditional decoder has a noise pattern consistent with the data distribution during the diffusion model training. This enables the conditional decoder to reconstruct high-quality images from audio-video fusion features. More importantly, the latent diffusion model is only used as a parameter optimizer during the training process and does not add any additional parameters during the inference process. The present invention can significantly improve the quality of reconstructed images without increasing the cost of model training and inference.

[0047] In addition, a Gaussian Dropout mechanism is added to improve the robustness of the diffusion model in complex scenarios. The present invention relies on a pre-trained diffusion model, calculates the mean square error (MSE) of the generated image and noise, and uses SDS to perform gradient updates on the conditional decoder parameters, thereby further optimizing the generation effect.

[0048] As a preferred embodiment, the loss function of the pre-trained video compression model is: , in, is the loss function of the pre-trained video compression model, is the L1 loss function, is the frequency domain L1 loss function, is the multi-scale structural similarity index loss function, is the probability density distillation loss function, For expectations, is noise, which follows the standard normal distribution , For data distribution x The input data sampled in are the parameters of the diffusion model, To predict the noise, x t For potential representation, t is the time step.

[0049] The goal of the video compression model is to minimize the original frame and reconstruct the frame The L1 loss function of the pixel value provides the original frame and reconstruct the frame The most direct measure of the difference between. For the frequency domain loss function, in the original frame and reconstruct the frame After Fourier transform, the L1 spectral feature loss is also calculated. The MS-SSIM loss function effectively retains the structural information (edges and details of the image). In addition, the SDS probability density distillation loss function improves the model's fidelity to details through dynamic noise injection and time step annealing, ensuring that high-frequency areas are more fully captured in multimodal fusion compression, thereby achieving high-quality image reconstruction.

[0050] In order to systematically evaluate and analyze the effectiveness of the present invention, multiple types of videos are used, including UCF101 (a dataset containing 16 action videos with a resolution of 320×240) and VoxCeleb2 (containing 14 single-person scene videos with a resolution of 224×224). The UCF101 dataset covers a variety of scenes, such as facial makeup, cello performance, swimming, etc., while VoxCeleb2 mainly consists of speaker scenes. In addition, in order to gain a deeper understanding of the performance of the present invention under different conditions, experiments are conducted at multiple resolutions and model sizes, and the model parameters are set to 3M, 5M, 10M, and 15M, respectively. By testing at different model sizes, it is intended to fully understand the adaptability and performance of the present invention under different computational costs and compression quality requirements. These results will provide strong support for evaluating the generalization ability of the framework, its applicability across datasets, and the effectiveness of integrating multimodal information.

[0051] In the experiment, Peak Signal-to-Noise Ratio (PSNR) and Fréchet Inception distance (FID) are used to measure the visual quality of the reconstructed video to ensure that the compressed video has high visual fidelity while retaining the original details and semantic information. In addition, bits per pixel (bpp) is used as a measure of video compression. The method of the present invention achieves state-of-the-art performance in both PSNR and FID indicators for compression and regression tasks.

[0052] What needs to be explained is: Peak signal-to-noise ratio is an indicator for evaluating the quality of compressed video.

[0053] The Fritschett Twin Distance is an indicator for evaluating the quality of images generated by generative models. It measures the similarity between generated images and real images by comparing their statistical features. The lower the FID value, the closer the distribution between the generated image and the real image, and the better the image quality.

[0054] The present invention processes audio and video signals jointly through multimodal fusion, fully utilizes key information in audio signals, enhances the expressiveness of video features, and enables the video to better retain important information during compression, achieving good results on the action dataset UCF101. At the same time, the present invention uses a pre-trained latent diffusion model to provide prior information for the decoder, combined with a probability density distillation loss function, thereby improving the reconstruction quality without increasing the inference calculation cost, further optimizing the video compression and generation process, and also adopts technologies such as multi-step annealing and dynamic noise modulation, while improving the compression rate (bpp), further ensuring that the final generated video has excellent visual quality.

[0055] Based on the present invention, the joint optimization of text features and video content can also improve the effect of video compression. In addition, more complex cross-modal attention mechanisms can be used to enhance information interaction between modalities. Dividing the diffusion process into layers, performing layer-by-layer generation, and gradually refining the generated content from low resolution to high resolution can also improve the efficiency and quality of video generation.

[0056] The video compression system provided by the present invention is described below. The video compression system described below and the video compression method described above can be referenced to each other.

[0057] Please refer to Figure 5 , Figure 5 A structural schematic diagram of a video compression system provided by the present invention.

[0058] The present invention also provides a video compression system, including: an acquisition module 501, used to acquire video data to be compressed and audio data corresponding to the video data to be compressed; a compression module 502, used to input the video data to be compressed and the audio data into a pre-trained video compression model, and acquire the compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is obtained by training based on a video training set.

[0059] Figure 6 An example of a structural diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 601, a communications interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communications interface 602 and the memory 603 communicate with each other through the communication bus 604. The processor 601 may call the logic instructions in the memory 603 to execute the video compression method, which includes: obtaining the video data to be compressed and the audio data corresponding to the video data to be compressed; inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining the compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data uses the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is trained based on the video training set.

[0060] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0061] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video compression method provided by the above-mentioned methods, and the method includes: obtaining video data to be compressed and audio data corresponding to the video data to be compressed; inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process audio data and video data, so that the video data utilizes the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is trained based on a video training set.

[0062] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video compression method provided by the above-mentioned methods, the method comprising: obtaining video data to be compressed and audio data corresponding to the video data to be compressed; inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; wherein the pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance the video compression perception; the pre-trained video compression model is trained based on a video training set.

[0063] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video compression method, characterized in that: include: Acquire video data to be compressed and audio data corresponding to the video data to be compressed; Inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; The pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance video compression perception; the pre-trained video compression model is trained based on a video training set.

2. The video compression method according to claim 1, characterized in that: The video compression model includes a video encoder, an audio encoder, an audio and video attention channel integrator and a conditional decoder; The video encoder is used to encode the video data to be compressed; The audio encoder is used to encode the audio data; The audio-video attention channel integrator is used to synchronize the encoded video data and the encoded audio data by using channel fusion, and to capture features by using an external attention mechanism to obtain audio-video fusion features; The conditional decoder is used to decode the audio-video fusion feature to obtain the compressed video data.

3. The video compression method according to claim 2, characterized in that: The audio and video attention channel integrator includes a projection layer, a channel fusion layer, an external attention module and a residual convolution layer; The projection layer is used to project the encoded audio data into an audio space so that the projected audio data has the same dimension as the encoded video data; The channel fusion layer is used to spatially fuse the projected audio data with the encoded video data; The external attention module is used to identify and filter relevant information in the spatially fused audio and video data to obtain the audio-video fusion feature; The residual convolution layer is used to enhance the visual information of the audio-video fusion feature.

4. The video compression method according to claim 2, characterized in that: The conditional decoder includes a time embedding module, a SNeRV block, a TAT residual block and a header layer; The time embedding module is used to convert the frame index corresponding to the audio-video fusion feature into a time embedding vector; The SNeRV block is used to convert the audio-video fusion feature into an intermediate representation feature; The TAT residual block is used to align the intermediate representation feature with the temporal embedding vector; The header layer is used to convert the features output by the TAT residual block into the compressed video data.

5. The video compression method according to claim 2, characterized in that: The video compression model also includes a conditional decoder optimization module; The conditional decoder optimization module uses a pre-trained diffusion model combined with a probability density distillation loss function to perform gradient descent optimization on the conditional decoder; The pre-trained diffusion model adopts linear and exponential annealing strategies to dynamically adjust the time step in the diffusion process.

6. The video compression method according to claim 2, characterized in that: The step of inputting the video data to be compressed and the audio data into a pre-trained video compression model to obtain compressed video data output by the video compression model includes: Inputting the video data to be compressed into the video encoder to obtain encoded video data output by the video encoder; Inputting the audio data into the audio encoder to obtain encoded audio data output by the audio encoder; Inputting the encoded video data and the encoded audio data into the audio-video attention channel integrator to obtain the audio-video fusion features output by the audio-video attention channel integrator; The audio-video fusion feature is input into the conditional decoder to obtain the compressed video data output by the conditional decoder.

7. The video compression method according to any one of claims 1 to 6, characterized in that: The loss function of the pre-trained video compression model is: ， in, is the loss function of the pre-trained video compression model, is the L1 loss function, is the frequency domain L1 loss function, is the multi-scale structural similarity index loss function, is the probability density distillation loss function, For expectations, is noise, which follows the standard normal distribution , For data distribution x The input data sampled in are the parameters of the diffusion model, To predict the noise, x t For potential representation, t is the time step.

8. A video compression system, characterized in that: include: An acquisition module, used for acquiring video data to be compressed and audio data corresponding to the video data to be compressed; A compression module, used for inputting the video data to be compressed and the audio data into a pre-trained video compression model, and obtaining compressed video data output by the video compression model; The pre-trained video compression model is used to jointly process the audio data and the video data, so that the video data utilizes the implicit semantics of the audio data to enhance video compression perception; the pre-trained video compression model is trained based on a video training set.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the video compression method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video compression method according to any one of claims 1 to 7 is implemented.

Images