Video processing method and device

Abstract

The invention provides a video processing method and device, and the method comprises the steps: inputting a plurality of video frames of a to-be-processed video into a trained target encoder which comprises a plurality of unit blocks connected in series; in each unit block, 2D convolution space compression and 3D convolution time sequence compression are sequentially carried out on the video frame to obtain a video feature output by the last unit block, and the feature dimension of the video feature is lower than the feature dimension of the video to be processed; processing the video features by adopting a processing model; and a trained target decoder is adopted to reconstruct a processing result of the processing model into a final video, and the target encoder and the target decoder can enable the final video to be matched with the video quality of the video to be processed. The video generation quality and the compression efficiency are balanced.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method and device. Background Art

[0002] Some processing models have shown strong capabilities in video generation tasks, such as the DiT (Diffusion Transformer) model, but their huge computational workload and long training and inference time limit their widespread application, especially in high-resolution video generation tasks. Efficient video compression technology is crucial to reducing the computational workload of processing models and shortening training and inference time. Existing video compression technologies mainly include traditional methods (such as H.264, H.265, VP9, ​​etc.) and deep learning-based methods (such as convolutional neural networks, recurrent neural networks, etc.).

[0003] Traditional compression methods are difficult to effectively reduce the complexity of processing models, resulting in long training and inference times. In order to speed up processing, simplified or inefficient compression strategies may be adopted, which often sacrifices generation quality. Therefore, when current processing models are applied to video generation tasks, it is often difficult to balance compression efficiency and generation quality. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a video processing method and device.

[0005] In a first aspect, the present application provides a video processing method, the method comprising:

[0006] Inputting a plurality of video frames of a to-be-processed video into a trained target encoder, wherein the target encoder comprises a plurality of serially connected unit blocks;

[0007] In each of the unit blocks, the video frames are sequentially subjected to 2D convolution spatial compression and 3D convolution temporal compression to obtain video features output by the last unit block, wherein the feature dimension of the video features is lower than the feature dimension of the video to be processed;

[0008] Processing the video features using a processing model;

[0009] The trained target decoder is used to reconstruct the processing result of the processing model into a final video, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

[0010] Optionally, in each of the unit blocks, the video frames are sequentially subjected to 2D convolution spatial compression and 3D convolution temporal compression, and the video features output by the last unit block are obtained, including:

[0011] Inputting the plurality of video frames into a first unit block in the target encoder;

[0012] Using the first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the multiple video frames, and output a 3D convolution result;

[0013] Use a non-first unit block to perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block in sequence until the last unit block outputs the 3D convolution result;

[0014] The 3D convolution result output by the last unit block is used as the video feature.

[0015] Optionally, using the non-first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block includes:

[0016] Obtain the 3D convolution result output by the previous unit block through a non-first unit block;

[0017] Using a 2D convolution kernel to perform local feature extraction and spatial feature compression on the frame image in the 3D convolution result to obtain multiple 2D convolution results;

[0018] Stacking the multiple 2D convolution results into a 3D feature map;

[0019] A 3D convolution kernel is used to extract dynamic change features and compress sequence features of the 3D feature map to obtain a 3D convolution result.

[0020] Optionally, using a 3D convolution kernel to perform sequence feature compression on the 3D feature map includes:

[0021] Dividing the 3D feature map into a plurality of frame images;

[0022] Determine a region identifier in each frame image, wherein the region identifier is used to indicate whether a pixel change between a same region of a current frame and an adjacent frame exceeds a change threshold;

[0023] Determining a compression ratio of a corresponding area according to the area identifier;

[0024] A 3D convolution kernel is used to compress the sequence features of the region according to the compression ratio.

[0025] Optionally, determining the compression ratio of the corresponding area according to the area identifier includes:

[0026] If the corresponding area is determined to be a violent motion area according to the area identifier, determining that the area adopts a first compression ratio, wherein the violent motion area is used to indicate that a pixel change exceeds a change threshold;

[0027] If the corresponding area is determined to be a smooth motion area according to the area identifier, the area is determined to use a second compression ratio, wherein the smooth motion area is used to indicate that the pixel change does not exceed the change threshold, and the second compression ratio is greater than the first compression ratio.

[0028] Optionally, the training process of the target encoder and the target decoder includes:

[0029] Acquire sample data, wherein the sample data includes a sample video and sample video frames decomposed from the sample video;

[0030] Using the sample data to train an initial encoder and an initial decoder to obtain an output video;

[0031] If the video difference between the output video and the sample video is greater than a threshold, the parameters in the initial encoder and the initial decoder are adjusted until the target encoder and the target decoder are obtained.

[0032] Optionally, using a trained target decoder to reconstruct the processing result of the processing model into a final video comprises:

[0033] Using a trained target decoder to decode the processing result of the processing model and reconstruct it into a video view sequence;

[0034] The video view sequences are merged into a final video.

[0035] In a second aspect, the present application provides a video processing method, the method comprising:

[0036] An input module, used for inputting a plurality of video frames of a video to be processed into a trained target encoder, wherein the target encoder comprises a plurality of unit blocks connected in series;

[0037] A convolution module, used for sequentially performing 2D convolution spatial compression and 3D convolution temporal compression on the video frame in each unit block to obtain a video feature output by the last unit block, wherein the feature dimension of the video feature is lower than the feature dimension of the video to be processed;

[0038] A processing module, used for processing the video features by using a processing model;

[0039] A reconstruction module is used to reconstruct the processing result of the processing model into a final video using a trained target decoder, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

[0040] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0041] Memory, used to store computer programs;

[0042] The processor is used to implement any of the steps of the video processing method when executing the program stored in the memory.

[0043] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the steps of the video processing method is implemented.

[0044] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0045] The method provided in the embodiment of the present application performs 2D spatial compression and 3D temporal compression on video frames through the unit blocks of the target encoder, thereby reducing redundant features of the video and thus reducing the feature dimension of the video, and the series connection of multiple unit blocks can iteratively reduce the feature dimension, so that the processing model can reduce the amount of calculation when processing video features with lower dimensions, thereby improving processing efficiency. The subsequent target decoder restores the video frames to the final video. Since the target encoder and target decoder have been trained, the quality of the finally generated video is equivalent to the quality of the original video to be processed, thereby balancing the video generation quality and compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a hardware environment of a video processing method provided in an embodiment of the present application;

[0049] Figure 2A flowchart of a video processing method provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the overall process of video processing provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0055] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a video processing method is provided.

[0056] Optionally, in the embodiment of the present application, the above video processing method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and the user uploads the video to be processed via the terminal. The server encodes, models and decodes the video to be processed and outputs the final video. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0057] The following will be combined with a specific implementation method to provide a video processing method provided by an embodiment of the present application in detail, which is applied to a server, such as Figure 2 As shown, the specific steps are as follows:

[0058] Step 201: input multiple video frames of a video to be processed into a trained target encoder, wherein the target encoder includes multiple serially connected unit blocks;

[0059] Step 202: performing 2D convolution spatial compression and 3D convolution temporal compression on the video frames in each unit block in turn to obtain the video features output by the last unit block, wherein the feature dimension of the video features is lower than the feature dimension of the video to be processed;

[0060] Step 203: using a processing model to process the video features;

[0061] Step 204: Use the trained target decoder to reconstruct the processing result of the processing model into a final video, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

[0062] The user uploads the video to be processed through the terminal, and the server decomposes the video to be processed into multiple video frames, and then inputs the multiple video frames into the trained target encoder. The encoder contains multiple serially connected unit blocks, each of which is responsible for gradually compressing and extracting 2D features and 3D features.

[0063] Specifically, the target encoder performs 2D convolution operations on a single frame or multiple frames to extract and compress spatial features, reduce the spatial redundancy of each frame, and then stacks the output of the 2D convolution into a 3D feature map and uses a 3D convolution kernel to perform a convolution operation on it to achieve downsampling and compression in the time dimension, thereby reducing the feature dimension through the unit block, and then gradually reducing the feature dimension through iterative convolution of multiple unit blocks in series, and finally obtaining the low-dimensional video features output by the last unit block.

[0064] The server inputs the low-dimensional video features into the processing model (such as the DiT model) for further processing, so that the processing model can operate in a smaller latent space, reducing the overall amount of computation and shortening the training and inference time. Finally, the server decodes the processing results of the processing model through the trained target decoder (for example, by deconvolution, upsampling, etc.), reconstructs the video view sequence, and then merges the video view sequence into the final video. The decoder is responsible for restoring the details in the process of returning from the low-dimensional latent space to the high-dimensional video space, and restores the image information as losslessly as possible. Since the target encoder and target decoder are both pre-trained, the encoder-decoder architecture ensures that the final video matches the quality of the original video, and effectively restores the details in the generated video, ensuring the authenticity and clarity of the visual effect.

[0065] The present application performs 2D spatial compression and 3D temporal compression on video frames through the unit blocks of the target encoder, thereby reducing the redundant features of the video and thus reducing the feature dimension of the video. The series connection of multiple unit blocks can iteratively reduce the feature dimension, so that the processing model can reduce the amount of calculation when processing video features with lower dimensions, thereby improving processing efficiency. The subsequent target decoder restores the video frames to the final video. Since the target encoder and target decoder have been trained, the quality of the final generated video is equivalent to that of the original video to be processed, thereby balancing the video generation quality and compression efficiency.

[0066] As an optional implementation, in step 202, 2D convolution spatial compression and 3D convolution temporal compression are sequentially performed on the video frames in each unit block, and obtaining the video features output by the last unit block includes the following steps:

[0067] Step S11: inputting a plurality of video frames into a first unit block in a target encoder;

[0068] Step S12: using the first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on multiple video frames, and outputting a 3D convolution result;

[0069] Step S13: using a non-first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block, until the last unit block outputs the 3D convolution result;

[0070] Step S14: The 3D convolution result output by the last unit block is used as the video feature.

[0071] The server inputs multiple video frames of the video to be processed into the first unit block in the target encoder, where each video frame is an independent 2D image, but together they constitute a continuous time series. In the first unit block, the unit block performs 2D convolution spatial compression and 3D convolution temporal compression on a single frame image or multiple frame images in sequence to obtain a 3D convolution result. In subsequent unit blocks, each unit block further compresses and extracts features based on the previous layer of unit blocks, gradually removes redundant information, and retains important spatiotemporal features. Through the serial processing of multiple unit blocks, it can capture high-level abstract features in the video more deeply, and improve the learning ability of the model. The server uses the 3D convolution result output by the last unit block as the final video feature. At this time, the feature dimension is significantly lower than the feature dimension of the original video, but it still retains sufficient spatiotemporal information.

[0072] In this application, by directly inputting multiple frames of video into the first unit block, the feature extraction process is quickly started, while ensuring that the information of all original frames can be fully transmitted. Combining 2D and 3D convolutions, it is possible to capture local features within a single frame and integrate the temporal relationship between frames to ensure the quality of the generated video. At the same time, the feature dimension is initially reduced, and then through the series processing of multiple unit blocks, redundant information is gradually removed while retaining important spatiotemporal features. The final low-dimensional video features not only contain rich spatiotemporal information, but also greatly reduce the amount of calculation, thereby improving video processing efficiency.

[0073] As an optional implementation, in step S12, using a non-first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block includes the following steps:

[0074] Step S21: obtaining the 3D convolution result output by the previous unit block through a non-first unit block;

[0075] Step S22: using a 2D convolution kernel to perform local feature extraction and spatial feature compression on the frame image in the 3D convolution result to obtain multiple 2D convolution results;

[0076] Step S23: stack multiple 2D convolution results into a 3D feature map;

[0077] Step S24: Use a 3D convolution kernel to extract dynamic change features and compress sequence features on the 3D feature map to obtain a 3D convolution result.

[0078] The non-first unit block obtains the 3D convolution result output by the previous unit block. The 3D convolution result contains a feature map after preliminary spatiotemporal compression, and each slice represents the spatial feature of a frame of image.

[0079] This unit block first uses a 2D convolution kernel to perform a convolution operation on each frame image (i.e., each 2D slice) in the 3D convolution result to extract local features, including visual elements such as edges and textures. The 2D convolution operation can compress the spatial redundancy of each frame image, remove unnecessary details, retain important information, and capture finer local features, thereby enhancing the model's ability to understand complex scenes.

[0080] Multiple 2D convolution results are restacked into a new 3D feature map in chronological order. Through the stacking operation, the time series relationship between frames is reconstructed to ensure the continuity and dynamic change characteristics of the video data. Then the new 3D feature map is convolved using a 3D convolution kernel to capture the dynamic change characteristics between frames, such as the trajectory of moving objects, changes in scenes, etc., and finally a 3D convolution result is obtained. This application can achieve downsampling and compression in the time dimension through 3D convolution operations, further reducing the data dimension while retaining important timing information. In addition, 3D convolution can effectively capture the movement of objects in the video and the changes in the scene, enhancing the realism and coherence of the generated video.

[0081] This application captures finer local features through 2D convolution, effectively removes spatial redundancy in each frame, and then reconstructs the temporal relationship between frames through stacking operations on the 2D convolution results, integrating features of different scales to form a 3D feature map containing rich spatiotemporal information, and then compresses in the time dimension through 3D convolution, and effectively captures the movement of objects in the video and changes in scenes. This application reduces the amount of data processed and improves computational efficiency through spatial compression of 2D convolution and temporal compression of 3D convolution.

[0082] As an optional implementation, in step S24, using a 3D convolution kernel to perform sequence feature compression on the 3D feature map includes the following steps:

[0083] Step S31: dividing the 3D feature map into multiple frame images;

[0084] Step S32: determining a region identifier in each frame image, wherein the region identifier is used to indicate whether a pixel change between the same region of the current frame and an adjacent frame exceeds a change threshold;

[0085] Step S33: determining the compression ratio of the corresponding area according to the area identifier;

[0086] Step S34: Use a 3D convolution kernel to compress the sequence features of the region according to the compression ratio.

[0087] The target encoder decomposes the obtained 3D feature map into a series of independent 2D frame images according to the time axis. Each frame image represents the spatial feature distribution at a time point, so as to perform more refined operations and analysis on the spatial features at each time point.

[0088] For each frame image, the pixel changes between it and the adjacent frames are analyzed, and different types of regions are marked according to the preset change threshold. Specifically, if the pixel changes in a region exceed the set threshold, the region is marked as a violent motion region; conversely, if the changes are not significant, it is marked as a gentle motion region.

[0089] Areas with intense motion refer to areas where pixel values ​​change significantly in a short period of time (i.e., within a few frames). For example, fast-moving objects, intense movements, or frequently flickering light sources. These areas show large differences between adjacent frames.

[0090] Slow motion areas refer to areas where the pixel values ​​in certain areas have almost no change or very small change in the same period of time, such as static backgrounds, slowly moving objects, etc. These areas show small differences between adjacent frames.

[0091] For the parts marked as areas with intense motion, a lower first compression ratio is used, that is, more details are retained. For the parts marked as areas with gentle motion, a higher second compression ratio is used, that is, redundant information is reduced. The compression strategy is adjusted according to the actual motion conditions of the area, which can not only ensure the high fidelity of areas with intense motion, but also effectively compress areas with gentle motion, reducing unnecessary calculations, and achieving a higher compression rate while ensuring that key information is not lost, thereby improving overall processing efficiency.

[0092] In this application, adjusting the compression ratio actually refers to downsampling or step-size adjustment of the time dimension during the 3D convolution process to reduce the number of frames processed. For example:

[0093] 1. Larger compression ratio: This can be understood as using a larger step size on the timeline (for example, jumping two frames or more each time), thereby reducing the total number of frames that need to be processed. This is effective for areas with slow motion, because these areas have small changes, and even skipping some frames will not significantly affect the visual quality.

[0094] 2. Smaller compression ratio: This means using a smaller step size (for example, only moving one frame at a time), ensuring higher temporal resolution, which is very important for capturing fast changes or complex movements.

[0095] In practical applications, for areas with gentle motion, a larger temporal compression ratio can be safely used, because even if the number of frames is reduced, not much visual information will be lost, and a larger time step can be set to reduce the amount of calculation and improve efficiency. For areas with intense motion, a higher temporal resolution is required to capture rapidly changing details, so the compression ratio should be reduced, and frame-by-frame processing may even be required to retain sufficient details and dynamic information.

[0096] As an optional implementation, the training process of the target encoder and the target decoder includes the following steps:

[0097] Step S41: acquiring sample data, wherein the sample data includes a sample video and sample video frames decomposed from the sample video;

[0098] Step S42: using sample data to train the initial encoder and the initial decoder to obtain an output video;

[0099] Step S43: If the video difference between the output video and the sample video is greater than a threshold, the parameters in the initial encoder and the initial decoder are adjusted until a target encoder and a target decoder are obtained.

[0100] The user collects representative sample videos that cover various scenarios and content types that may be encountered in the target application. The server decomposes each sample video into a series of frame images (ie, sample video frames).

[0101] The server uses the collected sample data as input to iteratively train the encoder and decoder in their initial state. In this process, the encoder is responsible for converting the original video into a compressed representation, while the decoder is responsible for reconstructing the video from this compressed representation. After multiple iterations, the internal parameters of the encoder and decoder are continuously optimized in order to make the final output video as close to the original sample video as possible.

[0102] After each training iteration, the difference between the output video and the corresponding sample video is calculated (such as using mean square error or other appropriate metrics). If this difference exceeds the preset threshold, it means that the performance of the current model is not good enough, and the parameters of the encoder and decoder need to be further adjusted to continue optimization. The above process is repeated until the difference between the output video and the sample video is controlled within an acceptable range. At this time, the encoder and decoder are the target model. Among them, the encoder and decoder can adopt the Transformer structure, and use the long-distance modeling ability of the Transformer to better capture the spatiotemporal dependencies of the video, thereby improving the quality of video reconstruction.

[0103] Through the training of the encoder and decoder, the final generated video is consistent with the original video in quality. In addition, by compressing the video in space and time, the redundancy of the time and space dimensions of the video data is reduced, the computational complexity of the processing model is reduced, and the efficiency of video generation and editing is improved.

[0104] Based on the same technical concept, the embodiment of the present application also provides a video processing flow, such as Figure 3 As shown, the following steps are included:

[0105] Step 301: Input a video frame sequence.

[0106] The video to be processed is decomposed into a series of frames to prepare basic data for subsequent time-space compression, ensuring that the image information at each time point can be processed separately.

[0107] Step 302: The encoder performs joint temporal and spatial compression (first block).

[0108] 2D spatial compression: Perform 2D convolution operations on each frame to extract and compress spatial features. The parameters of the 2D convolution kernel (such as size, step size, and number of channels) are adjusted according to the specific application scenario, which can reduce the spatial redundancy of each frame image, retain important local features, and improve computational efficiency.

[0109] 3D temporal compression: The outputs of multiple frames of 2D convolution are stacked into a 3D feature map, and a 3D convolution kernel is used to perform convolution operations on it to achieve downsampling and compression in the temporal dimension. By controlling the step size (for example, setting a step size greater than 1 in the temporal dimension), the data dimension can be further reduced. By capturing the temporal relationship between frames, the temporal redundancy of the video sequence is reduced while maintaining dynamically changing information.

[0110] Areas with relatively slow motion: Using a larger time dimension compression ratio can save computing resources, avoid over-processing of static or slowly changing scenes, and improve overall processing efficiency.

[0111] Areas with intense motion: Reduce the compression ratio to retain more detail information. This ensures that areas with rapid changes or complex motion retain sufficient details and prevents visual distortion caused by over-compression.

[0112] Step 303: The subsequent unit blocks perform 2D and 3D compression processing.

[0113] Each subsequent block receives the 3D feature map output by the previous block as input, repeats the above 2D spatial compression and 3D temporal compression process, and gradually reduces the feature dimension. By compressing the features layer by layer, it not only ensures the retention of important information, but also further reduces the computational complexity. As the compression level deepens, the feature representation becomes more compact and abstract, which helps the DiT model process more efficiently.

[0114] Step 304: low-dimensional video feature input processing model.

[0115] The low-dimensional representation after the joint temporal and spatial compression of multiple blocks is directly input into the processing model for training or reasoning. The processing model can operate in a smaller latent space, reducing the overall computational effort, shortening the training and reasoning time, while maintaining the quality of the generated video.

[0116] Step 305: The decoder reconstructs the video.

[0117] The output of the DiT model is reconstructed into the final video output through the decoder. The original video frame can be restored using methods such as deconvolution and upsampling. This ensures that the details are restored as losslessly as possible in the process of returning from the low-dimensional latent space to the high-dimensional video space, thereby ensuring the quality of the generated video.

[0118] Step 306: Output the generated video.

[0119] Merge the reconstructed video frames output by the decoder into the final video output. Provide high-quality generated video to meet user needs.

[0120] Based on the same technical concept, the embodiment of the present application also provides a video processing device, such as Figure 4 As shown, the device comprises:

[0121] An input module 401 is used to input a plurality of video frames of a video to be processed into a trained target encoder, wherein the target encoder comprises a plurality of unit blocks connected in series;

[0122] A convolution module 402 is used to perform 2D convolution spatial compression and 3D convolution temporal compression on the video frame in each unit block in turn to obtain the video features output by the last unit block, wherein the feature dimension of the video features is lower than the feature dimension of the video to be processed;

[0123] A processing module 403, used to process the video features using a processing model;

[0124] The reconstruction module 404 is used to reconstruct the processing result of the processing model into a final video using a trained target decoder, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

[0125] Optionally, the convolution module 402 is used to:

[0126] Inputting a plurality of video frames into a first unit block in a target encoder;

[0127] The first unit block is used to perform 2D convolution spatial compression and 3D convolution temporal compression on multiple video frames in sequence, and the 3D convolution result is output;

[0128] Use a non-first unit block to perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block in sequence until the last unit block outputs the 3D convolution result;

[0129] The 3D convolution result output by the last unit block is used as the video feature.

[0130] Optionally, the convolution module 402 is used to:

[0131] Obtain the 3D convolution result output by the previous unit block through a non-first unit block;

[0132] Use 2D convolution kernel to extract local features and compress spatial features of frame images in 3D convolution results to obtain multiple 2D convolution results;

[0133] Stack multiple 2D convolution results into a 3D feature map;

[0134] The 3D convolution kernel is used to extract dynamic change features and compress sequence features of the 3D feature map to obtain the 3D convolution result.

[0135] Optionally, the convolution module 402 is used to:

[0136] Divide the 3D feature map into multiple frame images;

[0137] Determine a region identifier in each frame image, wherein the region identifier is used to indicate whether a pixel change between a same region of a current frame and an adjacent frame exceeds a change threshold;

[0138] Determine the compression ratio of the corresponding area according to the area identifier;

[0139] The 3D convolution kernel is used to compress the sequence features of the region according to the compression ratio.

[0140] Optionally, the convolution module 402 is used to:

[0141] If the corresponding area is determined to be a violent motion area according to the area identifier, the first compression ratio is used for the determined area, wherein the violent motion area is used to indicate that the pixel change exceeds the change threshold;

[0142] If the corresponding area is determined to be a smooth motion area according to the area identifier, the second compression ratio is determined to be used for the area, wherein the smooth motion area is used to indicate that the pixel change does not exceed the change threshold, and the second compression ratio is greater than the first compression ratio.

[0143] Optionally, the device is also used for:

[0144] Acquire sample data, wherein the sample data includes a sample video and sample video frames decomposed from the sample video;

[0145] The sample data is used to train the initial encoder and the initial decoder to obtain an output video;

[0146] If the video difference between the output video and the sample video is greater than a threshold, the parameters in the initial encoder and the initial decoder are adjusted until a target encoder and a target decoder are obtained.

[0147] Optionally, the reconstruction module 404 is used to:

[0148] The trained target decoder is used to decode the processing result of the processing model and reconstruct the video sequence;

[0149] Merge video view sequences into a final video.

[0150] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0151] Memory 503, used for storing computer programs;

[0152] The processor 501 is used to implement the above steps when executing the program stored in the memory 503.

[0153] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] The communication interface is used for communication between the above electronic device and other devices.

[0155] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0156] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0157] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0158] In another embodiment of the present invention, a computer program product including instructions is provided, which enables the computer to execute any one of the methods in the above embodiments when the computer program product is executed on the computer.

[0159] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0160] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0161] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video processing method, characterized in that: The method comprises: Inputting a plurality of video frames of a to-be-processed video into a trained target encoder, wherein the target encoder comprises a plurality of serially connected unit blocks; In each of the unit blocks, the video frames are sequentially subjected to 2D convolution spatial compression and 3D convolution temporal compression to obtain video features output by the last unit block, wherein the feature dimension of the video features is lower than the feature dimension of the video to be processed; Processing the video features using a processing model; The trained target decoder is used to reconstruct the processing result of the processing model into a final video, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

2. The method according to claim 1, characterized in that In each of the unit blocks, the video frames are sequentially subjected to 2D convolution spatial compression and 3D convolution temporal compression, and the video features output by the last unit block are obtained, including: Inputting the plurality of video frames into a first unit block in the target encoder; Using the first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the multiple video frames, and output a 3D convolution result; Use a non-first unit block to perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block in sequence until the last unit block outputs the 3D convolution result; The 3D convolution result output by the last unit block is used as the video feature.

3. The method according to claim 2, characterized in that Using the non-first unit block to sequentially perform 2D convolution spatial compression and 3D convolution temporal compression on the 3D convolution result output by the previous unit block includes: Obtain the 3D convolution result output by the previous unit block through a non-first unit block; Using a 2D convolution kernel to perform local feature extraction and spatial feature compression on the frame image in the 3D convolution result to obtain multiple 2D convolution results; Stacking the multiple 2D convolution results into a 3D feature map; A 3D convolution kernel is used to extract dynamic change features and compress sequence features of the 3D feature map to obtain a 3D convolution result.

4. The method according to claim 3, characterized in that Using a 3D convolution kernel to perform sequence feature compression on the 3D feature map includes: Dividing the 3D feature map into a plurality of frame images; Determine a region identifier in each frame image, wherein the region identifier is used to indicate whether a pixel change between a same region of a current frame and an adjacent frame exceeds a change threshold; Determining a compression ratio of a corresponding area according to the area identifier; A 3D convolution kernel is used to compress the sequence features of the region according to the compression ratio.

5. The method according to claim 4, characterized in that Determining the compression ratio of the corresponding area according to the area identifier includes: If the corresponding area is determined to be a violent motion area according to the area identifier, determining that the area adopts a first compression ratio, wherein the violent motion area is used to indicate that a pixel change exceeds a change threshold; If the corresponding area is determined to be a smooth motion area according to the area identifier, the area is determined to use a second compression ratio, wherein the smooth motion area is used to indicate that the pixel change does not exceed the change threshold, and the second compression ratio is greater than the first compression ratio.

6. The method according to claim 1, characterized in that The training process of the target encoder and the target decoder includes: Acquire sample data, wherein the sample data includes a sample video and sample video frames decomposed from the sample video; Using the sample data to train an initial encoder and an initial decoder to obtain an output video; If the video difference between the output video and the sample video is greater than a threshold, the parameters in the initial encoder and the initial decoder are adjusted until the target encoder and the target decoder are obtained.

7. The method according to claim 1, characterized in that Using the trained target decoder to reconstruct the processing result of the processing model into a final video comprises: Using a trained target decoder to decode the processing result of the processing model and reconstruct it into a video view sequence; The video view sequences are merged into a final video.

8. A video processing method, characterized in that: The method comprises: An input module, used for inputting a plurality of video frames of a video to be processed into a trained target encoder, wherein the target encoder comprises a plurality of unit blocks connected in series; A convolution module, used for sequentially performing 2D convolution spatial compression and 3D convolution temporal compression on the video frame in each unit block to obtain a video feature output by the last unit block, wherein the feature dimension of the video feature is lower than the feature dimension of the video to be processed; A processing module, used for processing the video features by using a processing model; A reconstruction module is used to reconstruct the processing result of the processing model into a final video using a trained target decoder, wherein the target encoder and the target decoder can make the final video match the video quality of the video to be processed.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images