Video self-encoding method and device, electronic equipment and storage medium
By using a video autoencoding method and causal memory networks to process video frames, the problems of high computational complexity and encoding/decoding errors in existing technologies are solved, achieving more efficient and accurate video encoding and decoding, and reducing jitter artifacts in reconstructed videos.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-04-10
AI Technical Summary
Existing video encoding and decoding technologies suffer from high computational complexity, low encoding and decoding efficiency, and are prone to errors. In particular, the accumulation of prediction errors under complex motion conditions leads to jitter artifacts in the reconstructed video.
By employing a video autoencoding method, through steps such as pixel preprocessing, downsampling, feature encoding, feature decoding, and upsampling, a causal memory network is used to capture motion correlations in the video, generating a more temporally coherent latent video representation.
It reduces computational complexity, improves encoding and decoding accuracy, significantly reduces jitter artifacts in reconstructed videos, and enhances encoding and decoding efficiency.
Smart Images
Figure CN121842397A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and is applicable to the financial and medical fields. In particular, it relates to a video self-encoding method and apparatus, electronic device, and storage medium. Background Technology
[0002] In the field of video encoding and decoding, video bitrate can be reduced by encoding the video. For example, in financial claims scenarios, video evidence used by applicants for claims (such as accident scene videos, videos of interactions between vehicles and the environment) can be encoded and decoded to reduce memory usage. Similarly, in medical image management scenarios in the healthcare field, medical imaging videos (such as electrocardiogram videos, continuous echocardiograms, and multispectral endoscopic videos) can be encoded and decoded to reduce memory usage.
[0003] Related technologies primarily rely on traditional standards (such as H.265 and AV1) and neural network models based on 3D convolution. However, these solutions have the following significant drawbacks: Traditional video coding standards employ block-matching motion estimation and motion compensation techniques, requiring recursive searching of matching blocks in reference frames, resulting in high computational complexity and impacting encoding / decoding efficiency. Standard codecs such as those based on 3D convolution neural network models rely on motion vectors to transmit inter-frame information, but complex motions (such as occlusion and deformation) can easily lead to the accumulation of prediction errors, resulting in low encoding / decoding accuracy. Summary of the Invention
[0004] The main objective of this application is to propose a video self-encoding method, apparatus, electronic device, and storage medium that can solve the technical problems of low encoding and decoding efficiency and easy error occurrence. By effectively capturing motion correlation in the video, it reduces computational complexity and improves accuracy, and in particular, it can significantly reduce the occurrence of jitter artifacts in the reconstructed video.
[0005] To achieve the above objectives, a first aspect of this application proposes a video auto-encoding method, the method comprising: The original video is acquired, and pixel preprocessing is performed on each image frame of the original video to obtain an initial image feature sequence; wherein, the initial image feature sequence includes the initial image features of the image frame, and the image frame includes the current image frame and the previous image frame before the current image frame; The initial image features are downsampled to obtain downsampled image features; The encoded image features corresponding to the previous image frame are used as the first memory features, and the downsampled image features of the current image frame and the first memory features are used for feature encoding to obtain the encoded image features of the current image frame. The decoded image features corresponding to the previous image frame are used as the second memory features, and the encoded image features of the current image frame and the second memory features are used for feature decoding to obtain the decoded image features of the current image frame. The decoded image features are upsampled to obtain upsampled image features; The upsampled image features of all the image frames are post-processed at the pixel level to obtain the target video.
[0006] Optionally, the step of feature encoding the downsampled image features of the current image frame and the first memory features to obtain the encoded image features of the current image frame includes: Obtain a first causal memory network, which includes a first splicing layer, a first convolutional layer, a first skip connection layer, and a first activation function layer; The downsampled image features and the first memory features are spliced together by the first splicing layer to obtain the first spliced feature; The first convolutional feature is obtained by performing feature convolution on the first concatenated feature through the first convolutional layer; The first spliced feature and the first convolutional feature are connected by the first skip connection layer to obtain the first connected feature. The first connection feature is activated by the first activation function layer to obtain the encoded image features of the current image frame.
[0007] Optionally, before using the encoded image features corresponding to the previous image frame as the first memory feature, the method further includes: When the current image frame is the first frame of the original video, generate the encoded image features of the previous image frame whose elements are all 0 and whose dimensions are the same as the downsampled image features of the current image frame; If the current image frame is not the first frame of the original video, then the encoded image features are obtained by performing feature encoding based on the downsampled image features and the first memory features of the previous image frame.
[0008] Optionally, downsampling the initial image features to obtain downsampled image features includes: Channel expansion is performed on the channels of the initial image features to obtain temporally downsampled features; The width and height of the initial image features are downsampled to obtain spatial downsampled features; The downsampled image features are obtained by merging the temporal downsampling features and the spatial downsampling features.
[0009] Optionally, the channel expansion of the initial image features to obtain temporally downsampled features includes: The number of frames is determined based on the downsampling factor of the time downsampling, and image feature combinations are extracted from the initial image feature sequence based on the number of frames; any two image feature combinations do not overlap; The initial image features in the image feature combination are concatenated by channels to obtain the initial time downsampling features; The initial temporal downsampling features are mapped to the number of channels using a 1x1 convolutional layer to obtain temporal downsampling sub-features; The time-downsampled features are obtained by merging all the time-downsampled sub-features.
[0010] Optionally, the upsampling of the decoded image features to obtain upsampled image features includes: The upsampling factor is determined based on the downsampling factor; wherein the upsampling factor is equal to the downsampling factor; Spatial upsampled features are obtained by expanding the number of channels of the decoded image features by the upsampling factor through a 1x1 convolutional layer. The spatial upsampled features are arranged to obtain the upsampled image features; wherein the temporal and spatial dimensions of the upsampled image features are the same as those of the initial image features.
[0011] Optionally, the step of performing pixel preprocessing on the original video to obtain an initial image feature sequence includes: The feature tensor of the original video is split along the time dimension to obtain the first initial image features of the image frame; wherein, the first initial image features have spatial dimension and channel dimension; The adjacent s×s pixel blocks in the first initial image feature are expanded in the spatial dimension and arranged in the channel dimension to obtain the pixel rearrangement feature. The pixel rearrangement features are reduced by a factor of 1 / s in spatial dimension to obtain the second initial image features of the image frame; The initial image feature sequence is obtained by merging the second initial image features of each of the image frames.
[0012] To achieve the above objectives, a second aspect of this application provides a video autoencoding apparatus, the apparatus comprising: The video preprocessing module is used to acquire the original video and perform pixel preprocessing on each image frame of the original video to obtain an initial image feature sequence; wherein, the initial image feature sequence includes the initial image features of the image frame, and the image frame includes the current image frame and the previous image frame before the current image frame; The downsampling module is used to downsample the initial image features to obtain downsampled image features; The feature encoding module is used to take the encoded image features corresponding to the previous image frame as the first memory feature, and to perform feature encoding on the downsampled image features of the current image frame and the first memory feature to obtain the encoded image features of the current image frame. The feature decoding module is used to take the decoded image features corresponding to the previous image frame as the second memory features, and perform feature decoding on the encoded image features of the current image frame and the second memory features to obtain the decoded image features of the current image frame. The upsampling module is used to upsample the decoded image features to obtain upsampled image features; The video reconstruction module is used to perform pixel post-processing on the upsampled image features of all the image frames to obtain the target video.
[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the video self-encoding method described in the first aspect.
[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video auto-encoding method described in the first aspect.
[0015] The video self-encoding method, apparatus, electronic device, and storage medium proposed in this application sequentially perform pixel preprocessing, downsampling, feature encoding, feature decoding, upsampling, and pixel post-processing on the original video. Particularly during the feature encoding (decoding) process, the encoded image features (decoded image features) of the current image frame are generated based on the encoded image features (decoded image features) of the previous image frame. This allows information to propagate along the temporal dimension, thereby generating a more temporally coherent latent video representation and ultimately obtaining the reconstructed target video. In summary, this application effectively captures motion correlations in the video, reducing computational complexity and improving accuracy, particularly significantly reducing jitter artifacts in the reconstructed video.
[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0017] Figure 1 This is a flowchart of the video self-encoding method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step 101 in the document; Figure 3 yes Figure 1 The flowchart for step 102 in the document; Figure 4 yes Figure 3 The flowchart for step 301 in the document; Figure 5 yes Figure 1 The flowchart for step 103 in the document; Figure 6 yes Figure 1 The flowchart for step 105 in the document; Figure 7 This is a block diagram of the module structure of the video self-encoding device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0022] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0023] Currently, the video encoding and decoding field mainly relies on traditional standards (such as H.265 and AV1) and neural network models based on 3D convolution, but these solutions have the following significant drawbacks: (1) Low efficiency of temporal modeling: Traditional video coding standards use block matching motion estimation and motion compensation (MEMC) technology, which requires recursively searching for matching blocks in reference frames, resulting in extremely high computational complexity. For example, the coding complexity of HEVC is about 253% higher than that of H.264. While neural networks based on 3D convolution can automatically extract spatiotemporal features, the weight sharing mechanism of convolution kernels in the temporal dimension is difficult to capture long-range dependencies, and the large number of parameters leads to high training and inference costs.
[0024] (2) Memory and computing resource bottlenecks: Existing neural network encoders usually need to load all video frames at once for parallel processing, and the memory usage is proportional to the video length (O(T)), making it difficult to support long videos or high-resolution scenes. Some studies have attempted to reduce memory consumption through serialization, but often due to the lack of efficient memory management mechanisms, the inter-frame information transmission is insufficient, resulting in temporal inconsistencies in the reconstructed video (such as flickering and jitter).
[0025] (3) Insufficient flexibility and multi-scale generation capability: Traditional methods require retraining the model for different resolutions, frame rates or bit rates, lacking dynamic adaptation capability. For example, in real-time communication scenarios, in order to adapt to network fluctuations, it is often necessary to encode video streams of different qualities multiple times, resulting in high resource consumption. In addition, existing neural network models mostly adopt fixed upsampling / downsampling strategies, which make it difficult to take into account the controllable generation requirements of spatiotemporal dimensions (such as improving spatial resolution while maintaining temporal resolution).
[0026] (4) Insufficient guarantee of inter-frame continuity: Standard codecs rely on motion vectors to transmit inter-frame information, but complex motions (such as occlusion and deformation) can easily lead to the accumulation of prediction errors. Although some studies have introduced optical flow methods or recursive networks (such as LSTM) to enhance temporal consistency, optical flow estimation itself is computationally intensive, and the sequence processing mode of LSTM hinders parallelization, making it difficult to balance efficiency and quality.
[0027] This application belongs to the field of artificial intelligence technology, more specifically to the field of computer vision. It adopts an encoder-decoder symmetric architecture, but breaks through the traditional design by using the core idea of spatiotemporal separation. The encoder realizes the mapping from video to latent space through temporal pooling and spatial convolution, while the decoder completes the reconstruction through temporal growth and spatial upsampling.
[0028] The video self-encoding method provided in this application can be applied to terminals and servers, or it can be software running on the server. The server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or it can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video self-encoding method, etc., but is not limited to the above forms.
[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0030] This application provides a video self-encoding method, a video self-encoding device, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the video self-encoding method in the embodiments of this application is described.
[0031] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to the user's identity or characteristics, such as image data and video data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.
[0032] Reference Figure 1 , Figure 1 This is an optional flowchart of the video self-encoding method provided in the embodiments of this application, which may include, but is not limited to, steps 101 to 106.
[0033] Step 101: Obtain the original video, perform pixel preprocessing on each image frame of the original video, and obtain the initial image feature sequence; Step 102: Downsample the initial image features to obtain downsampled image features; Step 103: Use the encoded image features corresponding to the previous image frame as the first memory features, and encode the downsampled image features of the current image frame and the first memory features to obtain the encoded image features of the current image frame. Step 104: Use the decoded image features corresponding to the previous image frame as the second memory features, and perform feature decoding on the encoded image features and the second memory features of the current image frame to obtain the decoded image features of the current image frame. Step 105: Upsample the decoded image features to obtain upsampled image features; Step 106: Perform pixel post-processing on the upsampled image features of all image frames to obtain the target video.
[0034] Steps 101 to 106, as illustrated in the embodiments of this application, sequentially perform pixel preprocessing, downsampling, feature encoding, feature decoding, upsampling, and pixel post-processing on the original video. Particularly during the feature encoding (decoding) process, the encoded image features (decoded image features) of the current image frame are generated based on the encoded image features (decoded image features) of the previous image frame. This allows information to propagate along the temporal dimension, thereby generating a more temporally coherent latent video representation and ultimately obtaining the reconstructed target video. In summary, this application effectively captures motion correlations in the video, reducing computational complexity and improving accuracy, particularly significantly reducing jitter artifacts in the reconstructed video.
[0035] For example, in the financial claims process, video evidence used by the applicant for claims (such as accident scene videos, videos of the interaction between the vehicles and the environment) can be encoded and decoded to reduce memory usage.
[0036] For example, in medical image management scenarios in the healthcare field, medical image videos (such as electrocardiogram videos, continuous echocardiograms, multispectral endoscopic videos, etc.) can be encoded and decoded to reduce memory usage.
[0037] In step 101 of some embodiments, the original video is acquired, and pixel preprocessing is performed on each image frame of the original video to obtain an initial image feature sequence. The original video can be acquired using a camera. In the financial field, the original video can be video evidence such as accident scene videos or videos of the interaction between vehicles and the environment. In the medical field, the original video can be medical imaging videos such as electrocardiogram videos, continuous echocardiograms, and multispectral endoscopic videos.
[0038] The pixel preprocessing described above is used to rearrange the pixels of an image frame. The initial image feature sequence includes the initial image features of the image frame. An image frame includes the current image frame and the previous image frame preceding the current image frame. The original video includes at least two image frames. For example, the original video includes four image frames, numbered t0, t1, t2, and t3 in sequence. If the frame number of the current image frame is t0, there is no previous image frame; if the frame number of the current image frame is t1, the frame number of the previous image frame is t0; if the frame number of the current image frame is t2, the frame number of the previous image frame is t1; if the frame number of the current image frame is t3, the frame number of the previous image frame is t2.
[0039] In one embodiment, reference is made to Figure 2 Step 101 may include: Step 201: Split the feature tensor of the original video in the time dimension to obtain the first initial image features of the image frame; wherein, the first initial image features have spatial dimension and channel dimension; Step 202: Expand adjacent s×s pixel blocks in the first initial image feature in the spatial dimension and arrange them in the channel dimension to obtain pixel rearrangement features. Step 203: Reduce the spatial dimension of the pixel rearrangement features by a factor of 1 / s to obtain the second initial image features of the image frame; Step 204: Merge the second initial image features of each image frame to obtain the initial image feature sequence.
[0040] For example, when the original video (e.g., of shape [N,T,C,H,W], where N is the batch size, T is the number of time frames (also called the time dimension), C is the number of channels, H is the height, and W is the width) is input, the core objective is to reduce the computational load through spatial downsampling without losing information. This is mainly achieved through the following steps, which are similar to the inverse operation of "depthiable separable convolution" in the spatial dimension.
[0041] Frame sequence separation: First, the model unbinds the feature tensors of the input raw video along the time dimension T, processing each frame independently. This step ensures that preprocessing can be parallelized and applied to all frames.
[0042] Pixel rearrangement: This is the most crucial step. For each image frame (of shape [C,H,W]), the operation unfolds its adjacent s×s pixel blocks (e.g., 2x2 blocks) in the spatial dimension and rearranges them in the channel dimension. At this point, the number of channels C becomes s² times the original. Because each s×s pixel block has s² pixels, they are sequentially placed into the new channels. Continuing with the example of s=2, the number of channels changes from C to 4C.
[0043] Spatial downsampling: The height H and width W of the image frame are reduced to 1 / s of their original values. For example, when s=2, the spatial dimensions of [C,H,W] become [C,H / 2,W / 2].
[0044] Therefore, the final shape of the image frame changes from [C,H,W] to [C*s²,H / s,W / s]. The entire process only changes the arrangement of the data; the total amount of information (total number of pixels × number of channels) remains unchanged.
[0045] Frame sequence reconstruction: After all image frames have been processed independently, they are recombined into a feature tensor, which is the initial image feature vector. At this time, the shape of the feature tensor of the original video changes from [N,T,C,H,W] to [N,T,C*s²,H / s,W / s].
[0046] The advantage of the embodiments of steps 201 to 204 above is that adjusting the pixel arrangement of the image frame while keeping the total amount of information unchanged is beneficial for subsequent downsampling, feature encoding, feature decoding and upsampling, thereby improving the accuracy and efficiency of video auto-encoding.
[0047] In step 102 of some embodiments, the initial image features are downsampled to obtain downsampled image features. Downsampling refers to the process of reducing the number of sampling points or the resolution in data, that is, obtaining a low-dimensional / low-sampling-density representation from a high-dimensional / high-sampling-density dataset.
[0048] In one embodiment, reference is made to Figure 3 Step 102 may include: Step 301: Channel expansion is performed on the channels of the initial image features to obtain temporal downsampling features; Step 302: Downsample the width and height of the initial image features to obtain spatial downsampled features; Step 303: Merge the temporal downsampling features and spatial downsampling features to obtain downsampled image features.
[0049] The advantage of this embodiment is that by limiting downsampling to spatiotemporal separation downsampling, that is, downsampling in the time dimension (channel expansion) and downsampling in the spatial dimension, the number of parameters in subsequent feature encoding and feature decoding can be reduced. While maintaining performance, the size of the feature encoder and feature decoder is reduced, significantly improving the efficiency and quality of self-encoding.
[0050] In one embodiment, reference is made to Figure 4 Step 301 may include: Step 401: Determine the number of frames based on the downsampling factor of the time downsampling, and extract image feature combinations from the initial image feature sequence based on the number of frames; any two image feature combinations do not overlap; Step 402: Channel splicing is performed on each initial image feature in the image feature combination to obtain the initial time downsampling features; Step 403: The initial temporal downsampling features are mapped to the number of channels using a 1x1 convolutional layer to obtain temporal downsampling sub-features; Step 404: Merge all time-downsampled sub-features to obtain time-downsampled features.
[0051] The advantage of this embodiment is that by converting time-dimension downsampling into channel expansion, specifically by expanding the channels of the initial image features obtained after the aforementioned pixel preprocessing, information loss can be reduced and self-encoding efficiency can be improved.
[0052] For example, in the case of a temporal downsampling factor of 2, the downsampling factor is designed to be equal to the number of frames, in which case the number of frames is 2. Next, based on the number of frames being 2, the initial image features of 2 frames are extracted by combining image features, i.e., the features of two adjacent frames are combined as one image feature set. Then, the initial image features of the combined image features are concatenated along the channel dimension and fused using a 1×1 convolution. The formula is expressed as: TPool(X) = Conv_{1×1}(Concat(Frame_t,Frame_{t+1})). For a TPool operation with a downsampling factor of 2, the process can be represented as: Output=Conv1x1(Concat(Frame_t,Frame_{t+1})).
[0053] Specifically, the shape of the initial image features changes from [N,T,C,H,W] to [N,T / 2, 2*C,H,W], and then the number of channels is adjusted back to C (or adjusted to the target number of channels according to the model design) through 1x1 convolution, and the final output shape is [N,T / 2,C_out,H,W].
[0054] In one example, suppose the feature tensor shape of a video clip is [1,8,64,16,16] (1 sample, 8 frames, 64 channels, 16x16 resolution). After passing through a TPool layer with a downsampling factor of 2, it concatenates every 2 frames along the channel dimension (at this point, the shape is [1,4,128,16,16]), and then maps the number of channels to the target value (e.g., 128) through a 1x1 convolution. Finally, the time dimension is halved, and the output is [1,4,128,16, 16].
[0055] In one embodiment, before using the encoded image features corresponding to the previous image frame as the first memory feature in step 103, the video self-encoding method may further include: When the current image frame is the first frame of the original video, generate the encoded image features of the previous image frame whose elements are all 0 and whose dimensions are the same as the downsampled image features of the current image frame. If the current image frame is not the first frame of the original video, then the encoded image features are obtained by performing feature encoding based on the downsampled image features and the first memory features of the previous image frame.
[0056] The advantage of the above embodiments is that information can be propagated along the time dimension, thereby helping to generate more temporally coherent video latent representations and reconstructed frames.
[0057] For example, given an original video containing 4 frames, after initial convolution (spatiotemporal desampling), the feature map size for each image frame is [N=1, C=64, H=16, W=16]. The process of generating coded image features based on the previous image frame can include: (1) For the first frame (x_t0): Since there is no previous image frame, the encoded image features of the previous image frame can be initialized as a zero tensor with the same shape as x_t0 (i.e., all elements are 0).
[0058] (2) For the second frame (x_t1): If the previous image frame is x_t0, then obtain the encoded image feature output_t0 by encoding the downsampled image features based on x_t0 and the zero tensor with the same shape as x_t0.
[0059] (3) For the third frame (x_t2): If the previous image frame is x_t1, then obtain the downsampled image features based on x_t1 and the encoded image features output_t1 obtained by feature encoding with the first memory feature (the encoded image feature output_t0 of x_t0).
[0060] (3) For the 4th frame (x_t3): If the previous image frame is x_t2, then obtain the downsampled image features based on x_t2 and the encoded image features output_t2 obtained by feature encoding with the first memory feature (the encoded image features output_t1 of x_t1).
[0061] In step 103 of some embodiments, the encoded image features corresponding to the previous image frame are used as the first memory features, and the downsampled image features of the current image frame and the first memory features are encoded to obtain the encoded image features of the current image frame.
[0062] Specifically, the process of generating the encoded image features of the current image frame may include: (1) Process the first frame: take the zero tensor with the same shape as x_t0 as the first memory feature; perform feature encoding on the downsampled image feature of x_t0 and the first memory feature to obtain the encoded image feature output_t0 of x_t0.
[0063] (2) Process the second frame: use the encoded image feature output_t0 of x_t0 as the first memory feature; perform feature encoding on the downsampled image feature of x_t1 and the first memory feature to obtain the encoded image feature output_t1 of x_t1.
[0064] (3) Process the third frame: use the encoded image feature output_t1 of x_t1 as the first memory feature; perform feature encoding on the downsampled image feature of x_t2 and the first memory feature to obtain the encoded image feature output_t2 of x_t2.
[0065] (3) Process the 4th frame: use the encoded image feature output_t2 of x_t2 as the first memory feature; perform feature encoding on the downsampled image feature of x_t3 and the first memory feature to obtain the encoded image feature output_t3 of x_t3.
[0066] In one embodiment, reference is made to Figure 5 Step 103, which involves encoding the downsampled image features and the first memory features of the current image frame to obtain the encoded image features of the current image frame, may include: Step 501: Obtain the first causal memory network, which includes a first splicing layer, a first convolutional layer, a first skip connection layer, and a first activation function layer; Step 502: The downsampled image features and the first memory features are spliced together through the first splicing layer to obtain the first spliced features; Step 503: Perform feature convolution on the first concatenated features through the first convolutional layer to obtain the first convolutional features; Step 504: Connect the first concatenated feature and the first convolutional feature through the first skip connection layer to obtain the first connected feature; Step 505: Perform feature activation on the first connection feature through the first activation function layer to obtain the encoded image features of the current image frame.
[0067] It is understandable that the aforementioned feature convolution operation can simultaneously perceive the content of the second frame itself (x_t1) and the information of the first frame (output_t0), thus potentially capturing minute changes between the two frames (such as the movement of an object).
[0068] This embodiment has the following characteristics: (1) Causality: It only depends on the current frame and the previous frame, without the need for future frames, which meets the requirements of streaming processing. (2) Feature reuse mechanism: Input features and memory features are fused through skip connections to alleviate gradient vanishing. (3) Memory update rule: The replacement of memory features realizes lightweight updates of historical states and avoids redundant calculations. In addition, this embodiment can guarantee the inter-frame continuity of the video.
[0069] In step 104 of some embodiments, the decoded image features corresponding to the previous image frame are used as the second memory features, and the encoded image features and the second memory features of the current image frame are decoded to obtain the decoded image features of the current image frame.
[0070] In one embodiment, before using the decoded image features corresponding to the previous image frame as the second memory features in step 104, the video self-encoding method may further include: When the current image frame is the first frame of the original video, generate the decoded image features of the previous image frame, where all elements are 0 and the dimensions are the same as those of the decoded image features of the current image frame. If the current image frame is not the first frame of the original video, then the decoded image features are obtained by performing feature decoding based on the decoded image features of the previous image frame and the second memory features.
[0071] The advantage of the above embodiments is that information can be propagated along the time dimension, thereby helping to generate more temporally coherent video latent representations and reconstructed frames.
[0072] In one embodiment, step 104, which involves feature decoding of the encoded image features and the second memory features of the current image frame to obtain the decoded image features of the current image frame, may include: Obtain the second causal memory network, which includes a second splicing layer, a second convolutional layer, a second skip connection layer, and a second activation function layer; The encoded image features and the second memory features are concatenated through the second concatenation layer to obtain the second concatenation feature; The second concatenated features are obtained by performing feature convolution on the second convolutional layer; The second spliced feature and the second convolutional feature are connected by a second skip connection layer to obtain the second connected feature. The second connection features are activated by the second activation function layer to obtain the decoded image features of the current image frame.
[0073] This embodiment has the following characteristics: (1) Causality: It only depends on the current frame and the previous frame, without the need for future frames, which meets the requirements of streaming processing. (2) Feature reuse mechanism: Input features and memory features are fused through skip connections to alleviate gradient vanishing. (3) Memory update rule: The replacement of memory features realizes lightweight updates of historical states and avoids redundant calculations. In addition, this embodiment can guarantee the inter-frame continuity of the video.
[0074] In step 105 of some embodiments, the decoded image features are upsampled to obtain upsampled image features. Upsampling refers to the process of increasing the number of sampling points or the resolution in the data, that is, obtaining a high-density representation from a low-density data representation.
[0075] In one embodiment, reference is made to Figure 6 Step 105 may include: Step 601: Determine the upsampling factor based on the downsampling factor; wherein the upsampling factor is equal to the downsampling factor; Step 602: The number of channels of the decoded image features is expanded by an upsampling factor through a 1x1 convolutional layer to obtain spatial upsampled features; Step 603: Arrange the spatial upsampled features to obtain upsampled image features; wherein, the temporal and spatial dimensions of the upsampled image features are the same as those of the initial image features.
[0076] This embodiment is the reverse of step 102. It expands the number of channels by an upsampling factor equal to the downsampling factor using a 1×1 convolutional layer, and then splits the data into multiple frames. This design avoids redundant computation in 3D convolution and allows independent control of the temporal scaling. For an upsampling operation with a factor of 2, the process can be represented as follows: Output=Split(Conv1x1(Input)), The input shape is expanded from [N,T,C,H,W] through a 1x1 convolution to become [N,T,2*C,H,W], and then rearranged to [N,2*T,C,H,W].
[0077] For example, the shape of the decoded image features is [1,4,128,16,16]. After an upsampling layer with a multiplier of 2, a 1x1 convolution first expands the number of channels to 2*128=256 (the intermediate shape is [1,4,256,16,16]), and then these 256 channels are reinterpreted into two time steps, each with 128 channels. The final output shape becomes [1,8,128,16,16], and the temporal dimension is restored to 8 frames. Spatial dimension processing: standard convolution and deconvolution are used, but decoupled from temporal operations, and multi-scale feature extraction is achieved through a concatenated structure.
[0078] In step 106 of some embodiments, pixel post-processing is performed on the upsampled image features of all image frames to obtain the target video. Pixel post-processing is the final step in generating the final video. Specifically, the output tensor (typically of shape [N,T,C*s²,H / s,W / s]) from the decoder (used for feature decoding) is received, and operations almost exactly the opposite of preprocessing are performed to reconstruct the video frames of the original spatial dimensions.
[0079] Frame Sequence Separation: Similarly, the output tensor is first separated along the time dimension T, and each frame is processed independently. Pixel Rearrangement (Inverse Operation): This is the core restoration step. For each frame (shape [C*s², H / s, W / s]), the operation rearranges its data in the channel dimension back to the spatial dimension. Channel Shrinking: The number of channels is restored from C*s² to the original C. Spatial Upsampling: The height and width are restored from H / s and W / s to the original H and W, respectively. Therefore, the shape of a frame is successfully restored from [C*s², H / s, W / s] to [C, H, W]. This process is the perfect inverse transformation of pixel rearrangement. Frame Sequence Reassembly: All processed frames are recombined, and the final output video tensor has the shape [N, T, C, H, W], consistent with the original input size.
[0080] In one embodiment, the decoder supports independent control of spatiotemporal upsampling: (1) Spatial upsampling switch: whether bilinear upsampling is performed is controlled by the decoder_space_upscale parameter. (2) Temporal upsampling switch: the TGrow layer is enabled is controlled by the decoder_time_upscale parameter. This mechanism allows a single model to support multi-resolution and multi-frame-rate generation without retraining.
[0081] To balance memory and speed, this application supports two inference modes: (1) Parallel mode: all frames are processed simultaneously, utilizing GPU parallelism, suitable for short videos or scenarios with sufficient resources. (2) Sequence mode: frames are processed on demand based on a work queue, with a stable memory usage of O(1), supporting long video streaming processing.
[0082] In one example, the inference process in sequence mode includes the following steps: (1) Input the frame sequence of the original video; (2) Initialize the work queue according to the frame sequence; (3) Determine whether the work queue is empty; if it is not empty, proceed to step (4); if it is empty, proceed to step (11); (4) Take out the work item from the work queue; (5) Process the work item using one of the multiple predefined modules to obtain downsampled image features, or encoded image features, or decoded image features, or upsampled features; (6) Determine the type of the predefined module; if the type is a causal memory module, proceed to step (7); if the type is a downsampled module, proceed to step (8); if the type is an upsampled module, proceed to step (9); (7) Update the first memory feature or the second memory feature, and then proceed to step (10); (8) Cache the frames and aggregate them after determining that the downsampling multiple has been reached, and then proceed to step (10); (9) Expand the time dimension and split the frames, and then proceed to step (10); (10) Add a new item to the work queue, and then proceed to step (3); (11) Collect the output frames; (12) The video reconstruction is complete.
[0083] Based on the above embodiments, this application can achieve at least the following beneficial effects: (1) Significantly improve coding efficiency and quality: The spatiotemporal separation architecture reduces the number of parameters in 3D convolution, and the model size is reduced by about 40% while maintaining performance (based on comparative data of similar models). MemBlock's causal memory mechanism improves the inter-frame PSNR by about 2.1 dB and significantly reduces jitter artifacts. (2) Memory optimization and enhanced flexibility: In sequence mode, the memory usage of long videos is stable at O(1), supporting real-time processing of 4K videos (experiments show that processing 1000 frames of video only requires 1.2 GB of video memory, while traditional 3D CNN requires more than 8 GB). The controllable upsampling mechanism allows for dynamic adjustment of output specifications (such as directly generating 1080p@15fps from 480p@30fps), adapting to diverse terminal and network environments. (3) Expanding application scenarios: It can be deployed on edge devices (such as drones and surveillance cameras) to achieve low-power real-time encoding. As a preprocessor for video generation models (such as Diffusion Model), it effectively compresses the potential space and improves the generation speed.
[0084] (4) Standardization compatibility: The model output can be integrated into the transport stream of existing coding standards (such as H.265) without changing the decoder hardware.
[0085] Please see Figure 7 This application also provides a video self-encoding apparatus that can implement the above-described video self-encoding method. Figure 7 This is a block diagram of the module structure of a video self-encoding device provided in an embodiment of this application. The device includes: The video preprocessing module 701 is used to acquire the original video, perform pixel preprocessing on each image frame of the original video, and obtain an initial image feature sequence; wherein, the initial image feature sequence includes the initial image features of the image frame, and the image frame includes the current image frame and the previous image frame before the current image frame. The downsampling module 702 is used to downsample the initial image features to obtain downsampled image features; The feature encoding module 703 is used to take the encoded image features corresponding to the previous image frame as the first memory feature, and to perform feature encoding on the downsampled image features of the current image frame and the first memory feature to obtain the encoded image features of the current image frame. The feature decoding module 704 is used to take the decoded image features corresponding to the previous image frame as the second memory features, and perform feature decoding on the encoded image features and the second memory features of the current image frame to obtain the decoded image features of the current image frame. The upsampling module 705 is used to upsample the features of the decoded image to obtain upsampled image features; The video reconstruction module 706 is used to perform pixel post-processing on the upsampled image features of all image frames to obtain the target video.
[0086] It should be noted that the specific implementation of this video self-encoding device is basically the same as the specific implementation of the video self-encoding method described above, and will not be repeated here.
[0087] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned video self-encoding method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0088] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the video self-encoding method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.
[0089] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described video self-encoding method.
[0090] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0091] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0092] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0094] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0095] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0096] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0101] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A video auto-encoding method, characterized in that, The method includes: The original video is acquired, and pixel preprocessing is performed on each image frame of the original video to obtain an initial image feature sequence; wherein, the initial image feature sequence includes the initial image features of the image frame, and the image frame includes the current image frame and the previous image frame before the current image frame; The initial image features are downsampled to obtain downsampled image features; The encoded image features corresponding to the previous image frame are used as the first memory features, and the downsampled image features of the current image frame and the first memory features are used for feature encoding to obtain the encoded image features of the current image frame. The decoded image features corresponding to the previous image frame are used as the second memory features, and the encoded image features of the current image frame and the second memory features are used for feature decoding to obtain the decoded image features of the current image frame. The decoded image features are upsampled to obtain upsampled image features; The upsampled image features of all the image frames are post-processed at the pixel level to obtain the target video.
2. The method according to claim 1, characterized in that, The step of encoding the downsampled image features of the current image frame and the first memory features to obtain the encoded image features of the current image frame includes: Obtain a first causal memory network, which includes a first splicing layer, a first convolutional layer, a first skip connection layer, and a first activation function layer; The downsampled image features and the first memory features are spliced together by the first splicing layer to obtain the first spliced feature; The first convolutional feature is obtained by performing feature convolution on the first concatenated feature through the first convolutional layer; The first spliced feature and the first convolutional feature are connected by the first skip connection layer to obtain the first connected feature. The first connection feature is activated by the first activation function layer to obtain the encoded image features of the current image frame.
3. The method according to claim 1, characterized in that, Before using the encoded image features corresponding to the previous image frame as the first memory feature, the method further includes: When the current image frame is the first frame of the original video, generate the encoded image features of the previous image frame whose elements are all 0 and whose dimensions are the same as the downsampled image features of the current image frame; If the current image frame is not the first frame of the original video, then the encoded image features are obtained by performing feature encoding based on the downsampled image features and the first memory features of the previous image frame.
4. The method according to any one of claims 1 to 3, characterized in that, The step of downsampling the initial image features to obtain downsampled image features includes: Channel expansion is performed on the channels of the initial image features to obtain temporally downsampled features; The width and height of the initial image features are downsampled to obtain spatial downsampled features; The downsampled image features are obtained by merging the temporal downsampling features and the spatial downsampling features.
5. The method according to claim 4, characterized in that, The process of channel expansion of the initial image features to obtain temporally downsampled features includes: The number of frames is determined based on the downsampling factor of the time downsampling, and image feature combinations are extracted from the initial image feature sequence based on the number of frames; any two image feature combinations do not overlap; The initial image features in the image feature combination are concatenated by channels to obtain the initial time downsampling features; The initial temporal downsampling features are mapped to the number of channels using a 1x1 convolutional layer to obtain temporal downsampling sub-features; The time-downsampled features are obtained by merging all the time-downsampled sub-features.
6. The method according to claim 5, characterized in that, The upsampling of the decoded image features to obtain upsampled image features includes: The upsampling factor is determined based on the downsampling factor; wherein the upsampling factor is equal to the downsampling factor; Spatial upsampled features are obtained by expanding the number of channels of the decoded image features by the upsampling factor through a 1x1 convolutional layer. The spatial upsampled features are arranged to obtain the upsampled image features; wherein the temporal and spatial dimensions of the upsampled image features are the same as those of the initial image features.
7. The method according to any one of claims 1 to 3, characterized in that, The step of performing pixel preprocessing on the original video to obtain an initial image feature sequence includes: The feature tensor of the original video is split along the time dimension to obtain the first initial image features of the image frame; wherein, the first initial image features have spatial dimension and channel dimension; The adjacent s×s pixel blocks in the first initial image feature are expanded in the spatial dimension and arranged in the channel dimension to obtain the pixel rearrangement feature. The pixel rearrangement features are reduced by a factor of 1 / s in spatial dimension to obtain the second initial image features of the image frame; The initial image feature sequence is obtained by merging the second initial image features of each of the image frames.
8. A video auto-encoding device, characterized in that, The device includes: The video preprocessing module is used to acquire the original video and perform pixel preprocessing on each image frame of the original video to obtain an initial image feature sequence; wherein, the initial image feature sequence includes the initial image features of the image frame, and the image frame includes the current image frame and the previous image frame before the current image frame; The downsampling module is used to downsample the initial image features to obtain downsampled image features; The feature encoding module is used to take the encoded image features corresponding to the previous image frame as the first memory feature, and to perform feature encoding on the downsampled image features of the current image frame and the first memory feature to obtain the encoded image features of the current image frame. The feature decoding module is used to take the decoded image features corresponding to the previous image frame as the second memory features, and perform feature decoding on the encoded image features of the current image frame and the second memory features to obtain the decoded image features of the current image frame. The upsampling module is used to upsample the decoded image features to obtain upsampled image features; The video reconstruction module is used to perform pixel post-processing on the upsampled image features of all the image frames to obtain the target video.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video self-encoding method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video self-encoding method according to any one of claims 1 to 7.