A video coding method and system based on motion-guided temporal memory, a terminal device and a medium

Abstract

This invention discloses a video encoding / decoding method, system, terminal device, and medium based on motion-guided temporal memory, relating to the field of video encoding / decoding technology. The method includes: acquiring the current frame and historical reference frames of a video and obtaining motion information; extracting reference features from the historical reference frames; aligning the reference features based on the motion information to obtain aligned features; modeling the long-range temporal memory of the aligned features to obtain a long-term temporal context; acquiring a short-term temporal context and aggregating it with the long-term temporal context to obtain a long-short temporal context; and using this context as a priori to compress, encode, and reconstruct the current frame to obtain a reconstructed frame. This invention effectively models cross-frame long-range temporal dependencies, significantly improving the structural fidelity and semantic coherence of inter-frame prediction, optimizing the rate-distortion performance of video encoding / decoding, and achieving high-efficiency, high-fidelity video compression.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and in particular to a video encoding and decoding method, system, terminal device and medium based on motion-guided temporal memory. Background Technology

[0002] End-to-end video encoding and decoding technology based on deep learning is a research hotspot in the field of video compression. It is mainly divided into two types of compression frameworks: residual coding and conditional context coding. Among them, the conditional context coding framework has better rate-distortion performance because it can capture rich inter-frame spatiotemporal correlations.

[0003] Current mainstream end-to-end video encoding and decoding solutions mostly rely on the previous frame or two as short-term references for temporal modeling. Some methods that attempt to introduce long-term reference frames are either still limited to pixel-domain operations or directly use long-distance reference features that have not been aligned with feature-level motion. In scenarios with complex motion, occlusion, or non-rigid deformation, it is difficult to effectively mine long-range temporal correlations, which can easily lead to temporal information degradation and insufficient semantic consistency and structural integrity of inter-frame predictions. This seriously restricts the compression efficiency and reconstruction quality of video encoding and decoding.

[0004] Therefore, there is an urgent need for a video encoding and decoding method that achieves unified modeling of long and short-term temporal contexts guided by motion in the feature domain, in order to fill the gap in existing technologies. Summary of the Invention

[0005] The technical problem this invention aims to solve is that, in the field of video encoding and decoding, existing end-to-end video compression frameworks have significant deficiencies in their temporal modeling capabilities. Reliance on short-term reference frames leads to a narrow temporal receptive field, while operations on long-term reference frames suffer from pixel-domain limitations or feature misalignment issues. This makes it difficult to effectively mine long-range temporal dependencies across multiple frames in complex scenarios, resulting in poor inter-frame prediction semantic consistency and limitations in compression efficiency and reconstruction quality. Therefore, an effective solution is urgently needed to address these technical problems.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a video encoding and decoding method based on motion-guided temporal memory, the method comprising: Obtain the current frame and historical reference frames of the video, and extract motion information from the current frame and historical reference frames; Feature extraction is performed on the historical reference frames to obtain reference features; The reference features are aligned based on the motion information for motion-guided feature propagation to obtain aligned features. Long-range temporal memory modeling is performed on the aligned features to obtain the long-term temporal context; Obtain the short-term time series context, and aggregate the long-term time series context with the short-term time series context to obtain the long-short time series context; Using the long and short temporal contexts as a priori, the current frame is compressed, encoded, and reconstructed to obtain the reconstructed frame corresponding to the current frame.

[0007] In one implementation, obtaining the current frame and historical reference frames of the video, and obtaining motion information from the current frame and historical reference frames, includes: Obtain the current frame of the video and use the reconstructed frame corresponding to the previous moment as the historical reference frame; Based on the current frame and the historical reference frame, the spatiotemporal relationship between frames is modeled through a pre-trained motion estimation network to obtain the initial motion vector; The initial motion vector is encoded, compressed, and reconstructed to obtain the motion information.

[0008] In one implementation, the step of extracting features from the historical reference frame to obtain reference features includes: By using multi-scale convolution kernels, spatial features are extracted hierarchically from the historical reference frames to obtain features at different scales; The features at different scales are spliced ​​and fused together to obtain multi-scale depth features, which are then used as the reference features.

[0009] In one implementation, aligning the reference features based on the motion information for motion-guided feature propagation to obtain aligned features includes: Based on the motion information, determine the positional mapping relationship between the reference feature and the current frame feature coordinate space; Based on the position mapping relationship, a differentiable spatial deformation operation is performed on the reference feature to calibrate the reference feature to the feature coordinate space of the current frame in order to compensate for inter-frame motion displacement and obtain the aligned feature.

[0010] In one implementation, the step of modeling the aligned features using long-range temporal memory to obtain a long-term temporal context includes: By using gated memory units, temporal associations are established for the aligned features at different times, long-range temporal memory modeling is completed, and the long-term temporal context is obtained. The gated memory unit includes an input gate, a forget gate, and an output gate, which are used to retain high-confidence long-term time-series clues and suppress redundant time-series information.

[0011] In one implementation, the step of obtaining the short-term time series context and aggregating the long-term time series context with the short-term time series context to obtain a long-short time series context includes: Based on a preset number of reconstructed frames and their corresponding depth features from the preceding time of the current frame, the short-term temporal context is obtained through short-term temporal context feature extraction. Based on the motion information, a second spatial alignment is performed between the long-term temporal context and the short-term temporal context. The aligned long-term and short-term temporal contexts are then subjected to multi-scale feature aggregation to obtain the long and short-term temporal contexts.

[0012] In one implementation, the step of using the long and short temporal context as a priori to perform compression encoding and reconstruction decoding on the current frame to obtain the reconstructed frame corresponding to the current frame includes: The long and short temporal contexts are fused with the depth features of the current frame; The fused features are then subjected to entropy compression encoding to obtain the encoded bitstream; The encoded bitstream is reconstructed and decoded using the long and short temporal contexts as priors to obtain decoding features; The decoded feature map is restored to a pixel-domain frame to obtain the reconstructed frame.

[0013] Secondly, embodiments of the present invention also provide a video encoding / decoding system based on motion-guided temporal memory, the system comprising: The motion information acquisition module is used to acquire the current frame and historical reference frames of the video, and to acquire motion information from the current frame and historical reference frames; A reference feature extraction module is used to extract features from the historical reference frame to obtain reference features; The feature alignment module is used to align the reference features based on the motion information, and to perform motion-guided feature propagation to obtain the aligned features. The long-term temporal context acquisition module is used to perform long-range temporal memory modeling on the aligned features to obtain the long-term temporal context. The long and short time series context acquisition module is used to acquire the short-term time series context and aggregate the long-term time series context with the short-term time series context to obtain the long and short time series context. The encoding / decoding module is used to compress and encode the current frame and reconstruct and decode it based on the long and short temporal contexts to obtain the reconstructed frame corresponding to the current frame.

[0014] Thirdly, embodiments of the present invention also provide a terminal device, the terminal device including a memory, a processor, and a video encoding / decoding program based on motion-guided timing memory stored in the memory and executable on the processor, wherein when the processor executes the video encoding / decoding program based on motion-guided timing memory, it implements the steps of the video encoding / decoding method based on motion-guided timing memory as described in any of the above schemes.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a video encoding / decoding program based on motion-guided timing memory. When the video encoding / decoding program based on motion-guided timing memory is executed by a processor, it implements the steps of the video encoding / decoding method based on motion-guided timing memory as described in any of the above schemes.

[0016] Beneficial Effects: This invention discloses a video encoding / decoding method, system, terminal device, and medium based on motion-guided temporal memory, relating to the field of video encoding / decoding technology. The method first acquires the current frame and historical reference frames of the video, obtains motion information from the current frame and historical reference frames, and extracts features from the historical reference frames to obtain reference features. Subsequently, the reference features are aligned based on the motion information for motion-guided feature propagation to obtain aligned features, and long-range temporal memory modeling is performed on the aligned features to obtain a long-term temporal context. Then, a short-term temporal context is acquired, and the long-term temporal context is aggregated with the short-term temporal context to obtain a long-short temporal context. Finally, using the long-short temporal context as a priori, the current frame is compressed, encoded, and reconstructed to obtain the reconstructed frame corresponding to the current frame. This invention guides reference feature alignment and constructs long-short-term temporal contexts by using motion information, effectively modeling long-range temporal dependencies across frames in videos. It overcomes the limitations of existing technologies that rely on short-term references for temporal modeling, significantly improves the structural fidelity and semantic coherence of inter-frame predictions, optimizes the rate-distortion performance of video encoding and decoding, and maintains excellent compression performance even in complex motion and occlusion scenarios. It greatly improves video compression efficiency and reconstruction quality, achieving high-efficiency and high-fidelity video compression. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a specific implementation of the video encoding / decoding method based on motion-guided temporal memory provided in this invention.

[0018] Figure 2 This is a schematic diagram of the FG-ConvLSTM-based video encoding and decoding architecture of the motion-guided temporal memory-based video encoding and decoding method provided in an embodiment of the present invention.

[0019] Figure 3A schematic diagram of the FG-ConvLSTM framework for a motion-guided temporal memory-based video encoding and decoding method provided in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of a video encoding and decoding device based on motion-guided temporal memory provided in an embodiment of the present invention.

[0021] Figure 5 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0024] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0026] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0027] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] In the fields of computer vision and multimedia transmission technology, video encoding and decoding technology is the core link for achieving efficient transmission and storage of video data. Among them, end-to-end video compression technology based on deep learning has become an important research direction in this field. End-to-end optimized video compression frameworks, with their adaptive feature learning capabilities, have the potential to surpass the performance of traditional hand-designed codecs. Existing end-to-end video compression frameworks are mainly divided into two categories. One is the compression framework based on residual coding. This type of framework uses the residual information between the predicted frame and the actual frame as the core coding unit. It uses neural networks to efficiently model and compress the residual information, and can adaptively learn the temporal and spatial correlations of video sequences. It combines contextual information to generate high-quality reconstructed frames and reduces data redundancy by jointly optimizing the encoding and decoding processes. The other is the compression framework based on conditional context. This type of framework takes adjacent frames, motion information, and other conditional contexts as input, adaptively adjusts the coding strategy, accurately captures the complex temporal and spatial correlations between frames, and integrates the entire process of prediction, residual coding, and reconstruction through end-to-end optimization. It achieves better video reconstruction results at low bit rates, and therefore its rate-distortion performance is gradually surpassing that of compression frameworks based on residual coding.

[0029] Most current mainstream end-to-end video encoding and decoding methods rely on conditional context-based coding frameworks. However, the temporal context modeling stage of such methods still has obvious technical limitations. Its temporal modeling usually only uses the previous frame or the two frames before the current frame as reference frames, and generates short-term predictive features through optical flow estimation, deformable convolution, or local attention. The temporal receptive field of such designs is relatively narrow, and it lacks the ability to explicitly model the semantic evolution of video sequences across multiple frames. It is difficult to effectively handle video scenes with complex temporal structures, such as slow motion, periodic content recurrence, or long-term occlusion.

[0030] Some existing technologies attempt to introduce long-term reference frames to expand the temporal receptive field and try to mine long-range temporal correlations in video sequences to improve the accuracy of inter-frame prediction. However, such improvement schemes still have significant technical defects: First, some schemes are still limited to pixel domain operations, directly synthesizing predicted images based on optical flow or motion vectors. It is difficult to coordinate with learnable feature representations for optimization. Irreversible distortion is easily introduced in occlusion, fast motion, or texture-deficient areas, and the high-level semantic consistency of the video cannot be captured. Second, some schemes directly use long-distance reference features that have not undergone feature-level motion alignment. In complex motion, occlusion, or non-rigid deformation scenarios, historical reference features are prone to serious mismatch with the semantic space of the current frame, which cannot effectively mine long-range temporal correlations and may even introduce interference noise, causing the temporal information to degrade rapidly during transmission.

[0031] Specifically, some patents propose video encoding and decoding methods based on feature domain temporal modeling. These methods introduce long-term reference frames by stacking multiple layers of convolutional long short-term memory units to enhance contextual representation. However, they lack motion-guided spatiotemporal alignment of long-distance reference features, making them prone to feature semantic space mismatch in non-rigid motion and rapid scene changes. Furthermore, this approach uses the traditional residual coding paradigm, limiting its ability to express high-dimensional, multi-scale long-range contexts and failing to achieve adaptive weighting and fusion of temporal priors, resulting in low utilization efficiency of long-term reference information. Other patents design new motion information encoding and decoding and motion alignment methods, which effectively improve the representation of large motion information, but the parameters of these methods... The narrow temporal coverage of reference frames makes it difficult to effectively utilize long-term temporal reference information, thus limiting the upper limit of reference information utilization. In complex video scenes with high repetition, the room for improvement in compression performance is limited. Other related patents introduce a temporal modeling module into the neural video coding framework to utilize historical frame information, but it recursively passes the original reference features without motion compensation frame by frame, lacking explicit modeling of cross-frame motion. Furthermore, the standard convolution operation used has a fixed receptive field and static weights, which cannot adaptively respond to feature-level shifts caused by object motion and viewpoint changes. It is difficult to capture non-rigid and non-local temporal dependencies, which easily leads to temporal context distortion, restricting the accuracy of inter-frame prediction and compression efficiency.

[0032] In summary, the core technical problem of existing end-to-end video encoding and decoding technologies lies in their inability to effectively model long-range temporal dependencies across multiple frames in the feature domain. Existing solutions either rely solely on short-term reference frames, resulting in a narrow temporal receptive field, or suffer from limitations in pixel-domain operations or misalignment of feature-level motion when processing long-term reference frames. Ultimately, this leads to insufficient semantic consistency and structural integrity in inter-frame predictions, limited rate-distortion performance in complex dynamic video scenarios, and an inability to balance compression efficiency and reconstruction quality in video encoding and decoding, failing to meet the demands for high-efficiency, high-fidelity video compression. Therefore, an effective solution is urgently needed to address these technical problems.

[0033] This embodiment provides a video encoding and decoding method based on motion-guided temporal memory, which is implemented based on an end-to-end video encoding and decoding architecture. This architecture uses FG-ConvLSTM (Flow-Guided ConvLSTM, a feature flow-guided convolutional long short-term memory network) as the core temporal modeling unit.

[0034] Specifically, such as Figure 2As shown, the encoding / decoding architecture comprises five core components: a motion estimation module, a motion encoder-decoder network, an FG-ConvLSTM module, a contextual encoder-decoder network, an entropy model, and a probability estimation model. These components work collaboratively to complete the entire process of inter-frame motion information parsing, historical frame spatial feature extraction, cross-frame feature space calibration, long-range temporal memory modeling, long-short-term context fusion, and video frame compression encoding and pixel domain reconstruction. The architecture achieves spatial alignment compensation of historical features through motion information, dynamically filters and accumulates long-range temporal information across multiple frames based on the FG-ConvLSTM unit, integrates short-term local temporal features with long-term global temporal features to form a unified temporal prior, and incorporates this temporal prior into the encoding / decoding process to achieve high-efficiency compression and high-fidelity reconstruction of video data.

[0035] This embodiment provides a video encoding / decoding method based on motion-guided temporal memory, such as... Figure 1 As shown, the specific steps include the following: Step S100: Obtain the current frame and historical reference frames of the video, and obtain motion information from the current frame and historical reference frames.

[0036] In this embodiment, the current frame is a video frame in the video sequence that is at the moment of encoding / decoding processing. It is the core processing object of video encoding / decoding, and specifically it can be a video frame in the video sequence. A video sequence consists of a single frame image at a given moment and frame data containing multi-channel visual information. A historical reference frame is a video frame preceding the current frame that provides temporal reference information for the current frame; specifically, it can be a reconstructed frame from the previous moment, or original or reconstructed frames from multiple moments preceding the current frame. Motion information is a set of information characterizing the inter-frame motion relationship between the current frame and historical reference frames; specifically, it can be inter-frame motion vectors, optical flow information, and parameters characterizing inter-frame pixel or feature displacements.

[0037] Obtaining the current frame and historical reference frames of the video, and extracting motion information from the current frame and historical reference frames, is... Figure 2The FG-ConvLSTM encoding / decoding architecture shown here comprises the motion estimation module, the core input of the motion encoder-decoder network, and its basic processing steps. Specifically, it acquires the current frame and historical reference frames from the video stream, ensuring the integrity and temporal correlation of the frame data. Then, based on the acquired dual-frame data, it extracts inter-frame motion information, mining the motion correlations between frames from the video temporal sequence. This provides accurate prior motion information for subsequent motion-guided alignment in the feature domain. The motion information acquired in this way reflects the spatiotemporal positional changes between the current frame and the historical reference frames, effectively avoiding feature space mismatch caused by motion information deviations during subsequent feature alignment. Simultaneously, it limits the historical reference frames to temporal prior frames, ensuring the temporal logical consistency of subsequent temporal modeling.

[0038] In one implementation, the step of acquiring the current frame and historical reference frames of the video, and obtaining motion information from the current frame and historical reference frames, specifically includes the following steps: Step S110: Obtain the current frame of the video and use the reconstructed frame corresponding to the previous moment as the historical reference frame; Step S120: Based on the current frame and the historical reference frame, model the spatiotemporal relationship between frames through a pre-trained motion estimation network to obtain the initial motion vector; Step S130: Encode, compress, and reconstruct the initial motion vector to obtain the motion information.

[0039] In this embodiment, the current frame of the video is completed. The current frame is obtained and the reconstructed frame corresponding to the previous time step is matched. The reconstructed frame is designated as the historical reference frame. Using the reconstructed frame as the historical reference frame ensures that the feature information of the reference frame remains consistent with the feature domain of the encoding and decoding process, avoiding reference bias caused by domain differences between the original frame and the reconstructed frame. After obtaining the current frame and its corresponding historical reference frame, the spatiotemporal relationship between the frames is modeled using a pre-trained motion estimation network based on the dual-frame data to obtain the initial motion vector.

[0040] like Figure 2 As shown, this process corresponds to the motion estimation module in the FG-ConvLSTM codec architecture, which uses the current frame... Reconstructed frame from the previous moment As input, a pre-trained motion estimation network is used to mine the spatiotemporal positional correlations between frames. Specifically, the pre-trained motion estimation network can be the SPyNet model, which can accurately capture pixel-level displacements between frames, thereby outputting initial motion vectors that can characterize the motion correlations between frames. This vector is represented as reflecting the motion displacement relationship between the reconstructed frame at the previous moment and the current frame.

[0041] After obtaining the initial motion vector Then, it is input into the architecture's motion coding module, and after encoding processing, motion coding information is obtained. Subsequently, through Figure 2 The entropy model in the code performs probability estimation and feature compression to obtain the bitstream. Then, through the motion decoding module... The reconstructed motion vectors are obtained by performing reconstruction. The motion coding and motion decoding network employs a network architecture with a variational super-prior structure for the initial motion vector. Perform lossy compression. Figure 2 It shows Obtained through the motion coding module Then, it is obtained through the motion decoding module. The data processing flow, and It will be simultaneously input into the entropy model for probability estimation and compressed storage.

[0042] The method of encoding, compressing, and reconstructing the initial motion vectors can significantly reduce the storage and transmission costs of motion information while ensuring its validity, meeting the core requirement of data compression in video encoding and decoding. Simultaneously, using a pre-trained motion estimation network to model inter-frame spatiotemporal relationships improves the accuracy of the initial motion vectors, allowing the motion information to accurately reflect the actual motion state between frames. Using the reconstructed frame from the previous moment as a historical reference frame allows motion information extraction and subsequent feature domain processing to form a closed loop, avoiding incompatibility issues between pixel domain information and feature domain information in the original frame. The resulting motion information provides accurate motion priors for subsequent feature alignment, effectively ensuring the accuracy of subsequent motion-guided feature deformation operations. This allows feature alignment to accurately compensate for inter-frame motion displacement, providing a reliable motion foundation for subsequent long-range temporal memory modeling. The direct technical effect is improved feature alignment accuracy, reduced feature space mismatch caused by motion information deviations, and ultimately enhanced effectiveness of the entire long-range temporal modeling process.

[0043] Step S200: Extract features from the historical reference frame to obtain reference features.

[0044] In this embodiment, feature extraction of historical reference frames is a process of hierarchical and digital transformation and information filtering of the visual information of historical reference frames. Specifically, it can be feature mapping implemented through convolutional neural networks and multi-level information extraction completed by a multi-scale feature extraction module. Reference features are digital feature representations of intra-frame spatial semantic information obtained after feature extraction of historical reference frames. Specifically, they can be depth features, multi-scale spatial representations, and high-dimensional feature vectors containing intra-frame texture and structural information.

[0045] Feature extraction is performed on the historical reference frames to obtain reference features, providing input features for the FG-ConvLSTM module that meet the requirements of feature domain temporal modeling. In this embodiment, the feature extraction process of historical reference frames departs from traditional pixel domain operations, extracting effective information from historical reference frames in the feature domain. This eliminates the information redundancy problem of the pixel domain and mines feature information with high-level semantics in historical reference frames, allowing subsequent temporal modeling to be carried out based on the semantic level. This process, through targeted feature extraction, retains the core spatial semantic information in historical reference frames and filters out meaningless pixel noise, resulting in reference features with better expressive and semantic representation capabilities. Using these reference features as the basis for subsequent motion alignment and temporal modeling can improve the semantic consistency of cross-frame feature matching, avoid the irreversible distortion problem that pixel domain operations are prone to introduce in complex scenes, and make subsequent long-range temporal dependency mining more effective.

[0046] In one implementation, the step of extracting features from the historical reference frame to obtain reference features specifically includes the following steps: Step S210: Using multi-scale convolution kernels, perform spatial feature layering extraction on the historical reference frame to obtain features at different scales; Step S220: Perform channel splicing and fusion aggregation on the features of different scales to obtain multi-scale depth features as the reference features.

[0047] In this embodiment, multi-scale convolutional kernels are used to extract spatial features hierarchically from historical reference frames, obtaining features at different scales. Multi-scale convolutional kernels refer to a set of convolutional kernels with different sizes and receptive fields. Different sized kernels can capture spatial semantic information at different scales in the historical reference frames. Small kernels can capture fine texture details within the frame, while large kernels can capture global structural information. Through hierarchical extraction using multi-scale convolutional kernels, the spatial feature information of the historical reference frames is comprehensively mined, avoiding the one-sidedness of information extracted by single-scale convolutional kernels. After extracting features at different scales, channel concatenation and fusion aggregation are performed on the features at each scale to obtain multi-scale depth features. These multi-scale depth features are then used as reference features. Channel concatenation involves stacking the feature channels of features at different scales, while fusion aggregation integrates the information from the concatenated multi-channel features through feature fusion operations, allowing feature information at different scales to complement each other and form a more expressive feature representation.

[0048] like Figure 3As shown, this process corresponds to the Feature Inception (FIN, Multi-Scale Feature Initiation) module in the FG-ConvLSTM framework, which is the component for implementing multi-scale feature extraction. Furthermore, a ConvLSTM unit can be used to optimize and enhance the features.

[0049] The historical reference frame is the reconstructed frame from the previous moment. After performing basic deep feature extraction, we obtain This feature is from the reconstructed frame The basic visual features extracted from it are also Figure 3 The input features of the FIN module are the basic depth features retrieved from the cache. The input is fed into the FIN module, which employs three parallel feature processing branches to achieve multi-scale spatial feature hierarchical extraction. Each branch is a concatenated structure of a convolutional kernel, a ReLU activation function, and a convolutional kernel. The kernel sizes of the three branches are set to 1×1, 3×3, and 5×5, respectively, to capture... The system extracts spatial semantic information at different scales. Specifically, the 1×1 convolutional branch focuses on small-scale features such as fine intra-frame textures, the 3×3 convolutional branch captures medium-scale structural information, and the 5×5 convolutional branch extracts global large-scale semantic information, thus avoiding the one-sidedness of information extracted by single-scale convolutional kernels.

[0050] The multi-scale features after three-branch processing are respectively... , , It is indicated that the output features of the three branches are then superimposed and fused with information through channel splicing to obtain the basic reference feature. This feature is a multi-scale deep feature that only encodes the spatial semantic information of historical reference frames.

[0051] To further enhance the expressive power of reference features, the basic reference features are directly input into... Figure 3 A ConvLSTM unit connected after the FIN layer is used for short-term temporal dependency modeling. The gating mechanism of ConvLSTM is used to mine the short-term temporal correlation of the underlying reference features, realizing the fusion encoding of spatial multi-scale features and short-term temporal information, and finally generating multi-scale enhanced reference features. This feature simultaneously encodes both the spatial semantic information and short-term temporal information of the historical reference frames.

[0052] The multi-scale extraction and aggregation method of the FIN module enables the reference features to have excellent spatial semantic representation capabilities, effectively filter redundant noise in the pixel domain, and adapt to the basic requirements of cross-frame feature matching. Meanwhile, the short-term temporal modeling of the first ConvLSTM unit supplements the short-term temporal dependency information, making the reference features into spatial and temporal multi-dimensional features, effectively improving the temporal modeling capability of the FG-ConvLSTM module.

[0053] Step S300: Align the reference features based on the motion information for motion-guided feature propagation to obtain aligned features.

[0054] In this embodiment, motion-guided feature alignment utilizes extracted motion information to establish a mapping relationship between reference features of historical reference frames and the feature space of the current frame, achieving cross-frame feature space position matching. Specifically, this can be feature space calibration achieved through differentiable spatial warping operations or feature coordinate mapping based on motion vectors. Feature propagation is the process of transferring aligned effective feature information from historical reference frames to the feature domain of the current frame, achieving effective information flow. Specifically, this can be feature information transfer based on temporal correlation or historical information migration achieved through feature fusion. Aligned features are feature representations of reference features that, after motion-guided feature alignment and feature propagation, achieve position matching with the feature space of the current frame. Specifically, this can be depth features after spatial warping or multi-scale features that have completed cross-frame coordinate calibration.

[0055] In this embodiment, motion-guided feature alignment and feature propagation are achieved based on the motion information output by the motion encoder-decoder network in the FG-ConvLSTM architecture. Motion prior information is used to compensate for inter-frame motion displacement, resolving the mismatch between historical reference features and the current frame's feature space, thus enabling effective transfer of historical feature information into the current frame's feature domain. This process compensates for cross-frame feature space misalignment caused by object motion, camera displacement, and other factors, ensuring that historical reference feature information remains consistent with the current frame at the spatial semantic level. This effectively avoids the temporal information degradation problem caused by unaligned features directly participating in temporal modeling. The resulting aligned features provide an effective feature foundation for subsequent long-range temporal memory modeling, allowing the mining of long-range temporal dependencies to unfold within the aligned feature space, improving the accuracy of temporal modeling.

[0056] In one implementation, aligning the reference features based on the motion information for motion-guided feature propagation to obtain aligned features specifically includes the following steps: Step S310: Based on the motion information, determine the positional mapping relationship between the reference feature and the current frame feature coordinate space; Step S320: Based on the position mapping relationship, perform a differentiable spatial deformation operation on the reference feature to calibrate the reference feature to the feature coordinate space of the current frame to compensate for inter-frame motion displacement and obtain the aligned feature.

[0057] In this embodiment, as Figure 3 As shown in the warping of the differentiable space, based on the reconstructed motion vector... With multi-scale enhanced reference features The FG-ConvLSTM module performs motion-guided feature alignment, calibrating historical reference features to the current frame. The characteristic coordinate space is used to compensate for inter-frame motion displacement.

[0058] First, given a time series reference set ,For example ,right Each historical reference frame within ( ), through shared feature extractor The initial depth features are extracted using the following formula:

[0059] in, for Reference features of time-historical reference frames, shared feature extractor It shares the basic feature extraction network parameters with the FIN module to ensure the consistency of the feature space.

[0060] Subsequently, based on the reconstructed motion vectors For each Perform a differentiable space warping operation, mapping it to the current frame. The characteristic coordinate space, the deformation formula is:

[0061] in, This represents a deformation operation in a differentiable space, which can be implemented using bilinear sampling to ensure gradient propagation. For reference from the past A series of consecutive moments can be set. , Represents a set of reference frames Reference features, From Time's up The reconstructed motion vector at time step [time]. The aligned feature obtained after deformation is... Figure 3 In This feature is related to the current frame. The feature space matching is achieved, which solves the problem of mismatch between historical reference features and the current frame feature space.

[0062] The differentiability of differentiable spatial deformation operations allows these operations to be integrated into the end-to-end training process, enabling joint optimization of the entire encoder-decoder network and avoiding gradient breakage issues caused by non-differentiable operations. By calibrating historical reference features to the current frame's feature space through this operation, inter-frame displacement caused by factors such as object motion and camera displacement can be accurately compensated. This ensures that historical reference features remain consistent with the current frame at the spatial semantic level, effectively avoiding temporal information degradation and noise interference caused by misaligned features participating in temporal modeling. It provides a high-quality feature foundation for subsequent long-range temporal memory modeling through spatial matching. Its direct technical effect is to improve the semantic consistency of cross-frame feature matching and extend the effective time window for temporal modeling.

[0063] Step S400: Perform long-range temporal memory modeling on the aligned features to obtain the long-term temporal context.

[0064] In this embodiment, long-range temporal memory modeling involves temporal association mining, information filtering, and integration of aligned features at different times to achieve the process of modeling the temporal dependencies of a video sequence across multiple frames. Specifically, this can be achieved through the accumulation of temporal information using gated memory units and the temporal association of multi-frame features using a temporal fusion network. The long-term temporal context is a feature representation obtained after long-range temporal memory modeling, containing effective temporal information of the video sequence across multiple frames. It is the result of modeling long-range temporal dependencies and can specifically include multi-scale long-term memory features, a feature set that can provide long-range temporal priors for the current frame, and long-range temporal information features after gating and filtering.

[0065] In the temporal modeling stage of the FG-ConvLSTM module in this embodiment, long-range temporal dependency mining is performed through the gated memory unit inside the module, realizing the filtering and accumulation of temporal information across multiple frames. Temporal-dimensional information mining is performed on the spatially aligned features of multiple frames in the feature domain, overcoming the limitation of traditional temporal modeling that relies only on short-term reference frames, and dynamically filtering and accumulating high-confidence temporal information across multiple frames. This process, based on spatial alignment, achieves long-range information mining in the temporal dimension, effectively capturing the semantic evolution patterns across multiple frames in the video sequence and solving the problem of narrow temporal receptive field in traditional methods. The obtained long-term temporal context contains effective temporal information from distant frames in the video sequence, providing robust long-range temporal priors for subsequent long-short context aggregation, allowing subsequent context fusion to take into account both long-range global temporal information and short-range local detail information, thus improving the richness of the temporal context.

[0066] In one implementation, the step of performing long-range temporal memory modeling on the aligned features to obtain a long-term temporal context specifically includes the following steps: Step S410: Establish temporal associations for the aligned features at different times through gated memory units, complete long-range temporal memory modeling, and obtain the long-term temporal context; The gated memory unit includes an input gate, a forget gate, and an output gate, which are used to retain high-confidence long-term time-series clues and suppress redundant time-series information.

[0067] In this embodiment, based on aligned features ,Right now Figure 3 In Long-range temporal memory modeling is completed through the gated memory unit of the FG-ConvLSTM module, i.e., the ConvLSTM after the warp module, to obtain the long-range temporal context. .

[0068] Specifically, firstly, the aligned features ,Right now Based on temporal distance, it is divided into short-term alignment features. Long-term alignment features . Aligned features of neighboring times in the corresponding time series reference set The aligned features of distant moments in the corresponding time series reference set are expressed as follows:

[0069]

[0070] in, and yes The disjoint subsets correspond to short-term and long-term reference frames, respectively, enabling the model to learn different representations for frames that are closer in time and frames that are farther away.

[0071] Will and After being concatenated along the channel dimension, the data is input to a gated memory unit, which contains input gates. Forgotten Gate With output gate Through the dynamic control of three types of gate structures, the long-range time-series information can be filtered and accumulated.

[0072] The formula for calculating gated memory units is as follows:

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] in, It is the Sigmoid activation function. The hyperbolic tangent activation function is used. This represents the convolution operation. This represents element-wise multiplication. The convolution weights for each gate and candidate state are... For the corresponding bias term, This refers to the state at the previous moment. The current state. This refers to the long-term temporal context obtained after modeling with long-range temporal memory. .

[0079] The gated memory unit, the second ConvLSTM, deeply couples the motion alignment mechanism with the gated memory, breaking through the inefficient mode of traditional stacked ConvLSTMs that directly transmits misaligned features. It can safely and efficiently utilize reference information from dozens of frames ago, significantly expanding the effective time window for temporal modeling. Its direct technical effect is to effectively model long-range temporal dependencies across multiple frames in video sequences, retain high-confidence long-range temporal cues, suppress redundant temporal noise, and give the obtained long-term temporal context excellent temporal prior representation capabilities. This provides a robust long-range temporal foundation for subsequent aggregation of long and short-term temporal contexts, improving the ability to handle complex temporal scenarios such as slow motion, periodic content recurrence, and long-term occlusion.

[0080] Step S500: Obtain the short-term time series context and aggregate the long-term time series context with the short-term time series context to obtain the long-short time series context.

[0081] In this embodiment, short-term temporal context is a feature representation containing recent temporal information of the video sequence, extracted from reference frames or features of nearby time steps preceding the current frame. Specifically, it can be local temporal features extracted from 1-2 frames before the current frame or short-term memory features of nearby time-step frames. Temporal context aggregation is a process of deeply fusing the extracted long-term and short-term temporal contexts to integrate the two types of temporal information. Specifically, it can be context fusion combining multi-scale feature aggregation, channel splicing and residual fusion, or feature-weighted temporal information integration. Long and short-term temporal contexts are feature representations of the video sequence that fuse long and short-term temporal information after aggregating the long-term and short-term temporal contexts. Specifically, it can be multi-scale spatiotemporal context features or a feature set that can provide comprehensive temporal priors for encoding and decoding the current frame.

[0082] In this embodiment, the long-term temporal context output by the FG-ConvLSTM module is fused with the local short-term temporal context to provide a unified temporal prior for the architecture. Long-range and short-range temporal information of the video sequence are mined separately, and then deep fusion is achieved through targeted aggregation, taking into account both the detailed information of recent frames and the global temporal information of distant frames. This process effectively integrates the long- and short-term temporal features of the video sequence, solving the problem of incomplete representation by a single temporal context. The short-term temporal context can retain fine temporal details of the current frame's neighboring moments, while the long-term temporal context can provide global temporal patterns across multiple frames. The fusion of the two forms a richer and more discriminative spatiotemporal context representation. This long- and short-term temporal context can provide comprehensive temporal prior support for subsequent current frame compression encoding and reconstruction decoding, allowing the encoding and decoding process to unfold based on richer temporal information, improving encoding efficiency and reconstruction quality.

[0083] In one implementation, the step of obtaining the short-term time-series context and aggregating the long-term time-series context with the short-term time-series context to obtain a long-short time-series context specifically includes the following steps: Step S510: Based on the preset number of reconstructed frames and their corresponding depth features from the preceding time of the current frame, the short-term temporal context is obtained through short-term temporal context feature extraction; Step S520: Based on the motion information, perform secondary spatial alignment between the long-term temporal context and the short-term temporal context; Step S530: Perform multi-scale feature aggregation on the aligned long-term temporal context and short-term temporal context to obtain the long and short temporal contexts.

[0084] In this embodiment, as Figure 3 As shown, the time series context mining module is used to obtain long and short time series contexts.

[0085] Specifically, first obtain the current frame Reconstructed frames from the previous two time steps , Corresponding multi-scale depth features , Then, through the short context feature extraction process in the temporal context mining module, short-term temporal context features are extracted. .

[0086] Subsequently, acquire Figure 3 The long-term timing context of the ConvLSTM output shown Long-term temporal context features are extracted through the long context feature extraction process in the temporal context mining module. .

[0087] Subsequently, Under the guidance of [the relevant authorities], two types of multi-scale features were studied. , Perform a differentiable spatial warping operation to calibrate it to the current frame. The feature coordinate space compensates for spatial deviations caused by long-range temporal information transmission, and improves the spatial matching degree of the two types of contexts.

[0088] After completing the secondary spatial alignment, the two are concatenated along the channel dimension and integrated into a single-scale fused feature through residual fusion and feature weighting operations. Then, the fused features of the three scales are globally pooled and weighted to finally obtain the long and short temporal contexts. This context simultaneously encodes both the short-term fine-grained temporal details and the long-term global temporal patterns of the video sequence, possessing comprehensive temporal prior representation capabilities.

[0089] By aligning the long-term and short-term temporal contexts and performing multi-scale feature aggregation, deep fusion of long and short-term temporal information is achieved, solving the problem of incomplete representation of single temporal context information. The resulting long and short temporal contexts can provide strong semantic support for subsequent compression coding and reconstruction decoding, improve the adaptability of the encoding and decoding process to complex dynamic scenes, and at the same time ensure the semantic consistency and structural fidelity of the reconstructed frames.

[0090] Step S600: Using the long and short temporal context as a priori, compress and encode the current frame and reconstruct and decode it to obtain the reconstructed frame corresponding to the current frame.

[0091] In this embodiment, the temporal context prior uses the fused long and short temporal contexts as a reference to provide temporal guidance for the compression coding and reconstruction decoding process of the current frame. Specifically, it can be temporal prior features used to optimize entropy coding or spatiotemporal context information that provides semantic reference for reconstruction decoding. Compression coding is the process of compressing and encoding the feature information of the current frame, converting high-dimensional feature information into a compact bitstream. Specifically, it can be entropy compression coding, feature coding based on variational super-prior structures, or efficient coding combined with temporal priors. Reconstruction decoding is the process of decoding and restoring the encoded compact bitstream, regenerating the feature information of the current frame, and further restoring it into a video frame. Specifically, it can be feature decoding based on temporal priors or frame reconstruction from the feature domain to the pixel domain. The reconstructed frame is the video frame obtained after the current frame has undergone compression coding and reconstruction decoding. It is the final output of the entire encoding and decoding method. Specifically, it can be a single-frame image restored from the feature domain to the pixel domain or a reconstructed video frame that maintains high semantic consistency and structural fidelity.

[0092] In this embodiment, the core function of the context encoder-decoder network, entropy model, and probability estimation module in the FG-ConvLSTM encoding / decoding architecture is to integrate the long and short-term temporal priors generated by the architecture into the encoding and decoding process. Using the long and short-term temporal contexts as priors for compression encoding and reconstruction decoding of the current frame, the core technical idea is to seamlessly integrate the fused long and short-term temporal contexts into the conditional coding module. This leverages the temporal prior information to guide the compression encoding and reconstruction decoding of the current frame in the feature domain, achieving synergistic optimization of rate-distortion performance and visual fidelity. This process deeply integrates temporal prior information into the entire encoding and decoding process, allowing entropy encoding to more accurately capture the spatiotemporal distribution patterns of features, effectively reducing data redundancy and improving encoding efficiency. Simultaneously, during reconstruction decoding, the temporal context priors provide semantic references for feature restoration and frame reconstruction, ensuring the structural fidelity and semantic coherence of the reconstructed frame. This solves the problem of poor reconstruction quality in complex scenes using traditional encoding and decoding methods, ultimately achieving high-efficiency, high-fidelity video encoding and decoding, allowing the reconstructed frame to maintain excellent visual quality even at low bitrates.

[0093] In one implementation, the step of using the long and short temporal context as a priori to perform compression encoding and reconstruction decoding on the current frame to obtain the reconstructed frame corresponding to the current frame specifically includes the following steps: Step S610: Fuse the long and short temporal contexts with the depth features of the current frame; Step S620: Perform entropy compression encoding on the fused features to obtain the encoded bitstream; Step S630: Reconstruct and decode the encoded bitstream using the long and short temporal contexts as priors to obtain decoding features; Step S640: Restore the decoded feature map to a pixel domain frame to obtain the reconstructed frame.

[0094] In this embodiment, as Figure 2 As shown, the core functions of the context encoder-decoder network, entropy model and probability estimation module, and feature generator module are to utilize long and short temporal contexts. For the current frame Perform compression encoding and reconstruction decoding operations to obtain the reconstructed frame. .

[0095] Specifically, the current frame The input is fed into the context encoding module, where it undergoes encoding processing to obtain encoded features. These encoded features are then input into the entropy model and probability estimation modules. The entropy model employs a factorized entropy model based on the Laplace distribution, while simultaneously utilizing both super-prior and temporal priors, i.e., long and short temporal contexts. The encoded features are probabilistically modeled and entropy compressed to obtain the bitstream, enabling efficient storage and transmission of feature information. The hyper-prior is used to capture the current frame. Intra-frame spatial dependencies are used, while temporal priors are used to capture cross-frame temporal dependencies. The combination of the two improves the accuracy of probability estimation and compression efficiency.

[0096] Next, a reconstruction decoding operation is performed on the bitstream, and the bitstream is input to the context decoding module, in both long and short timing contexts. Under the guidance of the decoding process, the decoded features are obtained through decoding and reconstruction. .

[0097] Subsequently, the decoding features will be... The input is fed into the feature generator module, which can be an upsampling network composed of a deconvolutional neural network. Through layer-by-layer upsampling and feature mapping, the decoded features are generated. The process involves reconstructing the pixel domain from the feature domain, ultimately outputting the current frame. corresponding reconstructed frames .

[0098] By using long and short temporal contexts as priors to guide the entire encoding and decoding process, a deep integration of temporal priors and encoding / decoding modules is achieved. The direct technical effect is the synergistic optimization of rate-distortion performance and visual fidelity: under the guidance of temporal priors, entropy compression coding can more accurately capture the spatiotemporal distribution patterns of features, effectively reducing data redundancy, improving encoding efficiency, and lowering the bitrate; while reconstruction decoding, guided by temporal priors, can more accurately recover the semantic information of features, and the feature generator can accurately map and restore the decoded features to pixel-domain frames, ensuring the reconstruction of the frames... The structural fidelity and semantic coherence are improved. Ultimately, the entire encoding and decoding method achieves high-quality video reconstruction while maintaining a low bit rate, solving the problem of poor reconstruction quality at low bit rates in traditional encoding and decoding methods. It can still output reconstructed frames with excellent visual quality in complex dynamic video scenes, achieving high-efficiency and high-fidelity video encoding and decoding.

[0099] The Flow-Guided ConvLSTM (FG-ConvLSTM) proposed in this embodiment significantly enhances the long-range temporal modeling capability in video compression by deeply integrating the motion-guided feature alignment mechanism with the gated memory unit of ConvLSTM. This module utilizes the motion information recovered at the decoding end to perform differentiable warping on historical reference features and adaptively fuses long and short-term temporal contexts in the feature domain, effectively improving inter-frame prediction accuracy and reconstruction quality. However, in practical applications, if motion information is estimated solely based on a single pre-trained optical flow network (such as SPyNet), the generated motion vectors may be biased or invalid in complex scenarios such as occluded areas, large displacement movements, or drastic lighting changes, thus affecting the accuracy of feature alignment and limiting the performance ceiling of FG-ConvLSTM.

[0100] To address this, this embodiment further provides a multimodal motion enhancement strategy as an optional extension of FG-ConvLSTM: by fusing multi-source signals such as optical flow estimation, monocular depth prediction, semantic segmentation maps, or camera pose information, a more robust and geometrically aware motion compensation prior is constructed. For example, a lightweight depth estimation sub-network can be introduced to predict the scene depth map, and the motion field can be optimized by combining depth consistency constraints to effectively mitigate error propagation in occluded areas; or the semantic segmentation results can be used to apply independent motion modeling to different object instances to improve the processing capability of non-rigid motion. Such multimodal enhancement mechanisms can be seamlessly integrated into the front end of FG-ConvLSTM without changing its core gating fusion structure, significantly improving the system's generalization and robustness in complex dynamic environments.

[0101] In summary, the innovation of the method proposed in this embodiment compared with existing technologies lies in the design of a novel temporal modeling module—Flow-Guided ConvLSTM (FG-ConvLSTM)—which deeply couples the motion alignment mechanism with the gated memory unit of ConvLSTM. Specifically, aligned long-short-term reference features are input into the ConvLSTM unit, whose input gate, forget gate, and output gate dynamically adjust the retention and suppression of historical information according to the current reconstruction state: high-confidence, structurally consistent temporal cues are selectively enhanced, while mismatched or redundant features are actively filtered out. This mechanism breaks through the inefficient mode of traditional "stacked ConvLSTM directly passing unaligned features," achieving adaptive, robust, and efficient utilization of information within the extended temporal window. Experiments show that this architecture significantly outperforms existing short-term reference or pixel-domain long-term reference methods in challenging scenarios such as complex motion, slow changes, and long-term occlusion, achieving higher-quality video reconstruction while maintaining a low bitrate.

[0102] like Figure 4 As shown in the figure, this embodiment of the invention provides a video encoding and decoding system based on motion-guided temporal memory. The system includes: a motion information acquisition module 10, a reference feature extraction module 20, a feature alignment module 30, a long-term temporal context acquisition module 40, a long-short temporal context acquisition module 50, and an encoding and decoding module 60.

[0103] Specifically, the motion information acquisition module 10 is used to acquire the current frame and historical reference frames of the video, and to acquire motion information from the current frame and historical reference frames; the reference feature extraction module 20 is used to extract features from the historical reference frames to obtain reference features; the feature alignment module 30 is used to align the reference features based on the motion information, and to perform motion-guided feature propagation to obtain aligned features; the long-term temporal context acquisition module 40 is used to perform long-range temporal memory modeling on the aligned features to obtain long-term temporal context; the long-short temporal context acquisition module 50 is used to acquire short-term temporal context, and to aggregate the long-term temporal context and the short-term temporal context to obtain long-short temporal context; the encoding / decoding module 60 is used to perform compression encoding and reconstruction decoding on the current frame with the long-short temporal context as a priori to obtain the reconstructed frame corresponding to the current frame.

[0104] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video encoding / decoding method based on motion-guided timing memory. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0105] Those skilled in the art will understand that Figure 5 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0106] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including instructions for performing operations as described in the embodiments of the methods above.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video encoding / decoding method based on motion-guided temporal memory, characterized in that, The method includes: Obtain the current frame and historical reference frames of the video, and extract motion information from the current frame and historical reference frames; Feature extraction is performed on the historical reference frames to obtain reference features; The reference features are aligned based on the motion information for motion-guided feature propagation to obtain aligned features. Long-range temporal memory modeling is performed on the aligned features to obtain the long-term temporal context; Obtain the short-term time series context, and aggregate the long-term time series context with the short-term time series context to obtain the long-short time series context; Using the long and short temporal contexts as a priori, the current frame is compressed, encoded, and reconstructed to obtain the reconstructed frame corresponding to the current frame.

2. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The step of acquiring the current frame and historical reference frames of the video, and obtaining motion information from the current frame and historical reference frames, includes: Obtain the current frame of the video and use the reconstructed frame corresponding to the previous moment as the historical reference frame; Based on the current frame and the historical reference frame, the spatiotemporal relationship between frames is modeled through a pre-trained motion estimation network to obtain the initial motion vector; The initial motion vector is encoded, compressed, and reconstructed to obtain the motion information.

3. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The step of extracting features from the historical reference frames to obtain reference features includes: By using multi-scale convolution kernels, spatial features are extracted hierarchically from the historical reference frames to obtain features at different scales; The features at different scales are spliced ​​and fused together to obtain multi-scale depth features, which are then used as the reference features.

4. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The alignment of the reference features based on the motion information, used for motion-guided feature propagation, yields aligned features, including: Based on the motion information, determine the positional mapping relationship between the reference feature and the current frame feature coordinate space; Based on the position mapping relationship, a differentiable spatial deformation operation is performed on the reference feature to calibrate the reference feature to the feature coordinate space of the current frame in order to compensate for inter-frame motion displacement and obtain the aligned feature.

5. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The step of performing long-range temporal memory modeling on the aligned features to obtain a long-term temporal context includes: By using gated memory units, temporal associations are established for the aligned features at different times, long-range temporal memory modeling is completed, and the long-term temporal context is obtained. The gated memory unit includes an input gate, a forget gate, and an output gate, which are used to retain high-confidence long-term time-series clues and suppress redundant time-series information.

6. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The step of obtaining the short-term time series context and aggregating the long-term time series context with the short-term time series context to obtain the long-short-term time series context includes: Based on a preset number of reconstructed frames and their corresponding depth features from the preceding time of the current frame, the short-term temporal context is obtained through short-term temporal context feature extraction. Based on the motion information, a second spatial alignment is performed between the long-term temporal context and the short-term temporal context. The aligned long-term and short-term temporal contexts are then subjected to multi-scale feature aggregation to obtain the long and short-term temporal contexts.

7. The video encoding and decoding method based on motion-guided temporal memory according to claim 1, characterized in that, The step of compressing, encoding, and reconstructing the current frame using the long and short temporal context as a priori to obtain the reconstructed frame corresponding to the current frame includes: The long and short temporal contexts are fused with the depth features of the current frame; The fused features are then subjected to entropy compression encoding to obtain the encoded bitstream; The encoded bitstream is reconstructed and decoded using the long and short temporal contexts as priors to obtain decoding features; The decoded feature map is restored to a pixel-domain frame to obtain the reconstructed frame.

8. A video encoding and decoding system based on motion-guided temporal memory, characterized in that, The system includes: The motion information acquisition module is used to acquire the current frame and historical reference frames of the video, and to acquire motion information from the current frame and historical reference frames; A reference feature extraction module is used to extract features from the historical reference frame to obtain reference features; The feature alignment module is used to align the reference features based on the motion information, and to perform motion-guided feature propagation to obtain the aligned features. The long-term temporal context acquisition module is used to perform long-range temporal memory modeling on the aligned features to obtain the long-term temporal context. The long and short time series context acquisition module is used to acquire the short-term time series context and aggregate the long-term time series context with the short-term time series context to obtain the long and short time series context. The encoding / decoding module is used to compress and encode the current frame and reconstruct and decode it based on the long and short temporal contexts to obtain the reconstructed frame corresponding to the current frame.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a video encoding / decoding program based on motion-guided timing memory stored in the memory and executable on the processor. When the processor executes the video encoding / decoding program based on motion-guided timing memory, it implements the steps of the video encoding / decoding method based on motion-guided timing memory as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a video encoding / decoding program based on motion-guided timing memory. When the video encoding / decoding program based on motion-guided timing memory is executed by a processor, it implements the steps of the video encoding / decoding method based on motion-guided timing memory as described in any one of claims 1-7.

Images