An end-to-end video compression code rate control method and device based on context reliability mask, equipment and storage medium

Abstract

The application discloses an end-to-end video compression code rate control method and device based on context reliability mask, equipment and storage medium. A Lagrange multiplier λ is taken as a code rate control quantity, in a prediction frame coding process, reference frame reconstruction features are motion compensated to obtain aligned context features, and the context features are subjected to λ conditional channel modulation and spatial reliability mask weighting: a channel modulation coefficient is obtained by embedding and mapping of λ; a spatial reliability mask is output by a mask network input by the context alignment features, a motion vector and λ, context contribution of an occlusion, mismatch or high uncertainty area is inhibited, and residual bits are reduced. The application can realize finer context utilization and code rate allocation within a target code rate range, reduce residual overhead caused by misleading context, improve code rate hit stability and rate distortion performance, and is suitable for variable code rate transmission and bandwidth limited scenes.

Description

Technical Field

[0001] This invention relates to the field of video coding and bitrate control technology, and in particular to a bitrate control method, apparatus, electronic device, and computer-readable storage medium for end-to-end neural network video codecs. Background Technology

[0002] With the rapid growth of video data volume, video compression technology has become particularly important in environments with limited bandwidth or storage resources. Traditional video compression methods such as H.264 and HEVC, while showing significant progress in compression efficiency and video quality, still face the challenge of adjusting complexity and quality at a fixed bitrate. In practical applications, the complexity and variability of video content require encoding and decoding systems to dynamically adapt to different bitrate conditions to achieve higher compression efficiency and better visual quality.

[0003] In recent years, deep learning technology has been widely used in video coding, especially end-to-end video compression models. By directly optimizing the video compression process through neural networks, these methods can compress images without relying on traditional signal processing tools (such as transforms and quantization). In particular, end-to-end video compression models based on convolutional neural networks (CNNs) and autoencoders have achieved significant performance improvements in video compression due to their powerful feature learning capabilities.

[0004] However, despite significant progress in compression efficiency, end-to-end video compression methods still face challenges in multi-bitrate control and adaptive bitrate adjustment. Existing multi-bitrate control methods primarily rely on introducing Lagrange multipliers (…). This is used to control the trade-off between bitrate and reconstruction quality during the compression process. In traditional methods... This is a global scalar parameter used to control the balance between the target bitrate and distortion, and typically requires training a separate model for each bitrate target. While this approach offers relatively stable performance, it lacks flexibility and cannot effectively handle dynamic changes in video content, such as in areas with complex motion or rich texture. The control may not achieve the desired compression effect.

[0005] To address this issue, recent research has proposed end-to-end video compression methods based on feature modulation. Existing methods include dynamic bitrate control achieved through feature modulation, the core idea of ​​which is to learn and adjust the feature map during decoding to adapt to different bitrate targets. However, these methods still face the challenge of how to more finely utilize contextual information to further improve compression efficiency, especially regarding the reliability of contextual features. Specifically, how to evaluate and select "trustworthy" contextual features to avoid compression errors caused by misleading context has become a key challenge in improving video compression efficiency.

[0006] Furthermore, while existing end-to-end video compression methods train multiple models to adapt to different bitrate targets, this approach often incurs computational overhead and lacks the flexibility and efficiency of end-to-end compression. To overcome these issues, how to achieve multi-bitrate control within a single model and effectively utilize contextual reliability information to achieve adaptive and more efficient video compression has become a current research hotspot.

[0007] A comprehensive search of currently available technologies and comparative analysis revealed several technical issues with existing end-to-end video compression solutions. These issues include insufficient granularity and continuity of bitrate adjustment in scenarios with variable bitrate or target bitrate switching, fluctuations in bitrate response under different content and prediction structures, and unreliable contextual features in occluded / mismatched regions leading to entropy model condition mismatch, increased residual bitrate, and decreased reconstruction stability. Summary of the Invention

[0008] This application aims to propose a bitrate control method, apparatus, electronic device, and storage medium for end-to-end video compression. It addresses existing end-to-end video compression schemes that suffer from insufficient granularity and continuity of bitrate adjustment in variable bitrate or target bitrate switching scenarios, bitrate response fluctuations under different content and prediction structures, and problems arising from unreliable contextual features in occluded / mismatched regions leading to entropy model condition mismatch, increased residual bitrate, and decreased reconstruction stability. The goal is to minimize reconstruction distortion within a given target bitrate or bitrate range and improve bitrate control stability and engineering deployability.

[0009] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an end-to-end video compression bitrate control method, apparatus, device and storage medium based on context reliability mask.

[0010] Firstly: Provide a bitrate control method for end-to-end video compression. This application provides a bitrate control method for an end-to-end neural network video codec. The method uses bitrate control parameters during the prediction frame encoding process. As a rate-distortion tradeoff control, the same model covers multiple bitrate points, and more stable bitrate-quality control is achieved through adaptive utilization of the "reference reconstruction feature context". The method includes: (1) Obtain input and reference information: Obtain the video sequence or frame sequence to be encoded, and determine the current frame. Reference Reconstructed Frame and obtain bitrate control parameters. The aforementioned It can be a scalar, vector, or spatial distribution graph. -map).

[0011] (2) Contextual feature extraction and motion compensation alignment: Reference features are extracted from the reference reconstructed frame. And estimate motion vectors based on the current frame and the reference frame. ; The warped feature is obtained by performing motion compensation on the reference feature using motion vectors:

[0012] in This represents the motion compensation / warp alignment operator.

[0013] (3) Based on Context channel modulation (RefScale): Mapped to channel-wise modulation coefficients Channel modulation is performed on the alignment context features to obtain: ,

[0014] In some implementations, to ensure different training / testing settings Consistent numerical scales allow for... After normalization, it is used as a conditional input.

[0015] (4) Reliability-based spatial mask generation (RefMask) and weighted context: The context features after channel modulation Motion vectors and Input spatial mask network, output spatial mask And spatial weighting of the context features yields the reliability-aware context features: ,

[0016] The input to the masking network includes at least warped_feature, motion_vectors, and Conditional branching; in the current implementation, the network first... After convolution, joint features are extracted, and then... The scale is generated by embedding and MLP, and the mask is predicted after modulation of visual features. It also supports low-resolution prediction and upsampling recovery.

[0017] 5) Conditional coding and consistency decoding: Integrating reliability-aware contextual features The conditional information used for predicting frame encoding is used to generate latent variables, predict probabilistic models, and generate bitstreams through entropy encoding. At the decoding end, the reconstructed frame is recovered using the same conditional input as at the encoding end. .

[0018] In some implementations, the condition information It can be applied to entropy model conditions of residual branches, context refinement network inputs, or subsequent modules of motion-compensated context extraction paths to achieve adaptive behavior of "making fuller use of context in reliable regions and reducing dependence in unreliable regions".

[0019] (6) End-to-end training objective: The method can be trained according to the rate-distortion criterion, and its basic objective is: ,

[0020] in Indicates the bit rate term. This indicates a distortion term (e.g., MSE).

[0021] The second aspect: Joint modeling of low-resolution spatial masks and reliability (improving stability and computational efficiency) In some implementations, spatial masks are used to reduce computational complexity while maintaining an effective receptive field. It can be obtained by upsampling after low-resolution branch prediction, that is, first outputting the low-resolution mask. Then, it is restored to its original state through interpolation and upsampling. The spatial dimensions. This method has the following effects: 1) It expands the receptive field of the mask network, making it easier to perceive occlusion boundaries and large motion mismatch areas; 2) It brings natural smoothness through the interpolation process, reducing local noise in the mask; 3) It reduces computational overhead and facilitates deployment.

[0022] Furthermore, in some implementations, the "reliability" of the mask network is not a single metric explicitly calculated, but rather determined by... Implicit reliability characterization obtained from joint input: Provide evidence of whether the aligned context is locally continuous or anomalous; Provide evidence of the "risk of motion amplitude and geometric mismatch"; Provides a policy constraint on whether to use the context more aggressively or more conservatively under the current bitrate target (making the reliability judgment adaptive to the bitrate target).

[0023] Thirdly: Introduce TV / Temporal smooth constraints (to suppress mask timing jitter). In some implementations, to suppress temporal discontinuities (temporal jitter / flicker) in the spatial mask between adjacent frames and improve reconstruction quality stability and bitrate response stability, this application introduces spatial total variation (TV) and temporal consistency regularization into the mask during the training phase, resulting in an extended training objective: ,

[0024] in Represents the spatial gradient operator, This represents the result of aligning the mask motion compensation from the previous time step to the current time step based on the motion vector. These are the weighting coefficients.

[0025] By applying the above constraints, the phenomenon of "fluctuating mask values ​​in the same motion trajectory" can be reduced, making the context weighting strategy more consistent over time, thereby improving the temporal stability of video reconstruction.

[0026] Fourth aspect: Devices, electronic equipment and storage media (corresponding to the first aspect) The second and third aspects can be implemented corresponding to the method of the first aspect. This application also provides a bitrate control device for end-to-end video compression, including a reference feature extraction module, a motion estimation and motion compensation module, and... The system includes a conditional channel modulation module, a spatial mask generation and weighting module, and an encoding and decoding module; a reference feature extraction module, used to extract reference features from the reference reconstructed frame; and a motion estimation and motion compensation module, used to obtain motion vectors and generate aligned context features. The conditional channel modulation module is used to control parameters based on the bit rate. Generate channel modulation coefficients and perform channel modulation on the aligned context features; a spatial mask generation and weighting module is used to combine the channel-modulated context features and motion vectors with... The input masking network generates a spatial mask and performs spatial weighting to obtain reliability-aware context features; the encoding / decoding module is used to implement end-to-end encoding and end-to-end decoding of the current frame based on the reliability-aware context features.

[0027] This application also provides an electronic device, including a processor and a memory, wherein instructions stored in the memory, when executed by the processor, implement the method of any of the above embodiments; this application also provides a computer-readable storage medium, wherein a computer program stored thereon, when executed by a processor, implements the method of any of the above embodiments.

[0028] By adopting the above structure, the present invention has the following beneficial effects: 1) By using "context reliability space masking" to suppress the context contribution of occlusion, mismatch or high uncertainty regions, the residual overhead caused by misleading conditional information and the conditional mismatch of entropy model are reduced, thereby reducing the bit rate and improving rate distortion performance; 2) By introducing TV / Temporal smooth constraints, mask temporal consistency is improved, and the jitter of the adjacent frame context weighting strategy is reduced, thereby improving the stability of reconstruction quality and cross-content robustness; 3) The mask network input in this application only relies on the warped_feature, motion_vectors, and [other elements] already present in the encoding / decoding process. The conditional information is easily integrated with existing DCVC-like end-to-end architectures, and has good engineering feasibility and scalability.

[0029] In summary, based on the aforementioned technical problems, this invention proposes a context-reliability-aware rate control method. By combining feature modulation with spatial mask weighting, it further improves the flexibility and compression efficiency of multi-rate control. This method can adaptively select appropriate context information and effectively suppress mask flickering by introducing temporal consistency constraints, thereby ensuring the stability of video reconstruction quality and optimizing compression performance. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall process of an end-to-end video compression bitrate control method according to the present invention, including obtaining the current frame and the reference reconstructed frame, and based on the bitrate control parameters... The process involves context channel modulation, reliability-based spatial mask generation and weighting, and end-to-end entropy encoding and decoding reconstruction.

[0031] Figure 2 A schematic diagram of a masked network structure.

[0032] Figure 3 This invention is based on A schematic diagram of the context feature channel modulation (RefScale) structure for conditions, illustrating the alignment of context features. Channel modulation coefficient Processed And input the subsequent context to refine the relationship of the conditional branches of the network / entropy model.

[0033] Figure 4 This is a schematic diagram of a context reliability-aware spatial masking network (RefMask) structure according to the present invention, showing the modulation-aligned context features. Sports Information and bitrate control parameters Generate a spatial mask for the input. and through The reliability-aware context is obtained for subsequent context refinement of the network / entropy model conditional branches. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the following embodiments are only for explaining the present invention and do not constitute a limitation on the scope of protection of the present invention. Equivalent substitutions or modifications made by those skilled in the art to the step sequence, module combination, and network structure details of the various embodiments without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0035] In the description of this invention, the term "current frame" may refer to the first frame to be encoded. Frame image, denoted as "Reference reconstructed frame" can refer to the decoded first frame. The reconstructed image is denoted as . "Reference features" may refer to The feature representation obtained by the feature extraction network is denoted as "Motion vector / motion field" can refer to the pixel displacement field obtained by the motion estimation network, denoted as... "Alignment / motion compensation" refers to the operation of differentiable alignment of reference features based on motion vectors, denoted as... "Aligned context feature" can refer to the aligned feature, denoted as... "Rate control parameters" can refer to rate-distortion tradeoff parameters / Lagrange multipliers, denoted as... "Channel modulation coefficient" can refer to... The generated channel-scaled vector is denoted as A "spatial mask" can refer to a single-channel weighted map that weights contextual features in a spatial dimension, denoted as... Its value range is "Reliability-aware context features" can refer to the context features after masking and weighting, denoted as... Without causing ambiguity, the above symbols can also be used in the corresponding steps at the decoding end.

[0036] Example 1: A bitrate control method for end-to-end video compression (corresponding to...) Figure 1 and Figure 3 ) This embodiment provides a bitrate control method for end-to-end video compression, applied to a neural network video encoding / decoding system comprising an encoding end and a decoding end. This method uses bitrate control parameters... To control the amount of data, the same model can cover multiple target bitrate points. By channel modulation and reliability weighting of reference context features, the contextual misleading of occluded / mismatched regions can be reduced, thereby reducing the residual bitrate and improving reconstruction stability.

[0037] Step S101: Obtain the input frame and bitrate control parameters. Obtain the current frame to be encoded. and at least one reference reconstructed frame and obtain bitrate control parameters. In some implementations, Determined by external bitrate targets, network bandwidth, service class, or user configuration; in other implementations, It can be adaptively selected by the system's internal scheduling strategy. Can be in scalar form It can also be a spatial distribution form ( -map) .

[0038] Step S102: For Standardization processing is optional. To improve the generalization and stability of a single model across code rate points, some implementations perform standardization processing. Normalization / scaling is performed to obtain For example, to Mapping to a preset interval, or linearly scaling the logarithmic domain, makes the scale of the conditional input more stable. Unless otherwise specified in the following description, This can be understood as including the bitrate control parameters of this standardized processing.

[0039] Step S103: Reference Feature Extraction. Extract the reference reconstructed frame. Input the reference feature extraction network to obtain reference features In some implementations, the reference feature extraction network is a convolutional neural network, which outputs feature dimensions (e.g., number of channels) that match the subsequent context refinement network. ).

[0040] Step S104: Motion estimation and motion vector generation. The current frame... With reference reconstructed frame Input the motion estimation network to obtain motion vectors In some implementations, the motion estimation network is a multi-scale pyramid structure to improve the ability to estimate large displacements; in other implementations, it can... Injected as a conditional input into the motion estimation network, the motion estimation achieves different stability and detail trade-offs under different bitrate targets (e.g., a smooth and stable motion field at low bitrates, and retains more fine motion details at high bitrates).

[0041] Step S105: Motion compensation alignment generates warped features. Based on motion vectors. For reference features Perform motion-compensated alignment to obtain alignment context features: , in It can be a differentiable warp operation (e.g., bilinear sampling) for end-to-end training.

[0042] Step S106: Based on Context channel modulation (RefScale, corresponding to) Figure 3 ).Will The input channel modulation network generates the channel modulation coefficients. And perform channel-wise scaling on the alignment context features to obtain: ,

[0043] In some implementations, Scaling vector for output by channel (dimension and (The number of channels is consistent), and broadcast to the spatial dimension for element-wise multiplication; in some implementations, to avoid training oscillations caused by excessive conditional modulation in the early stages of training, the channel modulation network can be initialized with an approximate identity mapping, so that... Initially, the numbers are almost all 1s.

[0044] Step S107: Context Refinement and Conditional Encoding. The context features after channel modulation are... Input context refinement network to obtain refined context features (which can be denoted as...) The contextual features are then used as conditional information input to the predictive frame coding network and / or entropy model network to generate latent variable representations and perform entropy coding to generate a bitstream. The output bitstream from the encoder may contain a motion information bitstream and a residual information bitstream, where the residual information may be the latent variable entropy coding result.

[0045] Step S108: Decoding end consistency decoding. The decoding end receives the bitstream and parses it to obtain the required information. At the decoding end, a network structure consistent with the encoding end is used to decode the bitstream. (or under the same standardization method) Repeat steps S103 to S107 for alignment, modulation, and conditional modeling to complete the reconstruction and obtain the reconstruction result of the current frame. Because the encoding and decoding ends construct the condition information in the same way, encoding and decoding consistency is guaranteed.

[0046] In some implementations, the method of this embodiment performs end-to-end optimization according to the rate-distortion criterion, and its basic objective function can be expressed as: ,

[0047] in For the bit rate term (e.g., the expectation of the latent variable entropy). For distortion terms (such as mean squared error or other distortion measures). It is used to control the trade-off between bit rate and distortion, thereby achieving controllable encoding at multiple bit rate points.

[0048] In this embodiment, the context channel modulation in step S106 is introduced, so that the bitrate control is not limited to adjusting the compression intensity of the latent variable, but further acts on the intensity of the "conditional information itself", thereby improving the bitrate response stability under different content complexities.

[0049] Example 2: Context-aware Reliability Masking Network and its Generation Method (corresponding) Figure 2 and Figure 4 ) This embodiment provides a context reliability-aware masking network and its generation method, used to spatially weight reference context features in an end-to-end video compression model, thereby enabling the masking of different bitrate control parameters. The adaptive suppression of unreliable regions reduces the bit rate overhead caused by conditional mismatch and improves reconstruction stability. For ease of description, let the th... The frame's reference alignment context features are ,through The modulated features are The motion vector (optical flow) is The output space mask of the mask network is .

[0050] Step S201: Obtain the mask network input. Obtain (1) alignment context features. (2) Motion vector (3) Bitrate control parameters In one implementation, The warped feature is obtained after motion compensation and alignment of the reference features, and can be further processed by a channel scaling network to achieve a more advanced feature set based on... Context modulation; This is the motion vector output for the model's motion branch and reproducible on the decoding side.

[0051] Step S202: Perform format unification and size alignment on the input. In one implementation, when for -map (dimension is) When ), its scalar form is obtained by converging it in the spatial dimension. , as input for conditional branches; when When in scalar / vector form, use directly. In one implementation, when... and When the spatial resolution is inconsistent, for Perform interpolation to satisfy This is to ensure the robustness of subsequent splicing operations.

[0052] Step S203: Construct a visual joint input and perform downsampling feature extraction. and By concatenating the data along the channel dimension, we obtain the joint input: .

[0053] In one implementation, to expand the receptive field and reduce computational cost, a step size of [missing information] is used. convolution pairs Perform downsampling (where ), to obtain low-resolution visual features: ,

[0054] in It can be composed of two or more layers of convolutions and nonlinear activations. In this way, the mask network can comprehensively observe "texture breakage / mismatch signs" (from...) in the low-resolution domain. ) and “risk of drastic changes in movement / occlusion” (from This forms an implicit reliability characterization.

[0055] Step S204: For Continuous embedding is performed to generate conditional modulation coefficients. In one implementation, The input sine embedding module obtains the embedding vector. The channel modulation vector is obtained through a multilayer perceptron. To ensure stability during the initial training phase, the modulation vector can be mapped to a scaling factor centered at 1 using a sigmoid algorithm. ,

[0056] And expand it to With Broadcast multiplication. This mapping makes when hour, This allows the masking network to approach identity modulation during the initialization phase, avoiding the influence of... The introduction of excessively strong perturbations in the conditions leads to training oscillations.

[0057] Step S205: Perform conditional modulation and generate a low-resolution mask. In one implementation, the visual features are modulated channel by channel to obtain: ,

[0058] And Input mask to generate a subnetwork to obtain a low-resolution mask: ,

[0059] in It can be a two-layer convolutional structure: the first layer extracts local structural information, and the second layer outputs a single channel and is normalized by Sigmoid.

[0060] Step S206: Upsample the low-resolution mask to obtain the final mask and perform context weighting. When At that time, Upsampling yields: ,

[0061] in Bilinear interpolation can be used; bilinear interpolation has a natural smoothing effect, which helps reduce noise and fragmented regions in the mask space. Subsequently, the context features are spatially weighted using the mask to obtain the reliability-aware context features: , And Input context refinement networks and / or conditional entropy models as temporal priors for subsequent latent variable probability distribution prediction and entropy encoding.

[0062] Step S207: Key Initialization Strategy (Optional). In one implementation, to avoid the sudden loss of context information due to random mask outputs approaching 0 in the early stages of training, the mask network can be initialized as follows: the parameters of the last layer of the conditional branch are initialized to 0, making... Initially close to 1; and initialize the mask output layer bias to a positive number, so that The initial approach is close to all-pass, thus ensuring that the model has a stable baseline reconstruction capability before gradually learning to suppress unreliable regions.

[0063] In some implementations, the input to the masking network does not require an explicit provision of a separate "reliability" scalar graph. Because... It already includes contextual consistency cues after motion compensation. Directly characterizes the intensity of motion and boundary changes, while The strength of the "preservation / suppression" strategy for context is adjusted under different target bit rates. Therefore, the combination of the three can form an implicit reliability estimate and drive mask learning.

[0064] In some implementations, the mask The application of this feature is not limited to the direct weighting of warped features. It can also be applied to the input / output features of the context refinement network or as the modulation weight of the conditional entropy model to further influence the conditional distribution prediction of the probability model, thereby achieving bitrate savings.

[0065] Example 3: TV / Temporal smoothness constraint and training method for masks This embodiment provides a method for spatial masks. The training method of temporal smoothness constraint and spatial total variation (TV) constraint is used to suppress the temporal discontinuity of the mask between adjacent frames, improve the temporal consistency of the context weighting strategy, and thus improve the stability of reconstruction quality and bitrate control.

[0066] Step S301: Construct the base rate-distortion training objective. Train the end-to-end codec system in Example 1 or Example 2 to construct the base rate-distortion objective function: , in For bit rate, For distortion terms, This is a bitrate control parameter.

[0067] Step S302: Construct the spatial total variation (TV) regularization term. To constrain excessive fragmentation and noise in the spatial dimension of the mask, a spatial TV regularization is introduced in one implementation: ,

[0068] in This represents the spatial gradient operator. This regularization encourages the mask to form a more coherent regional structure, making the spatial pattern of "occlusion / mismatch region suppression and reliable region preservation" more stable.

[0069] Step S303: Construct a temporal consistency regularization term based on motion alignment. Since the video object moves over time, in one implementation, motion vectors are first used... Mask of the previous moment Align with the current frame coordinate system to obtain And based on this, construct a time consistency regular expression: , This regular expression is used to constrain the mask values ​​of "regions on the same motion trajectory" to remain consistent in adjacent frames, suppressing high-frequency fluctuations in the mask over time, thereby reducing temporal jitter in the reconstruction results and improving cross-frame stability.

[0070] Step S304: Construct an extended training objective and perform end-to-end optimization. In one implementation, the regularization terms from steps S302 and S303 are weighted and superimposed onto the base rate-distortion objective to obtain the extended training objective: , in represents the weight coefficients. End-to-end training is performed using this objective function, enabling the masked network to achieve stronger spatial coherence and temporal consistency while satisfying rate-distortion optimization.

[0071] Step S305: Weight coefficients and training strategy (optional). In some implementations, It can be fixed; in some other implementations, The mask can be gradually increased during the training phase to avoid overly constraining the mask in the early stages of training, making it difficult to learn effective reliability patterns; in other implementations... Can be followed Adaptive changes allow for greater emphasis on mask stability and suppression of misleading contexts at low bitrates, while prioritizing detail preservation at high bitrates.

[0072] By using the TV / Temporal smooth constraint in this embodiment, the temporal discontinuity of the mask in adjacent frames can be significantly reduced, the consistency of the context weighting strategy can be improved, thereby enhancing the subjective and objective stability of the reconstruction results and improving the predictability of bitrate control.

[0073] Example 4: Representation, position of action, and system integration variants (corresponding) Figures 1-3 ) This embodiment provides several optional variations to expand the applicability and scope of protection of the present invention in different end-to-end video compression architectures.

[0074] Step S401: Multiple representations of . In one implementation, It is a global scalar used to control the rate-distortion tradeoff across the entire frame; in another implementation, In vector form, it is used to apply different bitrate strategies to different layers or submodules; in another implementation, for -map is used to apply differentiated bitrate strategies to different spatial regions, and can be obtained through spatial aggregation. As input for conditional branches, while retaining -map serves as a spatial conditional input for certain network layers.

[0075] Step S402: Variations in the application of modulation and masking. In one implementation, channel modulation and spatial masking are applied to the warped feature (e.g., S106 in Embodiment 1 and S206 in Embodiment 2); in another implementation, the mask is applied to the input or output of the context refinement network to adjust the processing intensity of the refinement network for unreliable regions; in yet another implementation, the mask or its derived features are used as conditional input weights of the entropy model / temporal prior network to influence probability distribution prediction and further reduce the bit rate.

[0076] Step S403: Low-resolution masking and multi-scale masking variants. In one implementation, the masking network employs a downsampling ratio. One approach is to perform low-resolution prediction and upsampling recovery; another approach is to use multi-scale masks (coarse to fine) to balance large-scale occlusion judgment and fine-grained boundary localization; yet another approach is to generate masks only in key layers or key temporal modules to further reduce computational overhead.

[0077] Step S404: Consistent integration between the encoder and decoder. Any of the above variations can use consistent input construction and network parameters at both the encoder and decoder ends to ensure mask consistency. Contextual features This ensures reproducibility, thereby guaranteeing encoding and decoding consistency.

[0078] Example 5: Implementation of the device, electronic device and storage medium; This example provides implementations of the device, electronic device and storage medium corresponding to the above method.

[0079] Step S501: Provide an end-to-end video compression bitrate control device. The device includes: a data acquisition module, a reference feature extraction module, a motion estimation and motion compensation module, and The system includes a conditional channel modulation module, a mask generation and weighting module, and an encoding / decoding module. Among these: The data acquisition module is used to acquire , and ; The reference feature extraction module is used to obtain from the reference reconstructed frame ; The motion estimation and motion compensation module is used to obtain And generate ; The conditional channel modulation module is used to generate And obtained ; The mask generation and weighting module is used to generate And obtained ; The codec module is used for... After completing the latent variable generation, entropy coding output bitstream, and consistency decoding, the following results are obtained: .

[0080] Step S502: Provide an electronic device. The electronic device includes a processor and a memory, the memory storing a computer program, which, when executed on the processor, causes the electronic device to perform the method steps described in any one of Embodiments 1 to 4.

[0081] Step S503: Provide a computer-readable storage medium. The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of Embodiments 1 to 4.

[0082] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions are used to implement the end-to-end video compression bitrate control method described in Embodiment 1, Embodiment 2, or Embodiment 3, thereby controlling the bitrate under given bitrate parameters. Motion compensation alignment based on reference context features is completed under (or target bit rate constraints). The system employs context channel modulation, reliability spatial mask generation and weighting based on context and motion information, bitstream generation, and consistent decoding reconstruction. In some implementations, the total variation (TV) and temporal smoothness constraints are further utilized to suppress temporal discontinuities in the mask between adjacent frames, thereby improving reconstruction quality stability and bitrate response stability.

[0083] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit the scope of protection of this application. Those skilled in the art can make various modifications or substitutions to the above embodiments without departing from the concept of this application, for example, by changing the bitrate control parameters. The representation is changed from scalar to vector or spatial distribution diagram. -map), replacing the downsampling factor of the mask network from 2 to 4 or adopting a multi-scale mask structure, replacing the spatial weighted position from alignment context features to the input / output of the context refinement network or entropy model conditional features, and changing the temporal consistency regularization from motion alignment-based... Distance replaced with Distance or Huber loss, replacing the distortion metric from mean square error to the distortion term corresponding to MS-SSIM, or replacing the modulation network from channel scaling to affine modulation / gated modulation, etc., should all fall within the protection scope of this application.

[0084] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. An end-to-end video compression bitrate control method based on context reliability masking, applied to a neural network video encoding / decoding system including an encoding end and a decoding end, characterized in that: It includes the following steps: 1) Obtain the current frame to be encoded and at least one reference reconstructed frame Reference features are extracted from the reference reconstructed frame. ; 2) Based on the current frame With reference reconstructed frame Estimate motion information to obtain motion vectors And motion compensation is performed on the reference features to obtain the aligned context features. ; 3) Obtain bitrate control parameters ,Depend on Generate channel modulation coefficients The aligned context features are then subjected to channel modulation to obtain the channel-modulated context features. ; 4) Modulate the context features of the channel Motion vectors With bitrate control parameters Input a masking network, output a spatial mask. The spatial mask is then used to spatially weight the context features to obtain the reliability-aware context features. The spatial weighting satisfies: ， 5) Based on the aforementioned reliability-aware context features For the current frame End-to-end encoding is performed to obtain a bitstream, and at the decoding end, the bitstream and the reliability-aware context features are used as the basis for the decoding. Perform end-to-end decoding to obtain the reconstruction result of the current frame. .

2. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: The channel modulation coefficient mentioned in step 3) From the The embedding mapping and multilayer perceptron generation, and the channel modulation satisfy: ， in This is a scaled vector output by channel and broadcast to spatial dimensions.

3. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: The bit rate control parameters It can be any of the following: scalar, vector, or spatial distribution graph; when When the spatial distribution map is obtained, it is spatially converged to obtain scalar or vector forms for generating channel modulation coefficients and / or as input to mask network conditions.

4. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: Step 4) The masking network includes a visual branch and a conditional branch: the visual branch is used to... and Extract joint representation from the splicing features, and use conditional branching to... Embedding and modulation parameters are generated to make the spatial mask It adapts to the target bit rate.

5. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: Step 4) Spatial mask This is obtained by upsampling after low-resolution prediction; that is, the masking network first outputs at low resolution. Then, through interpolation and upsampling, the result is obtained. This expands the receptive field and reduces computational complexity.

6. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: The space mask The range of values ​​is It is used to suppress the context contribution of occluded regions, mismatched regions, or highly uncertain regions, thereby reducing the residual bit overhead caused by misleading context.

7. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: Step 5) The end-to-end encoding includes: based on Generate a latent variable representation for the current frame and incorporate it into the entropy model. As conditional information to predict probability distributions, the latent variables are then arithmetically or range-coded to generate a bitstream.

8. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: The method introduces a temporal smoothness constraint during the training phase, applying total variational regularization and temporal consistency regularization to the mask to ensure that the training objective satisfies: ， in For bit rate, For distortion terms, Represents the spatial gradient operator, Representation based on motion vectors Alignment operator, These are the weighting coefficients.

9. The end-to-end video compression bitrate control method based on context reliability mask according to claim 8, characterized in that: In the aforementioned time consistency regularization... The mask from the previous time step is obtained by aligning the motion vector with the motion compensation, which reduces the high-frequency fluctuation of the mask in the time dimension and improves the stability of reconstruction quality.

10. The end-to-end video compression bitrate control method based on context reliability mask according to claim 1, characterized in that: The channel modulation and spatial weighting in steps 3) and 4) must be applied at least at one of the following locations: a) Aligned context features of the reference features after motion compensation; b) Aligning contextual features with subsequent contextual refinement network input; c) As conditional input features for entropy models or temporal prior networks.

11. An end-to-end video compression bitrate control device based on context reliability masking, characterized in that, It includes: The reference feature extraction module is used to extract reference features from the reference reconstructed frame; The motion estimation and motion compensation module is used to obtain motion vectors and generate aligned context features; The channel modulation module is used to control parameters according to the bit rate. Generate channel modulation coefficients and perform channel modulation on the alignment context features; The mask generation and weighting module is used to combine the channel-modulated context features and motion vectors with... The input masking network generates a spatial mask and performs spatial weighting to obtain reliability-aware context features; The encoding / decoding module is used to implement end-to-end encoding and end-to-end decoding of the current frame based on reliability-aware context features.

12. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is run on the processor, it causes the electronic device to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run on a processor, performs the method according to any one of claims 1 to 10.

Images