Frame quantization coefficient adjustment methods and video stream encoding methods

Abstract

This application provides a method for adjusting the quantization coefficients of a frame and a method for encoding a video stream. The method includes: acquiring a target image block corresponding to a target frame; determining model selection parameters corresponding to the target image block, wherein the model selection parameters include at least one of the following: block size parameters, and other parameters; retrieving a target model corresponding to the model selection parameters from multiple candidate models; determining input parameters corresponding to the target model, wherein the input parameters include target block coefficients of the target image block, target quantization parameters, and initial quantization coefficients, and the target block coefficients include target amplitude coefficients and target sign coefficients; inputting the input parameters into the target model to obtain a predicted adjustment value for the target image block; and adjusting the initial quantization coefficients based on the predicted adjustment value to obtain the target quantization coefficients. This solves the technical problem in related technologies of balancing compression efficiency and computational complexity.

Description

Technical Field

[0001] This application relates to the field of frame processing technology, and in particular to a method for adjusting the quantization coefficients of a frame and a method for encoding a video stream. Background Technology

[0002] In video coding technology, quantization is a core step in balancing coding compression efficiency and image quality. It directly determines the final coding bitrate and the decoded image quality by compressing the frequency domain coefficients of image blocks. To ensure quantization effectiveness, rate-distortion optimized quantization (RDOQ) is commonly used in related technologies. This scheme requires iteratively and exponentially searching for all possible quantization coefficient adjustment values ​​and calculating the rate-distortion cost for each value to determine the optimal solution.

[0003] However, the exponential traversal characteristic of traditional RDOQ schemes leads to extremely high computational complexity. The computational load increases exponentially with the number of non-zero coefficients within an image patch, resulting not only in enormous computational consumption but also significant processing latency. This makes traditional schemes unsuitable for real-time encoding requirements in scenarios with limited computing power, such as mobile and edge computing. This creates a core contradiction: pursuing high compression efficiency requires sacrificing real-time performance, while ensuring real-time performance necessitates compromising compression quality. The inability to achieve a precise balance between compression efficiency and computational complexity has become a key bottleneck restricting the practical application of real-time video coding technology. Summary of the Invention

[0004] This application provides a method for adjusting the quantization coefficients of a frame and a method for encoding a video stream, in order to solve the technical problem in related technologies that it is difficult to balance compression efficiency and computational complexity.

[0005] In a first aspect, embodiments of this application provide a method for adjusting the quantization coefficients of a frame, comprising: acquiring a target image patch corresponding to a target frame; determining model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: a patch size parameter, and other parameters; retrieving a target model corresponding to the model selection parameters from a plurality of candidate models, wherein the plurality of candidate models are obtained by training corresponding initial models, and the plurality of initial models correspond to different model architectures; determining input parameters corresponding to the target model, wherein the input parameters include target patch coefficients of the target image patch, a target quantization parameter, and an initial quantization coefficient, the target patch coefficients including a target amplitude coefficient and a target sign coefficient, the target amplitude coefficient representing the signal intensity of the image patch, the target sign coefficient representing the signal direction of the image patch, and the initial quantization coefficients being obtained by performing a compression operation on the target patch coefficients based on the target quantization parameter; inputting the input parameters into the target model to obtain a predicted adjustment value for the target image patch; and adjusting the initial quantization coefficients based on the predicted adjustment value to obtain the target quantization coefficients.

[0006] Secondly, embodiments of this application provide a video stream encoding method, comprising: acquiring an initial video stream, wherein the initial video stream includes multiple video frames, the multiple video frames including a target frame; determining a target image block corresponding to the target frame; determining model selection parameters corresponding to the target image block, wherein the model selection parameters include at least one of the following: a block size parameter, and other parameters; retrieving a target model corresponding to the model selection parameters from multiple candidate models, wherein the multiple candidate models are obtained by training corresponding initial models, and the multiple initial models correspond to different model architectures; determining input parameters corresponding to the target model, wherein the... The input parameters include target block coefficients of the target image patch, target quantization parameters, and initial quantization coefficients. The target block coefficients include target amplitude coefficients and target sign coefficients. The target amplitude coefficients represent the signal strength of the image patch, and the target sign coefficients represent the signal direction of the image patch. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. The input parameters are input to the target model to obtain the prediction adjustment value of the target image patch. The initial quantization coefficients are adjusted based on the prediction adjustment value to obtain the target quantization coefficients. An encoding operation is performed based on the target quantization coefficients to obtain the target video stream.

[0007] Thirdly, embodiments of this application provide a frame quantization coefficient adjustment device, including: a first acquisition module, used to acquire a target image block corresponding to a target frame; A first determining module is used to determine model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: patch size parameters, other parameters; a first retrieving module is used to retrieve a target model corresponding to the model selection parameters from multiple candidate models, wherein the multiple candidate models are obtained by training corresponding initial models, and the multiple initial models correspond to different model architectures; a second determining module is used to determine input parameters corresponding to the target model, wherein the input parameters include target patch coefficients, target quantization parameters, and initial quantization coefficients of the target image patch, wherein the target patch coefficients include target amplitude coefficients and target sign coefficients, wherein the target amplitude coefficients represent the signal intensity of the image patch, the target sign coefficients represent the signal direction of the image patch, and the initial quantization coefficients are obtained by performing a compression operation on the target patch coefficients based on the target quantization parameters; a third determining module is used to input the input parameters into the target model to obtain a predicted adjustment value for the target image patch; a first adjustment module is used to adjust the initial quantization coefficients based on the predicted adjustment value to obtain target quantization coefficients.

[0008] Fourthly, embodiments of this application provide a video stream encoding apparatus, comprising: a second acquisition module for acquiring an initial video stream, wherein the initial video stream includes multiple video frames, and the multiple video frames include a target frame; a fourth determination module for determining a target image block corresponding to the target frame; a fifth determination module for determining model selection parameters corresponding to the target image block, wherein the model selection parameters include at least one of the following: block size parameters, and other parameters; a second retrieval module for retrieving a target model corresponding to the model selection parameters from multiple candidate models, wherein the multiple candidate models are obtained by training corresponding initial models, and the multiple initial models correspond to different model architectures; and a sixth determination module for determining a target model corresponding to the target model. The input parameters include target block coefficients, target quantization parameters, and initial quantization coefficients for the target image patch. The target block coefficients include target amplitude coefficients and target sign coefficients. The target amplitude coefficients represent the signal strength of the image patch, and the target sign coefficients represent the signal direction of the image patch. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. A seventh determining module is used to input the input parameters into the target model to obtain the predicted adjustment value of the target image patch. A second adjusting module is used to adjust the initial quantization coefficients based on the predicted adjustment value to obtain the target quantization coefficients. An eighth determining module is used to perform an encoding operation based on the target quantization coefficients to obtain the target video stream.

[0009] Fifthly, embodiments of this application provide a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the method described in any of the above-mentioned embodiments.

[0010] Sixthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processing component, implement the method described in any of the above-mentioned cases.

[0011] In this embodiment, a target image block corresponding to a target frame is obtained; model selection parameters corresponding to the target image block are determined, wherein the model selection parameters include at least one of the following: block size parameters, other parameters; a target model corresponding to the model selection parameters is retrieved from multiple candidate models, wherein the multiple candidate models are obtained by training corresponding initial models, and the multiple initial models correspond to different model architectures; input parameters corresponding to the target model are determined, wherein the input parameters include target block coefficients of the target image block, target quantization parameters, and initial quantization coefficients, the target block coefficients include target amplitude coefficients and target sign coefficients, the target amplitude coefficients represent the signal intensity of the image block, the target sign coefficients represent the signal direction of the image block, and the initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters; the input parameters are input to the target model to obtain the prediction adjustment value of the target image block; the initial quantization coefficients are adjusted according to the prediction adjustment value to obtain the target quantization coefficients. In this embodiment of the invention, a multi-architecture candidate model adaptation and one-time prediction quantization adjustment value method is adopted. By selecting the matching target model based on the model selection parameters of the target image block, the input parameters are input into the target model, and the optimal prediction adjustment value is directly output to optimize the target quantization coefficient. This replaces the exponential traversal search of traditional iterative rate-distortion optimization quantization, achieving the goal of reducing computational complexity while ensuring compression efficiency, i.e., rate-distortion performance, and adapting to real-time hardware encoding. This achieves a precise balance between compression efficiency and real-time encoding requirements, with controllable computational complexity and improved hardware adaptability. In turn, it solves the technical problem in related technologies that it is difficult to balance compression efficiency and computational complexity.

[0012] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a frame quantization coefficient adjustment method provided in this application is shown; Figure 2 A flowchart of a video stream encoding method provided in this application is shown; Figure 3 A schematic diagram of the architecture of the U-Net neural network model for size quantization adjustment prediction of 16x16 blocks and 32x32 blocks provided by an optional embodiment of the present invention; Figure 4 This is a schematic diagram of the architecture of a fully convolutional neural network (DCNN) for 4x4 and 8x8 block size quantization adjustment prediction provided by an optional embodiment of the present invention. Figure 5 This is a schematic diagram of the training process of the DNN-RDOQ model provided in an optional embodiment of the present invention; Figure 6 This is a schematic diagram of the multi-model DNN-RDOQ framework provided by an optional embodiment of the present invention integrated into the AV1 / libaom encoder pipeline; Figure 7 This is a schematic diagram of the process of porting the Rate Distortion Optimized Quantization (RDOQ) algorithm component in the HEVC standard to the libaom encoder pipeline provided by an optional embodiment of the present invention. Figure 8 A schematic diagram of the structure of a frame quantization coefficient adjustment device provided in this application is shown; Figure 9 A schematic diagram of the structure of a video stream encoding device provided in this application is shown. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0017] For ease of reference, some terms used in this description are defined as follows. The terms presented and their respective definitions are not strictly limited to these definitions—a term may be further defined by its use in this disclosure. The term “example” as used herein means used as an example, instance, or illustration. Any aspect or design described herein as “exemplary” should not necessarily be construed as superior to other aspects or designs. Rather, the term “exemplary” is used to present the concept in a concrete manner. In this application and the appended claims, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise stated or clearly apparent from the context, “X uses A or B” is intended to mean any natural inclusive arrangement. That is, “X uses A or B” is satisfied if X employs A, X employs B, or X employs both A and B. As used herein, at least one of A or B means at least one of A, or at least one of B, or at least one of A and B. In other words, this phrase is disjunctive. The articles “a” and “an” used in this application and in the appended claims should generally be interpreted as “one or more”, unless otherwise stated or clearly apparent from the context that they are in the singular form.

[0018] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] Figure 1 A flowchart of a frame quantization coefficient adjustment method provided in this application is shown, as follows: Figure 1 As shown, the method may include the following steps: 101. Obtain the target image block corresponding to the target frame; The target frame refers to the specific video frame that needs to be quantized and optimized. It is the frame-level carrier for all subsequent block-level processing and clarifies the scope and boundaries of the overall processing.

[0020] Among them, the target image block refers to the basic processing unit obtained by splitting the target frame into preset sizes, such as 4×4, 8×8, 16×16, etc. It is the core object of block-level optimization in video coding, such as quantization, transformation, and model inference, which can make subsequent processing more focused on local pixel features and improve processing accuracy.

[0021] This step locates and extracts the core processing objects to be optimized from the initial video stream. Breaking the complete target frame into image blocks decomposes the complex frame-level optimization task into simpler block-level tasks, reducing the complexity of single-step processing and adapting to the processing needs of subsequent models for blocks of different sizes. This lays the foundation for accurate model selection and quantization optimization in the future.

[0022] 102. Determine the model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: patch size parameter, other parameters; Among them, the model selection parameters refer to the set of core parameters used to select the appropriate model from multiple candidate models. They serve as a bridge to establish the matching relationship between image patch features and model architecture, ensuring that the model called subsequently can accurately adapt to the processing requirements of the current patch.

[0023] Among them, the block size parameter refers to the specific size information of the target image block, which is one of the core bases for model selection. Different architectures have different processing accuracy and efficiency for image blocks of different sizes. This parameter can directly match the size adaptation capability of the model.

[0024] Other parameters refer to supplementary parameters that affect model adaptability, in addition to block size. Specifically, they may include the frame type of the frame to which the target image block belongs, such as intra-frame / inter-frame, and the corresponding transform mode, such as Discrete Cosine Transform (DCT) and Adaptive Discrete Sine Transform (ADST). These parameters can further refine the processing scenario of the image block and improve the comprehensiveness and accuracy of model selection.

[0025] This step extracts key parameters related to the characteristics of the target image patch, providing a basis for subsequent model selection. For the target image patch obtained in step 101, its core characteristics are analyzed, and patch size parameters and other supplementary parameters are extracted to form a complete set of model selection parameters.

[0026] This step, by accurately extracting model selection parameters, avoids blindly calling candidate models and ensures that the subsequently called models are highly compatible with the characteristics of the current target image patch. For example, large image patches are matched with models that have efficient global feature extraction capabilities, and image patches corresponding to frames within a frame are matched with models that are compatible with spatial prediction features. This provides a guarantee for the accuracy and efficiency of subsequent model inference, while reducing invalid inference caused by model mismatch and lowering the overall computational cost.

[0027] 103. From multiple candidate models, retrieve the target model corresponding to the model selection parameters. The multiple candidate models are obtained by training the corresponding initial models, and the multiple initial models correspond to different model architectures. Among them, multiple candidate models refer to models formed by targeted training of multiple initial models with different architectures. Each model is adapted to specific image patch characteristics, such as specific size and specific frame type, providing diverse adaptation schemes for model selection. The initial model refers to the basic neural network model that has not been trained. Its different model architectures, such as lightweight multilayer perceptron (MLP) and deep separable convolutional network (Depthwise Conv), determine the model's inference efficiency, feature extraction capability, and adaptability to different scenarios. The original intention of different architectures is to match the processing needs of different image patches.

[0028] The target model refers to the model selected from multiple candidate models that is perfectly compatible with the model selection parameters determined in step 102. It is the core tool for generating the predicted adjusted values, and its compatibility directly determines the accuracy of the predicted adjusted values.

[0029] Among them, model architecture refers to the layer structure design of neural network models, such as the number of hidden layers, the number of neurons, and the type of convolution kernel. It is the carrier of the core capabilities of the model. Models with different architectures have differences in computational complexity and feature extraction accuracy, and can be adapted to different needs such as real-time encoding and high-precision optimization.

[0030] This step completes accurate model matching and retrieval based on the model selection parameters. Using the model selection parameters determined in step 102, the target model that best matches the parameter characteristics is selected from multiple candidate models and retrieved.

[0031] This step, through parameter matching and architecture adaptation model selection logic, ensures that the selected target model can maximize its advantages. For example, a lightweight model is selected for small image patches to balance accuracy and inference speed; a high-precision model is selected for complex scene image patches to ensure optimization effect. At the same time, multiple candidate models with different architectures cover diverse processing scenarios, avoiding the limitation of a single model adapting to all scenarios. This not only improves the accuracy of predicted adjustment values, but also controls the computational complexity according to scenario requirements, laying the foundation for balancing compression efficiency and real-time performance in the future.

[0032] 104. Determine the input parameters corresponding to the target model. The input parameters include the target block coefficients of the target image block, the target quantization parameters and the initial quantization coefficients. The target block coefficients include the target amplitude coefficients and the target sign coefficients. The target amplitude coefficients represent the signal strength of the image block and the target sign coefficients represent the signal direction of the image block. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. Among them, the input parameters refer to the set of core features extracted from the target image patch and the encoding parameters to meet the inference requirements of the target model. They are the basis for model learning and outputting predicted adjustment values, and their completeness and accuracy directly determine the effect of model inference.

[0033] Among them, the target block coefficients, also known as the transform coefficients corresponding to the target image block, are the core data reflecting the signal characteristics of the image block. Optionally, they can refer to the frequency domain residual coefficients obtained after the target image block has been transformed. Compared with the spatial domain pixel values, the frequency domain coefficients can more accurately reflect the energy distribution of the image block and provide effective features for the model to learn the quantization optimization rules.

[0034] Among them, the target quantization parameter, also known as the target QP, refers to the encoding control parameter used to control the degree of quantization. Its value determines the size of the quantization step size. That is, the larger the value, the coarser the quantization, the higher the compression ratio but the worse the image quality; the smaller the value, the better. It is the core basis for generating the initial quantization coefficients and also an important reference for the model to judge the compression requirements.

[0035] The initial quantization coefficient refers to the coefficient obtained after performing quantization operation on the target block coefficient based on the target quantization parameter. It serves as the benchmark coefficient for subsequent optimization and provides a clear correction target for the predicted adjustment value.

[0036] Among them, the target amplitude coefficient refers to the absolute value of the target block coefficient, which is used to characterize the energy intensity of the frequency domain signal of the image block. It enables the model to accurately perceive the signal importance at different positions of the image block, that is, the larger the amplitude, the greater the impact of the signal on the image quality. Among them, the target sign coefficient refers to the positive or negative sign of the target block coefficient, which is used to characterize the direction of the frequency domain signal of the image block. It fully preserves the original features of the frequency domain coefficient and avoids model learning bias caused by missing sign information.

[0037] This step extracts all the input features required by the target model, clarifying the source and meaning of each input parameter. First, target block coefficients are extracted from the frequency domain features of the target image patch, and amplitude and sign coefficients are separated. Then, the target quantization parameters used to generate the initial quantization coefficients are obtained. Finally, the initial quantization coefficients obtained through the quantization operation are determined, forming a complete set of input parameters.

[0038] Complete input parameters provide comprehensive feature support for the target model. The amplitude and sign features of the target block coefficients fully reflect the signal characteristics of the image block. The target quantization parameters clarify the current compression requirements. The initial quantization coefficients provide an optimization benchmark. The combination of these three allows the model to accurately learn the mapping relationship between signal features, compression requirements, and quantization optimization, avoiding deviations in the predicted adjustment values ​​due to missing features, and ensuring the generation of the optimal predicted adjustment values ​​in the future.

[0039] 105. Input the input parameters into the target model to obtain the predicted adjustment value of the target image patch; The predicted adjustment value refers to the adjustment amount obtained by the target model based on the input parameters and used to correct the initial quantization coefficients. Optionally, the typical value is 0 or -1. It is the core output that replaces the traditional iterative RDOQ in traversing and searching for adjustment values. It can directly guide the optimization of the initial quantization coefficients without exponential traversal calculations.

[0040] This step quickly generates the optimization direction and adjustment amount of the initial quantization coefficients. The input parameters are then fed into the target model, which can directly output predicted adjustment values ​​adapted to the current target image patch.

[0041] This step, compared to the traditional RDOQ approach which iterates through all possible adjustment values ​​and calculates rate-distortion costs, directly obtains the predicted adjustment value through a single model inference. This reduces the computational complexity from exponential (O(2^N), where N is the number of non-zero coefficients within the block) to linear (O(1)), significantly reducing computational overhead and improving the efficiency of quantization optimization. Simultaneously, relying on the accurate training of the target model, the output predicted adjustment value closely matches the rate-distortion optimization criteria, ensuring that the subsequently optimized quantization coefficients possess excellent compression efficiency. This achieves a balance between efficient inference and accurate optimization, providing crucial support for adapting to real-time coding requirements.

[0042] 106. Adjust the initial quantization coefficients based on the predicted adjustment values ​​to obtain the target quantization coefficients.

[0043] Among them, the target quantization coefficient refers to the final quantization coefficient obtained after the initial quantization coefficient is corrected by the prediction adjustment value. It is the core coding data that takes into account both compression efficiency and image quality. It will be used as the input of entropy coding in the subsequent coding process, which directly affects the final coding bitrate and the degree of image distortion.

[0044] This step, based on the predicted adjustment values ​​output by the model, accurately optimizes the initial quantization coefficients. Using the predicted adjustment values ​​as the basis for correction, the initial quantization coefficients are adjusted coefficient by coefficient to ultimately obtain the optimized target quantization coefficients.

[0045] This step, through precise adjustments, ensures that the corrected target quantization coefficients minimize the encoding bitrate and improve compression efficiency while maintaining image quality (e.g., low distortion), thus solving the problem of balancing compression rate and image quality inherent in traditional quantization coefficients. Furthermore, compared to the iterative adjustment process of traditional RDOQ, this step is simpler and more direct, requiring no additional rate-distortion calculations, further reducing computational complexity and improving encoding efficiency. Moreover, as core data in subsequent encoding processes, the target quantization coefficients' superior rate-distortion performance directly enhances the overall video encoding compression effect, while the efficient adjustment process makes the encoding workflow more adaptable to the demands of real-time hardware encoding.

[0046] Through the above steps, the target image patch corresponding to the target frame is obtained; the model selection parameters corresponding to the target image patch are determined, wherein the model selection parameters include at least one of the following: patch size parameters, other parameters; from multiple candidate models, the target model corresponding to the model selection parameters is retrieved, wherein the multiple candidate models are obtained by training the corresponding initial models, and the multiple initial models correspond to different model architectures; the input parameters corresponding to the target model are determined, wherein the input parameters include the target block coefficients of the target image patch, the target quantization parameters, and the initial quantization coefficients, the target block coefficients include the target amplitude coefficients and the target sign coefficients, the target amplitude coefficients represent the signal intensity of the image patch, the target sign coefficients represent the signal direction of the image patch, and the initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters; the input parameters are input into the target model to obtain the prediction adjustment value of the target image patch; the initial quantization coefficients are adjusted according to the prediction adjustment value to obtain the target quantization coefficients. In this embodiment of the invention, a multi-architecture candidate model adaptation and one-time prediction quantization adjustment value method is adopted. By selecting the matching target model based on the model selection parameters of the target image block, the input parameters are input into the target model, and the optimal prediction adjustment value is directly output to optimize the target quantization coefficient. This replaces the exponential traversal search of traditional iterative rate-distortion optimization quantization, achieving the goal of reducing computational complexity while ensuring compression efficiency, i.e., rate-distortion performance, and adapting to real-time hardware encoding. This achieves a precise balance between compression efficiency and real-time encoding requirements, with controllable computational complexity and improved hardware adaptability. In turn, it solves the technical problem in related technologies that it is difficult to balance compression efficiency and computational complexity.

[0047] As an optional embodiment, before retrieving the target model corresponding to the model selection parameters from multiple candidate models, the method includes: determining multiple candidate block sizes when the model selection parameters include block size parameters; determining architecture selection parameters corresponding to the multiple candidate block sizes, wherein the architecture selection parameters include at least one of the following: block size characteristic parameters, computational efficiency parameters, and quality-complexity balance parameters; determining initial models corresponding to the multiple candidate block sizes based on the corresponding architecture selection parameters; and training the corresponding initial models using the corresponding sample data to obtain multiple candidate models.

[0048] Among them, candidate block size refers to a variety of image block size options preset to adapt to different image block processing scenarios, covering diverse needs from small-sized detail-dense blocks to large-sized global feature blocks, providing clear size adaptation targets for subsequent targeted model architecture design.

[0049] Among them, the architecture selection parameters refer to the set of core decision-making criteria used to determine the initial model architecture that matches a specific candidate block size. It is a key bridge connecting block size characteristics and model architecture design, ensuring that the initial model architecture can accurately adapt to the processing requirements of the corresponding block size and avoiding blindness in architecture design.

[0050] Among them, the block size characteristic parameter refers to the parameter that characterizes the inherent properties of different candidate block sizes. Specifically, it may include pixel correlation within the block, signal energy distribution characteristics, detail density, etc. For example, small-sized blocks have strong pixel correlation and high detail density, while large-sized blocks have significant global features and more dispersed energy distribution, providing a direct basis for the selection of the layer structure and feature extraction method of the model architecture.

[0051] Among them, computational efficiency parameters refer to parameters that characterize the performance of the model inference process, such as computing power consumption, inference speed, and memory usage. They are directly related to the real-time encoding requirements of hardware, such as the computing power limit of mobile hardware and the latency requirements of real-time transmission, and provide constraints for the lightweight design of model architecture and operator selection.

[0052] Among them, the quality and complexity balance parameter refers to the parameter used to balance the quality of model output quantization optimization, such as rate-distortion performance and model computational complexity. Its core is to control the computational cost of the model while ensuring that the compression quality does not decrease significantly, so as to avoid extreme situations such as optimal quality but excessive complexity that prevents real-time operation or excessively low complexity but substandard quality.

[0053] Among them, sample data refers to the training dataset that matches a specific candidate block size and encoding scenario. It includes core data such as the transformation coefficients, quantization parameters, initial quantization coefficients, optimal quantization adjustment values, and corresponding rate-distortion cost index of the image block at that size. Its specificity directly determines the effect and generalization ability of model training.

[0054] In this implementation, multiple candidate models precisely adapted to various candidate block sizes are constructed, providing a high-quality model foundation for subsequent selection of the target model based on model selection parameters. First, based on the block-level processing requirements of video coding, multiple candidate block sizes are determined, clarifying the size range suitable for the model. For each candidate block size, its block size characteristic parameters are analyzed, and combined with the computational efficiency requirements of hardware coding and the goal of balancing quality and complexity, corresponding architecture selection parameters are extracted. Based on the architecture selection parameters, a matching initial model architecture is designed for each candidate block size. For example, small blocks correspond to a lightweight MLP architecture, balancing inference speed and detailed feature extraction; large blocks correspond to a DepthwiseConv+ global pooling architecture, improving global feature capture capabilities while controlling complexity, ensuring that the initial model architecture is highly compatible with the block size characteristics and coding requirements. Sample data corresponding to each candidate block size is collected and organized, and the corresponding initial model is trained using this sample data, ultimately obtaining multiple candidate models precisely adapted to different candidate block sizes.

[0055] This approach enables precise adaptation between model architecture and block size. By using block size characteristic parameters to guide the initial model architecture design, each candidate model is specifically adapted to the features of blocks of a particular size. This avoids the low specificity problem caused by a single model adapting to all models, thus improving the accuracy of model inference. Furthermore, computational efficiency parameters and quality-complexity balance parameters are incorporated into the architecture selection parameters. Complexity is constrained during the model design phase, ensuring that the trained candidate models not only possess excellent quantization optimization quality but also meet the computational power and latency requirements of real-time hardware encoding. This avoids the risk of a model meeting quality standards but being too complex to be practical. Moreover, training with sample data corresponding to the candidate block size ensures that the quantization optimization rules learned by the model are more closely aligned with the actual encoding scenarios of blocks of that size. Through the design of multiple candidate block sizes and the training of corresponding models, a candidate model set covering the entire size range is formed, adapting to target image blocks of different sizes in the target frame. This provides ample high-quality model support for subsequent precise retrieval of target models based on model selection parameters, further enhancing the overall solution's adaptability to diverse encoding scenarios.

[0056] As an optional embodiment, inputting input parameters into the target model to obtain the predicted adjustment value of the target image patch includes: when the patch size parameter is greater than a first size threshold and the target model is a first model, inputting the input parameters into the encoding layer structure of the first model to obtain multi-scale shrinkage features, wherein the first model further includes a bottleneck layer structure, a decoding layer structure, and a first output layer structure, and the multi-scale shrinkage features represent the signal features of the target image patch at multiple scales after different downsampling processes and spatial size compression; inputting the multi-scale shrinkage features into the bottleneck layer structure to obtain global context features, wherein the global context features are features characterizing the global distribution pattern and cross-location dependency of the target image patch; inputting the global context features into the decoding layer structure to obtain multi-scale expansion features, wherein the multi-scale expansion features represent the signal features of multiple sizes obtained after fusing the global context features and the multi-scale shrinkage features and restoring the spatial size; and inputting the multi-scale expansion features into the first output layer structure to obtain the predicted adjustment value.

[0057] The first size threshold, which is a pre-set critical value (such as 16×16) used to distinguish the size of target image blocks, is the core basis for determining whether to enable the first model. By clarifying the size division standard, large blocks can be targeted to adopt a model architecture adapted to global feature processing.

[0058] The first model refers to a neural network model specifically designed for processing large target image blocks with a size greater than a first size threshold. Its architecture includes an encoding layer structure, a bottleneck layer structure, a decoding layer structure, and a first output layer structure, which can accurately adapt to the global feature distribution and cross-location dependency capture requirements of large blocks.

[0059] The coding layer structure refers to the set of network layers in the first model that are responsible for multi-scale feature extraction and dimensionality compression of the input parameters. It includes convolutional layers and pooling layers of different lengths. The feature space size can be shrunk by progressive downsampling operations while retaining key signal features, laying the foundation for subsequent global feature extraction.

[0060] Among them, multi-scale shrinkage features refer to signal features at multiple scales obtained after the input parameters are downsampled and compressed in spatial size by different coding layer structures. The features at each scale correspond to information at different levels of the target image block, such as coarse-grained global energy distribution and fine-grained local detail features, which can fully cover the signal characteristics of large-size blocks.

[0061] Downsampling refers to the process of reducing the spatial resolution of feature maps in the coding layer structure by adjusting the convolution stride and pooling. This can reduce the amount of feature data, improve computational efficiency, and expand the receptive field of subsequent network layers, helping to capture global features of large blocks.

[0062] Among them, the bottleneck layer structure refers to the core network layer in the first model that connects the encoding layer structure and the decoding layer structure, such as the attention mechanism layer and global pooling. It focuses on mining the global distribution pattern and cross-positional dependency relationship in multi-scale shrinkage features, and can fuse and refine feature information scattered at different scales.

[0063] Global context features refer to the features representing the global distribution patterns and cross-positional dependencies of target image patches after multi-scale shrinkage features are processed by a bottleneck layer structure. This overcomes the limitations of local features and accurately captures the correlation information of coefficients at different positions within large-size patches, such as the energy complementarity between low-frequency and high-frequency coefficients. Global distribution patterns refer to the overall distribution characteristics of all coefficients within a target image patch, not the local attributes of a single coefficient, but rather the statistical and trend-based patterns covering the entire patch. For example, within an image patch, this can manifest as the numerical range distribution of coefficients, regional density distribution, or energy proportion distribution. Cross-positional dependencies refer to the correlation constraints between coefficients at different positions within an image patch. That is, the state of a coefficient at a certain position, such as its amplitude or whether it is zero, will influence and logically relate to the states of coefficients at other positions. For example, within an image patch, this can manifest as the amplitude correlation of coefficients at adjacent positions, the energy complementarity correlation of coefficients in different regions, or the distributional synergy of coefficients at discontinuous positions.

[0064] Among them, cross-location dependency refers to the signal features at different spatial locations in the target image patch, in order to determine the key features that affect the accuracy of quantization optimization.

[0065] The decoding layer structure refers to the set of network layers in the first model that are responsible for multi-scale feature recovery and fusion of global context features, such as transposed convolutional layers and upsampling layers. These layers can gradually recover the feature space size and simultaneously fuse global context features with multi-scale shrinkage features output by the encoding layer across scales.

[0066] Among them, multi-scale extended features refer to signal features of multiple sizes obtained by fusing global context features with multi-scale contraction features and restoring spatial size after global context features are processed by the decoding layer structure. This not only preserves the global distribution pattern but also restores local detail information, achieving accurate fusion of global and local features.

[0067] Among them, restoring spatial size refers to the processing method in the decoding layer structure that improves the spatial resolution of the feature map through operations such as upsampling and transposed convolution. This can restore the feature size compressed by the coding layer to a dimension that matches the input parameters, ensuring that the subsequent output layer can accurately map to the predicted adjustment value of each coefficient.

[0068] The first output layer structure refers to the final network layer in the first model that is responsible for mapping multi-scale extended features to predicted adjustment values ​​of the target image patch, such as a fully connected layer or a sigmoid activation layer. Its output dimension is consistent with the number of coefficients of the target image patch, and it can directly output the adjustment amount (0 or -1) for each coefficient.

[0069] In this embodiment, a precise prediction adjustment value generation process is provided for large target image patches whose size exceeds a first size threshold. First, the size type of the target image patch is determined by the first size threshold. When the patch size parameter is greater than the first size threshold, a first model specifically adapted for large patches is activated to ensure that the model architecture matches the patch size characteristics. The input parameters are then fed into the encoding layer of the first model. The encoding layer gradually shrinks the feature space size through multi-step downsampling, generating multi-scale shrunken features containing information at different scales. This reduces computational complexity while comprehensively capturing the global and local features of large blocks. The multi-scale shrunken features are then fed into the bottleneck layer. The bottleneck layer uses attention mechanisms and global pooling to uncover hidden global distribution patterns and cross-positional dependencies in the features, outputting condensed global context features. The global context features are then fed into the decoding layer. The decoding layer gradually restores the feature space size through upsampling and transposed convolution, while simultaneously fusing the global context features with the multi-scale shrunken features output by the encoding layer across scales, generating multi-scale extended features that take into account both global patterns and local details. Finally, the multi-scale extended features are fed into the first output layer. The output layer uses dimension mapping and activation function processing to directly generate predicted adjustment values ​​that correspond one-to-one with the coefficients of the target image blocks, completing the optimization inference of the quantization adjustment values ​​for large blocks.

[0070] This approach addresses the core challenge of large blocks: complex global feature distribution and significant cross-positional dependencies. Therefore, the first model employs an encoding, bottleneck, and decoding architecture. The encoding layer captures multi-scale features, the bottleneck layer extracts global dependencies, and the decoding layer fuses global and local features, resolving the issue of insufficient feature capture when small-sized models process large blocks. Furthermore, multi-scale shrinkage features comprehensively cover the hierarchical information of large blocks, global context features overcome the limitations of local features, and multi-scale expansion features achieve deep fusion of global and local features. This three-layer progressive feature processing allows the first model to accurately learn the quantization optimization rules of large blocks, resulting in output prediction adjustments that better align with rate-distortion optimization criteria, ensuring high compression efficiency for large blocks. Additionally, the downsampling process in the encoding layer reduces the amount of intermediate feature data, lowering overall computational overhead. The bottleneck layer focuses on core global features, avoiding redundant computation, and the precise upsampling in the decoding layer restores feature size without introducing invalid information, achieving a balance between accurate processing of large blocks and controllable computational efficiency, thus adapting to real-time hardware encoding requirements.

[0071] As an optional embodiment, inputting input parameters into the target model to obtain the predicted adjustment value of the target image patch includes: when the patch size parameter is less than a first size threshold and the target model is a second model, inputting the input parameters into the input layer structure of the second model to obtain initial fusion features, wherein the second model further includes a hidden convolutional layer structure and a second output layer structure; inputting the initial fusion features into the hidden convolutional layer structure to obtain target fusion features, wherein the target fusion features are features representing the fusion of local signal features and compression requirement features, and the compression requirement features are features matching the compression requirements of the target quantization parameters; and inputting the target fusion features into the second output layer structure to obtain the predicted adjustment value.

[0072] The second model refers to a lightweight neural network model specifically designed for processing small target image patches with a size smaller than the first size threshold. Its architecture includes an input layer structure, a hidden convolutional layer structure, and a second output layer structure. It can accurately adapt to the characteristics of dense local features and weak cross-location dependence of small patches, while balancing inference speed and quantization optimization accuracy.

[0073] The input layer structure refers to the first layer of the network in the second model that is responsible for receiving input parameters and performing preliminary feature integration. It can map heterogeneous input parameters such as target block coefficients, target quantization parameters, and initial quantization coefficients into feature vectors of a unified dimension, providing standardized input for subsequent local feature extraction.

[0074] Among them, the initial fusion feature refers to the unified dimensional feature obtained after the input parameters are processed by the input layer structure. It is a preliminary integration of the block signal features and compression requirement features in the input parameters, which eliminates the dimensional differences of heterogeneous parameters and lays the foundation for the extraction of local features of the hidden convolutional layer.

[0075] Among them, the hidden convolutional layer structure refers to the core network layer in the second model that is responsible for extracting local signal features and fusing compression requirements, such as deep convolutional layers with small convolutional kernels and lightweight activation layers. The size of its convolutional kernel is adapted to the local feature range of small blocks, which can efficiently capture the signal correlation between adjacent positions within the block, while avoiding redundant calculations.

[0076] Among them, the target fusion feature refers to the feature obtained after the initial fusion feature is processed by the hidden convolutional layer structure. It is a deep fusion of the local signal features of small blocks and the compression requirement features that adapt to the target quantization parameters. It not only preserves the key information of the detailed signal within the block, but also reflects the compression strength requirements of the current encoding.

[0077] Among them, local signal features refer to the signals at adjacent positions in small-sized target image blocks, such as the correlation characteristics and detailed information of frequency domain coefficients, such as the local energy distribution corresponding to edges and textures within small-sized blocks. These are the core basis for quantization optimization of small-sized blocks, and their extraction accuracy directly affects the accuracy of the predicted adjustment values.

[0078] Among them, compression requirement features refer to features generated based on target quantization parameters that match the current coding compression intensity requirements, such as quantization step size features corresponding to QP values ​​and bitrate control requirement features. These features enable the model to clearly define the current compression target and make the predicted adjustment values ​​conform to the actual coding requirements.

[0079] The second output layer structure refers to the final network layer in the second model that is responsible for mapping the target fusion features to the predicted adjustment values. It has few network parameters and low computational cost, and can quickly complete the dimensional mapping from features to adjustment values, outputting adjustment values ​​that correspond one-to-one with the coefficients of small-sized blocks.

[0080] This optional embodiment provides an efficient and lightweight prediction adjustment value generation process for small-sized target image patches with a size smaller than a first size threshold. The core logic is as follows: In this embodiment, the size type of the target image patch is determined by a first size threshold. When the patch size parameter is smaller than the first size threshold, a second model specifically adapted for small patches is activated to ensure that the model architecture is highly matched with the characteristics and lightweight requirements of small patches. The input parameters are input to the input layer structure of the second model. The input layer converts heterogeneous input parameters into initial fusion features of a unified dimension through dimensional mapping and preliminary integration, eliminating dimensional barriers between parameters and providing standardized input for subsequent local feature extraction. The initial fusion features are input to the hidden convolutional layer structure. The hidden convolutional layer uses small convolutional kernels adapted to small patches to efficiently capture local signal features at adjacent positions within the patch, while incorporating compression requirements to generate target fusion features that combine local details and compression objectives. The target fusion features are input to the second output layer structure. The output layer processes the features through lightweight dimensional mapping and activation functions to quickly convert them into predicted adjustment values ​​that correspond one-to-one with the coefficients of the small patches, completing the quantization adjustment value optimization inference for small patches.

[0081] This approach precisely adapts to the feature processing needs of small blocks. Since the core characteristics of small blocks are dense local features and weak cross-location dependencies, a complex global feature capture architecture is unnecessary. The second model, through a simplified architecture of an input layer, a lightweight hidden convolutional layer, and an output layer, focuses on local signal feature extraction, avoiding wasted computational power. Furthermore, each layer of the second model can adopt a lightweight design, such as simplified dimension mapping in the input layer, small convolutional kernels in the hidden convolutional layers, and simplified parameters in the output layer. This significantly reduces the computational load and memory consumption of model inference, making single-block inference time controllable. Simultaneously, the accurate capture of local features by the hidden convolutional layers and the fusion of features requiring compression ensure the optimization accuracy of the predicted adjustment values, enabling small blocks to maintain excellent rate-distortion performance with low computational overhead. Moreover, the lightweight model architecture has low hardware computational requirements and can be directly deployed on mobile devices, edge devices, and other hardware devices with limited computing power, without relying on high-performance GPUs / NPUs. This adapts to the real-time encoding needs of diverse hardware and expands the application scenario coverage of the overall solution.

[0082] As an optional embodiment, the input parameters are input to the target model to obtain the predicted adjustment value of the target image patch, including: taking the absolute value of the target block coefficients to obtain unsigned block coefficients, where the unsigned block coefficients represent the block coefficients that retain the target amplitude coefficients after removing the sign attribute corresponding to the target sign coefficients; normalizing the unsigned block coefficients and the initial quantization coefficients respectively to obtain normalized block coefficients and normalized quantization coefficients; encoding the target quantization parameters to obtain coded quantization parameters, where the encoding process includes one-hot encoding; and inputting the normalized block coefficients, normalized quantization coefficients, and coded quantization parameters as input parameters to the target model to obtain the predicted adjustment value.

[0083] Among them, unsigned block coefficients refer to the block coefficients obtained after taking the absolute value of the target block coefficients. The core is to remove the positive and negative sign attributes corresponding to the target sign coefficients and retain only the target amplitude coefficients that represent the signal strength. This can eliminate the interference of sign differences on model inference and allow the model to focus more on the amplitude features that are directly related to quantization optimization.

[0084] Normalization refers to the preprocessing operation that maps unsigned block coefficients and initial quantization coefficients to a uniform numerical range, such as [0,1] or [-1,1], according to preset rules, such as Min-Max normalization or Z-Score normalization. This can eliminate scale differences and dimensional effects between different parameters.

[0085] Among them, the normalized block coefficients refer to the coefficients obtained after the unsigned block coefficients have been normalized. The numerical range is uniform, which can avoid the model learning bias caused by the large difference in amplitude coefficients of different image blocks, such as the large amplitude of bright blocks and the small amplitude of dark blocks.

[0086] Among them, the normalized quantization coefficient refers to the coefficient obtained after the initial quantization coefficient is normalized. It is on the same numerical scale as the normalized block coefficient, which allows the model to more accurately capture the correlation between the two and improve the feature fusion effect.

[0087] Encoding processing refers to the preprocessing operation of converting the format of the target quantization parameters. Its core purpose is to convert the discrete quantization parameter values ​​into a feature form that fits the model input, making it easier for the model to understand and learn the mapping relationship between the quantization parameters and the predicted adjustment values.

[0088] One-hot encoding refers to a specific implementation of encoding, which converts discrete target quantization parameters into binary one-hot vectors, thus avoiding ordinal bias caused by discrete values.

[0089] Among them, the encoded quantization parameter refers to the feature vector obtained after encoding the target quantization parameter, such as one-hot encoding. It is a quantization parameter representation form that adapts to the model input, can accurately convey the quantization intensity requirement, and has stronger compatibility with normalized block coefficients and normalized quantization coefficients.

[0090] In this embodiment, the original input parameters are preprocessed to generate standardized input features adapted to the target model, thereby accurately obtaining the predicted adjustment values. First, the target block coefficients are subjected to absolute value processing. Since the sign attribute of the target block coefficients has little impact on the decision of the quantization adjustment value, while the amplitude coefficient is the core factor determining the direction of quantization optimization—that is, the larger the amplitude, the more significant the impact on image quality, requiring more caution during quantization—absolute value processing is used to remove the sign attribute, resulting in unsigned block coefficients that retain only amplitude information, simplifying the model learning task. Then, the unsigned block coefficients and the initial quantization coefficients are normalized. The original unsigned block coefficients exhibit significant scale differences due to variations in the brightness and content complexity of image blocks, and the initial quantization coefficients also exhibit different numerical ranges depending on the target quantization parameters. This scale discrepancy can cause the model to overemphasize large numerical parameters and ignore the effective information of small numerical parameters. Normalization maps both to a unified range, eliminating scale and dimension interference, resulting in normalized block coefficients and normalized quantization coefficients. The target quantization parameters are then encoded. Since these parameters are discrete integers, directly inputting them into the model might lead to misinterpretation of a linear ordinal relationship. One-hot encoding converts them into binary vectors, accurately representing the discrete properties of each quantization parameter and avoiding ordinal bias, resulting in encoded quantization parameters. Finally, the preprocessed normalized block coefficients, normalized quantization coefficients, and encoded quantization parameters are input into the target model. The standardized input features eliminate interference from the original parameters, enabling the model to output accurate predicted adjustment values.

[0091] This approach improves model prediction accuracy. By focusing on the core feature of amplitude through absolute value extraction, invalid interference from sign attributes is eliminated; normalization makes parameters at different scales comparable, avoiding model learning bias; one-hot encoding accurately represents the discrete properties of quantization parameters, aligning with the nonlinearity of quantization step size in actual encoding. The combination of these three elements makes the input parameters more closely match the core logic of quantization optimization, allowing the model to learn more accurate mapping relationships, thereby improving the accuracy of predicted adjustment values. Furthermore, it accelerates model training and inference efficiency. Normalization makes the model's loss function gradient smoother, avoiding gradient explosion or vanishing, and accelerating training convergence; preprocessed input parameters have a uniform scale and less redundant information, eliminating the need for the model to expend additional computing power to process invalid features, reducing computational overhead during the inference stage, and adapting to real-time encoding requirements.

[0092] As an optional embodiment, adjusting the initial quantization coefficient based on the predicted adjustment value to obtain the target quantization coefficient includes: performing an addition calculation on the initial quantization coefficient and the predicted adjustment value to obtain an unsigned quantization coefficient; and performing element-wise multiplication on the unsigned quantization coefficient and the target signed coefficient to obtain the target quantization coefficient.

[0093] Among them, the addition calculation process refers to the operation of adding the initial quantization coefficient and the predicted adjustment value element by element according to the corresponding position. It is the core means of correcting the initial quantization coefficient with the predicted adjustment value. It is simple and efficient to operate and can quickly realize the optimization and adjustment of the quantization coefficient amplitude.

[0094] Among them, the unsigned quantization coefficient refers to the coefficient obtained by adding the initial quantization coefficient and the predicted adjustment value. Since the predicted adjustment value is derived from the unsigned block coefficient that retains only amplitude information, and the amplitude characteristics of the initial quantization coefficient have been optimized, this coefficient only retains amplitude information and has no positive or negative sign attribute.

[0095] Among them, element-wise multiplication refers to the operation of multiplying the unsigned quantization coefficients with the target sign coefficients one by one according to their corresponding positions. It is a key step in restoring the original signal direction of the unsigned quantization coefficients, which can accurately match the original sign of each coefficient and ensure the integrity of the signal characteristics.

[0096] Among them, the target symbol coefficients refer to the coefficients that are separated from the target block coefficients and represent the positive and negative directions of the original frequency domain signal. They are the core data that preserves the directional attributes of the original signal. When combined with the unsigned quantization coefficients, they can completely restore the characteristics of the optimized frequency domain coefficients.

[0097] In this embodiment, the target quantization coefficients, balancing optimization and signal integrity, are accurately obtained through amplitude correction and sign restoration. First, an addition calculation is performed. Since the predicted adjustment value is derived from the target model based on unsigned block coefficients that retain only amplitude information, its core function is to optimize the amplitude characteristics of the quantization coefficients. Therefore, the predicted adjustment value is added element-wise to the initial quantization coefficients. This addition operation directly corrects the amplitude of the initial quantization coefficients, resulting in unsigned quantization coefficients that retain only the optimized amplitude information. This avoids interference from sign attributes with adjustment accuracy, and the addition operation is simple, efficient, and allows for rapid correction. Next, an element-wise multiplication process is performed. Since the unsigned quantization coefficients only contain optimized amplitude information and lack the directional characteristics of the original signal, directly using them as encoded data would lead to signal distortion. The target sign coefficients, however, retain the original sign attributes of the target block coefficients. Therefore, the two are multiplied element-wise, matching each unsigned quantization coefficient to its corresponding original sign, restoring the complete target quantization coefficients that combine optimized amplitude and original direction. By completing the signal characteristics, it is ensured that the optimized quantization coefficients do not lose the core attributes of the original signal.

[0098] This method achieves precise optimization of quantization coefficients while maintaining signal integrity. Through addition, the amplitude of the initial quantization coefficients is accurately corrected using predicted adjustment values, ensuring the amplitude characteristics align with the distortion optimization criteria. Element-by-element multiplication then restores the original symbols, preventing signal distortion caused by symbol loss. The resulting target quantization coefficients possess excellent compression optimization while fully preserving the core features of the original signal. Furthermore, the computation is highly efficient, adapting to real-time encoding requirements. Both addition and element-by-element multiplication are basic arithmetic operations, requiring no complex iterations or reasoning processes, resulting in low computational complexity and short processing time, significantly reducing the overall computational overhead of quantization optimization.

[0099] As an optional embodiment, before inputting the input parameters into the target model to obtain the predicted adjusted value of the target image patch, the method further includes: determining a target loss function, wherein the target loss function includes a binary cross-entropy loss, which is used to measure the difference between the model's predicted adjusted value and the actual adjusted value; and training the corresponding initial model using the corresponding sample data based on the target loss function to obtain the target model.

[0100] The target loss function, which measures the difference between the predicted adjusted value and the actual adjusted value of the model output, is the core basis for guiding the optimization of the initial model parameters. It can provide a clear optimization direction for model training and ensure the accuracy of the model output after training.

[0101] Among them, binary cross-entropy loss refers to a specific implementation of the target loss function, which is specifically adapted to binary classification outputs such as predicted adjustment values. For example, in scenarios where the value is typically 0 or -1, it can accurately quantify the difference in probability distribution between the model's predicted results and the true labels, thereby optimizing the model's ability to discriminate binary classification tasks.

[0102] Among them, the model predicted adjustment value refers to the adjustment value predicted by the model for the input sample during the training process. It is one of the core inputs for calculating the loss value. The difference between the model prediction adjustment value and the actual adjustment value directly determines the magnitude of the loss value, and thus affects the direction of parameter update.

[0103] The true adjustment value refers to the globally optimal adjustment value, which serves as a standard reference.

[0104] In this embodiment, by determining a suitable loss function and training with sample data, the basic initial model is optimized into a target model with accurate prediction capabilities. First, the target loss function is determined. Since the core output of the model in this scheme is the predicted adjustment value, which is a binary classification result (e.g., outputting 0 or -1), traditional mean squared error loss and other functions are not suitable for binary classification scenarios and cannot accurately measure prediction bias. Therefore, binary cross-entropy loss is chosen as the target loss function. It can accurately capture the deviation between the predicted adjustment value and the true adjustment value by quantifying the difference in probability distribution, providing precise optimization guidance for model training. Next, model training is conducted based on the target loss function. The input features of the sample data are input into the initial model to obtain the predicted adjustment value output by the model. The loss value between the predicted adjustment value and the true adjustment value is calculated using the target loss function to determine the magnitude of the model's prediction bias. Finally, based on the loss value, the network parameters of the initial model are iteratively updated using the backpropagation algorithm; that is, the larger the loss value, the larger the parameter adjustment range, until the loss value converges to a preset threshold, resulting in the target model.

[0105] This approach ensures the accuracy of model predictions. Binary cross-entropy loss, tailored to the characteristics of binary classification tasks, accurately quantifies the difference between predicted and actual adjusted values, avoiding the optimization bias of traditional loss functions in binary classification scenarios and providing precise guidance for model parameter updates. The objective loss function provides a clear optimization direction for model training, avoiding blind updates of model parameters, accelerating loss value convergence, and shortening the training cycle.

[0106] As an optional embodiment, before inputting the input parameters into the target model to obtain the predicted adjustment value of the target image patch, the method further includes: training the initial model using corresponding sample data to obtain an intermediate training model; determining a multi-dimensional validation metric corresponding to the target model, wherein the multi-dimensional validation metric includes at least one of the following: classification accuracy metric, theoretical rate-distortion performance metric, and actual rate-distortion evaluation metric; performing multi-dimensional performance testing on the intermediate training model based on the multi-dimensional validation metric to obtain a model performance evaluation result; and updating the model parameters of the intermediate training model if the model performance evaluation result is unsuccessful, until a target model with a corresponding successful model performance evaluation result is obtained.

[0107] Intermediate training models refer to transitional models obtained after initial training of the initial model using corresponding sample data. Their network parameters have initially learned the quantitative optimization rules in the sample data, but have not undergone comprehensive performance verification, or have been verified but still do not meet the requirements of the evaluation results.

[0108] Among them, multi-dimensional validation metrics refer to a set of metrics used to comprehensively evaluate the performance of intermediate training models from different dimensions. They can overcome the limitations of single-metric evaluation, ensure that the evaluation results cover the core performance requirements of the model, and provide a comprehensive basis for judging whether the model meets the standards.

[0109] Among them, the classification accuracy index refers to the metric that measures the degree of matching between the predicted adjusted value output by the intermediate training model and the actual adjusted value. Specifically, it is the ratio of the number of correctly matched samples to the total number of samples, which directly reflects the accuracy of the model's prediction of the quantized adjusted value.

[0110] Among them, theoretical rate-distortion performance indicators refer to rate-distortion related indicators calculated based on the predicted adjustment values ​​of the model output, such as theoretical bit rate and theoretical distortion. They are used to evaluate the compression efficiency potential of the quantization coefficients after model optimization from a theoretical perspective, and provide a basis for preliminary judgment of the model's rate-distortion performance.

[0111] Among them, the actual rate-distortion evaluation metric refers to the true rate-distortion metric obtained by deploying the intermediate trained model into the actual encoding process and conducting encoding tests. This includes metrics such as the actual encoding bitrate, the PSNR / SSIM distortion of the decoded image, and the BD-Rate value. It accurately reflects the compression efficiency performance of the model in real-world application scenarios. Furthermore, the introduction of the actual rate-distortion evaluation metric allows model testing to directly connect with the real encoding process, enabling the early detection of potential problems in actual deployment. These problems can be addressed through iterative optimization, avoiding the need for redesign after model training due to substandard performance, thus reducing development costs.

[0112] Among them, the model performance evaluation result refers to the comprehensive evaluation conclusion obtained after testing the intermediate training model based on multi-dimensional validation indicators. It is divided into two categories: evaluation pass and evaluation fail, which directly determines whether the intermediate training model needs further optimization.

[0113] Among them, model parameters refer to the learnable parameters in the intermediate training model network layers, such as convolutional kernel weights, fully connected layer coefficients, bias terms, etc. Their values ​​directly determine the model's prediction performance and rate-distortion optimization effect.

[0114] In this embodiment, a closed-loop process of multi-dimensional verification and iterative parameter optimization training ensures that the final target model has reliable performance and is suitable for actual coding requirements. First, an intermediate training model is obtained. Based on sample data from the corresponding scenario, the initial model is initially trained, allowing the model parameters to learn the mapping rules and obtain an intermediate training model with basic prediction capabilities. Then, multi-dimensional verification metrics are determined to avoid the problem that a single metric cannot comprehensively evaluate model performance. For example, looking only at classification accuracy may result in accurate predictions but high actual coding bitrate; looking only at rate-distortion performance may result in high compression efficiency but poor prediction stability. Therefore, a multi-dimensional metric system is selected, including classification accuracy, theoretical rate-distortion performance, and actual rate-distortion evaluation metrics, to achieve comprehensive coverage of model performance. The test results from the three dimensions are combined to determine whether the model has reached a preset performance threshold, such as classification accuracy ≥ 95% and BD-Rate improvement ≤ 0.5%, resulting in a model performance evaluation result of passing or failing the evaluation. If the evaluation result is unsuccessful, it means that the parameters of the intermediate training model have not been optimized properly. Based on the problems exposed in the multi-dimensional test, such as inaccurate prediction of certain block size samples or poor performance in high QP scene rate distortion, the model parameters need to be updated in a targeted manner, such as adjusting the learning rate and increasing the training weight of the corresponding scene samples. Then, training and multi-dimensional testing should be carried out again, and the training should be iterated repeatedly until the model performance evaluation result is successful, and the target model is obtained.

[0115] This approach, through a multi-dimensional validation index system, ensures that the adjusted values ​​of the model output closely match the optimal solution in terms of prediction accuracy, and also ensures that the model meets compression efficiency requirements in terms of theoretical and practical rate-distortion performance. This gives the target model the dual advantages of accurate prediction and efficient compression. Furthermore, the model performance evaluation results can clearly pinpoint the shortcomings of the intermediate training model; for example, low classification accuracy leads to optimization of the prediction logic, and poor practical rate-distortion leads to optimization of feature learning weights. This provides a clear direction for model parameter updates, avoids blindly adjusting parameters, improves the efficiency of iterative optimization, and shortens the development cycle of the target model.

[0116] As an optional embodiment, before obtaining the target image block corresponding to the target frame, the method further includes: obtaining an initial video stream; determining multiple video frames corresponding to the initial video stream; determining predicted image blocks corresponding to the multiple video frames based on the original image blocks corresponding to the multiple video frames respectively, wherein the multiple original image blocks correspond one-to-one with the multiple predicted image blocks, and the predicted image blocks include a first image block and / or a second image block, wherein the first image block is obtained by spatial correlation prediction based on the reference image blocks corresponding to the reconstructed adjacent blocks for intra-frames in the multiple video frames, and the second image block is obtained by motion compensation prediction based on the reference image blocks corresponding to the encoded reference frames for inter-frames in the multiple video frames; determining residual image blocks corresponding to the multiple video frames based on the multiple original image blocks and the multiple predicted image blocks, wherein the residual image blocks are used to represent the pixel information difference data between the corresponding original image blocks and the corresponding predicted image blocks; and determining processed image blocks corresponding to the multiple video frames based on the residual image blocks corresponding to the multiple video frames respectively, wherein the multiple processed image blocks include the target image block.

[0117] The initial video stream refers to the raw video data stream that has not undergone encoding optimization. It contains a continuous sequence of visual frames and related video attribute information. It is the raw data input for the entire encoding optimization process and provides basic data support for subsequent frame-level and block-level processing.

[0118] Among them, a video frame refers to a continuous single image obtained after the initial video stream is split.

[0119] The original image block refers to the original pixel block obtained after splitting each video frame according to a preset size. It is the original object of block-level processing in video encoding and retains the original pixel information of the local video frame.

[0120] Among them, the predicted image patch refers to the image patch predicted based on the correlation features of the original image patch. It is used to replace the redundant pixel information in the original image patch and is the core carrier for reducing video data redundancy. It corresponds one-to-one with the original image patch.

[0121] The first image block refers to the predicted image block generated for the intra-frame in the video frame. Its generation relies on the reference image block of the reconstructed adjacent block and is obtained through spatial correlation prediction, adapting to the spatial pixel correlation features of the intra-frame.

[0122] Intraframes refer to frames in a video sequence that can be encoded independently without relying on other video frames, such as I-frames. The correlation of their pixel information is mainly reflected in the spatial dimension of adjacency, without the need for the temporal dimension of interframe correlation.

[0123] Among them, the reconstructed adjacent blocks refer to the image blocks in the frame that are adjacent to the current original image block and have been encoded and reconstructed. Their pixel information has been verified through preprocessing and can be used as a reliable spatial reference to ensure the accuracy of intra-frame prediction.

[0124] Among them, the reference image block refers to the baseline image block extracted from the reconstructed adjacent blocks or the encoded reference frame, which is used to generate the prediction image block. It is the core reference carrier for prediction calculation, and its reliability directly determines the accuracy of the prediction image block.

[0125] Spatial correlation prediction refers to a prediction calculation method based on the spatial adjacency correlation of pixels within a frame, such as the brightness and chromaticity continuity of adjacent pixels. By reusing the pixel information of adjacent reconstructed blocks, spatial redundancy within the frame is reduced.

[0126] The second image block refers to the predicted image block generated for the inter-frame in the video frame. Its generation relies on the reference image block of the encoded reference frame and is obtained through motion compensation prediction, adapting to the temporal correlation features of the inter-frame.

[0127] Inter-frames refer to frames in a video sequence that require previously encoded frames to complete encoding, such as P-frames and B-frames. The correlation of their pixel information is mainly reflected in the inter-frame correlation in the time dimension, and redundant information from previous frames can be reused.

[0128] Among them, the encoded reference frame refers to the video frame in the video sequence that is located before the current frame and has completed the encoding process. Its pixel information has been standardized and can be used as a time dimension reference benchmark for inter-frame prediction to support the realization of motion compensation prediction.

[0129] Motion compensation prediction refers to a calculation method that generates predicted image patches by detecting the motion trajectory between the current frame and the encoded reference frame and mapping the corresponding region in the reference frame to the current frame. This method can effectively reduce the temporal redundancy between frames.

[0130] Among them, the residual image block refers to the image block obtained by calculating the difference between the original image block and the corresponding predicted image block pixel by pixel. It is the core data that characterizes the difference between the two pixel information, such as the difference data in dimensions such as brightness and chromaticity. By removing redundant information in the original image block, the effective difference information that needs to be encoded is focused.

[0131] Among them, the processed image block refers to the image block obtained after screening or further processing based on the residual image block, which is used for subsequent core processes such as quantization optimization. The screening criteria can be determined in combination with the coding requirements.

[0132] This embodiment describes the preprocessing flow from the original video stream to residual image blocks, and finally to processed image blocks containing the target image blocks. First, the initial video stream is acquired, and multiple video frames are determined. Predicted image blocks corresponding to the original image blocks are generated. For each original image block obtained from the splitting, the type of video frame it belongs to is first distinguished, such as intra-frame / inter-frame. Pixels in intra-frames have spatial adjacency relationships; therefore, spatial correlation prediction is performed based on the reference image blocks of reconstructed adjacent blocks to generate the first image block. Pixels in inter-frames have temporal inter-frame correlations; therefore, motion compensation prediction is performed based on the reference image blocks of the encoded reference frames to generate the second image block. This spatial / temporal correlation reuse maximizes the removal of redundant information from the original image blocks. Then, the difference between each original image block and its corresponding predicted image block is calculated pixel-by-pixel to obtain residual image blocks. The residual image blocks only retain the pixel information difference data between the two, significantly reducing data redundancy. Furthermore, the difference data better reflects the core details and features of the image, making it an efficient processing target for subsequent encoding optimization. Finally, processed image blocks are obtained by filtering or simple processing from the residual image blocks.

[0133] This method, through a preprocessing step-by-step transformation of the original video stream into target image patches required for subsequent processes, bridges the gap between the original data input and the core quantization optimization stage. This significantly reduces data redundancy and improves subsequent processing efficiency. The predicted image patches generated through intra-frame spatial correlation prediction and inter-frame motion compensation prediction accurately reuse the spatial and temporal redundancy information of the original image patches. The residual image patches calculated subsequently only retain the difference data, resulting in a much smaller data volume than the original image patches. This significantly reduces the computational overhead of subsequent model inference and quantization optimization, meeting the efficiency requirements of real-time encoding. Furthermore, it adapts to the encoding requirements of different frame types, enhancing the process's versatility. For different correlation characteristics of intra-frame and inter-frame frames, spatial correlation prediction and motion compensation prediction are used to generate corresponding predicted image patches, ensuring prediction accuracy.

[0134] As an optional embodiment, after adjusting the initial quantization coefficients based on the predicted adjustment value to obtain the target quantization coefficients, the method further includes: decompressing the target quantization coefficients to obtain inverse quantization coefficients; determining inverse residual image blocks based on the inverse quantization coefficients; and determining reference image blocks based on the inverse residual image blocks and the corresponding predicted image blocks.

[0135] Among them, decompression processing refers to the inverse quantization operation performed on the target quantization coefficients in the frequency domain. The core is to restore the compressed target quantization coefficients to inverse quantization coefficients in the frequency domain through a preset inverse quantization matrix and calculation rules, thereby eliminating the signal compression distortion caused by the quantization operation.

[0136] Among them, the inverse quantization coefficient refers to the frequency domain coefficient obtained after the target quantization coefficient is decompressed, that is, after inverse quantization. It is the result of restoring the frequency domain residual information, retaining the core energy characteristics of the residual signal, and providing frequency domain data support for the subsequent generation of inverse residual image blocks. Among them, the inverse residual image block refers to the spatial domain image block obtained after inverse transformation based on the inverse quantization coefficients. It is the restored form of the residual image block and represents the restoration result of the difference information between the original image block and the predicted image block. Its integrity directly affects the accuracy of the subsequent reference image block.

[0137] The reference image block refers to the reconstructed image block obtained by pixel-level fusion of the inverse residual image block and the corresponding predicted image block. It has complete spatial domain pixel information and can be used as the prediction benchmark for subsequent coded frames, especially inter-frame frames, providing a reliable reference for the prediction of subsequent image blocks.

[0138] As an optional embodiment, before inputting the input parameters into the target model to obtain the predicted adjustment value of the target image patch, the method further includes: obtaining an initial training dataset, wherein the initial training dataset includes sample block coefficients of the sample image patch, sample quantization parameters, initial sample quantization coefficients, target sample quantization coefficients, and a predetermined rate-distortion cost index corresponding to the target sample quantization coefficients, and setting the predetermined sample quantization coefficients as the target sample quantization coefficients; updating the initial training dataset to obtain the target training dataset, wherein the update operation includes determining multiple candidate sample adjustment values ​​based on the initial sample quantization coefficients, determining candidate sample quantization coefficients corresponding to the multiple candidate sample adjustment values ​​respectively, determining candidate rate-distortion cost indices corresponding to the multiple candidate sample quantization coefficients respectively, and, if there is a candidate rate-distortion cost index lower than the predetermined rate-distortion cost index, setting the predetermined sample quantization coefficient as the candidate sample quantization coefficient corresponding to the lowest candidate rate-distortion cost index.

[0139] The initial training dataset refers to the original data set used for model training, which includes various core features and label-related data corresponding to sample image patches.

[0140] Among them, sample image patches refer to image patch samples extracted from various video scenes for model training, covering different sizes and content complexities, such as static, dynamic, bright and dark areas, providing a diverse scene adaptation basis for model learning.

[0141] Among them, the sample block coefficients refer to the frequency domain residual coefficients obtained after the sample image block is transformed. They include the sample amplitude coefficients and the sample sign coefficients. They are the core data that characterizes the signal features of the sample image block and provide the basis for the model input features.

[0142] Among them, the sample quantization parameter, namely sample QP, refers to the encoding control parameter used to generate the initial sample quantization coefficients, which characterizes the preset compression strength requirement of the sample image patch.

[0143] The initial sample quantization coefficient refers to the baseline quantization coefficient obtained after performing quantization operations on the sample block coefficients based on the sample quantization parameters. It is the basis for generating subsequent candidate sample quantization coefficients, and its initial state may not have reached the optimal rate-distortion performance.

[0144] Among them, the target sample quantization coefficient refers to the preset quantization coefficient in the initial training dataset, which serves as the model training label. It will be optimized to the optimal quantization coefficient through subsequent update operations, providing a precise target reference for model learning.

[0145] Among them, the predetermined rate-distortion cost index refers to the rate-distortion cost assessment value corresponding to the initially set target sample quantization coefficient. It can comprehensively consider the coding bitrate and image distortion, and is the core indicator for measuring the quality of the quantization coefficient. The lower the value, the better the balance between compression efficiency and image quality.

[0146] Among them, the predetermined sample quantization coefficient refers to the benchmark coefficient that is temporarily set as the target sample quantization coefficient in the initial stage. It is the comparison benchmark for subsequent update operations, and its initial value may be the initial sample quantization coefficient.

[0147] The update operation refers to a series of processes that optimize and correct the quantization coefficients of the target samples in the initial training dataset. The core purpose is to select the optimal quantization coefficients with the lowest rate-distortion cost and improve the label quality of the training dataset.

[0148] Among them, the candidate sample adjustment value refers to the potential adjustment amount designed for the initial sample quantization coefficient. It is the core variable for generating the candidate sample quantization coefficient and covers the possible optimization directions of the quantization coefficient.

[0149] Among them, the candidate sample quantization coefficient refers to the various possible quantization coefficients obtained after the initial sample quantization coefficient and the candidate sample adjustment value are calculated. It is the basis for the subsequent calculation of the candidate rate distortion cost index and provides a variety of choices for screening the optimal quantization coefficient.

[0150] Among them, the candidate rate distortion cost index refers to the rate distortion cost assessment value calculated for each candidate sample quantization coefficient. It is the direct basis for judging the quality of the candidate sample quantization coefficient. By comparing it with the predetermined rate distortion cost index, better quantization coefficients can be selected.

[0151] The target training dataset refers to the final training dataset obtained after the initial training dataset has been optimized through update operations. Its core advantage is that the quantization coefficients of the target samples are all the optimal quantization coefficients of the corresponding samples.

[0152] In this embodiment, a target training dataset with accurate labels is constructed through initial dataset acquisition and label optimization and update. First, the initial training dataset is acquired and a predetermined sample quantization coefficient is set. Then, based on the initial sample quantization coefficient, all possible candidate sample adjustment values ​​are generated. Each candidate sample adjustment value is combined with the initial sample quantization coefficient through calculation to obtain the corresponding candidate sample quantization coefficient. Subsequently, for each candidate sample quantization coefficient, the corresponding candidate rate-distortion cost index is calculated by combining the encoding bitrate and image distortion, providing sufficient comparison samples for selecting the optimal label. Then, all candidate rate-distortion cost indices are compared with the predetermined rate-distortion cost index to determine if there is a candidate rate-distortion cost index with a lower value. If so, it indicates the existence of a quantization coefficient better than the current predetermined sample quantization coefficient. At this time, the predetermined sample quantization coefficient is updated to the candidate sample quantization coefficient corresponding to the lowest candidate rate-distortion cost index, and this is used as the new target sample quantization coefficient. The above candidate generation and comparison update process is repeated until no better candidate exists. Finally, after a full sample update, the initial training dataset becomes the target training dataset, ensuring that the target quantization coefficient of each sample is the optimal solution in the current scene through traversal optimization, eliminating the local optima problem that may exist in the initial label.

[0153] This approach improves the quality of training dataset labels, ensuring model learning accuracy. The preset target sample quantization coefficients in the initial training dataset may not be optimal. By iterating through candidate adjustment values ​​through update operations, the optimal quantization coefficient with the lowest rate-distortion cost is selected as the label, ensuring the target training dataset labels possess global optimality. The model, trained on these high-quality labels, accurately learns the mapping relationship between input features and the optimal quantization adjustment, avoiding prediction distortion caused by label bias. Furthermore, through such training, the output prediction adjustment values ​​naturally guide towards quantization coefficients with lower rate-distortion costs, fundamentally improving the rate-distortion performance of subsequent quantization optimizations and addressing the problem of traditional models focusing only on prediction accuracy while neglecting actual encoding performance.

[0154] As an optional embodiment, before retrieving the target model corresponding to the model selection parameters from multiple candidate models, the method further includes: obtaining multiple initial models; determining initial shared layers corresponding to the multiple initial models respectively; training the initial shared layers using the corresponding shared sample data to obtain the target shared layer; obtaining intermediate adjustment models corresponding to the multiple initial models based on the shared layers; and training the corresponding intermediate adjustment models using the corresponding sample data to obtain multiple candidate models.

[0155] The initial shared layer refers to the basic network layer with the same structure and common function in multiple initial models. It is used to extract common features of various input parameters, which can realize the reuse of feature extraction logic and avoid repeated development and training.

[0156] Among them, shared sample data refers to a general training dataset that covers multiple encoding scenarios, different block sizes, different quantization parameters, and different content image blocks. It contains input features and corresponding labels under various scenarios, providing diverse data support for the initial shared layer to learn general features.

[0157] Among them, the target sharing layer refers to the network layer with stable and general feature extraction capabilities obtained after the initial sharing layer is trained with shared sample data. Its parameters have been optimized to adapt to the common features of multiple scenarios and can be directly integrated into each initial model for reuse.

[0158] The intermediate adjustment model refers to the transitional model obtained by integrating the target shared layer into the corresponding initial model. It retains the exclusive architecture of the initial model while incorporating the trained shared layer, thus possessing both general feature extraction capabilities and scene adaptation potential.

[0159] In this embodiment, a transfer learning approach combining unified training of a shared layer and targeted training of a dedicated layer efficiently constructs multiple candidate models adaptable to various scenarios. First, multiple initial models are acquired, and their common functional requirements are analyzed to determine the initial shared layer shared by all models. Shared sample data covering multiple scenarios is input into the initial shared layer, and the parameters within the layer are optimized through backpropagation, allowing the shared layer to learn the common features of the input parameters under different scenarios. Once the performance of the shared layer on the general feature extraction task is stable, the target shared layer is obtained. This allows for single training and reuse of multiple models, avoiding repeated training of the basic feature extraction layer for each initial model and significantly reducing overall training overhead. The trained target shared layer is then integrated into the corresponding multiple initial models, replacing the original initial shared layer to obtain intermediate adjustment models. Subsequently, corresponding specialized sample data is matched to each intermediate adjustment model, and the dedicated architecture parameters of the intermediate adjustment model are optimized using this specialized sample data, allowing the model to accurately adapt to the processing requirements of specific scenarios. Once the performance of each model in its dedicated scenario reaches the preset requirements, multiple candidate models are obtained.

[0160] This approach significantly reduces training computational overhead and improves model building efficiency. The initial shared layer only needs to be trained once with shared sample data to obtain the target shared layer, eliminating the need to train a separate basic feature extraction layer for each initial model and avoiding a large amount of repetitive computation. Subsequent intermediate adjustment models only need to optimize specific parameters, significantly reducing the training workload and greatly shortening the overall construction cycle of multiple candidate models, thus meeting the needs of efficient development. The target shared layer is trained based on shared sample data from multiple scenarios, learning the common features of different scenarios. Integrating these features into each model allows each candidate model to adapt to its specific scenario while possessing the generalization ability to cope with variations in similar scenarios, reducing the risk of overfitting. The targeted training phase of the intermediate adjustment models optimizes specific architecture parameters with specialized sample data, enabling each candidate model to accurately capture the characteristics of its specific scenario and ensuring the accuracy of the predicted adjustment values.

[0161] As an optional embodiment, before determining the input parameters corresponding to the target model, the method includes: if the target block coefficients are of spatial domain type, transforming the target block coefficients of spatial domain type to obtain the target block coefficients of frequency domain type.

[0162] Among them, the spatial domain type refers to the feature representation type of the target block coefficients, which is based on the pixel spatial location. Under this type, the coefficients directly reflect the pixel distribution information of the image block, and the correlation between pixels is strong. It is the original feature form of the image block and provides the basic object for subsequent transformation processing.

[0163] Among them, the frequency domain type refers to the feature representation type of the target block coefficients in terms of frequency components. Under this type, the coefficients are distributed according to the high and low frequency of the signal, and the energy is mainly concentrated in the low frequency coefficients, such as the main outline of the image, which is suitable for the needs of quantization optimization and model feature extraction.

[0164] In this embodiment, the dimensionality transformation from spatial domain coefficients to frequency domain coefficients is completed. Before determining the input parameters of the target model, the representation type of the target block coefficients is first clarified. If it is a spatial domain type, further processing is required because the core problem of spatial domain coefficients is the strong correlation between pixels and the dispersed energy distribution. Directly using them as model input will lead to two key problems: first, there is a lot of redundant information, increasing the difficulty of model learning and computational overhead; second, it cannot accurately reflect the energy importance of the signal, making it difficult for the model to learn effective quantization optimization rules. Therefore, a transformation process is performed to achieve dimensionality transformation. For the target block coefficients of the spatial domain type, the orthogonal transformation algorithm commonly used in coding standards is used for processing. The core characteristic of orthogonal transformation is that it can concentrate the dispersed pixel difference energy in the spatial domain to a few low-frequency coefficients in the frequency domain without losing key information, while weakening the energy of high-frequency coefficients. That is, redundant features in the spatial domain can be transformed into efficient features in the frequency domain through mathematical transformation, resulting in target block coefficients in the frequency domain type. After the transformation process, the spatial domain coefficients are formally converted into frequency domain coefficients, which allows the model to quickly focus on key features and improve learning efficiency.

[0165] This application also provides a video stream encoding method. Figure 2 A flowchart of a video stream encoding method provided in this application is shown, as follows: Figure 2 As shown, the method may include the following steps: 201. Obtain the initial video stream, which includes multiple video frames, including the target frame. 202, Determine the target image block corresponding to the target frame; 203. Determine the model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: patch size parameter, other parameters; 204. From multiple candidate models, retrieve the target model corresponding to the model selection parameters. The multiple candidate models are obtained by training the corresponding initial models, and the multiple initial models correspond to different model architectures. 205. Determine the input parameters corresponding to the target model. The input parameters include the target block coefficients of the target image patch, the target quantization parameters and the initial quantization coefficients. The target block coefficients include the target amplitude coefficients and the target sign coefficients. The target amplitude coefficients represent the signal strength of the image patch and the target sign coefficients represent the signal direction of the image patch. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. 206. Input the input parameters into the target model to obtain the predicted adjustment value of the target image patch; 207. Adjust the initial quantization coefficients based on the predicted adjustment values ​​to obtain the target quantization coefficients; 208. Based on the target quantization coefficient, perform the encoding operation to obtain the target video stream.

[0166] Through the above steps, an initial video stream is obtained, which includes multiple video frames, including a target frame; a target image patch corresponding to the target frame is determined; model selection parameters corresponding to the target image patch are determined, where the model selection parameters include at least one of the following: patch size parameter, other parameters; a target model corresponding to the model selection parameters is retrieved from multiple candidate models, where the multiple candidate models are trained on the corresponding initial models, and the multiple initial models correspond to different model architectures; input parameters corresponding to the target model are determined, where the input parameters include target block coefficients of the target image patch, target quantization parameters, and initial quantization coefficients, where the target block coefficients include target amplitude coefficients and target sign coefficients, the target amplitude coefficients represent the signal strength of the image patch, the target sign coefficients represent the signal direction of the image patch, and the initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters; the input parameters are input to the target model to obtain the predicted adjustment value of the target image patch; the initial quantization coefficients are adjusted according to the predicted adjustment value to obtain the target quantization coefficients; and an encoding operation is performed based on the target quantization coefficients to obtain the target video stream. In this embodiment of the invention, a multi-architecture candidate model adaptation and one-time prediction quantization adjustment value method is adopted. By selecting the matching target model based on the model selection parameters of the target image block, the input parameters are input into the target model, and the optimal prediction adjustment value is directly output to optimize the target quantization coefficient. This replaces the exponential traversal search of traditional iterative rate-distortion optimization quantization, achieving the goal of reducing computational complexity while ensuring compression efficiency, i.e., rate-distortion performance, and adapting to real-time hardware encoding. This achieves a precise balance between compression efficiency and real-time encoding requirements, with controllable computational complexity and improved hardware adaptability. In turn, it solves the technical problem in related technologies that it is difficult to balance compression efficiency and computational complexity.

[0167] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0168] In video coding, quantization is a crucial step in balancing compression efficiency and image quality. It directly determines the final bitrate and decoded image quality by compressing the frequency domain coefficients of image blocks. The Alliance for Open Media (AOM) developed the AV1 video codec, an open-source, royalty-free successor to VP9 and a competitor to HEVC / H.265. AV1 offers significant compression improvements over previous codecs and is widely used in streaming video applications. Unlike proprietary codecs like HEVC, AV1's open nature allows it to be implemented in hardware encoders across various platforms. Quantization is a key component in video coding, reducing precision through scaling and rounding transformations of coefficients to achieve compression. Standard Scalar Quantization (SQ) applies a uniform quantization step size to all coefficients in a block, resulting in low computational cost, but it is not optimal in terms of rate-distortion performance. Rate-Distortion Optimized Quantization (RDOQ) is an advanced technique that optimizes the trade-off between bit rate and visual distortion by selectively adjusting the quantization level of individual transform coefficients. RDOQ has been successfully implemented in HEVC encoders, significantly reducing bit rate (typically by 2-4%) compared to scalar quantization. However, traditional RDOQ algorithms involve iterative, sequential processing and repetitive entropy coding evaluation, making them computationally expensive and unsuitable for real-time hardware encoders. Therefore, its inability to achieve a precise balance between compression efficiency and computational complexity has become a key bottleneck restricting the practical application of real-time video coding technology.

[0169] Furthermore, this application finds that although RDOQ has proven effective in HEVC, directly porting RDOQ to AV1 faces several unique challenges: 1) Different block partitioning: AV1 uses a more flexible block partitioning scheme than HEVC, supporting non-square blocks and a wider range of block sizes (4x4 to 64x64, including rectangular blocks). 2) Multiple transform types: AV1 supports multiple transform types besides DCT, including ADST (Asymmetric Discrete Sine Transform), with different transforms selectable in the horizontal and vertical directions. 3) Different entropy coding: AV1 uses range coding instead of HEVC's CABAC (Context Adaptive Binary Arithmetic Coding), requiring different rate estimation methods. 4) Open-source implementation: The implementation in libaom (AV1's reference implementation) has a significantly different encoder architecture than HEVC's HM reference software. 5) Hardware constraints: Modern video coding hardware increasingly integrates Neural Processing Units (NPUs) to support AI acceleration capabilities, creating opportunities for neural network solutions that were not considered in the design of HEVC.

[0170] In view of this, this application proposes an optional implementation of a method and system for adjusting the quantization coefficients of frames. Optionally, it can be referred to as a method and system for performing rate-distortion optimized quantization in an AV1 video encoder, which uses a deep neural network optimized for hardware implementation. The system adapts the RDOQ technique in HEVC to the AV1 codec by employing a multi-model neural network framework to accommodate the unique characteristics of AV1, including its flexible block partitioning, multiple transform types (DCT, ADST), and range-based entropy coding. This framework utilizes a block-size-specific neural network model integrated into the libaom reference implementation, dynamically selecting different architectures based on the AV1 block size (U-Net for larger blocks, and fully convolutional neural networks for smaller blocks). The neural network predicts the quantization adjustment of the scalar quantization transform coefficients, approximating the computationally expensive iterative RDOQ, while enabling parallel execution on Neural Processing Units (NPUs). This method makes RDOQ suitable for hardware-based real-time AV1 encoders, and is particularly beneficial for the open-source and royalty-free AV1 codecs used in modern video applications. By introducing the advantages of RDOQ into the AV1 encoder, while taking into account AV1's unique encoding structure and transformations, the computational complexity is reduced to achieve real-time encoding. It can also utilize available NPU hardware for parallel processing, maintain compatibility with the open-source libaom implementation, and provide a hardware-friendly solution suitable for embedded systems. Figure 3 The schematic diagram of the architecture of the U-Net neural network model for size quantization adjustment prediction of 16x16 blocks and 32x32 blocks provided by the optional embodiments of the present invention is as follows: Figure 4 This is a schematic diagram of the architecture of a fully convolutional neural network (DCNN) for quantization adjustment prediction of 4x4 and 8x8 block sizes provided by an optional embodiment of the present invention, demonstrating a lightweight design with smaller blocks. Figure 5 This is a schematic diagram of the training process of the DNN-RDOQ model provided by an optional embodiment of the present invention, including data generation from HEVC RDOQ, adaptation to AV1, and training of a neural network with AV1-specific characteristics. Figure 6 This is a schematic diagram of the multi-model DNN-RDOQ framework provided by an optional embodiment of the present invention integrated into the AV1 / libaom encoder pipeline, illustrating the dynamic model selection based on block size and the processing flow through transformation, quantization, neural network inference and entropy coding stages. Figure 7 This is a schematic diagram illustrating the process of porting the Rate-Distortion Optimized Quantization (RDOQ) algorithm component from the HEVC standard to the libaom encoder pipeline, as provided in an optional embodiment of the present invention. Figure 3-7 As shown, the optional embodiments of this application will be described below.

[0171] The core of the method provided by the optional embodiments of the present invention is reflected in the following aspects: (a) AV1-specific adaptation This system is specifically designed for the AV1 coding structure. Its core adaptability includes: adapting to AV1's flexible block partitioning mechanism, supporting multiple transform types (DCT, ADST) used by AV1, being compatible with AV1's range-based entropy coding scheme, and being able to be seamlessly integrated with the libaom encoder pipeline.

[0172] (ii) Multi-model architecture A multi-neural network model architecture is adopted, and each model is optimized for specific block sizes of AV1 (4×4, 8×8, 16×16, 32×32). During the encoding process, the corresponding model can be dynamically matched and called according to the current block size being processed.

[0173] (iii) Block size adaptive complexity A complexity adaptation strategy with block size adaptation is adopted to match different neural network architectures for blocks of different sizes: Specifically, for larger blocks such as 16×16 and 32×32, the U-Net architecture is adopted, and the compression gain of such blocks is sufficient to support the computational overhead of complex models; for smaller blocks such as 4×4 and 8×8, Fully Convolutional Neural Networks (FCNN) are adopted to prioritize computational efficiency.

[0174] (iv) Hardware optimization design The framework employs a hardware-optimized design, specifically tailored for the hardware implementation of Neural Processing Units (NPUs); relying solely on standard arithmetic operations, it can run efficiently on existing NPU hardware without requiring the design of additional dedicated RDOQ circuits.

[0175] The method and process are described below: For each transform coefficient block in the AV1 encoder: S1. Transformation Stage: After completing intra / inter-frame prediction, the residual block is transformed using the transform type (DCT or ADST) selected by AV1; S2. Initial Quantization: Apply the standard scalar quantization algorithm to generate initial quantization coefficients; S3. Model Selection: Based on the size of the current processing block, retrieve a suitable pre-trained neural network model from the model library; S4. Neural Network Inference: The selected model receives the original transform coefficients (absolute values), scalar quantization coefficients, and quantization parameters (QP) as inputs, and outputs the predicted adjustment value (usually 0 or -1) for each coefficient. S5. Adjustment Application: Apply the predicted adjustment value to the scalar quantization coefficient to obtain the final RDOQ optimized quantization coefficient; S6. Sign Recovery: Recovering the signs of coefficients that have been stripped before neural network processing; S7. AV1-specific post-processing: Integrated with AV1-specific encoding functions, specifically including: identifying the position of the last non-zero coefficient, ensuring compatibility with the AV1 coefficient encoding order, and achieving seamless integration with the AV1 entropy encoding system.

[0176] Regarding model training methods, neural network models are trained in the following ways: 1) HEVC RDOQ-based data generation: Training signals are generated based on the mature HEVC RDOQ implementation scheme and adapted to the AV1 coding structure; 2) AV1-specific training data: Training data is constructed using video sequences encoded with libaom to capture the unique AV1 pattern in transform coefficients and quantization decisions; 3) Block-size-specific training: Separate training is conducted for different block sizes to precisely optimize the model performance for each size; 4) QP-specific models: Train dedicated models for different quantization parameters (QPs) to adapt to different compression intensity requirements.

[0177] This invention provides a neural network-based RDOQ synthesis system specifically designed for AV1 video codecs. The system consists of multiple integrated components that work collaboratively to provide a hardware-friendly, high-performance quantization optimization scheme. Details are as follows: (a) AV1 encoder integration This system can be seamlessly integrated into the AV1 encoder pipeline. The integration node is located in the quantization stage, between the transformation module and the entropy coding module.

[0178] Regarding the adaptation to AV1-specific features, the integration of this system fully considers the unique technical characteristics of AV1, specifically including: Flexible block partitioning adaptation: AV1 supports multiple block sizes through a recursive partitioning scheme. This system can handle 4×4, 8×8, 16×16 and 32×32 square blocks - these are the mainstream transform unit sizes in AV1, perfectly adapting to its flexible partitioning requirements; Multi-transformation type compatibility: AV1 can employ different transformation types (DCT, ADST, IDTX) in the horizontal and vertical directions. The neural network model of this system is specially trained to handle the coefficients output by different transformation types; by using the absolute values ​​of the coefficients as input, the model is insensitive to transformation types and does not require separate optimization for a single transformation type. Superblock structure adaptation: AV1 uses superblocks (64×64 or 128×128) for recursive partitioning. In the encoding process, this system processes each transformation unit independently to accurately adapt to the recursive partitioning logic of superblocks.

[0179] (II) Multi-model Neural Network Framework The core innovation of this system lies in the use of multiple dedicated neural network models, rather than a single general-purpose model. This architecture design has multiple technical advantages.

[0180] 1. Model Library Structure The system maintains a structured pre-trained model library. The models are organized in multiple dimensions, such as block size dimension: covering the mainstream AV1 transform unit sizes such as 4×4, 8×8, 16×16, and 32×32; quantization parameter (QP) range dimension: adapting to different compression intensity requirements (e.g., typical values ​​of QP such as 5, 10, 21, and 30).

[0181] 2. Dynamic Model Selection Mechanism During the encoding process, the system completes model adaptation and invocation according to the following steps: determine the block size of the current processing transformation unit; obtain the quantization parameters (QP) corresponding to the current encoding; select pre-trained models that match the above multiple dimensions from the model library; if the target model is not loaded into the NPU (Neural Processing Unit), then perform model loading first; and call the selected model to perform inference computation.

[0182] 3. The core reason for adopting a multi-model architecture Block size specificity adaptation: The statistical distribution characteristics of the transformation coefficients of transformation units of different sizes are significantly different. Dedicated models can learn the exclusive patterns of each size to improve prediction accuracy. Computational efficiency optimization: Smaller blocks (such as 4×4, 8×8) use a simpler model, which effectively reduces the computational overhead of blocks with high frequency of occurrence, and adapts to real-time encoding requirements; Quality and complexity balance: Larger blocks (such as 16×16, 32×32) have greater compression gain potential, and their performance improvement is sufficient to support the computational power consumption of complex models such as U-Net, achieving the optimal trade-off between "high complexity and high gain".

[0183] (III) Neural Network Architecture The system employs two main neural network architectures, each suited to different block sizes and computational constraints.

[0184] (1) Large-scale U-Net architecture (same as the first model above) For 16x16 and 32x32 blocks, the system uses a U-Net-based architecture, such as... Figure 3 As shown.

[0185] Regarding architectural details: The U-Net model includes: Encoder path (shrinkage) (same as the above coding layer structure): includes an initial convolutional layer (3x3 kernel, ReLU activation), followed by two downsampling operations: the first downsampling is max pooling (2x2) followed by convolution, and the second downsampling is max pooling (2x2) followed by convolution; the channel depth of each downsampling stage gradually increases (8 → 16 → 32 channels).

[0186] Bottleneck layer (same as the bottleneck layer structure above): This is the deepest layer in the network, with the highest channel depth (32 channels), and can capture the global context features of the coefficient blocks.

[0187] Decoder path (extension) (same as the decoding layer structure above): includes two upsampling steps: the first upsampling is bilinear upsampling followed by convolution, and the second upsampling is bilinear upsampling followed by convolution; each upsampling stage includes a skip connection to the corresponding encoder layer, and the channel depth decreases in reverse (32→16→8 channels).

[0188] Output layer (same as the first output layer structure above): a single-channel output is generated by the final 1x1 convolution, and each output value corresponds to the adjustment of the predicted logit (0 or -1).

[0189] It should be noted that during the input processing stage, U-Net receives two concatenated input channels: the normalized absolute values ​​of the transform coefficients and the normalized absolute values ​​of the scalar quantization coefficients; in addition, the QP value is provided as a global conditional parameter and can be injected into multiple network layers.

[0190] Using the above neural network architecture can bring at least the following beneficial effects: Skip connections preserve spatial information at different scales, encoder-decoder structures capture local patterns and global context, and are suitable for complex spatial dependencies in larger transform units.

[0191] The number of parameters involved is approximately 3.23 million (300 channels) for the 16x16 model.

[0192] (2) Small-block fully convolutional neural network (FCNN) (same as the second model above) For 4x4 and 8x8 blocks, the system uses a lighter-weight FCNN architecture, such as... Figure 4 As shown.

[0193] Regarding architectural details: The FCNN model includes: Input layer (same as the input layer structure described above): receives the transform coefficients and quantization coefficients of the connection, with an input size of Block_size × Block_size × 2 channels. Hidden convolutional layers (same as the hidden convolutional layer structure above): 8x8 blocks correspond to 4 hidden layers (N=4), each with 256 channels; 4x4 blocks correspond to 3 hidden layers (N=3), each with 256 channels; all hidden layers use 3x3 convolutions, and the spatial size remains unchanged through zero padding.

[0194] Output layer (same as the second output layer structure above): generates a single-channel output through 1x1 convolution, with each coefficient position corresponding to a binary classification logit (adjusted to 0 or -1).

[0195] It should be noted that the network can also be configured with batch normalization, ReLU activation, and QP conditional processing; among them, QP conditional processing provides the QP value as input to each convolutional layer, which can be achieved through additional input channels or conditional batch normalization.

[0196] Using the above neural network architecture can bring at least the following beneficial effects: The simple feedforward architecture minimizes computational overhead, all coefficients are processed in parallel (without order dependency), fewer parameters reduce memory usage and inference time, and it is suitable for hardware implementations using standard convolution operations.

[0197] The number of parameters involved is approximately 1.19 million for the 4x4 model and approximately 1.78 million for the 8x8 model.

[0198] (iv) Input processing and normalization Coefficient sign processing: Before processing by the neural network, the coefficient sign is removed (absolute value is taken). The neural network only predicts the adjustment of the coefficient amplitude. After the adjustment prediction is completed, the sign is reapplied. This processing method reduces the complexity of the learning problem and the number of different input-output patterns (similar to the above absolute value processing of the target block coefficients to obtain unsigned block coefficients, where unsigned block coefficients represent the block coefficients that retain the target amplitude coefficients after removing the sign attribute corresponding to the target sign coefficients, and the subsequent element-wise multiplication of the unsigned quantized coefficients and the target sign coefficients to obtain the target quantized coefficients).

[0199] Normalization: All inputs are normalized based on statistical information calculated from the training set: For transformation coefficients, mean subtraction and standard deviation division are used; for quantization coefficients, the same normalization scheme is used; and each QP and block size corresponds to a separate normalization parameter (the unsigned block coefficients and initial quantization coefficients are normalized as described above to obtain normalized block coefficients and normalized quantization coefficients).

[0200] Quantization parameter encoding: QP values ​​are encoded and provided to the network. Specific encoding methods include: one-hot encoding of discrete QP values, or normalization to the range of [0,1] to achieve continuous representation; the encoded QP values ​​can be provided as global context to all layers, or injected into specific layers (similar to the above encoding of target quantization parameters to obtain encoded quantization parameters, wherein the encoding process includes one-hot encoding).

[0201] (v) Output processing and adjustment applications Binary classification: The output of the neural network is the logit matrix corresponding to the position of each transformation coefficient.

[0202] For each position: when Logit>0, the prediction is adjusted to 0 (no change), and when Logit≤0, the prediction is adjusted to -1 (the magnitude is reduced).

[0203] The target quantization coefficient can be determined as follows: q_final (final optimized quantization coefficient, also the target quantization coefficient) = q_scalar (initial scalar quantization coefficient, also the initial quantization coefficient) + Δ (same as the prediction adjustment value).

[0204] It should be noted that additional compliance processing can be performed to meet the additional processing requirements for AV1 compatibility. This includes ensuring that coefficient values ​​remain within a valid range, updating the position of the last non-zero coefficient if adjustments change it, and maintaining compatibility with the AV1 coefficient encoding order (raster scanning with configurable modes).

[0205] (vi) Training methods (1) Training data generation: Phase 1: HEVC RDOQ Initial Training Signal Construction Using HEVC RDOQ as the initial training signal, the training process begins with the implementation of the mature RDOQ in HEVC: Encode a diverse set of video sequences using an HM (HEVC test model) with RDOQ enabled; For each transform unit, record the following data: transform coefficients before quantization, scalar quantization coefficients, RDOQ optimization coefficients, QP value, and transform type; The difference between the RDOQ optimization coefficients and the scalar quantization coefficients is the training label.

[0206] Phase 2: AV1 Adaptation Processing The training data exported from HEVC is adapted to the AV1 encoding context. The specific steps are as follows: The video sequence was encoded using the ported RDOQ algorithm and libaom encoder; Collect AV1-specific training data, including: output coefficients of DCT and ADST transforms in AV1, patterns unique to the AV1 coding structure, and statistical data reflecting the entropy coding characteristics of AV1; Training data needs to be filtered and augmented: only blocks with non-zero coefficients are retained; data is augmented through rotation, reflection, etc. (while maintaining transformation properties); a balancing strategy is used to solve severe class imbalance problems (most adjustment values ​​are 0).

[0207] Optionally, Phase 3: Training Data Optimization To further improve RDOQ baseline performance, an optional greedy search procedure can be applied, as follows: For each 4x4 coefficient group within the block, exhaustively search for a better quantization level; Each candidate solution is evaluated using the real rate-distortion cost. If a better solution (lower RD cost) is found, the training objective is updated. This method can generate "expert" labels that outperform the standard RDOQ (similar to updating the initial training dataset to obtain the target training dataset, wherein the update operation includes determining multiple candidate sample adjustment values ​​based on the initial sample quantization coefficients, determining the candidate sample quantization coefficients corresponding to the multiple candidate sample adjustment values, determining the candidate rate-distortion cost index corresponding to the multiple candidate sample quantization coefficients, and setting the predetermined sample quantization coefficient to the candidate sample quantization coefficient corresponding to the lowest candidate rate-distortion cost index when there is a candidate rate-distortion cost index lower than the predetermined rate-distortion cost index).

[0208] (2) Neural network training: Determine the loss function: The main training objective is the binary cross-entropy loss, calculated by coefficients (same as the above for determining the target loss function, where the target loss function includes the binary cross-entropy loss, which is used to measure the difference between the model's predicted adjusted value and the actual adjusted value).

[0209] L=-Σ[y_ilog(p_i)+(1-y_i)log(1-p_i)] Where y_i: the true adjusted value (0 or 1, where 1 represents an adjustment of -1), and p_i: the predicted adjusted value from the model (logit_i).

[0210] For the core configuration of model training: The optimizer used for model training is Adam, and the learning rate is fixed at 3×10. -4 The batch size employs a dynamic adaptation strategy, flexibly adjusting according to the image patch size. Specifically, smaller image patches are matched with larger batch sizes to improve training and data utilization efficiency. The regularization process uses a multi-dimensional combination strategy: the core incorporates 10⁻ 6 L2 weight decay suppresses model overfitting, while batch normalization stabilizes the gradient distribution during training; Dropout is an optional strategy that is only enabled in larger models to further enhance the model's generalization ability.

[0211] Regarding the model training execution process: Before training officially begins, the network weights are initialized, using either Xavier or He initialization methods to ensure a reasonable initial parameter distribution. The training process is executed iteratively in cycles: within each training cycle, the training dataset is first randomly shuffled to avoid data order bias affecting model learning; for each mini-batch of data, forward propagation (calculating model prediction adjustments), loss calculation (quantifying the difference between predicted and true values), and backpropagation (solving parameter gradients) are performed sequentially, and the network weights are iteratively updated using the Adam optimizer. After each training cycle, model performance is evaluated on the validation set. If the validation set performance improves, the current optimal model weights are saved immediately. The overall training process continues until the model converges or terminates when the early stopping criteria are triggered (e.g., validation set performance shows no improvement for several consecutive cycles).

[0212] For differentiated training strategies: To adapt to the needs of different encoding scenarios, the model adopts a scenario-specific training approach: on the one hand, dedicated model versions are trained for different image patch sizes such as 4×4, 8×8, 16×16, and 32×32; on the other hand, independent models are also trained for different QP (quantization parameter) ranges, ensuring that each model accurately adapts to the quantization optimization requirements of a specific scenario. Furthermore, each model can selectively share early network layers, leveraging transfer learning to reuse general feature extraction capabilities, reducing the training overhead of different configuration models, and simultaneously improving the model's generalization ability (similar to determining the initial shared layers corresponding to multiple initial models; training the initial shared layers using corresponding shared sample data to obtain the target shared layer; and obtaining intermediate adjustment models corresponding to multiple initial models based on the shared layers).

[0213] (vii) Validation and Model Selection The model performance validation constructs a multi-dimensional, hierarchical evaluation system, which corely covers classification accuracy, rate-distortion performance, and offline rate-distortion (RD) (similar to the multi-dimensional validation metrics corresponding to the target model determined above, wherein the multi-dimensional validation metrics include at least one of the following: classification accuracy metric, theoretical rate-distortion performance metric, and actual rate-distortion evaluation metric; based on the multi-dimensional validation metrics, the intermediate training model is subjected to multi-dimensional performance testing to obtain the model performance evaluation results; if the model performance evaluation result is unsuccessful, the model parameters of the intermediate training model are updated until the target model with the corresponding model performance evaluation result is obtained).

[0214] In terms of classification accuracy, not only is the overall percentage of the model correctly predicting the adjusted values ​​statistically analyzed, but the evaluation dimensions are further refined: first, the accuracy of each class split by the adjusted value category (with a focus on the minority classes with an adjusted value of -1 to avoid prediction bias of the model for niche adjusted types); second, the accuracy weighted by coefficient position to adapt to the differentiated impact of coefficients at different positions in image blocks on the coding effect, and to comprehensively measure the model's prediction accuracy.

[0215] Rate-distortion performance evaluation uses BD rate (Bjøntegaard Delta Rate) as the core indicator and compares it with the scalar quantization baseline. This indicator needs to be evaluated on the reserved verification video sequence and calculated separately for different QP (quantization parameter) values ​​to ensure that the performance is covered under different compression intensities.

[0216] Offline RD evaluation employs an independent offline encoding method: the model prediction results are used to encode the verification video (not yet integrated into the encoder), the actual RD performance after encoding is measured, and the results are compared with HMRDOQ (HEVC reference model rate-distortion optimized quantization) and scalar quantization baselines to verify the actual effect of the model in real encoding scenarios.

[0217] In the model selection process, tiered screening criteria were established for each configuration to ensure the selection of the optimal model that balances performance and practicality: the primary selection criterion is the BD rate performance on the validation set, prioritizing the model with the best BD rate to ensure encoding compression efficiency; the secondary criterion is the classification accuracy on non-zero coefficients to supplement the validation model's ability to predict key coefficient adjustment values ​​and avoid prediction bias in core features; the tertiary criterion is the model size and inference time, taking into account the lightweight and real-time requirements of hardware deployment scenarios, and finally determining the model with the best overall performance for each configuration.

[0218] (viii) Runtime Operation 1. Frame Input The video frames to be encoded are input into the AV1 encoder as the raw data basis for the entire encoding process (same as above for obtaining the initial video stream; determining multiple video frames corresponding to the initial video stream).

[0219] 2. Predictive Processing Perform differential prediction based on the type of video frame to generate predicted image patches (similar to the above, determining the predicted image patches corresponding to multiple video frames based on the original image patches corresponding to each video frame respectively): Intraframe: Based on the reconstructed adjacent image blocks, a prediction block is generated through spatial prediction (same as the first image block mentioned above, where the first image block is obtained by spatial correlation prediction based on the reference image blocks corresponding to the reconstructed adjacent blocks for intraframes in multiple video frames). Inter-frame: Based on the encoded reference frame, a prediction block is generated through motion compensation prediction (same as the second image block mentioned above, the second image block is obtained by performing motion compensation prediction on the reference image block corresponding to the encoded reference frame for inter-frames in multiple video frames).

[0220] 3. Residual Calculation Calculate the pixel information difference between the original image block and the above-mentioned prediction block to obtain the residual image block, and focus on the non-redundant difference data to be encoded (similar to the above, based on multiple original image blocks and multiple prediction image blocks, determine the residual image blocks corresponding to multiple video frames respectively).

[0221] 4. Transformation Selection and Application The AV1 encoder autonomously selects the transform type (DCT, ADST, or a combination of both) based on the signal characteristics of the residual image block, and applies the selected transform method to the residual image block to complete the conversion from the spatial domain to the frequency domain and generate transform coefficients.

[0222] 5. Scalar quantization Standard scalar quantization is used as the initial quantization step: Combine the current QP (quantization parameter) and quantization matrix, perform scalar quantization on the transform coefficients to generate initial quantization coefficients (q_scalar).

[0223] 6. Optimization of RDOQ in Neural Networks (Core Component) 6.1 Input Preparation Extract the absolute values ​​of the transform coefficients (stripping away the sign attribute and retaining only the amplitude feature), and store the sign information of the coefficients separately; The quantized values ​​obtained from the transformation coefficients and scalar quantization are normalized respectively; Extract the QP value used in the current encoding to complete the preprocessing of the model input features.

[0224] 6.2 Model Selection Determine the size (W×H) of the current image patch being processed; Select a model from the pre-trained model library that is suitable for this block size; If the target model is not loaded into the NPU (Neural Processing Unit), the model loading will be completed first.

[0225] 6.3 Model Reasoning The preprocessed features are integrated into an input tensor (the dimensions of which include [transformation coefficients, quantization coefficients, QP]). Perform neural network forward propagation computation on the NPU; Output the logit value (unnormalized prediction score) for each coefficient position.

[0226] 6.4 Adjusting the forecast Thresholding is applied to the logit value to convert it into a binary adjusted value (containing only 0 or -1): An adjustment value of 0: retains the initial value obtained from scalar quantization; An adjustment value of -1: reduces the magnitude of the corresponding coefficient by 1.

[0227] 6.5 Adjusting the application The predicted adjustment value is added to the scalar quantization coefficient to optimize the coefficient amplitude. Restore the previously stored coefficient signs and restore the direction attributes of the coefficients; The adjusted coefficient values ​​are cropped to the valid range allowed by AV1 encoding.

[0228] 6.6 Post-processing Update the position of the "last non-zero coefficient" in the image patch as needed; Verify and ensure that the adjusted coefficients fully comply with the format and syntax requirements of AV1 encoding.

[0229] 7. Entropy Coding The AV1 standard range encoder is used to encode the final quantization coefficients optimized by the neural network RDOQ; the encoding process strictly follows the coefficient scanning order and syntax rules specified by AV1 to generate a compressed bitstream.

[0230] 8. Reconstruction Processing Perform inverse quantization and inverse transform operations on the final quantization coefficients to restore the residual image patch in the spatial domain; The reconstructed residual image patch is superimposed with the predicted patch to complete the reconstruction of the image patch (similar to the above decompression process of the target quantization coefficient to obtain the inverse quantization coefficient; based on the inverse quantization coefficient, the inverse residual image patch is determined; based on the inverse residual image patch and the corresponding predicted image patch, the reference image patch is determined). The reconstructed image patches are stored in the frame buffer as reference data for subsequent frame prediction.

[0231] (ix) Hardware implementation design flow of a neural network-based RDOQ system This system is designed for efficient adaptation to the neural processing unit (NPU) of modern coding hardware. It optimizes hardware utilization efficiency from the dimensions of batch processing, model management, quantization, complexity control, and memory scheduling. The specific design process is as follows: 1. Efficient batch processing workflow of NPU To maximize the parallel computing capabilities of the NPU, a collaborative coding pipeline is implemented using the following process: During the encoding process, image blocks of the same size are collected and classified in real time, and blocks of the same size are grouped into batches to be processed. Deploy a buffer mechanism and perform fine-grained pipeline management to balance the block collection rhythm with the NPU processing speed and avoid pipeline interruptions; By submitting blocks of the same size in batches to the NPU, the parallel computing capabilities of the NPU can be used to process multiple blocks simultaneously, significantly improving the utilization rate of NPU resources.

[0232] 2. Model caching optimization process To address the overhead of model loading / switching, a lightweight caching strategy is designed: Statistical analysis of the frequency of occurrence of each block size in the coding scenario helps identify frequently used models. Preload commonly used models and keep them resident in NPU memory to reduce the time loss from frequent loading / unloading; Based on real-time statistics of block size, the size of subsequent blocks to be processed is predicted, and the corresponding model is preloaded to further reduce model switching overhead.

[0233] 3. Neural Network Quantization Optimization Process With the goal of "low precision + high throughput", quantization adaptation is performed: During the model training phase, quantization-aware training is introduced to adapt the model to the weight quantization logic and minimize the loss of accuracy. The trained network weights are quantized into INT8 / INT16 format to compress the weight data size. Deploy the quantized model to the NPU to leverage the NPU's efficient processing capabilities for low-precision data, while reducing memory usage and increasing data throughput.

[0234] 4. Hardware adaptation design for computational complexity Achieving complexity optimization and parallelization through architectural design: Compared to the traditional RDOQ (complexity O(N²×M), where N is the block size and M is the number of iterations), the complexity of the neural network RDOQ is designed to be O(N²×K) (K is the fixed number of operations for each coefficient). Based on the forward propagation characteristics of neural networks, the processing logic is designed as an O(1) iteration (requiring only a single forward computation), and all coefficient processing is fully parallelized, adapting to the parallel computing architecture of NPU.

[0235] 5. Refined management process for memory requirements Controlling memory overhead from storage and runtime perspectives: Model storage: All pre-trained models are stored uniformly in the encoder's non-volatile memory, occupying only a few MB in total, thus avoiding the consumption of runtime memory; Runtime memory scheduling: Allocate only the minimum input / output buffers required for neural network inference; The model parameters are loaded into the NPU memory only when the model is activated (processing the corresponding size block), and released as needed after processing is complete; Memory resources are reused with the existing encoder pipeline to ensure that additional memory overhead during hardware operation is minimized.

[0236] For AV1-specific optimization design: To address the technical characteristics of the AV1 encoder, this neural network-based RDOQ system has undergone multi-dimensional adaptation and optimization to ensure a balance between functional compatibility and coding efficiency. Regarding transform type processing, AV1 supports multiple modes, including DCT (Discrete Cosine Transform), ADST (Asymmetric Discrete Sine Transform), IDTX (Identity Transform), and hybrid transformations (such as horizontal DCT + vertical ADST). The system extracts the absolute values ​​of the coefficients as neural network input, making the model largely insensitive to transform types—the distribution pattern of coefficient amplitudes already carries core features, eliminating the need for separate optimization for individual transform types. If subsequent experiments demonstrate that additional complexity can bring significant performance gains, it can also be extended to train dedicated models for different transform types.

[0237] Regarding block size adaptation, although the current system focuses on square blocks such as 4×4, 8×8, 16×16, and 32×32, the framework has the flexibility to expand to non-square blocks, which can be achieved in three ways: First, the filling method, which fills the non-square blocks to the nearest square size, and then extracts the effective area after model processing; second, the dedicated model method, which trains a dedicated model for common non-square sizes such as 4×8 and 8×16; and third, the adaptive pooling method, which introduces an adaptive pooling layer into the network to accommodate variable input sizes.

[0238] In its integration with AV1 rate control, the system follows the encoder's native logic: the QP value is dynamically determined by the rate control algorithm, the neural network model is pre-trained for a specific QP range, and the appropriate model is automatically selected at runtime based on the current QP. For QP values ​​that do not fully match the training range, performance can be further optimized through optional inter-model interpolation schemes. Simultaneously, the system strictly ensures compatibility with various AV1 encoding tools, including dedicated modes such as CDEF (Constrained Direction Enhancement Filter), loop recovery, film grain synthesis, and screen content encoding. Because the RDOQ system runs during the quantization stage, earlier than the execution flow of the aforementioned post-processing tools, full compatibility is achieved without modifying the original encoding tool logic.

[0239] Performance characteristic verification: Based on the implementation and experimental results on the Netint / UBC platform, the performance characteristics of this system are as follows: In terms of compression performance, compared with scalar quantization, it can achieve an average improvement of 1-2% in BD rate, reaching 40-50% of the performance of traditional iterative RDOQ (the specific value varies with video content and QP value), and performs best in the medium QP range (20-30). In terms of prediction accuracy, different sized models show stable accuracy: the non-zero coefficient prediction accuracy of the 32×32 model reaches 97%, the 16×16 U-Net model is 96%, and the 8×8 and 4×4 FCNN models are approximately 92%.

[0240] In terms of computational performance, the system significantly outperforms traditional iterative RDOQ, supporting not only inter-block parallel processing but also real-time encoding requirements through NPU acceleration, completely resolving the pain points of high computational complexity and large latency in traditional solutions. Regarding visual quality, the system maintains objective visual effects comparable to iterative RDOQ, without introducing significant coding artifacts; simultaneously, thanks to a superior rate-distortion tradeoff strategy, the subjective visual quality of the video is further improved, achieving a dual optimization of "efficiency and quality."

[0241] Industrial applicability: This invention provides practical and efficient technical support for AV1 video encoding in various industrial scenarios. Its core advantages are reflected in three dimensions: hardware adaptation, real-time application implementation, and open-source ecosystem compatibility, and it has broad industrial deployment value.

[0242] In the field of hardware video encoders, modern hardware encoders are increasingly integrating Neural Processing Units (NPUs) to enable AI acceleration. This invention perfectly aligns with this technological trend—eliminating the need for dedicated, area-dense RDOQ hardware circuitry, allowing hardware encoders to leverage the compression efficiency advantages of RDOQ, significantly reducing hardware design and manufacturing costs. This feature is highly practical in various scenarios: suitable for mobile devices such as smartphones and tablets, meeting their lightweight requirements for real-time video capture and streaming media transmission; compatible with IP camera monitoring systems that simultaneously encode multiple video streams, balancing encoding efficiency and hardware load; supporting broadcast equipment dedicated to live streaming and streaming media, ensuring professional-grade encoding quality; and applicable to cloud encoding services for large-scale server-side video transcoding, improving cluster encoding throughput.

[0243] In real-time video applications, the inherent parallelism of neural network inference enables this invention to provide compression performance close to that of traditional RDOQ while ensuring real-time encoding: for bandwidth-constrained video conferencing, enhanced compression efficiency can optimize call fluency; for live streaming scenarios such as game streaming, sports broadcasting, and live events, higher quality video transmission can be provided while reducing bandwidth consumption; at the same time, it can adapt to the efficient encoding requirements of high-resolution VR / AR content, helping to promote the implementation of immersive experiences.

[0244] In the open-source video ecosystem, AV1, as an open-source and royalty-free codec, is gradually becoming the industry mainstream. This invention further promotes the widespread adoption of AV1 by enabling low-cost deployment of advanced RDOQ compression technology in hardware implementation: it can enhance the video playback efficiency of web browsers and improve the user's viewing experience; help streaming media services reduce the bandwidth costs of content delivery and optimize operating expenses; and it can also seamlessly integrate with mainstream AV1 open-source implementations such as libaom, providing high-performance encoding solution support for open-source projects.

[0245] In summary, this invention is a practical and directly deployable technical solution that successfully bridges the gap between the theoretical compression advantages of traditional RDOQ and the resource constraints of real-world hardware video coding systems. Furthermore, it is deeply adapted to the AV1 codec standard, providing an efficient and flexible solution for various industrial-grade video coding applications.

[0246] It should be noted that before this system is built, there is also the relevant content of building each subsystem in sequence: The technical implementation of this invention relies on the sequential construction of four subsystems, with each subsystem progressing layer by layer and supporting the others: 1) libaom+RDOQ: A branch of the libaom project (AV1 codec reference implementation), the core of which is to port the HEVC RDOQ algorithm to the AV1 encoder pipeline, filling the gap that libaom natively lacks RDOQ functionality; 2) Deep learning model: Define a dual architecture design of U-Net and FCNN, and optimize model complexity for different block sizes; 3) DNN-RDOQ: A deep neural network trained for a specific QP, specifically for predicting the RDOQ quantization adjustment value of the AV1 encoder pipeline; 4) libaom+DNN-RDOQ: A branch of libaom+RDOQ, which integrates DNN to replace the traditional RDOQ, thereby achieving more efficient quantization decision-making.

[0247] The libaom+RDOQ subsystem uses libaom's built-in Trellis quantization as the model training baseline. This quantization scheme is natively integrated into libaom. Adding RDOQ can further optimize the quantization effect, but it will increase the computational overhead. The parallel characteristics of DNN can compensate for this performance loss. The core reason for choosing libaom as the basic branch is that it is the official reference implementation of AV1 and meets the customer's explicit technical requirements.

[0248] Through experimental comparison and verification, the following design can be adopted to achieve better results: 1) Model selection was verified through multiple rounds of comparison: The selected U-Net and FCNN architectures have prediction accuracy close to that of more complex models, but have significantly smaller parameter scale, are more hardware-friendly, and are more suitable for real-time coding scenarios.

[0249] 2) QP parameter selection and adaptation: The core set of adapted QP values ​​is {5, 10, 21, 30}. Reason for selection: Effective training data can be generated with a small sample size. High QP values ​​can generate sparse coefficient matrices, covering key compression scenarios. It supports adapting to other QP value requirements by training additional neural networks, and has strong scalability.

[0250] It should be noted that, for the integration method, the libaom+DNN-RDOQ subsystem can embed the DNN into the AV1 encoder pipeline through the libtorch C++ API, directly replacing the original RDOQ algorithm and achieving efficient decision-making for quantization coefficient increments; The above settings can also meet at least the following requirements: after integration, the functional requirements FR-1 (precise quantization optimization) and FR-2 (neural network replacing traditional algorithm) can be met, and the non-functional requirements NR-1 (peak signal-to-noise ratio PSNR meets the standard libaom) and NR-2 (latency meets the standard libaom+RDOQ) can be met.

[0251] The following beneficial effects can be achieved through the above optional implementation methods: 1) AV1-specific design: For the first time, RDOQ based on neural networks was specifically applied to AV1 to address its unique characteristics.

[0252] 2) Hardware feasibility: By utilizing NPUs instead of requiring dedicated, expensive RDOQ circuitry, the hardware AV1 encoder can achieve the advantages of RDOQ.

[0253] 3) Parallel processing: Unlike the sequential traditional RDOQ, it can be fully parallelized within each block.

[0254] 4) Reduced complexity: It eliminates iterative search and repetitive entropy encoding evaluation.

[0255] 5) Open source integration: Designed to be compatible with libaom and support the open source video ecosystem.

[0256] 6) Scalable performance: Balance quality and computational cost for different model complexities with different block sizes.

[0257] According to an embodiment of this application, a frame quantization coefficient adjustment device is provided. Figure 8 This application provides a schematic diagram of the structure of a frame quantization coefficient adjustment device. Figure 8 As shown, the device includes: a first acquisition module 801, a first determination module 802, a first retrieval module 803, a second determination module 804, a third determination module 805, and a first adjustment module 806. The device will be described below.

[0258] The first acquisition module 801 is used to acquire the target image block corresponding to the target frame; The first determining module 802 is connected to the first acquiring module 801 and is used to determine the model selection parameters corresponding to the target image block. The model selection parameters include at least one of the following: block size parameters, and other parameters. The first retrieval module 803 is connected to the first determination module 802 and is used to retrieve the target model corresponding to the model selection parameters from multiple candidate models. The multiple candidate models are obtained by training the corresponding initial models, and the multiple initial models correspond to different model architectures. The second determining module 804 is connected to the first fetching module 803 and is used to determine the input parameters corresponding to the target model. The input parameters include the target block coefficients of the target image block, the target quantization parameters and the initial quantization coefficients. The target block coefficients include the target amplitude coefficients and the target sign coefficients. The target amplitude coefficients represent the signal strength of the image block and the target sign coefficients represent the signal direction of the image block. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. The third determining module 805, connected to the second determining module 804, is used to input the input parameters into the target model to obtain the predicted adjustment value of the target image patch; The first adjustment module 806, connected to the third determination module 805, is used to adjust the initial quantization coefficient based on the predicted adjustment value to obtain the target quantization coefficient.

[0259] It should be noted that the first acquisition module 801, the first determination module 802, the first retrieval module 803, the second determination module 804, the third determination module 805, and the first adjustment module 806 mentioned above correspond to steps 101 to 106 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, and their implementation principles and technical effects will not be elaborated further. The specific methods by which each module and unit in the device of the above embodiments performs operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0260] According to an embodiment of this application, a video stream encoding apparatus is provided. Figure 9 This application provides a schematic diagram of the structure of a video stream encoding device. Figure 9 As shown, the device includes: a second acquisition module 901, a fourth determination module 902, a fifth determination module 903, a second retrieval module 904, a sixth determination module 905, a seventh determination module 906, a second adjustment module 907, and an eighth determination module 908. The device will be described below.

[0261] The second acquisition module 901 is used to acquire an initial video stream, wherein the initial video stream includes multiple video frames, and the multiple video frames include a target frame; The fourth determining module 902, connected to the second obtaining module 901, is used to determine the target image block corresponding to the target frame; The fifth determining module 903, connected to the fourth determining module 902, is used to determine the model selection parameters corresponding to the target image block, wherein the model selection parameters include at least one of the following: block size parameters, other parameters; The second retrieval module 904 is connected to the fifth determination module 903 mentioned above, and is used to retrieve the target model corresponding to the model selection parameters from multiple candidate models. The multiple candidate models are obtained by training the corresponding initial models, and the multiple initial models correspond to different model architectures. The sixth determining module 905, connected to the second fetching module 904, is used to determine the input parameters corresponding to the target model. The input parameters include the target block coefficients of the target image block, the target quantization parameters and the initial quantization coefficients. The target block coefficients include the target amplitude coefficients and the target sign coefficients. The target amplitude coefficients represent the signal strength of the image block and the target sign coefficients represent the signal direction of the image block. The initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters. The seventh determining module 906, connected to the sixth determining module 905, is used to input the input parameters into the target model to obtain the predicted adjustment value of the target image patch; The second adjustment module 907, connected to the seventh determination module 906, is used to adjust the initial quantization coefficient based on the predicted adjustment value to obtain the target quantization coefficient. The eighth determining module 908 is connected to the second adjusting module 907 and is used to obtain the target video stream based on the target quantization coefficient.

[0262] It should be noted that the second acquisition module 901, the fourth determination module 902, the fifth determination module 903, the second retrieval module 904, the sixth determination module 905, the seventh determination module 906, the second adjustment module 907, and the eighth determination module 908 mentioned above correspond to steps 201 to 208 in the embodiments. The instances and application scenarios implemented by these modules and their corresponding steps are the same, and their implementation principles and technical effects will not be elaborated further. The specific methods by which each module and unit in the device of the above embodiments performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0263] This application also provides a computer device, which may include a storage component and a processing component; The storage component contains one or more computer instructions, wherein one or more computer instructions are invoked and executed by the processing component to implement any of the methods described above.

[0264] Of course, computer equipment may also include other components, such as input / output interfaces, display components, communication components, etc.

[0265] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0266] The processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0267] Storage components are configured to store various types of data to support operations on the terminal. Storage components can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0268] The display component can be an electroluminescent (EL) element, a liquid crystal display or a microdisplay with a similar structure, or a retina-direct display or a similar laser scanning display.

[0269] It should be noted that the aforementioned computing device implementation method or processing method can be a physical device or an elastic computing host provided by a cloud computing platform. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device.

[0270] When the above-mentioned computing device implements the above method, it can be specifically implemented as an electronic device. An electronic device can refer to a device used by a user that has the computing, Internet access, communication and other functions required by the user, such as a mobile phone, tablet computer, personal computer, wearable device, etc.

[0271] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0272] This application also provides a computer-readable storage medium storing a computer program that, when executed by a computer, can implement the above-described method. This computer-readable medium may be included in the electronic device described in the above embodiments; alternatively, it may exist independently and not be assembled into the electronic device.

[0273] This application also provides a computer program product comprising a computer program carried on a computer-readable storage medium, which, when executed by a computer, can implement the methods described above. In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the various functions defined in the system of this application.

[0274] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0275] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0276] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0277] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0278] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for adjusting the quantization coefficients of a frame, characterized in that, include: Obtain the target image block corresponding to the target frame; Determine the model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: patch size parameters, other parameters; From multiple candidate models, a target model corresponding to the model selection parameters is retrieved, wherein the multiple candidate models are obtained by training a corresponding initial model, and the multiple initial models correspond to different model architectures; Determine the input parameters corresponding to the target model, wherein the input parameters include the target block coefficients of the target image patch, the target quantization parameters and the initial quantization coefficients, the target block coefficients include the target amplitude coefficients and the target sign coefficients, the target amplitude coefficients represent the signal intensity of the image patch, the target sign coefficients represent the signal direction of the image patch, and the initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters; The input parameters are input into the target model to obtain the predicted adjustment value of the target image patch; The initial quantization coefficient is adjusted based on the predicted adjustment value to obtain the target quantization coefficient.

2. The method according to claim 1, characterized in that, Before retrieving the target model corresponding to the model selection parameters from multiple candidate models, the process includes: When the model selection parameters include block size parameters, multiple candidate block sizes are determined; Determine the architecture selection parameters corresponding to the various candidate block sizes, wherein the architecture selection parameters include at least one of the following: block size characteristic parameters, computational efficiency parameters, and quality-complexity balance parameters; Based on the parameters selected according to the corresponding architecture, an initial model corresponding to each of the various candidate block sizes is determined; The corresponding initial model is trained using the corresponding sample data to obtain the multiple candidate models.

3. The method according to claim 1, characterized in that, The input parameters are input into the target model to obtain the predicted adjustment value of the target image patch, including: When the block size parameter is greater than a first size threshold and the target model is a first model, the input parameter is input to the encoding layer structure of the first model to obtain multi-scale shrinkage features. The first model further includes a bottleneck layer structure, a decoding layer structure, and a first output layer structure. The multi-scale shrinkage features represent the signal features at multiple scales obtained after the target image block undergoes different downsampling processes and compressed spatial sizes. The multi-scale shrinkage features are input into the bottleneck layer structure to obtain global context features, wherein the global context features are features that characterize the global distribution pattern and cross-location dependency relationship of the target image patch; The global context features are input into the decoding layer structure to obtain multi-scale extended features, wherein the multi-scale extended features represent signal features of multiple sizes obtained by fusing the global context features and the multi-scale contraction features and restoring the spatial size; The multi-scale extended features are input into the first output layer structure to obtain the predicted adjustment value.

4. The method according to claim 1, characterized in that, The input parameters are input into the target model to obtain the predicted adjustment value of the target image patch, including: When the block size parameter is less than the first size threshold and the target model is the second model, the input parameters are input to the input layer structure of the second model to obtain the initial fusion features. The second model also includes a hidden convolutional layer structure and a second output layer structure. The initial fusion feature is input into the hidden convolutional layer structure to obtain the target fusion feature, wherein the target fusion feature is a feature representing the fusion of local signal features and compression requirement features, and the compression requirement feature is a feature matching the compression requirement of the target quantization parameters; The target fusion features are input into the second output layer structure to obtain the prediction adjustment value.

5. The method according to claim 1, characterized in that, The input parameters are input into the target model to obtain the predicted adjustment value of the target image patch, including: The absolute value of the target block coefficients is taken to obtain unsigned block coefficients, wherein the unsigned block coefficients represent the block coefficients that retain the target amplitude coefficients after removing the sign attribute corresponding to the target sign coefficients; The unsigned block coefficients and the initial quantization coefficients are normalized respectively to obtain normalized block coefficients and normalized quantization coefficients; The target quantization parameters are encoded to obtain encoded quantization parameters, wherein the encoding process includes one-hot encoding. The normalized block coefficients, the normalized quantization coefficients, and the encoded quantization parameters are input into the target model as input parameters to obtain the predicted adjustment value.

6. The method according to claim 5, characterized in that, The initial quantization coefficient is adjusted based on the predicted adjustment value to obtain the target quantization coefficient, including: The initial quantization coefficients and the predicted adjustment values ​​are added together to obtain unsigned quantization coefficients. The unsigned quantization coefficient and the target signed coefficient are multiplied element-wise to obtain the target quantization coefficient.

7. The method according to claim 1, characterized in that, Before inputting the input parameters into the target model to obtain the predicted adjusted value of the target image patch, the method further includes: Determine the target loss function, wherein the target loss function includes binary cross-entropy loss, which is used to measure the difference between the model's predicted adjusted value and the actual adjusted value; Based on the target loss function, the corresponding initial model is trained using the corresponding sample data to obtain the target model.

8. The method according to claim 1, characterized in that, Before inputting the input parameters into the target model to obtain the predicted adjusted value of the target image patch, the method further includes: The initial model is trained using the corresponding sample data to obtain the intermediate training model; Determine the multi-dimensional validation metrics corresponding to the target model, wherein the multi-dimensional validation metrics include at least one of the following: classification accuracy metrics, theoretical rate-distortion performance metrics, and actual rate-distortion evaluation metrics; Based on the aforementioned multi-dimensional verification metrics, the intermediate training model is subjected to multi-dimensional performance testing to obtain model performance evaluation results. If the model performance evaluation result is unsuccessful, the model parameters of the intermediate training model are updated until the target model with a passing performance evaluation result is obtained.

9. The method according to claim 1, characterized in that, Before obtaining the target image patch corresponding to the target frame, the following steps are also included: Obtain the initial video stream; Determine multiple video frames corresponding to the initial video stream; Based on the original image blocks corresponding to the plurality of video frames, predictive image blocks corresponding to the plurality of video frames are determined, wherein the plurality of original image blocks and the plurality of predictive image blocks correspond one-to-one. The predictive image blocks include a first image block and / or a second image block. The first image block is obtained by spatial correlation prediction based on the reference image blocks corresponding to the reconstructed adjacent blocks for intra-frames in the plurality of video frames. The second image block is obtained by motion compensation prediction based on the reference image blocks corresponding to the encoded reference frames for inter-frames in the plurality of video frames. Based on the plurality of original image blocks and the plurality of predicted image blocks, residual image blocks corresponding to the plurality of video frames are determined respectively, wherein the residual image blocks are used to represent the pixel information difference data between the corresponding original image blocks and the corresponding predicted image blocks; Based on the residual image blocks corresponding to the plurality of video frames respectively, a processing image block corresponding to the plurality of video frames is determined, wherein the plurality of processing image blocks includes the target image block.

10. The method according to claim 1, characterized in that, After adjusting the initial quantization coefficient based on the predicted adjustment value to obtain the target quantization coefficient, the method further includes: The target quantization coefficients are decompressed to obtain inverse quantization coefficients; Based on the inverse quantization coefficients, determine the inverse residual image block; A reference image block is determined based on the inverse residual image block and the corresponding predicted image block.

11. The method according to claim 1, characterized in that, Before inputting the input parameters into the target model to obtain the predicted adjusted value of the target image patch, the method further includes: Obtain an initial training dataset, wherein the initial training dataset includes sample block coefficients of sample image patches, sample quantization parameters, initial sample quantization coefficients, target sample quantization coefficients, and a predetermined rate distortion cost index corresponding to the target sample quantization coefficients, and set the predetermined sample quantization coefficients as the target sample quantization coefficients; The initial training dataset is updated to obtain the target training dataset. The update operation includes determining multiple candidate sample adjustment values ​​based on the initial sample quantization coefficients, determining the candidate sample quantization coefficients corresponding to the multiple candidate sample adjustment values, determining the candidate rate-distortion cost index corresponding to the multiple candidate sample quantization coefficients, and setting the predetermined sample quantization coefficient as the candidate sample quantization coefficient corresponding to the lowest candidate rate-distortion cost index when there is a candidate rate-distortion cost index lower than the predetermined rate-distortion cost index.

12. The method according to claim 1, characterized in that, Before retrieving the target model corresponding to the model selection parameters from multiple candidate models, the process further includes: Obtain multiple initial models; Determine the initial shared layer corresponding to each of the multiple initial models; The initial shared layer is trained using the corresponding shared sample data to obtain the target shared layer; Based on the shared layer, an intermediate adjusted model corresponding to the plurality of initial models is obtained; The corresponding intermediate adjustment model is trained using the corresponding sample data to obtain the multiple candidate models.

13. The method according to any one of claims 1 to 12, characterized in that, Before determining the input parameters corresponding to the target model, the process includes: When the target block coefficients are of spatial domain type, the target block coefficients of spatial domain type are transformed to obtain the target block coefficients of frequency domain type.

14. A video stream encoding method, characterized in that, include: Acquire an initial video stream, wherein the initial video stream includes multiple video frames, and the multiple video frames include a target frame; Determine the target image block corresponding to the target frame; Determine the model selection parameters corresponding to the target image patch, wherein the model selection parameters include at least one of the following: patch size parameters, other parameters; From multiple candidate models, a target model corresponding to the model selection parameters is retrieved, wherein the multiple candidate models are obtained by training a corresponding initial model, and the multiple initial models correspond to different model architectures; Determine the input parameters corresponding to the target model, wherein the input parameters include the target block coefficients of the target image patch, the target quantization parameters and the initial quantization coefficients, the target block coefficients include the target amplitude coefficients and the target sign coefficients, the target amplitude coefficients represent the signal intensity of the image patch, the target sign coefficients represent the signal direction of the image patch, and the initial quantization coefficients are obtained by performing a compression operation on the target block coefficients based on the target quantization parameters; The input parameters are input into the target model to obtain the predicted adjustment value of the target image patch; The initial quantization coefficient is adjusted based on the predicted adjustment value to obtain the target quantization coefficient; Based on the target quantization coefficients, an encoding operation is performed to obtain the target video stream.

15. A computing device, characterized in that, This includes processing components and storage components; The storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement the method as described in any one of claims 1 to 14.

16. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processing component, implements the method as described in any one of claims 1 to 14.

Images