Quantization parameter prediction method for perceptual optimization of video coding based on attention mechanism

Abstract

The application provides a perceptual optimization video coding quantization parameter prediction method based on an attention mechanism. The original video is divided into multiple frames and sequentially sent into a lightweight backbone network to extract deep features, and then a learnable query vector is combined to automatically detect a key interest region in a corresponding target detection model, generate a soft attention map, and multiply the original features by channel weighting to obtain an enhanced weighted feature map. On this basis, the weighted feature map is cut into several fixed-size image blocks, tokens are generated by linear mapping and position encoding, and input into a multi-layer lightweight Transformer encoder for global context modeling. Finally, the multi-layer perception network is used to regress the quantization parameter prediction value block by block, and combined into a frame-level quantization parameter allocation map. The video coding quantization parameter generated by the method can significantly improve the subjective visual quality of the target interest region and the overall picture while ensuring coding efficiency.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for perceptually optimized video coding quantization parameter (QP) prediction based on an attention mechanism, for improving the visual perception quality and efficiency of video coding. Background Technology

[0002] With the continuous increase in video resolution and frame rate, the demand for high-quality, low-bitrate video is growing. Traditional video coding standards such as H.264 / AVC and H.265 / HEVC employ fixed QP strategies based on linear models or frame complexity, but they neglect the importance of different regions in the image and the characteristics of human vision. This results in insufficient protection of details in key target areas (such as faces and vehicles), leading to perceptual quality blind spots. Furthermore, traditional QP control methods are often frame-level or macroblock-level overall adjustments, unable to perform fine-grained optimization for high-interest areas. This results in the loss of details in key areas, affecting subjective perceptual quality.

[0003] In recent years, research has attempted to improve QP prediction and allocation using deep learning methods. For example, some methods use convolutional neural networks to predict the optimal QP value for video frames or fine-tune the QP of coded blocks to reduce bitrate while maintaining perceptual quality. However, these deep learning methods are typically complex and computationally expensive, making them difficult to meet the requirements of real-time coding and resource-constrained edge devices. Moreover, existing deep models often only consider global features, failing to effectively combine visual information from salient regions and thus failing to fully leverage the advantages of attention mechanisms in perceptual quality optimization. Therefore, there is a lack of a perceptual optimization method for video coding QP that simultaneously considers region of interest detection and lightweight implementation. Summary of the Invention

[0004] To address the shortcomings of existing technologies, such as Figure 2 The present invention proposes a perceptual optimization video coding quantization parameter prediction method based on an attention mechanism, which includes:

[0005] Video input steps: Divide the original video into N video frames { , … };

[0006] Feature extraction steps: Extract the original feature map of each video frame using a convolutional neural network. , … }; Original feature map of each frame With a set of prior boxes The input is fed into the decoder, where K represents the maximum number of candidate boxes that the detector can output, and d is the dimension of the hidden vector;

[0007] The object detection step, this decoder uses a multi-head attention mechanism to... With each query vector The interaction mechanism automatically focuses on key regions of interest in the image during the interaction process, and outputs a set of corresponding detection boxes and a semantic embedding vector for each detection. ;

[0008] The weighted multiplication step generates a binary mask for each detection box on the spatial grid of the original feature map, and the union of all masks is obtained. After Gaussian smoothing, a soft attention map is obtained. , … }, then compare it with the original feature map { , … The weighted feature map is obtained by multiplying the features by channel weights. , … Based on the weighted feature maps, the image is divided into M fixed-size, non-overlapping image patches. }, where the integer j ranges from 1 to M, and a token is generated through linear mapping and positional encoding. };

[0009] The encoding steps involve using an L-layer Transformer encoder to capture tokens globally using multi-head self-attention and a feedforward network. The context and inter-frame relationships of};

[0010] Prediction step, output of the last layer of the Transformer encoder { The quantization parameter values ​​of the corresponding patches are mapped one by one through a multilayer perceptron. The position of the corresponding patch in the frame is encoded using the quantization parameter value to obtain the encoded video of the original video.

[0011] The aforementioned attention-based perceptual optimization video coding quantization parameter prediction method, wherein the weighted feature map ,in It is a soft attention map after Gaussian filtering, where λ is a coefficient that controls the enhancement intensity.

[0012] The perceptual optimization video coding quantization parameter prediction method based on attention mechanism is described in which, after the binary mask is generated, a Gaussian kernel or bilateral filtering can be applied to the mask edge to smooth the transition region; the Transformer encoder has ≤6 layers and ≤4 heads per layer, and has a local or sparse attention mechanism.

[0013] The aforementioned attention-based perceptual optimization video coding quantization parameter prediction method, in its feature extraction step, employs a convolutional neural network to perform feature encoding on each frame of the image, thereby extracting the corresponding depth space feature map. , … }:

[0014]

[0015] in, This represents a convolutional neural network, where H′ and W′ are the height and width of the feature map, respectively, and C is the number of channels; convolutional neural networks use a network architecture that shares parameters.

[0016] The aforementioned attention-based perceptual optimization video coding quantization parameter prediction method, wherein the weighted multiplication step includes:

[0017] In the original feature map Above, a corresponding binary mask image is generated based on each bounding box. All mask images are joined along the channel dimension and smoothed using a Gaussian function to obtain a continuous soft attention map. Original feature map Will be related to attention map Perform channel-by-channel weighted fusion to obtain a weighted feature map. , where ⊙ represents element-wise multiplication by channel;

[0018] The image is divided into M fixed-size, non-overlapping image patches by combining the feature maps of each frame. Each image patch has a spatial size of P×P. Each image patch is flattened and converted into a fixed-dimensional embedding vector using a linear mapping function. A corresponding positional encoding is added to each token to form the input sequence. ,in This represents the token sequence corresponding to the i-th frame image. ;

[0019] The encoding steps include:

[0020] This Transformer encoder employs a multi-head self-attention mechanism and a feedforward network structure:

[0021]

[0022] in, This represents the sequence of hidden states output by the Transformer encoder, with each vector... This represents the global features extracted by the j-th patch after fusion context;

[0023] The prediction steps include:

[0024] The output of the last layer of the Transformer encoder { The input is fed into this multilayer perceptron, which consists of multiple fully connected layers, to predict the quantization parameter QP value for each image patch. Where j increases sequentially from 1 to M; QP prediction values ​​for all image patches It is reorganized into a frame-level QP allocation map to guide subsequent video encoders to adaptively adjust quantization parameters at the block level, thereby achieving fine compression of key regions.

[0025] The perceptual optimization video coding quantization parameter prediction method based on the attention mechanism further includes a model training step:

[0026] Based on the pre-labeled target categories and bounding boxes and the predicted categories and candidate boxes output by the decoder, a target detection loss consisting of target detection loss, QP prediction loss and perceptual quality loss is constructed, and the decoder is trained.

[0027] The target detection loss :

[0028]

[0029] in, Represents the category cross-entropy loss. The L1 loss represents the location of the bounding box. This represents the generalized IOU loss, where λ1 and λ2 are weighting coefficients;

[0030] Let the predicted value of the j-th patch in the i-th frame be... Its true QP label is The defined intra-frame QP prediction loss is:

[0031]

[0032] To further improve the perceptual quality of the video, a perceptual loss term based on the video quality score is introduced during the optimization process. Let the video quality score between the original frame and the reconstructed frame be... The target reference score is Then the perceived quality loss is:

[0033]

[0034] The overall joint loss function is constructed as follows:

[0035]

[0036] in, , , These are the weight coefficients for each subtask; the convolutional neural network, the detector, the decoder, the Transformer encoder, and the multilayer perceptron are trained using this total joint loss function.

[0037] like Figure 3 As shown, this invention also proposes a perceptual optimization video coding quantization parameter prediction device based on an attention mechanism, which includes:

[0038] The video input module divides the original video into N video frames. , … };

[0039] The feature extraction module extracts the original feature map of each video frame using a convolutional neural network. , … }; Original feature map of each frame With a set of prior boxes The input is fed into the decoder, where K represents the maximum number of candidate boxes that the detector can output, and d is the dimension of the hidden vector;

[0040] The object detection module and the decoder utilize a multi-head attention mechanism to... With each query vector The interaction mechanism automatically focuses on key regions of interest in the image during the interaction process, and outputs a set of corresponding detection boxes and a semantic embedding vector for each detection. ;

[0041] The weighted multiplication module generates a binary mask for each detection box on the spatial grid of the original feature map, and then performs a union of all masks. After Gaussian smoothing, a soft attention map is obtained. , … }, then compare it with the original feature map { , … The weighted feature map is obtained by multiplying the features by channel weights. , … Based on the weighted feature maps, the image is divided into M fixed-size, non-overlapping image patches. }, where the integer j ranges from 1 to M, and a token is generated through linear mapping and positional encoding. };

[0042] The encoding module employs an L-layer Transformer encoder that utilizes multi-head self-attention and a feedforward network to capture tokens globally. The context and inter-frame relationships of};

[0043] The output of the last layer of the prediction module and the Transformer encoder { The quantization parameter values ​​of the corresponding patches are mapped one by one through a multilayer perceptron. The position of the corresponding patch in the frame is encoded using the quantization parameter value to obtain the encoded video of the original video.

[0044] The present invention also proposes an electronic device, including the aforementioned attention-based perception-optimized video coding quantization parameter prediction device, which may be connected to an information display device for displaying the coded video with user-set display parameters, attributes, or through an artificial intelligence model.

[0045] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the attention-based perceptual optimization video coding quantization parameter prediction method.

[0046] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the perceptual optimization video coding quantization parameter prediction method based on attention mechanism described above.

[0047] As can be seen from the above solutions, the advantages of the present invention are:

[0048] The attention-based perceptual optimization video coding quantization parameter prediction method described in this invention divides the original video into multiple frames and sequentially feeds them into a lightweight backbone network to extract deep features. Then, by combining learnable object queries with an improved DETR decoder, key regions of interest (ROIs) are automatically detected, generating a soft ROI attention map. This soft ROI attention map is then multiplied with the original features by channel weighting to obtain an enhanced weighted feature map. Based on this, the system divides the weighted feature map into several fixed-size patches, generates tokens through linear mapping and positional encoding, inputs them into a multi-layer lightweight Transformer encoder for global context modeling, and finally, regresses the predicted quantization parameters block by block through a multi-layer perceptual network, combining them into a frame-level QP allocation map. Driven by a joint loss function, this method achieves adaptive quantization intensity control that prioritizes ROI perception, integrates global context fusion, and balances lightweight real-time performance. Using the video quantization parameters output by this method for video coding can effectively improve the subjective visual quality of the video while ensuring coding efficiency. Attached Figure Description

[0049] Figure 1 A flowchart of the overall process for predicting quantization parameters of perceptually optimized video coding based on the attention mechanism according to the present invention is shown.

[0050] Figure 2 This is a flowchart of the method of the present invention;

[0051] Figure 3 This is a step diagram of the device of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0053] Figure 5 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0054] Figure 6 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0055] Figure label:

[0056] A - First electronic device;

[0057] B-Attention-based perceptual optimization video coding quantization parameter prediction device;

[0058] C-Data acquisition equipment;

[0059] D-Information display device;

[0060] 1000 - Second electronic device;

[0061] Ⅰ-Computational Unit;

[0062] II-ROM;

[0063] III-RAM;

[0064] N-bus;

[0065] V-Interface;

[0066] VI - Input Unit;

[0067] VII - Output Unit;

[0068] VIII - Storage medium;

[0069] IX - Communication Unit. Detailed Implementation

[0070] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0071] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0073] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0074] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0075] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0076] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0077] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0078] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0079] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0080] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0081] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] This invention provides a perceptual optimization method for predicting quantization parameters in video coding based on an attention mechanism. The method divides the original video into multiple frames and sequentially feeds them into a lightweight backbone network to extract deep features. Then, it combines these frames with learnable query vectors to automatically detect key regions of interest (ROIs) in a corresponding object detection model, generating a soft attention map. This soft attention map is then multiplied by the original features in a channel-weighted manner to obtain an enhanced weighted feature map. Based on this, the weighted feature map is divided into several fixed-size image blocks. Tokens are generated through linear mapping and positional encoding, and then input into a multi-layer lightweight Transformer encoder for global context modeling. Finally, the predicted quantization parameters are regressed block by block through a multi-layer perceptual network and combined to form a frame-level quantization parameter allocation map. In this process, through joint optimization of ROI priority loss and global consistency loss, adaptive quantization intensity control is achieved, balancing ROI perception priority, global context fusion, and lightweight real-time performance. The video coding quantization parameters generated using this method can significantly improve the subjective visual quality of the target ROI and the overall image while ensuring coding efficiency.

[0086] This invention provides a perceptual optimization method for predicting quantization parameters (QPs) in video coding based on an attention mechanism. By automatically detecting key regions of interest (ROIs) in images and combining them with a lightweight Transformer network, it achieves QP prediction at the image patch level. This improves the subjective visual quality of the video while maintaining coding efficiency, and takes into account the computational resource constraints of edge devices, effectively solving the technical problems existing in the prior art. QP refers to the coding quantization parameters, such as frame rate, resolution, bit rate, and other image compression parameters.

[0087] Specifically, this invention provides a perceptual optimization video coding quantization parameter prediction method based on an attention mechanism. This method includes the following steps: a video input step, a backbone feature extraction step, a target detection step, a ROI attention map generation step, a Transformer coding step, a QP prediction step, and a joint loss function optimization step. In the video input step, the original video is divided into multiple video frames. , … (N is an integer and N≥2), these N video frames are first fed into the backbone feature extraction step; in the feature extraction step, each frame is processed by a lightweight convolutional neural network with shared parameters to extract a depth spatial feature map. , … This preserves the consistency of features between frames and local details; the integer i ranges from 1 to N, and each frame's feature map... With a set of learnable object queries (Similar to "anchor boxes" or "prior boxes" in traditional detectors) are input together into the improved DETR decoder, where K represents the maximum number of candidate boxes output by the detector, and d is the dimension of the hidden vector; the decoder uses a multi-head attention mechanism to... With each query vector Interaction, where 0 represents the initial query vector (layer 0). During the interaction, the network automatically focuses on key regions of interest such as faces and vehicles in the image through a self-attention mechanism, and outputs the corresponding set of object detection boxes and the semantic embedding vector corresponding to each object detection box. Subsequently, a binary mask is generated for each detection box on the spatial grid of the original feature map, and the union of all masks is calculated. After Gaussian smoothing, a soft attention map is obtained. , … }, then combine it with the feature map { , … The channel-weighted multiplication enhances the response of the region of interest and suppresses background noise, thus obtaining a weighted feature map. , … Based on the weighted feature maps, the image is divided into M fixed-size, non-overlapping patches. }, where the integer j ranges from 1 to M, and a series of tokens are generated through linear mapping and positional encoding. }, This represents the initial token representation of the j-th patch in the i-th frame. These token sequences are then input into a lightweight L-layer Transformer encoder, which uses multi-head self-attention and a feedforward network to capture context and inter-frame relationships globally. The encoder's final layer outputs { The predicted quantization parameters for each corresponding patch are mapped through a multilayer perceptron (MLP) regression model. The quantization parameters (qp) of each patch are used to construct a frame-level Qp map. The predicted quantization parameters of multiple patches across a video frame are combined to form the frame-level Qp allocation map. The multi-frame Qp map obtained through this process adaptively adjusts the video quantization intensity.

[0088] To ensure detection accuracy, QP prediction accuracy, and visual quality of the compressed video, a joint loss function was adopted during the training phase. This function combines DETR classification and bounding box regression loss, QP mean squared error regression loss, and VMAF-based perceptual quality loss. This joint optimization was applied to the backbone network, detection head, Transformer encoder, and QP prediction head, thereby significantly improving the visual fidelity of key areas and overall perceptual quality while maintaining real-time performance.

[0089] Preferably, the original video may include video data of different types, different resolutions, and in various scenarios.

[0090] Preferably, the backbone feature extraction step employs a lightweight MobileNetV3 convolutional branch to reduce the number of model parameters while maintaining feature representation capability; the extracted 3D feature maps of the video frames are... Where H′ and W′ are the height and width of the feature map, respectively, and C is the number of channels. As i increases from 1 to N, N feature maps are formed. , … .

[0091] Preferably, the weighted feature map is denoted as ,in It is the weighted feature value of the i-th frame at spatial location (u,v) and channel c; It is the target detection bounding box (xk,i,yk,i,wk,i,hk,i). It is a soft attention map that has undergone Gaussian filtering. λ is a coefficient that controls the enhancement intensity. The initial value can be set to 0.5 and adaptively adjusted during training through inverse gradient. Optionally, λ can be dynamically adjusted according to indicators such as scene brightness and motion amplitude to improve robustness in weak texture or high motion scenes.

[0092] Optionally, after the binary mask is generated, the ROI attention map generation step may apply a learnable Gaussian kernel or bilateral filtering to the mask edges to smooth the transition region.

[0093] Preferably, the DETR decoder is an efficient variant that enables a detection frame rate of ≥25 FPS; the Transformer encoder is a lightweight structure with L≤6 layers and ≤4 heads per layer, and may optionally introduce local or sparse attention mechanisms.

[0094] Preferably, the joint loss function used during the training phase Weight It can be set to 1:5:2 to balance detection accuracy and perception quality.

[0095] In summary, the attention-based perceptual optimization video coding quantization parameter prediction method described in this invention divides the original video into multiple frames and sequentially feeds them into a lightweight backbone network to extract deep features. Then, by combining learnable object queries, key regions of interest (ROIs) are automatically detected in an improved DETR decoder, generating a soft ROI attention map. This attention map is then multiplied by the original features in a channel-weighted manner to obtain an enhanced weighted feature map. Based on this, the system divides the weighted feature map into several fixed-size patches, generates tokens through linear mapping and positional encoding, inputs them into a multi-layer lightweight Transformer encoder for global context modeling, and finally, regresses the predicted quantization parameters block by block through a multi-layer perceptual network, combining them into a frame-level QP allocation map. Driven by a joint loss function, this method achieves adaptive quantization intensity control that prioritizes ROI perception, integrates global context, and balances lightweight real-time performance. Using the video quantization parameters output by this method for video coding can effectively improve the subjective visual quality of the video while ensuring coding efficiency.

[0096] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0097] In summary, this invention provides a perceptually optimized video coding quantization parameter prediction method based on an attention mechanism. This method effectively mines perceptually important region information (such as faces, moving subjects, etc.) in videos by introducing an object detection mechanism and an attention-guided strategy. It combines a lightweight convolutional feature extraction network and a multi-layer Transformer structure to enhance semantic and contextual modeling capabilities for key regions of interest while maintaining computational efficiency. By guiding the encoder to focus on important regions in spatial and temporal dimensions, it achieves fine-grained prediction of quantization parameters (QP) at the block level, thereby enabling adaptive adjustment of video quantization parameters. Furthermore, this method designs a joint optimization strategy based on multi-loss fusion, simultaneously introducing object detection loss, QP regression loss, and perceptual quality loss based on the VMAF metric during the training phase. These losses jointly guide the parameter learning of the backbone network, the Transformer encoder, and the QP prediction step, thereby significantly improving the visual fidelity of the encoded video while maintaining compression ratio. The following will combine... Figure 1 The present invention describes in detail the specific structure and processing flow of the perceptual optimization video coding quantization parameter prediction method based on the attention mechanism according to the present invention.

[0098] Figure 1 A flowchart illustrating the implementation of the perceptual optimization video coding quantization parameter prediction method based on the attention mechanism according to the present invention is shown.

[0099] like Figure 1 As shown, the original video first enters the video input step for video processing, dividing the continuous video content into N video frames according to the timeline. , … }, where N is an integer and N≥2. The subsequent N video frames will be fed into the backbone feature extraction step to extract frame-level spatial semantic information. In the backbone feature extraction step, a lightweight convolutional neural network is used to encode features for each frame, thereby extracting the corresponding depth spatial feature map { , … },satisfy:

[0100]

[0101] in, The backbone network is represented by H′ and W′, which are the height and width of the feature map, respectively, and C is the number of channels. This network architecture with shared parameters ensures consistency and efficiency in cross-frame processing while preserving local structural and semantic information, providing stable input for subsequent target region extraction.

[0102] It should be noted that the video input step and feature extraction step described in this invention have strong adaptability and can support video data from different sources, including but not limited to video data of different types, different resolutions and in various scenarios.

[0103] For example, in intelligent surveillance applications, video content may contain key targets such as pedestrians, cars, and traffic signs on the street. This invention can accurately identify these targets through subsequent region of interest detection steps and guide the coding system to allocate higher coding resources to key areas, thereby maintaining good image quality at low bit rates.

[0104] like Figure 1 As shown, in obtaining the depth spatial feature map of each frame of video { , … Afterwards, these feature maps will be compared with a set of learnable query vectors (object queries). The improved DETR decoder step is input together, where K represents the maximum number of targets the model can detect, and d is the dimension of the query vector. This step uses a multi-head cross-attention mechanism to align and match each query vector with the region information in the feature map, thereby extracting key regions of interest in the image. The final output of the DETR decoder includes: a set of K bounding boxes { ... }, where 1 ≤ i ≤ N, represents the region of interest detected in the i-th frame; the semantic embedding vector corresponding to each bounding box is { ... }

[0105] Subsequently, in the original feature map Above, a corresponding binary mask image is generated based on each bounding box. All mask images are unioned along the channel dimension and smoothed using a Gaussian function to obtain a soft attention map with continuous values. This represents the importance of each spatial location in the current frame. To highlight regions of interest and suppress redundant background information, the original feature map... Will be related to attention map Perform channel-by-channel weighted fusion to obtain a weighted feature map. , where ⊙ represents element-wise multiplication by channel. The final output is { , … This will be used for subsequent Transformer encoder processing.

[0106] It should be noted that the object detection step described in this invention, by introducing an improved DETR decoder, can automatically model salient regions in video frames without the need for manual anchor box design. The object queries are a set of learnable position embedding vectors, which are continuously optimized during training and ultimately used in the decoder to interact with image features to predict the object's category and bounding box coordinates. Simultaneously, the attention map generation process requires no external supervision, relying entirely on the semantic alignment results within the network, thus exhibiting good adaptability. Pre-labeling of objects of interest in the image is unnecessary. DETR can be trained on the object queries and DETR decoder without manual annotation because the network automatically generates "labels" through self-supervised or pseudo-supervised pre-training, rather than obtaining them from manual annotation.

[0107] like Figure 1 As shown, in obtaining the weighted feature map Then, combining the feature maps of each frame, the image is divided into M fixed-size, non-overlapping image patches. Each image patch has a spatial size of P×P, and each image patch is flattened and converted into a fixed-dimensional embedding vector using a linear mapping function. To preserve positional information, corresponding positional encoding is further added to each token, thus forming the input sequence. ,in This represents the token sequence corresponding to the i-th frame image. .

[0108] Subsequently, the token sequence is fed into a lightweight Transformer encoder containing L layers. This encoder employs a multi-head self-attention mechanism and a feedforward network structure to model cross-patch and intra-frame global context information.

[0109]

[0110] in, This represents the sequence of hidden states output by the encoder. Each vector... This represents the global features extracted by the j-th patch after fusion context.

[0111] Next, the above output feature sequence The input is fed into a multilayer perceptron (MLP) consisting of multiple fully connected layers, which predicts the quantization parameter (QP) value for each image patch one by one. , where j increases sequentially from 1 to M. The final QP prediction values ​​for all image patches. It is reorganized into a frame-level QP allocation map to guide subsequent video encoders to adaptively adjust quantization parameters at the block level, thereby achieving fine compression of key regions.

[0112] like Figure 1 As shown, during model training, to achieve high-precision region of interest detection, intra-frame block-level QP prediction, and overall optimization of the final video perceptual quality, this invention constructs a joint loss function composed of multiple sub-objectives. This function achieves synergistic optimization by simultaneously minimizing three key loss terms: object detection loss, QP prediction loss, and perceptual quality loss.

[0113] Specifically, the loss in the object detection part consists of the standard object detection loss function of the DETR network, including matching loss, bounding box regression loss, and classification cross-entropy loss, which can be expressed as:

[0114]

[0115] in, Represents the category cross-entropy loss. The L1 loss represents the location of the bounding box. Let λ1 and λ2 represent the generalized IOU loss, where λ1 and λ2 are weighting coefficients.

[0116] It's important to note that object detection losses include class cross-entropy, L1 regression, and GIoU, requiring truth-labeled object classes and bounding boxes. The "unsupervised" approach mentioned earlier refers to the pre-training phase, where the query vector and decoder are initialized using self-supervised / pseudo-supervised tasks, followed by fine-tuning of the detection head. These two methods are not contradictory; it's a process of pre-training + supervised fine-tuning. First, establish a solid foundation for detection using unsupervised methods, then refine the model using supervised methods.

[0117] The generalized IOU loss introduces a penalty term on top of the standard IoU for the non-overlapping regions within the bounding boxes of the predicted and ground truth boxes (i.e., the maximum rectangle range of C). Here, A is the predicted box (candidate box), B is the ground truth box (pre-labeled bounding box), and C is the minimum closure rectangle region of A and B.

[0118] GIoU=IoU(A,B)−∣C\(A∪B)∣ / ∣C∣

[0119] In the formula, C\(A∪B) represents an element that exists in set C but not in set A or set B, and \ represents the difference between the sets.

[0120] Meanwhile, in order to regress the predicted value of the quantization parameter QP for each patch, let the predicted value of the j-th patch in the i-th frame be... Its true QP label is The defined intra-frame QP prediction loss is:

[0121]

[0122] To further improve the perceptual quality of the video, a perceptual loss term based on VMAF (Video Multi-method Assessment Fusion) is introduced during the optimization process. Let the VMAF score between the original frame and the reconstructed frame be... VMAF is a metric that measures the perceived image quality of a reconstructed frame relative to the original frame, with a score of 0-100. The target reference score is... The VMAF score, which is usually automatically calculated / estimated by the encoder or training framework, is used as a reference target for the "perceptual loss" during training. The perceptual quality loss is then:

[0123]

[0124] Combining the above loss terms, a total joint loss function is constructed. The backbone feature extraction network, DETR detection module, Transformer encoder, and multilayer perceptron were trained as follows:

[0125]

[0126] in, , , These are the weight coefficients for each subtask, and the weight relationship between detection accuracy, QP regression error, and perception quality optimization is balanced by adjusting the hyperparameters.

[0127] Up to this point, the method provided by this invention has completed feature modeling, key region perception, and quantization parameter prediction of the input video through an end-to-end trained deep neural network model. The model's final output is a frame-level QP allocation map for each frame of the image. Each allocation map contains the prediction quantization parameters for all image patches. .

[0128] Next, these predicted QP allocation maps will be used as a guide and input into the video encoder to achieve block-level adaptive quantization control, thereby significantly improving the overall video compression efficiency and perceived quality while ensuring the visual quality of key areas.

[0129] This concludes the basic introduction to the attention-based perceptual optimization video coding quantization parameter prediction method provided by this invention. In summary, this method divides the original video into multiple frames and sequentially feeds them into a lightweight backbone network to extract deep features. Then, by combining learnable object queries, key regions of interest (ROIs) are automatically detected in an improved DETR decoder, generating a soft ROI attention map. This attention map is then multiplied by the original features in a channel-weighted manner to obtain an enhanced weighted feature map. Based on this, the system divides the weighted feature map into several fixed-size patches. Tokens are generated through linear mapping and positional encoding, input into a multi-layer lightweight Transformer encoder for global context modeling, and finally, the predicted quantization parameters are regressed block by block through a multi-layer perceptual network and combined into a frame-level QP allocation map. Driven by a joint loss function, this method achieves adaptive quantization intensity control that prioritizes ROI perception, integrates global context, and balances lightweight real-time performance. Using the video quantization parameters output by this method for video coding can effectively improve the subjective visual quality of the video while ensuring coding efficiency.

[0130] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0131] like Figure 3 As shown, this invention also proposes a perceptual optimization video coding quantization parameter prediction device based on an attention mechanism, which includes:

[0132] The video input module divides the original video into N video frames. , … };

[0133] The feature extraction module extracts the original feature map of each video frame using a convolutional neural network. , … }; Original feature map of each frame With a set of prior boxes The input is fed into the decoder, where K represents the maximum number of candidate boxes that the detector can output, and d is the dimension of the hidden vector;

[0134] The object detection module and the decoder utilize a multi-head attention mechanism to... With each query vector The interaction mechanism automatically focuses on key regions of interest in the image during the interaction process, and outputs a set of corresponding detection boxes and a semantic embedding vector for each detection. ;

[0135] The weighted multiplication module generates a binary mask for each detection box on the spatial grid of the original feature map, and then performs a union of all masks. After Gaussian smoothing, a soft attention map is obtained. , … }, then compare it with the original feature map { , … The weighted feature map is obtained by multiplying the features by channel weights. , … Based on the weighted feature maps, the image is divided into M fixed-size, non-overlapping image patches. }, where the integer j ranges from 1 to M, and a token is generated through linear mapping and positional encoding. };

[0136] The encoding module employs an L-layer Transformer encoder that utilizes multi-head self-attention and a feedforward network to capture tokens globally. The context and inter-frame relationships of};

[0137] The output of the last layer of the prediction module and the Transformer encoder { The quantization parameter values ​​of the corresponding patches are mapped one by one through a multilayer perceptron. The corresponding patch is encoded within the frame using the quantization parameter value to obtain the encoded video of the original video. This allows for different encoding strategies to be applied to different patches within the frame, using high-definition encoding for patches involving critical targets and high-compression encoding for patches containing non-critical targets, thus balancing subjective visual quality, video size, and encoding efficiency.

[0138] like Figure 4 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the aforementioned attention mechanism-based perception-optimized video coding quantization parameter prediction device B.

[0139] like Figure 5As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire the video to be compressed and encoded, and the information display device D is used to display the encoded video obtained by the dynamic color coding of the present invention.

[0140] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0141] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the attention-based perceptual optimization video coding quantization parameter prediction method provided by the above methods.

[0142] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that performs the aforementioned attention-based perceptual optimization video coding quantization parameter prediction method. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0143] Figure 6 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0144] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0145] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S6. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0147] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. An attention mechanism based perceptual optimization video coding quantization parameter prediction method, characterized in that, Comprise: video inputting step, dividing the original video into N video frames {X1, X2, …, XN}; N}​ Feature extraction steps: Extract the original feature maps {F1, F2, ..., F...} of each video frame using a convolutional neural network. N }; Original feature map F for each frame i With a set of prior boxes The input is fed into the decoder, where K represents the maximum number of candidate boxes that the detector can output, and d is the dimension of the hidden vector; Target detection step, the decoder passes F i with each query vector interaction, in the interaction process, automatically focus on the key interest area in the image through self-attention mechanism, and output the corresponding detection frame set and the semantic embedding vector corresponding to each detection b k,i is the target detection frame (x k,i ,y k,i ,w k,i ,h k,i ) The weighted multiplication step, generating a binary mask for each detection box on the spatial grid of the original feature map and taking the union of all masks, and obtaining a soft attention map {R1, R2, …, R N} after Gaussian smoothing processing, and then multiplying it with the original feature map {F1, F2, …, F N} by channel to obtain the weighted feature map On the basis of the weighted feature map, the image is divided into M fixed size and non-overlapping image blocks patch {x j}, and the integer j ranges from 1 to M, and is generated by linear mapping and position coding represents the initial token representation of the jth patch in the ith frame image; The encoding step, the Transformer encoder with L layers, captures the context and inter-frame relationships in a global scope using multi-head self-attention and feed-forward networks ; the output of the last layer of the Transformer encoder mapping out quantization parameter values corresponding to patches through a multi-layer perception network encoding the location of the corresponding patch in the frame picture with the quantization parameter value to obtain the encoded video of the original video. 2.The attention mechanism based perception optimized video coding quantization parameter prediction method of claim 1, wherein, The weighted feature map wherein is a soft attention map after a Gaussian filtering operation, λ is a coefficient for controlling the enhancement intensity, F i (u, v, c) is the feature value of the i-th frame at the spatial position (u, v) and the channel c. 3.The attention mechanism based perception optimized video coding quantization parameter prediction method of claim 1, wherein, After the binary mask is generated, a Gaussian kernel or a bilateral filter can be applied to the mask edge to smooth the transition area; the number of layers of the Transformer encoder is L≤6 layers, the number of heads of each layer is ≤4, and a local or sparse attention mechanism is used. 4.The attention mechanism based perception optimized video coding quantization parameter prediction method of claim 1, wherein, In the feature extraction step, the convolutional neural network is used to encode features of each frame of image, thereby extracting corresponding deep spatial feature maps {F1, F2, …, F N} wherein, represents a convolutional neural network, H' and W' are the height and width of the feature map respectively, and C is the number of channels; the convolutional neural network is through a network architecture of shared parameters.

5. The method of claim 1, wherein the method is based on an attention mechanism. The weighted multiplication step comprises: In the original feature map Above, a corresponding binary mask image is generated based on each bounding box. H' and W' are the height and width of the feature map, respectively; all mask maps are unioned along the channel dimension and smoothed using a Gaussian function to obtain a continuous soft attention map. Original feature map F i Will be related to attention map Perform channel-by-channel weighted fusion to obtain a weighted feature map. Where ⊙ represents element-wise multiplication by channel; The image is divided into M fixed-size and non-overlapping image blocks patch in combination with the feature map of each frame, the spatial size of each image block is PXP, and each image block is flattened and converted into a fixed-dimension embedding vector through a linear mapping function; a corresponding position code is added to each token to form an input sequence T i = {t i,1 , t i,2 , ..., t i,M} , wherein T i represents a token sequence corresponding to the i-th frame image, The encoding step comprises: The Transformer encoder adopts a multi-head self-attention mechanism and a feedforward network structure: wherein Z i represents a sequence of hidden states output by the Transformer encoder, each vector z i,j represents the global feature extracted by the jth patch after fusing the context; The prediction step comprises: The output of the last layer of the Transformer encoder The output of the last layer of the Transformer encoder Wherein j increases from 1 to M in turn, MLP is a multi-layer perception network; all image block QP prediction values The image block QP prediction values are reorganized into a frame-level QP allocation map for guiding the subsequent video encoder to adaptively adjust the quantization parameter at the block level, thereby realizing fine compression of the key area.

6. The method of claim 1 to 5, wherein the method is based on attention mechanism. Further comprising a model training step: According to the pre-labeled target category and boundary box and the prediction category and candidate box output by the decoder, a target detection loss L is constructed The target detection loss wherein, represents a category cross-entropy loss, represents an L1 loss of the boundary box position, represents a generalized IOU loss, and λ1 and λ2 are weight coefficients. Let the predicted value of the jth patch in the ith frame be with its true QP label being q i,j The defined Intra QP prediction loss is: To further improve the perceptual quality of the video, a perceptual loss term based on the video quality score is introduced in the optimization process, where the video quality score between the original frame and the reconstructed frame is denoted as v i , the target reference score is denoted as , and the perceptual quality loss is: The total joint loss function is constructed as follows: Wherein, α, β, γ are the weight coefficients of each subtask; the total joint loss function is used to train the convolutional neural network, the detector, the decoder, the Transformer encoder and the multi-layer perception network.

7. An attention mechanism based perception optimized video coding quantization parameter prediction apparatus, characterized in that, Comprise: a video input module, dividing the original video into N video frames {X1, X2, …, XN} N}; The feature extraction module extracts the original feature maps {F1, F2, ..., F...} of each video frame using a convolutional neural network. N }; Original feature map F for each frame i With a set of prior boxes The input is fed into the decoder, where K represents the maximum number of candidate boxes that the detector can output, and d is the dimension of the hidden vector; The target detection module, the decoder, through a multi-head attention mechanism, will F i With each query vector Interaction, in the interaction process, through the self-attention mechanism, automatically focus on the key interest area in the image, and output the corresponding detection frame set and the semantic embedding vector corresponding to each detection b k,i Is the target detection frame (x k,i ,y k,i ,w k,i ,h k,i ) The weighted multiplication module generates a binary mask for each detection box on the spatial grid of the original feature map and takes the union of all masks, and obtains a soft attention map {R1, R2,..., R N} through Gaussian smoothing processing, and then performs channel-wise weighted multiplication with the original feature map {F1, F2,..., F N} to obtain a weighted feature map On the basis of the weighted feature map, the image is divided into M fixed-size and non-overlapping image blocks patch {x j}, and the integer j ranges from 1 to M, and is generated by linear mapping and position encoding represents the initial token representation of the jth patch in the ith frame image; The encoding module, a Transformer encoder with L layers, utilizes multi-head self-attention and feed-forward networks to capture context and inter-frame relationships in a global scope; The output of the last layer of the prediction module, the Transformer encoder The quantization parameter value corresponding to the patch is mapped through a multi-layer perception network one by one The position of the corresponding patch in the frame picture is encoded with the quantization parameter value to obtain the encoded video of the original video.

8. An electronic device, comprising: The electronic device or the information display device connected thereto is used to display the encoded video with display parameters, attributes set by a user or through an artificial intelligence model. 9.A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method for predicting a quantization parameter of a perception-optimized video encoding based on an attention mechanism according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method for predicting a quantization parameter of a perception-optimized video encoding based on an attention mechanism according to any one of claims 1-6.

Images