Image quality enhancement method and device, model training method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image quality enhancement method and device, a model training method and device, a storage medium and a program product. In the image quality enhancement method, a reconstructed frame obtained by decoding a compressed frame and decoding intermediate state data generated in the decoding process can be obtained, and the decoding intermediate state data comprises pixel data and coding parameters. Performing feature processing on the pixel data and the reconstruction frame based on coding prior information in the coding parameters to obtain fusion features; and inputting the fusion feature into the target enhancement model to obtain residual data, and generating an enhancement frame according to the reconstruction frame and the residual data. Pixel data in a decoding process can provide pixel-level time domain or space domain prior information for enriching feature expression, and coding parameters can provide coding prior information for guiding feature processing, so that a target enhancement model can perceive distortion characteristics of a reconstructed frame based on the prior information, compression distortion can be recovered more accurately, and the compression efficiency is improved. And the quality of a picture with compression loss is improved.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a method, device, storage medium, and program product for image enhancement and model training. Background Technology

[0002] With the widespread adoption of high-definition and ultra-high-definition video, efficiently transmitting video under limited bandwidth has become a key challenge. In some applications, newer international video coding standards, such as H.265 / HEVC and H.266 / VVC, are widely used for video encoding. Compared to earlier standards (such as H.264 / AVC), these newer international video coding standards can significantly reduce the bitrate while maintaining the same image quality, thus improving compression efficiency. However, at lower bitrates, the strong compression used during encoding can lead to distortions in the decoded video, such as blockiness, blurring, and ringing, affecting the viewing experience.

[0003] Some solutions use deep learning-based post-processing techniques to correct compression distortion. However, existing deep learning-based post-processing techniques have limited ability to correct compression distortion and are insufficient to significantly improve image quality. Therefore, a better solution is needed. Summary of the Invention

[0004] This application provides an image quality enhancement and model training method, device, storage medium, and program product to improve the quality of images with compression loss.

[0005] This application provides an image quality enhancement method, comprising: acquiring a reconstructed frame obtained by decoding a compressed frame and decoding intermediate state data generated during the decoding process of the compressed frame, wherein the compressed frame is obtained by encoding an original frame, and the decoding intermediate state data includes at least one pixel data and at least one encoding parameter; performing feature processing on the reconstructed frame and the at least one pixel data based on encoding prior information in the at least one encoding parameter to obtain fusion features; inputting the fusion features into a target enhancement model to obtain residual data; and generating a first enhancement frame based on the reconstructed frame and the residual data.

[0006] This application also provides a model training method, comprising: acquiring reconstructed frame samples obtained by decoding compressed frame samples and decoding intermediate state data samples generated during the decoding process of the compressed frame samples, wherein the compressed frame samples are obtained by encoding the original frame samples, and the decoding intermediate state data samples include at least one pixel data sample and at least one encoding parameter sample; performing feature processing on the reconstructed frame samples and the at least one pixel data sample based on the encoding prior information in the at least one encoding parameter sample to obtain fused feature samples; inputting the fused feature samples into a target augmentation model to obtain residual data samples; generating a first augmentation frame sample based on the reconstructed frame samples and the residual data samples; determining the augmentation loss of the target augmentation model based on the first augmentation frame sample and the original frame samples; and training the target augmentation model with the goal of optimizing the augmentation loss.

[0007] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.

[0008] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0009] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0010] In this embodiment, the reconstructed frame obtained by decoding the compressed frame and the intermediate decoding state data generated during the decoding process can be acquired. The intermediate decoding state data includes pixel data and encoding parameters. Based on the encoding prior information in the encoding parameters, feature processing can be performed on the pixel data and the reconstructed frame to obtain fused features. The fused features are input into the target enhancement model to obtain residual data, and an enhanced frame is generated based on the reconstructed frame and the residual data. In this process, the pixel data during the decoding process can provide the target enhancement model with pixel-level temporal or spatial prior information to enrich the feature expression, and the encoding parameters can provide encoding prior information to guide feature processing. This allows the target enhancement model to perceive the distortion characteristics of the reconstructed frame based on this prior information, thereby more accurately restoring compression distortion and improving the quality of images with compression loss. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic flowchart of an image quality enhancement method provided for an exemplary embodiment of this application; Figure 2 A schematic diagram of the trimming process provided for an exemplary embodiment of this application; Figure 3 A schematic diagram illustrating cross-block cell clipping and non-cross-block cell clipping provided for an exemplary embodiment of this application; Figure 4 A schematic flowchart of a model training method provided for an exemplary embodiment of this application; Figure 5 A schematic diagram of a model training process provided for an exemplary embodiment of this application; Figure 6 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one. “A plurality” generally includes at least two, but does not exclude the inclusion of at least one.

[0014] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0015] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0016] In some post-processing techniques that utilize deep learning for compression distortion restoration, convolutional neural networks (CNNs) and residual learning strategies are employed to achieve enhancement operations such as denoising and deblocking by predicting the distortion residuals introduced by compression. This approach has achieved certain performance improvements in objective metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). However, this method typically uses a single model to process all color channels uniformly, failing to fully consider the significant differences between luminance (Y) and chrominance (U / V) channels in terms of data distribution, human eye sensitivity, and distortion characteristics. Furthermore, most of these schemes are trained and inferred based on low-precision 8-bit data, ignoring the richer dynamic range and detailed information contained in the intermediate states of higher bit depth (e.g., 10-bit) decoding, thus limiting the model's ability to model and restore subtle compression distortions. Consequently, it is difficult to significantly improve image quality.

[0017] To address the aforementioned technical problems, a solution is provided in some embodiments of this application. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating an exemplary embodiment of an image quality enhancement method provided in this application. The method may include, for example: Figure 1 The steps shown are as follows: Step 101: Obtain the reconstructed frame obtained by decoding the compressed frame and the intermediate decoding state data generated during the decoding process of the compressed frame. The compressed frame is obtained by encoding the original frame. The intermediate decoding state data includes at least one pixel data and at least one encoding parameter.

[0019] Step 102: Based on the prior information of the encoding in the at least one encoding parameter, perform feature processing on the reconstructed frame and the at least one pixel data to obtain fused features.

[0020] Step 103: Input the fused features into the target enhancement model to obtain residual data.

[0021] Step 104: Generate a first enhanced frame based on the reconstructed frame and the residual data.

[0022] This application is used to enhance compressed frames to improve their image quality.

[0023] In step 101, the original frame refers to the uncompressed input image or video frame, i.e., the raw image data received by the encoder. The original frame has full spatial resolution and a specified bit depth (e.g., 8-bit, 10-bit). Bit depth describes the precision that each pixel's color or luminance component can represent, i.e., how many bits are used to store the value of a pixel component. The compressed frame refers to the bitstream generated by the encoder after encoding the original frame; it is frame data existing in compressed bitstream form. A compressed frame is a set of syntax elements stored in the bitstream as binary encoding. Syntax elements may include: prediction mode, motion vector, quantization coefficients, quantization parameters (QP), etc. The compressed frame can be restored to pixel data by the decoder. The reconstructed frame refers to the image frame recovered by the decoder after completing steps such as inverse quantization, inverse transform, prediction synthesis, and loop filtering. The reconstructed frame is used both for final display and as a reference frame for subsequent frames in inter-frame prediction.

[0024] In this embodiment, the intermediate state data refers to the internal pixel or syntax information generated during the encoder's decoding of the compressed frame to obtain the reconstructed frame, which has not yet completed the entire processing flow. This data reflects the gradual recovery process of the image content by the decoder during the decoding stage and is used as prior information to guide the image quality enhancement process.

[0025] In this embodiment, the intermediate state data for decoding may include at least one pixel data and at least one encoding parameter. Pixel data refers to intermediate data existing in the form of pixel values ​​during the encoding or reconstruction process. This intermediate data is located in the pixel domain and has a definite spatial location and luminance / chrominance values. Pixel data can reflect the reconstruction or prediction state of pixel-level content and can provide pixel-level spatial or temporal prior information.

[0026] In video encoding, encoding parameters refer to metadata or control information used to control or describe encoding decisions, syntax structure, and compression operations. Encoding parameters do not directly represent pixel values, but they determine how pixel data is predicted, transformed, quantized, and filtered, providing prior information about content, structure, distortion characteristics, and motion characteristics.

[0027] In this embodiment, the intermediate decoding state data may include, but is not limited to, at least one of the following: prediction frames, residual blocks, initial reconstruction frames, coding unit (CU) partitioning maps, CU depth maps, filter intensity maps, motion vector fields, and quantization parameter maps. The prediction frame consists of prediction blocks, which are prediction signals generated through intra-frame or inter-frame prediction and used to synthesize reconstruction blocks with the residuals. The residual block refers to the residual signal obtained after inverse quantization and inverse transform of the quantization coefficients parsed from the bitstream; it is the difference between the original pixel block and the prediction block, reflecting information not covered by the prediction. Intra-frame prediction frames can provide prior information related to spatial structure and local content modeling, which can be used to effectively guide tasks such as reconstruction enhancement, artifact removal, and texture restoration. Inter-frame prediction frames embody the inter-frame prediction mode and can serve as prior information for the temporal prediction mode to guide the enhancement or repair process of the reconstruction frame. The initial reconstruction frame consists of initial reconstruction blocks, which are the reconstruction results after adding the prediction blocks and residual blocks, before undergoing loop filtering (such as deblocking filtering). The CU partition map describes the partitioning structure of coding units (CUs) in each region of an image, reflecting the quadtree / multi-type tree segmentation decision used during encoding, and providing prior information on spatial structure partitioning. The CU depth map represents the partitioning depth of the CU to which each pixel belongs (e.g., depth 0 represents 64×64, depth 3 represents 8×8), characterizing the local coding granularity and providing prior information on local content complexity. The higher the complexity of a CU, the greater its partitioning depth. The filter intensity map refers to the boundary strength used for deblocking filtering, such as the spatial distribution map of boundary strength (Bs) values ​​in H.265 / HEVC, providing prior information on the distortion type and restoration intention determined by the encoder. The motion vector field describes the distribution of motion vectors corresponding to each block in inter-frame prediction, providing prior information related to temporal motion trajectories and consistency. The quantization parameter map is used to represent the spatial distribution of QP values ​​for each CU or Transform Block (TB), reflecting the local compression intensity and providing prior information on local distortion / fidelity.

[0028] In the intermediate state data of decoding, at least one pixel data may include at least one of the predicted frame generated during the decoding process to obtain the reconstructed frame and the initial reconstructed frame. At least one encoding parameter may include at least one of the following: CU partition map, CU depth map, prediction mode, quantization parameter, motion vector field, and filter intensity map. In step 102, feature processing refers to a series of transformation, selection, enhancement, or reconstruction operations performed on the input data or intermediate representation to generate a feature representation more suitable for downstream tasks. The feature processing process may include at least one of feature extraction, feature adjustment, and feature mapping. Feature extraction refers to using a neural network-based feature extractor to map the input data (i.e., the reconstructed frame and the intermediate state data of decoding) into low-dimensional or high-dimensional semantic feature vectors to capture the inherent structure and implicit information of the input data. Feature adjustment may include at least one of the following operations: feature weighting, feature filtering, feature normalization, feature fusion, and feature enhancement. Encoding prior information refers to prior information generated by the encoder during the video or image compression process that is related to at least one of the following intermediate decision information: the structural organization of the coding unit, distortion characteristics, block partitioning strategy, and content characteristics (such as texture complexity, residual distribution, etc.). This information encapsulates the encoder's understanding of the original content and structure of the original frame, providing effective guidance signals for the enhancement or restoration process of the reconstructed frame. In this step, feature processing of the reconstructed frame and the at least one pixel data based on the prior coding information in the at least one coding parameter refers to using the prior coding information provided by the at least one coding parameter, such as content prior information, structural prior information, or semantic guidance, to adjust or influence the feature processing of the reconstructed frame and the at least one pixel data. This adjustment or influence can act on one or more stages of the feature processing process. For example, the at least one coding parameter can be used to adjust the stage of feature extraction of the reconstructed frame and the at least one pixel data, or the at least one coding parameter can be used to adjust the stage of feature adjustment of the reconstructed frame and the at least one pixel data. This embodiment does not limit this.

[0029] In this step, feature extraction is performed on the reconstructed frame and pixel data based on the prior information in the encoding parameters. This facilitates the perception encoder's understanding of the original content and structure of the original frame during the encoding process, thereby uncovering the causes of compression distortion. The uncovered information can then be used as guidance for image enhancement, resulting in more accurate image enhancement.

[0030] In step 103, the target augmentation model is trained based on sample data. During training, the target augmentation model learns the ability to output more accurate residual data based on the fusion features of the input reconstructed frame and the decoded intermediate state data. This residual data reflects the difference between the original frame and the reconstructed frame, and is used to compensate for information not covered during reconstruction. The residual data is used to compensate for missing information in the reconstructed frame, thereby enhancing the image quality of the reconstructed frame. The specific training process of the target model will be described in subsequent embodiments and will not be repeated here.

[0031] In step 104, the reconstructed frame and the residual data can be directly superimposed to generate the first enhanced frame. Alternatively, the reconstructed frame and the residual data can be weighted according to a set weighting coefficient to generate the first enhanced frame. The first enhanced frame is the enhanced result of the reconstructed frame, which can be used for direct display or as a reference frame for the next frame.

[0032] In this embodiment, the reconstructed frame obtained by decoding the compressed frame and the intermediate decoding state data generated during the decoding process can be acquired. The intermediate decoding state data includes pixel data and encoding parameters. Based on the encoding prior information in the encoding parameters, feature processing can be performed on the pixel data and the reconstructed frame to obtain fused features. The fused features are input into the target enhancement model to obtain residual data, and an enhanced frame is generated based on the reconstructed frame and the residual data. In this process, the pixel data during the decoding process can provide the target enhancement model with pixel-level temporal or spatial prior information to enrich the feature expression, and the encoding parameters can provide encoding prior information to guide feature processing. This allows the target enhancement model to perceive the distortion characteristics of the reconstructed frame based on this prior information, thereby more accurately restoring compression distortion and improving the quality of images with compression loss.

[0033] In some application scenarios, different quantization parameters may be used to control the quantization step size when encoding the original frame. A larger quantization parameter results in a larger quantization step size, lower bitrate, and greater distortion; conversely, a smaller quantization parameter results in a smaller quantization step size, higher bitrate, and less distortion. Reconstructed frames generated under different quantization parameters exhibit differentiated compression distortion characteristics, thus the difficulty and optimization objectives for distortion enhancement also differ. In this embodiment, the quantization parameters can be divided into different quantization parameter segments, and corresponding enhancement models can be trained for different quantization parameter segments. For example, in some embodiments, the quantization parameter range (0~63) can be divided into 7 parameter segments, and the lengths of different parameter segments can be the same or different.

[0034] Based on this, a model selection operation can be performed before inputting the fused feature into the target augmentation model. Optionally, the target quantization parameter segment to which the quantization parameter used to encode the original frame belongs can be determined from at least one quantization parameter segment. Quantization parameters are usually explicitly encoded in the compressed frame. The decoder can obtain the quantization parameters during the decoding process of the compressed frame to perform inverse quantization based on the quantization parameters. After determining the target quantization parameter segment to which the quantization parameter belongs, the target augmentation model can be determined from the augmentation models corresponding to multiple quantization parameter segments based on the target quantization parameter segment. It should be understood that the training process of augmentation models corresponding to different quantization parameter segments is obtained by training with sample data corresponding to different quantization parameter segments. The structure of augmentation models corresponding to different quantization parameter segments can be the same or different, and this embodiment does not impose any restrictions.

[0035] Based on this implementation method, the reconstructed frames corresponding to different quantization parameter segments can be enhanced independently and differentiated, so that the enhancement processing of the reconstructed frames under the quantization parameter segment matches the compression distortion characteristics of the reconstructed frames, thereby achieving fine-grained and highly matched image quality restoration operations.

[0036] In digital images or videos, pixel colors are typically described based on color components defined by a specific color model. A color component refers to the basic channels or dimensions used to represent color information. Different color models use different numbers and types of color components to describe pixel colors. For example, the RGB model (Red-Green-Blue) uses color components R (Red), G (Green), and B (Blue) to describe pixel colors, while the YU / V model (Luminance-Chroma) uses Y (Luminance component) and chromaticity components to describe pixel colors. The chromaticity components include U (Blue color difference component) and (Red color difference component).

[0037] Taking the YU / V model as an example, when compressing the original frame, pixels can be converted from the YU / V model to the YCbCr color space. Leveraging the human eye's greater sensitivity to brightness and less sensitivity to chroma, the chroma components (Cb, Cr) can be downsampled (e.g., using a 4:2:0 chroma sampling format). This significantly reduces data volume while maintaining subjective visual quality, achieving efficient compression. Different color components exhibit significant differences in data distribution. For instance, in terms of numerical distribution, when the original frame has low saturation, the Y component after conversion to the YCbCr color space has a wider distribution range and changes significantly with image brightness, while the Cb / Cr components are concentrated around the median (e.g., 128). In terms of spatial frequency, the Y component contains rich high-frequency details (such as edges and textures), while the Cb / Cr component is dominated by low frequencies and exhibits gentler spatial variations. In terms of information entropy, the Y component has high information entropy and low redundancy, making it difficult to compress; the Cb / Cr component has low information entropy and high redundancy, making it easy to compress. The aforementioned differences result in different compression distortion characteristics for different color components.

[0038] In some alternative embodiments, taking into account the different characteristics of compression distortion in different color components of the image, corresponding enhancement models can be trained for different color components. That is, the target enhancement model may include enhancement models for multiple color components individually.

[0039] Based on this, during the feature processing of the reconstructed frame and the decoding intermediate state data to obtain the fusion features, the fusion features corresponding to different color components can be obtained separately. Optionally, the reconstructed frames corresponding to each of the multiple color components can be determined from the reconstructed frame, and at least one pixel data and at least one encoding parameter corresponding to each of the multiple color components can be determined from the decoding intermediate state data. Then, based on the encoding prior information in the at least one encoding parameter corresponding to each of the multiple color components, feature extraction is performed on the reconstructed frames and at least one pixel data corresponding to the multiple color components to obtain the fusion features of each of the multiple color components. For example, when the multiple color components include a first color component and a second color component, the first reconstructed frame corresponding to the first color component and the second reconstructed frame corresponding to the second color component can be determined from the reconstructed frame, and the first decoding intermediate state data corresponding to the first color component and the second decoding intermediate state data corresponding to the second color component can be determined from the decoding intermediate state data. Subsequently, based on the prior coding information in at least one coding parameter in the first decoding intermediate state data, feature extraction can be performed on at least one pixel data in the first reconstructed frame and the first decoding intermediate state data to obtain the first fusion feature corresponding to the first color component. Similarly, based on the prior coding information in at least one coding parameter in the second decoding intermediate state data, feature extraction can be performed on at least one pixel data in the second reconstructed frame and the second decoding intermediate state data to obtain the second fusion feature. Likewise, when multiple color components contain color components other than the first and second color components, the above feature extraction operation can be performed on the other color components, which will not be elaborated further.

[0040] Based on this implementation method, prior information corresponding to each of the multiple color components can be extracted differentially through the reconstructed frames and decoding intermediate state data corresponding to each color component. This facilitates differentiated guidance of the image enhancement process for different color components, thereby achieving fine-grained image quality restoration.

[0041] The following will use any one of the multiple color components as an example to illustrate an optional implementation method for obtaining the fusion features of any one color component.

[0042] Optionally, for any one of the multiple color components, at least one pixel data and at least one encoding parameter corresponding to that color component can be determined from the decoding intermediate state data corresponding to that color component. Optionally, if the color component is a luminance component, then the at least one encoding parameter corresponding to that color component includes at least one of a decoding unit partition map, a decoding unit depth map, and a filter intensity map used in the process of decoding to obtain the reconstructed frame. If the color component is a chrominance component, then the at least one encoding parameter corresponding to that color component includes a filter intensity map used in the process of decoding to obtain the reconstructed frame. Based on this implementation, the different distortion characteristics of different color components can be utilized, and different decoding intermediate state data can be used for different color components, thereby facilitating targeted, differentiated, and high-quality image enhancement of the reconstructed frames corresponding to different color components.

[0043] After determining at least one pixel data and at least one encoding parameter corresponding to the color component, preliminary features of the reconstructed frame, the at least one pixel data, and the at least one encoding parameter corresponding to the color component can be extracted. In this step, a neural network-based feature extractor can be used to perform the preliminary feature extraction operation, without limiting the type and structure of the neural network. In some embodiments, the reconstructed frame, the at least one pixel data, and the at least one encoding parameter corresponding to the color component can be normalized to the [0, 1] interval, and then the feature extractor uses pixel rearrangement downsampling and convolutional layers to extract preliminary features. Pixel rearrangement downsampling refers to using the inverse process of pixel rearrangement to achieve downsampling, that is, by periodically reorganizing pixel positions, the high-resolution feature map is converted into a low-resolution feature map with more channels. This step is used to map the reconstructed frame, the at least one pixel data, and the at least one encoding parameter corresponding to the color component into the feature space to obtain their semantic features. Subsequently, along the channel dimension, the preliminary features of the reconstructed frame corresponding to the color component and the preliminary features of each of the at least one pixel data can be concatenated to obtain the first concatenated feature. Then, along the channel dimension, the preliminary features of each of the at least one encoding parameter corresponding to the color component can be concatenated to obtain the second concatenated feature. Concatenation along the channel dimension refers to stacking multiple feature maps together while maintaining their original size to form a multi-channel feature map.

[0044] After obtaining the first and second stitched features, feature extraction is performed on both features to obtain pixel features and encoding parameter features. In this step, a neural network-based feature extractor can be used, employing convolution to perform feature extraction operations on the first and second stitched features respectively. For example, for the luminance component, the preliminary features of the reconstructed frame, predicted frame, and preliminary reconstructed frame corresponding to the luminance component can be stitched together along the channel dimension, and convolution calculations are performed on the stitched features to form multi-channel pixel features. Similarly, the preliminary features of the CU partition map, CU depth map, and filter intensity map corresponding to the luminance component can be stitched together along the channel dimension, and convolution calculations are performed on the stitched features to form encoding parameter features. Pixel features characterize the pixel's inherent features and its compression distortion performance. Encoding parameter features characterize the encoder's structure, complexity, and compression behavior information during the encoding decision process, essentially acting as a "log" in the encoding process. Based on these two features, the target enhancement model can accurately identify the causes of compression distortion, thereby facilitating precise and high-quality image restoration.

[0045] After obtaining the pixel features and coding parameter features, the pixel features can be weighted based on the coding prior information in the coding parameter features to obtain the fusion features of the color components. Optionally, the pixel features and coding parameter features can be directly weighted and summed according to specified weighting summation coefficients to obtain the fusion features of the color components. Alternatively, coding prior information can be extracted from the coding parameter features, and the pixel features can be dynamically weighted based on the coding prior information to obtain the fusion features of the color components.

[0046] In some optional embodiments, an attention mechanism can be used to calculate the encoded parameter features to obtain an attention weight matrix, which represents the prior information of the encoded parameter features. After obtaining the attention weights, the attention weight matrix can be used to perform weighted calculations on the pixel features to obtain the fusion features of the color component. Optionally, the possible methods for calculating the encoded parameter features using the attention mechanism may include using the Softmax function or the Sigmoid function to calculate the encoded parameter features; this embodiment is not limited to these methods. For example, a linear transformation can be performed on the encoded parameter features to calculate their scores; then, the Sigmoid function can be used to convert the scores of the encoded parameter features into values ​​within the range of (0, 1), which are used as the attention weights for each pixel position.

[0047] Weighting pixel features involves assigning a weight to each channel of the pixel feature and then scaling the pixel values ​​of that channel using that weight. Assume the input feature map is F∈R. B×C×H×WB×C×H×W represents the number of channels in the feature map along its length, height, and width. The dynamically generated attention weight matrix is ​​w∈R. B×C×1×1 B×C×1×1 indicates that each channel has a scalar weight. The weighted input is then: Fout=w⊙F, where ⊙ represents broadcast multiplication, meaning all H×W pixels in each channel are multiplied by the same weight coefficient.

[0048] Based on this implementation, pixel features can be dynamically weighted using encoding parameter features, thereby deeply fusing the structural information of the compressed domain with the content information of the pixel domain. The encoding parameter features express the encoder's judgment of local image complexity, motion intensity, and distortion risk. Dynamically weighting pixel features using this information allows subsequent image enhancement processing to focus on areas that require enhancement or repair. Secondly, encoding parameter features can introduce prior coding information, providing clues about compression distortion to the image enhancement process, thus improving the enhancement effect. Furthermore, pixel features themselves struggle to distinguish between "real texture" and "compression artifacts" (such as blockiness and ringing). Encoding parameter features can help determine the potential location and type of artifacts, guiding subsequent image enhancement processing to focus on these areas.

[0049] In some optional embodiments, before performing feature processing based on the reconstructed frame and the intermediate decoding state data to obtain fused features, the reconstructed frame and the intermediate decoding state data can be cropped. Optionally, the reconstructed frame and the intermediate decoding state data can be cropped synchronously according to the target cropping start position and the target cropping resolution. In this embodiment, the target cropping start position can be randomly selected, specified, or determined according to a specific algorithm; this embodiment is not limited. The target cropping resolution corresponds to the expected resolution of the reconstructed frame. The expected resolution refers to the resolution that the final output reconstructed frame is expected to achieve in tasks such as video or image encoding / decoding and image restoration; for example, the expected resolution could be 1920×1080, 640×480, etc. The expected resolution can be manually specified. Figure 2 As shown, the target cropping start position can be spatial coordinates (x, y), and the target cropping resolution can be N*N, where N is a positive integer. Synchronous cropping of the reconstructed frame and the decoding intermediate state data, based on the target cropping start position and resolution, ensures that all channels are strictly aligned in spatial position, thus maintaining pixel-level consistency between the image content in the reconstructed frame and the prior information in the decoding intermediate state data. During the cropping process, geometric enhancement operations that disrupt spatial consistency, such as rotation or flipping, are not performed on the cropped blocks to avoid distortion of CU partitioning information, CU depth information, and filtering intensity, etc., in the encoded semantics.

[0050] In some optional embodiments, the target cropping resolution can be 1024×1024 to accommodate 1080p output requirements. That is, before inputting the reconstructed frame and the intermediate decoding data into the feature extractor for feature extraction, a 1024×1024 block can be cropped to the right and downwards from the same randomly selected spatial coordinates (x, y), without exceeding the original resolution boundaries. After cropping, no geometric enhancement operation is performed on the cropped block.

[0051] Based on this cropping method, on the one hand, irrelevant data in the reconstructed frame and decoded intermediate state data can be extracted, thereby reducing the amount of input data in subsequent stages and reducing computational overhead. On the other hand, cropping can remove non-content areas at the edges, allowing subsequent stages to focus on the effective content areas.

[0052] It is also worth noting that, because VVC processes data in blocks using coding units (CUs), the CU partitioning map, CU depth map, and filter intensity map all exhibit a clear block structure, such as... Figure 3 As shown. In this embodiment, the target cropping resolution corresponds to the expected resolution of the reconstructed frame. This expected resolution is usually a higher resolution, which helps to reduce the probability of cropping across block units, making it easier to retain more complete coding block structure information, thereby ensuring the spatial continuity and context richness of the coding parameters.

[0053] After obtaining the fusion features of each of the multiple color components based on the method provided in the foregoing embodiments, the fusion features of each of the multiple color components can be input into the enhancement model corresponding to the multiple color components in the target enhancement model to obtain the residual data of each of the multiple color components. For example, when the multiple color components include a first color component and a second color component, the first fusion feature corresponding to the first color component can be input into the first enhancement model corresponding to the first color component in the target enhancement model to obtain the first residual data; the second fusion feature corresponding to the second color component can be input into the second enhancement model corresponding to the second color component in the target enhancement model to obtain the second residual data.

[0054] Among them, the enhancement models corresponding to multiple color components have learned the ability to calculate the residual data corresponding to each of the multiple color components based on the fusion features of the multiple color components. Therefore, they can perform image enhancement differently based on the distortion characteristics corresponding to different color components, thereby improving image quality more accurately.

[0055] Optionally, after obtaining the residual data of each of the multiple color components based on the foregoing embodiments, the reconstructed frames corresponding to each of the multiple color components can be superimposed with the residual data of each of the multiple color components to obtain the enhanced frames of each of the multiple color components. For example, when the multiple color components include a first color component and a second color component, the first reconstructed frame and the first residual data corresponding to the first color component can be superimposed to obtain the first enhanced frame corresponding to the first color component; the second reconstructed frame and the second residual data corresponding to the second color component can be superimposed to obtain the second enhanced frame corresponding to the second color component. Then, the enhanced frames of each of the multiple color components are combined to obtain the first enhanced frame.

[0056] Based on this implementation method, when different color components have different distortion characteristics, the residual data of each color component is used to repair the reconstructed frames corresponding to different color components, achieving precise alignment between distortion characteristics and residual compensation. In this process, it is ensured that the enhancement of the luminance component does not affect the chrominance component, and the correction of the chrominance component does not interfere with the luminance structure, thus improving the overall image quality.

[0057] In the above and following embodiments of this application, the bit depth corresponding to the reconstructed frame and the decoded intermediate state data may be the same as or different from the bit depth corresponding to the original frame. For ease of description and distinction, the bit depth corresponding to the reconstructed frame and the decoded intermediate state data can be denoted as the first bit depth, and the bit depth corresponding to the original frame can be denoted as the second bit depth, where the second bit depth is the bit depth of the desired output enhanced frame. In some optional embodiments, the first bit depth may be greater than the second bit depth. That is, when extracting the reconstructed frame and the decoded intermediate state data from the decoder, the reconstructed frame and the decoded intermediate state data with a bit depth greater than that of the original frame and the desired output enhanced frame can be extracted. Assuming that the second bit depth corresponding to the original frame and the desired output enhanced frame is 8 bits, then the first bit depth corresponding to the reconstructed frame and the decoded intermediate state data can be 10 bits, 12 bits, or a larger bit depth, which is not limited in this embodiment. A larger bit depth can usually express more detailed information, and therefore can provide more prior information for the image enhancement process, thus helping to improve image quality.

[0058] Based on this, in some embodiments, after generating a first enhanced frame based on the reconstructed frame and the residual data, the first enhanced frame can be quantized according to the second bit depth to obtain a second enhanced frame adapted to the second bit depth, thereby ensuring that the quantized second enhanced frame meets the output requirements. For example, if the first bit depth is 10 bits and the second bit depth is 8 bits, when quantizing the first enhanced frame, the pixel values ​​represented by 10 bits in the first enhanced frame can be compressed to an 8-bit (0-255) range and stored using the uint8 data type, thereby obtaining an 8-bit second enhanced frame. Specifically, the first enhanced frame can be... Multiply by the maximum pixel value corresponding to the target bit depth (i.e., the second bit depth), map to an integer range, and then perform a rounding operation. The specific formula is as follows: in, This is the second enhanced frame after analog quantization. This represents the maximum pixel value, corresponding to an 8-bit bit depth. , This is the floor function.

[0059] Based on this implementation method, by using a reconstructed frame with a larger bit depth and decoding intermediate state data as prior information, a second enhanced image frame with a lower bit depth that matches the expected output bit depth can be obtained by quantizing the first enhanced frame, thus meeting the output requirements.

[0060] In addition to the foregoing embodiments, this application also provides a model training method for training the target augmentation model described in the foregoing embodiments. The following will be combined with... Figure 4 The training method for this model is illustrated by example.

[0061] like Figure 4 As shown, the main model training methods include: Step 401: Obtain the reconstructed frame sample obtained by decoding the compressed frame sample and the decoding intermediate state data sample generated during the decoding process of the compressed frame sample. The compressed frame sample is obtained by encoding the original frame sample. The decoding intermediate state data sample includes at least one pixel data sample and at least one encoding parameter sample.

[0062] Step 402: Based on the prior coding information in the at least one coding parameter sample, perform feature processing on the reconstructed frame sample and the at least one pixel data sample to obtain fused feature samples.

[0063] Step 403: Input the fused feature samples into the target enhancement model to obtain residual data samples.

[0064] Step 404: Generate a first enhanced frame sample based on the reconstructed frame sample and the residual data sample.

[0065] Step 405: Determine the enhancement loss of the target enhancement model based on the first enhanced frame sample and the original frame sample.

[0066] Step 406: Train the target augmentation model with the goal of optimizing the augmentation loss.

[0067] The optional implementations of steps 401-404 can be found in the descriptions related to steps 101-104 in the foregoing embodiments, and will not be repeated here. The target enhancement model trained in this embodiment is used to enhance the reconstructed frames of the compressed frames described in the foregoing embodiments.

[0068] In some optional embodiments, if the reconstructed frames corresponding to different quantization parameter segments are enhanced independently and differentially, enhancement models corresponding to different quantization parameter segments can be trained. Taking the original frame described in the aforementioned embodiment as an example, if the quantization parameters used to encode the original frame belong to the target quantization parameter segment, then the target enhancement model corresponding to the target quantization parameter segment can be trained using the sample data corresponding to the target quantization parameter segment. That is, the quantization parameter segment to which the quantization parameters used to encode the compressed frame sample belong is the same as the quantization parameter segment to which the quantization parameters used to encode the compressed frame belong. This allows the model to learn the encoding distortion characteristics corresponding to the target quantization parameter segment, thereby independently enhancing the reconstructed frame of the compressed frame corresponding to the target quantization parameter segment. It should be noted that this embodiment uses the target enhancement model as an example to illustrate the training method of the enhancement model corresponding to any quantization parameter segment. Enhancement models corresponding to different quantization parameter segments can be trained using training data corresponding to different quantization parameter segments and a training method similar to the target enhancement model, which will not be elaborated further. Based on this implementation method, enhancement models corresponding to different quantization parameter segments can be trained independently and differentiatedly, so that the enhancement processing of the reconstructed frame under the quantization parameter segment matches the compression distortion characteristics of the reconstructed frame, thereby achieving fine-grained and highly matched image quality restoration operations.

[0069] Figure 5 The training process of the target augmentation model is illustrated below. Figure 5 The optional implementation methods of this embodiment will be described by way of example.

[0070] Optionally, taking the luminance component as an example, the reconstructed frame sample corresponding to the luminance component obtained by decoding the compressed frame sample, and the intermediate decoding state data sample corresponding to the luminance component generated during the decoding process can be obtained. Figure 5 As shown, the intermediate state data samples for decoding include: the predicted frame, the initial reconstructed frame, the filter intensity map, the CU partition map, and the CU depth map obtained during the decoding process of this compressed frame sample. Figure 5As shown, starting from the target cropping start position (x, y), the reconstructed frame samples and each decoded intermediate state data sample can be synchronously cropped using the target cropping resolution. The target cropping resolution can be 1024*1024, or any other cropping resolution corresponding to the expected output resolution. In this embodiment, the target cropping start position (x, y) refers to the upper left corner of the cropping box. This position can be randomly selected, specified, or determined according to a specific algorithm; this embodiment does not impose any restrictions. During the cropping process, no geometric enhancement operations that disrupt spatial consistency, such as rotation or flipping, are performed on the cropped blocks to avoid distortion of CU partitioning information, CU depth information, and filtering intensity, etc., in the coding semantics.

[0071] After cropping, preliminary features are extracted from the reconstructed frame samples, predicted frames, initial reconstructed frames, filter intensity maps, CU partition maps, and CU depth maps. In this step, a neural network-based feature extractor can be used to perform the preliminary feature extraction operation; the type and structure of the neural network are not limited. Figure 5 As shown, after cropping, the reconstructed frame samples, predicted frames, initial reconstructed frames, filter intensity maps, CU partition maps, and CU depth maps can be downsampled by a factor of 4 (i.e., ...). Figure 5 As shown in the 4×↓ diagram, pixel rearrangement is performed. Then, along the channel dimension, the preliminary features of the reconstructed frame sample, the preliminary features of the predicted frame, and the preliminary features of the initial reconstructed frame are concatenated to obtain the first concatenated feature. And along the channel dimension, the preliminary features of the filter intensity map, the preliminary features of the CU partition map, and the preliminary features of the CU depth map are concatenated to obtain the second concatenated feature. After obtaining the first and second concatenated features, they can be calculated using convolution (e.g., ...). Figure 5 As shown in the diagram, Conv extracts features from the first and second concatenation features to obtain pixel features and coding parameter features. After obtaining the pixel features and coding parameter features, the coding parameter features can be used as prior information, and under their guidance, weighted calculations are performed on the pixel features to obtain the fused features of the luminance component. For example... Figure 5 As shown, the Sigmoid function can be used to calculate the attention weight for each pixel location based on the encoded parameter features. Then, the attention weight for each pixel location is multiplied by the pixel value of the corresponding channel in the pixel feature, dynamically weighting the pixel features using the encoded parameter features to obtain the fused feature. Figure 5 As shown, the weighted fused features can be added to the unweighted pixel features, and after convolution, input into the backbone network of the target enhancement model to obtain the residual data of the luminance component. The residual data can be added to the features input to the target enhancement model and then upsampled by a factor of 2 (i.e., Figure 5The result (2×↑) is then fed into multiple neural network layers for computation. Each neural network layer contains a convolutional layer and an activation function (such as...). Figure 5 The algorithm consists of LReLU (as shown). After calculation by multiple neural network layers, the final residual data is output. This residual data is superimposed with the reconstructed frame samples to obtain the enhanced frame samples. Then, the loss of the target augmentation model can be calculated based on the enhanced frame samples and the original frame samples, and the target augmentation model can be optimized based on this loss.

[0072] In some optional embodiments, the bit depth corresponding to the reconstructed frame samples and the decoded intermediate state data samples may be the same as or different from the bit depth corresponding to the original frame samples. The bit depth corresponding to the reconstructed frame samples and the decoded intermediate state data samples is the same as the bit depth corresponding to the reconstructed frame and the decoded intermediate state data, corresponding to the first bit depth. The bit depth corresponding to the original frame samples is the same as the bit depth corresponding to the original frame, corresponding to the second bit depth. In some optional embodiments, the first bit depth may be greater than the second bit depth. That is, when extracting the reconstructed frame samples and the decoded intermediate state data samples from the decoder, reconstructed frame samples and decoded intermediate state data samples with a bit depth greater than that of the original frame samples can be extracted. For example, if the second bit depth corresponding to the original frame samples is 8 bits, then the first bit depth corresponding to the reconstructed frame samples and the decoded intermediate state data samples can be 10 bits, 12 bits, or a greater bit depth; this embodiment is not limited to this. A greater bit depth can usually express more detailed information, thus providing more prior information for the image enhancement process and helping to improve image quality.

[0073] Based on this, in some optional embodiments, before determining the enhancement loss of the target enhancement model according to the first enhanced frame sample and the original frame sample, the first enhanced frame sample can be quantized according to the second bit depth to obtain a second enhanced frame sample adapted to the second bit depth. The process of quantizing the first enhanced frame sample can be referred to the process of quantizing the first enhanced frame described in the foregoing embodiments, and will not be repeated here.

[0074] After determining the second enhanced frame sample, the enhancement loss of the target enhancement model can be determined based on the second enhanced frame sample and the original frame sample.

[0075] Based on this implementation, during the training phase of the target augmentation model, inputting reconstructed frame samples with greater bit depth and decoding intermediate state data samples into the target augmentation model enables the target augmentation model to explore the intrinsic relationship between reconstructed frame samples and residual data samples under the guidance of decoding intermediate state data samples containing more detailed information and prior information, thereby learning to improve the image quality of reconstructed frames based on the input decoding intermediate data.

[0076] In some optional embodiments, one way to determine the enhancement loss of the target enhancement model based on the second enhanced frame sample and the original frame sample may include: calculating the mean absolute error (MAE) loss between the second enhanced frame sample and the original frame sample, and determining a first enhancement loss based on the mean absolute error loss and a first auxiliary loss, the first auxiliary loss including at least one of edge loss and stationary wavelet transform (SWT) loss. The mean absolute error loss, also known as the L1 norm loss, measures the average absolute error between the predicted and true values. The edge loss measures the difference between horizontal and vertical edges in the second enhanced frame sample and the original frame sample. When calculating the edge loss, the Sobel edge extraction operator can be applied to the second enhanced frame sample and the original frame sample respectively to extract gradient magnitude maps (i.e., edge intensity maps), and the edge loss is determined based on the difference between the gradient magnitude maps of the second enhanced frame sample and the original frame sample. The SWT loss is used to decompose the image into sub-bands of multiple scales and directions (such as low-frequency approximation and high-frequency details) through a stationary wavelet transform, and then calculate the difference between the predicted image (i.e. the second enhanced frame sample) and the real image (original frame sample) in the wavelet domain.

[0077] Optionally, the first enhancement loss is calculated as shown in the following formula: in, Describing the L1 norm, Indicates marginal loss. Indicates SWT loss. , This represents the weighting coefficient. Represents the original frame sample. This indicates the second enhanced frame sample.

[0078] The formula for calculating edge loss is as follows: in, Indicates the original frame sample The gradient magnitude map (i.e. edge intensity map) extracted after applying the Sobel operator.

[0079] The formula for calculating SWT loss is as follows: in, The number of frequency bands generated by the SWT transform. For the corresponding control coefficient, This represents the desired outcome. In this implementation, coarse tuning is performed using L1 loss combined with Sobel edge loss and SWT high-frequency loss, which enables the target enhancement model to converge quickly.

[0080] In some optional embodiments, the training process of the target model may include two stages. After the first enhancement loss converges to a specified range, a second training stage may be entered, in which a second enhancement loss is used to fine-tune the target enhancement model. Optionally, the mean squared error (MSE) loss of the second enhanced frame sample and the original frame sample may be calculated, and the second enhancement loss is determined based on the MSE loss and the second auxiliary loss. The second enhancement loss includes at least one of edge loss, stationary wavelet transform loss, and structural similarity loss.

[0081] Among them, MSE loss is used to measure the average of the squared errors between the predicted value (i.e., the second enhanced frame sample) and the true value (the original frame sample), and is also known as L2 loss. Structural similarity loss is a perceptual quality metric loss function used for image reconstruction tasks. It is based on the SSIM (Structural Similarity Index Measure) and is designed to more closely approximate the human visual system in measuring the similarity between two images.

[0082] Optionally, the first enhancement loss is calculated as shown in the following formula: The formula for calculating structural similarity loss is as follows: in, It is the weighting coefficient of structural similarity loss.

[0083] Based on this implementation, in the second training phase of the model, the mean squared error loss and SSIM loss can be switched to improve structural fidelity. During training, random pruning of the input data can be retained, while enhancement methods that disrupt spatial alignment, such as rotation and flipping, can be disabled to ensure the integrity of encoded prior information.

[0084] It's worth noting that the first and second enhancement losses are calculated based on the second-depth enhanced frame samples and the original frame samples, while the target enhancement model is based on the first-depth reconstructed frame samples and decoded intermediate state data samples for forward propagation. Therefore, in the model's backpropagation stage, a STE (Straight-Through Estimator) can be used to backpropagate the gradient, "straight-through" the gradient from the second-depth representation back to the first-depth representation. The formula is then applied to the first-depth enhanced frame samples. During the quantification process, The function is discontinuous at the transition point, therefore, STE is used to backpropagate the gradient during backpropagation, i.e., let In other words, regardless of the quantified... Whether the numerical value changes or not, the partial derivative is set to 1. Therefore, the gradient discontinuity of the quantization operation can be ignored, and the downstream error can be propagated to the network. This allows the model to perceive the accuracy loss caused by quantization during the training phase, improving its robustness in real-integer environments during inference.

[0085] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0086] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0088] Figure 6 This illustration shows a structural diagram of an electronic device provided in an exemplary embodiment of this application. This electronic device is applicable to the image enhancement method and model training method provided in the foregoing embodiments. Figure 6 As shown, the electronic device includes: a memory 601, a processor 602, and a communication component 603.

[0089] Memory 601 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0090] In some alternative embodiments, Figure 6 The illustrated electronic device is used to perform an image enhancement method. A processor 602, coupled to a memory 601, is used to execute a computer program in the memory 601 to: acquire a reconstructed frame obtained by decoding a compressed frame and intermediate decoding state data generated during the decoding process of the compressed frame, wherein the compressed frame is obtained by encoding the original frame, and the intermediate decoding state data includes at least one pixel data and at least one encoding parameter; perform feature processing on the reconstructed frame and the at least one pixel data based on encoding prior information in the at least one encoding parameter to obtain fusion features; input the fusion features into a target enhancement model to obtain residual data; and generate a first enhanced frame based on the reconstructed frame and the residual data.

[0091] Optionally, before inputting the fused features into the target enhancement model, the processor 602 is further configured to: determine, from at least one quantization parameter segment, the target quantization parameter segment to which the quantization parameter used to encode the original frame belongs; and determine the target enhancement model from enhancement models corresponding to multiple quantization parameter segments based on the target quantization parameter segment.

[0092] Optionally, when the processor 602 performs feature processing on the reconstructed frame and the at least one pixel data based on the encoding prior information in the at least one encoding parameter to obtain fusion features, it specifically performs the following steps: determining the reconstructed frames corresponding to each of the multiple color components from the reconstructed frame; determining at least one pixel data and at least one encoding parameter corresponding to each of the multiple color components from the decoding intermediate state data; and extracting features from the reconstructed frames and at least one pixel data corresponding to each of the multiple color components based on the encoding prior information in the at least one encoding parameter corresponding to each of the multiple color components to obtain fusion features for each of the multiple color components.

[0093] Optionally, when the processor 602 extracts features from the reconstructed frames and at least one pixel data corresponding to the plurality of color components based on the prior coding information in at least one coding parameter corresponding to each of the plurality of color components to obtain the fusion features of each of the plurality of color components, the processor 602 specifically performs the following steps: for any color component among the plurality of color components, extracts preliminary features from the reconstructed frame, at least one pixel data, and at least one coding parameter corresponding to the color component; in the channel dimension, concatenates the preliminary features of the reconstructed frame and the preliminary features of the at least one pixel data to obtain a first concatenated feature; in the channel dimension, concatenates the preliminary features of the at least one coding parameter corresponding to the color component to obtain a second concatenated feature; extracts features from the first concatenated feature and the second concatenated feature respectively to obtain pixel features and coding parameter features; and weights the pixel features based on the prior coding information in the coding parameter features to obtain the fusion features of the color components.

[0094] Optionally, the at least one pixel data includes at least one of a prediction frame and an initial reconstructed frame generated during the decoding process to obtain the reconstructed frame; if the color component is a luminance component, then the at least one encoding parameter includes at least one of a decoding unit partition map, a decoding unit depth map, and a filter intensity map used during the decoding process to obtain the reconstructed frame; if the color component is a chrominance component, then the at least one encoding parameter includes a filter intensity map used during the decoding process to obtain the reconstructed frame.

[0095] Optionally, when the processor 602 weights the pixel features based on the prior information of the encoding parameters to obtain the fusion features of the color components, it specifically performs the following: calculates the encoding parameter features using an attention mechanism to obtain an attention weight matrix; and uses the attention weight matrix to perform weighted calculations on the pixel features to obtain the fusion features of the color components.

[0096] Optionally, before the processor 602 performs feature processing on the reconstructed frame and the at least one pixel data based on the encoding prior information in the at least one encoding parameter to obtain the fused features, it is further configured to: synchronously crop the reconstructed frame and the decoding intermediate state data according to the target cropping start position and the target cropping resolution, wherein the target cropping resolution corresponds to the expected resolution corresponding to the reconstructed frame.

[0097] Optionally, when the processor 602 inputs the fusion features into the target enhancement model to obtain residual data, it is specifically used to: input the fusion features of each of the multiple color components into the enhancement model corresponding to the multiple color components in the target enhancement model to obtain the residual data of each of the multiple color components.

[0098] Optionally, the reconstructed frame includes reconstructed frames corresponding to each of the plurality of color components; when the processor 602 generates the first enhanced frame based on the reconstructed frame and the residual data, it is specifically configured to: superimpose the reconstructed frames corresponding to each of the plurality of color components with the residual data of each of the plurality of color components to obtain the enhanced frames of each of the plurality of color components; and combine the enhanced frames of each of the plurality of color components to obtain the first enhanced frame.

[0099] Optionally, the reconstructed frame and the decoded intermediate state data correspond to the first bit depth, and the original frame corresponds to the second bit depth of the expected output, wherein the first bit depth is greater than the second bit depth; when the processor 602 generates the first enhanced frame based on the reconstructed frame and the residual data, it is specifically used to: quantize the first enhanced frame based on the second bit depth to obtain a first enhanced frame that is adapted to the second bit depth.

[0100] Optionally, before inputting the fused features into the target augmentation model, the processor 602 is further configured to: acquire reconstructed frame samples obtained by decoding compressed frame samples and decoding intermediate state data samples generated during the decoding process of the compressed frame samples, wherein the compressed frame samples are encoded from the original frame samples, and the decoding intermediate state data samples include at least one pixel data sample and at least one encoding parameter sample; perform feature processing on the reconstructed frame samples and the at least one pixel data sample based on the encoding prior information in the at least one encoding parameter sample to obtain fused feature samples; input the fused feature samples into the target augmentation model to obtain residual data samples; generate a first augmented frame sample based on the reconstructed frame samples and the residual data samples; determine the augmentation loss of the target augmentation model based on the first augmented frame sample and the original frame samples; and train the target augmentation model with the goal of optimizing the augmentation loss.

[0101] Optionally, the quantization parameter segment to which the quantization parameters used to encode the compressed frame sample belong is the same as the quantization parameter segment to which the quantization parameters used to encode the compressed frame belong.

[0102] Optionally, the reconstructed frame sample and the decoded intermediate state data sample correspond to the first bit depth, and the compressed frame sample corresponds to the second bit depth, wherein the first bit depth is greater than the second bit depth; the processor 602, in determining the enhancement loss of the target enhancement model based on the first enhanced frame sample and the original frame sample, is further configured to: quantize the first enhanced frame sample according to the second bit depth to obtain a second enhanced frame sample adapted to the second bit depth; and determine the enhancement loss of the target enhancement model based on the second enhanced frame sample and the original frame sample.

[0103] Optionally, when the processor 602 determines the enhancement loss of the target enhancement model based on the second enhanced frame sample and the original frame sample, it is specifically configured to: calculate the mean absolute error loss between the second enhanced frame sample and the original frame sample, and determine a first enhancement loss based on the mean absolute error loss and a first auxiliary loss, wherein the first auxiliary loss includes at least one of edge loss and stationary wavelet transform loss.

[0104] Optionally, the processor 602 is further configured to: calculate the mean square error loss of the second enhanced frame sample and the original frame sample after the first enhancement loss converges to a specified range; determine the second enhancement loss based on the mean square error loss and the second auxiliary loss, wherein the second enhancement loss includes at least one of edge loss, stationary wavelet transform loss and structural similarity loss.

[0105] In some alternative embodiments, Figure 6 The illustrated electronic device is used to execute a model training method. Processor 602, coupled to memory 601, is used to execute a computer program in memory 601 for: acquiring reconstructed frame samples obtained by decoding compressed frame samples and decoding intermediate state data samples generated during the decoding process of the compressed frame samples, wherein the compressed frame samples are obtained by encoding the original frame samples, and the decoding intermediate state data samples include at least one pixel data sample and at least one encoding parameter sample; performing feature processing on the reconstructed frame samples and the at least one pixel data sample based on encoding prior information in the at least one encoding parameter sample to obtain fused feature samples; inputting the fused feature samples into a target augmentation model to obtain residual data samples; generating a first augmented frame sample based on the reconstructed frame samples and the residual data samples; determining the augmentation loss of the target augmentation model based on the first augmented frame sample and the original frame samples; and training the target augmentation model with the goal of optimizing the augmentation loss.

[0106] Furthermore, such as Figure 6As shown, the electronic device also includes other components such as a power supply component 604, a display component 605, and an audio component 606. Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components shown. Figure 6 In this embodiment, the components within the dashed boxes are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 6 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 6 The component within the dashed box.

[0107] The memory 601 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0108] The communication component 603 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0109] The power supply component 604 is used to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0110] The display component includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0111] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0112] In this embodiment, the reconstructed frame obtained by decoding the compressed frame and the intermediate decoding state data generated during the decoding process can be acquired. The intermediate decoding state data includes pixel data and encoding parameters. Based on the encoding prior information in the encoding parameters, feature processing can be performed on the pixel data and the reconstructed frame to obtain fused features. The fused features are input into the target enhancement model to obtain residual data, and an enhanced frame is generated based on the reconstructed frame and the residual data. In this process, the pixel data during the decoding process can provide the target enhancement model with pixel-level temporal or spatial prior information to enrich the feature expression, and the encoding parameters can provide encoding prior information to guide feature processing. This allows the target enhancement model to perceive the distortion characteristics of the reconstructed frame based on this prior information, thereby more accurately restoring compression distortion and improving the quality of images with compression loss.

[0113] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0115] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for enhancing image quality, characterized in that, include: The reconstructed frame obtained by decoding the compressed frame and the intermediate decoding state data generated during the decoding process of the compressed frame are obtained. The compressed frame is obtained by encoding the original frame. The intermediate decoding state data includes at least one pixel data and at least one encoding parameter. Based on the prior coding information in the at least one coding parameter, feature processing is performed on the reconstructed frame and the at least one pixel data to obtain fused features; The fused features are input into the target enhancement model to obtain residual data; A first enhanced frame is generated based on the reconstructed frame and the residual data.

2. The method according to claim 1, characterized in that, Before inputting the fused features into the target enhancement model, the following steps are also included: From at least one quantization parameter segment, determine the target quantization parameter segment to which the quantization parameters used to encode the original frame belong; Based on the target quantization parameter segment, the target enhancement model is determined from the enhancement models corresponding to multiple quantization parameter segments.

3. The method according to claim 1, characterized in that, Based on the prior coding information in the at least one coding parameter, feature processing is performed on the reconstructed frame and the at least one pixel data to obtain fused features, including: From the reconstructed frames, determine the reconstructed frames corresponding to each of the multiple color components; From the decoded intermediate state data, determine at least one pixel data and at least one encoding parameter corresponding to each of the plurality of color components; Based on the prior coding information in at least one coding parameter corresponding to each of the multiple color components, feature extraction is performed on the reconstructed frames and at least one pixel data corresponding to the multiple color components to obtain the fusion features of each of the multiple color components.

4. The method according to claim 3, characterized in that, Based on the prior coding information in at least one coding parameter corresponding to each of the multiple color components, feature extraction is performed on the reconstructed frames and at least one pixel data corresponding to the multiple color components respectively to obtain the fusion features of each of the multiple color components, including: For any one of the multiple color components, extract the preliminary features of the reconstructed frame, at least one pixel data and at least one encoding parameter corresponding to the color component. In the channel dimension, the preliminary features of the reconstructed frame corresponding to the color component and the preliminary features of each of the at least one pixel data are spliced ​​together to obtain the first spliced ​​feature; In the channel dimension, the preliminary features of each of the at least one encoding parameter corresponding to the color component are concatenated to obtain the second concatenated feature; Feature extraction is performed on the first splicing feature and the second splicing feature respectively to obtain pixel features and encoding parameter features; Based on the prior coding information in the coding parameter features, the pixel features are weighted to obtain the fusion features of the color components.

5. The method according to claim 4, characterized in that, The at least one pixel data includes at least one of the prediction frame generated during the decoding process to obtain the reconstructed frame and the initial reconstructed frame; If the color component is a luminance component, then the at least one encoding parameter includes at least one of the following: a decoding unit partition map, a decoding unit depth map, and a filter intensity map used in the process of decoding to obtain the reconstructed frame; If the color component is a chrominance component, then the at least one encoding parameter includes: a filter intensity map used in the process of decoding to obtain the reconstructed frame.

6. The method according to claim 4, characterized in that, Based on the prior coding information in the coding parameter features, the pixel features are weighted to obtain the fusion features of the color components, including: An attention mechanism is used to calculate the features of the encoded parameters to obtain an attention weight matrix; The attention weight matrix is ​​used to perform weighted calculations on the pixel features to obtain the fusion features of the color components.

7. The method according to claim 3, characterized in that, Before performing feature processing on the reconstructed frame and the at least one pixel data based on the prior coding information in the at least one coding parameter to obtain the fused features, the method further includes: Based on the target cropping start position and the target cropping resolution, the reconstructed frame and the decoded intermediate state data are cropped synchronously, and the target cropping resolution corresponds to the expected resolution of the reconstructed frame.

8. The method according to claim 3, characterized in that, The fused features are input into the target enhancement model to obtain residual data, including: The fusion features of each of the multiple color components are input into the enhancement model corresponding to the multiple color components in the target enhancement model to obtain the residual data of each of the multiple color components.

9. The method according to claim 8, characterized in that, The reconstructed frame includes the reconstructed frame corresponding to each of the multiple color components; Based on the reconstructed frame and the residual data, a first enhanced frame is generated, including: The reconstructed frames corresponding to each of the multiple color components are superimposed with the residual data of each of the multiple color components to obtain the enhanced frames of each of the multiple color components. The enhanced frames of each of the multiple color components are combined to obtain the first enhanced frame.

10. The method according to claim 1, characterized in that, The reconstructed frame and the decoded intermediate state data correspond to the first bit depth, and the original frame corresponds to the second bit depth of the expected output. The first bit depth is greater than the second bit depth. Based on the reconstructed frame and the residual data, a first enhanced frame is generated, including: Based on the second bit depth, the first enhanced frame is quantized to obtain a first enhanced frame that is adapted to the second bit depth.

11. The method according to any one of claims 1-9, characterized in that, Before inputting the fused features into the target enhancement model, the following is also included: The reconstructed frame sample obtained by decoding the compressed frame sample and the intermediate decoding state data sample generated during the decoding process of the compressed frame sample are obtained. The compressed frame sample is obtained by encoding the original frame sample. The intermediate decoding state data sample includes at least one pixel data sample and at least one encoding parameter sample. Based on the prior coding information in the at least one coding parameter sample, feature processing is performed on the reconstructed frame sample and the at least one pixel data sample to obtain a fused feature sample. The fused feature samples are input into the target enhancement model to obtain residual data samples; Based on the reconstructed frame sample and the residual data sample, a first enhanced frame sample is generated; The enhancement loss of the target enhancement model is determined based on the first enhanced frame sample and the original frame sample. The target augmentation model is trained with the goal of optimizing the augmentation loss.

12. The method according to claim 11, characterized in that, The quantization parameter segment to which the quantization parameters used to encode the compressed frame sample belong is the same as the quantization parameter segment to which the quantization parameters used to encode the compressed frame belong.

13. The method according to claim 11, characterized in that, The reconstructed frame sample and the decoded intermediate state data sample correspond to the first bit depth, and the compressed frame sample corresponds to the second bit depth. The first bit depth is greater than the second bit depth. Determining the enhancement loss of the target enhancement model based on the first enhanced frame sample and the original frame sample further includes: The first enhanced frame sample is quantized according to the second bit depth to obtain a second enhanced frame sample that is adapted to the second bit depth. The enhancement loss of the target enhancement model is determined based on the second enhanced frame sample and the original frame sample.

14. The method according to claim 13, characterized in that, Based on the second enhanced frame sample and the original frame sample, the enhancement loss of the target enhancement model is determined, including: Calculate the mean absolute error loss between the second enhanced frame sample and the original frame sample, and determine the first enhancement loss based on the mean absolute error loss and the first auxiliary loss, wherein the first auxiliary loss includes at least one of edge loss and stationary wavelet transform loss.

15. The method according to claim 14, characterized in that, Also includes: After the first enhancement loss converges to a specified range, the mean square error loss of the second enhanced frame sample and the original frame sample is calculated. A second enhancement loss is determined based on the mean square error loss and the second auxiliary loss, wherein the second enhancement loss includes at least one of edge loss, stationary wavelet transform loss, and structural similarity loss.

16. A model training method, characterized in that, include: The reconstructed frame sample obtained by decoding the compressed frame sample and the intermediate decoding state data sample generated during the decoding process of the compressed frame sample are obtained. The compressed frame sample is obtained by encoding the original frame sample. The intermediate decoding state data sample includes at least one pixel data sample and at least one encoding parameter sample. Based on the prior coding information in the at least one coding parameter sample, feature processing is performed on the reconstructed frame sample and the at least one pixel data sample to obtain a fused feature sample. The fused feature samples are input into the target enhancement model to obtain residual data samples; Based on the reconstructed frame sample and the residual data sample, a first enhanced frame sample is generated; The enhancement loss of the target enhancement model is determined based on the first enhanced frame sample and the original frame sample. The target augmentation model is trained with the goal of optimizing the augmentation loss.

17. An electronic device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-16.

18. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-16.

19. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, enables the implementation of the steps in the method according to any one of claims 1-16.

Images