Adaptive compensation method in video image encoding process and related device

Abstract

The present disclosure provides an adaptive compensation method in a video image encoding process and a related device, and relates to the technical fields of cloud computing and video transcoding. The method comprises the following steps: obtaining an original pixel value array of an original pixel block to be encoded and a reconstructed pixel value array of a reconstructed pixel block obtained after the original pixel block is reconstructed; determining an encoding category to which each pixel in the reconstructed pixel block belongs, and respectively determining a target pixel value of each pixel when the pixel satisfies at least two optimization objectives simultaneously; for each pixel in each encoding category, determining a first offset value according to an original pixel value of the pixel in the original pixel value array and a reconstructed pixel value of the pixel in the reconstructed pixel value array; determining a second offset value according to the target pixel value of the pixel and the reconstructed pixel value of the pixel; determining an offset value interval of the pixel according to the sizes of the first offset value and the second offset value; performing frequency statistics on integer offset values respectively contained in the offset value intervals of all pixels in each encoding category, and determining an expected offset value of the corresponding encoding category based on the obtained statistical result; and compensating the corresponding reconstructed pixel value of each pixel according to the expected offset value corresponding to the encoding category to which the pixel belongs.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to the fields of cloud computing and video transcoding technology, and particularly to an adaptive compensation method, apparatus, electronic device, computer-readable storage medium, and computer program product in the video image encoding process. Background Technology

[0002] Since the digitization of visual information, how to efficiently compress it for storage and distribution has been a key technical issue. To enable the compressed bitstreams to be interoperable and decoded in a standardized manner, international organizations have established international standards for video coding, gradually forming a hybrid coding framework based on technologies such as prediction, transform, quantization, entropy coding, and post-processing filtering.

[0003] In traditional image and video coding frameworks, the sample adaptive compensation technique is used to perform post-processing filtering on the reconstructed image. By superimposing a certain offset on the reconstructed value, the distortion between the coded reconstructed image and the original image is reduced. Summary of the Invention

[0004] This disclosure provides an adaptive compensation method, apparatus, electronic device, computer-readable storage medium, and computer program product for video image encoding.

[0005] In a first aspect, embodiments of this disclosure propose an adaptive compensation method in the video image encoding process, comprising: obtaining an array of original pixel values ​​of an original pixel block in a state to be encoded during loop filtering; obtaining an array of reconstructed pixel values ​​of a reconstructed pixel block obtained after reconstructing the original pixel block; determining the encoding category to which each pixel in the reconstructed pixel block belongs, and determining the target pixel value of each pixel when it simultaneously satisfies at least two optimization objectives; for each pixel under each encoding category, determining a first offset value based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array; determining a second offset value based on the target pixel value and the reconstructed pixel value; determining the offset value range of the pixel based on the magnitude of the first offset value and the second offset value; performing frequency statistics on the integer offset values ​​contained in the offset value range of each pixel under each encoding category, and determining the expected offset value of the corresponding encoding category based on the obtained statistical results; and compensating the corresponding reconstructed pixel value of each pixel according to the expected offset value corresponding to its encoding category.

[0006] Secondly, embodiments of this disclosure propose an adaptive compensation device in a video image encoding process, comprising: an original pixel value acquisition unit configured to acquire an array of original pixel values ​​of an original pixel block in a state to be encoded during loop filtering; a reconstructed pixel value acquisition unit configured to acquire an array of reconstructed pixel values ​​of a reconstructed pixel block obtained after reconstructing the original pixel block; an encoding category determination and multi-objective optimization unit configured to determine the encoding category to which each pixel in the reconstructed pixel block belongs, and to determine the target pixel value of each pixel when it simultaneously satisfies at least two optimization objectives; and an offset value interval calculation unit configured to calculate the offset value interval for each pixel under each encoding category according to the original pixel value. A pixel is assigned a first offset value based on its original pixel value in the original pixel value array and its reconstructed pixel value in the reconstructed pixel value array. A second offset value is determined based on the target pixel value and the reconstructed pixel value of the pixel. The offset value range of the pixel is determined based on the magnitude of the first offset value and the second offset value. The expected offset value calculation unit is configured to perform frequency statistics on the integer offset values ​​contained in the offset value range of each pixel under each coding category, and determine the expected offset value of the corresponding coding category based on the obtained statistical results. The pixel value compensation unit is configured to compensate the corresponding reconstructed pixel value of each pixel according to the expected offset value corresponding to its coding category.

[0007] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the adaptive compensation method in the video image encoding process as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the adaptive compensation method in the video image encoding process as described in the first aspect when executed.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the steps of the adaptive compensation method in the video image encoding process as described in the first aspect.

[0010] The adaptive compensation scheme in the video image encoding process provided in this embodiment, for the original pixel block to be encoded during the loop filtering process, firstly obtains the original pixel value array of the original pixel block and the reconstructed pixel block array obtained after reconstruction. Then, it determines the target pixel value of each pixel constituting the reconstructed pixel block when it simultaneously satisfies multiple optimization objectives. Next, after determining the encoding category to which each pixel belongs, for each pixel under each encoding category, two offset values ​​are calculated based on the reconstructed pixel value, the original pixel value, and the target pixel value, respectively. The integer offset values ​​included in the offset value range determined are then determined. Furthermore, the expected offset value of the corresponding encoding category is calculated based on the frequency of occurrence of each integer offset value in each offset value range under the corresponding category. Thus, adaptive compensation is performed according to the expected offset value corresponding to the encoding category. In other words, this method can use offset values ​​that consider multiple optimization objectives to compensate for pixel values, thereby achieving better encoding performance.

[0011] For each pixel in each coding category, the frequency of integer offset values ​​within their respective offset ranges is statistically analyzed, and the expected offset value for the corresponding coding category is determined based on the statistical results.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0014] Figure 1 This is an exemplary system architecture to which this disclosure can be applied;

[0015] Figure 2 A flowchart of an adaptive compensation method in the video image encoding process provided in this disclosure embodiment;

[0016] Figure 3 A flowchart illustrating a method for calculating an offset value range for each pixel under each coding category, provided in an embodiment of this disclosure;

[0017] Figure 4 A flowchart illustrating a method for calculating the desired offset value corresponding to each coding category based on the offset value range of all pixels under each coding category, as provided in this embodiment of the disclosure;

[0018] Figure 5A flowchart illustrating another method for calculating the desired offset value corresponding to each coding category based on the offset value range of all pixels under each coding category, as provided in this embodiment of the disclosure;

[0019] Figure 6 A structural block diagram of an adaptive compensation device in the video image encoding process provided in this disclosure embodiment;

[0020] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for performing an adaptive compensation method in the video image encoding process, as provided in an embodiment of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0022] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the adaptive compensation methods, apparatuses, electronic devices, and computer-readable storage media in the video image encoding process of this disclosure can be applied.

[0024] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0025] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include video image encoding applications, video image decoding applications, and instant messaging applications.

[0026] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0027] Server 105 can provide various services through its built-in applications. Taking a video image encoding application as an example, when running this application, server 105 can achieve the following: First, obtain the original pixel value array of the original pixel block in the loop filtering process of video image encoding, which is in the state to be encoded; next, obtain the reconstructed pixel value array of the reconstructed pixel block obtained after reconstructing the original pixel block; then, determine the encoding category to which each pixel in the reconstructed pixel block belongs, and determine the target pixel value of each pixel when it simultaneously satisfies at least two optimization objectives; then, for each encoding category... For each pixel in the array, a first offset value is determined based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array. A second offset value is determined based on the target pixel value and the reconstructed pixel value. The offset value range of the pixel is determined based on the magnitude of the first and second offset values. Next, the frequency of integer offset values ​​contained in the offset value range of each pixel under each coding category is statistically analyzed, and the expected offset value of the corresponding coding category is determined based on the obtained statistical results. Finally, the corresponding reconstructed pixel value is compensated for each pixel according to the expected offset value corresponding to its coding category.

[0028] Furthermore, the server 105 can also send the encoding result corresponding to the expected reconstructed pixel block obtained after compensation processing to the terminal devices 101, 102, and 103, so that the terminal devices can decode the video image through the built-in video image decoding application and present the corresponding video image to the user through the display screen.

[0029] It should be noted that the raw pixel blocks to be encoded can be obtained from the terminal devices 101, 102, and 103 via the network 104, or they can be pre-stored locally on the server 105 through various means. Therefore, when the server 105 detects that this data is already stored locally (for example, when it starts processing previously stored encoding tasks), it can choose to obtain this data directly from the local storage. In this case, the exemplary system architecture 100 may also exclude the terminal devices 101, 102, and 103 and the network 104.

[0030] Since video image encoding requires significant computing resources and power, the adaptive compensation method in the video image encoding process provided in the subsequent embodiments of this disclosure is generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the adaptive compensation device in the video image encoding process is also generally located in the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also perform the aforementioned calculations performed by the server 105 through the video image encoding applications installed on them, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the video image encoding application determines that the terminal device it is using has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the adaptive compensation device in the video image encoding process can also be located in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0032] Please refer to Figure 2 , Figure 2 A flowchart of an adaptive compensation method in a video image encoding process provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0033] Step 201: Obtain the original pixel value array of the original pixel block in the encoding state during the loop filtering process;

[0034] This step aims to be performed by the entity executing the adaptive compensation method in the video image encoding process (e.g., Figure 1The server 105 shown obtains the array of original pixel values ​​of the original pixel blocks in the loop filtering process of video image encoding.

[0035] The loop filtering process clarifies the specific stage and scenario in which the action described in this step occurs. In modern hybrid video coding frameworks (such as H.264 / AVC, H.265 / HEVC, or AV1), loop filtering is a crucial step in the encoder reconstruction loop, primarily including deblocking filtering and sample adaptive compensation. The current stage refers to the encoder logic about to enter or currently in the sample adaptive compensation sub-filtering stage. At this point, the encoder has completed the prediction, transformation, quantization, and inverse quantization and inverse transformation of the current block, obtaining a preliminary, distorted "reconstructed pixel block." The solution provided in this embodiment is triggered at this node, aiming to optimize the preliminary reconstructed block by using a better compensation value determination scheme.

[0036] Here, the raw pixel block refers to the image region block (e.g., a maximum coding unit or a transform block) that is currently being processed from the original video frame to be compressed. It represents the source data without any compression or distortion. The limitation of being in a state to be encoded indicates that the raw pixel block is the object that the encoder pipeline is currently processing. On the other hand, although the data of the block is being used for calculation in the encoder's internal logic, its corresponding raw pixel values ​​are usually pre-stored in a specific buffer or data structure to ensure that they can be accessed quickly and accurately when reference is needed. That is, it does not mean that the block has not yet started to be encoded, but emphasizes that its raw data is still available and ready for reference in the filtering stage.

[0037] The raw pixel value array is a structured and specific representation of the data contained in the raw pixel block. The pixel value is the brightness and / or chromaticity component value of each pixel. These values ​​are the most original light and color information captured by the sensor or generated by the computer. The raw pixel values ​​usually come from the raw video frame data loaded during encoder initialization. In memory, the raw pixels of the whole frame are divided into multiple blocks, and a corresponding buffer is maintained for each block or the whole frame. The array specifies the organization of these pixel values ​​in the encoder's memory. It is usually a matrix data structure whose width and height correspond exactly to the size of the video block being processed (e.g., 4x4, 8x8, 16x16, etc.). The "array" form makes it easy for the encoder to efficiently locate and access the raw value of any pixel by row and column index.

[0038] To obtain the original pixel value array, the encoder can extract the portion of original pixel data that completely corresponds to the spatial location of the current "reconstructed pixel block" from the original frame buffer through memory addressing, based on the position (such as its coordinates in the frame) and size information of the block currently being processed, and organize it into the above "array" form, and load it into the register or local cache of the processing unit for subsequent calculation.

[0039] This step describes how, when the encoder process reaches the sample adaptive compensation stage of the loop filtering stage, the corresponding uncompressed raw pixel data is located and extracted from the original video frame data based on the location information of the currently processed reconstruction block, and then organized into a matrix form to lay the foundation for the execution of subsequent steps.

[0040] Step 202: Obtain the array of reconstructed pixel values ​​of the reconstructed pixel block obtained after reconstructing the original pixel block;

[0041] Based on step 201, this step aims to obtain the array of reconstructed pixel values ​​of the reconstructed pixel block obtained by the aforementioned execution entity after the original pixel block has been reconstructed.

[0042] The reconstruction operation is the core closed-loop operation in the hybrid video coding framework. Specifically, the encoder predicts the original pixel block (intra-frame or inter-frame prediction) to obtain residuals, then transforms and quantizes the residuals to obtain encodeable coefficients. Next, for subsequent prediction reference, the encoder immediately performs inverse quantization and inverse transform, adding the obtained residuals to the predicted values ​​to reconstruct a decoded pixel block. This process is the reconstruction process described in this step. The resulting reconstructed pixel block typically represents the original pixel block after lossy compression and decoding, inevitably introducing distortion caused by quantization and other operations. It represents the final image seen by the decoder without any post-processing (such as sample adaptive compensation). There is a difference between it and the original pixel block, and this difference is precisely what the subsequent filtering stages need to try to compensate for.

[0043] The data for the reconstructed pixel block originates from the output buffer of the reconstruction loop within the encoder. In the encoding pipeline, after a block completes inverse quantization, inverse transform, and addition with the predicted value, its reconstruction result is temporarily stored in a designated memory area. This step performs an acquisition operation when the sample adaptive compensation filter is invoked. It reads the latest reconstruction result from the buffer before any sample adaptive compensation filtering has been applied, and all compensation calculation strategies provided in subsequent steps are based on this unfiltered baseline state. The aim is to optimize this state using the scheme provided in this embodiment, thereby producing a better final reconstruction result.

[0044] Step 203: Determine the coding category of each pixel in the reconstructed pixel block, and determine the target pixel value of each pixel when at least two optimization objectives are met simultaneously;

[0045] Based on step 202, this step aims to determine the coding category of each pixel in the reconstructed pixel block by the aforementioned execution entity, and to determine the target pixel value of each pixel when at least two optimization objectives are met simultaneously.

[0046] In determining the coding category of each pixel, the process typically involves classifying each pixel in the reconstructed pixel block based on established rules of current coding classification standards. This classification aims to group pixels with similar visual characteristics to facilitate the application of a unified compensation strategy. Taking the HEVC / H.265 encoder as an example, its filtering modes include band offset and edge offset. Band offset classifies pixels into up to 32 categories (bands) based on the predefined range within which their reconstructed brightness values ​​fall, primarily used to address color or brightness distortion in smooth areas. Edge offset classifies pixels into specific edge direction patterns (usually 5 categories) based on the gradient relationship between the pixel's reconstructed brightness values ​​and those of its immediate neighbors (e.g., left, top, right, bottom, center), mainly used to identify and enhance edge structures in the image. Typically, a pixel will only be assigned to either band offset or edge offset; that is, it will either belong to one of the aforementioned 32 categories or one of the aforementioned 5 categories.

[0047] Of course, besides directly using the existing encoding classification standard of the video encoder, a target encoding classification standard can also be determined based on the existing encoding classification standard of the video encoder. Specifically, the encoding classification standard of the existing video encoder can be adjusted according to custom requirements to obtain the target encoding classification standard. This adjustment according to custom requirements can include: increasing the number of subclasses under a specified encoding category (e.g., increasing the 32 subclasses mentioned above to 33), adjusting the coverage of each subclass under a specified encoding category, etc. Then, the encoding category to which each pixel belongs is determined using the target encoding classification standard.

[0048] In the operation of determining the target pixel value for each pixel when simultaneously satisfying at least two optimization objectives, this operation calculates a "target pixel value" for each pixel, which is crucial for achieving multi-objective collaborative optimization. It is defined as: the ideal value that the pixel should be reconstructed to under the common constraint of satisfying at least two different optimization objectives.

[0049] One possible implementation method, including but not limited to, is as follows: First, based on the obtained actual optimization requirements, determine an optimization target set containing at least two different optimization objectives; then, for each pixel constituting the reconstructed pixel block, determine the target pixel value that simultaneously satisfies all optimization objectives in the optimization target set. The optimization objectives can include: optimization objectives that require reference to the original image and optimization objectives that do not require reference to the original image (of course, they can also be divided according to other classification dimensions, such as whether they mainly improve the viewing experience for the human eye). Optimization objectives that require reference to the original image can further include: Mean Squared Error (MSE, calculated as the mean of the squared differences at the pixel level; a smaller value indicates lower distortion), Peak Signal-to-Noise Ratio (PSNR, a logarithmic value based on MSE; a larger value indicates better quality), Structural Similarity Index (SSIM, measuring the similarity of brightness, contrast, and structure, which better aligns with subjective human perception), Feature Similarity Index (FSIM), Gradient Magnitude Similarity Deviation (GMSD), and Video Multimethod Assessment. Fusion, VMAF (a comprehensive index that integrates multiple basic metrics and is highly correlated with human visual ratings), requires at least one of the following optimization objectives without reference to the original image: sharpness enhancement (improving detail clarity through edge enhancement and super-resolution techniques, such as using the Laplacian operator to detect edges), noise suppression (reducing compression noise, such as noise reduction algorithms based on nonlocal mean filtering), color enhancement (adjusting saturation / contrast, such as adaptive histogram equalization, to improve overly dark or overexposed areas), artifact removal (eliminating texture distortion caused by compression, such as using generative adversarial networks to repair mosaic areas), and subjective quality assessment models (using at least one of the following metrics: BRISQUE (blind image spatial quality evaluator), NIQE (natural image quality evaluator), and CONTRIQUE (contrast-based no-reference quality assessment).

[0050] In practical terms, the encoder can calculate an expected value for each pixel that satisfies the best balance of these objectives based on a preset set of optimization targets (such as T = [PSNR, SSIM]) through an internal decision-making mechanism. This value may not actually exist, but is a calculated theoretical optimal value. It serves as the expectation that guides the direction of compensation, adding a new optimization dimension to the traditional method that only relies on the original pixel value.

[0051] When the optimization objectives are specifically divided into two categories, namely optimization objectives that require reference to the original image and optimization objectives that do not require reference to the original image, if the actual resolution of the original image corresponding to the original pixel block is higher than the preset resolution, at least one of the optimization objectives that require reference to the original image can be determined as the optimization objective when obtaining the target pixel value of the pixel; if the actual resolution of the original image corresponding to the original pixel block is lower than the preset resolution, at least one of the optimization objectives that do not require reference to the original image can be determined as the optimization objective when obtaining the target pixel value of the pixel.

[0052] Step 204: For each pixel under each coding category, determine the first offset value based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array; determine the second offset value based on the target pixel value and the reconstructed pixel value; determine the offset value range of the pixel based on the magnitude of the first offset value and the second offset value.

[0053] Building upon step 203, this step aims to have the aforementioned execution entity calculate a feasible range of compensation values ​​for each pixel. First, a first offset value is calculated: the original pixel value minus its reconstructed pixel value. This value represents the traditional compensation direction when only minimizing a single distortion (such as PSNR) is considered. Second, a second offset value is calculated: the multi-objective optimization ideal value (target pixel value) of the pixel minus its reconstructed pixel value. This value represents the ideal compensation direction when multiple image quality metrics are optimized collaboratively. Finally, by comparing these two offset values, the smaller one is taken as the lower limit of the feasible range, and the larger one as the upper limit, thereby determining the offset value range for the pixel. This range collectively constrains all possible compensation values ​​under the conditions of basic fidelity and multi-objective guidance.

[0054] It should be noted that since the reconstructed pixel block is obtained based on the original pixel block after reconstruction, although the pixels that make up the reconstructed pixel block may not be completely the same as the pixels that make up the original pixel block, there must be a corresponding relationship between the two that corresponds to the reconstruction operation. Therefore, for each pixel that makes up the reconstructed pixel block, the value of the original pixel can still be determined through this correspondence.

[0055] Step 205: Count the frequency of integer offset values ​​contained in the offset value range of each pixel under each coding category, and determine the expected offset value of the corresponding coding category based on the obtained statistical results;

[0056] Based on step 204, this step aims to perform frequency statistics on the integer offset values ​​contained in the offset value range of each pixel under each coding category, and determine the expected offset value of the corresponding coding category based on the obtained statistical results.

[0057] The frequency statistics target all pixels under a specific coding category (such as an edge direction class or a brightness stripe class). The specific processing logic is as follows: check the offset value range of each pixel one by one (e.g., [a, b]). For each integer offset value (i.e., a, a+1, a+2, ..., b) that falls within this range, a count is accumulated for it. This means that as long as an integer offset value is covered by the range of a certain pixel, its statistical frequency increases by 1. After traversing all pixels of this class, each integer offset value has a corresponding frequency count.

[0058] The determination of the expected offset value is a data processing based on this statistical result, which involves combining all possible integer offset values ​​with their corresponding frequencies to calculate the statistical expected value of the category (usually the quotient of the sum of the products of each integer offset value and its frequency and the sum of all frequencies). Mathematically, this expected value represents the average optimal direction of all feasible compensation values ​​for all pixels under the multi-objective constraints of this category, and is therefore selected as the unified and final compensation value for this category.

[0059] Step 206: For each pixel, compensate the corresponding reconstructed pixel value according to the expected offset value corresponding to its coding category.

[0060] Building upon step 205, this step aims to allow the aforementioned executing entity to fully utilize its output to actually correct the reconstructed pixel value array. Specifically, for each pixel, based on its determined encoding category, the expected offset value calculated in step 205 is indexed to uniquely correspond to that category. This value is then directly added to the pixel's current reconstructed value; the addition operation is "compensation." This allows the final reconstructed image effect to be improved not only by optimizing a single distortion target in the traditional sense, but also by enhancing the overall visual experience under the combined effect of multiple predetermined optimization targets, while maintaining full compatibility with existing encoding standard syntax (since only one compensation value is written for each category).

[0061] The adaptive compensation method in the video image encoding process provided in this disclosure, for the original pixel block to be encoded during the loop filtering process, firstly obtains the original pixel value array of the original pixel block and the reconstructed pixel block array of the reconstructed pixel block. Then, it determines the target pixel value of each pixel constituting the reconstructed pixel block when it simultaneously satisfies multiple optimization objectives. Next, after determining the encoding category to which each pixel belongs, for each pixel under each encoding category, two offset values ​​are calculated based on the reconstructed pixel value, the original pixel value, and the target pixel value, respectively. The integer offset values ​​included in the determined offset value range are then determined. Furthermore, the expected offset value for the corresponding encoding category is calculated based on the frequency of occurrence of each integer offset value in each offset value range under the corresponding category. Adaptive compensation is then performed according to the expected offset value corresponding to the encoding category. In other words, this method can use offset values ​​that consider multiple optimization objectives for pixel value compensation, thereby achieving better encoding performance.

[0062] To further understand step 204, please refer to [the relevant documentation / reference]. Figure 3 , Figure 3 A flowchart of a method for calculating an offset value range for each pixel under each coding category, provided for embodiments of this disclosure, wherein process 300 includes the following steps:

[0063] Step 310: For each pixel under each coding category, perform the following offset value range calculation steps:

[0064] Step 311: Subtract the reconstructed pixel value from the original pixel value of the pixel to obtain the first offset value;

[0065] This step performs the core calculation of the traditional compensation algorithm: subtracting the reconstructed pixel value from the original pixel value of the current pixel to obtain the first offset value. This operation quantifies the absolute error between the current reconstructed value and the lossless original image. This value represents the most direct and basic compensation direction for this pixel when the sole objective is to "minimize distortion".

[0066] Step 312: Subtract the reconstructed pixel value from the target pixel value of the pixel to obtain the second offset value;

[0067] This step embodies the newly introduced multi-objective optimization concept: subtracting the reconstructed pixel value from the target pixel value (i.e., the ideal value determined by multiple preset optimization objectives) yields the second offset value. This operation quantifies the gap between the current reconstructed value and the theoretically optimal value for multiple objectives. This value represents a more advanced and intelligent compensation direction for the pixel after comprehensively considering multiple image quality indicators.

[0068] Step 313: Determine the offset value range of the pixel by using the smaller of the first offset value and the second offset value as the lower limit of the range and the larger value as the upper limit of the range.

[0069] This step is responsible for fusing the two offset values ​​to generate a decision interval. The rule is: compare the numerical values ​​of the first and second offset values, take the smaller one as the lower limit of the pixel offset value interval, and take the larger one as the upper limit. The interval [lower limit, upper limit] thus determined has a clear physical meaning: each integer offset value within the interval is a "feasible" compensation scheme for the pixel, and its compensation effect is either better than or equal to the single-objective optimization result, or better than or equal to the multi-objective optimization result, thus forming a set of candidate solutions for Pareto improvement. This interval provides the data basis for subsequent calculation of statistical expected value.

[0070] This embodiment lays the foundation for subsequent statistical optimization through steps 310-313, namely, calculating a feasible range (offset value interval) of compensation value for each pixel, so as to further calculate the expected offset value corresponding to each coding category.

[0071] To further understand process 205, please also refer to Figure 4 , Figure 4 A flowchart of a method for calculating the expected offset value corresponding to each coding category based on the offset value range of all pixels under each coding category, provided in this embodiment of the disclosure, includes the following steps in process 400:

[0072] Step 410: For the offset value range of all pixels under each coding category, perform the following expected offset value calculation steps:

[0073] Step 411: Iterate through all integer offset values ​​within the offset range;

[0074] This step aims to address the specific encoding category currently being processed (e.g., all pixels classified as "specific edge direction"). Instead of operating on a range of individual pixels, the operation involves iterating through all integer offset values ​​that could be encompassed by the offset range of any pixel. These integer offset values ​​form a candidate value set. The reason this embodiment only iterates through integer offset values ​​is that, in most cases, the smallest unit of offset value is an integer, and decimals are not present. If a smaller unit of measurement, such as a decimal, exists, adaptive adjustments can be made; this is not strictly limited here.

[0075] Step 412: For each integer offset value, count the frequency of occurrence of each pixel in the offset value range under each coding category, and calculate the occurrence frequency based on the frequency.

[0076] In this step, for each integer offset value in the candidate set from the previous step, the system checks whether it falls within the offset value range of each pixel in that category. Each time the integer offset value falls within a pixel's range, it is counted once; the sum of these counts represents the frequency of that integer offset value in this encoded category. Then, the frequency of each integer offset value is divided by the total frequency of all pixels in that category (i.e., the sum of the frequencies of all integer offset values) to obtain its occurrence frequency. The frequency characterizes the prevalence of the compensation value being accepted under multi-objective constraints across all pixels.

[0077] Step 413: Calculate the expected offset value corresponding to each coding category based on the frequency of occurrence of each integer offset value under each coding category.

[0078] This step treats each integer offset value as a possible output, uses its corresponding frequency as its probability weight, and calculates the weighted average (i.e., the mathematical expectation) of all these integer offset values. This calculated expectation value is the expected offset value for that coding category. It represents the compensation value that, from a statistical point of view, best balances the multi-objective constraints of all pixels, and will be ultimately written into the bitstream and used for filtering.

[0079] This embodiment, through the specific implementation scheme described in steps 410-413, consolidates the offset value intervals that represent the feasible domain of multiple targets and are independently calculated for each pixel in the previous steps into a unified and statistically optimal compensation value for each type of pixel, thereby achieving a more accurate determination of the compensation value.

[0080] Based on any of the above embodiments, after the compensation action described in step 206, the compensated pixel value after compensating the corresponding reconstructed pixel value for each pixel according to the expected offset value corresponding to its coding category can be determined, and the pixel block composed of pixels whose pixel value is the compensated pixel value can be further determined as the expected reconstructed pixel block.

[0081] Specifically, after completing the compensation operation for all pixels, the encoder will enter the result confirmation and data reconstruction stage. In this stage, the compensated pixel value is first determined by algebraically adding the expected offset value corresponding to the coding category of each pixel to its original reconstructed pixel value. The result is the final optimized value of the pixel. This value integrates the basic fidelity and multi-target guidance, representing the ideal state of the pixel after processing in this embodiment.

[0082] Subsequently, the encoder constructs the desired reconstruction pixel block, aiming to reorganize all compensated pixel values ​​into a complete, two-dimensional pixel block structure according to their original spatial relationships. This newly constructed pixel block is the desired reconstruction pixel block, which represents the best overall reconstruction quality that the current coding block can achieve after innovative multi-objective guided sampling adaptive compensation filtering. This pixel block is then written into the frame buffer, serving both as a reference frame for subsequent inter-frame prediction and as optimized reconstructed image data output by the entire coding loop. Its image quality surpasses the results obtained by traditional single-objective optimization methods in multiple objective metrics and subjective user experience.

[0083] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario:

[0084] Conventional sample-based adaptive compensation includes two types: edge compensation and stripe compensation. Both methods first classify each reconstructed pixel, then calculate the compensation value for each class, and finally write the compensation value for each class into the bitstream. During decoding, the compensation value is decoded from the bitstream and superimposed according to the class of the reconstructed pixel.

[0085] In traditional methods, the compensation value for each category is typically determined using least squares. This involves calculating the difference between the original and reconstructed pixel values ​​and then averaging the differences for pixels of the same category as the compensation value to minimize distortion. Because this method has sufficient theoretical support and low complexity, encoders using sample-based adaptive compensation techniques generally employ this compensation value determination algorithm. However, this algorithm essentially optimizes the MSE / PSNR metric, therefore, this compensation value calculation method cannot simultaneously address multiple optimization objectives, such as improving subjective performance or enhancing other objective quality evaluation metrics that are more consistent with subjective experience while maintaining distortion reduction.

[0086] This embodiment, based on the characteristics of sample-point adaptive compensation and the principle of multi-objective optimization, proposes a guided sample-point adaptive compensation value determination algorithm that uses guidance information as the optimization target carrier. This algorithm can obtain the preferences during sample-point adaptive compensation according to a predetermined set of optimization targets, and determine the Pareto optimal sample-point compensation value for each pixel. Then, it determines the expected compensation value for each type of pixel, which is used as the final compensation value for encoding and filtering. The main differences between this embodiment and existing traditional sample-point adaptive compensation value solving algorithms are as follows:

[0087] 1) The optimization objective is extended from traditional single-objective optimization to multi-objective optimization, thereby obtaining the Pareto optimal solution that dominates the traditional sample-point adaptive compensation algorithm; 2) It is not limited to optimizing MSE / PSNR, but can be extended to other optimization objectives, such as objective evaluation indicators that better conform to the subjective characteristics of human eyes. The complete technical process is as follows:

[0088] 1. Define the set of optimization objectives ,in To optimize the number of objectives.

[0089] 2. For a raw pixel block to be encoded ,in The height of this pixel block. This represents the width of the pixel block. The reconstructed pixel block before sample adaptive filtering is obtained after reconstruction. The pixel is classified according to the encoding standard specification to obtain the category of each pixel. ,in The pixel category is represented by a finite set. For example, when applied in an HEVC / H.265 encoder, the filtering modes include strip compensation and edge compensation. In strip compensation, each pixel has 32 categories, i.e. In edge compensation, each pixel has 5 categories, namely... .

[0090] 3. Based on the defined optimization objective set To obtain the guidance information of each pixel , indicating the corresponding optimization objective set The target is to reconstruct the pixel values.

[0091] 4. Iterate through all possible categories For all pixels belonging to this class Calculate the original pixel value Pixel values ​​of the reconstructed pixel block before sample adaptive filtering Get the offset value Calculate the target pixel value for reconstruction Pixel values ​​of the reconstructed pixel block before sample adaptive filtering Get the offset value The smaller of the two offset values ​​is The larger value This gives us a range. For all integers falling within this range, increment the frequency count by one.

[0092] 5. For each category The statistical expectation is calculated based on the frequency of the statistics and written as the offset value of the category into the generated bitstream.

[0093] The solutions described in steps 4 and 5 above can be found in [reference needed]. Figure 5 , Figure 5 A flowchart illustrating another method for calculating the desired offset value corresponding to each coding category based on the offset value range of all pixels under each coding category, as provided in this disclosure embodiment.

[0094] In the field of image and video coding, the evaluation of video coding and reconstruction results can be broadly categorized into subjective quality assessment and objective quality assessment. Subjective evaluation metrics, based on user experience, are the closest to human visual characteristics and user experience, but they suffer from two significant drawbacks: firstly, conducting subjective experiments requires substantial human and material resources; secondly, they cannot be applied to encoder optimization. Therefore, objective quality assessment aims to find a directly calculable method to measure video reconstruction quality. Over the years, objective quality evaluation metrics have evolved from Peak Signal-to-Noise Ratio (PSNR) based on signal fidelity to metrics such as Structural Similarity (SSIM) and Video Multi-Method Evaluation Fusion (VMAF), which are more correlated with human subjective experience. However, currently, none of these metrics can comprehensively measure video reconstruction quality; they cannot completely replace it. Therefore, in practical applications, multiple objective evaluation metrics are often used in conjunction for reference to more accurately measure the performance of image and video encoders.

[0095] After years of development, existing video encoders still largely rely on distortion based on signal fidelity as the basis for rate-distortion optimization, especially in solving sample adaptive compensation filtering, due to considerations such as computational complexity. This makes it impossible to effectively explore the performance under a multi-objective evaluation system during the encoding process.

[0096] Based on the traditional sample-point adaptive compensation filtering, this embodiment can obtain the compensation solver corresponding to the Pareto optimal solution that can dominate the traditional sample-point adaptive filtering solution with low complexity, based on guidance information. This is beneficial for better coding performance under multiple objective quality collaborative evaluation systems and achieves an advantage over competing products.

[0097] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an adaptive compensation device in the video image encoding process. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0098] like Figure 6As shown, the adaptive compensation device 600 in the video image encoding process of this embodiment may include: an original pixel value acquisition unit 601, a reconstructed pixel value acquisition unit 602, an encoding category determination and multi-objective optimization unit 603, an offset value interval calculation unit 604, a desired offset value calculation unit 605, and a pixel value compensation unit 606. The original pixel value acquisition unit 601 is configured to acquire the original pixel value array of the original pixel block in the state to be encoded during loop filtering; the reconstructed pixel value acquisition unit 602 is configured to acquire the reconstructed pixel value array of the reconstructed pixel block obtained after reconstructing the original pixel block; the encoding category determination and multi-objective optimization unit 603 is configured to determine the encoding category to which each pixel in the reconstructed pixel block belongs, and to determine the target pixel value of each pixel when at least two optimization objectives are simultaneously satisfied; the offset value interval calculation unit 604 is configured to, for each pixel under each encoding category, calculate the target pixel value based on the pixel's corresponding value in the original pixel value array. The original pixel value and the reconstructed pixel value in the reconstructed pixel value array are used to determine a first offset value; a second offset value is determined based on the target pixel value and the reconstructed pixel value of the pixel; the offset value range of the pixel is determined based on the magnitude of the first offset value and the second offset value; the expected offset value calculation unit 605 is configured to perform frequency statistics on the integer offset values ​​contained in the offset value range of each pixel under each coding category, and determine the expected offset value of the corresponding coding category based on the obtained statistical results; the pixel value compensation unit 606 is configured to compensate the corresponding reconstructed pixel value of each pixel according to the expected offset value corresponding to its coding category.

[0099] In this embodiment, the specific processing and technical effects of the following components in the adaptive compensation device 600 during the video image encoding process—including the original pixel value acquisition unit 601, the reconstructed pixel value acquisition unit 602, the encoding category determination and multi-objective optimization unit 603, the offset value interval calculation unit 604, the desired offset value calculation unit 605, and the pixel value compensation unit 606—can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201-206 in the corresponding embodiments will not be repeated here.

[0100] In some other optional implementations of this embodiment, the encoding category determination and multi-objective optimization unit 603 includes a multi-objective optimization subunit configured to determine the target pixel value of each pixel when at least two optimization objectives are simultaneously satisfied. The multi-objective optimization subunit can be further configured to:

[0101] Based on the actual optimization requirements obtained, determine a set of optimization objectives that includes at least two different optimization goals;

[0102] For each pixel that constitutes the reconstructed pixel block, determine the target pixel value that simultaneously satisfies all optimization objectives in the optimization objective set.

[0103] In some other optional implementations of this embodiment, the optimization objective may include: optimization objectives that require reference to the original image and optimization objectives that do not require reference to the original image. Optimization objectives that require reference to the original image include at least one of the following: mean square error, peak signal-to-noise ratio, structural similarity, feature similarity, gradient magnitude similarity deviation, and video multi-method evaluation fusion. Optimization objectives that do not require reference to the original image include at least one of the following: sharpness enhancement, noise suppression, color enhancement, artifact repair, and subjective quality assessment model.

[0104] In some other optional implementations of this embodiment, in response to the actual resolution of the original image corresponding to the original pixel block being higher than the preset resolution, the optimization target required for determining the target pixel value of the obtained pixel includes at least one of the optimization targets that need to refer to the original image.

[0105] In some other optional implementations of this embodiment, in response to the actual resolution of the original image corresponding to the original pixel block being lower than a preset resolution, the optimization objective required for determining the target pixel value of the obtained pixel includes at least one of the optimization objectives that do not require reference to the original image.

[0106] In some other optional implementations of this embodiment, the coding category determination and multi-objective optimization unit 603 includes a coding category determination subunit configured to determine the coding category to which each pixel in the reconstructed pixel block belongs. The coding category determination subunit may include:

[0107] The target classification standard determination module is configured to determine the target encoding classification standard based on the encoding classification standard of the existing video encoder;

[0108] The encoding category determination module is configured to determine the encoding category of each pixel using the target encoding classification criteria.

[0109] In some other optional implementations of this embodiment, the target classification criterion determination module can be further configured as follows:

[0110] The existing video encoder's encoding classification standard is adjusted according to custom requirements to obtain the target encoding classification standard. The adjustment of the classification standard according to custom requirements includes: increasing the number of subclasses under the specified encoding category and adjusting the coverage of each subclass under the specified encoding category.

[0111] In some other optional implementations of this embodiment, the offset value interval calculation unit 604 can be further configured as follows:

[0112] For each pixel under each coding category, the following offset value range calculation steps are performed:

[0113] Subtract the reconstructed pixel value from the original pixel value of the pixel to obtain the first offset value;

[0114] Subtract the reconstructed pixel value from the target pixel value to obtain the second offset value;

[0115] The offset value range of the pixel is determined by using the smaller of the first offset value and the second offset value as the lower limit of the range and the larger value as the upper limit of the range.

[0116] The offset value interval calculation unit, the desired offset value calculation unit 605, can be further configured as follows:

[0117] For each pixel's offset value range under each coding category, the following steps are performed to calculate the expected offset value:

[0118] Iterate through all integer offset values ​​within the specified offset range;

[0119] For each integer offset value, the frequency of occurrence of each pixel in the offset value range under each coding category is counted, and the occurrence frequency is calculated based on the frequency.

[0120] The expected offset value corresponding to each encoding category is calculated based on the frequency of occurrence of each integer offset value in each encoding category.

[0121] In some other optional implementations of this embodiment, the adaptive compensation device 600 in the video image encoding process may further include:

[0122] The compensated pixel value determination unit is configured to determine the compensated pixel value after compensating the corresponding reconstructed pixel value for each pixel according to the expected offset value corresponding to its coding category;

[0123] The desired reconstruction pixel block determination unit is configured to determine the pixel block formed by pixels whose pixel value is the compensated pixel value as the desired reconstruction pixel block.

[0124] This embodiment exists as a device embodiment corresponding to the above method embodiment. This embodiment provides an adaptive compensation device for the video image encoding process. ，For the original pixel block to be encoded during the loop filtering process, the original pixel value array and the reconstructed pixel block array are first obtained. Then, the target pixel value of each pixel constituting the reconstructed pixel block is determined when multiple optimization objectives are simultaneously satisfied. Next, after determining the coding category to which each pixel belongs, for each pixel in each coding category, two offset values ​​are calculated based on the reconstructed pixel value, the original pixel value, and the target pixel value, respectively. The integer offset values ​​included in the offset value range are determined based on the offset value range determined in this way. Then, the expected offset value of the corresponding coding category is calculated based on the frequency of occurrence of each integer offset value in each offset value range in the corresponding category. Thus, adaptive compensation is performed according to the expected offset value corresponding to the coding category. In other words, this method can use offset values ​​that consider multiple optimization objectives to compensate for pixel values, thereby achieving better coding performance.

[0125] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the adaptive compensation method in the video image encoding process described in any of the above embodiments when executed.

[0126] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the adaptive compensation method in the video image encoding process described in any of the above embodiments when executed.

[0127] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the adaptive compensation method in the video image encoding process described in any of the above embodiments.

[0128] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0130] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the adaptive compensation method in the video image encoding process. For example, in some embodiments, the adaptive compensation method in the video image encoding process can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the adaptive compensation method in the video image encoding process described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured, by any other suitable means (e.g., by means of firmware), to perform an adaptive compensation method during the video image encoding process.

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0138] According to the technical solution of this disclosure, for the original pixel block to be encoded during the loop filtering process, firstly, the original pixel value array of the original pixel block and the reconstructed pixel block array of the reconstructed pixel block are obtained. Then, the target pixel value of each pixel constituting the reconstructed pixel block is determined when multiple optimization objectives are simultaneously satisfied. Next, after determining the coding category to which each pixel belongs, for each pixel under each coding category, two offset values ​​are calculated based on the reconstructed pixel value, the original pixel value, and the target pixel value, respectively. The integer offset values ​​included in the offset value range determined therefrom are then determined. Furthermore, the expected offset value of the corresponding coding category is calculated based on the frequency of occurrence of each integer offset value in each offset value range under the corresponding category. Thus, adaptive compensation is performed according to the expected offset value corresponding to the coding category. That is, in this way, offset values ​​that consider multiple optimization objectives can be used for pixel value compensation, thereby achieving better coding performance.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An adaptive compensation method in the video image encoding process, comprising: Obtain the array of original pixel values ​​for the original pixel blocks in the state to be encoded during the loop filtering process; Obtain the array of reconstructed pixel values ​​of the reconstructed pixel block obtained after reconstructing the original pixel block; The coding category of each pixel in the reconstructed pixel block is determined, and the target pixel value of each pixel is determined when at least two optimization objectives are satisfied simultaneously. The optimization objectives include: optimization objectives that require reference to the original image and optimization objectives that do not require reference to the original image. The optimization objectives that require reference to the original image include at least one of the following: mean square error, peak signal-to-noise ratio, structural similarity, feature similarity, gradient magnitude similarity deviation, and video multi-method evaluation fusion. The optimization objectives that do not require reference to the original image include at least one of the following: sharpness enhancement, noise suppression, color enhancement, artifact repair, and subjective quality assessment model. For each pixel under each of the coding categories, a first offset value is determined based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array corresponding to the pixel; a second offset value is determined based on the target pixel value and the reconstructed pixel value of the pixel; and the offset value range of the pixel is determined based on the magnitude of the first offset value and the second offset value. Frequency statistics are performed on the integer offset values ​​contained in the offset value range of each pixel under each coding category, and the expected offset value of the corresponding coding category is determined based on the obtained statistical results. For each pixel, the corresponding reconstructed pixel value is compensated according to the expected offset value corresponding to its coding category.

2. The method according to claim 1, wherein, The step of determining the target pixel value for each pixel when it simultaneously satisfies at least two optimization objectives includes: Based on the actual optimization requirements obtained, determine a set of optimization objectives that includes at least two different optimization goals; For each pixel constituting the reconstructed pixel block, determine the target pixel value that simultaneously satisfies all optimization objectives in the optimization objective set.

3. The method according to claim 1, wherein, In response to the fact that the actual resolution of the original image corresponding to the original pixel block is higher than the preset resolution, the optimization objective required to determine the target pixel value of the pixel includes at least one of the optimization objectives of the original image reference class.

4. The method according to claim 1, wherein, In response to the fact that the actual resolution of the original image corresponding to the original pixel block is lower than the preset resolution, the optimization objective required to determine the target pixel value of the pixel includes at least one of the optimization objectives that do not require reference to the original image.

5. The method according to claim 1, wherein, Determining the encoding category of each pixel in the reconstructed pixel block includes: Based on existing video encoder coding classification standards, determine the target coding classification standard; The coding category to which each pixel belongs is determined using the target coding classification criteria.

6. The method according to claim 5, wherein, The determination of the target encoding classification standard based on existing video encoder encoding classification standards includes: The existing video encoder's encoding classification standard is adjusted according to custom requirements to obtain the target encoding classification standard; wherein, adjusting the classification standard according to the custom requirements includes: increasing the number of subclasses under the specified encoding category and adjusting the coverage of each subclass under the specified encoding category.

7. The method according to claim 1, wherein, For each pixel under each of the coding categories, a first offset value is determined based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array corresponding to that pixel. The second offset value is determined based on the target pixel value and the reconstructed pixel value of the pixel; The offset value range of the pixel is determined based on the magnitudes of the first offset value and the second offset value, including: For each pixel under each of the aforementioned coding categories, the following offset value range calculation steps are performed: The first offset value is obtained by subtracting the reconstructed pixel value from the original pixel value of the pixel. The second offset value is obtained by subtracting the reconstructed pixel value from the target pixel value of the pixel. The offset value range of the pixel is determined by using the smaller of the first offset value and the second offset value as the lower limit of the range and the larger value as the upper limit of the range.

8. The method according to claim 1, wherein, The step of performing frequency statistics on the integer offset values ​​contained in the offset value intervals of all pixels under each coding category, and determining the expected offset value of the corresponding coding category based on the obtained statistical results, includes: For each pixel's offset value range under each of the aforementioned coding categories, the following expected offset value calculation steps are performed: Iterate through all integer offset values ​​within the specified offset range; For each integer offset value, the frequency of occurrence of each pixel in the offset value range under each encoding category is counted, and the occurrence frequency is calculated based on the frequency. The expected offset value corresponding to each encoding category is calculated based on the frequency of occurrence of each integer offset value under each encoding category.

9. The method according to any one of claims 1-8, further comprising: Determine the compensated pixel value after compensating the corresponding reconstructed pixel value for each pixel according to the expected offset value corresponding to its coding category; The pixel block formed by pixels whose pixel value is the compensated pixel value is determined as the desired reconstructed pixel block.

10. An adaptive compensation device in a video image encoding process, comprising: The raw pixel value acquisition unit is configured to acquire the raw pixel value array of the raw pixel block in the encoding state during the loop filtering process; The reconstructed pixel value acquisition unit is configured to acquire the reconstructed pixel value array of the reconstructed pixel block obtained after the original pixel block is reconstructed; The coding category determination and multi-objective optimization unit is configured to determine the coding category to which each pixel in the reconstructed pixel block belongs, and to determine the target pixel value of each pixel when it simultaneously satisfies at least two optimization objectives. The optimization objectives include: optimization objectives requiring reference to the original image and optimization objectives not requiring reference to the original image. The optimization objectives requiring reference to the original image include at least one of the following: mean square error, peak signal-to-noise ratio, structural similarity, feature similarity, gradient magnitude similarity deviation, and video multi-method evaluation fusion. The optimization objectives not requiring reference to the original image include at least one of the following: sharpness enhancement, noise suppression, color enhancement, artifact repair, and subjective quality assessment model. The offset value interval calculation unit is configured to, for each pixel under each coding category, determine a first offset value based on the original pixel value in the original pixel value array and the reconstructed pixel value in the reconstructed pixel value array corresponding to the pixel; determine a second offset value based on the target pixel value and the reconstructed pixel value of the pixel; and determine the offset value interval of the pixel based on the magnitude of the first offset value and the second offset value. The expected offset value calculation unit is configured to perform frequency statistics on the integer offset values ​​contained in the offset value intervals of all pixels under each coding category, and determine the expected offset value of the corresponding coding category based on the obtained statistical results. The pixel value compensation unit is configured to compensate the corresponding reconstructed pixel value for each pixel according to the expected offset value corresponding to its coding category.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the adaptive compensation method in the video image encoding process according to any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the adaptive compensation method in the video image encoding process of any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the adaptive compensation method in the video image encoding process according to any one of claims 1-9.

Images