An image processing method, an electronic device, a storage medium, and a program product

Abstract

The application provides an image processing method, an electronic device, a storage medium and a program product. The method comprises the following steps: acquiring a pixel to be processed corresponding to an image block; filtering the pixel according to the correlation between the pixel and adjacent pixels to determine a filtered pixel; and filtering the pixel to be processed corresponding to the image block according to the correlation between the pixel and adjacent pixels. The method improves or eliminates the problem of low image processing accuracy caused by uneven pixel value distribution, and improves the coding and decoding efficiency of the image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, storage medium, and program product. Background Technology

[0002] Inter-predictor correction (IPC) is a coding tool in AVS4 used to compensate for illumination differences between encoded and reference frames. Since video images exhibit illumination differences in the temporal domain, IPC can compensate for these differences, thereby improving coding efficiency. Currently, AVS4's IPC algorithm has shortcomings in the following aspects: the four pixel pairs used to derive the illumination compensation model coefficients are either distributed near the beginning and end positions of the top row and / or left column of the block, or evenly distributed (e.g., at positions 0, 1 / 4, 2 / 4, 3 / 4, etc.). When the pixel value distribution of the block is uneven and has high dispersion, the selected pixels affect the derived model coefficients, leading to low accuracy. Therefore, how to solve the problem of uneven pixel value distribution is an issue that needs to be addressed. Summary of the Invention

[0003] This application provides an image processing method, electronic device, storage medium, and program product to solve the problem of uneven pixel distribution.

[0004] To achieve the above objectives, embodiments of this application provide an image processing method, including:

[0005] Obtain the pixels to be processed corresponding to the image patch;

[0006] The pixel is filtered based on its correlation with neighboring pixels to determine the filtered pixel.

[0007] To achieve the above objectives, embodiments of this application provide another image processing method applied at the encoding end, including:

[0008] Acquire an image to be encoded, the image to be encoded comprising data blocks;

[0009] Motion compensation is performed on the data blocks to obtain the prediction blocks;

[0010] The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method provided in any embodiment of this application.

[0011] The image is encoded based on the processing results to form the encoded image.

[0012] To achieve the above objectives, embodiments of this application provide an image processing method applied at a decoding end, comprising:

[0013] Acquire an image to be decoded, the image to be decoded comprising data blocks;

[0014] Motion compensation is performed on the data blocks to obtain the prediction blocks;

[0015] The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method provided in any embodiment of this application.

[0016] The image is decoded based on the processing results to form the decoded image.

[0017] To achieve the above objectives, embodiments of this application provide an electronic device, including: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the image processing method as described in any one of the embodiments of this application.

[0018] To achieve the above objectives, embodiments of this application provide a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the image processing method described in any one of the embodiments of this application.

[0019] To achieve the above objectives, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method described in any one of the embodiments of this application.

[0020] The image processing method, electronic device, storage medium, and program product provided in this application embodiment obtain the pixels to be processed corresponding to the image block; filter the pixels according to the correlation between the pixels and neighboring pixels to determine the filtered pixels; and filter the pixels to be processed corresponding to the image block according to the correlation between the pixels and neighboring pixels, thereby improving or eliminating the problem of low image processing accuracy caused by uneven pixel value distribution and improving image encoding and decoding efficiency.

[0021] Further details regarding the above embodiments and other aspects of this application, as well as their implementations, are provided in the accompanying drawings, detailed description, and claims. Attached Figure Description

[0022] Figure 1a This is an example image of pixels taken from an IC_TL model;

[0023] Figure 1b Example image of pixels taken for another IC_TL model;

[0024] Figure 2a Here is an example image of a pixel pair;

[0025] Figure 2b Here is an example image of another type of pixel pair;

[0026] Figure 3 A flowchart illustrating an image processing method provided in one embodiment;

[0027] Figure 4 An example diagram of a prediction block provided for one embodiment;

[0028] Figure 5 An example diagram of a current block is provided for one embodiment;

[0029] Figure 6 Example diagram of another prediction block provided for one embodiment;

[0030] Figure 7 A schematic diagram of a prediction block after filtering is provided in one embodiment;

[0031] Figure 8 Example diagram of another current block provided for one embodiment;

[0032] Figure 9 A flowchart of another image processing method provided in one embodiment;

[0033] Figure 10 A flowchart of another image processing method provided in one embodiment;

[0034] Figure 11a An example diagram of a smoothing filter for a prediction block is provided for one embodiment;

[0035] Figure 11b An example diagram of a smoothing filter for a current block is provided for one embodiment;

[0036] Figure 12a An example diagram of a prediction block before downsampling is provided for one embodiment;

[0037] Figure 12b An example diagram of a prediction block after downsampling is provided for one embodiment;

[0038] Figure 13a An example diagram is provided for one embodiment of the current block before downsampling;

[0039] Figure 13b An example diagram showing the current block downsampled as provided in one embodiment;

[0040] Figure 14a An example diagram of a prediction block provided for one embodiment;

[0041] Figure 14b An example diagram of a current block is provided for one embodiment;

[0042] Figure 15a An example diagram of a prediction block provided for one embodiment;

[0043] Figure 15b An example diagram of a current block is provided for one embodiment;

[0044] Figure 16a An example diagram of a prediction block before downsampling is provided for one embodiment;

[0045] Figure 16b An example diagram of a predicted block after downsampling is provided for one embodiment;

[0046] Figure 17a An example diagram is provided for one embodiment of the current block before downsampling;

[0047] Figure 17b An example diagram showing the current block downsampled as provided in one embodiment;

[0048] Figure 18a An example diagram of a prediction block before downsampling is provided for one embodiment;

[0049] Figure 18b An example diagram of a predicted block after downsampling is provided for one embodiment;

[0050] Figure 19a An example diagram is provided for one embodiment of the current block before downsampling;

[0051] Figure 19b An example diagram showing the current block downsampled as provided in one embodiment;

[0052] Figure 20 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment;

[0053] Figure 21 A schematic diagram of the structure of another image processing apparatus provided in one embodiment;

[0054] Figure 22 A schematic diagram of the structure of another image processing apparatus provided in one embodiment;

[0055] Figure 23 This is a schematic diagram of the structure of an electronic device provided in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be arbitrarily combined with each other.

[0057] IPC technology is a coding method proposed in AVS4 for direct / skip mode to compensate for illumination differences between coded frames and reference frames. Its technical features include:

[0058] (1) Correct the predicted blocks after motion compensation in direct / skip mode (IPC is applied to the predicted blocks obtained by motion compensation) to make the reconstructed image closer to the original image.

[0059] (2) Three illumination compensation models are adopted, namely IC_TL, IC_T and IC_L. IC_TL calculates the compensation model by taking the reconstructed pixels on the top and left sides and the predicted pixels on the left side. IC_T calculates the compensation model by taking the reconstructed pixels on the top side and the predicted pixels on the right side. IC_L calculates the compensation model by taking the reconstructed pixels on the left side and the predicted pixels on the left side. Figure 1a An example image of the pixels taken by the IC_TL model is provided, which shows the pixels taken by the predicted block in the image. Figure 1b An example image of the pixels taken by another IC_TL model is provided, which shows the pixels taken by the data blocks in the image.

[0060] (3) Based on the availability of adjacent pixel blocks, four available pixel pairs are obtained in three cases. a and b are calculated using the four available pixel pairs. After obtaining a and b, the corrected prediction value is obtained using the following formula:

[0061] Pred′(x,y)=a*pred(x,y)+b

[0062] Where pred(x,y) is the predicted value of the current block in the current frame located at (x,y), a is the scaling factor, b is the offset factor, and Pred′(x,y) is the final predicted value of (x,y).

[0063] The AVS4 IPC algorithm derives the four-pixel pairs of the illumination compensation model coefficients, for example, by distributing them near the beginning and end of the upper row and / or left column of the block, or by distributing them evenly, for example, at positions 0, 1 / 4, 2 / 4, 3 / 4, etc. Figure 2a An example image of pixel pairs is provided, as shown by the black dots in the image. Figure 2a This displays the pixel pairs of the predicted block; Figure 2b Another example image of pixel pairs is provided, as shown by the black dots in the image. Figure 2b This displays pixel pairs of data blocks.

[0064] The derivation process of parameters a and b is shown below:

[0065] 1) Count the pixels in the left row and top column of the current encoding CU block, referring to the pixels in the top row and left column inside the CU block, and then sort them in ascending order. This results in two sets of sorted pixel values, denoted as CountRec[N] and CountPred[N], where N is the number of pixels.

[0066] 2) In these two sets of data, select two smaller points and two larger points respectively, using the following formula:

[0067] For the two smaller points:

[0068] minIndex1 = (N == 4) ? 0:1

[0069] Rec min1 =CountRec[minIndex1]

[0070] Ref min1 =CountRef[minIndex1]

[0071] minIndex2=(N-2)>>2+1

[0072] Rec min2 =CountRec[minIndex2]

[0073] Ref min2 =CountRef[minIndex2]

[0074] For the two larger points:

[0075] maxIndex1=(N==4)? N-1:N-2

[0076] Rec max1 =CountRec[maxIndex1]

[0077] Ref max1 =CountRef[maxIndex1]

[0078] maxIndex2=N-(N-2)>>2-2

[0079] Rec max2 =CountRec[maxIndex2]

[0080] Ref max2 =CountRef[maxIndex2]

[0081] 3) After obtaining the corresponding pixel value, the calculation process is as follows:

[0082] y max =(Rec max1 +Rec max2 +1)>>1

[0083] y min =(Rec min1 +Rec min2 +1)>>1

[0084] x max =(Ref max1 +Ref max2 +1)>>1

[0085] x min =(Ref min1 +Rec min2 +1)>>1

[0086]

[0087] b = y min -a*x min

[0088] When selecting 4 pairs of available pixels, the availability of the top and left pixels needs to be considered, which can be divided into the following 3 cases:

[0089] 1) If both the top and left pixels of the current block are "available", then select 2 pixel pairs from the top and 2 pixel pairs from the left.

[0090] 2) If only the top pixel is available in the current block, then all 4 pixel pairs are selected from the top, and the selected positions are: 0 / 4, 1 / 4, 2 / 4, 3 / 4 of the width;

[0091] 3) If only the left pixel is available in the current block, then all 4 pixel pairs are selected from the left, and the selected positions are: 0 / 4, 1 / 4, 2 / 4, 3 / 4 of the height.

[0092] (4) In terms of high-level syntax representation, IPC sets up two levels of syntax flags for control: frame-level and block-level. Frame-level syntax flags are only applied to Skip mode. If the frame-level flag of the current frame is negative, the skip mode of the current frame will not attempt illumination compensation processing; otherwise, illumination compensation prediction will be attempted. The frame-level flag is determined before the current frame is encoded, based on the brightness histogram (SAD) of the current frame and each reference frame. If the SAD is greater than a preset threshold, the frame-level flag is enabled. Block-level syntax flags are applied to direct / skip mode. The encoder uses RDO to decide whether to use illumination compensation for the current block. If illumination compensation is required, it needs to be indicated in the bitstream to indicate the use of illumination compensation and the illumination compensation model. The block-level syntax flags are shown in Table 1 below.

[0093] Table 1 Block-level syntax identifiers

[0094] IC_Index IC_flag IC_index0 IC_index1 No_IC 0 - - IC_TL 1 1 - IC_T 1 0 1 IC_L 1 0 0

[0095] The filtering provided in this application is applicable to the derivation of IPC model coefficients, and also applicable to other encoding tools in AVS, such as TSCPM.

[0096] Figure 3 A flowchart of an image processing method provided in one embodiment, such as Figure 3 As shown, the method includes S110-S120:

[0097] S110. Obtain the pixels to be processed corresponding to the image block.

[0098] In this context, an image block can be understood as a region obtained by dividing a frame of an image. For example, during image encoding, a frame of an image is acquired and divided according to certain rules to obtain different image blocks. An image block includes one or more pixels, usually multiple pixels, and one image block is a processing unit. For example, in the embodiments of this application, an image block can refer to an image block used to derive the illumination compensation model coefficients during inter-frame prediction value correction of an image; this image block can be an encoding block or a decoding block.

[0099] When processing an image, different image patches are obtained. These patches can be data blocks from the original image or images obtained by processing data blocks from the original image, such as predicted patches. For each image patch, the number of pixels to be processed can be one or more; for example, the pixels to be processed might be the pixels in the top row of the image patch, which are pixels inside the image patch; or, the pixels to be processed might be the pixels in the top row of the image patch, which are pixels outside the image patch, and so on.

[0100] S120. Filter the pixel based on the correlation between the pixel and its neighboring pixels to determine the filtered pixel.

[0101] In this process, neighboring pixels can be pixels adjacent to a given pixel or pixels within a certain distance range. The process involves analyzing the pixel to be processed and determining the neighboring pixels for each pixel. For a given pixel, the influence of neighboring pixels at different distances may differ; that is, the correlation between neighboring pixels at different distances and the pixel varies. For example, the closer the neighboring pixels are, the greater their correlation. The process involves analyzing the pixel and its corresponding neighboring pixels, and filtering the pixel based on the correlation between the pixel and its neighboring pixels to obtain the filtered pixel. For example, the pixel can be weighted according to its distance from other neighboring pixels to obtain the filtered pixel.

[0102] The correlation between pixels and their neighboring pixels can also be called spatial correlation.

[0103] In this embodiment, when filtering pixels to be processed, filtering is performed based on the pixel and its corresponding neighboring pixels to obtain filtered pixels. When there are multiple pixels to be processed, all pixels are processed sequentially to obtain filtered pixels.

[0104] The image processing method provided in this application embodiment obtains the pixels to be processed corresponding to an image block; filters the pixels based on the correlation between the pixels and neighboring pixels to determine the filtered pixels; and filters the pixels to be processed corresponding to the image block based on the correlation between the pixels and neighboring pixels, thereby improving or eliminating the problem of low image processing accuracy caused by uneven pixel value distribution and improving image encoding and decoding efficiency.

[0105] In some embodiments, the image block includes at least one of the following:

[0106] Data blocks are obtained by dividing the image;

[0107] Prediction blocks are obtained by predicting data blocks obtained from dividing the image.

[0108] In this process, an image block can be at least one of a data block and a prediction block. That is, when performing image processing using the method provided in this application embodiment, data blocks can be processed, prediction blocks can be processed, or both data blocks and prediction blocks can be processed. The image is divided into regions, each of which is a data block. During image processing, each processed data block can be recorded as the current block. Prediction is performed on the data blocks to obtain prediction blocks; for example, motion compensation is applied to the data blocks to obtain prediction blocks.

[0109] An image is divided into different data blocks. During image processing, each data block and its corresponding prediction block are processed sequentially. In each processing iteration, the data block is treated as the current block, and the prediction block is the corresponding prediction block. In image decoding, the current block can be called the current decoding unit or decoding block; in image encoding, the current block can be called the current encoding unit or encoding block.

[0110] In some embodiments, the pixels to be processed include at least one of the following:

[0111] Predict the pixels in the top row of the block;

[0112] Predict the pixels in the leftmost column of the block;

[0113] The pixels in the row above the predicted block;

[0114] The pixels in the left column of the predicted block;

[0115] The number of pixels in the row above the data block;

[0116] The pixels in the left column of the data block;

[0117] The pixels in the top two rows of the data block;

[0118] The pixels in the two columns to the left of the data block.

[0119] The pixels to be processed are at least one of the above, which can be a row of pixels, a column of pixels, or multiple rows or columns of pixels. The above pixel rows or columns can be combined.

[0120] For example, when the image patch is a prediction patch, the pixels to be processed are: the pixels in the top row of the prediction patch and the pixels in the left column of the prediction patch. This is equivalent to processing one row and one column of pixels. Alternatively, the pixels in the top row of the prediction patch, the pixels in the top row of the prediction patch, the pixels in the left column of the prediction patch, and the pixels in the left column of the prediction patch. This is equivalent to processing two rows and two columns of pixels.

[0121] When the image block is a data block, the pixels to be processed are: the pixels in the top row of the data block and the pixels in the left column of the data block. This is equivalent to processing one row and one column of pixels; or, the pixels in the top two rows of the data block and the pixels in the top two columns of the data block. This is equivalent to processing two rows and two columns of pixels.

[0122] In some embodiments, filtering a pixel based on its correlation with neighboring pixels to determine the filtered pixel includes:

[0123] If the filtered pixel coordinates are not the first or last pixel coordinates, determine the first pixel and at least two first neighboring pixels corresponding to the pixel coordinates, and perform a weighted operation based on the first pixel and the first neighboring pixels to obtain the filtered pixel corresponding to the pixel coordinates.

[0124] Given that the filtered pixel coordinates are the first and last pixel coordinates, determine the second pixel and at least one second neighboring pixel corresponding to the pixel coordinates. Perform a weighted operation based on the second pixel and the second neighboring pixel corresponding to the pixel coordinates to obtain the filtered pixel corresponding to the pixel coordinates.

[0125] In this context, the first pixel can be understood as the pixel corresponding to the filtered pixel coordinates. The first pixel can be the pixel whose pixel coordinates correspond to the image or image patch before filtering. Alternatively, in the case of downsampling or upsampling filtering, the first pixel can be the pixel whose pixel coordinates are mapped to the image or image patch before filtering. The first neighboring pixel can be understood as a pixel adjacent to the first pixel, such as left-right or top-bottom adjacent pixels. The second pixel can be the pixel whose pixel coordinates correspond to the image or image patch before filtering. Alternatively, in the case of downsampling or upsampling filtering, the second pixel can be the pixel whose pixel coordinates are mapped to the image or image patch before filtering. The second neighboring pixel can be understood as a pixel adjacent to the first pixel, such as left-right or top-bottom adjacent pixels.

[0126] During the filtering process, the pixel coordinates of the same pixel may change before and after filtering. For example, during downsampling, the size of the image block before filtering is 16*16, and the size of the image block after filtering is also 16*16. In this case, the coordinates of the filtered pixels can correspond one-to-one with the coordinates of the pixels before filtering. However, during downsampling, the size of the image block before filtering is 16*16, and the size of the image block after filtering is 8*8. In this case, the coordinates of the filtered pixels cannot correspond one-to-one with the coordinates of the pixels before filtering. In this embodiment, the determination of the filtered pixels is based on the pixel coordinates after filtering. That is, the first pixel and the first neighboring pixel corresponding to the filtered pixel in the image block before filtering are determined based on the pixel coordinates after filtering, or the second pixel and the second neighboring pixel corresponding to the filtered pixel in the image block before filtering are determined based on the pixel coordinates after filtering.

[0127] The filtered pixel coordinates are determined, and the corresponding filtered pixel can be determined using the method provided in this application for each pixel coordinate. It is determined whether the pixel coordinate is the first or last pixel coordinate (i.e., the first or last pixel of a row or column). If the filtered pixel coordinate is not the first or last pixel coordinate, its corresponding first pixel and first neighboring pixels are determined based on the pixel coordinate. The first pixel can be directly mapped based on the pixel coordinate, and at least two first neighboring pixels are determined based on the pixel coordinate. For example, the first neighboring pixels are one or more of the following: vertically adjacent, horizontally adjacent, or pixels within a preset distance. A weighted operation is performed on the first pixel and the first neighboring pixels, and the result of the weighted operation is used as the filtered pixel corresponding to the pixel coordinate.

[0128] When the filtered pixel coordinates are the coordinates of the first and last pixels, the corresponding second pixel and second neighboring pixel are determined based on the pixel coordinates. The second pixel can be directly mapped based on its pixel coordinates. At least one second neighboring pixel is determined based on the pixel coordinates; for example, the second neighboring pixel can be one or more of the following: above neighbor, below neighbor, left neighbor, and right neighbor. A weighted operation is performed on the second pixel and the second neighboring pixel, and the result of the weighted operation is used as the filtered pixel corresponding to the pixel coordinates.

[0129] In some embodiments, the first pixel includes at least one of the following:

[0130] The pixel corresponding to the pixel coordinates;

[0131] The pixel coordinates corresponding to a preset multiple of the pixel coordinates, where the preset multiple is determined based on the sampling multiple.

[0132] The first pixel can be the pixel whose pixel coordinates are directly mapped to the corresponding coordinates in the image patch. For example, if the size of the image patch does not change before and after filtering, and the pixel coordinates after filtering correspond one-to-one with the pixel coordinates before filtering, then the first pixel is the pixel corresponding to the pixel coordinates in the image patch. For instance, the pixel coordinates after filtering are (x, 0), and the pixel value of (x, 0) before filtering is P. x,0 Since the pixel coordinates correspond one-to-one before and after filtering, the first pixel in this case is P. x,0 .

[0133] The first pixel can also be a pixel determined based on pixel coordinates and a preset multiple, where the preset multiple is determined by the sampling multiple and is the reciprocal of the sampling multiple. For example, when downsampling an image block, reducing a 16*16 image block to an 8*8 image block, the pixel coordinates before filtering are twice the pixel coordinates after filtering, the sampling multiple is 1 / 2, and the preset multiple is 2. The pixel coordinates after filtering are multiplied by the preset multiple to obtain their corresponding coordinates in the image block before filtering, and this pixel with the corresponding coordinates is taken as the first pixel. For example, if the pixel coordinates after filtering are (x, 0), and the pixel at (2x, 0) is P before filtering. 2x,0 Since the filtered pixel coordinates (x, 0) are mapped back to the unfiltered coordinates (2x, 0) according to the sampling factor, the first pixel in this case is P. 2x,0 .

[0134] In some embodiments, the first neighboring pixel includes at least one of the following:

[0135] The two pixels whose pixel coordinates are adjacent to each other on the left and right;

[0136] Two pixels whose pixel coordinates are adjacent vertically;

[0137] The two pixels to the left and right of the coordinates corresponding to a preset multiple of the pixel coordinates;

[0138] The two pixels directly above and below each other at a preset multiple of the pixel coordinates;

[0139] The first and last two pixels of the row containing the pixel coordinates;

[0140] The first and last two pixels of the column containing the pixel coordinates;

[0141] The pixels above the two adjacent pixels to the left and right of the preset multiple of the pixel coordinates, and the pixels above the coordinates corresponding to the preset multiple of the pixel coordinates;

[0142] The pixel to the left of the two adjacent pixels above and below the preset multiple of the pixel coordinates, and the pixel to the left of the preset multiple of the pixel coordinates;

[0143] The preset multiple is determined based on the sampling multiple.

[0144] The first neighboring pixel can be any one of the above, or any combination of several. For example, if the filtered pixel coordinates are a pixel in a pixel row, then its corresponding first neighboring pixel can be one of the following: 1. The two pixels to the left and right of the pixel coordinates; 2. The two pixels to the left and right of the coordinates corresponding to a preset multiple of the pixel coordinates; 3. The two pixels to the left and right of the pixel coordinates, and the two pixels at the beginning and end of the row where the pixel coordinates are located; 4. The two pixels to the left and right of the coordinates corresponding to a preset multiple of the pixel coordinates, and the two pixels at the beginning and end of the row where the pixel coordinates are located; 5. The two pixels to the left and right of the coordinates corresponding to a preset multiple of the pixel coordinates, the pixel above the two pixels to the left and right of the coordinates corresponding to a preset multiple of the pixel coordinates, and the pixel above the coordinates corresponding to a preset multiple of the pixel coordinates.

[0145] For example, if the filtered pixel coordinates are a pixel in a pixel column, then its corresponding first neighboring pixel can be one of the following: 1. Two pixels directly above and below the pixel coordinates; 2. Two pixels directly above and below the coordinates corresponding to a preset multiple of the pixel coordinates; 3. Two pixels directly above and below the pixel coordinates, and the first and last pixels of the column containing the pixel coordinates; 4. Two pixels directly above and below the coordinates corresponding to a preset multiple of the pixel coordinates, and the first and last pixels of the column containing the pixel coordinates; 5. Two pixels directly above and below the coordinates corresponding to a preset multiple of the pixel coordinates, the pixel to the left of the two pixels directly above and below the coordinates corresponding to a preset multiple of the pixel coordinates, and the pixel to the left of the coordinates corresponding to a preset multiple of the pixel coordinates.

[0146] In some embodiments, the second pixel includes at least one of the following:

[0147] The pixel corresponding to the pixel coordinates;

[0148] The first coordinate corresponds to the pixel, and the first coordinate is determined based on the pixel coordinate and the sampling factor.

[0149] The second pixel can be a pixel whose pixel coordinates are directly mapped to the corresponding coordinates in the image patch. For example, if the size of the image patch remains unchanged before and after filtering, and the pixel coordinates after filtering correspond one-to-one with the pixel coordinates before filtering, then the second pixel is the pixel corresponding to the pixel coordinates in the image patch. For instance, if the pixel coordinates after filtering are (0, 0), then the pixel at (0, 0) before filtering is P. 0,0 Since the pixel coordinates correspond one-to-one before and after filtering, the first pixel in this case is P. 0,0The second pixel can also be the pixel whose coordinates have been transformed and mapped to the corresponding pixel in the image or image patch before filtering. The second pixel is the pixel with the first coordinate, which is the coordinate of the pixel in the image patch before sampling. The first coordinate can be determined based on the pixel coordinate and the sampling factor. For example, the second coordinate is the reciprocal of the sampling factor * pixel coordinate + 1. For instance, the filtered pixel coordinates are (w-1, 0), and the pixel (2w-1, 0) before filtering is P. 2w-1,0 Since the filtered pixel coordinates (w-1, 0) are mapped back to the unfiltered coordinates based on the sampling factor, their first coordinates are (2w-1, 0). In this case, the second pixel is P. 2w-1,0 .

[0150] In some embodiments, the second neighboring pixel includes at least one of the following:

[0151] The pixel to the left of the pixel coordinates;

[0152] The pixel to the right of the pixel coordinates;

[0153] The pixel adjacent to the pixel coordinate above;

[0154] The pixel adjacent to the pixel coordinate below;

[0155] The pixel to the left of the first coordinate;

[0156] The pixel to the right of the first coordinate;

[0157] The pixel adjacent to the first coordinate;

[0158] The pixel adjacent to the first coordinate;

[0159] The first coordinate is determined based on the pixel coordinate and the sampling factor.

[0160] The second neighboring pixel can be any one of the above, or any combination thereof, and can be determined based on the specific position of the filtered coordinates. The specific position of the filtered coordinates determines whether there are pixels above, below, to the left, and to the right of the pixel, thus determining its corresponding second neighboring pixel. For example, if the filtered pixel coordinates are the last pixel in the top row, this pixel, being the only pixel adjacent to it on its left, has its corresponding second neighboring pixel as the pixel to its left. For example, if the filtered pixel coordinates are a pixel in a pixel row, its corresponding second neighboring pixel can be one of the following: 1. The pixel to the left of the first pixel coordinate; 2. The pixel to the right of the first pixel coordinate; 3. The pixel above the first pixel coordinate; 4. The pixel to the left of the first pixel coordinate; 5. The pixel to the right of the first pixel coordinate; 6. The pixel above the first pixel coordinate. If the filtered pixel coordinates are a pixel in a pixel column, then its corresponding second neighboring pixel can be one of the following: 1. The pixel above the pixel coordinates; 2. The pixel below the pixel coordinates; 3. The pixel to the left of the pixel coordinates; 4. The pixel above the pixel coordinates and the pixel to the left of the pixel coordinates; 5. The pixel below the first coordinate; 6. The pixel above the first coordinate; 7. The pixel to the left of the first coordinate; 8. The pixel above the first coordinate and the pixel to the left of the first coordinate.

[0161] In some embodiments, the weighted calculation based on the first pixel and the first neighboring pixels includes:

[0162] The first pixel and its first neighboring pixel are weighted according to their respective weighting coefficients, and the weighted result is then rounded down.

[0163] The weighting coefficients are determined based on one or more factors, such as the number of first neighboring pixels, the coordinates of the first neighboring pixels, and the position of the filtered pixel coordinates. Different weighting coefficients can be used when selecting different numbers and positional relationships of first neighboring pixels for filtering. Different weighting coefficients can be preset. After determining the first neighboring pixels, an appropriate weighting coefficient is selected, and a weighted operation is performed on the first pixel and its first neighboring pixels according to the weighting coefficients. The weighting operation can be a weighted summation, and then the sum is divided by the sum of the weighting coefficients to obtain the mean. The mean may contain decimals; finally, the weighted result is rounded down to the nearest integer, and the resulting value is the filtered pixel.

[0164] In some embodiments, a weighted calculation is performed based on the second pixel and the second neighboring pixel, including:

[0165] The second pixel and its second neighboring pixel are weighted according to their respective weighting coefficients, and the weighted result is then rounded down.

[0166] Similarly, weighting coefficients for the second pixel and the second neighboring pixel are preset, and the first pixel and the first neighboring pixel are weighted according to the weighting coefficients. The weighted result is then rounded down to obtain the filtered pixel.

[0167] The image block processing in this application can be mainly divided into the following methods: smoothing filtering of the top row and / or left column of pixels of the block; and downsampling smoothing filtering of the top row and / or left column of pixels of the block.

[0168] For example, Figure 4An example image of a prediction block is provided, where the circles in the image represent pixels that need to be smoothed, including the top row and / or the left column. 1. Taking the example where the image size does not change before and after smoothing, as shown in the figure, with the coordinates of B as (x, 0) and the filtered pixel coordinates as (x, 0), the filtering process is explained as follows: When (x, 0) is not the coordinates of the first or last pixel, the first pixel is the pixel of B (i.e., the pixel corresponding to the pixel coordinate), and the first neighboring pixels are A and C (i.e., the two pixels adjacent to each other on the left and right), or the first neighboring pixels are A, C, D, and E (i.e., the two pixels adjacent to each other on the left and right, and the two pixels at the beginning and end). 2. Taking downsampling smoothing filtering as an example, the image size changes before and after sampling, as shown in the figure. Taking the coordinates of B as (2x, 0) and the filtered pixel coordinates as (x, 0) as an example, the filtering process is explained as follows: When (x, 0) is not the first or last pixel, the first pixel is the pixel of B (i.e., the pixel corresponding to the coordinate of a preset multiple of the pixel coordinates), the first neighboring pixels are the pixels of A and C (i.e., the two pixels adjacent to each other on the left and right of the coordinates corresponding to the preset multiple of the pixel coordinates), or the first neighboring pixels are the pixels of A, C, D, and E (i.e., the two pixels adjacent to each other on the left and right of the coordinates corresponding to the preset multiple of the pixel coordinates, and the two pixels at the beginning and end of the column). The filtering principle of the pixels in the left column is the same as that of the top row, that is, the filtered pixels are determined by combining the two pixels adjacent to each other on the left and right of the pixel coordinates, the two pixels at the beginning and end of the column, and the two pixels adjacent to each other on the left and right of the coordinates corresponding to the preset multiple of the pixel coordinates. For the specific implementation, please refer to the filtering of the pixels in the top row.When (x, 0) are the first and last pixel coordinates, if the image size remains unchanged before and after smoothing filtering, pixels D, E, and F in the image represent the first and last pixels before filtering, and their coordinates correspond one-to-one with the first and last pixel coordinates after filtering. The second pixel corresponding to the filtered pixel coordinate D'(0, 0) is the pixel of D. D' has two second neighboring pixels: one to the right of D and one below D. During filtering, one of these can be selected, or both can be used. The second pixel corresponding to the filtered pixel coordinate E'(15, 0) is the pixel of E, and the second neighboring pixel of E' is the pixel to the left of E. The second pixel corresponding to the filtered pixel coordinate F'(0, 15) is the pixel of F, and the second neighboring pixel of F' is the pixel above F. One; If the image size changes before and after smoothing filtering, taking an image size of 16*16 before filtering and 8*8 after filtering as an example, the second pixel corresponding to the filtered pixel D'(0,0) is the pixel of D, where the coordinate of D is the first coordinate. There are two second neighboring pixels, namely the one to the right of D and the one below D. During filtering, one of them can be selected for filtering, or both can be used; the second pixel corresponding to the filtered pixel E'(7,0) is the pixel of E, and the second neighboring pixel corresponding to E' is the one to the left of E, where the coordinate of E is the first coordinate; the second pixel corresponding to the filtered pixel F'(0,7) is the pixel of F, and the second neighboring pixel corresponding to F' is the one above F, where the coordinate of F is the first coordinate.

[0169] For example, Figure 5 An example image of the current block is provided. The filtering process for the current block is basically the same as that for the prediction block, except that the current block has four pixels at the beginning and end, and each pixel at the beginning and end corresponds to one second neighbor pixel. For example, when the image size does not change, the second neighbor pixel is the pixel to the right, left, below, or above the pixel coordinate. Alternatively, when the width of the filtered image block is half the width of the original image block and the height of the filtered image block is half the height of the original image block, the second neighbor pixel is the pixel to the right, left, below, or above the first coordinate. The first coordinate is the coordinate of the four beginning and end pixels before filtering, which correspond to the four beginning and end pixels after filtering. The filtering process for the current block can be referred to the above description.

[0170] For example, Figure 6Another example image of the prediction block is provided. The circles in the image represent pixels that need to be smoothed, including the top two rows and / or the left two columns. Taking downsampling smoothing filtering as an example, the image size changes before and after sampling, as shown in the figure. Taking the coordinates of E as (2x, 0) and the filtered pixel coordinates as (x, 0) as an example, the filtering process is explained as follows: When (x, 0) is not the first or last pixel, the first pixel is the pixel of E (i.e., the pixel corresponding to the coordinate of a preset multiple of the pixel coordinate), and the first neighboring pixels are the pixels of A, B, C, D, and F (i.e., the pixels above the two adjacent pixels corresponding to the preset multiple of the pixel coordinate, the pixels above the preset multiple of the pixel coordinate, and the two adjacent pixels corresponding to the preset multiple of the pixel coordinate). The filtering principle of the left column of pixels is the same as that of the top row. The first pixel is the pixel N, the first neighboring pixels are the two pixels above and below the preset multiple of the pixel coordinates (O and P), the pixels to the left of the two pixels above and below the preset multiple of the pixel coordinates (Q and S), and the pixels to the left of the preset multiple of the pixel coordinates (R). Figure 7 A schematic diagram of the filtered prediction block is provided. Figure 6 The prediction block in the middle is obtained by filtering Figure 7 The prediction block in the image. When the filtered pixel coordinates (x, 0) are the coordinates of the first and last pixels, for example, in the three pixels H', K', and L' in the image, the second pixel corresponding to H' is the pixel of H, and there are two corresponding second neighboring pixels, namely the pixel above and the pixel to the left of H, such as... Figure 6 In G and I, the coordinates of H are the first coordinates corresponding to H'; the second pixel corresponding to K' is the pixel of K, and the second neighboring pixel is the pixel above K, as shown in the example. Figure 6 In the graph, the pixel of J is the first coordinate corresponding to K'; the second pixel corresponding to L' is the pixel of L, and the second neighboring pixel is the pixel to the left of L, i.e., the pixel of M, where the coordinate of L is the first coordinate corresponding to L'.

[0171] For example, Figure 8 Another example image of the current block is provided. The filtering process for the current block is basically the same as that for the prediction block, except that the current block has four pixels at the beginning and end. Each pixel at the beginning and end has one second neighbor pixel, which is either the pixel above or to the left of the pixel. For the first and last pixels in a pixel row, the second neighbor pixel is the pixel above the pixel; for the first and last pixels in a pixel column, the second neighbor pixel is the pixel to the left of the pixel. The filtering process for the current block can be found in the description above.

[0172] In some embodiments, before obtaining the pixels to be processed corresponding to the image block, the method further includes:

[0173] Obtain image filtering indication information;

[0174] The decision to filter the corresponding image block is based on the filtering instruction information.

[0175] The filtering indication information can be understood as information indicating whether or not filtering should be performed. For example, the filtering indication information indicates that the image block should be filtered using the filtering method provided in this application. The filtering indication information can be generated in advance. For example, it can be obtained during the encoding process through one or more methods, such as receiving filtering indication information sent by other nodes, processing the image to determine the filtering indication information, or manually setting the filtering indication information. After obtaining the filtering indication information, it is determined whether to filter the image block in the image based on the information indicated by the filtering indication information. For example, a filtering indication information of 1 indicates that the image block in the image should be filtered, a filtering indication information of 0 indicates that the image block in the image should not be filtered, and so on.

[0176] For example, during the image encoding process at the encoding end, the encoding end processes the image and determines whether to use the filtering method provided in this application embodiment for filtering, and generates filtering indication information, which is then written into the bitstream. When the decoding end receives the bitstream for decoding, it can directly determine whether to use the filtering method provided in this application embodiment for filtering processing based on the parsed filtering indication information.

[0177] In some embodiments, the filtering indication information includes at least one of the following:

[0178] Sequence-level identification information;

[0179] Frame-level identification information.

[0180] The sequence-level identifier information is used to indicate whether the image is filtered in the sequence header. For example, 1 indicates filtering and 0 indicates no filtering. The frame-level identifier information is used to indicate whether the image is filtered in the inter-frame prediction image header. For example, 1 indicates filtering and 0 indicates no filtering.

[0181] In some embodiments, determining whether to filter the image patch corresponding to the image based on the filtering indication information includes:

[0182] If both sequence-level and frame-level identifiers indicate filtering, then the corresponding image block of the image is selected for filtering.

[0183] Analyze the filtering indication information to determine the specific information indicated by the sequence-level and frame-level identifiers. If both the sequence-level and frame-level identifiers indicate filtering, then filtering is applied to the corresponding image block. For example, if both the sequence-level and frame-level identifiers are 1, then filtering is applied to the corresponding image block.

[0184] The image processing method provided in this application obtains the pixels to be processed corresponding to the image block; determines whether the filtered pixel is the first or last pixel coordinate; for non-first or last pixel coordinates, there are at least two neighboring pixels; a weighted operation is performed based on the corresponding first pixel and the first neighboring pixel to obtain the filtered pixel corresponding to the pixel coordinate; for first or last pixel coordinates, there is at least one neighboring pixel; a weighted operation is performed based on the corresponding second pixel and the second neighboring pixel to achieve smooth filtering or downsampling smooth filtering of row and column pixels, reducing or eliminating the influence caused by uneven distribution of pixel values ​​in the upper row and / or left column of the block, improving or eliminating the problem of low image processing accuracy caused by uneven pixel value distribution, and improving the image encoding and decoding efficiency.

[0185] Figure 9 A flowchart of another image processing method provided in one embodiment, such as Figure 9 As shown, the image processing method described in this application embodiment is applied to the encoding end, and the method includes S210-S240:

[0186] S210. Obtain the image to be encoded, which includes data blocks.

[0187] In the image encoding process, an image to be encoded is acquired, which includes data blocks. In this embodiment, the image to be encoded can refer to the original image or an intermediate value obtained during the encoding process at the encoding end. During processing, the image is divided into different data blocks, and each data block can be used as the current encoding unit for encoding. The image to be encoded includes one or more data blocks, and each data block can be processed accordingly using the following methods.

[0188] S220. Perform motion compensation based on the data block to obtain the prediction block.

[0189] Determine motion information, such as MV information, and perform motion compensation on the data block based on the motion information to obtain the prediction block.

[0190] S230. The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method provided in any embodiment of this application.

[0191] The data block and the prediction block are used as image blocks. The image blocks are processed using the method provided in any embodiment of this application, that is, the pixels to be processed corresponding to the image block are obtained, the pixels are filtered according to the correlation between the pixels and the neighboring pixels, and the filtered pixels are determined.

[0192] When filtering an image block, the processing can be performed directly, or the corresponding filtering indication information can be obtained. The filtering indication information is used to determine whether to filter the image block. If it is determined that filtering is required, the method provided in any of the above embodiments is used for filtering. If filtering is not required, no filtering is performed, and the next step is executed directly.

[0193] S240. Based on the processing results, the image is encoded to form the encoded image.

[0194] If the data block and prediction block are treated as image blocks and filtered using the method provided in the embodiments of this application, the processing result is determined based on the filtered pixels. If the data block and prediction block are not treated as image blocks and filtered using the method provided in the embodiments of this application, the processing result is determined based on the pixels of the data block and prediction block before processing. Image encoding is performed based on the processing result, i.e., the pixels before or after filtering. For example, an illumination compensation model is determined based on the filtered pixels, and then image encoding is performed to form an encoded image.

[0195] Each data block can be processed using the steps described above. Finally, based on the processing results of all data blocks and their corresponding prediction blocks, an encoded image is formed.

[0196] In some embodiments, image encoding is performed based on the processing result to form an encoded image, including:

[0197] Inter-frame prediction value correction is performed based on the processing results;

[0198] Based on the correction results, an illumination compensation model is selected, and the index of the illumination compensation model is written into the bitstream to form the encoded image.

[0199] Based on the processing results, inter-frame prediction values ​​are corrected using Rate Distortion Optimization (RDO). A suitable illumination compensation model is then selected based on the correction results; for example, the illumination compensation model with the lowest RDO cost is chosen. The index of the selected illumination compensation model is written into the bitstream to form the encoded image.

[0200] This application provides an image processing method. In the image encoding process, data blocks and prediction blocks are treated as image blocks and processed by the image processing method provided in any of the above embodiments to achieve smooth filtering of data blocks and prediction blocks. By correcting the inter-frame prediction values, the pixels involved in the derivation of illumination compensation model coefficients are smoothed using spatial domain correlation. This improves or eliminates the problem of low accuracy in model coefficient derivation caused by uneven pixel value distribution, thereby improving encoding efficiency.

[0201] Figure 10 A flowchart of another image processing method provided in one embodiment, such as Figure 10 As shown, the image processing method of this application embodiment is applied to the decoding end, and the method includes S310-S340:

[0202] S310. Obtain the image to be decoded, which includes data blocks.

[0203] In the image decoding process, an image to be decoded is acquired, which includes data blocks. In this embodiment, the image to be decoded can refer to the reconstructed image parsed from the bitstream by the decoding end, or it can refer to intermediate values ​​obtained during the decoding process. The image is divided into different data blocks during processing, and each data block can be used as the current decoding unit for decoding. The image to be decoded includes one or more data blocks, and each data block can be processed accordingly using the following methods.

[0204] S320. Perform motion compensation based on the data block to obtain the prediction block.

[0205] Motion information is obtained by parsing the bitstream, and motion compensation is performed on the data blocks based on the motion information to obtain the prediction blocks.

[0206] S330. The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method provided in any embodiment of this application.

[0207] The data block and the prediction block are used as image blocks. The image blocks are processed using the method provided in any embodiment of this application, that is, the pixels to be processed corresponding to the image block are obtained, the pixels are filtered according to the correlation between the pixels and the neighboring pixels, and the filtered pixels are determined.

[0208] When filtering an image block, the processing can be performed directly, or the corresponding filtering indication information can be obtained. The filtering indication information is used to determine whether to filter the image block. If it is determined that filtering is required, the method provided in any of the above embodiments is used for filtering. If filtering is not required, no filtering is performed, and the next step is executed directly.

[0209] S340. Decode the image based on the processing result to form the decoded image.

[0210] If the data block and prediction block are treated as image blocks and filtered using the method provided in the embodiments of this application, the processing result is determined based on the filtered pixels. If the data block and prediction block are not treated as image blocks and are not filtered using the method provided in the embodiments of this application, the processing result is determined based on the pixels of the data block and prediction block before processing. Image decoding is performed based on the processing result, i.e., the pixels before or after filtering. For example, the prediction block is corrected based on the processing result, and the decoded pixels are determined based on the correction result to form the decoded image.

[0211] Each data block can be processed using the steps described above. Finally, based on the processing results of all data blocks and their corresponding prediction blocks, a decoded image is formed.

[0212] In some embodiments, image decoding is performed based on the processing result to form a decoded image, including:

[0213] The index and residual pixels of the illumination compensation model are obtained by parsing the bitstream corresponding to the image to be decoded;

[0214] The predicted blocks are corrected based on the processing results and the index of the illumination compensation model to obtain the corrected predicted pixels;

[0215] The decoded image is determined based on the corrected predicted pixels and residual pixels.

[0216] The bitstream of the image to be decoded is parsed to determine the index of the illumination compensation model and the residual pixels. During image encoding, the encoder writes the index of the illumination compensation model into the bitstream. The illumination compensation model is determined based on the index, and its parameters are determined based on the processing results, thus obtaining the complete illumination compensation model. The prediction blocks are then corrected using the illumination compensation model to obtain the corrected prediction pixels. The corrected prediction pixels are added to the residual pixels to obtain the reconstructed pixels corresponding to the data blocks. Finally, the decoded image is determined based on the reconstructed pixels corresponding to all data blocks.

[0217] This application provides an image processing method. In the image decoding process, data blocks and prediction blocks are treated as image blocks and processed by the image processing method provided in any of the above embodiments to achieve smooth filtering of data blocks and prediction blocks. By correcting the inter-frame prediction values, the pixels involved in the derivation of illumination compensation model coefficients are smoothed using spatial domain correlation. This improves or eliminates the problem of low accuracy in model coefficient derivation caused by uneven pixel value distribution, thereby improving decoding efficiency.

[0218] In the embodiments of this application, both the encoding and decoding processes process data blocks and prediction blocks. For easy distinction, the data blocks can be referred to as the first data block and the second data block, and the prediction blocks can be referred to as the first prediction block and the second prediction block, etc. That is, the data of the same type involved in the encoding and decoding process can be distinguished by the first and second methods.

[0219] The image processing process is illustrated through the following examples:

[0220] The process of smoothing filtering is illustrated using the example of applying smoothing filtering to the current block and the predicted block. Here, the current block refers to the data block in the image currently undergoing smoothing filtering.

[0221] Example 1

[0222] First, use a three-tap filter (1, 2, 1) / 4 to smooth the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the predicted block. Then, use the smoothed pixels to derive the illumination compensation model coefficients. Figure 11a An example diagram of smoothing filtering for a prediction block is provided. Figure 11b An example diagram of smoothing filtering for the current block is provided; such as... Figure 11a and Figure 11b As shown in the figure, the gray pixels are the pixels to be filtered, including the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the prediction block.

[0223] The filtering process is shown in the following formula, P x-1,0 P x,0 P x+1,0 P 0,y-1 P 0,y and P 0,y+1 For pixels before filtering, PF x,0 and PF 0,y The filtered pixels:

[0224] PF x,0 =(P x-1,0 +2*P x,0 +P x+1,0 +2)>>2

[0225] PF 0,y =(P 0,y-1 +2*P 0,y +P 0,y+1 +2)>>2

[0226] Where x = (1, 2, ..., w-1), y = (1, 2, ..., h-1), w and h represent the block width and block height, respectively. When processing a row of pixels, the y-coordinate of this row of pixels is 0, and the x-coordinate ranges from 1 to w-1; when processing a column of pixels, the x-coordinate of this column of pixels is 0, and the y-coordinate ranges from 1 to h-1. From the diagram and the formula, it can be seen that for pixels that are not at the beginning or end, their filtered pixel coordinates do not change. For pixel (x, 0), the filtered pixel coordinates are determined based on the pixel coordinates of this pixel and its neighboring pixels. From the range of x and y values, it can be seen that the first and last pixels do not apply the above filtering process; the first and last pixels apply a two-tap filter (3, 1) / 4, as shown below:

[0227] PF 0,0 =(3*P 0,0 +P 1,0 +2)>>2

[0228] PF w-1,0 =(3*P w-1,0 +P w-2,0 +2)>>2

[0229] PF 0,h-1 =(3*P 0,h-1 +P 0,h-2 +2)>>2

[0230] For a prediction block, there are a total of 3 pixels at the beginning and end. For the pixel in the upper left corner, its filtered pixel can be determined based on the pixel at that point (i.e., P). 0,0 ) and its right-adjacent pixel (i.e., P) 1,0 It can be determined either by the pixel of this point (i.e., P). 0,0 ) and its adjacent pixels below (i.e., P) 0,1 ) Determine. For the last pixel in the bottom row (i.e., the last pixel in the left column), its filtered pixel can be determined based on the pixel at this point (i.e., P). 0,h-1 ) and its adjacent pixels above (i.e., P) 0,h-2 The filter can be determined by the last pixel in the top column (i.e., the last pixel in the top row). w-1,0 ) and its left-adjacent pixel (i.e., P) w-2,0 )Sure.

[0231] For the current block, there are a total of 4 pixels at the beginning and end. For the pixels in the row above, the filtered pixel value of the first pixel is determined by the pixel at this point and its right-hand neighbor, and the filtered pixel value of the last pixel is determined by the pixel at this point and its left-hand neighbor. For the pixels in the column to the left, the filtered pixel value of the first pixel is determined by the pixel at this point and its lower-hand neighbor, and the filtered pixel value of the last pixel is determined by the pixel at this point and its upper-hand neighbor.

[0232] The smoothed and filtered pixels are used in the subsequent derivation of the illumination compensation model coefficients, while the method for selecting pixel pairs remains unchanged.

[0233] To identify whether the smoothing filtering method provided in this application embodiment is used, two levels of syntax flags, sequence-level and frame-level, are set for control. Control flags `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are set in the sequence header and inter-frame prediction image header, respectively, as shown in the table below. For the encoding end, the illumination compensation coefficient derivation method based on pixel smoothing filtering provided in this invention can only take effect when both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1. For the decoding end, `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are first parsed from the bitstream. If both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1, then the same illumination compensation coefficient derivation process based on pixel smoothing filtering as the encoding end is executed, ensuring consistency between encoding and decoding. Additionally, as shown in the table below, ipc_sf_enable_flag only takes effect when inter_pc_enable_flag is 1, i.e., inter-frame prediction correction is enabled.

[0234] Table 2 shows the sequence header with the added IPC_SF_enable_flag control flag.

[0235] Sequence Header Definition descriptor sequence_header(){ video_sequence_start_code f(32) profile_id u(8) level_id u(8) ............... inter_pc_enable_flag u(1) if(InterPcEnableFlag){ ipc_sf_enable_flag u(1) } ...............

[0236] Table 3 shows the inter-frame prediction image headers with the picture_ipc_sf_enable_flag control flag added.

[0237] Inter-frame prediction image head definition descriptor inter_picture_header(){ inter_picture_start_code f(32) random_access_decodable_flag u(1) bbv_delay u(32) picture_coding_type u(2) decode_order_index u(8) ............... if(InterPcEnableFlag&&IpcSfEnableFlag){ u(1) picture_ipc_sf_enable_flag } ...............

[0238] Example 2

[0239] First, a three-tap filter (1, 2, 1) / 4 is used to downsample and smooth the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the predicted block. Then, the smoothed pixels are used to derive the illumination compensation model coefficients. Figure 12a An example image of the prediction block before downsampling is provided. Figure 12b An example image of a prediction block after downsampling is provided; Figure 13a An example image is provided before the current block is downsampled. Figure 13b An example image of the current block after downsampling is provided; Figure 12b The white pixels in the image are obtained through... Figure 12a The black pixels in the image are calculated after smoothing filtering. Figure 13b The white pixels in the image are obtained through... Figure 13a The black pixels in the image are calculated after smoothing filtering. Figure 12a and Figure 13a The gray pixels shown are those that participated in the downsampling smoothing filter, including the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the prediction block.

[0240] The downsampling filtering process is shown in the following formula:

[0241] PF x,0 =(P 2x-1,0 +2*P 2x,0 +P 2x+1,0 +2)>>2

[0242] PF 0,y =(P 0,2y-1 +2*P 0,2y +P 0,2y+1 +2)>>2

[0243] In the above equation, P on the right side of the equation 2x-1,0 P 2x,0 P 2x+1,0 P 0,2y-1 P 0,2y and P 0,3y+1 It comes from the pre-downsampling filter block ( Figure 12a and Figure 13a The pixels of the block shown, PF x,0 PF 0,y These are the pixels after downsampling and filtering. Figure 12b and Figure 13b (As shown by the white pixels in the middle).

[0244] Where x = (1, 2, ..., w-1), y = (1, 2, ..., h-1), w and h represent the block width and block height after downsampling filtering, respectively. From the range of x and y values, it can be seen that the first and last pixels do not apply the above filtering process; instead, the first and last pixels apply a two-tap filter (3, 1) / 4, as shown below:

[0245] PF 0,0 =(P 0,1 +2*P 0,0 +P 1,0 +2)>>2

[0246] PF w-1,0 =(3*P 2w-1,0 +P 2w-2,0 +2)>>2

[0247] PF 0,h-1 =(3*P 0,2h-1 +P 0,2h-2 +2)>>2

[0248] The implementation principle for pixel filtering at the beginning and end of the prediction block and the current block can be referred to in Example 1. For the prediction block, since the pixel in its upper left corner has two adjacent pixels, its filtering can be achieved through the adjacent pixels on the right and below. That is, for pixel (0,0), its filtered pixel can be calculated according to P. 0,1 P 0,0 and P 1,0 Confirmed. For the other two beginning and ending pixels in the prediction block, and the four beginning and ending pixels in the current block, their filtered pixels can be calculated from the unfiltered beginning and ending pixels and the pixel of the one adjacent pixel.

[0249] The pixels after downsampling and smoothing filtering are used in the subsequent derivation of illumination compensation model coefficients, while the method for selecting pixel pairs remains unchanged.

[0250] To identify whether the smoothing filtering method provided in this application embodiment is used, two levels of syntax flags, sequence-level and frame-level, are set for control. Control flags `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are set in the sequence header and inter-frame prediction image header, respectively, as shown in the table below. For the encoding end, the illumination compensation coefficient derivation method based on pixel smoothing filtering provided in this invention can only take effect when both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1. For the decoding end, `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are first parsed from the bitstream. If both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1, then the same illumination compensation coefficient derivation process based on pixel smoothing filtering as the encoding end is executed, ensuring consistency between encoding and decoding. Additionally, as shown in the table below, `ipc_sf_enable_flag` only takes effect when `inter_pc_enable_flag` is 1, i.e., inter-frame prediction correction is enabled. For the specific syntax structure, please refer to Tables 2 and 3.

[0251] Example 3

[0252] Considering the reconstructed pixels in the row above and / or column to the left of the current block, and the influence of the first and last pixels in the predicted pixels in the row above and / or column to the left of the predicted block, a five-tap filter (1, 3, 8, 3, 1) / 16 is first used to smooth the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the predicted block. Then, the smoothed pixels are used to derive the illumination compensation model coefficients. Figure 14a An example graph of a prediction block is provided. Figure 14b An example image of the current block is provided. The five pixels connected by the arrows at both ends are the extracted pixels. Pixels A, B, and C in the image represent the beginning and end pixels. A schematic diagram of smoothing filtering is shown below. Figure 14a and Figure 14b As shown in the figure, the gray pixels are the pixels to be filtered, including the reconstructed pixels above and / or to the left of the current block, and the predicted pixels in the row above and / or column to the left of the prediction block.

[0253] The filtering process is shown in the following formula, PH 0,0 P x-1,0 P x,0 P x+1,0 P w-1,0 PV 0,0 P 0,y-1 P 0,y P0,y+1 and P 0,h-1 For pixels before filtering, PF x,0 and PF 0,y The filtered pixels:

[0254] PF x,0 =(PH) 0,0 +3*P x-1,0 +8*P x,0 +3*P x+1,0 +P w-1,0 +8)>>4

[0255] PF 0,y =(PV) 0,0 +3*P 0,y-1 +8*P 0,y +3*P 0,y+1 +P 0,h-1 +8)>>4

[0256] Where x = (1, 2, ..., w-1), y = (1, 2, ..., h-1), and w and h represent the block width and block height, respectively. From the range of x and y values, it can be seen that the first and last pixels do not apply the above filtering process; instead, the first and last pixels apply a two-tap filter (3, 1) / 4, as shown below:

[0257] PF 0,0 =(3*P 0,0 +P 1,0 +2)>>2

[0258] PF w-1,0 =(3*P w-1,0 +P w-2,0 +2)>>2

[0259] PF 0,h-1 =(3*P 0,h-1 +P 0,h-2 +2)>>2

[0260] The pixel values ​​at the beginning and end can be calculated using the same method as those in the above embodiment.

[0261] The smoothed and filtered pixels are used in the subsequent derivation of the illumination compensation model coefficients, while the method for selecting pixel pairs remains unchanged.

[0262] To identify whether the smoothing filtering method provided in this application embodiment is used, two levels of syntax flags, sequence-level and frame-level, are set for control. Control flags `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are set in the sequence header and inter-frame prediction image header, respectively, as shown in the table below. For the encoding end, the illumination compensation coefficient derivation method based on pixel smoothing filtering provided in this invention can only take effect when both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1. For the decoding end, `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are first parsed from the bitstream. If both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1, then the same illumination compensation coefficient derivation process based on pixel smoothing filtering as the encoding end is executed, ensuring consistency between encoding and decoding. Additionally, as shown in the table below, `ipc_sf_enable_flag` only takes effect when `inter_pc_enable_flag` is 1, i.e., inter-frame prediction correction is enabled. For the specific syntax structure, please refer to Tables 2 and 3.

[0263] Example 4

[0264] Considering the reconstructed pixels in the row above and / or column to the left of the current block, and the influence of the first and last pixels in the predicted pixels in the row above and / or column to the left of the predicted block, a five-tap filter (1, 3, 8, 3, 1) / 16 is first used to downsample and smooth the reconstructed pixels in the row above and / or column to the left of the current block, and the predicted pixels in the row above and / or column to the left of the predicted block. Then, the downsampled and smoothed pixels are used to derive the illumination compensation model coefficients. Figure 15a An example graph of a prediction block is provided. Figure 15b An example image of the current block is provided, where pixels A, B, and C are the beginning and end pixels. Figure 16a An example image is provided before prediction block downsampling. Figure 16b An example image of a predicted block after downsampling is provided; Figure 17a A sample image is provided before downsampling of the current block, and 17b provides a sample image after downsampling of the current block. Figure 16b The white pixels in the image are obtained through... Figure 16a The black pixels and the first and last pixels in 17b are calculated after downsampling and smoothing filtering. The white pixels in 17b are obtained through... Figure 17aThe black pixels and the first and last pixels in the figure are calculated after downsampling and smoothing filtering. The gray pixels shown in the figure are the pixels that participated in downsampling and smoothing filtering, including the reconstructed pixels above and / or to the left of the current block, and the predicted pixels in the row above and / or column to the left of the prediction block.

[0265] The downsampling filtering process is shown in the following formula:

[0266] PF x,0 =(PH) 0,0 +3*P 2x-1,0 +8*P 2x,0 +3*P 2x+1,0 +P 2w-1,0 +8)>>4

[0267] PF 0,y =(PV) 0,0 +3*P 0,2y-1 +8*P 0,2y +3*P 0,2y+1 +P 0,2h-1 +8)>>4

[0268] In the above equation, PH on the right side of the equation 0,0 P 2x-1,0 P 2x,0 P 2x+1,0 P 2w-1,0 PV 0,0 P 0,2y-1 P 0,2y P 0,2y+1 and P 0,2h-1 It comes from the pre-downsampling filter block ( Figure 16a or Figure 17a The pixels of the block shown, PF x,0 PF 0,y These are the pixels after downsampling and filtering. Figure 16b Or as shown by the white pixel in 17b).

[0269] Where x = (1, 2, ..., w-1), y = (1, 2, ..., h-1), w and h represent the block width and block height after downsampling filtering, respectively. From the range of x and y values, it can be seen that the first and last pixels do not apply the above filtering process; instead, the first and last pixels apply a two-tap filter (3, 1) / 4, as shown below:

[0270] PF 0,0 =(P 0,1 +2*P 0,0 +P 1,0 +2)>>2

[0271] PF w-1,0 =(3*P 2w-1,0 +P2w-2,0 +2)>>2

[0272] PF 0,h-1 =(3*P 0,2h-1 +P 0,2h-2 +2)>>2

[0273] The pixel values ​​at the beginning and end can be calculated using the same method as those in the above embodiment.

[0274] The pixels after downsampling and smoothing filtering are used in the subsequent derivation of illumination compensation model coefficients, while the method for selecting pixel pairs remains unchanged.

[0275] To identify whether the smoothing filtering method provided in this application embodiment is used, two levels of syntax flags, sequence-level and frame-level, are set for control. Control flags `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are set in the sequence header and inter-frame prediction image header, respectively, as shown in the table below. For the encoding end, the illumination compensation coefficient derivation method based on pixel smoothing filtering provided in this invention can only take effect when both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1. For the decoding end, `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are first parsed from the bitstream. If both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1, then the same illumination compensation coefficient derivation process based on pixel smoothing filtering as the encoding end is executed, ensuring consistency between encoding and decoding. Additionally, as shown in the table below, `ipc_sf_enable_flag` only takes effect when `inter_pc_enable_flag` is 1, i.e., inter-frame prediction correction is enabled. For the specific syntax structure, please refer to Tables 2 and 3.

[0276] Example 5

[0277] First, a six-tap filter (1, 2, 1, 1, 2, 1) / 8 is used to downsample and smooth the reconstructed pixels in the two rows above and / or the two columns to the left of the current block, and the predicted pixels in the two rows above and / or the two columns to the left of the predicted block. Then, the smoothed pixels are used to derive the illumination compensation model coefficients. Figure 18a An example image is provided before prediction block downsampling. Figure 18b An example image of a predicted block after downsampling is provided; Figure 19a A sample image is provided before downsampling of the current block, and 19b provides a sample image after downsampling of the current block. A schematic diagram of downsampling smoothing filtering is shown in the figure. Figure 18b The white pixels in the image are obtained through... Figure 18aThe black pixels in the image are calculated after smoothing filtering. Figure 19b The white pixels in the image are obtained through... Figure 19a The black pixels in the image are calculated after smoothing filtering. The gray pixels shown in the image are the pixels that participated in the downsampling smoothing filtering, including the reconstructed pixels in the two rows above and / or the two columns to the left of the current block, and the predicted pixels in the two rows above and / or the two columns to the left of the prediction block.

[0278] The filtering process is shown in the following formula, P 2x-1,0 P 2x,0 P 2x+1,0 P 2x-1-stride,0 P 2x-stride,0 P 2x+1-stride,0 P -1,2y-1 P -1,2y P -1,2y+1 P 0,2y-1 P 0,2y P 0,2y+1 For pixels before filtering, PF x,0 and PF 0,y The filtered pixels:

[0279] PF x,0 =(P 2x-1,0 +2*P 2x,0 +P 2x+1,0 +P 2x-1-stride,0 +2*P 2x-stride,0 +P 2x+1-stride,0 +4)>>3

[0280] PF 0,y =(P -1,2y-1 +2*P -1,2y +P -1,2y+1 +P 0,2y-1 +2*P 0,2y +P 0,2y+1 +4)>>3

[0281] Where x = (1, 2, ..., w-1), y = (1, 2, ..., h-1), w and h represent the filtered block width and height, respectively, and stride is the span before filtering. For example, Figure 18a The span between A and C is 2, and the span between A and D is 8. From the range of x and y values, we know that the first and last pixels do not apply the filtering process described above; instead, the first and last pixels apply a two-tap filter (1, 1) / 2, as shown below:

[0282] PF 0,0 =(P -1,0 +2P 0,0 +P 0,-stride +2)>>2

[0283] PFw-1,0 =(P 2w-1,0 +P 2w-1,-stride +1)>>1

[0284] PF 0,h-1 =(P 0,2h-1 +P -1,2h-1 +1)>>1

[0285] like Figure 18a and Figure 18b As shown, for pixel rows that are not at the beginning or end, taking pixel point E with coordinates (2x, 0) as an example, the filtered pixel coordinates (x, 0) are determined based on the pixels A(2x-1-stride, 0), B(2x-stride, 0), C(2x+1-stride, 0), D(2x-1, 0), E(2x, 0), and F(2x+1, 0). Figure 18b The white pixels in the middle pixel row. For pixel columns that are not at the beginning or end, taking pixel N with coordinates (0, 2y) as an example, the filtered pixel coordinates (0, y) are determined based on the pixels at N(0, 2y), O(0, 2y-1), P(0, 2y+1), Q(-1, 2y-1), R(-1, 2y), and S(-1, 2y+1). Figure 18b The white pixels in the middle pixel column.

[0286] like Figure 18a and 18b As shown, there are three pixels at the beginning and end of the filtered graph. The pixels at the beginning and end of the filtered graph are H', K' and L'. The pixel corresponding to H'(0,0) is determined by I, G and H; the pixel corresponding to K'(w-1,0) is determined by K and J; and the pixel corresponding to L'(0,h-1) is determined by L and M.

[0287] like Figure 19a and 19b As shown, there are four pixels at the beginning and end after filtering. The pixels at the beginning and end after filtering are H', K', L', and T'. The pixel corresponding to H' is determined by G and H; the pixel corresponding to K' is determined by K and J; the pixel corresponding to T' is determined by T and I; and the pixel corresponding to L' is determined by L and M.

[0288] The smoothed and filtered pixels are used in the subsequent derivation of the illumination compensation model coefficients, while the method for selecting pixel pairs remains unchanged.

[0289] To identify whether the smoothing filtering method provided in this application embodiment is used, two levels of syntax flags, sequence-level and frame-level, are set for control. Control flags `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are set in the sequence header and inter-frame prediction image header, respectively, as shown in the table below. For the encoding end, the illumination compensation coefficient derivation method based on pixel smoothing filtering provided in this invention can only take effect when both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1. For the decoding end, `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are first parsed from the bitstream. If both `ipc_sf_enable_flag` and `picture_ipc_sf_enable_flag` are 1, then the same illumination compensation coefficient derivation process based on pixel smoothing filtering as the encoding end is executed, ensuring consistency between encoding and decoding. Additionally, as shown in the table below, `ipc_sf_enable_flag` only takes effect when `inter_pc_enable_flag` is 1, i.e., inter-frame prediction correction is enabled. For the specific syntax structure, please refer to Tables 2 and 3.

[0290] Figure 20 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment, such as... Figure 20 As shown, the device includes a pixel acquisition module 410 and a filtering module 420.

[0291] The pixel acquisition module 410 is used to acquire the pixels to be processed corresponding to the image block;

[0292] The filtering module 420 is used to filter the pixel based on the correlation between the pixel and its neighboring pixels to determine the filtered pixel.

[0293] The image processing apparatus provided in this application obtains the pixels to be processed corresponding to an image block; filters the pixels based on the correlation between the pixels and neighboring pixels to determine the filtered pixels; and filters the pixels to be processed corresponding to the image block based on the correlation between the pixels and neighboring pixels, thereby improving or eliminating the problem of low image processing accuracy caused by uneven pixel value distribution and improving image encoding and decoding efficiency.

[0294] Image blocks include at least one of the following:

[0295] Data blocks, which are obtained by dividing the image;

[0296] The prediction block is obtained by predicting data blocks obtained by dividing the image.

[0297] In some embodiments, the pixel to be processed includes at least one of the following:

[0298] Predict the pixels in the top row of the block;

[0299] Predict the pixels in the leftmost column of the block;

[0300] The pixels in the row above the predicted block;

[0301] The pixels in the left column of the predicted block;

[0302] The number of pixels in the row above the data block;

[0303] The pixels in the left column of the data block;

[0304] The pixels in the top two rows of the data block;

[0305] The pixels in the two columns to the left of the data block.

[0306] In some embodiments, filtering the pixel based on its correlation with neighboring pixels to determine the filtered pixel includes:

[0307] If the filtered pixel coordinates are not the first or last pixel coordinates, determine the first pixel and at least two first neighboring pixels corresponding to the pixel coordinates, and perform a weighted operation based on the first pixel and the first neighboring pixels to obtain the filtered pixel corresponding to the pixel coordinates.

[0308] When the filtered pixel coordinates are the first and last pixel coordinates, the second pixel and at least one second neighboring pixel corresponding to the pixel coordinates are determined. A weighted operation is performed on the second pixel and the second neighboring pixel to obtain the filtered pixel corresponding to the pixel coordinates.

[0309] In some embodiments, the first pixel includes at least one of the following:

[0310] The pixel corresponding to the pixel coordinates;

[0311] The pixel coordinates corresponding to a preset multiple of the pixel coordinates, wherein the preset multiple is determined based on the sampling multiple.

[0312] In some embodiments, the first neighboring pixel includes at least one of the following:

[0313] The two pixels whose pixel coordinates are adjacent to each other on the left and right;

[0314] The two pixels whose pixel coordinates are adjacent vertically;

[0315] The two pixels to the left and right of the coordinates corresponding to the preset multiple of the pixel coordinates;

[0316] The two pixels directly above and below each other whose coordinates correspond to a preset multiple of the pixel coordinates;

[0317] The first and last two pixels of the row containing the pixel coordinates;

[0318] The first and last two pixels of the column containing the pixel coordinates;

[0319] The pixels above the two adjacent pixels on the left and right sides of the preset multiple of the pixel coordinates, and the pixels above the coordinates corresponding to the preset multiple of the pixel coordinates;

[0320] The pixel to the left of the two adjacent pixels above and below the preset multiple of the pixel coordinates, and the pixel to the left of the preset multiple of the pixel coordinates.

[0321] The preset multiple is determined based on the sampling multiple.

[0322] In some embodiments, the second pixel includes at least one of the following:

[0323] The pixel corresponding to the pixel coordinates;

[0324] The first coordinate corresponds to the pixel, and the first coordinate is determined based on the pixel coordinate and the sampling factor.

[0325] In some embodiments, the second neighboring pixel includes at least one of the following:

[0326] The pixel adjacent to the left of the pixel coordinate;

[0327] The pixel adjacent to the right of the pixel coordinate;

[0328] The pixel adjacent to the pixel coordinate above;

[0329] The pixel adjacent to the pixel coordinate below;

[0330] The pixel to the left of the first coordinate;

[0331] The pixel to the right of the first coordinate;

[0332] The pixel adjacent to the first coordinate above;

[0333] The pixel immediately below the first coordinate;

[0334] The first coordinate is determined based on the pixel coordinate and the sampling factor.

[0335] In some embodiments, the weighted operation based on the first pixel and the first neighboring pixels includes:

[0336] The first pixel and the first neighboring pixel are weighted according to the corresponding weighting coefficients, and the weighted result is rounded down.

[0337] In some embodiments, the weighted operation based on the second pixel and the second neighboring pixels includes:

[0338] The second pixel and the second neighboring pixel are weighted according to the corresponding weighting coefficients, and the weighted result is rounded down.

[0339] In some embodiments, before obtaining the pixels to be processed corresponding to the image block, the method further includes:

[0340] Obtain image filtering indication information;

[0341] Whether to filter the image block corresponding to the image is determined based on the filtering indication information.

[0342] In some embodiments, the filtering indication information includes at least one of the following:

[0343] Sequence-level identification information;

[0344] Frame-level identification information.

[0345] In some embodiments, determining whether to filter the image patch corresponding to the image based on the filtering indication information includes:

[0346] If both the sequence-level identifier information and the frame-level identifier information indicate filtering, then it is determined that the image block corresponding to the image should be filtered.

[0347] The image processing apparatus proposed in this embodiment belongs to the same inventive concept as the image processing method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in any of the above embodiments. Furthermore, this embodiment has the same beneficial effects as performing the image processing method.

[0348] Figure 21 This is a schematic diagram of another image processing apparatus provided in one embodiment, which is applied to the encoding end, such as... Figure 5 As shown, the device includes: a first image acquisition module 510, a first motion compensation module 520, a first processing module 530, and an encoding module 540.

[0349] The first image acquisition module 510 is used to acquire an image to be encoded, wherein the image to be encoded includes data blocks;

[0350] The first motion compensation module 520 is used to perform motion compensation based on the data block to obtain a prediction block;

[0351] The first processing module 530 is used to treat the data block and the prediction block as image blocks, and to perform image processing on the image blocks using the image processing method provided in any embodiment of this application.

[0352] The encoding module 540 is used to encode the image based on the processing results to form an encoded image.

[0353] This application provides an image processing apparatus that, during image encoding, treats data blocks and prediction blocks as image blocks and processes them using the image processing methods provided in any of the above embodiments. This achieves smooth filtering of data blocks and prediction blocks, corrects inter-frame prediction values, and uses spatial correlation to perform smooth filtering on pixels involved in the derivation of illumination compensation model coefficients. This improves or eliminates the problem of low accuracy in model coefficient derivation caused by uneven pixel value distribution, thereby improving encoding efficiency.

[0354] In some embodiments, the step of encoding the image based on the processing result to form an encoded image includes:

[0355] Inter-frame prediction value correction is performed based on the processing results;

[0356] Based on the correction results, an illumination compensation model is selected, and the index of the illumination compensation model is written into the bitstream to form the encoded image.

[0357] The image processing apparatus proposed in this embodiment belongs to the same inventive concept as the image processing method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in any of the above embodiments. Furthermore, this embodiment has the same beneficial effects as performing the image processing method.

[0358] Figure 22 This is a schematic diagram of another image processing apparatus provided in one embodiment, which is applied to the decoding end, such as... Figure 5 As shown, the device includes: a second image acquisition module 610, a second motion compensation module 620, a second processing module 630, and a decoding module 640.

[0359] The second image acquisition module 610 is used to acquire an image to be decoded, the image to be decoded including data blocks;

[0360] The second motion compensation module 620 is used to perform motion compensation based on the data block to obtain a prediction block;

[0361] The second processing module 630 is used to treat the data block and the prediction block as image blocks, and to perform image processing on the image blocks using the image processing method provided in any embodiment of this application.

[0362] The decoding module 640 is used to decode the image based on the processing result to form a decoded image.

[0363] This application provides an image processing apparatus that, during image decoding, treats data blocks and prediction blocks as image blocks and processes them using the image processing methods provided in any of the above embodiments. This achieves smooth filtering of data blocks and prediction blocks, corrects inter-frame prediction values, and uses spatial correlation to perform smooth filtering on pixels involved in the derivation of illumination compensation model coefficients. This improves or eliminates the problem of low accuracy in model coefficient derivation caused by uneven pixel value distribution, thereby improving decoding efficiency.

[0364] In some embodiments, the step of decoding the image based on the processing result to form a decoded image includes:

[0365] The index and residual pixels of the illumination compensation model are obtained by parsing the bitstream corresponding to the image to be decoded;

[0366] The predicted block is corrected based on the processing results and the index of the illumination compensation model to obtain the corrected predicted pixels;

[0367] The decoded image is determined based on the corrected predicted pixels and residual pixels.

[0368] The image processing apparatus proposed in this embodiment belongs to the same inventive concept as the image processing method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in any of the above embodiments. Furthermore, this embodiment has the same beneficial effects as performing the image processing method.

[0369] This application also provides an electronic device. Figure 23 A schematic diagram of the structure of an electronic device is provided as an embodiment, such as... Figure 23 As shown, the electronic device provided in this application includes a processor 710, a memory 720, and a computer program stored in the memory and executable on the processor. When the processor 710 executes the program, it implements the image processing method described above.

[0370] The electronic device may also include a memory 720; the processor 710 in the electronic device may be one or more. Figure 23 Taking a processor 710 as an example; a memory 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement the information transmission method as described in the embodiments of this application.

[0371] The electronic device also includes: a communication device 730, an input device 740, and an output device 750.

[0372] The processor 710, memory 720, communication device 730, input device 740, and output device 750 in the electronic device can be connected via a bus or other means. Figure 23 Taking the example of a connection between China and Israel via a bus.

[0373] Input device 740 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 750 may include display devices such as a display screen.

[0374] The communication device 730 may include a receiver and a transmitter. The communication device 730 is configured to perform information transmission and reception communication under the control of the processor 710.

[0375] The memory 720, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the image processing method described in the embodiments of this application (e.g., pixel acquisition module 410 and filtering module 420 in an image processing device, or first image acquisition module 510, first motion compensation module 520, first processing module 530 and encoding module 540 in an image processing device, or second image acquisition module 610, second motion compensation module 620, second processing module 630 and decoding module 640 in an image processing device). The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include memory remotely located relative to the processor 710, and these remote memories can be connected to the electronic device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0376] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements any of the image processing methods described in this application.

[0377] Optionally, the image processing method includes: acquiring the pixel to be processed corresponding to the image block; filtering the pixel based on the correlation between the pixel and its neighboring pixels to determine the filtered pixel.

[0378] Optionally, the image processing method is applied at the encoding end and includes: acquiring an image to be encoded, the image to be encoded including data blocks; performing motion compensation based on the data blocks to obtain prediction blocks; using the data blocks and the prediction blocks as image blocks, performing image processing on the image blocks using the image processing method provided in any embodiment of this application; and performing image encoding based on the processing result to form an encoded image.

[0379] Optionally, the image processing method is applied to the decoding end and includes: acquiring an image to be decoded, the image to be decoded including data blocks; performing motion compensation based on the data blocks to obtain prediction blocks; using the data blocks and the prediction blocks as image blocks, performing image processing on the image blocks using the image processing method provided in any embodiment of this application; and performing image decoding based on the processing result to form a decoded image.

[0380] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0381] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0382] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0383] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method described in any one of the embodiments of this application.

[0384] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0385] The above description is merely an exemplary embodiment of this application and is not intended to limit the scope of protection of this application.

[0386] Those skilled in the art will understand that the term user terminal encompasses any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, portable web browsers, or vehicle-mounted mobile stations.

[0387] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although this application is not limited thereto.

[0388] Embodiments of this application can be implemented by executing computer program instructions through the data processor of a mobile device, for example, in a processor entity, or through hardware, or through a combination of software and hardware. The computer program instructions can be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages.

[0389] Any block diagram of logical flow in the accompanying drawings of this application may represent program steps, or may represent interconnected logic circuits, modules, and functions, or may represent a combination of program steps and logic circuits, modules, and functions. The computer program may be stored on memory. Memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as, but not limited to, read-only memory (ROM), random access memory (RAM), optical storage devices and systems (Digital Video Disc (DVD) or Compact Disk (CD), etc.). Computer-readable media may include non-transitory storage media. Data processors may be of any type suitable to the local technical environment, such as, but not limited to, general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and processors based on multi-core processor architectures.

[0390] A detailed description of exemplary embodiments of this application has been provided above through exemplary and non-limiting examples. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, without departing from the scope of this application. Therefore, the proper scope of this application will be determined by the claims.

Claims

1. An image processing method, characterized in that, include: Obtain the pixels to be processed corresponding to the image patch; The pixel is filtered based on its correlation with neighboring pixels to determine the filtered pixel.

2. The image processing method according to claim 1, characterized in that, The image block includes at least one of the following: Data blocks, which are obtained by dividing the image; The prediction block is obtained by predicting data blocks obtained by dividing the image.

3. The image processing method according to claim 2, characterized in that, The pixels to be processed include at least one of the following: Predict the pixels in the top row of the block; Predict the pixels in the leftmost column of the block; The pixels in the row above the predicted block; The pixels in the left column of the predicted block; The number of pixels in the row above the data block; The pixels in the left column of the data block; The pixels in the top two rows of the data block; The pixels in the two columns to the left of the data block.

4. The image processing method according to claim 1, characterized in that, The step of filtering the pixel based on its correlation with neighboring pixels to determine the filtered pixel includes: If the filtered pixel coordinates are not the first or last pixel coordinates, determine the first pixel and at least two first neighboring pixels corresponding to the pixel coordinates, and perform a weighted operation based on the first pixel and the first neighboring pixels to obtain the filtered pixel corresponding to the pixel coordinates. When the filtered pixel coordinates are the first and last pixel coordinates, the second pixel and at least one second neighboring pixel corresponding to the pixel coordinates are determined. A weighted operation is performed on the second pixel and the second neighboring pixel to obtain the filtered pixel corresponding to the pixel coordinates.

5. The image processing method according to claim 4, characterized in that, The first pixel includes at least one of the following: The pixel corresponding to the pixel coordinates; The pixel coordinates corresponding to a preset multiple of the pixel coordinates, wherein the preset multiple is determined based on the sampling multiple.

6. The image processing method according to claim 4, characterized in that, The first neighboring pixel includes at least one of the following: The two pixels whose pixel coordinates are adjacent to each other on the left and right; The two pixels whose pixel coordinates are adjacent vertically; The two pixels to the left and right of the coordinates corresponding to the preset multiple of the pixel coordinates; The two pixels directly above and below each other whose coordinates correspond to a preset multiple of the pixel coordinates; The first and last two pixels of the row containing the pixel coordinates; The first and last two pixels of the column containing the pixel coordinates; The pixels above the two adjacent pixels on the left and right sides of the preset multiple of the pixel coordinates, and the pixels above the coordinates corresponding to the preset multiple of the pixel coordinates; The pixel to the left of the two adjacent pixels above and below the preset multiple of the pixel coordinates, and the pixel to the left of the preset multiple of the pixel coordinates. The preset multiple is determined based on the sampling multiple.

7. The image processing method according to claim 4, characterized in that, The second pixel includes at least one of the following: The pixel corresponding to the pixel coordinates; The first coordinate corresponds to the pixel, and the first coordinate is determined based on the pixel coordinate and the sampling factor.

8. The image processing method according to claim 4, characterized in that, The second neighboring pixel includes at least one of the following: The pixel adjacent to the left of the pixel coordinate; The pixel adjacent to the right of the pixel coordinate; The pixel adjacent to the pixel coordinate above; The pixel adjacent to the pixel coordinate below; The pixel to the left of the first coordinate; The pixel to the right of the first coordinate; The pixel adjacent to the first coordinate above; The pixel immediately below the first coordinate; The first coordinate is determined based on the pixel coordinate and the sampling factor.

9. The image processing method according to claim 4, characterized in that, The weighted calculation based on the first pixel and the first neighboring pixels includes: The first pixel and the first neighboring pixel are weighted according to the corresponding weighting coefficients, and the weighted result is rounded down. The weighted calculation based on the second pixel and the second neighboring pixels includes: The second pixel and the second neighboring pixel are weighted according to the corresponding weighting coefficients, and the weighted result is rounded down.

10. The image processing method according to any one of claims 1-9, characterized in that, Before obtaining the pixels to be processed corresponding to the image patch, the following steps are also included: Obtain image filtering indication information; Whether to filter the image block corresponding to the image is determined based on the filtering indication information.

11. The image processing method according to claim 10, characterized in that, The filtering indication information includes at least one of the following: Sequence-level identification information; Frame-level identification information.

12. The image processing method according to claim 11, characterized in that, The step of determining whether to filter the image block corresponding to the image based on the filtering indication information includes: If both the sequence-level identifier information and the frame-level identifier information indicate filtering, then it is determined that the image block corresponding to the image should be filtered.

13. An image processing method, characterized in that, Applied to the encoding end, including: Acquire an image to be encoded, the image to be encoded comprising data blocks; Motion compensation is performed on the data blocks to obtain the prediction blocks; The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method according to any one of claims 1-12. The image is encoded based on the processing results to form the encoded image.

14. The image processing method according to claim 13, characterized in that, The step of encoding the image based on the processing result to form the encoded image includes: Inter-frame prediction value correction is performed based on the processing results; Based on the correction results, an illumination compensation model is selected, and the index of the illumination compensation model is written into the bitstream to form the encoded image.

15. An image processing method, characterized in that, Applied to the decoding end, including: Acquire an image to be decoded, the image to be decoded comprising data blocks; Motion compensation is performed on the data blocks to obtain the prediction blocks; The data block and the prediction block are used as image blocks, and the image blocks are processed using the image processing method according to any one of claims 1-12. The image is decoded based on the processing results to form the decoded image.

16. The image processing method according to claim 15, characterized in that, The step of decoding the image based on the processing result to form a decoded image includes: The index and residual pixels of the illumination compensation model are obtained by parsing the bitstream corresponding to the image to be decoded; The predicted block is corrected based on the processing results and the index of the illumination compensation model to obtain the corrected predicted pixels; The decoded image is determined based on the corrected predicted pixels and residual pixels.

17. An electronic device, characterized in that, include: The image processing method as described in any one of claims 1-16 includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When executed by the processor, the program implements the steps of the image processing method as described in any one of claims 1-16.

18. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the image processing method according to any one of claims 1-16.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-16.

Images