Image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a storage medium, relates to the field of image processing, and solves the problem that after an image is decoded and filtered, the image quality of different permission areas in the image is poor. The image processing method is executed by the electronic equipment and comprises the following steps: dividing a to-be-processed image into a plurality of image sub-regions, and obtaining an authority level of each image sub-region; and if the permission levels of at least two adjacent image sub-regions in the plurality of image sub-regions are different, filtering the connected boundary between the at least two image sub-regions.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] In video encoding and decoding, the purpose of filtering is to achieve smooth noise reduction and detail removal within video image blocks, while preserving image edges to the greatest extent possible. Typically, electronic devices filter the boundaries of multiple regions in an image to preserve image edges. However, different users have different viewing permissions for different regions of an image. If a low-privilege user does not have permission to view a high-privilege region in the image, the electronic device will not filter the boundary between the high-privilege region and the low-privilege region during the video decoding process, resulting in poor image quality. Summary of the Invention

[0003] The present application provides an image processing method, device, electronic device and storage medium, which solve the problem of poor image quality in different permission areas of an image after decoding and filtering.

[0004] This application adopts the following technical solution.

[0005] In a first aspect, the present application provides an image processing method, which is executed by an electronic device, including: dividing the image to be processed into multiple image sub-regions, and obtaining the authority level of each image sub-region; if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering the connecting boundaries between the at least two image sub-regions.

[0006] In this embodiment, the electronic device divides the image to be processed into multiple image subregions, obtains the permission level of each image subregion, and then determines the permission level of each image subregion. If the permission levels of at least two adjacent image subregions within the multiple image subregions are different, this indicates that the two image subregions may still have not been filtered after the filtering process during the decoding process, resulting in low quality of the processed image. Therefore, based on the decoded information, such as the division of the permission regions, the electronic device further filters the boundaries between at least two image subregions, thereby improving the image quality of the different permission regions in the processed image.

[0007] In some embodiments, the permission levels include a first permission level, a second permission level, and a zero permission level; an image sub-region of the zero permission level is an image sub-region that can be viewed by any user.

[0008] In some embodiments, before dividing the image to be processed into multiple image sub-regions and obtaining the permission level of each image sub-region, the process includes: receiving a decoded code stream of the image to be processed, decoding based on the decoded code stream, and obtaining the decoded image to be processed.

[0009] In some embodiments, the filtering process includes at least one of: deblocking effect vertical filtering, deblocking effect horizontal filtering, sample adaptive offset filtering, adaptive loop filtering and neural network filtering.

[0010] In some embodiments, there is a vertical boundary between at least two image sub-regions, and the filtering process includes vertical filtering for deblocking effects. The filtering process for the boundary between at least two image sub-regions includes: determining a target area to be filtered on the vertical side of a first sub-region among the at least two image sub-regions; the vertical side is the left or right side of the boundary between the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the right side of the boundary, the area to be filtered on the left side of the boundary, the area to be filtered on the right side of the boundary, and the area to be filtered on the left side of the boundary; and performing vertical filtering for deblocking effects on the target area to be filtered.

[0011] In some embodiments, the at least two adjacent image subregions further include a second subregion, which is horizontally adjacent to the first subregion. Horizontal adjacency herein means that the second subregion is located to the left or right of the first subregion in the horizontal direction. The at least two adjacent image subregions may have different permission levels, including: the permission level of the first subregion is lower than the permission level of the second subregion; or the permission level of the first subregion is higher than the permission level of the second subregion; or the permission level of the first subregion is not equal to the permission level of the second subregion; or the permission level of the first subregion is not equal to the permission level of the second subregion, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.

[0012] In some embodiments, there is a horizontal boundary between at least two image sub-regions, and the filtering process includes horizontal filtering for deblocking effects. The filtering process for the boundary between at least two image sub-regions includes: determining a target area to be filtered on the horizontal side of a first sub-region among the at least two image sub-regions; the horizontal side is the upper side or the lower side of the boundary between the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the lower side of the boundary, the area to be filtered on the upper side of the boundary, the area to be filtered on the lower side of the boundary, and the area to be filtered on the upper side of the boundary; and performing horizontal filtering for deblocking effects on the target area to be filtered.

[0013] In some embodiments, the at least two adjacent image subregions further include a third subregion, which is vertically adjacent to the first subregion. Vertically adjacent here means that the third subregion is located horizontally above or below the first subregion. The at least two adjacent image subregions have different permission levels, including: the permission level of the first subregion is lower than the permission level of the third subregion; or the permission level of the first subregion is higher than the permission level of the third subregion; or the permission level of the first subregion is not equal to the permission level of the third subregion; or the permission level of the first subregion is not equal to the permission level of the third subregion, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.

[0014] In some embodiments, the filtering process includes sample adaptive compensation filtering, and the above-mentioned filtering process for the connecting boundary between at least two image sub-regions includes: determining the target area to be filtered from the surrounding side of the first sub-region in the at least two image sub-regions according to the set filtering order; the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the connecting boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area in the first sub-region compared to the connecting boundary, the second side area outside the first sub-region compared to the connecting boundary, the first side area and the second side area; according to the set filtering order, the target area to be filtered is subjected to sample adaptive compensation filtering.

[0015] In some embodiments, the filtering process includes adaptive loop filtering, and the above-mentioned filtering process for the connecting boundary between at least two image sub-regions includes: determining the target area to be filtered from the surrounding side of the first sub-region in the at least two image sub-regions according to the set filtering order; the surrounding side includes, in sequence: the left side, the upper side, the right side and the lower side of the connecting boundary between the first sub-region and the other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the connecting boundary, the second side area outside the first sub-region compared to the connecting boundary, the first side area and the second side area; according to the set filtering order, the target area to be filtered is adaptive loop filtering.

[0016] In some embodiments, based on the decoded information of the image to be processed, it is determined whether the first sub-region of at least two image sub-regions needs to be subjected to neural network-based filtering processing; when the first sub-region needs to be subjected to neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

[0017] In some embodiments, when the first sub-region needs to be filtered based on a neural network, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network, and the permission level of at least one image sub-region adjacent to the first sub-region is input into the neural network.

[0018] In some embodiments, the electronic device may obtain information of an image sub-region that is lower than or equal to the user's permission level based on the user's permission level; the information of the image region includes pixel values ​​of the image sub-region.

[0019] In some embodiments, if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering processing is performed on the connecting boundary between the at least two image sub-regions, including: filtering processing of the luminance channel component of the connecting boundary between the at least two image sub-regions; or, filtering processing of the chrominance channel component of the connecting boundary between the at least two image sub-regions; or, filtering processing of the luminance channel component and the chrominance channel component of the connecting boundary between the at least two image sub-regions.

[0020] In a second aspect, the present application provides an image processing device, which includes: an image division unit and an image filtering unit; the image division unit is used to divide the image to be processed into multiple image sub-regions; an image decoding unit is used to obtain the authority level of each image sub-region; and the image filtering unit is used to filter the adjacent boundaries between at least two image sub-regions if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different.

[0021] In some embodiments, the permission level of each image sub-region is used to indicate: whether each image sub-region is a set permission region, and if each image sub-region is a permission region, the electronic device determines the permission level of each image sub-region; the permission level includes a first permission level and a second permission level.

[0022] In some embodiments, the image decoding unit is further configured to receive a decoded code stream of the image to be processed, perform decoding based on the decoded code stream, and obtain a decoded image to be processed.

[0023] In some embodiments, the image filtering unit is further used to determine a target area to be filtered on the vertical side of a first sub-region in at least two image sub-regions; the vertical side is the left or right side of the adjacent boundary of the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the right side of the adjacent boundary, the area to be filtered on the left side of the adjacent boundary, the area to be filtered on the right side of the adjacent boundary, and the area to be filtered on the left side of the adjacent boundary; the image filtering unit is further used to perform vertical filtering to remove blocking effects on the target area to be filtered.

[0024] In some embodiments, the at least two adjacent image subregions further include a second subregion that is horizontally adjacent to the first subregion, where horizontal adjacency refers to the second subregion being located horizontally to the left or right of the first subregion. The at least two adjacent image subregions have different permission levels, including: the permission level of the first subregion is lower than the permission level of the second subregion; or the permission level of the first subregion is higher than the permission level of the second subregion; or the permission level of the first subregion is not equal to the permission level of the second subregion; or the permission level of the first subregion is not equal to the permission level of the second subregion, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.

[0025] In some embodiments, the image filtering unit is further used to determine a target area to be filtered on the horizontal side of a first sub-region in at least two image sub-regions; the horizontal side is the upper side or lower side of the boundary between the first sub-regions, and the target area to be filtered is any one of the following: the area to be filtered on the lower side of the boundary, the area to be filtered on the upper side of the boundary, the area to be filtered on the lower side of the boundary, and the area to be filtered on the upper side; the image filtering unit is further used to perform horizontal filtering to remove blocking effects on the target area to be filtered.

[0026] In some embodiments, the at least two adjacent image subregions further include a third subregion, which is vertically adjacent to the first subregion. Vertically adjacent here means that the third subregion is located horizontally above or below the first subregion. The at least two adjacent image subregions have different permission levels, including: the permission level of the first subregion is lower than the permission level of the third subregion; or the permission level of the first subregion is higher than the permission level of the third subregion; or the permission level of the first subregion is not equal to the permission level of the third subregion; or the permission level of the first subregion is not equal to the permission level of the third subregion, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.

[0027] In some embodiments, the image filtering unit is further used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the boundary, the second side area outside the first sub-region compared to the boundary, the first side area and the second side area; the image filtering unit is further used to perform sample adaptive compensation filtering on the target area to be filtered according to the set filtering order.

[0028] In some embodiments, the image filtering unit is further used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the boundary, the second side area outside the first sub-region compared to the boundary, the first side area and the second side area; the image filtering unit is further used to perform adaptive loop filtering on the target area to be filtered according to the set filtering order.

[0029] In some embodiments, the image filtering unit is further used to determine whether it is necessary to perform neural network-based filtering processing on the first sub-region of at least two image sub-regions based on the decoded information of the image to be processed; when the first sub-region needs to perform neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

[0030] In some embodiments, the image filtering unit is further configured to input the permission level of at least one image sub-region adjacent to the first sub-region into the neural network.

[0031] In some embodiments, the image filtering unit is further used to perform filtering processing on the luminance channel component of the connecting boundary between at least two image sub-regions; or, to perform filtering processing on the chrominance channel component of the connecting boundary between at least two image sub-regions; or, to perform filtering processing on the luminance channel component and the chrominance channel component of the connecting boundary between at least two image sub-regions.

[0032] In a third aspect, the present application provides an electronic device comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; wherein, when the processor executes the computer instructions, the image processing device performs the image processing method as in the first aspect and any possible implementation thereof.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an image processing device, the image processing device implements the image processing method of the first aspect and any possible implementation thereof.

[0034] In a fifth aspect, the present application provides a computer program product that, when executed on an image processing device, causes the image processing device to perform the image processing method of the first aspect and any possible implementation thereof. The beneficial effects of aspects 2 through 5 above can be found in the corresponding description of aspect 1 and are not further elaborated here.

[0035] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of an image processing process framework provided in this application;

[0037] Figure 2 A schematic diagram of a deblocking filter area provided in this application;

[0038] Figure 3 A filter block boundary diagram provided in this application;

[0039] Figure 4 A schematic diagram of a filter provided in this application;

[0040] Figure 5 A schematic structural diagram of an image processing device provided in this application;

[0041] Figure 6 A flowchart of an image processing method provided in this application;

[0042] Figure 7 A schematic diagram of a detection area provided in this application;

[0043] Figure 8 A schematic diagram of a filtering area provided in this application;

[0044] Figure 9 A flowchart of another image processing method provided by this application;

[0045] Figure 10 A schematic diagram of another filtering area provided in this application;

[0046] Figure 11 A flowchart of another image processing method provided by this application;

[0047] Figure 12 A schematic diagram of another filtering area provided in this application;

[0048] Figure 13 A flowchart of another image processing method provided by this application;

[0049] Figure 14A schematic diagram of another filtering area provided in this application;

[0050] Figure 15 A flowchart of another image processing method provided by this application;

[0051] Figure 16 A schematic diagram of another filtering area provided in this application;

[0052] Figure 17 A flowchart of another image processing method provided by this application;

[0053] Figure 18 A flowchart of another image processing method provided by this application;

[0054] Figure 19 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in the embodiments of this application as "exemplarily" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0058] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application and the main processes of video encoding and decoding are briefly explained below.

[0059] 1. Technical terms

[0060] 1. Deblocking filter (DBF): used to remove the block boundary effect caused by block coding, including vertical deblocking filter and horizontal deblocking filter.

[0061] 2. Sample adaptive offset (SAO): By classifying the sample based on its pixel value and the gradient value of the surrounding block, different compensation values ​​are added to the pixel values ​​of each category, making the reconstructed image closer to the original image. The basic principle of SAO is to compensate the peak pixels in the reconstructed curve by adding negative values ​​and to supplement the valley pixels by adding positive values. SAO uses the Collect Transfer Unit (CTU) as the basic unit and includes two major types of compensation: Edge Offset (EO) and Band Offset (BO). In addition, parameter fusion technology is also introduced. Sample offset compensation filtering includes basic sample offset compensation (the above-mentioned SAO), enhanced sample offset compensation (ESAO), and cross-component sample offset compensation (CCSAO, only for chroma).

[0062] 3. Adaptive loop filter (ALF): This filter enhances the reconstructed image using a Wiener filter, making it closer to the original image. The ALF can also use the Enhanced Adaptive Correction Filter (EALF).

[0063] 4. Neural Network (NN): A neural network is a computational model composed of a large number of interconnected nodes (or neurons). In an artificial neural network, neuronal processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. Processing units in the network are divided into three types: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the system's processing results; and hidden units are located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information is reflected in the connections between the network's processing units. Artificial neural networks are a non-programmed, brain-like information processing method. Their essence is to achieve parallel and distributed information processing capabilities through network transformations and dynamic behavior, mimicking the information processing functions of the human brain to varying degrees and levels. Currently, commonly used neural networks in the field of video processing include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0064] 2. The main process of video encoding and decoding

[0065] like Figure 1 As shown, taking video coding as an example, video coding generally includes prediction, transformation, quantization, entropy coding and other processes. Furthermore, the coding process can also be carried out according to Figure 1 The framework in .

[0066] Prediction can be divided into intra-frame prediction and inter-frame prediction. Intra-frame prediction uses surrounding coded blocks as references to predict the current uncoded block, effectively removing redundancy in the spatial domain. Inter-frame prediction uses adjacent coded images to predict the current image, effectively removing redundancy in the temporal domain.

[0067] Transformation involves converting an image from the spatial domain to the transform domain, representing it using transform coefficients. Most images contain a large number of flat areas and slowly varying regions. Appropriate transformations can shift the image from a dispersed distribution in the spatial domain to a relatively concentrated distribution in the transform domain, removing frequency-domain correlations between signals. Combined with quantization, this effectively compresses the bitstream.

[0068] Entropy coding is a lossless encoding method that converts a series of element symbols into a binary stream for transmission or storage. The input symbols may include quantized transform coefficients, motion vector information, prediction mode information, and transform and quantization-related syntax. Entropy coding effectively removes redundancy in video element symbols.

[0069] The above explanation uses encoding as an example. Video decoding and encoding are parallel processes. Video decoding typically involves entropy decoding, prediction, inverse quantization, inverse transform, and filtering. The implementation principles of each process are the same or similar to those of entropy encoding. Filtering includes deblocking filtering, sample adaptive offset filtering, adaptive loop filtering, and neural network-based filtering.

[0070] 3. Deblocking filter (DBF)

[0071] The implementation of DBF filtering is briefly described below.

[0072] DBF filtering processing includes two steps: filtering decision and filtering operation.

[0073] The filtering decision involves: 1) obtaining the boundary filter strength (BS value); 2) determining whether to enable or disable filtering; and 3) selecting the appropriate filter strength. For chroma, only step 1) is performed, and the BS value for luma is directly reused. For chroma, filtering is performed only when the BS value is 2 (i.e., at least one of the image regions on either side is in intra mode).

[0074] The filtering operations include: 1) strong filtering and weak filtering for luminance; 2) filtering for chrominance.

[0075] The boundary to be filtered that meets any of the following conditions does not require DBF filtering:

[0076] Condition 1: If the boundary to be filtered is an image boundary, then the boundary does not need to be filtered.

[0077] Condition 2: If the boundary to be filtered is a slice boundary and the cross-slice loop filtering enable flag is 0, then filtering is not required for this boundary.

[0078] Condition 3: If the boundary to be filtered is a luminance filtering boundary, and the luminance filtering boundary is not a boundary of a luminance coding block or a luminance transform block, then the boundary does not need to be filtered.

[0079] Condition 4: If the boundary to be filtered is a chroma filter boundary, and the chroma filter boundary is not the boundary of a chroma coding block or a chroma transform block, and the luminance filter boundary corresponding to the chroma filter boundary is not the boundary of a luminance coding block, then the boundary does not need to be filtered.

[0080] Condition 5: If the boundary to be filtered is a luminance filtering boundary, the sub-block transform flag of the coding unit where the luminance filtering boundary is located is 1, and the luminance filtering boundary is not a boundary of a luminance coding block, then the boundary does not need to be filtered.

[0081] Condition 6: If the string copy intra prediction mode flag of the coding unit where the boundary to be filtered is located is 1, then the boundary does not need to be filtered.

[0082] DBF filtering generally performs deblocking vertical filtering and deblocking horizontal filtering in units of 8x8, and DBF filtering can filter up to 3 pixels on both sides of the boundary of the current block (image area to be filtered), and can filter up to 4 pixels on both sides of the boundary, so vertical / horizontal filtering of different blocks does not affect each other and can be performed in parallel. Figure 2 As shown, for the current 8x8 block, vertical filtering is first performed on the 3 columns on the left side of the current block and the 3 columns on the right side of the left block, and then horizontal filtering is performed on the 3 rows above the current block and the 3 rows below the upper block.

[0083] (1) The derivation process of boundary strength, that is, BS value, is as follows:

[0084] like Figure 3 As shown in FIG, a schematic diagram of the vertical boundary (vertical boundary) and the horizontal boundary (horizontal boundary) of the current block is shown, and the 8 pixel samples on both sides of the vertical boundary / horizontal boundary are recorded as p0, p1, p2, p3 and q0, q1, q2, q3 respectively. It should be noted that when calculating the boundary filter strength BS value, the boundary filter strength BS value is calculated for the pixels of the current block, that is, the boundary is the boundary of the pixels of the current block, not the boundary of the current luminance block or the current chrominance block.

[0085] Boundary filter strength determination method 1:

[0086] If all of the following conditions are met, the boundary filter strength BS value is equal to 0.

[0087] a) The quantization coefficients of the transform blocks of the coding units where p0 and q0 are located are both 0. If p0 (or q0) is a luma sample and the coding unit where p0 (or q0) is located contains only luma samples, then the coding unit where p0 (or q0) is located refers to the luma coding unit containing p0 (or q0); if p0 (or q0) is a chroma sample and the coding unit where p0 (or q0) is located contains only chroma samples, then the coding unit where p0 (or q0) is located refers to the coding unit containing the luma sample corresponding to p0 (or q0); otherwise (i.e., the coding unit where p0 or q0 is located contains both luma and chroma samples), then the coding unit where p0 (or q0) is located refers to the coding unit containing p0 (or q0).

[0088] b) The prediction type of the coding unit where p0 and q0 are located is not intra-frame.

[0089] c) Record B P and B Q are the 4×4 luminance coding blocks where p0 and q0 are located, B P and B Q The motion information satisfies the following conditions 1 and 2 at the same time or satisfies conditions 3, 4 and 5 at the same time.

[0090] 1)B P and B Q The L0 reference indexes of the corresponding spatial motion information storage units are all equal to -1, or B P and B Q The reference frames corresponding to the L0 reference indexes of the spatial motion information storage units are the same frame and the differences of all components of the L0 motion vectors of the spatial motion information storage units are less than an integer pixel.

[0091] 2)B P and B Q The L1 reference indexes of the corresponding spatial motion information storage units are all equal to -1, or B P and B Q The reference frames corresponding to the L1 reference indexes of the spatial motion information storage units are the same frame, and the differences of all components of the L1 motion vectors of the spatial motion information storage units are less than an integer pixel.

[0092] 3) Meet one of the following conditions:

[0093] B P The L0 reference index of the corresponding spatial motion information storage unit is equal to -1 and BQ The L1 reference index of the corresponding spatial motion information storage unit is equal to -1.

[0094] B P The reference frame corresponding to the L0 reference index of the corresponding spatial motion information storage unit is the same as the reference frame B Q The reference frame corresponding to the L1 reference index of the corresponding spatial motion information storage unit is the same frame and B P The L0 motion vector and B of the corresponding spatial motion information storage unit Q The differences of all components of the L1 motion vector of the corresponding spatial motion information storage unit are all less than an integer pixel.

[0095] 4) Meet one of the following conditions:

[0096] Condition 1: B Q The L0 reference index of the corresponding spatial motion information storage unit is equal to -1 and B P The L1 reference index of the corresponding spatial motion information storage unit is equal to -1.

[0097] Condition 2: B Q The reference frame corresponding to the L0 reference index of the corresponding spatial motion information storage unit is the same as the reference frame B P The reference frame corresponding to the L1 reference index of the corresponding spatial motion information storage unit is the same frame and B Q The L0 motion vector and B of the corresponding spatial motion information storage unit P The differences of all components of the L1 motion vector of the corresponding spatial motion information storage unit are all less than an integer pixel.

[0098] 5)B P The reference frame and B corresponding to the L0 reference index of the corresponding spatial motion information storage unit Q The reference frames corresponding to the L0 reference indexes of the corresponding spatial motion information storage units are not the same frame; B P The reference frame and B corresponding to the L1 reference index of the corresponding spatial motion information storage unit Q The reference frames corresponding to the L1 reference indexes of the corresponding spatial motion information storage units are not the same frame.

[0099] Otherwise, the boundary filter strength Bs value is calculated according to the second boundary filter strength determination method.

[0100] Boundary filter strength determination method 2:

[0101] Step 1: Calculate the average quantization parameter QP of the coding unit where p0 and q0 are located avIf it is a luma sample, the quantization parameter of the luma coding block should be used; if it is a chroma sample, the quantization parameter of the chroma coding block should be used. Let the quantization parameter of the coding unit where p0 is located be QP p , the quantization parameter of the coding unit where q0 is located is QP q , the average quantization parameter is:

[0102] <![CDATA[QP av =(QP p +QP q +1)>>1]]>

[0103] Step 2: Calculate indexes IndexA and IndexB.

[0104]

[0105] Step 3: Look up the table based on IndexA and IndexB to get the values ​​of α' and β' respectively, and then get the values ​​of α and β based on BitDepth.

[0106]

[0107] For example, ">>" is a right shift operation, which is used to replace division, that is, ">>5" is equivalent to dividing by 25 (that is, 32). In addition, in the embodiment of the present application, multiplication (that is, "*") can be replaced by left shift in actual implementation. For example, a multiplied by 4 can be replaced by a left shift of 2 bits, that is, by a<<2; a multiplied by 10 can be replaced by (a<<3)+(a<<1).

[0108] Exemplarily, “<<” is a left shift operation, which is used to replace multiplication, that is, “a<<2” is equivalent to multiplication by 22 (ie, 4).

[0109] For example, considering that when a division operation is implemented by shifting, the operation result is usually rounded directly.

[0110] That is, when the result of the operation is a non-integer between N and N+1, the result is taken as N. Considering that when the decimal part is greater than 0.5, the accuracy of taking the result as N+1 will be higher, therefore, in order to improve the accuracy of the determined pixel value, when performing the calculation, 1 / 2 of the denominator (i.e., the dividend) can be added to the numerator of the above weighted sum to achieve the effect of rounding.

[0111] Step 4: If DeblockingFilterType is 1 and Abs(p0-q0) is greater than or equal to 4×α, then Bs is equal to 0; otherwise, calculate the Bs value as follows.

[0112] 1) Set the values ​​of fL and fR to 0 and calculate fS.

[0113]

[0114] 2) Determine the Bs value based on fS, and the following situations exist.

[0115] Case 1: When fS is equal to 6, if Abs(p0-p1) is less than or equal to β / 4 and Abs(q0-q1) is less than or equal to β / 4 and Abs(p0-q0) is less than α, then the Bs value is equal to 4; otherwise, the Bs value is equal to 3.

[0116] Case 2: When fS is equal to 6 and DeblockingFilterType is equal to 1, if Abs(p0-p1) is less than or equal to β / 4 and Abs(q0-q1) is less than or equal to β / 4 and Abs(p0-p3) is less than or equal to β / 2 and Abs(q0-q3) is less than or equal to β / 2 and Abs(p0-q0) is less than α, then the Bs value is equal to 4; otherwise, the Bs value is equal to 3.

[0117] Case 3: When fS is equal to 5, if p0 is equal to p1 and q0 is equal to q1, then the Bs value is equal to 3; otherwise, the Bs value is equal to 2.

[0118] Case 4: When fS is equal to 5 and DeblockingFilterType is equal to 1, if p0 is equal to p1 and q0 is equal to q1 and Abs(p2-q2) is less than α, then the Bs value is equal to 3; otherwise, the Bs value is equal to 2.

[0119] Case 5: When fS is equal to 4, if fL is equal to 2, then the Bs value is equal to 2; otherwise, the Bs value is equal to 1.

[0120] Case 6: When fS is equal to 3, if Abs(p1–q1) is less than β, then the Bs value is equal to 1; otherwise, the Bs value is equal to 0.

[0121] Case 7: When fS is other values, the Bs value is equal to 0.

[0122] 3) If the Bs value obtained in step 2) is not equal to 0 and the filtered boundary is a chroma coding block boundary, the Bs value is reduced by 1.

[0123] (2) Determine the deblocking filter adjustment parameters

[0124] Deblocking filter adjustment parameters DbrThresold, DbrOffset0, DbrOffset1, DbrAltOffset0, and DbrAltOffset1 are determined.

[0125] If the current boundary is a vertical boundary and PictureDbrVEnableFlag is 1, or if the current boundary is a horizontal boundary and PictureDbrHEnableFlag is 1, the value of PictureDbrEnableFlag is 1; otherwise, the value of PictureDbrEnableFlag is 0.

[0126] If the current boundary is a vertical boundary and PictureAltDbrVEnableFlag is 1, or if the current boundary is a horizontal boundary and PictureAltDbrHEnableFlag is 1, the value of PictureAltDbrEnableFlag is 1; otherwise, the value of PictureAltDbrEnableFlag is 0.

[0127] For vertical boundaries:

[0128]

[0129] For horizontal boundaries:

[0130]

[0131] (3) Boundary filtering process when the brightness component Bs is equal to 4

[0132] When the boundary filter strength Bs is 4, the calculation process for filtering p0, p1, p2 and q0, q1, q2 is as follows (P0, P1, P2 and Q0, Q1, Q2 are filtered values):

[0133]

[0134] If the value of PictureDbrEnableFlag is 1, adjust the values ​​of P0, P1, P2, Q0, Q1, and Q2 according to 3.2.11.

[0135] (IV) Boundary filtering process when the brightness component Bs is equal to 3

[0136] When the boundary filter strength Bs is 3, the calculation process for filtering p0, p1 and q0, q1 is as follows (P0, P1 and Q0, Q1 are the filtered values):

[0137]

[0138] If the value of PictureDbrEnableFlag is 1, adjust the values ​​of P0, P1, Q0, and Q1 according to 3.2.11.

[0139] (V) Boundary filtering process when the brightness component Bs is equal to 2

[0140] When the boundary filter strength Bs is 2, the calculation process for filtering p0 and q0 is as follows (P0 and Q0 are the filtered values):

[0141]

[0142] If the value of PictureDbrEnableFlag is 1, adjust the values ​​of P0 and Q0 according to 3.2.11.

[0143] (6) Boundary filtering process when the brightness component Bs is equal to 1

[0144] When the value of the boundary filter strength Bs is 1, the calculation process of filtering p0 and q0 is as follows (P0 and Q0 are the filtered values):

[0145]

[0146] If the value of PictureDbrEnableFlag is 1, adjust the values ​​of P0 and Q0 according to 3.2.11.

[0147] (VII) Boundary filtering process when the brightness component Bs is equal to 0

[0148] When the value of the boundary filter strength Bs is 0 and the value of PictureAltDbrEnableFlag is 1, the calculation process for the filter adjustment of p0 and q0 is as follows (P0 and Q0 are the values ​​after the filter adjustment):

[0149]

[0150] (8) Boundary filtering process when the chrominance component Bs is greater than 0

[0151] When the value of the boundary filter strength Bs is greater than 0, the calculation process of filtering p0 and q0 is as follows (P0 and Q0 are the filtered values):

[0152]

[0153]

[0154] When the value of the boundary filter strength Bs is equal to 3, the calculation process of filtering p1 and q1 is as follows (P1 and Q1 are the filtered values):

[0155]

[0156] (IX) Deblocking filter adjustment process

[0157] Adjust P i and Q i (i can be 0, 1 or 2) value:

[0158]

[0159] Clip1(x) means to limit x to the range [0, 2^(bit_depth)-1] (inclusive). bit_depth represents the bit depth of the image, which is usually 8, 10, 12, etc.

[0160] 4. Sample Adaptive Offset (SAO)

[0161] SAO filtering is used to eliminate the ringing effect. The ringing effect is caused by the quantization distortion of high-frequency AC coefficients, which will produce ripples around the edges after decoding. The larger the transform block size, the more obvious the ringing effect. The basic principle of SAO is to compensate the peak pixels in the reconstructed curve by adding negative values ​​and supplement the valley pixels by adding positive values. SAO uses the Collect Transfer Unit (CTU) as the basic unit and includes two major types of compensation: edge offset (EO) and sideband offset (BO). In addition, parameter fusion technology is also introduced.

[0162] The sample offset compensation filtering includes basic sample offset compensation (the above-mentioned SAO), enhanced sample offset compensation (the above-mentioned ESAO), and cross-component sample offset compensation (the above-mentioned CCSAO, which is performed only for chroma).

[0163] The basic process of SAO:

[0164] If SAO is enabled for the current component of the current block, the basic sample value offset compensation unit is first derived, then the basic sample value offset compensation information corresponding to the current basic sample value offset compensation unit is derived, and finally, each component of each sample within the current basic sample value offset compensation unit is operated to obtain the offset sample value. Otherwise, the value of the corresponding component of the filtered sample is directly used as the offset value of the sample component.

[0165] The basic process of ESAO (ESAO and SAO are mutually exclusive, that is, if SAO is enabled, ESAO is not enabled):

[0166] If ESAO is enabled for the current component of the current block, the enhanced sample value offset compensation unit is first derived, and then each component of each sample in the enhanced sample value offset compensation unit is operated to obtain the offset sample value; otherwise, the value of the corresponding component of the filtered sample is directly used as the value of the sample component after offset.

[0167] The basic process of CCSAO:

[0168] If CCSAO is enabled for the current component of the current block, the cross-component sample offset compensation unit is first derived, and then the cross-component sample offset compensation unit is operated on each component sample to obtain the cross-component offset sample value. Otherwise, the value of the component corresponding to the offset sample is directly used as the value of the sample component after the cross-component offset.

[0169] For chroma pixels that require CCSAO, the classification of the current chroma pixel is determined based on the deblocking filtered (adjusted) luminance value (Y5 described above) at the corresponding position and the current chroma pixel (the pixel after ESAO). Based on the classification, a compensation value for the current chroma pixel is determined, and this compensation value is added to the current chroma pixel to obtain the chroma pixel value after CCSAO filtering.

[0170] 5. Adaptive loop filter (ALF)

[0171] ALF filtering is a Wiener filter that calculates the optimal filter in the mean squared sense based on the original and distorted signals. The adaptive correction filter (ALF) can be used with or without the enhanced adaptive correction filter (EALF).

[0172] The basic process of ALF is as follows:

[0173] If the current component of the current image does not enable ALF, the value of the offset sample component is directly used as the value of the corresponding reconstructed sample component; otherwise, adaptive correction filtering is performed on the corresponding offset sample component.

[0174] The adaptive correction filter (ACCF) unit is derived from the LCU and processed sequentially in raster scan order. First, the ACC coefficients for each component are decoded. The ACCF unit is then derived and the ACCF coefficient index for the luminance component of the current ACCF unit is determined. Finally, the ACCF unit's luminance and chrominance components are adaptively filtered to produce a reconstructed sample.

[0175] If EALF is enabled for the current image, each set of filter coefficients for the ALF of the current image has 15, with a shape like Figure 4 As shown in (a), the maximum number of adaptive correction filters for the brightness component of the current image is 64; otherwise, each group of filter coefficients of the ALF of the current image has 9, and the shape Figure 4 As shown in (b) in FIG, the maximum number of adaptive correction filters for the brightness component of the current image is 16.

[0176] Whether it is DBF filtering, SAO filtering or ALF filtering, they are all classified based on the current pixel value of the adjacent boundary to be filtered, or the relationship between the pixel value of the current block and the pixel value of the adjacent blocks around the current block (for example, the level of authority), and then different filtering operations are performed based on different categories.

[0177] As mentioned earlier, in video encoding and decoding, filtering aims to achieve smooth noise reduction and detail removal within video image blocks while preserving image edges to the greatest extent possible. Typically, electronic devices filter the boundaries of multiple regions within an image to preserve image edges.

[0178] To protect user data security, general images can be configured with permission areas (e.g., high-permission level areas, low-permission level areas) and non-permission areas (zero-permission level areas). Permission areas can include multiple permission areas of varying levels. A permission area refers to an area where only users with the relevant permissions can correctly decode the image content. For example, only users with high permissions can correctly view image areas of any permission level, while users with only low permissions cannot view image areas of higher permission levels and can only view image areas of low permission levels and zero-permission level areas.

[0179] Therefore, since different users have different viewing permissions for different areas in the image, when a low-authority user does not have viewing permissions for high-authority areas in the image, the electronic device will not filter the adjacent boundaries of image areas with different permission levels during the video decoding process, resulting in poor image quality.

[0180] To address this issue, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium. This method uses an electronic device to divide an image to be processed into multiple image sub-regions, obtain the permission level of each image sub-region, and determine the permission level of each image sub-region. If the permission levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, this indicates that the two image sub-regions may still have not been filtered after filtering during the decoding process, resulting in low quality of the image to be processed. Therefore, based on the decoded information, such as the division of permission regions, the electronic device can further filter the adjacent boundaries between at least two image sub-regions, thereby improving the image quality of different permission regions in the image to be processed.

[0181] The image processing method provided in the embodiment of the present application can be applied to Figure 5 In the image processing device 11 shown, the image processing device 11 includes an image dividing unit 101 , an image filtering unit 102 , and an image decoding unit 103 .

[0182] The image division unit 101 is used to divide the image to be processed into multiple image sub-regions.

[0183] The image decoding unit 103 is configured to obtain the permission level of each image sub-region.

[0184] The image filtering unit 102 is configured to perform filtering processing on a boundary between at least two adjacent image sub-regions among the plurality of image sub-regions if the permission levels of at least two adjacent image sub-regions are different.

[0185] The image decoding unit 103 is further configured to receive a decoded code stream of the image to be processed, perform decoding based on the decoded code stream, and obtain a decoded image to be processed.

[0186] In some embodiments, the image filtering unit 102 is further used to determine a target area to be filtered on the vertical side of a first sub-region in at least two image sub-regions; the vertical side is the left or right side of the adjacent boundary of the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the right side of the adjacent boundary, the area to be filtered on the left side of the adjacent boundary, the area to be filtered on the right side of the adjacent boundary, and the area to be filtered on the left side of the adjacent boundary; the image filtering unit 102 is further used to perform deblocking vertical filtering on the target area to be filtered.

[0187] In some embodiments, the image filtering unit 102 is further used to determine a target area to be filtered on the horizontal side of a first sub-region in at least two image sub-regions; the horizontal side is the upper side or lower side of the boundary between the first sub-regions, and the target area to be filtered is any one of the following: the area to be filtered on the lower side of the boundary, the area to be filtered on the upper side of the boundary, the area to be filtered on the lower side of the boundary, and the area to be filtered on the upper side; the image filtering unit 102 is further used to perform horizontal filtering to remove blocking effects on the target area to be filtered.

[0188] In some embodiments, the image filtering unit 102 is further used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the boundary, the second side area outside the first sub-region compared to the boundary, the first side area and the second side area; the image filtering unit 102 is further used to perform sample adaptive compensation filtering on the target area to be filtered according to the set filtering order.

[0189] In some embodiments, the image filtering unit 102 is further used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the connected boundary, the second side area outside the first sub-region compared to the connected boundary, the first side area and the second side area; the image filtering unit 102 is further used to perform adaptive loop filtering on the target area to be filtered according to the set filtering order.

[0190] In some embodiments, the image filtering unit 102 is further used to determine whether it is necessary to perform neural network-based filtering processing on the first sub-region of at least two image sub-regions based on the decoded information of the image to be processed; when the first sub-region needs to perform neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

[0191] In some embodiments, the image filtering unit 102 is further configured to input the permission level of at least one image sub-region adjacent to the first sub-region into the neural network.

[0192] In some embodiments, the image filtering unit 102 is further configured to perform filtering processing on the luminance channel component of the adjacent boundary between at least two image sub-regions; or, perform filtering processing on the chrominance channel component of the adjacent boundary between at least two image sub-regions; or, perform filtering processing on the luminance channel component and the chrominance channel component of the adjacent boundary between at least two image sub-regions.

[0193] Figure 6 This is a flow chart of an image processing method provided in an embodiment of the present application. For example, the image processing method provided in this application can be applied to Figure 5 The image processing device shown in FIG. Figure 6 As shown, the image processing method provided by this application may specifically include the following steps:

[0194] S101: Divide an image to be processed into multiple image sub-regions, and obtain the permission level of each image sub-region.

[0195] The permission levels are divided into at least two different permission levels.

[0196] In some embodiments, an image may be provided with a permission area and a non-permission area (zero permission level area). The image sub-area of ​​the non-permission area (zero permission level area) is an image sub-area that any user can view. Among them, the permission area may include multiple permission areas of different high and low levels. For example, the permission area may include a first permission level area (high permission level area), a second permission level area (medium permission level area), and a third permission level area (low permission level area). The permission level of the first permission level area is higher than the permission level of the second permission level area, and the permission level of the second permission level area is higher than the permission level of the third permission level area. Each permission level may also be divided into multiple sub-permission levels, which is not specifically limited in this embodiment.

[0197] In some embodiments, the permission level of a target area in an image is configured based on the category of the target area. For example, a person's face in an image is a high-privilege area, the upper body is a low-privilege area, and the lower body is a zero-privilege area; the license plate in a vehicle image is a high-privilege area, the windows are a low-privilege area, and the vehicle body is a zero-privilege area.

[0198] In some embodiments, the permission levels of the image sub-regions in the image to be processed can be divided into high permission level, low permission level region and zero permission level region. For example, Figure 7 As shown, the plurality of image sub-regions may include a low permission level region, a high permission level region, and a zero permission level region.

[0199] It is understandable that different users have different viewing permissions for different areas in the image. High-authority users can view image sub-areas at any permission level (for example, high-authority level, low-authority level areas, and zero-authority level areas), low-authority users can view image sub-areas at low-authority levels and image sub-areas at zero-authority levels, and unauthorized users can only view image sub-areas at zero-authority levels.

[0200] In some embodiments, before dividing the image to be processed into a plurality of image sub-regions, the image decoding unit 103 is used to receive a decoded code stream of the image to be processed, and decoding is performed based on the decoded code stream to obtain a decoded image to be processed.

[0201] In other embodiments, when the image to be processed includes a target image region, and the target image region is used as the filtering processing object, the image decoding unit 103 receives the decoded code stream of the image to be processed, and decodes the target image region based on the decoded code stream. After the decoding of the target image region is completed, it is not necessary to wait for the decoding of the entire image to be processed to be completed. The target image region can be directly divided into multiple image sub-regions to obtain the permission level of each image sub-region.

[0202] In other embodiments, when the image to be processed includes a target image area, when the target image area is used as the filtering processing object, the target image area can be taken as the center and expanded to the surrounding sides (left, right, top or bottom) to obtain a detection area with a width CW and a height CH. The above detection area is divided to obtain multiple image sub-block areas with NxN (N is preferably 2 or 4 or 8) pixel units, wherein the authority level within each image sub-block area is the same. The expanded pixel units of the detection area in the four directions can be different or completely the same.

[0203] For example, Figure 8 As shown in (a), the detection area can be centered on the target image area and extended to the left and upward by T (T is preferably 4, 8 or 16) unit pixels.

[0204] For example, Figure 8 As shown in (b) in the figure, the detection area can also be Figure 8 An upper right area is added to the detection area of ​​(a) as the overall detection area.

[0205] S102: If the authority levels of at least two adjacent image sub-regions among the plurality of image sub-regions are different, perform filtering processing on a boundary (boundary to be filtered) between the at least two image sub-regions.

[0206] In some embodiments, when the image to be processed includes a target image area that is the object of filtering processing, after the target image area is decoded using the image decoding unit 103, the target image area can be directly divided into multiple image sub-areas. If the authority levels of at least two adjacent image sub-areas in the multiple image sub-areas are different, the connecting boundary between the two image sub-areas can be directly filtered without waiting for the entire image to be processed to be decoded before performing filtering processing.

[0207] In some embodiments, the filtering process includes at least one of: deblocking effect vertical filtering, deblocking effect horizontal filtering, sample adaptive offset filtering, adaptive loop filtering and neural network filtering.

[0208] In some embodiments, the at least two adjacent image sub-regions may be two connected image sub-regions, or may be two non-connected image sub-regions but relatively close to each other.

[0209] In some embodiments, the boundary between at least two image sub-regions may be filtered for the luminance channel component; or, the boundary between at least two image sub-regions may be filtered for the chrominance channel component; or, the boundary between at least two image sub-regions may be filtered for both the luminance channel component and the chrominance channel component.

[0210] In some embodiments, the image processing device can obtain image area information of any permission level in the image to be processed, as well as image area information of non-authorization areas; or, the image processing device can obtain image area information of the second permission level in the image to be processed, as well as image area information of non-authorization areas.

[0211] It can be understood that by dividing the image to be processed into multiple image subregions and obtaining the permission level of each image subregion, and then determining the permission level of each image subregion, if the permission levels of at least two adjacent image subregions within the multiple image subregions are different, it indicates that these two image subregions may still have not been filtered after the filtering process during the decoding process, resulting in low quality of the processed image. Therefore, based on the decoded information such as the division of the permission regions, the boundary between at least two image subregions is further filtered to improve the image quality of different permission regions in the image to be processed.

[0212] In some embodiments, if there is a vertical boundary between the at least two image sub-regions (i.e., the two image sub-regions are horizontally connected), the filtering process includes deblocking vertical filtering, and if the conditions for DBF filtering are met, deblocking vertical filtering can be performed on the boundary between the at least two image sub-regions. See Figure 9 The decoding method provided in this embodiment includes the following steps Sa1 to Sa2.

[0213] Sa1. Determine a target area to be filtered on a vertical side of a first sub-region in at least two image sub-regions.

[0214] Among them, the vertical side is the left or right side of the boundary of the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the right side of the boundary, the area to be filtered on the left side of the boundary, the area to be filtered on the right side of the boundary, and the area to be filtered on the left side.

[0215] Sa2. Perform deblocking vertical filtering on the target area to be filtered.

[0216] In some embodiments, the at least two adjacent image sub-regions further include a second sub-region, and the second sub-region is horizontally adjacent to the first sub-region. Here, horizontal adjacent means that the second sub-region is located to the left or right of the first sub-region in the horizontal direction. The at least two adjacent image sub-regions have different permission levels, including any of the following situations:

[0217] 1. The permission level of the first sub-area is lower than that of the second sub-area.

[0218] 2. The permission level of the first sub-area is higher than the permission level of the second sub-area.

[0219] 3. The permission level of the first sub-area is not equal to the permission level of the second sub-area.

[0220] 4. The authority level of the first sub-area is not equal to the authority level of the second sub-area, and the filtering strength (BS value) of the adjacent boundary is greater than or equal to the preset filtering strength.

[0221] like Figure 10 As shown in (a), when the second sub-region is located on the left side of the first sub-region, the target area to be filtered is subjected to deblocking vertical filtering according to the different authority levels of the at least two adjacent image sub-regions, which is described in detail.

[0222] In Example 1, when the permission level of the first sub-region is lower than the permission level of the second sub-region, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The right side of the area to be filtered is a pixel area n columns wide to the right of the adjacent boundary, and n is preferably 1.

[0223] In Example 2, when the permission level of the first sub-region is higher than the permission level of the second sub-region, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The right side of the area to be filtered is a pixel area n columns wide to the right of the adjacent boundary, and n is preferably 1.

[0224] In a third embodiment, when the authority level of the first sub-region is not lower than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The right side of the area to be filtered is a pixel area n columns wide to the right of the adjacent boundary, and n is preferably 1.

[0225] In a fourth embodiment, when the authority level of the first sub-region is not equal to the authority level of the second sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The right area to be filtered is a pixel area n columns wide to the right of the adjacent boundary, and n is preferably 1.

[0226] Among them, Figure 10 As shown in (b), in the first to fourth embodiments, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-area).

[0227] In a fifth embodiment, when the permission level of the first sub-region is lower than the permission level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The left area to be filtered is a pixel area n columns wide to the left of the adjacent boundary, and n is preferably 1.

[0228] In Example 6, when the permission level of the first sub-region is higher than the permission level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The left area to be filtered is a pixel area n columns wide to the left of the adjacent boundary, and n is preferably 1.

[0229] In Example 7, when the authority level of the first sub-region is not equal to the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The left area to be filtered is a pixel area n columns wide to the left of the adjacent boundary, and n is preferably 1.

[0230] In the eighth embodiment, when the authority level of the first sub-region is not equal to the authority level of the second sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered to the left of the adjacent boundary (within the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The left area to be filtered is a pixel area n columns wide to the left of the adjacent boundary, and n is preferably 1.

[0231] Among them, Figure 10 As shown in (c), in the fifth to eighth embodiments, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-area).

[0232] In a ninth embodiment, when the authority level of the first subregion is lower than the authority level of the second subregion, the target area to be filtered is the area to be filtered on the right and left sides of the adjacent boundary (within the first region and the second subregion), and deblocking vertical filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n columns to the right and n sides to the left of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0233] In Example 10, when the authority level of the first sub-region is higher than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the right side and the area to be filtered on the left side of the adjacent boundary (within the first region and the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n columns to the right and n sides to the left of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0234] In Example 11, when the authority level of the first sub-region is not equal to the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the right side and the area to be filtered on the left side of the adjacent boundary (within the first region and the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n columns to the right and n sides to the left of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0235] Example 12: When the authority level of the first sub-region is not equal to the authority level of the second sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered on the right side and the area to be filtered on the left side of the adjacent boundary (within the first region and the second sub-region), and deblocking vertical filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area with a width of 2n, which is n columns to the right and n sides to the left of the adjacent boundary, and n is preferably 1.

[0236] Among them, Figure 10 As shown in (d), in the ninth to twelfth embodiments, the target area to be filtered is the right area to be filtered and the left area to be filtered at the adjacent boundaries (within the first area and the second sub-area).

[0237] It is understandable that if there is a vertical boundary between the at least two image sub-regions mentioned above, and the filtering process includes vertical filtering for deblocking effects, the target area to be filtered on the vertical side of the first sub-region in the at least two image sub-regions can be determined, and the target area to be filtered can be vertically filtered for deblocking effects. Particularly, based on the authority levels of the two image sub-regions, the corresponding target area to be filtered can be selected for vertical filtering for deblocking effects. In the case where the image processing device obtains information about the corresponding image area according to the authority level of the user, this method can meet the needs of users of any authority level to perform vertical filtering for deblocking effects on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

[0238] In some embodiments, if there is a horizontal boundary between the at least two image sub-regions (i.e., the two image sub-regions are vertically connected), the filtering process includes deblocking effect horizontal filtering, and if the DBF filtering conditions are met, the deblocking effect horizontal filtering is performed on the boundary between the at least two image sub-regions. Figure 11 The decoding method provided in this embodiment includes the following steps Sb1 to Sb2.

[0239] Sb1. Determine a target area to be filtered on the horizontal side of a first sub-region in at least two image sub-regions.

[0240] Among them, the horizontal side is the upper side or lower side of the boundary of the first sub-region, and the target area to be filtered is any one of the following: the upper side area to be filtered of the boundary, the lower side area to be filtered of the boundary, the upper side area to be filtered of the boundary, and the lower side area to be filtered of the boundary.

[0241] Sb2. Perform deblocking horizontal filtering on the target area to be filtered.

[0242] In some embodiments, the at least two adjacent image sub-regions further include a third sub-region, and the third sub-region is vertically adjacent to the first sub-region. Vertically adjacent here means that the third sub-region is located above or below the first sub-region in the horizontal direction. The permission levels of the at least two adjacent image sub-regions are different, including any of the following situations:

[0243] 1. The permission level of the first sub-area is lower than that of the third sub-area.

[0244] 2. The permission level of the first sub-area is higher than the permission level of the third sub-area.

[0245] 3. The permission level of the first sub-area is not equal to the permission level of the third sub-area.

[0246] 4. The authority level of the first sub-area is not equal to the authority level of the third sub-area, and the filtering strength (BS value) of the adjacent boundaries is greater than or equal to the preset filtering strength.

[0247] like Figure 12 As shown in (a), the third sub-region is located above the first sub-region. The following describes in detail the deblocking effect horizontal filtering of the target area to be filtered based on the different authority levels of the at least two adjacent image sub-regions.

[0248] In Example 13, when the permission level of the first sub-region is lower than the permission level of the third sub-region, the target area to be filtered is the area to be filtered below the adjacent boundary (within the first sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The lower area to be filtered is a pixel area with a width of n rows upward from the adjacent boundary, where n is preferably 1.

[0249] In Example 14, when the permission level of the first sub-region is higher than the permission level of the third sub-region, the target area to be filtered is the area to be filtered below the adjacent boundary (within the first sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The lower area to be filtered is a pixel area with a width of n rows upward from the adjacent boundary, where n is preferably 1.

[0250] In Example 15, when the permission level of the first sub-region is not lower than the permission level of the third sub-region, the target area to be filtered is the area to be filtered below the adjacent boundary (within the first sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The lower area to be filtered is a pixel area with a width of n rows upward from the adjacent boundary, where n is preferably 1.

[0251] Example 16: When the authority level of the first sub-region is not equal to the authority level of the third sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered below the adjacent boundary (within the first sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The lower area to be filtered is a pixel area with a width of n rows upward from the adjacent boundary, where n is preferably 1.

[0252] Among them, Figure 12 As shown in (b), in the thirteenth to sixteenth embodiments, the target area to be filtered is the area to be filtered at the lower side of the adjacent boundary (within the first sub-area).

[0253] In Example 17, when the permission level of the first sub-region is lower than the permission level of the third sub-region, the target area to be filtered is the area to be filtered above the adjacent boundary (within the third sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The upper area to be filtered is the area of ​​pixels with a width of n rows below the adjacent boundary, where n is preferably 1.

[0254] In Example 18, when the permission level of the first sub-region is higher than the permission level of the third sub-region, the target area to be filtered is the area to be filtered above the adjacent boundary (within the third sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The upper area to be filtered is the area of ​​pixels with a width of n rows below the adjacent boundary, where n is preferably 1.

[0255] In Example 19, when the permission level of the first sub-region is not equal to the permission level of the third sub-region, the target area to be filtered is the area to be filtered above the adjacent boundary (within the third sub-region), and deblocking horizontal filtering is performed on the target area to be filtered. The upper area to be filtered is the area of ​​pixels with a width of n rows below the adjacent boundary, where n is preferably 1.

[0256] Example 20: When the authority level of the first sub-region is not equal to the authority level of the third sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the upper area to be filtered of the adjacent boundary (within the third sub-region), and deblocking effect horizontal filtering is performed on the target area to be filtered. The upper area to be filtered is the pixel area with a width of n rows below the adjacent boundary, and n is preferably 1.

[0257] Among them, Figure 12As shown in (c), in the seventeenth to twentieth embodiments, the target area to be filtered is the area to be filtered on the upper side of the adjacent boundary (within the third sub-area).

[0258] In Example 21, when the permission level of the first subregion is lower than the permission level of the third subregion, the target area to be filtered is the lower and upper areas to be filtered of the adjacent boundary (within the first region and the third subregion), and horizontal deblocking filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n rows upward and n sides downward of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0259] In Example 22, when the authority level of the first subregion is higher than the authority level of the third subregion, the target area to be filtered is the lower and upper areas to be filtered (within the first region and the third subregion) of the adjacent boundary, and horizontal deblocking filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n rows upward and n sides downward of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0260] In Example 23, when the authority level of the first subregion is not equal to the authority level of the third subregion, the target area to be filtered is the lower and upper areas to be filtered of the adjacent boundary (within the first region and the third subregion), and deblocking horizontal filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area n rows upward and n sides downward of the adjacent boundary, with a width of 2n, where n is preferably 1.

[0261] Example 24: When the authority level of the first sub-region is not equal to the authority level of the third sub-region and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the lower and upper areas to be filtered of the adjacent boundary (within the first region and the third sub-region), and deblocking effect horizontal filtering is performed on the target area to be filtered. The target area to be filtered is a pixel area with a width of 2n, n rows upward and n sides downward of the adjacent boundary, where n is preferably 1.

[0262] Among them, Figure 12 As shown in (d), in Examples 21 to 24, the target area to be filtered is the upper area to be filtered and the lower area to be filtered of the adjacent boundaries (within the first area and the third sub-area).

[0263] It is understandable that if there is a horizontal boundary between the at least two image sub-regions mentioned above, and the filtering process includes deblocking effect horizontal filtering, the target area to be filtered on the horizontal side of the first sub-region in the at least two image sub-regions can be determined, and the target area to be filtered can be subjected to deblocking effect horizontal filtering. Particularly, based on the authority levels of the two image sub-regions, the corresponding target area to be filtered can be selected for deblocking effect horizontal filtering. In the case where the image processing device obtains the information of the corresponding image area according to the user's authority level, this method can meet the needs of users of any authority level to perform deblocking effect horizontal filtering on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

[0264] In some embodiments, the filtering process includes sample adaptive compensation filtering. If the permission levels of at least two adjacent image sub-regions in the plurality of image sub-regions are different, sample adaptive compensation filtering is performed on the boundary between the at least two image sub-regions. Figure 13 The decoding method provided in this embodiment includes the following steps Sc1 to Sc2.

[0265] Sc1. Determine a target area to be filtered from the periphery of a first sub-area in at least two image sub-areas according to a set filtering order.

[0266] Among them, the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions. When the authority level of the first sub-region is different from the authority level of other sub-regions, the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the boundary, the second side area outside the first sub-region compared to the boundary, the first side area and the second side area.

[0267] In some embodiments, the permission levels of other sub-regions surrounding the first sub-region may be determined in sequence according to a set filtering order. If the permission levels of other sub-regions surrounding the first sub-region are different from the permission level of the first sub-region, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

[0268] Case 1: If other sub-regions with different authority levels from the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-region.

[0269] Case 2: If other sub-regions with different authority levels from the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-region.

[0270] The above two situations can exist at the same time.

[0271] For example, Figure 14 As shown, the first sub-region is surrounded by an adjacent sub-region A on the left, an adjacent sub-region B on the upper left, an adjacent sub-region C on the upper side, and an adjacent sub-region D on the upper right. According to the filtering order from right to left and then from top to bottom, the authority levels of other sub-regions around the first sub-region are judged in turn. When the authority levels of adjacent sub-regions A and C are different from the authority level of the first sub-region, and when the authority levels of adjacent sub-regions B and D are the same as the authority level of the first sub-region, the target area to be filtered is determined to be a column pixel area with a width of 1 in the left direction and a column pixel area with a width of 1 in the right direction of the boundary connecting to the adjacent sub-region A, and a row pixel area with a width of 1 in the upward direction and a row pixel area with a width of 1 in the downward direction of the boundary connecting to the adjacent sub-region C, that is, Figure 14 The shaded area is shown.

[0272] In some other embodiments, when the authority levels of other sub-areas around the first sub-area are higher than the authority level of the first sub-area, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

[0273] Case 1: If other sub-regions with a higher authority level than the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-region.

[0274] Case 2: If other sub-regions with a higher authority level than the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-region.

[0275] The above two situations can exist at the same time.

[0276] Sc2. Perform sample adaptive compensation filtering on the target area to be filtered according to the set filtering order.

[0277] In some embodiments, as described above Figure 14 As shown, according to the filtering order from right to left and then from top to bottom, the target area to be filtered is Figure 14 The shaded area in the image is subjected to sample adaptive compensation filtering.

[0278] In some embodiments, the adjacent sub-regions around the first sub-region that have not been filtered retain their original pixel values. Figure 14 As shown, the pixel values ​​of the boundary between the first sub-region and the adjacent sub-region B and the boundary between the first sub-region and the adjacent sub-region D remain unchanged.

[0279] It is understandable that if the authority levels of at least two adjacent image sub-regions in a plurality of image sub-regions are different, the target area to be filtered can be determined from the surrounding side of the first sub-region of the at least two image sub-regions according to the set filtering order, and the target area to be filtered can be subjected to sample adaptive compensation filtering according to the set filtering order. Among them, based on the high and low authority levels of the adjacent sub-regions on the surrounding side of the first sub-region, the corresponding target area to be filtered can be selected for sample adaptive compensation filtering. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, this method can meet the needs of users of any authority level to perform sample adaptive compensation filtering on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

[0280] In some embodiments, the filtering process includes adaptive loop filtering. If the permission levels of at least two adjacent image sub-regions in the plurality of image sub-regions are different, adaptive loop filtering is performed on the boundary between the at least two image sub-regions. Figure 15 The decoding method provided in this embodiment includes the following steps Sd1 to Sd2.

[0281] Sd1. Determine a target area to be filtered from the periphery of a first sub-area in at least two image sub-areas according to a set filtering order.

[0282] Among them, the surrounding side includes, in sequence: the left side, upper side, right side and lower side of the boundary between the first sub-region and other sub-regions. When the authority level of the first sub-region is different from the authority level of other sub-regions, the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the boundary, the second side area outside the first sub-region compared to the boundary, the first side area and the second side area.

[0283] In some embodiments, the permission levels of other sub-regions surrounding the first sub-region may be determined in sequence according to a set filtering order. If the permission levels of other sub-regions surrounding the first sub-region are different from the permission level of the first sub-region, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

[0284] Case 1: If other sub-regions with different authority levels from the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-region.

[0285] Case 2: If other sub-regions with different authority levels from the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-region.

[0286] The above two situations can exist at the same time.

[0287] For example, Figure 16 As shown, the first subregion is surrounded by an adjacent subregion A on the left, an adjacent subregion B on the upper left, an adjacent subregion C on the upper side, and an adjacent subregion D on the upper right. According to the filtering order from right to left and then from top to bottom, the authority levels of other subregions around the first subregion are judged in turn. When the authority levels of adjacent subregions B, C, and D are different from the authority level of the first subregion, and when the authority level of adjacent subregion A is the same as the authority level of the first subregion, the target area to be filtered is determined to be a row pixel area with a width of 1 in the downward direction and a row pixel area with a width of 1 in the upward direction of the boundaries connecting the adjacent subregions B, C, and D, that is, Figure 16 The shaded area is shown.

[0288] In some other embodiments, when the authority levels of other sub-areas around the first sub-area are higher than the authority level of the first sub-area, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

[0289] Case 1: If other sub-regions with a higher authority level than the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-region.

[0290] Case 2: If other sub-regions with a higher authority level than the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-region.

[0291] The above two situations can exist at the same time.

[0292] Sd2: Perform adaptive loop filtering on the target area to be filtered according to the set filtering order.

[0293] In some embodiments, as described above Figure 16 As shown, according to the filtering order from right to left and then from top to bottom, the target area to be filtered is Figure 16 The shaded area in the image is used for adaptive loop filtering.

[0294] In some embodiments, the adjacent sub-regions around the first sub-region that have not been filtered retain their original pixel values. Figure 16 As shown, the pixel value of the boundary between the first sub-region and the adjacent sub-region A remains unchanged.

[0295] It is understandable that if the authority levels of at least two adjacent image sub-regions in a plurality of image sub-regions are different, the target area to be filtered can be determined from the surrounding side of the first sub-region in the at least two image sub-regions according to the set filtering order, and the target area to be filtered can be subjected to sample adaptive compensation filtering according to the set filtering order. Among them, based on the high and low authority levels of the adjacent sub-regions on the surrounding side of the first sub-region, the corresponding target area to be filtered can be selected for adaptive loop filtering. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, this method can meet the needs of users of any authority level to perform adaptive loop filtering on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

[0296] In some embodiments, the filtering process includes a neural network-based filtering process. If the permission levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, the boundary between the at least two image sub-regions may be subjected to a neural network-based filtering process. Figure 17 The decoding method provided in this embodiment includes the following steps Se1 to Se2.

[0297] Se1. When the first sub-region needs to be subjected to filtering processing based on a neural network, the pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network.

[0298] In some embodiments, the pixel value of the first sub-area can be a reconstructed pixel value, which is the pixel value of the first sub-area after decoding and filtering. It can be the pixel value after decoding and other filtering (such as deblocking vertical filtering). This embodiment does not impose any specific restrictions.

[0299] In some embodiments, when the first sub-region requires neural network-based filtering, the reconstructed pixel value of the first sub-region, the permission level of the first sub-region, and other decoded information are input into the neural network. The other decoded information includes the aforementioned boundary filter strength Bs value, the quantization coefficients of the coding unit transform block, and the like. This embodiment does not impose any specific limitations.

[0300] In some embodiments, the neural network may be a convolutional neural network (CNN) or other neural networks, which is not specifically limited in this embodiment.

[0301] In some embodiments, step Se1 may include step Se12.

[0302] Se12. Input the permission level of at least one image sub-region adjacent to the first sub-region into the neural network.

[0303] For example, the above Figure 16 The permission level of the adjacent sub-region A, the permission level of the adjacent sub-region B, the permission level of the adjacent sub-region C, and the permission level of the adjacent sub-region D around the first sub-region in are all input into the neural network.

[0304] Se2. Obtain the filtered pixels of the first sub-region of the output.

[0305] like Figure 18 As shown, the reconstructed pixel value of the first sub-region, the authority level of the first sub-region, the authority level of at least one image sub-region adjacent to the first sub-region, and other decoding information are input into the neural network to obtain the output filtered reconstructed pixel of the first sub-region.

[0306] It is understood that based on the permission level of the image sub-region, the corresponding target filtering region can be selected for neural network filtering. In the case where the image processing device obtains information about the corresponding image region based on the user's permission level, this method can meet the needs of users of any permission level to perform neural network filtering on the decoded image again, improving the image quality of different permission regions in the processed image, thereby enhancing the user experience at each permission level.

[0307] In some embodiments, the process of executing Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, and Sd1 to Sd2 can be performed in sequence, that is, the output of Sa1 to Sa2 is the input of Sb1 to Sb2, the output of Sb1 to Sb2 is the input of Sc1 to Sc2, and the output of Sc1 to Sc2 is the input of Sd1 to Sd2.

[0308] In other embodiments, the process of executing Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, Sd1 to Sd2, and Se1 to Se2 can be performed in sequence, that is, the output of Sa1 to Sa2 is the input of Sb1 to Sb2, the output of Sb1 to Sb2 is the input of Sc1 to Sc2, the output of Sc1 to Sc2 is the input of Sd1 to Sd2, and the output of Sd1 to Sd2 is the input of Se1 to Se2.

[0309] In other embodiments, the above-mentioned Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, Sd1 to Sd2, and Se1 to Se2 can be performed in any combination.

[0310] It is understood that based on the permission level of the image sub-region, the corresponding target filtering region can be selected for the above-mentioned filtering process. Furthermore, based on the filtering requirements of the image to be processed, the pixels on one side or both sides of the adjacent boundaries of the image sub-region can be filtered. Furthermore, different filtering methods can be combined and filtered in the order in which they are combined to improve the image quality of different permission regions in the image to be processed, thereby enhancing the user experience.

[0311] The present application also provides an electronic device, such as Figure 19 As shown, Figure 19 This is a schematic diagram of the structure of an electronic device provided in this application. The electronic device 12 includes a processor 201 and a communication interface 202. The processor 201 and the communication interface 202 are coupled to each other. It is understood that the communication interface 202 can be a transceiver or an input / output interface. Optionally, the electronic device 12 may also include a memory 203 for storing instructions executed by the processor 201, or storing input data required by the processor 201 to execute instructions, or storing data generated after the processor 201 executes instructions.

[0312] In some embodiments, the processor 201 and the communication interface 202 are used to perform the functions of the above-mentioned image division unit 101 , image filtering unit 102 , and image decoding unit 103 .

[0313] The specific connection medium between the communication interface 202, the processor 201 and the memory 203 is not limited in the embodiment of the present application. Figure 17 The communication interface 202, the processor 201 and the memory 203 are connected via a bus 204. Figure 19 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 19Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0314] The memory 203 can be used to store software programs and modules, such as program instructions / modules corresponding to the decoding method or encoding method provided in the embodiments of the present application. The processor 201 executes the software programs and modules stored in the memory 203 to perform various functional applications and data processing. The communication interface 202 can be used to communicate signaling or data with other devices. In this application, the electronic device 12 can have multiple communication interfaces 202.

[0315] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), a neural processing unit (NPU) or a graphics processing unit (GPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0316] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and storage medium can also exist as discrete components in a network device or a terminal device.

[0317] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0318] In the various embodiments of the present application, unless otherwise specified or logically conflicting, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. In the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of the present application, the character " / " indicates that the previous and next associated objects are in a "division" relationship.

[0319] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

Claims

1. An image processing method, characterized in that: The image processing method is performed by an electronic device and includes: Divide the image to be processed into multiple image sub-regions and obtain the permission level of each image sub-region; If the permission levels of at least two adjacent image sub-regions among the plurality of image sub-regions are different, performing filtering processing on a boundary between the at least two image sub-regions; The filtering process includes: sample adaptive compensation filtering and adaptive loop filtering.

2. The method according to claim 1, characterized in that The permission levels include a first permission level, a second permission level, and a zero permission level. The image sub-area at the zero permission level is an image sub-area that can be viewed by users of any permission level.

3. The method according to claim 1 or 2, characterized in that Before dividing the image to be processed into multiple image sub-regions and obtaining the permission level of each image sub-region, the following steps are included: A decoded code stream of the image to be processed is received, decoding is performed based on the decoded code stream, and a decoded image to be processed and decoding information of the image to be processed are obtained.

4. The method according to claim 1 or 2, characterized in that The filtering process further includes: at least one of: deblocking effect vertical filtering, deblocking effect horizontal filtering and neural network filtering.

5. The method according to claim 1, wherein In a case where the filtering process is the sample adaptive compensation filtering, the filtering process on the boundary between the at least two image sub-regions includes: Determining a target area to be filtered from a surrounding side of a first sub-region among the at least two image sub-regions according to a set filtering order; the surrounding side sequentially includes: a left side, an upper side, a right side, and a lower side of a boundary between the first sub-region and other sub-regions; the authority level of the first sub-region is different from the authority levels of the other sub-regions; and the target area to be filtered is any one of the following: a first side area within the first sub-region relative to the boundary; a second side area outside the first sub-region relative to the boundary; the first side area; and the second side area. According to the set filtering order, sample adaptive compensation filtering is performed on the target area to be filtered.

6. The method according to claim 1, wherein In a case where the filtering process is adaptive loop filtering, the filtering process on the boundary between the at least two image sub-regions includes: Determining a target area to be filtered from a surrounding side of a first sub-region among the at least two image sub-regions according to a set filtering order; the surrounding side sequentially includes: a left side, an upper side, a right side, and a lower side of a boundary between the first sub-region and other sub-regions; the authority level of the first sub-region is different from the authority levels of the other sub-regions; and the target area to be filtered is any one of the following: a first side area within the first sub-region relative to the boundary; a second side area outside the first sub-region relative to the boundary; the first side area; and the second side area. According to the set filtering order, adaptive loop filtering is performed on the target area to be filtered.

7. The method according to claim 1, characterized in that The electronic device obtains information of an image sub-region that is lower than or equal to the user's authority level according to the user's authority level; the information of the image sub-region includes pixel values ​​of the image sub-region.

8. The method according to claim 1, characterized in that If the permission levels of at least two adjacent image sub-regions among the plurality of image sub-regions are different, filtering the boundary between the at least two adjacent image sub-regions includes: performing filtering processing on a brightness channel component on a boundary between the at least two image sub-regions; Alternatively, filtering of the chrominance channel components is performed on the boundary between the at least two image sub-regions; Alternatively, filtering processing of the luminance channel component and the chrominance channel component is performed on the boundary between the at least two image sub-regions.

9. An image processing device, characterized in that: The image processing device includes: an image division unit, an image filtering unit, and an image decoding unit; The image division unit, the image filtering unit, and the image decoding unit are used to implement the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instruction, and when the computer program or instruction is executed by an electronic device, the method according to any one of claims 1 to 8 is implemented.

Images