Image processing method and apparatus, electronic device, and computer readable storage medium

Abstract

Disclosed in the present application are an image processing method and apparatus, an electronic device, and a computer readable storage medium. The method comprises: obtaining a target video frame and coding information of the target video frame, the coding information comprising a macro block position of a macro block in the target video frame; on the basis of the macro block position, determining a boundary position of the boundary of the macro block in the target video frame; and, on the basis of the boundary position, determining a block effect area from the target video frame, the block effect area being an area in which a block effect occurs.

Description

Image processing method and device, electronic device and computer-readable storage medium

[0001] This disclosure claims an application filed with the China Patent Office on March 14, 2023, with application number 202310241877.9, and the invention name is “Image processing method and device, electronic device and computer-readable storage medium”, all of which are incorporated by reference in this disclosure. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0003] Image compression may cause blocking artifacts, which obviously degrade image quality. Therefore, to improve image quality, it is necessary to eliminate blocking artifacts. Before eliminating blocking artifacts, it is necessary to identify the areas where blocking artifacts occur. Therefore, how to identify the areas where blocking artifacts occur is of great significance.

[0004] Summary of the Invention

[0005] The present application provides an image processing method and apparatus, an electronic device, and a computer-readable storage medium.

[0006] In a first aspect, an image processing method is provided, the method comprising:

[0007] Acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0008] Determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position;

[0009] A blocking effect area is determined from the target video frame based on the boundary position, where the blocking effect area is an area where blocking effect occurs.

[0010] In combination with any embodiment of the present application, determining the blocking effect area from the target video frame based on the boundary position includes:

[0011] determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0012] Constructing a pixel neighborhood of a preset size with the pixel to be confirmed in the target video frame as the center;

[0013] determining a target number of said boundary pixels within said pixel neighborhood;

[0014] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0015] The blocking area is determined according to the blocking pixels in the target video frame.

[0016] In combination with any embodiment of the present application, when the target number is greater than or equal to the first threshold, before determining that the pixel to be confirmed is a blocking pixel, the method further includes:

[0017] Determine an average gradient of the boundary pixels within the pixel neighborhood to obtain a first gradient;

[0018] Determine the average gradient of pixels in the pixel neighborhood excluding the boundary pixels to obtain a second gradient;

[0019] The step of determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to a first threshold includes:

[0020] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, the pixel to be confirmed is determined to be the blocking pixel, and the preset value is greater than 1.

[0021] In combination with any embodiment of the present application, determining the average gradient of the boundary pixels in the pixel neighborhood to obtain the first gradient includes:

[0022] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0023] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0024] In combination with any embodiment of the present application, determining the blocking effect area from the target video frame based on the boundary position includes:

[0025] Determine, in the target video frame, pixels whose distance from the boundary position is less than or equal to a second threshold as blocking pixels;

[0026] The blocking area is determined according to the blocking pixels in the target video frame.

[0027] In combination with any embodiment of the present application, determining that a pixel in the target video frame whose distance to the boundary position is less than or equal to a second threshold is a blocking pixel includes:

[0028] determining a flat area from the target video frame;

[0029] Pixels in the flat area whose distance from the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0030] In combination with any embodiment of the present application, after determining the blocking effect area from the target video frame based on the boundary position, the method further includes:

[0031] Smoothing the blocking area in the target video frame to obtain an enhanced video frame.

[0032] In combination with any embodiment of the present application, the step of smoothing the blockiness region in the target video frame to obtain an enhanced video frame includes:

[0033] A sharpening process is performed on the area except the blocking effect area in the target video frame, and a smoothing process is performed on the blocking effect area in the target video frame to obtain the enhanced video frame.

[0034] In a second aspect, an image processing apparatus is provided, the apparatus comprising:

[0035] an acquiring unit, configured to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0036] a determining unit, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position;

[0037] The determining unit is further configured to determine a blocking effect region from the target video frame based on the boundary position, where the blocking effect region is a region where blocking effects occur.

[0038] In combination with any embodiment of the present application, the determining unit is configured to:

[0039] determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0040] Constructing a pixel neighborhood of a preset size with the pixel to be confirmed in the target video frame as the center;

[0041] determining a target number of said boundary pixels within said pixel neighborhood;

[0042] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0043] The blocking area is determined according to the blocking pixels in the target video frame.

[0044] In combination with any embodiment of the present application, the determining unit is further configured to:

[0045] Determine an average gradient of the boundary pixels within the pixel neighborhood to obtain a first gradient;

[0046] Determine the average gradient of pixels in the pixel neighborhood excluding the boundary pixels to obtain a second gradient;

[0047] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, the pixel to be confirmed is determined to be the blocking pixel, and the preset value is greater than 1.

[0048] In combination with any embodiment of the present application, the determining unit is configured to:

[0049] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0050] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0051] In combination with any embodiment of the present application, the determining unit is configured to:

[0052] Determine, in the target video frame, pixels whose distance from the boundary position is less than or equal to a second threshold as blocking pixels;

[0053] The blocking area is determined according to the blocking pixels in the target video frame.

[0054] In combination with any embodiment of the present application, the determining unit is configured to:

[0055] determining a flat area from the target video frame;

[0056] Pixels in the flat area whose distance from the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0057] In combination with any embodiment of the present application, the device further includes:

[0058] The processing unit is configured to perform smoothing on the blocking area in the target video frame to obtain an enhanced video frame.

[0059] In combination with any embodiment of the present application, the processing unit is configured to:

[0060] A sharpening process is performed on the area except the blocking effect area in the target video frame, and a smoothing process is performed on the blocking effect area in the target video frame to obtain the enhanced video frame.

[0061] In a third aspect, an electronic device is provided, characterized in that it includes: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0062] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments described above.

[0063] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the first aspect and any of its embodiments described above.

[0064] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the above-mentioned first aspect and any embodiment thereof.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0066] In the present application, after obtaining the target video frame and the encoding information of the target video frame, the image processing device uses the macroblock positions in the encoding information to determine the boundary positions of the macroblocks in the target video frame. Furthermore, because blocking artifacts are often prone to occur at macroblock boundaries, and the closer the area to the macroblock boundary is, the higher the probability of the occurrence of blocking artifacts, the image processing device can determine the blocking artifact area from the target video frame based on the boundary position, thereby achieving the determination of the blocking artifact area in the target video frame based on the encoding information of the target video frame.

[0067] Since the encoding information of the target video frame is information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount for determining the blocking effect area in the target video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0069] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0070] FIG1 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0071] FIG2 is a schematic diagram of a target video frame provided in an embodiment of the present application;

[0072] FIG3 is a flow chart of another image processing method provided in an embodiment of the present application;

[0073] FIG4 is a schematic diagram of a boundary map E obtained by determining the boundary position in the target video frame shown in FIG2 , provided by an embodiment of the present application;

[0074] FIG5 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0075] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0078] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0079] Image compression may cause blocking effects to appear in the image. For example, encoding a video may cause blocking effects to appear in the video frames of the video. The appearance of blocking effects will obviously reduce the image quality. Therefore, in order to improve the image quality, the blocking effects in the image need to be eliminated. Before eliminating the blocking effects in the image, it is necessary to determine the area where the blocking effects appear (hereinafter referred to as the blocking effect area) from the image. Current technologies generally use specific blocking effect area detection algorithms to determine the blocking effect area from the image. This results in a large amount of data processing and low efficiency in determining the blocking effect area from the image. Based on this, the embodiments of the present application provide an image processing method to reduce the amount of data processing required to determine the blocking effect area from the image and improve efficiency.

[0080] The execution subject of the embodiments of the present application is an image processing device, wherein the image processing device can be any electronic device that can execute the technical solution disclosed in the method embodiments of the present application. Optionally, the image processing device can be one of the following: a mobile phone, a computer, a tablet computer, or a wearable smart device.

[0081] It should be understood that the method embodiments of the present application can also be implemented by a processor executing computer program code. The embodiments of the present application are described below in conjunction with the accompanying drawings. Please refer to Figure 1, which is a flow chart of an image processing method provided in the embodiments of the present application.

[0082] 101. Obtain a target video frame and encoding information of the target video frame.

[0083] In the embodiment of the present application, the target video frame is any frame in the encoded video obtained by encoding the video to be encoded. The target video frame is obtained by encoding the video frame to be encoded in the video to be encoded. For example, in the process of encoding video A to obtain video B, the first frame of video B is obtained by encoding the first frame of video A. In this case, video A is the video to be encoded, and video B is the encoded video. If the first frame of video A is the video frame to be encoded, then the first frame of video B is the target video frame.

[0084] In an embodiment of the present application, the encoding information of the target video includes information for encoding a video frame to be encoded, wherein the encoding information includes the macroblock position of the macroblock in the target video frame, and the macroblock position represents the position of the macroblock in the target video frame. A macroblock is a coding unit for video encoding. For example, a target video frame is obtained by encoding a video frame to be encoded. Then, in the process of encoding the video frame to be encoded to obtain the target video frame, the video frame to be encoded is divided into at least two macroblocks, and then each macroblock is encoded separately to obtain the target video frame. For example, FIG2 shows a target video frame, in which some macroblocks have been visualized. Specifically, there is an orange square on the back of the hand at the bottom of FIG2 , and the square includes multiple small squares, wherein each small square is a macroblock.

[0085] In one implementation of acquiring a target video frame, an image processing apparatus receives a target video frame input by a user through an input component to acquire the target video frame. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0086] In another implementation of obtaining a target video frame, an image processing device receives a target video frame sent by a terminal to obtain the target video frame. Optionally, the terminal can be any of the following: a mobile phone, a computer, a tablet computer, or a server. For example, the terminal is a server, and a communication connection exists between the server and the image processing device. The server transmits the encoded video to the image processing device via this communication connection, and the image processing device obtains the target video frame by receiving the encoded video.

[0087] In an implementation method of obtaining encoding information of a target video frame, an image processing apparatus receives a target video frame input by a user through an input component to obtain the target video frame.

[0088] In another implementation of obtaining encoding information for a target video frame, an image processing device receives a target video frame sent by a terminal and obtains the target video frame. For example, the terminal is a server, and a communication connection exists between the server and the image processing device. The server transmits the encoded video to the image processing device via the communication connection. In this case, the encoded video carries the encoding information of the encoded video, and the image processing device uses the encoding information of the encoded video as the encoding information of the target video frame.

[0089] 102. Determine a boundary position of a macroblock in the target video frame based on the macroblock position.

[0090] In the embodiment of the present application, the boundary position is the position of the macroblock boundary in the target video frame. The image processing device can determine the position of the macroblock boundary in the target video frame based on the macroblock position, thereby determining the boundary position.

[0091] 103. Determine a blocking effect area from the target video frame based on the boundary position.

[0092] In the embodiment of the present application, the blocking effect area is an area in the target video frame where blocking effects occur. For example, in the target video frame shown in FIG2 , blocking effects are present on the back of the hand. Since blocking effects are generally prone to occur at macroblock boundaries, and the closer the area is to the macroblock boundary, the higher the probability of occurrence of blocking effects, the image processing device can determine the blocking effect area from the target video frame based on the boundary position.

[0093] In an embodiment of the present application, after obtaining the target video frame and the encoding information of the target video frame, the image processing device uses the macroblock positions in the encoding information to determine the boundary positions of the macroblocks in the target video frame. Furthermore, because blocking artifacts are often prone to occur at macroblock boundaries, and the closer the area to the macroblock boundary is, the higher the probability of blockiness, the image processing device can determine the blocking artifact area from the target video frame based on the boundary position. This allows the blocking artifact area in the target video frame to be determined based on the encoding information of the target video frame.

[0094] Since the encoding information of the target video frame is information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount for determining the blocking effect area in the target video frame.

[0095] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 103:

[0096] 201. Determine boundary pixels located on the boundary of the macroblock based on the boundary position.

[0097] In the embodiment of the present application, pixels located on the boundary of a macroblock are referred to as boundary pixels. The image processing apparatus may use pixels located at the boundary of a target video frame as boundary pixels.

[0098] 202. Construct a pixel neighborhood with a preset size, with the pixel to be confirmed in the target video frame as the center.

[0099] In the embodiment of the present application, the pixel to be confirmed can be any pixel in the target video frame. The preset size is a pre-set size. Optionally, the preset size is N×N, where N is an odd number. For example, if the preset size is 5×5, the size of the pixel neighborhood is 5×5. For another example, if the preset size is 7×7, the size of the pixel neighborhood is 7×7. The center of the pixel neighborhood is the pixel to be confirmed, that is, the pixel to be confirmed is the geometric center of the pixel neighborhood.

[0100] 203. Determine a target number of the boundary pixels within the pixel neighborhood.

[0101] In the embodiment of the present application, the target number is the number of boundary pixels in the pixel neighborhood.

[0102] 204 . When the target number is greater than or equal to a first threshold, determine that the pixel to be confirmed is a blocking pixel.

[0103] Since the size of the pixel neighborhood is a preset size, the size of the pixel neighborhood is fixed, that is, the number of pixels within the pixel neighborhood is fixed. Therefore, a large number of boundary pixels within the pixel neighborhood indicates that the center of the pixel neighborhood is close to the macroblock boundary, which means that the probability of blocking artifacts occurring at the location of the unidentified pixel is high. Conversely, a small number of boundary pixels within the pixel neighborhood indicates that the center of the pixel neighborhood is far from the macroblock boundary, which means that the probability of blocking artifacts occurring at the location of the unidentified pixel is low.

[0104] In an embodiment of the present application, the image processing device determines whether the number of boundary pixels in a pixel neighborhood is large or small based on a first threshold value. Specifically, if the number of boundary pixels in a pixel neighborhood is greater than or equal to the first threshold value, it means that the number of boundaries in the pixel neighborhood is large; if the number of boundary pixels in a pixel neighborhood is less than the first threshold value, it means that the number of boundaries in the pixel neighborhood is small.

[0105] Therefore, when the target number is greater than or equal to the first threshold, the image processing device determines that the probability of blockiness occurring at the location of the pixel to be confirmed is high, and further determines that the pixel to be confirmed is a blocky pixel. Conversely, when the target number is less than the first threshold, the image processing device determines that the probability of blockiness occurring at the location of the pixel to be confirmed is low, and further determines that the pixel to be confirmed is not a blocky pixel. In this embodiment of the present application, a blocky pixel is a pixel with a high probability of blockiness occurring.

[0106] 205 : Determine the blocking effect area according to the blocking effect pixels in the target video frame.

[0107] In a possible implementation, the image processing apparatus uses a pixel region including blocking pixels as the blocking region. Alternatively, the image processing apparatus uses a pixel region composed of blocking pixels as the blocking region.

[0108] In this embodiment, the image processing device determines boundary pixels located on the macroblock boundary based on the boundary position, constructs a pixel neighborhood of a preset size centered on the pixel to be confirmed in the target video frame, and then determines a target number of boundary pixels within the pixel neighborhood. The device then determines whether the pixel to be confirmed is a blocking pixel based on the relationship between the target number and a first threshold. Specifically, if the target number is greater than or equal to the first threshold, the pixel to be confirmed is determined to be a blocking pixel. That is, the image processing device can determine whether each pixel in the target video frame is a blocking pixel based on steps 201 to 204. After determining all blocking pixels in the target video frame, the device can determine a blocking region based on the blocking pixels in the target video frame.

[0109] As an optional implementation manner, the image processing apparatus further performs the following steps before performing step 204:

[0110] 301. Determine an average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient.

[0111] In one possible implementation, the image processing device may determine the gradient of each boundary pixel in the pixel neighborhood respectively, and then determine the average value of the gradients of the boundary pixels in the pixel neighborhood to obtain the first gradient.

[0112] In another possible implementation, the image processing device determines the gradient sum of boundary pixels within a pixel neighborhood using a Sobel operator, and then determines the quotient of the gradient sum of the boundary pixels and a target number to obtain a first gradient.

[0113] 302. Determine the average gradient of pixels in the pixel neighborhood excluding the boundary pixels to obtain a second gradient.

[0114] For convenience of expression, pixels other than boundary pixels within a pixel neighborhood are referred to as non-boundary pixels below. In one possible implementation, the image processing device may separately determine the gradient of each non-boundary pixel within the pixel neighborhood, and then determine the average value of the gradients of the non-boundary pixels within the pixel neighborhood to obtain a second gradient.

[0115] In another possible implementation, the image processing device determines the gradient sum of non-boundary pixels in a pixel neighborhood using a Sobel operator, determines the number of non-boundary pixels in the pixel neighborhood, and then determines the quotient of the gradient sum of the non-boundary pixels and the number of non-boundary pixels to obtain a second gradient.

[0116] When the first gradient and the second gradient are obtained, the image processing apparatus performs the following steps during step 204:

[0117] 303 : When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, determine that the pixel to be confirmed is the blocking pixel.

[0118] The target video frame may contain regions with large gradients (hereinafter referred to as large gradient regions). However, if blocking artifacts occur in large gradient regions, the blocking artifacts have little impact on the display quality of the target video frame. Specifically, if blocking artifacts occur in large gradient regions, the human eye generally cannot perceive the presence of blocking artifacts. Therefore, rather than detecting blocking artifacts within large gradient regions, detection may be performed within regions other than large gradient regions (hereinafter referred to as flat regions). Subsequently, by removing blocking artifacts from the flat regions, the display quality of the target video frame is improved, i.e., the image quality of the target video frame is improved, thereby reducing the amount of data processing required to improve the image quality of the target video frame.

[0119] Because the gradient of non-boundary pixels in a large gradient region may be large, while the gradient of non-boundary pixels in a flat region is usually small, when the gradient of boundary pixels is large, the difference between the average gradient of boundary pixels in the pixel neighborhood and the average gradient of non-boundary pixels in the pixel neighborhood can be used as a basis to determine whether the pixel neighborhood is a large gradient region or a flat region. Specifically, if the difference between the average gradient of boundary pixels in the pixel neighborhood and the average gradient of non-boundary pixels in the pixel neighborhood is large, that is, the difference between the first gradient and the second gradient is large, it indicates that the pixel neighborhood is a flat region. If the difference between the average gradient of boundary pixels in the pixel neighborhood and the average gradient of non-boundary pixels in the pixel neighborhood is small, that is, the difference between the first gradient and the second gradient is small, it indicates that the pixel neighborhood is a large gradient region.

[0120] In an embodiment of the present application, the image processing device determines that the difference between the first gradient and the second gradient is large by determining whether the first gradient is greater than the product of the second gradient and a preset value, where the preset value is greater than 1. Specifically, the image processing device determines that the difference between the first gradient and the second gradient is large when the first gradient is greater than the product of the second gradient and the preset value, and determines that the difference between the first gradient and the second gradient is small when the first gradient is less than or equal to the product of the second gradient and the preset value. Thus, when the first gradient is greater than the product of the second gradient and the preset value, the image processing device determines that the pixel neighborhood is a flat area, i.e., the pixel to be confirmed is a pixel within the flat area, and thus can determine whether the pixel to be confirmed is a blocking pixel. When the first gradient is less than or equal to the product of the second gradient and the preset value, the image processing device determines that the pixel neighborhood is a large gradient area, i.e., the pixel to be confirmed is a pixel within the large gradient area, and thus can determine whether the pixel to be confirmed is a blocking pixel.

[0121] Therefore, the image processing apparatus determines that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to the first threshold and the first gradient is greater than the product of the second gradient and the preset value.

[0122] In this embodiment, before determining whether a pixel to be confirmed is a blocking pixel, the image processing device determines the average gradient of boundary pixels within the pixel's neighborhood to obtain a first gradient, and determines the average gradient of non-boundary pixels within the pixel's neighborhood to obtain a second gradient. Based on the difference between the first and second gradients, the device then determines whether the pixel's neighborhood is a region with a large gradient or a flat region. If the pixel's neighborhood is a flat region, the device determines that the pixel to be confirmed is a pixel within the flat region, thereby further determining whether the pixel to be confirmed is a blocking pixel. Therefore, if the number of targets in the image is greater than or equal to a first threshold and the first gradient is greater than the product of the second gradient and a preset value, the device determines that the pixel to be confirmed is a blocking pixel. This allows the blocking pixel to be a pixel within a flat region with a high probability of exhibiting blocking.

[0123] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 103:

[0124] 401 : Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels.

[0125] As described in step 103, blocking effects are generally more likely to occur at the boundaries of macroblocks, and the closer the area is to the macroblock boundary, the higher the probability of the blocking effect occurring. Therefore, the image processing apparatus may select pixels close to the macroblock boundary from the target video frame as blocking effect pixels. In other words, pixels close to the boundary in the target video frame may be used as blocking effect pixels.

[0126] In the embodiment of the present application, the image processing device determines whether the distance between a pixel in the target video frame and the boundary position is close or far based on a second threshold. Specifically, if the distance between a pixel in the target video frame and the boundary position is less than or equal to the second threshold, it indicates that the distance between the pixel and the boundary position is close, and the pixel can be determined to be a blocking pixel. Conversely, if the distance between the pixel in the target video frame and the boundary position is greater than the second threshold, it indicates that the distance between the pixel and the boundary position is far, and the pixel can be determined to be non-blocking pixel. Therefore, the image processing device determines that a pixel in the target video frame whose distance to the boundary position is less than or equal to the second threshold is a blocking pixel.

[0127] 402 : Determine the blocking effect area according to the blocking effect pixels in the target video frame.

[0128] The implementation of this step can refer to the implementation of step 205, which will not be repeated here.

[0129] In this embodiment, the image processing apparatus determines whether a pixel in the target video frame is a blocking pixel based on the magnitude relationship between the distance from the pixel in the target video frame to the boundary position and a second threshold. Specifically, if the distance from the pixel in the target video frame to the boundary position is less than or equal to the second threshold, the pixel in the target video frame is determined to be a blocking pixel. In other words, the image processing apparatus can determine whether each pixel in the target video frame is a blocking pixel based on step 401. After determining all blocking pixels in the target video frame, the image processing apparatus can determine a blocking area based on the blocking pixels in the target video frame.

[0130] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 401:

[0131] 501. Determine a flat area from the target video frame.

[0132] In the embodiment of the present application, a flat area is an area with a small gradient, that is, an area other than the large gradient area described above. In one possible implementation, the average gradient of pixels in the flat area is less than or equal to a third threshold. If the average gradient of pixels in the flat area is less than or equal to the third threshold, it means that the average gradient of pixels in the flat area is small, that is, the flat area is an area with a small gradient.

[0133] In another possible implementation, the flat region is a region in the target video frame where the grayscale variation is less than or equal to a fourth threshold. If the grayscale variation in the flat region is less than or equal to the fourth threshold, it indicates that the flat region is a region with a small gradient.

[0134] 502 : Determine pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold as the blocking pixel.

[0135] As described in step 303, when blocking artifacts occur in regions with large gradients, the human eye typically cannot detect the presence of blocking artifacts. Therefore, rather than detecting blocking artifacts within the large gradient regions, the system can detect blocking artifacts within regions other than the large gradient regions (hereinafter referred to as flat regions). This can subsequently improve the display quality of the target video frame by removing blocking artifacts within the flat regions, thereby improving the image quality of the target video frame and reducing the amount of data processing required to improve the image quality of the target video frame. Therefore, in step 502, the image processing device identifies blocking artifact pixels within the flat regions of the target video frame. Specifically, the image processing device identifies pixels within the flat regions whose distance to a boundary is less than or equal to a second threshold as blocking artifact pixels.

[0136] In this embodiment, before determining whether a pixel to be confirmed is a blocking pixel, the image processing device determines a flat area from the target video frame, and then determines whether the pixels in the flat area are blocking pixels. Specifically, pixels in the flat area whose distance to a boundary position is less than or equal to a second threshold are determined to be blocking pixels. In this way, blocking pixels can be pixels with a high probability of blocking occurring in the flat area.

[0137] As an optional implementation manner, after determining the blocking area from the target video frame based on the boundary position, the image processing apparatus further performs the following steps:

[0138] 601. Perform smoothing on the blocking area in the target video frame to obtain an enhanced video frame.

[0139] Because the blocking effect area in the target video frame may have blocking effects, thereby causing the image quality of the target video frame to deteriorate, the blocking effects in the blocking effect area can be eliminated by smoothing the blocking effect area in the target video frame. Therefore, the image processing device can eliminate the blocking effects in the blocking effect area by smoothing the blocking effect area in the target video frame, thereby improving the image quality of the target video frame.

[0140] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 601:

[0141] 701 : Perform sharpening processing on the area except the blocking effect area in the target video frame, and perform smoothing processing on the blocking effect area in the target video frame to obtain the enhanced video frame.

[0142] In step 701, the image processing apparatus performs a smoothing process on the blocking region in the target video frame while also performing a sharpening process on the region other than the blocking region in the target video frame. This can eliminate blocking artifacts in the target video frame while sharpening the edges in the target video frame, thereby improving the image quality of the target video frame while also sharpening the edges in the target video frame. This avoids the situation where the boundaries of the blocking region are also sharpened due to the sharpening process on all pixels in the target video frame.

[0143] In one possible implementation, the image processing device performs sharpening processing on the target video frame according to the following formula, so as to achieve smoothing of the block effect area in the target video frame and sharpening of the area other than the block effect area in the target video frame to obtain an enhanced video frame: pro =A+gain×(Af(A))…Formula (1)

[0144] Among them, A prorepresents the enhanced video frame, A represents the target video frame, f(A) represents the intermediate frequency signal and the low frequency signal in the target video frame, and optionally, f(·) represents Gaussian filtering, i.e., f(A) represents the intermediate frequency signal and the low frequency signal in the target video frame obtained by performing Gaussian filtering on the target video frame to remove the high frequency signal in the target video frame.

[0145] Gain represents gain. Optionally, for the block effect area in the target video frame, the gain value is -1. In this case, A pro =f(A), that is, in the process of sharpening the target video frame to obtain the enhanced video frame, the high-frequency signals in the blocking area of ​​the target video frame are removed, while the low-frequency and medium-frequency signals in the blocking area of ​​the target video frame are retained. In other words, the blocking area of ​​the target video frame is smoothed. For areas other than the blocking area of ​​the target video frame, the value of gain is a positive number.

[0146] Based on the technical solution provided in the embodiment of the present application, the embodiment of the present application also provides a possible application scenario. Please refer to Figure 3, which shows a flow chart of another image processing method provided in the embodiment of the present application.

[0147] As shown in Figure 3, after a bitstream is input to an image processing device, the device decodes the bitstream to obtain an encoded video, where the encoded video includes a decoded video frame A. Decoded video frame A is used as the target video frame. Based on the encoding information of the encoded video, encoding information of the target video frame can be obtained. Then, based on the macroblock positions in the encoding information of the target video frame, the possible locations of blocking artifacts in the target video frame can be obtained, that is, the locations of macroblock boundaries in the target video frame can be obtained.

[0148] Optionally, the image processing apparatus obtains a boundary map E by determining boundary positions in the target video frame. For example, the image shown in FIG4 is a boundary map E, which is obtained by determining boundary positions in the target video frame shown in FIG2 . In the boundary map E shown in FIG4 , black lines indicate locations in the target video frame where blocking artifacts may occur. Specifically, the black lines are macroblock boundaries. It should be understood that the boundaries of two adjacent macroblocks do not overlap, and therefore the black lines include two rows of pixels, or two columns of pixels, i.e., the black lines include the boundaries of two adjacent macroblocks.

[0149] Optionally, in the boundary map E, for any pixel p, E(p)=1 indicates that the pixel p is in the area covered by the black line, and E(p)=0 indicates that the pixel p is in the area not covered by the black line. Specifically, the pixel value in the boundary map E can be expressed by the following formula:

[0150] The image processing device then calculates the blocking area of ​​the target video frame based on the boundary positions in the boundary map E, thereby determining the blocking area in the target video frame (equivalent to step 103 described above) and obtaining a blocking map B. The blocking map B includes position information of the blocking area in the target video frame.

[0151] Optionally, when the pixel values ​​in the boundary map E are represented by formula (2), and the image processing apparatus determines the blocking area in the target video frame by executing steps 201, 202, 203, 301, 302, and 303, the image processing apparatus may determine whether the pixel to be confirmed in the target video frame is a blocking pixel by using the following formula:

[0152] Among them, sum1 represents the target number, optional, Where N×N represents the preset size, E(q i )=1 represents the boundary pixel, E(q i )=0 indicates non-boundary pixels. grad i represents the gradient of the i-th pixel in the pixel neighborhood constructed with the pixel to be confirmed as the center, represents the gradient sum of the boundary pixels in the neighborhood of this pixel, represents the gradient sum of non-boundary pixels in the neighborhood of the pixel, and α is the preset value in step 303.

[0153] Finally, after obtaining the blocking image B, the image processing device performs sharpening calculation on the target video frame according to the blocking image B (the implementation process of the sharpening calculation can be seen in step 701 described above) to obtain an enhanced video frame (i.e., the enhanced video frame A_pro in Figure 3).

[0154] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0155] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0156] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0157] Please refer to FIG5 , which is a schematic structural diagram of an image processing device 1 provided in an embodiment of the present application. The image processing device 1 includes an acquisition unit 11 and a determination unit 12 . Optionally, the image processing device 1 further includes a processing unit 13 . Specifically:

[0158] An acquiring unit 11 is configured to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0159] A determining unit 12, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position;

[0160] The determining unit 12 is further configured to determine a blocking effect region from the target video frame based on the boundary position, where the blocking effect region is a region where blocking effects occur.

[0161] In combination with any embodiment of the present application, the determining unit 12 is configured to:

[0162] determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0163] Constructing a pixel neighborhood of a preset size with the pixel to be confirmed in the target video frame as the center;

[0164] determining a target number of said boundary pixels within said pixel neighborhood;

[0165] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0166] The blocking area is determined according to the blocking pixels in the target video frame.

[0167] In combination with any embodiment of the present application, the determining unit 12 is further configured to:

[0168] Determine an average gradient of the boundary pixels within the pixel neighborhood to obtain a first gradient;

[0169] Determine the average gradient of pixels in the pixel neighborhood excluding the boundary pixels to obtain a second gradient;

[0170] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, the pixel to be confirmed is determined to be the blocking pixel, and the preset value is greater than 1.

[0171] In combination with any embodiment of the present application, the determining unit 12 is configured to:

[0172] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0173] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0174] In combination with any embodiment of the present application, the determining unit 12 is configured to:

[0175] Determine, in the target video frame, pixels whose distance from the boundary position is less than or equal to a second threshold as blocking pixels;

[0176] The blocking area is determined according to the blocking pixels in the target video frame.

[0177] In combination with any embodiment of the present application, the determining unit 12 is configured to:

[0178] determining a flat area from the target video frame;

[0179] Pixels in the flat area whose distance from the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0180] In combination with any embodiment of the present application, the device further includes:

[0181] The processing unit 13 is configured to perform smoothing on the blocking area in the target video frame to obtain an enhanced video frame.

[0182] In combination with any embodiment of the present application, the processing unit 13 is configured to:

[0183] A sharpening process is performed on the area except the blocking effect area in the target video frame, and a smoothing process is performed on the blocking effect area in the target video frame to obtain the enhanced video frame.

[0184] In an embodiment of the present application, after obtaining the target video frame and the encoding information of the target video frame, the image processing device uses the macroblock positions in the encoding information to determine the boundary positions of the macroblocks in the target video frame. Furthermore, because blocking artifacts are often prone to occur at macroblock boundaries, and the closer the area to the macroblock boundary is, the higher the probability of blockiness, the image processing device can determine the blocking artifact area from the target video frame based on the boundary position. This allows the blocking artifact area in the target video frame to be determined based on the encoding information of the target video frame.

[0185] Since the encoding information of the target video frame is information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount for determining the blocking effect area in the target video frame.

[0186] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0187] Figure 6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, through various interfaces, transmission lines, buses, etc.

[0188] The processor 21 may include one or more processors, for example, one or more central processing units (CPUs). In the case where the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Alternatively, the processor 21 may be a processor group consisting of multiple CPUs, wherein the multiple processors are coupled to each other via one or more buses. Alternatively, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0189] The memory 22 can be used to store computer program instructions and various computer program codes, including the program code for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0190] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0191] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0192] It is understood that FIG6 merely illustrates a simplified design of an electronic device. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the scope of protection of the present application.

[0193] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0194] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of this application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0198] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0199] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by a computer program instructing related hardware to perform the processes. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame; Determine the boundary position of the boundary of the macroblock in the target video frame based on the macroblock position; A blocking effect area is determined from the target video frame based on the boundary position, where the blocking effect area is an area where the blocking effect occurs.

2. The method according to claim 1, characterized in that The determining the blocking effect area from the target video frame based on the boundary position includes: Determining boundary pixels located on the boundary of the macroblock based on the boundary position; Constructing a pixel neighborhood with a preset size with the pixel to be confirmed in the target video frame as the center; determining a target number of said boundary pixels within said pixel neighborhood; When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel; The blocking effect area is determined according to the blocking effect pixels in the target video frame.

3. The method according to claim 2, characterized in that Before determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to the first threshold, the method further includes: Determine the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient; Determine the average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient; The step of determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to a first threshold comprises: When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, it is determined that the pixel to be confirmed is the blocking pixel, and the preset value is greater than 1.

4. The method according to claim 3, characterized in that The determining the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient includes: Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator; The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

5. The method according to claim 1, characterized in that The determining the blocking effect area from the target video frame based on the boundary position includes: Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels; The blocking effect area is determined according to the blocking effect pixels in the target video frame.

6. The method according to claim 5, characterized in that The step of determining that a pixel in the target video frame whose distance to the boundary position is less than or equal to a second threshold is a blocking pixel includes: determining a flat area from the target video frame; The pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

7. The method according to any one of claims 1 to 6, characterized in that After determining the blockiness region from the target video frame based on the boundary position, the method further includes: The block effect area in the target video frame is smoothed to obtain an enhanced video frame.

8. The method according to claim 7, characterized in that The step of smoothing the block effect area in the target video frame to obtain an enhanced video frame includes: The area other than the block effect area in the target video frame is sharpened, and the block effect area in the target video frame is smoothed to obtain the enhanced video frame.

9. An image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame; A determination unit, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position; The determining unit is further configured to determine a blocking effect region from the target video frame based on the boundary position, wherein the blocking effect region is a region where blocking effects occur.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 8.

Images