Image Processing Method, Apparatus, Device, and Storage Medium

Abstract

The present disclosure relates to an image processing method, apparatus, device, and storage medium. The image processing method includes: obtaining a first chrominance component value, where the first chrominance component value refers to a value obtained by upsampling the chrominance component of a pixel point in a video frame; performing a predetermined optimization process on the first chrominance component to obtain a second chrominance component value; performing edge intensity detection on the pixel point to obtain an edge intensity value of the luminance component and an edge intensity value of the chrominance component of the pixel point; and determining the value of the chrominance component of the pixel point as the second chrominance component value when the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition.

Description

Technical Field

[0001] The present disclosure relates to the field of video technology, and more particularly, to an image processing method, apparatus, device, and storage medium. Background Art

[0002] In the current video coding and decoding framework, most encoders convert the RGB image information collected at the shooting end to the YUV color gamut for encoding. In the YUV color gamut, compared with the luminance component Y, the chrominance components UV carry a relatively high degree of information redundancy. Therefore, in order to save bandwidth, most encoders downsample the UV components and then send them to the encoder for encoding. Subsequently, at the decoding end, the decoded YUV image needs to upsample the chrominance components to ensure that the three YUV components have the same size dimension for RGB conversion and display at the terminal.

[0003] Generally, the chrominance upsampling at the terminal is often completed by simple nearest neighbor interpolation algorithm or bilinear interpolation algorithm. These two algorithms are mature and simple, and can handle most natural scenes, which is a cost-effective solution. However, the images obtained by low-pass filtering upsampling methods such as the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm will have various problems such as color bleeding and missing, affecting the final image quality and not meeting the requirements of scenes with high color processing requirements. Summary of the Invention

[0004] The present disclosure provides an image processing method, apparatus, device, and storage medium to at least solve the problems in the above related technologies, or may not solve any of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including: obtaining a first chrominance component value, where the first chrominance component value is a value obtained by upsampling the chrominance component of a pixel point in a video frame; performing a predetermined optimization process on the first chrominance component value to obtain a second chrominance component value; performing edge intensity detection on the pixel point to obtain an edge intensity value of the luminance component and an edge intensity value of the chrominance component of the pixel point; and determining the value of the chrominance component of the pixel point as the second chrominance component value when the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition.

[0006] Optionally, the image processing method may further include: when the edge intensity value of the luminance component does not meet the first predetermined condition, and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not meet the second predetermined condition, determining the value of the chrominance component of the pixel as the first chrominance component value. Optionally, the first predetermined condition may be that the edge intensity value of the luminance component is equal to or greater than a first predetermined threshold; the second predetermined condition may be that the edge intensity value of the chrominance component is not higher than a second predetermined threshold of the edge intensity value of the luminance component.

[0007] Optionally, the predetermined optimization process may include linear prediction processing and guided filtering processing.

[0008] Optionally, the performing a predetermined optimization process on the first chrominance component to obtain a second chrominance component value may include: performing linear prediction processing on the first chrominance component to obtain a first chrominance component after linear prediction processing; using the value of the luminance component to perform guided filtering processing on the first chrominance component after linear prediction processing to obtain the second chrominance component.

[0009] Optionally, the performing linear prediction processing on the first chrominance component to obtain a first chrominance component after linear prediction processing may include: obtaining the values of the luminance components and the upsampled values of the chrominance components of neighboring pixels within a predetermined neighborhood of the pixel in the video frame; establishing a linear model based on the values of the luminance components and the upsampled values of the chrominance components of the neighboring pixels, where the linear model is used to characterize the linear relationship between the value of the luminance component and the value of the chrominance component; using the linear model and the value of the luminance component of the pixel to obtain the first chrominance component after linear prediction processing.

[0010] Optionally, the chrominance component may include a U component and a V component; where, when the edge intensity value of the luminance component meets the first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component meets the second predetermined condition, determining the value of the chrominance component of the pixel as the second chrominance component value may include: when the edge intensity value of the luminance component meets the first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the luminance component meets the second predetermined condition, determining the value of the U component of the pixel as the second chrominance component value of the U component; when the edge intensity value of the luminance component meets the first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the luminance component meets the second predetermined condition, determining the value of the V component of the pixel as the second chrominance component value of the V component.

[0011] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including: an acquisition unit configured to acquire a first chrominance component value, where the first chrominance component value refers to a value obtained by upsampling the chrominance component of a pixel point in a video frame; an optimization processing unit configured to perform a predetermined optimization process on the first chrominance component value to obtain a second chrominance component value; an edge intensity detection unit configured to perform edge intensity detection on the pixel point to obtain an edge intensity value of the luminance component and an edge intensity value of the chrominance component of the pixel point; and a determination unit configured to determine the chrominance component value of the pixel point as the second chrominance component value when the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition.

[0012] Optionally, the determination unit may be configured to determine the chrominance component value of the pixel point as the first chrominance component value when the edge intensity value of the luminance component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not satisfy the second predetermined condition.

[0013] Optionally, the first predetermined condition may be that the edge intensity value of the luminance component is equal to or greater than a first predetermined threshold; the second predetermined condition may be that the edge intensity value of the chrominance component is not higher than a second predetermined threshold of the edge intensity value of the luminance component.

[0014] Optionally, the predetermined optimization process may include linear prediction processing and guided filtering processing.

[0015] Optionally, the optimization processing unit may be configured to perform linear prediction processing on the first chrominance component to obtain a first chrominance component after linear prediction processing; and perform guided filtering processing on the first chrominance component after linear prediction processing by using the value of the luminance component to obtain the second chrominance component.

[0016] Optionally, the optimization processing unit may be configured to acquire the values of the luminance components and the upsampled values of the chrominance components of adjacent pixel points within a predetermined neighborhood of the pixel point; establish a linear model based on the values of the luminance components and the upsampled values of the chrominance components of the adjacent pixel points, where the linear model is used to characterize the linear relationship between the value of the luminance component and the value of the chrominance component; and obtain the first chrominance component after linear prediction processing by using the linear model and the value of the luminance component of the pixel point.

[0017] Optionally, the chrominance component may include a U component and a V component; wherein, the determining unit is configured to: when the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the luminance component satisfies the second predetermined condition, determine the value of the U component of the pixel as the second chrominance component value of the U component; when the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the luminance component satisfies the second predetermined condition, determine the value of the V component of the pixel as the second chrominance component value of the V component.

[0018] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: at least one processor; at least one memory storing computer-executable instructions, wherein, when the computer-executable instructions are run by the at least one processor, the at least one processor is caused to execute the image processing method according to the present disclosure.

[0019] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, when the instructions in the computer-readable storage medium are run by at least one processor, the at least one processor is caused to execute the image processing method according to the present disclosure.

[0020] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including computer instructions, which when executed by at least one processor implement the image processing method according to the present disclosure.

[0021] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0022] According to the image processing method and the image processing device of the present disclosure, the chrominance upsampling method can be adaptively adjusted based on the edge intensity information of the luminance component and the chrominance component of the pixel points in the video image, so that the original chrominance component value can be selected to be retained or the original chrominance component value can be optimized (for example, linear prediction processing, guided filtering processing, etc.) according to the edge intensity information, thereby improving the effect of optimizing the chrominance component value, effectively solving the problems of color bleeding between color blocks and increased noise in the processed image existing in the prior art, and being able to achieve the beneficial effects of more obvious improvement in the color area missing situation and less color bleeding phenomenon (for example, the text effect is more plump, the color boundary in the color block junction area is clearer, and the text jaggedness is reduced, etc.), and being able to meet the scenarios with high requirements for color processing such as video conferencing and game live streaming.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0025] Figure 1 is a schematic diagram showing the conversion process between RGB and YUV 4:2:0.

[0026] Figure 2 is a schematic diagram showing various sampling methods.

[0027] Figure 3 is a flowchart showing an image processing method according to an exemplary embodiment of the present disclosure.

[0028] Figure 4 is a schematic diagram showing linear prediction processing.

[0029] Figure 5 is a flowchart showing adaptive upsampling based on edge intensity according to an exemplary embodiment of the present disclosure.

[0030] Figure 6 is a schematic diagram showing the process of a chrominance upsampling method according to an exemplary embodiment of the present disclosure.

[0031] Figure 7 is a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0032] Figure 8 is a block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure. Detailed Embodiments

[0033] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order different from those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] It should be noted here that "at least one of several items" as used in this disclosure all represent three parallel situations, namely "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example, "performing at least one of step one and step two" means the following three parallel situations: (1) performing step one; (2) performing step two; (3) performing step one and step two.

[0036] In the existing video coding and decoding framework, most encoders will convert the RGB image information collected at the shooting end to the YUV color gamut for encoding. To save bandwidth, most encoders will downsample the UV components and then send them to the encoder for encoding. For example, according to different downsampling ratios, mainstream encoders usually support two chroma downsampling methods: YUV4:2:0 and YUV4:2:2. Subsequently, at the consumer end, the decoded YUV image needs to upsample the chroma components to ensure that the three YUV components have the same size dimension for RGB conversion and then display at the terminal.

[0037] Compared with YUV4:2:2, YUV4:2:0 can better balance the bitrate and quality. Therefore, at the production end, the YUV4:2:0 downsampling method is more widely adopted. Figure 1 is a schematic diagram showing the conversion process between RGB and YUV 4:2:0. Refer to Figure 1 , first, at the encoding end, the image data in the full RGB chroma format is subjected to chroma space conversion to obtain image data in the full YCbCr 4:4:4 chroma format. Subsequently, the image data in the full YCbCr 4:4:4 chroma format can be subjected to chroma downsampling to obtain image data in the YCbCr 4:2:0 chroma format. Subsequently, at the consumer end, the decoded YCbCr 4:2:0 chroma format image data is subjected to chroma upsampling to obtain image data in the full YCbCr 4:4:4 chroma format, and then chroma space conversion is performed to obtain the reconstructed full RGB chroma format image data.

[0038] Figure 2 is a schematic diagram showing various sampling methods. Refer to Figure 2 , the black dots represent the luminance components, and the circles represent the chroma components. In the YUV4:4:4 chroma format, the sizes of the three YUV components are the same. In the YUV4:2:2 chroma format, assuming the image size of the Y component is W×H, the sizes of the chroma components U and V are respectively W / 2×H. In the YUV4:2:0 chroma format, the sizes of the chroma components U and V are reduced to half of the luminance component Y, that is, W / 2×H / 2.

[0039] The chrominance upsampling of a general terminal is usually completed by a simple nearest neighbor interpolation algorithm or a bilinear interpolation algorithm. These two algorithms are mature and simple, and can handle most natural scenes, which is a solution with relatively high cost performance. However, with the increasing popularity of video conferencing and the emergence of game live broadcast scenarios, the method of obtaining complete chrominance components by using nearest neighbor upsampling or bilinear upsampling can no longer meet the requirements of the scenarios. In video conferencing scenarios and game scenarios, the color components are more abundant, the color edges of the content are more sharp, and the high-frequency components are more abundant. The images obtained by using low-pass filtering upsampling methods such as nearest neighbor or bilinear will have various problems such as color bleeding and missing, affecting the final image quality.

[0040] Since the edge information of the encoded luminance component is usually much better than that of the chrominance component, and there is often a high correlation between the chrominance component and the luminance component. A commonly used current solution is to use the gradient information of the luminance component to perform guided filtering on the chrominance component to improve the edge clarity of the chrominance component. However, the processed result of this method is prone to virtual shadows at the edges of text, and the problem of color erosion at the junction of large color blocks cannot be well solved. And affected by the clarity of the luminance component itself, if the edge information of the luminance component in this area is severely damaged, the chrominance component in this area will be further weakened and is easily weakened.

[0041] In addition, another method idea using the correlation between the luminance component and the chrominance component is as follows: Assume that the values between the local luminance component and the chrominance component are linearly related. For example, U = A1×Y + B1, V = A2×Y + B2. Then, the relevant linear parameters A1, A2, B1, and B2 can be fitted by using the YUV pixel values in the local area, and the chrominance components U and V of the chrominance component at the corresponding position can be predicted by using the parameters A1, A2, B1, and B2 and the Y component, so as to achieve the purpose of improving the chrominance edge clarity and distinctiveness. Although this method can effectively solve the problem of mutual erosion between large color blocks. However, since the parameters A1, A2, B1, and B2 are local estimated values, noise errors are easily introduced, resulting in noise points, so it often gives people a feeling that the picture is relatively "dirty".

[0042] To solve the above technical problems, the present disclosure proposes an image processing method and an image processing apparatus. Specifically, based on the edge intensity information of the luminance component and the chrominance component of the pixel points in the video image, the chrominance upsampling method can be adaptively adjusted, so that the original chrominance component value can be selected to be retained or the original chrominance component value can be optimized (for example, linear prediction processing, guided filtering processing, etc.) according to the edge intensity information, thereby improving the effect of optimizing the chrominance component value and effectively solving the problems such as color bleeding between color blocks and increased noise in the processed image in the existing problems. It can achieve the beneficial effects of more obvious improvement in the color area missing situation and more reduction in the color bleeding phenomenon (for example, the text effect is more plump, the color boundary in the color block junction area is clearer, and the text jaggedness is reduced, etc.), and can meet the scenarios with high requirements for color processing such as video conferencing and game live streaming. The following will refer to Figures 3 to 8 to describe in detail the image processing method and the image processing apparatus according to the present disclosure.

[0043] Figure 3 is a flowchart showing an image processing method according to an exemplary embodiment of the present disclosure.

[0044] Referring to Figure 3 , in step 301, a first chrominance component value can be obtained, where the first chrominance component value refers to the value obtained after upsampling the chrominance component of the pixel points in the video frame. Here, the video frame can be the video data obtained by the decoding end after decoding the encoded video data received from the encoding end. Since the chrominance component is downsampled during encoding, during decoding, it is necessary to upsample the chrominance component to ensure that the chrominance component has the same size dimension as the luminance component. For example, existing upsampling algorithms such as the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm can be used to upsample the chrominance component. Therefore, the first chrominance component value refers to the value obtained after upsampling the chrominance component of the pixel points in the video frame.

[0045] In step 302, a predetermined optimization process can be performed on the first chrominance component value to obtain a second chrominance component value.

[0046] According to an exemplary embodiment of the present disclosure, the predetermined optimization process can include at least one of linear prediction processing, guided filtering processing, edge adaptive interpolation processing, etc. Of course, the predetermined optimization process of the present disclosure is not limited to the above processes, and can also be any chrominance optimization process that can improve the reconstructed image quality. When the predetermined optimization process includes multiple processes, the present disclosure does not limit the order of the multiple processes.

[0047] According to an exemplary embodiment of the present disclosure, the predetermined optimization process may include first performing linear prediction processing and then performing guided filtering processing. Specifically, linear prediction processing may be performed on the first chrominance component to obtain the first chrominance component after linear prediction processing, and then, using the value of the luminance component, guided filtering processing may be performed on the first chrominance component after linear prediction processing to obtain the second chrominance component. Here, the linear prediction processing can preferably solve the chrominance erosion problem. The guided filtering processing can remove some noise points that appear in the result of the linear prediction processing, making the picture cleaner and tidier on the basis of ensuring the sharpness of the chrominance component. Therefore, performing linear prediction processing first and then guided filtering processing can achieve a better optimization effect. Of course, the present disclosure is not limited to the above processing sequence, and guided filtering processing may also be performed first and then linear prediction processing. In addition, the linear prediction processing is performed on the chrominance components of the pixel points of the YUV444 video frame obtained by initial upsampling, and only one round of parameter prediction is performed, which improves the parallelism of the algorithm.

[0048] Regarding the linear prediction processing, for each pixel point, using the hidden linear relationship between the luminance component and the chrominance component, based on the luminance-chrominance component pairs within the neighborhood of each chrominance component, the linear relationship parameters between each chrominance component and its corresponding luminance component may be estimated. Using the estimated linear relationship parameters, the corresponding value of each chrominance component is predicted as the chrominance upsampling result. For example, the linear parameters between the chrominance component and the luminance component are as follows:

[0049] U = A1×Y + B1

[0050] V = A2×Y + B2

[0051] Where Y represents the luminance component Y, U represents the chrominance component U, V represents the chrominance component V, and A1, A2, B1, and B2 represent linear parameters.

[0052] According to an exemplary embodiment of the present disclosure, the values of the luminance components and the upsampled values of the chrominance components of the neighboring pixel points within the predetermined neighborhood of the pixel point in the video frame may be obtained; based on the values of the luminance components and the upsampled values of the chrominance components of the neighboring pixel points, a linear model is established, where the linear model is used to characterize the linear relationship between the value of the luminance component and the value of the chrominance component; using the linear model and the value of the luminance component of the pixel point, the first chrominance component after linear prediction processing is obtained. Below, reference will be made to Figure 4 to specifically describe the process of the linear prediction processing.

[0053] Figure 4 is a schematic diagram showing the linear prediction processing. Although Figure 4Only the linear prediction processing procedure for the chrominance component U(x, y) is shown, but this procedure is equally applicable to the chrominance component V(x, y), where (x, y) represents the pixel coordinate value.

[0054] Referring to Figure 4 , in step 401, pairs of Y and U components within an N×N neighborhood can be obtained. Here, N can be 3, 5, 7, etc. For example, when N is 3, for the pixel point (x, y), the values of the luminance components and the chrominance components of 8 neighboring pixel points within the 3×3 neighborhood centered on it can be obtained (at this time, the values of the chrominance components are the chrominance components after upsampling).

[0055] In step 402, the values of parameters a and b in U = a×Y + b can be solved based on the obtained pairs of Y and U components. That is to say, the values of the luminance components and the chrominance components of the obtained neighboring pixel points can be substituted into the above formula to estimate the values of parameters a and b.

[0056] In step 403, the value of the chrominance component U can be updated based on U1(x, y) = a×Y(x, y) + b. That is to say, based on the estimated values of parameters a and b, this linear model can be obtained, and the value of the luminance component Y(x, y) of the pixel point (x, y) can be substituted into this linear model to obtain the updated value U1(x, y) of the chrominance component U.

[0057] Returning to the reference Figure 3 , in step 303, edge intensity detection is performed on the pixel point to obtain the edge intensity value of the luminance component and the edge intensity value of the chrominance component of the pixel point. Here, the method of edge intensity detection in the present disclosure is not limited, and any feasible edge intensity detection method can be used to perform step 303.

[0058] After obtaining the edge intensity value of the luminance component and the edge intensity value of the chrominance component of the pixel point, based on the edge intensity value of the luminance component and the comparison result between the edge intensity value of the luminance component and the edge intensity value of the chrominance component, the value of the chrominance component of the pixel point can be determined as the first chrominance component value or the second chrominance component value. Here, in order to solve the situation where the edge information of some luminance components is weaker than that of the chrominance components in the above-mentioned predetermined optimization processing, the gradient intensity of each pixel point in the video frame can be detected (that is, the edge intensity detection in step 303), and whether to retain the original chrominance component value (that is, the value obtained after upsampling, that is, the first chrominance component value) can be determined according to the comparison of the edge intensity values of the luminance component and the chrominance component.

[0059] Specifically, in step 304, when the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies the second predetermined condition, the value of the chrominance component of the pixel can be determined as the second chrominance component value. On the other hand, when the edge intensity value of the luminance component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not satisfy the second predetermined condition, the value of the chrominance component of the pixel can be determined as the first chrominance component value.

[0060] According to an exemplary embodiment of the present disclosure, the first predetermined condition may be that the edge intensity value of the luminance component is equal to or greater than the first predetermined threshold. The second predetermined condition may be that the edge intensity value of the chrominance component is not higher than the second predetermined threshold of the edge intensity value of the luminance component. Here, the first predetermined threshold and the second predetermined threshold can be set in advance according to needs, experience, or experiments. In addition, the present disclosure is not limited to the above comparison method, and it may also be that when the edge intensity value of the luminance component is lower than the first predetermined threshold and the edge intensity value of the chrominance component is higher than the edge intensity value of the luminance component, the value of the chrominance component of the pixel is determined as the first chrominance component value; and so on. Next, the processing method of adaptive upsampling based on edge intensity will be described with reference to Figure 5 in detail.

[0061] In addition, it should be noted that the present disclosure does not limit the execution order of step 303 and steps 301 and 302. For example, steps 301 and 302 can be executed first, and then step 303 can be executed, or step 301 can be executed first, then step 303, and then step 302, or step 303 can be executed first, and then steps 301 and 302, or step 303 can be executed in parallel with steps 301 and 302, etc. For example, the step of performing edge intensity detection on the pixel can be executed first. When the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies the second predetermined condition, the step of performing a predetermined optimization process on the first chrominance component value is then executed, so as to determine the value of the chrominance component of the pixel as the second chrominance component value.

[0062] Figure 5 is a flowchart showing adaptive upsampling based on edge intensity according to an exemplary embodiment of the present disclosure. Although Figure 5 only shows the process of adaptive upsampling based on edge intensity for the chrominance component U, this process is equally applicable to the chrominance component V.

[0063] Referring to Figure 5 , in step 501, edge intensity detection can be performed on the luminance component Y and the chrominance component U of the pixel to obtain the edge intensity value Gy of the luminance component Y and the edge intensity value Gu of the chrominance component U.

[0064] In step 502, it can be determined whether the edge intensity value Gy of the luminance component Y is lower than a first predetermined threshold C, and whether the edge intensity value Gu of the chrominance component U is higher than a second predetermined threshold D which is the edge intensity value Gy of the luminance component Y.

[0065] In step 503, when it is determined that the edge intensity value Gy of the luminance component Y is lower than the first predetermined threshold C and the edge intensity value Gu of the chrominance component U is higher than the second predetermined threshold D which is the edge intensity value Gy of the luminance component Y, the U value obtained by nearest neighbor upsampling is output, that is, the first chrominance component value of the chrominance component U. Of course, in addition to nearest neighbor upsampling, other feasible upsampling methods can also be used, such as bilinear interpolation upsampling, etc.

[0066] In step 504, otherwise, the U value obtained through linear prediction and guided filtering is output, that is, the second chrominance component value of the chrominance component U. Of course, in addition to linear prediction and guided filtering, other feasible optimization processing schemes can also be used for the predetermined optimization processing.

[0067] Return reference Figure 1 , according to an exemplary embodiment of the present disclosure, the chrominance component may include a U component and a V component. The same above-mentioned processing method can be respectively adopted for the U component and the V component to finally determine the values of the U component and the V component of the pixel point. For example, the first chrominance component value of the U component and the first chrominance component value of the V component can be respectively obtained; the first chrominance component value of the U component and the first chrominance component value of the V component are respectively subjected to a predetermined optimization process to obtain the second chrominance component value of the U component and the second chrominance component value of the V component; three-channel edge intensity detection is performed on the pixel point to obtain the edge intensity value of the Y component, the edge intensity value of the U component, and the edge intensity value of the V component of the pixel point; when the edge intensity value of the Y component satisfies a first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the Y component satisfies a second predetermined condition, the value of the U component of the pixel point is determined to be the second chrominance component value of the U component. In addition, when the edge intensity value of the Y component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the U component and the edge intensity value of the Y component does not satisfy the second predetermined condition, the value of the U component of the pixel point is determined to be the first chrominance component value of the U component; when the edge intensity value of the Y component satisfies a first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the Y component satisfies a second predetermined condition, the value of the V component of the pixel point is determined to be the second chrominance component value of the V component. In addition, when the edge intensity value of the Y component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the V component and the edge intensity value of the Y component does not satisfy the second predetermined condition, the value of the V component of the pixel point is determined to be the first chrominance component value of the V component.

[0068] Next, specific embodiments of the image processing method according to the present disclosure will be described in detail with reference to Figure 6

[0069] Figure 6 FIG. is a schematic diagram showing the process of a chrominance upsampling method according to an exemplary embodiment of the present disclosure.

[0070] With reference to Figure 6 , in step 601, the chrominance components of the pixel points in the input video frame in YUV420p format can be upsampled by nearest neighbor interpolation to obtain a video frame in YUV444 format. Here, the format of the input video frame is not limited to YUV420p, and may also be YUV420i, YUV422p, YUV422i, etc. In addition, the upsampling method is not limited to nearest neighbor interpolation, and may also be bilinear upsampling, etc.

[0071] In step 602, linear prediction can be performed on the chrominance components of the pixel points in the YUV444 format video frame.

[0072] In step 603, guided filtering can be performed on the chrominance components obtained by linear prediction.

[0073] In step 604, three-channel edge intensity detection can be performed on the pixel points in the YUV444 format video frame. Here, step 604 can be executed in parallel with steps 602 and 603.

[0074] In step 605, the U value and V value of the chrominance component can be adaptively determined according to the comparison result of the edge intensities of the luminance component and the chrominance component. Specifically, when the edge intensity value of the Y component is lower than the first predetermined threshold and the edge intensity value of the U component is higher than the second predetermined threshold of the edge intensity value of the Y component, the value of the U component is determined as the U component value output in step 601; otherwise, the value of the U component is determined as the U component value output in step 603. When the edge intensity value of the Y component is lower than the first predetermined threshold and the edge intensity value of the V component is higher than the second predetermined threshold of the edge intensity value of the Y component, the value of the V component is determined as the V component value output in step 601; otherwise, the value of the V component is determined as the V component value output in step 603.

[0075] Figure 7 FIG. is a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0076] With reference to Figure 7

[0077] , the image processing apparatus 700 according to an exemplary embodiment of the present disclosure may include an acquisition unit 701, an optimization processing unit 702, an edge intensity detection unit 703, and a determination unit 704.The acquisition unit 701 can acquire a first chrominance component value, where the first chrominance component value refers to the value obtained by upsampling the chrominance component of a pixel point in a video frame. Here, the video frame can be the video data obtained by the decoding end after decoding the encoded video data received from the encoding end. Since the chrominance component is downsampled during encoding, during decoding, it is necessary to upsample the chrominance component to ensure that the chrominance component has the same size dimension as the luminance component. For example, existing upsampling algorithms such as the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm can be used to upsample the chrominance component. Therefore, the first chrominance component value refers to the value obtained by upsampling the chrominance component of a pixel point in the video frame.

[0078] The optimization processing unit 702 can perform a predetermined optimization process on the first chrominance component value to obtain a second chrominance component value.

[0079] According to an exemplary embodiment of the present disclosure, the predetermined optimization process may include at least one of linear prediction processing, guided filtering processing, edge self-adaptive interpolation processing, etc. Of course, the predetermined optimization process of the present disclosure is not limited to the above processes, and can also be any chrominance optimization process that can improve the reconstructed image quality. When the predetermined optimization process includes multiple processes, the present disclosure also does not limit the order of the multiple processes.

[0080] According to an exemplary embodiment of the present disclosure, the predetermined optimization process may include first performing linear prediction processing and then performing guided filtering processing. Specifically, the optimization processing unit 702 can perform linear prediction processing on the first chrominance component to obtain the first chrominance component after linear prediction processing, and then use the value of the luminance component to perform guided filtering processing on the first chrominance component after linear prediction processing to obtain the second chrominance component. Here, the linear prediction processing can better solve the chrominance erosion problem. The guided filtering processing can remove some noise points that appear in the result of the linear prediction processing, making the picture cleaner and tidier on the basis of ensuring the sharpness of the chrominance component. Therefore, performing linear prediction processing first and then guided filtering processing can achieve a better optimization effect. Of course, the present disclosure is not limited to the above processing order, and guided filtering processing can also be performed first and then linear prediction processing. In addition, the linear prediction processing is performed on the chrominance component of the pixel points of the YUV444 video frame obtained by initial upsampling, and only one round of parameter prediction is performed, which improves the parallelism of the algorithm.

[0081] According to an exemplary embodiment of the present disclosure, the optimization processing unit 702 may obtain the values of the luminance components and the upsampled values of the chrominance components of neighboring pixel points within a predetermined neighborhood of a pixel point in a video frame; establish a linear model based on the values of the luminance components and the upsampled values of the chrominance components of the neighboring pixel points, where the linear model is used to characterize the linear relationship between the value of the luminance component and the value of the chrominance component; and obtain a first chrominance component after linear prediction processing by using the linear model and the value of the luminance component of the pixel point.

[0082] The edge intensity detection unit 703 may perform edge intensity detection on the pixel point to obtain the edge intensity value of the luminance component and the edge intensity value of the chrominance component of the pixel point. Here, the present disclosure does not limit the method of edge intensity detection, and the edge intensity detection unit 703 may use any feasible edge intensity detection method.

[0083] After the edge intensity detection unit 703 obtains the edge intensity value of the luminance component and the edge intensity value of the chrominance component of the pixel point, the determination unit 704 may determine the value of the chrominance component of the pixel point as a first chrominance component value or a second chrominance component value based on the edge intensity value of the luminance component and the comparison result between the edge intensity value of the luminance component and the edge intensity value of the chrominance component. Here, in order to solve the situation that the edge information of some luminance components is weaker than that of chrominance components in the above-mentioned predetermined optimization processing, the gradient intensity of each pixel point in the video frame may be detected, and whether to retain the original chrominance component value (i.e., the value obtained by upsampling, i.e., the first chrominance component value) may be determined according to the comparison of the edge intensity values of the luminance component and the chrominance component.

[0084] Specifically, the determination unit 704 may determine the value of the chrominance component of the pixel point as the second chrominance component value when the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition. On the other hand, the determination unit 704 may determine the value of the chrominance component of the pixel point as the first chrominance component value when the edge intensity value of the luminance component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not satisfy the second predetermined condition.

[0085] According to an exemplary embodiment of the present disclosure, the first predetermined condition may be that the edge intensity value of the luminance component is equal to or greater than a first predetermined threshold. The second predetermined condition may be that the edge intensity value of the chrominance component is not higher than a second predetermined threshold of the edge intensity value of the luminance component. Here, the first predetermined threshold and the second predetermined threshold may be set in advance according to needs, experience, or experiments. In addition, the present disclosure is not limited to the above comparison method, and may also be that when the edge intensity value of the luminance component is lower than the first predetermined threshold and the edge intensity value of the chrominance component is higher than the edge intensity value of the luminance component, the value of the chrominance component of the pixel is determined as the first chrominance component value; and so on.

[0086] In addition, it should be noted that the present disclosure does not limit the execution order of the edge intensity detection unit 703, the acquisition unit 701, and the optimization processing unit 702. For example, the acquisition unit 701 and the optimization processing unit 702 may perform operations first, and then the edge intensity detection unit 703 performs operations, or the acquisition unit 701 may perform operations first, then the edge intensity detection unit 703 performs operations, and then the optimization processing unit 702 performs operations, or the edge intensity detection unit 703 may perform operations first, and then the acquisition unit 701 and the optimization processing unit 702 perform operations, or the edge intensity detection unit 703 may perform operations in parallel with the acquisition unit 701 and the optimization processing unit 702, etc. For example, the edge intensity detection unit 703 may first perform the step of detecting the edge intensity of the pixel. When the edge intensity value of the luminance component meets the first predetermined condition or the comparison result of the edge intensity value of the chrominance component and the edge intensity value of the luminance component meets the second predetermined condition, the optimization processing unit 702 then performs a predetermined optimization process on the first chrominance component value, so that the determination unit 704 determines the value of the chrominance component of the pixel as the second chrominance component value.

[0087] According to an exemplary embodiment of the present disclosure, the chrominance component may include a U component and a V component. The same above-mentioned processing method may be respectively adopted for the U component and the V component, and finally the values of the U component and the V component of the pixel are determined. For example, the obtaining unit 701 may respectively obtain the first chrominance component value of the U component and the first chrominance component value of the V component; the optimization processing unit 702 respectively performs a predetermined optimization process on the first chrominance component value of the U component and the first chrominance component value of the V component to obtain the second chrominance component value of the U component and the second chrominance component value of the V component; the edge intensity detection unit 703 performs three-channel edge intensity detection on the pixel to obtain the edge intensity value of the Y component, the edge intensity value of the U component, and the edge intensity value of the V component of the pixel; the determination unit 704 determines the value of the U component of the pixel as the second chrominance component value of the U component when the edge intensity value of the Y component satisfies the first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the Y component satisfies the second predetermined condition. In addition, when the edge intensity value of the Y component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the U component and the edge intensity value of the Y component does not satisfy the second predetermined condition, the value of the U component of the pixel is determined as the first chrominance component value of the U component; the determination unit 704 determines the value of the V component of the pixel as the second chrominance component value of the V component when the edge intensity value of the Y component satisfies the first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the Y component satisfies the second predetermined condition. In addition, when the edge intensity value of the Y component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the V component and the edge intensity value of the Y component does not satisfy the second predetermined condition, the value of the V component of the pixel is determined as the first chrominance component value of the V component.

[0088] Figure 8 is a block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure.

[0089] Referring to Figure 8 , the electronic device 800 includes at least one memory 801 and at least one processor 802. A set of computer-executable instructions is stored in the at least one memory 801. When the set of computer-executable instructions is executed by the at least one processor 802, an image processing method according to an exemplary embodiment of the present disclosure is executed.

[0090] As an example, the electronic device 800 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instruction set. Here, the electronic device 800 does not have to be a single electronic device, and can also be a collection of devices or circuits that can execute the above instructions (or instruction sets) individually or jointly. The electronic device 800 can also be a part of an integrated control system or a system manager, or can be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).

[0091] In the electronic device 800, the processor 802 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0092] The processor 802 can run instructions or code stored in the memory 801, where the memory 801 can also store data. The instructions and data can also be sent and received over the network via a network interface device, where the network interface device can use any known transmission protocol.

[0093] The memory 801 can be integrated with the processor 802. For example, RAM or flash memory can be arranged within an integrated circuit microprocessor, etc. In addition, the memory 801 can include a separate device, such as an external disk drive, a storage array, or other storage devices that can be used by any database system. The memory 801 and the processor 802 can be operatively coupled or can communicate with each other, for example, through an I / O port, a network connection, etc., such that the processor 802 can read files stored in the memory.

[0094] In addition, the electronic device 800 can also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 800 can be connected to each other via a bus and / or a network.

[0095] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are run by at least one processor, the at least one processor is caused to execute the image processing method according to the present disclosure. Examples of the computer-readable storage medium herein include: read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray or optical disc memory, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store in a non-transitory manner a computer program and any associated data, data files, and data structures and to provide the computer program and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the computer program. The computer program in the above computer-readable storage medium may run in an environment deployed in computer devices such as a client, host, proxy device, server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0096] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including computer instructions executable by at least one processor to complete the image processing method according to the exemplary embodiment of the present disclosure.

[0097] According to the image processing method and image processing device of the present disclosure, the chroma upsampling method can be adaptively adjusted based on the edge intensity information of the luminance component and chrominance component of the pixel points in the video image, so that the original chrominance component value can be selected to be retained or the original chrominance component value can be optimized (for example, linear prediction processing, guided filtering processing, etc.) according to the edge intensity information, thereby improving the effect of optimizing the chrominance component value, effectively solving problems such as color bleeding between color blocks and increased noise in the processed image in the existing problems, and being able to achieve more obvious improvement in the color area missing situation and more reduction in the color bleeding phenomenon (for example, the text effect is more plump, the color boundary in the color block junction area is clearer, the text jaggedness is reduced, etc.), and being able to meet scenarios with high requirements for color processing such as video conferencing and game live streaming.

[0098] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only to be considered exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0099] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Obtaining a first chrominance component value, where the first chrominance component value is a value obtained by upsampling the chrominance component of a pixel point in a video frame; Performing chrominance optimization processing on the first chrominance component value to obtain a second chrominance component value; Performing edge intensity detection on the pixel point to obtain an edge intensity value of the luminance component and an edge intensity value of the chrominance component of the pixel point; When the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition, determining the value of the chrominance component of the pixel point as the second chrominance component value; When the edge intensity value of the luminance component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not satisfy the second predetermined condition, determining the value of the chrominance component of the pixel point as the first chrominance component value.

2. The image processing method according to claim 1, characterized in that, The first predetermined condition is that the edge intensity value of the luminance component is equal to or greater than a first predetermined threshold; The second predetermined condition is that the edge intensity value of the chrominance component is not higher than a second predetermined threshold of the edge intensity value of the luminance component.

3. The image processing method according to claim 1, wherein The chrominance optimization processing includes linear prediction processing and guided filtering processing.

4. The image processing method according to claim 3, wherein The performing chrominance optimization processing on the first chrominance component to obtain a second chrominance component value includes: Performing linear prediction processing on the first chrominance component to obtain a first chrominance component after linear prediction processing; Using the value of the luminance component to perform guided filtering processing on the first chrominance component after linear prediction processing to obtain the second chrominance component.

5. The image processing method according to claim 4, wherein, The performing linear prediction processing on the first chrominance component to obtain a first chrominance component after linear prediction processing includes: Obtaining the values of the luminance components and the upsampled values of the chrominance components of neighboring pixel points within a predetermined neighborhood of the pixel point in the video frame; Based on the values of the luminance components and the upsampled values of the chrominance components of the neighboring pixel points, establishing a linear model, where the linear model is used to represent the linear relationship between the value of the luminance component and the value of the chrominance component; Using the linear model and the value of the luminance component of the pixel point to obtain the first chrominance component after linear prediction processing.

6. The image processing method according to any one of claims 1 to 5, characterized in that The chrominance component includes U component and V component; Among them, the when the edge intensity value of the luminance component satisfies a first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies a second predetermined condition, determining the value of the chrominance component of the pixel point as the second chrominance component value includes: When the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the luminance component satisfies the second predetermined condition, determining the value of the U component of the pixel point as the second chrominance component value of the U component; When the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the luminance component satisfies the second predetermined condition, the value of the V component of the pixel is determined as the second chrominance component value of the V component.

7. An image processing apparatus, characterized in that, Including: An acquisition unit configured to acquire a first chrominance component value, where the first chrominance component value refers to the value obtained by upsampling the chrominance component of a pixel in a video frame; An optimization processing unit configured to perform chrominance optimization processing on the first chrominance component value to obtain a second chrominance component value; An edge intensity detection unit configured to perform edge intensity detection on the pixel to obtain the edge intensity value of the luminance component and the edge intensity value of the chrominance component of the pixel; A determination unit configured to: when the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component satisfies the second predetermined condition, determine the value of the chrominance component of the pixel as the second chrominance component value; when the edge intensity value of the luminance component does not satisfy the first predetermined condition and the comparison result between the edge intensity value of the chrominance component and the edge intensity value of the luminance component does not satisfy the second predetermined condition, determine the value of the chrominance component of the pixel as the first chrominance component value.

8. The image processing apparatus according to claim 7, wherein The first predetermined condition is that the edge intensity value of the luminance component is equal to or greater than a first predetermined threshold; The second predetermined condition is that the edge intensity value of the chrominance component is not higher than a second predetermined threshold of the edge intensity value of the luminance component.

9. The image processing apparatus according to claim 7, wherein The chrominance optimization processing includes linear prediction processing and guided filtering processing.

10. The image processing apparatus according to claim 9, wherein The optimization processing unit is configured to: Perform linear prediction processing on the first chrominance component to obtain the first chrominance component after linear prediction processing; Use the value of the luminance component to perform guided filtering processing on the first chrominance component after linear prediction processing to obtain the second chrominance component.

11. The image processing apparatus according to claim 10, wherein The optimization processing unit is configured to: Acquire the values of the luminance component and the upsampled values of the chrominance component of adjacent pixels within a predetermined neighborhood of the pixel; Based on the values of the luminance component and the upsampled values of the chrominance component of the adjacent pixels, establish a linear model, where the linear model is used to characterize the linear relationship between the value of the luminance component and the value of the chrominance component; Use the linear model and the value of the luminance component of the pixel to obtain the first chrominance component after linear prediction processing.

12. The image processing apparatus according to any one of claims 7 to 11, characterized in that, The chrominance component includes a U component and a V component; Wherein, the determination unit is configured to: when the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the U component and the edge intensity value of the luminance component satisfies the second predetermined condition, determine the value of the U component of the pixel as the second chrominance component value of the U component; When the edge intensity value of the luminance component satisfies the first predetermined condition or the comparison result between the edge intensity value of the V component and the edge intensity value of the luminance component satisfies the second predetermined condition, the value of the V component of the pixel is determined as the second chrominance component value of the V component.

13. An electronic device, characterized in that, Comprising: At least one processor; At least one memory storing computer-executable instructions, wherein, when the computer-executable instructions are run by the at least one processor, the at least one processor is caused to execute the image processing method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are run by at least one processor, the at least one processor is caused to execute the image processing method according to any one of claims 1 to 6.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images