Image chromaticity restoration processing method and device

Abstract

The embodiment of the invention provides an image chromaticity restoration processing method and device. The method comprises the following steps: generating a binarized mask value based on a Y channel of a first image in a YUV420 format; copying and up-sampling a U channel and a V channel of the first image to obtain a second image in a YUV444 format; and performing chromaticity restoration on the second image according to the mask value to obtain a third image. According to the method, the problems that in the prior art, up-sampling is achieved through low-pass filtering, the calculation amount is large and complex can be solved, different up-sampling strategies are applied to different areas of the image, the chromaticity restoration result with the subjective quality and objective indexes remarkably improved is obtained, and the method has the advantages of being good in restoration effect, small in calculation amount and low in calculation complexity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and more specifically, to a method and device for image chromaticity restoration processing. Background Art

[0002] YUV is a color encoding method, where "Y" represents brightness, "U" and "V" represent chrominance, and is used to express color and saturation. Common formats include YUV444, YUV422, and YUV420. YUV444 is a lossless format for both chrominance and brightness, that is, each Y component corresponds to a set of UV components.

[0003] Taking advantage of the fact that the human eye is sensitive to brightness but not to chrominance, the amount of UV data is reduced in the YUV format, so as to achieve the purpose of compressing the overall data volume without affecting the user's viewing experience. In the YUV422 format, every two Y components share a set of UV components, and in the YUV420 format, every four Y components share a set of UV components. The storage space occupied by the same image in the YUV420 format is only half of that in the YUV444 format or RGB format, so the YUV420 video image consumes less bandwidth during transmission.

[0004] The YUV420 format video images received by the terminal suffer from chroma loss due to chroma downsampling, while upsampling methods such as simple copy upsampling, linear interpolation, bilinear interpolation, bicubic interpolation and lanczos interpolation have limited restoration effects, resulting in a poor user experience.

[0005] The method for solving the chroma upsampling error in the related art is to first copy and upsample the input YUV420 format image, low-pass filter and upsample the data of adjacent fields of the same polarity in the UV channel; low-pass filter and upsample the data of adjacent fields of opposite polarity in the UV channel, and finally determine the upsampling error through the upsampling error detection unit and then perform low-pass filtering. This method upsamples the YUV420 format to the YUV422 format, and the upsampling calculation is large and complex through low-pass filtering.

[0006] As for the problem that upsampling by low-pass filtering in related technologies is computationally intensive and complex, no solution has been proposed yet. Summary of the invention

[0007] The embodiments of the present application provide a method and device for image chromaticity restoration processing, so as to at least solve the problem that upsampling by low-pass filtering in related technologies requires large and complex calculations.

[0008] According to an embodiment of the present application, a method for image chromaticity restoration is provided, the method comprising:

[0009] Generate a binary mask value based on the Y channel of the input image in YUV420 format;

[0010] Copy and upsample the U channel and the V channel of the first image to obtain an input image in a YUV444 format;

[0011] The second image is chromatically restored according to the mask value to obtain a third image.

[0012] According to another embodiment of the present application, there is also provided an image chromaticity restoration processing device, the device comprising:

[0013] A generation module, used to generate a binary mask value based on the Y channel of the input image in YUV420 format;

[0014] A sampling module, used for copying and up-sampling the U channel and the V channel of the first image to obtain an input image in a YUV444 format;

[0015] The chromaticity restoration module is used to perform chromaticity restoration on the second image according to the mask value to obtain a third image.

[0016] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0017] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0018] In the embodiment of the present application, a binary mask value is generated based on the Y channel of a first image in a YUV420 format; the U channel and the V channel of the first image are copied and upsampled to obtain a second image in a YUV444 format; and the second image is chromatically restored according to the mask value to obtain a third image. This can solve the problem of large and complex upsampling calculations through low-pass filtering in related technologies, and different upsampling strategies are applied to different areas of the image to obtain a chromatization restoration result with significantly improved subjective quality and objective indicators. This has the effects of good restoration effect, small calculation amount, and low computational complexity, and can achieve high-quality chromatization restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a hardware structure block diagram of a computer device of the image chromaticity restoration processing method according to an embodiment of the present application;

[0020] Figure 2is a flow chart of an image chromaticity restoration processing method according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of an image chromaticity restoration processing method according to an optional embodiment of the present application;

[0022] Figure 4 is a schematic diagram of average downsampling according to this embodiment;

[0023] Figure 5 is a schematic diagram of U channel replication upsampling according to this embodiment;

[0024] Figure 6 is a flowchart of upsampling a YUV420 image according to this embodiment;

[0025] Figure 7 is a schematic diagram of four points a, b, c, d to be restored and their five nearest neighboring points in a 2×2 unit of a U channel according to this embodiment;

[0026] Figure 8 is a block diagram of an image chromaticity restoration processing device according to an embodiment of the present application;

[0027] Fig. 9 It is a block diagram of an image chromaticity restoration processing device according to an optional embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a computer device or a similar computing device. Taking running on a computer device as an example, Figure 1 is a hardware structure block diagram of a computer device for the image chromaticity restoration processing method of an embodiment of the present application, such as Figure 1 As shown, the computer device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device) and a memory 104 for storing data, wherein the above-mentioned computer device may also include a transmission device 106 for communication functions and an input and output device 108. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer device. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image chromaticity restoration processing method in the embodiment of the present application. The processor 102 executes various functional applications and single board matching by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a method for image chromaticity restoration processing running on the above-mentioned computer device is provided. Figure 2 is a flowchart of an image chromaticity restoration processing method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0034] Step S202, generating a binary mask value based on the Y channel of the first image in YUV420 format;

[0035] In the above step S202, the format of the first image is YUV420, and every four Y components in the YUV420 format share a set of UV components. A binary mask value of the first image is generated based on the Y channel of the first image, and the mask value may also be called a mask value.

[0036] Step S204, copying and up-sampling the U channel and the V channel of the first image to obtain a second image in a YUV444 format;

[0037] In this embodiment, the above step S204 may specifically include: traversing the U channel and the V channel of the first image respectively with a 2×2 size area as the minimum unit, and copying the U channel value and the V channel value in each minimum unit 3 times to convert the first image into a second image. The format of the obtained second image is YUV444, which is a lossless format for both chroma and brightness, that is, each Y component corresponds to a group of UV components.

[0038] Step S206, performing chromaticity restoration on the second image according to the mask value to obtain a third image.

[0039] Through the above steps S202 to S206, the problem of large and complex upsampling calculations through low-pass filtering in related technologies can be solved. Different upsampling strategies are applied to different areas of the image to obtain chromaticity restoration results with significantly improved subjective quality and objective indicators. It has the effects of good restoration effect, small amount of calculation, low computational complexity, and can achieve high-quality chromaticity restoration.

[0040] In this embodiment, the above step S202 may specifically include:

[0041] S2021, traversing the Y channel of the first image with a preset size area as the minimum unit through the following steps, and obtaining a binary mask value of the first image after the traversal is completed: determining a brightness value in a target unit; setting a mask value corresponding to a pixel point in the target unit to 0 or 1 according to the brightness value in the target unit, and dividing the target unit into a chromaticity smooth area or a chromaticity non-smooth area according to the mask value, wherein the current minimum unit is the target unit; or

[0042] S2022, generating a binary mask value of the first image based on the Y channel of the first image through a machine learning clustering algorithm, a deep learning algorithm, or a pre-trained neural network model.

[0043] Furthermore, the above S2021 may specifically include: determining a target value based on the brightness value within the target unit, wherein the target value is one of the following: the difference between the maximum brightness value and the minimum brightness value in the brightness values ​​within the target unit, the square of the difference between the maximum brightness value and the minimum brightness value, the ratio of the maximum brightness value to the minimum brightness value, the standard deviation of the brightness values ​​within the target unit, and the variance of the brightness values ​​within the target unit; if the target value is less than a first preset threshold, setting the mask value corresponding to the pixel point in the target unit to 0, and dividing the target unit into a chromaticity smooth area; if the target value is greater than or equal to the first preset threshold, setting the mask value corresponding to the pixel point in the target unit area to 1, and dividing the target unit into a chromaticity non-smooth area.

[0044] If the above-mentioned preset size area is a 2×2 unit, the target unit contains 4 pixels. Specifically, the target value is determined according to the brightness value in the 2×2 unit, and the target value is one of the following: the difference between the maximum brightness value and the minimum brightness value in the brightness values ​​in the 2×2 unit, the square of the difference between the maximum brightness value and the minimum brightness value, the ratio of the maximum brightness value to the minimum brightness value, the standard deviation of the brightness values ​​in the 2×2 unit, and the variance of the brightness values ​​in the 2×2 unit; if the target value is less than the first preset threshold, the mask value corresponding to the 4 pixels in the target unit is set to 0, and the target unit is divided into a chromaticity smooth area; if the target value is greater than or equal to the first preset threshold, the mask value corresponding to the 4 pixels in the target unit area is set to 1, and the target unit is divided into a chromaticity non-smooth area.

[0045] In an optional embodiment, the method further includes: acquiring the resolution of the first image; determining whether the width and height values ​​of the first image are an even number based on the resolution; if they are an even number, executing the above step S2021; if they are not an even number, adjusting the width and height values ​​of the first image so that the width and height values ​​of the first image are an even number, and then executing the above step S2021.

[0046] In this embodiment, the above step S206 may specifically include:

[0047] The U channel and V channel of the second image are traversed at least once with a 2×2 area as the minimum unit through the following steps, and a third image is obtained after the traversal is completed, wherein the current minimum unit is the target unit: if the mask values ​​corresponding to the four pixels in the traversed target unit are all 0, there is no need to perform chromaticity restoration on the target unit, and the next unit is directly traversed; if the mask values ​​corresponding to the four pixels in the traversed target unit are not all 0, the U channel values ​​and V channel values ​​of the four pixels in the target unit are restored to obtain the chromaticity restoration result of the target unit, and further, according to the m nearest neighbors of the target unit, The method comprises the following steps: performing chromaticity restoration on the U channel values ​​and V channel values ​​of the four pixels in the target unit using m pixels, wherein m is an integer greater than or equal to 1; detecting the mask values ​​corresponding to the four pixels in the target unit after the chromaticity restoration; obtaining the chromaticity restoration result of the target unit when the mask values ​​corresponding to the four pixels in the target unit are all 0; and adjusting the U channel values, V channel values ​​and the corresponding mask values ​​of one or more pixels in the four pixels according to the mask values ​​corresponding to the four pixels in the target unit to obtain the chromaticity restoration result of the target unit when the mask values ​​corresponding to the four pixels in the target unit are not all 0.

[0048] In an optional embodiment, the above-mentioned chromaticity restoration of the U channel value and the V channel value of the four pixels in the target unit according to the m pixels closest to the target unit may specifically include: performing the following steps on each of the four pixels in the target unit, wherein the pixel being executed is the first target pixel: respectively calculating the absolute value or square abs_y of the difference between the brightness value of the first target pixel and the brightness values ​​of the m pixels, and the absolute value or square abs_u and abs_v of the difference between the U channel value and the V channel value of the first target pixel and the U channel value and the V channel value of the m pixels; if there is a first target point among the m pixels that satisfies abs_y=0 and the corresponding mask value is 0, replacing the U channel value and the V channel value of the first target pixel with the arithmetic mean of the U channel value and the V channel value corresponding to the first target point, and the U channel value and the V channel value corresponding to the first target pixel are replaced by the arithmetic mean of the U channel value and the V channel value corresponding to the first target point. The mask value is changed to 0; if there is no first target point satisfying abs_y=0 and the corresponding mask value is 0 among the m pixel points, and there is a second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and the corresponding mask value is 0, the U channel value and V channel value of the first target pixel point are replaced with the arithmetic mean of the U channel value and V channel value corresponding to the second target point, and the mask value corresponding to the first target pixel point is changed to 0; if there is no first target point satisfying abs_y=0 and the corresponding mask value is 0 and there is no second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and the corresponding mask value is 0, the U channel value and V channel value of the first target pixel point are kept unchanged, and the mask value corresponding to the first target pixel point remains unchanged.

[0049] In another optional embodiment, the above-mentioned adjusting the U channel value, V channel value and corresponding mask value of one or more pixel points among the four pixel points in the target unit according to the mask values ​​corresponding to the four pixel points in the target unit to obtain the chromaticity restoration result of the target unit includes: if there are n pixel points whose corresponding mask value is 1 among the four pixel points in the target unit, the U channel value and V channel value of the n pixel points whose corresponding mask value is 1 are replaced by U0 minus the sum of the U channel values ​​of other (4-n) points and V0 minus the sum of the V channel values ​​of other (4-n) points, respectively, and the mask values ​​of the n pixel points whose corresponding mask value is 1 are changed to 0, wherein U0 is the sum of the U channel values ​​of the four pixel points in the target unit, V0 is the sum of the V channel values ​​of the four pixel points in the target unit, and the value of n is 1, 2, or 3.

[0050] Furthermore, if there is a pair of pixels whose mask value should be 1 among the 4 pixels in the target unit, the U channel value and V channel value of the pair of pixels whose mask value should be 1 are replaced with U0 minus the sum of the U channel values ​​of the other 3 points and V0 minus the sum of the V channel values ​​of the other 3 points, respectively, and the mask value of the pair of pixels whose mask value should be 1 is changed to 0, where U0 is the sum of the U channel values ​​of the 4 pixels in the target unit, and V0 is the sum of the V channel values ​​of the 4 pixels in the target unit; if there are two corresponding pixels whose mask value is 1 among the 4 pixels in the target unit, and the corresponding Y values ​​of the two corresponding pixels whose mask value is 1 are equal, the U channel values ​​of the two corresponding pixels whose mask value is 1 are replaced with U 0 minus the arithmetic mean of the sum of the U channel values ​​of the other two pixels in the four pixels, the V channel value is replaced with V0 minus the arithmetic mean of the sum of the V channel values ​​of the other two pixels, and the corresponding mask values ​​of the two corresponding pixels with mask values ​​of 1 are changed to 0; if there are three corresponding pixels with a mask value of 1 among the four pixels in the target unit, and the corresponding Y values ​​of the three corresponding pixels with mask values ​​of 1 are equal, the U channel values ​​of the three corresponding pixels with mask values ​​of 1 are replaced with U0 minus the arithmetic mean of the U channel value of another pixel in the four pixels, and the V channel value is replaced with V0 minus the arithmetic mean of the sum of the V channel values ​​of another pixel, and the corresponding mask values ​​of the three corresponding pixels with mask values ​​of 1 are changed to 0.

[0051] Figure 3 is a flowchart of an image chromaticity restoration processing method according to an optional embodiment of the present application. Figure 3 As shown, the process also includes the following steps:

[0052] Step S302: Compensate and correct the third image according to the brightness information and position information of the third image to obtain a fourth image in a YUV444 format.

[0053] In this embodiment, the above step S302 may specifically include: taking a 2×2 size area as the minimum unit to traverse the U channel and V channel of the third image respectively, and compensating and correcting the second target pixel with a mask value of 1 among the four pixels in the target unit according to the brightness information and position information of the third image, wherein the target unit is the current minimum unit. Further, determining the normalized weight value of the m pixels closest to the second target pixel and the brightness information combined with the position information of the second target pixel at the second target pixel; and determining the product of the normalized weight value and the sum of the U channel value and the V channel value of the four pixels in the target unit as the U channel value and the V channel value after compensation and correction of the second target pixel.

[0054] The following describes this embodiment in detail by taking the preset size area as a 2x2 size area (corresponding to the above-mentioned 2x2 unit) and m equal to 5, that is, sampling the 5 nearest pixel points as an example.

[0055] Figure 4 is a schematic diagram of average downsampling according to this embodiment, such as Figure 4 As shown, the YUV420 format video image obtained by average downsampling is widely used in video transmission. This embodiment performs chroma upsampling restoration based on the YUV420 format video image obtained by average downsampling, and the YUV420 format image obtained by average downsampling can minimize the mean square error of the chroma upsampling restoration result.

[0056] The Y channel brightness information of the YUV420 format video image is used to generate a binary mask. Based on the prior assumption that the smaller the difference between the brightness values ​​of a pixel and its five nearest neighboring pixels, the smaller the chromaticity difference, the binary mask value can divide the image into a chromaticity smooth area and a chromaticity non-smooth area, and different upsampling strategies are applied to different areas. Figure 5 is a schematic diagram of U channel replication upsampling according to this embodiment, such as Figure 5 As shown, the UV channel is traversed with a 2×2 area as a unit, and the UV channel value in each unit is copied 3 times, that is, there are 4 identical UV channel values ​​in the 2×2 unit, so that the YUV420 format is converted to the YUV444 format; further, chroma upsampling restoration is performed on the YUV444 format video image obtained by copying and upsampling based on the mask value, and the result of copying and upsampling is used in the chroma smooth area, and chroma upsampling restoration is performed based on brightness information in the chroma non-smooth area; finally, since the prior assumption is still not applicable to a small number of scenes, the restoration result will be further compensated and corrected in combination with brightness and position information to obtain the final chroma restoration result (that is, the third image mentioned above).

[0057] Figure 6 is a flowchart of upsampling the YUV420 image according to this embodiment, as shown in Figure 6 As shown, the specific implementation steps of upsampling the YUV420 format video image based on the brightness information mask are as follows:

[0058] The first step is to generate a binary mask value based on the Y channel brightness information for the image in YUV420 format (corresponding to the first image above):

[0059] 1. Get the resolution of the input image and determine whether the width and height of the image are even numbers. If they are odd numbers, truncate them to even numbers.

[0060] 2. Traverse the Y channel with a 2×2 area as the smallest unit;

[0061] 3. Calculate the difference delta_y between the maximum brightness value y_max and the minimum brightness value y_min in each 2×2 cell. If delta_y is less than the set threshold t, the cell area is determined to be a chromaticity smoothing area, and the mask values ​​corresponding to the four positions in the cell are set to 0, otherwise they are set to 1. After the traversal is completed, a 0-1 binary mask value is obtained.

[0062] The mask value may then be updated regularly or irregularly in the above manner.

[0063] The second step is to copy and upsample the image in YUV420 format based on the U and V channels:

[0064] 1. Take the 2×2 area as the smallest unit to traverse the U and V channels respectively, and copy the U and V channel values ​​in each smallest unit 3 times, that is, there are 4 identical U and V channel values ​​in the 2×2 unit, that is, the YUV420 format is converted to the YUV444 format. And for each 2×2 unit, calculate the sum of the I and V channel values ​​of the 4 pixels in the unit:

[0065] U0=U a +U b +U c +U d

[0066] V0=V a +V b +V c +V d ;

[0067] The V channel copy upsampling is similar to that of the U channel and will not be described in detail here.

[0068] The third step is to restore the chromaticity based on the YUV444 image obtained by upsampling in the second step (corresponding to the second image above) according to the mask value:

[0069] 1. Take the 2×2 area as the smallest unit to traverse the U and V channels respectively. Each 2×2 unit has a corresponding mask value. If the mask values ​​corresponding to the four pixels in the unit are all 0, it is considered that the area does not need to be further restored and the next unit is traversed directly. Otherwise, the IV values ​​of the corresponding four pixels a, b, c, and d in the unit are restored;

[0070] 2. Figure 7 is a schematic diagram of the four points a, b, c, d to be restored and their five nearest points in the U channel 2×2 unit according to this embodiment, as shown in FIG. Figure 7 As shown in the figure, the restoration of the I and V values ​​of the four pixels a, b, c, and d in the 2×2 unit refers to the 5 nearest pixels. The same is true for the V channel. Taking point a as an example, the chromaticity restoration specifically includes:

[0071] (1) Calculate the absolute value abs_y of the difference between the brightness value of point a and the brightness values ​​of its five nearest neighbors, and the absolute value abs_u, abs_v of the difference between the U and V values ​​of point a and the U and V values ​​of the five nearest neighbors;

[0072] (2) If there is a point among the five points that satisfies abs_y=0 and the corresponding mask value is 0, the U and V values ​​of point a are replaced by the arithmetic mean of the U and V values ​​corresponding to the pixels that meet the above conditions, and the mask value corresponding to point a becomes 0;

[0073] (3) If there is no point among the five points that satisfies the condition in (2), then set the thresholds t1 and t2, and look for a point that satisfies abs_y<t1, abs_u<t2, abs_v<t2 and whose corresponding mask value is 0. If there is a point that satisfies the condition, the U and V values ​​of point a are replaced by the arithmetic mean of the U and V values ​​corresponding to the pixels that meet the condition, and the mask value corresponding to point a becomes 0.

[0074] (4) If conditions (2) and (3) are not met, the U and V values ​​of point a remain unchanged, the corresponding mask value remains unchanged, and the UV of the next position in the 2×2 unit is restored.

[0075] 3. In the 2×2 unit that needs to be restored, perform the operation in step 2 on the four pixels a, b, c, and d in turn. After completion, check the mask values ​​corresponding to the four pixels. If there are still points with a corresponding mask value of 1, perform the following operations:

[0076] (1) If one of the four pixels a, b, c, and d has a mask value of 1, the U and V values ​​of the point are replaced by I0 minus the sum of the U values ​​of the other three points and V0 minus the sum of the V values ​​of the other three points, respectively, and the mask value of the position becomes 0; assuming that the mask value of point a is 1, then:

[0077]

[0078]

[0079] (2) If the mask value corresponding to two of the four pixels a, b, c, and d is 1, and the Y values ​​corresponding to these two points are equal, then the U value of these two points is replaced by the arithmetic mean of U0 minus the sum of the U values ​​of the other two points, and the V value is replaced by the arithmetic mean of V0 minus the sum of the V values ​​of the other two positions, and the mask value corresponding to these two pixels becomes 0; assuming that the mask values ​​of a and b are 1 and the Y values ​​are equal, then:

[0080]

[0081]

[0082] (3) If the mask value corresponding to three of the four pixels a, b, c, and d is 1, and the Y values ​​corresponding to these three points are equal, then the I values ​​of these three points are replaced by the arithmetic mean of U0 minus the U value of another point, and the V values ​​are replaced by the arithmetic mean of V0 minus the sum of the V values ​​of another point, and the mask values ​​corresponding to these three pixels become 0; assuming that the mask values ​​of a, b, and c are 1 and the Y values ​​are equal, then:

[0083]

[0084]

[0085] 4. Traverse the video image at least once (specifically, it can be 2 times, 3 times, etc.) to perform the operations of steps 1-3, and output the result of the third step.

[0086] The fourth step is to perform chromaticity compensation correction on the chromaticity restoration result (corresponding to the third image above) to obtain and output an image in YUV444 format (corresponding to the fourth image above):

[0087] 1. Based on the output of the third step, the U and V channels are traversed respectively with a 2×2 area as the smallest unit. If there is a point with a corresponding mask value of 1 among the 4 pixels in the unit, further compensation correction is performed. Take the U channel value of point a as an example (the same applies to the V channel value):

[0088] (1) Calculate the absolute value of the difference between the brightness value corresponding to point a and the average brightness value of points b, c, and d, as well as the absolute value of the difference between the brightness values ​​of its five nearest points:

[0089]

[0090]

[0091] E 2_1 =|Y a -Y 2_1 |

[0092]

[0093] E 4_1 =|Y a -Y 4_1 |

[0094]

[0095] Define that the distances of the 5 points closest to any pixel point in the unit in the counterclockwise direction from the point are D2=1, D4=1, like Figure 7 As shown, Indicates that the distance from a is of The position corresponds to the Y value, Y 2_1 Indicates that the distance from a is 1 The position corresponds to the Y value, and the other symbols are the same. D1, D2, D3, D4, and D5 can also be set to other values ​​that meet the conditions D1 = D3 = D5, D2 = D4, and D1 > D2.

[0096] (2) Calculate the normalized weights of the brightness information of point a and the brightness information of the five points closest to point a at point a:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] (3) Calculate the weight of the U value of point a and the U value position information of its five nearest points at point a:

[0104]

[0105] Where D0 is the distance between the point to be restored and the center of the 2×2 unit. Based on D1-D5, the above recommended values ​​can be used. D0 can be

[0106] 2. Similar to step 1, calculate the weights of brightness information and position information at points b, c, and d respectively. The calculation method of brightness information weight at points b, c, and d is similar to that of point a. The calculation formula of position information weight is as follows:

[0107]

[0108]

[0109]

[0110] 3. Calculate the normalized weight of the brightness information of the four pixels a, b, c, and d in the unit combined with the position information. Take point a as an example, and the same applies to points b, c, and d. Define:

[0111]

[0112] So the normalized weight calculation formula is:

[0113]

[0114]

[0115]

[0116] Among them, δ is the brightness information threshold, which means that when the brightness difference is small, the position information is dominant, and when the brightness difference is large, the brightness information is dominant.

[0117] 4. For the position in the unit corresponding to the mask value of 1, the compensation correction formula of the U channel value is:

[0118]

[0119]

[0120]

[0121]

[0122] The formula for compensation correction of the V channel value is the same. After completing one traversal, the result is rounded to the integer and the final color restoration result is output. The rounding strategy for 8-bit data is rounding, and the value is truncated within the corresponding range according to the specific form of the input YUV, which is TV range or Full range.

[0123] In this embodiment, the upsampling of the image in YUV420 format based on brightness information requires the application platform to support basic operations of digital image processing, such as image reading and storage, pixel addition, subtraction, multiplication and division calculations, image display, etc. Different upsampling strategies can be applied to different areas of the image to obtain chromaticity restoration results with significantly improved subjective quality and objective indicators. It has good restoration effect, small calculation amount and low calculation complexity, and has significant advantages over upsampling methods such as simple copy upsampling, linear interpolation, bilinear interpolation, bicubic interpolation and 1anczos interpolation.

[0124] The present application also provides an image chromaticity restoration processing device. Figure 8 is a block diagram of an image chromaticity restoration processing device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0125] A generating module 82, configured to generate a binary mask value based on the Y channel of the first image in the YUV420 format;

[0126] A sampling module 84 is used to copy and upsample the I channel and the V channel of the first image to obtain a second image in a YUV444 format;

[0127] The chromaticity restoration module 86 is used to perform chromaticity restoration on the second image according to the mask value to obtain a third image.

[0128] In one embodiment, the generating module 82 includes:

[0129] The first traversal submodule is used to traverse the Y channel of the first image with a preset size area as the minimum unit through the following steps, and obtain the binary mask value of the first image after the traversal is completed: determine the brightness value in the target unit; set the mask value corresponding to the pixel point in the target unit to 0 or 1 according to the brightness value in the target unit, and divide the target unit into a chromaticity smooth area or a chromaticity non-smooth area according to the mask value, wherein the current minimum unit is the target unit; or

[0130] A generating submodule is used to generate a binary mask value of the first image based on the Y channel of the first image through a machine learning clustering algorithm, a deep learning algorithm or a pre-trained neural network model.

[0131] In one embodiment, the first traversal submodule is further used to determine a target value according to the brightness value in the target unit, wherein the target value is one of the following: the difference between the maximum brightness value and the minimum brightness value in the brightness values ​​in the target unit, the square of the difference between the maximum brightness value and the minimum brightness value, the ratio of the maximum brightness value to the minimum brightness value, the standard deviation of the brightness values ​​in the target unit, and the variance of the brightness values ​​in the target unit; if the target value is less than a first preset threshold, the mask value corresponding to the pixel point in the target unit is set to 0, and the target unit is divided into a chromaticity smooth area; if the target value is greater than or equal to the first preset threshold, the mask value corresponding to the pixel point in the target unit area is set to 1, and the target unit is divided into a chromaticity non-smooth area.

[0132] In one embodiment, the device further comprises:

[0133] An acquisition submodule, used to acquire the resolution of the first image;

[0134] The adjustment submodule is used to determine the width and height of the first image to be an even number according to the resolution or to adjust the width and height of the first image to be an even number.

[0135] In one embodiment, the sampling module 84 is further used to traverse the U channel and the V channel of the first image respectively with a 2×2 size area as the minimum unit, and copy the U channel value and the V channel value in each minimum unit three times to convert the first image into the second image.

[0136] In one embodiment, the chromaticity restoration module 86 includes:

[0137] The second traversal submodule is used to traverse the U channel and the V channel of the second image at least once with a 2×2 size area as the minimum unit through the following steps, and obtain the third image after the traversal is completed, wherein the current minimum unit is the target unit:

[0138] If the mask values ​​corresponding to the four pixels in the traversed target unit are all 0, there is no need to perform chromaticity restoration on the target unit, and the next unit is traversed directly;

[0139] If the mask values ​​corresponding to the four pixels in the traversed target unit are not all 0, the U channel values ​​and the V channel values ​​of the four pixels in the target unit are chromatically restored to obtain the chromatically restored result of the target unit.

[0140] In one embodiment, the second traversal submodule includes:

[0141] A chromaticity restoration unit, used to perform chromaticity restoration on U channel values ​​and V channel values ​​of four pixels in the target unit according to m pixels closest to the target unit, where m is an integer greater than or equal to 1;

[0142] A detection unit, used for detecting mask values ​​corresponding to four pixels in the target unit after chromaticity restoration;

[0143] An adjustment unit is used to obtain a chromaticity restoration result of the target unit when the mask values ​​corresponding to the four pixels in the target unit are all 0; when the mask values ​​corresponding to the four pixels in the target unit are not all 0, adjust the U channel value, V channel value and corresponding mask value of one or more of the four pixels according to the mask values ​​corresponding to the four pixels in the target unit to obtain the chromaticity restoration result of the target unit.

[0144] In one embodiment, the chromaticity restoration unit is further configured to perform the following steps on each of the four pixel points in the target unit, wherein the pixel point being performed is the first target pixel point:

[0145] Calculate the absolute value or square of the difference between the brightness value of the first target pixel and the brightness values ​​of the m pixels, abs_y, and the absolute value or square of the difference between the U channel value, V channel value of the first target pixel and the U channel value, V channel value of the m pixels, abs_u, abs_v;

[0146] If there is a first target point among the m pixels that satisfies abs_y=0 and whose corresponding mask value is 0, the U channel value and the V channel value of the first target pixel point are replaced with the arithmetic mean of the U channel value and the V channel value corresponding to the first target point, and the mask value corresponding to the first target pixel point is changed to 0;

[0147] If there is no first target point satisfying abs_y=0 and whose corresponding mask value is 0 among the m pixel points, and there is a second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and whose corresponding mask value is 0, the U channel value and the V channel value of the first target pixel point are replaced with the arithmetic mean of the U channel value and the V channel value corresponding to the second target point, and the mask value corresponding to the first target pixel point is changed to 0;

[0148] If there is no first target point satisfying abs_y=0 and the corresponding mask value is 0, and there is no second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and the corresponding mask value is 0, the U channel value and V channel value of the first target pixel point are kept unchanged, and the mask value corresponding to the first target pixel point remains unchanged.

[0149] In one embodiment, the adjustment unit is also used to replace the U channel values ​​and V channel values ​​of the n pixel points with corresponding mask values ​​of 1 with U0 minus the sum of the U channel values ​​of other (4-n) points and V0 minus the sum of the V channel values ​​of other (4-n) points, respectively, if there are n pixel points with corresponding mask values ​​of 1 among the 4 pixel points in the target unit, and the mask values ​​of the n pixel points with corresponding mask values ​​of 1 are changed to 0, wherein U0 is the sum of the U channel values ​​of the 4 pixel points in the target unit, V0 is the sum of the V channel values ​​of the 4 pixel points in the target unit, and the value of n is 1, 2, or 3.

[0150] Fig. 9 is a block diagram of an image chromaticity restoration processing device according to an optional embodiment of the present application, such as Fig. 9 As shown, the device also includes:

[0151] The compensation correction module 92 is used to perform compensation correction on the third image according to the brightness information and position information of the third image to obtain the fourth image in the YUV444 format.

[0152] In one embodiment, the compensation correction module is also used to traverse the U channel and V channel of the third image respectively with a 2×2 size area as the minimum unit, and perform compensation correction on the second target pixel point with a mask value of 1 among the 4 pixel points in the target unit according to the brightness information and position information of the third image, wherein the target unit is the current minimum unit.

[0153] In one embodiment, the compensation correction module includes:

[0154] A first determination submodule is used to determine a normalized weight value of the brightness information of the second target pixel combined with the position information of the m pixels closest to the second target pixel and the second target pixel at the second target pixel;

[0155] The second determination submodule is used to determine the product of the normalized weight value and the sum of the U channel value and the V channel value of the four pixels in the target unit as the U channel value and the V channel value of the second target pixel after compensation correction.

[0156] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0157] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0158] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0159] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0160] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0161] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0162] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for image chromaticity restoration, characterized in that: The method comprises: Generate a binary mask value based on the Y channel of the first image in YUV420 format; Copy and upsample the U channel and the V channel of the first image to obtain a second image in a YUV444 format; The second image is chromatically restored according to the mask value to obtain a third image.

2. The method according to claim 1, characterized in that: The binary mask value generated based on the Y channel of the first image in YUV420 format includes: The Y channel of the first image is traversed with a preset size area as the minimum unit through the following steps, and after the traversal is completed, a binary mask value of the first image is obtained: determining a brightness value in a target unit; setting a mask value corresponding to a pixel point in the target unit to 0 or 1 according to the brightness value in the target unit, and dividing the target unit into a chromaticity smooth area or a chromaticity non-smooth area according to the mask value, wherein the current minimum unit is the target unit; or A binary mask value of the first image is generated based on the Y channel of the first image through a machine learning clustering algorithm, a deep learning algorithm or a pre-trained neural network model.

3. The method according to claim 2, characterized in that The step of setting the mask value corresponding to the pixel point in the target unit to 0 or 1 according to the brightness value in the target unit, and dividing the target unit into a chromaticity smooth area or a chromaticity non-smooth area according to the mask value includes: Determine a target value according to the brightness values ​​in the target unit, wherein the target value is one of the following: a difference between a maximum brightness value and a minimum brightness value among the brightness values ​​in the target unit, a square of a difference between the maximum brightness value and the minimum brightness value, a ratio of the maximum brightness value to the minimum brightness value, a standard deviation of the brightness values ​​in the target unit, and a variance of the brightness values ​​in the target unit; If the target value is less than a first preset threshold, the mask value corresponding to the pixel point in the target unit is set to 0, and the target unit is divided into a chromaticity smoothing area; If the target value is greater than or equal to the first preset threshold, the mask value corresponding to the pixel point in the target unit area is set to 1, and the target unit is divided into a chromaticity non-smooth area.

4. The method according to claim 2, characterized in that: After traversing the Y channel of the first image with a preset size area as the minimum unit through the following steps, and obtaining the binary mask value of the first image after the traversal is completed, the method further includes: Acquire the resolution of the first image; The width and height of the first image are determined to be an even number according to the resolution, or the width and height of the first image are adjusted to be an even number.

5. The method according to claim 1, characterized in that The U channel and the V channel of the first image are copied and up-sampled to obtain the second image in the YUV444 format, including: The U channel and the V channel of the first image are traversed respectively with a 2×2 size area as the minimum unit, and the U channel value and the V channel value in each minimum unit are copied 3 times to convert the first image into the second image.

6. The method according to claim 1, characterized in that Performing chromaticity restoration on the second image according to the mask value to obtain the third image comprises: The U channel and the V channel of the second image are traversed at least once respectively with a 2×2 size area as the minimum unit through the following steps, and the third image is obtained after the traversal is completed, wherein the current minimum unit is the target unit: If the mask values ​​corresponding to the four pixels in the traversed target unit are all 0, there is no need to perform chromaticity restoration on the target unit, and the next unit is traversed directly; If the mask values ​​corresponding to the four pixels in the traversed target unit are not all 0, the U channel values ​​and the V channel values ​​of the four pixels in the target unit are chromatically restored to obtain the chromatically restored result of the target unit.

7. The method according to claim 6, characterized in that Performing chromaticity restoration on the U channel values ​​and the V channel values ​​of the four pixels in the target unit to obtain the chromaticity restoration result of the target unit includes: Perform chromaticity restoration on U channel values ​​and V channel values ​​of four pixels in the target unit according to the m pixels closest to the target unit, respectively, where m is an integer greater than or equal to 1; After chromaticity restoration, detecting mask values ​​corresponding to four pixels in the target unit; When the mask values ​​corresponding to the four pixels in the target unit are all 0, a chromaticity restoration result of the target unit is obtained; When the mask values ​​corresponding to the four pixels in the target unit are not all 0, the U channel value, V channel value and corresponding mask value of one or more of the four pixels are adjusted according to the mask values ​​corresponding to the four pixels in the target unit to obtain the chromaticity restoration result of the target unit.

8. The method according to claim 7, characterized in that Performing chromaticity restoration on U channel values ​​and V channel values ​​of four pixels in the target unit according to the m pixels closest to the target unit respectively includes: The following steps are performed for each of the four pixel points in the target unit, wherein the pixel point being performed is the first target pixel point: Calculate the absolute value or square of the difference between the brightness value of the first target pixel and the brightness values ​​of the m pixels, abs_y, and the absolute value or square of the difference between the U channel value, V channel value of the first target pixel and the U channel value, V channel value of the m pixels, abs_u, abs_v; If there is a first target point among the m pixels that satisfies abs_y=0 and whose corresponding mask value is 0, the U channel value and the V channel value of the first target pixel point are replaced with the arithmetic mean of the U channel value and the V channel value corresponding to the first target point, and the mask value corresponding to the first target pixel point is changed to 0; If there is no first target point satisfying abs_y=0 and whose corresponding mask value is 0 among the m pixel points, and there is a second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and whose corresponding mask value is 0, the U channel value and the V channel value of the first target pixel point are replaced with the arithmetic mean of the U channel value and the V channel value corresponding to the second target point, and the mask value corresponding to the first target pixel point is changed to 0; If there is no first target point satisfying abs_y=0 and the corresponding mask value is 0, and there is no second target point satisfying abs_y<second preset threshold, abs_u<third preset threshold, abs_v<third preset threshold and the corresponding mask value is 0, the U channel value and V channel value of the first target pixel point are kept unchanged, and the mask value corresponding to the first target pixel point remains unchanged.

9. The method according to claim 7, characterized in that: Adjusting the U channel value, the V channel value and the corresponding mask value of one or more of the four pixels according to the mask values ​​corresponding to the four pixels in the target unit to obtain the chromaticity restoration result of the target unit includes: If there are n pixels with a corresponding mask value of 1 among the 4 pixels in the target unit, the U channel values ​​and V channel values ​​of the n pixels with a corresponding mask value of 1 are replaced by U 0 minus the sum of the U channel values ​​of the other (4-n) points, V 0 minus the sum of the V channel values ​​of the other (4-n) points, the mask values ​​of the n pixels corresponding to the mask value of 1 are changed to 0, where: U 0 is the sum of the U channel values ​​of the four pixels in the target unit, V 0 is the sum of the V channel values ​​of the 4 pixels in the target unit, and the value of n is 1, 2, or 3.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: The third image is compensated and corrected according to the brightness information and position information of the third image to obtain a fourth image in the YUV444 format.

11. The method according to claim 10, characterized in that The third image is compensated and corrected according to the brightness information and the position information of the third image to obtain the fourth image in the YUV444 format, including: The U channel and the V channel of the third image are traversed respectively with a 2×2 size area as the minimum unit, and compensation correction is performed on the second target pixel point with a mask value of 1 among the 4 pixel points in the target unit according to the brightness information and position information of the third image, wherein the target unit is the current minimum unit.

12. The method according to claim 11, characterized in that Taking a 2×2 size area as the minimum unit, traversing the U channel and the V channel of the third image respectively, and compensating and correcting the second target pixel point with a mask value of 1 among the four pixel points in the target unit according to the brightness information and the position information of the third image, comprises: Determine a normalized weight value of the brightness information of the second target pixel combined with the position information of the m pixels closest to the second target pixel and the second target pixel; The product of the normalized weight value and the sum of the U channel value and the V channel value of the four pixels in the target unit is determined as the U channel value and the V channel value of the second target pixel after compensation correction.

13. An image chromaticity restoration processing device, characterized in that: The device comprises: A generating module, used for generating a binary mask value based on a Y channel of a first image in a YUV420 format; A sampling module, used for copying and up-sampling the U channel and the V channel of the first image to obtain a second image in a YUV444 format; The chromaticity restoration module is used to perform chromaticity restoration on the second image according to the mask value to obtain a third image.

14. A computer-readable storage medium, wherein a computer program is stored in the storage medium, wherein: The computer program is arranged to execute the method as claimed in any one of claims 1 to 12 when executed.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 12.

Images