Image Processing Method and Readable Storage Medium

Abstract

The present invention provides an image processing method and a readable storage medium. Among them, the image processing method includes: acquiring RAW data based on a QPD array or an image sensor similar to a QPD array; calculating the non-uniformity of at least a part of the units; obtaining a flare correction value corresponding to each of the units / pixels based on the non-uniformity; and correcting the RAW data based on the flare correction value. With such a configuration, the flare and ordinary object light are distinguished based on the non-uniformity caused by the flare to the pixels within the same unit, and then the original image is corrected for flare based on this. The final obtained image has a good flare elimination effect and is not limited to a specific external environment, flare color, and flare mode, solving the problem of poor applicability of the existing methods for eliminating lens flare.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image processing method and a readable storage medium. Background Art

[0002] As Figure 1 and Figure 2 shown, when taking a photo with a camera facing a strong light source, the obtained photo may contain lens flare artifacts. The flare appears in various patterns (halos, streaks, color bands, haze, etc.), and the flare also appears in various colors. Figure 1 and Figure 2 The arrows in indicate the position of the flare.

[0003] Eliminating flare can increase the clarity of the photo and improve the user experience. However, there are some problems with the current algorithms for eliminating lens flare. For example, they can only weaken the intensity of the flare, but cannot eliminate the flare color; they can only eliminate flare of a specific color, and the elimination effect of flares of other colors is not ideal; they can only eliminate flare of a specific pattern (for example, only haze-like flare can be eliminated), and the elimination effect of flares of other patterns is not ideal. Or, eliminating flare depends on specific preconditions, such as specific training data sets, specific lighting conditions, etc.

[0004] In summary, the methods for eliminating lens flare in the prior art have poor applicability and cannot be applied to working scenarios with different patterns, different colors, and different external environments. Summary of the Invention

[0005] The purpose of the present invention is to provide an image processing method and a readable storage medium to solve the problem of poor applicability of the methods for eliminating lens flare in the prior art.

[0006] To solve the above technical problems, the present invention provides an image processing method for weakening or eliminating lens flare. The image processing method includes the following steps: obtaining RAW data based on an image sensor of a preset array, where the preset array includes a plurality of 2*2 units, the color channels of the 4 pixels in each unit are the same, and the 4 pixels in the same unit share a microlens; calculating the non-uniformity of at least a part of the units, where the non-uniformity is calculated based on the gray value difference between the 4 pixels of the unit itself; obtaining a flare correction value based on the non-uniformity; and obtaining the corrected RAW data based on the flare correction value.

[0007] Optionally, the image processing method further includes the following step: generating RAW data of a Bayer array based on the corrected RAW data.

[0008] Optionally, the flare correction value belongs to the unit or belongs to the pixel.

[0009] When the flare correction value belongs to the unit, the steps of obtaining the flare correction value based on the degree of imbalance include: calculating a flare estimation value of each unit based on the degree of imbalance; calculating a flare confidence value of each unit based on the degree of imbalance; and calculating the flare correction value based on the flare estimation value of the unit and the flare confidence value of the unit.

[0010] When the flare correction value belongs to the pixel, the steps of obtaining the flare correction value based on the degree of imbalance include: calculating a flare estimation value of each unit based on the degree of imbalance; calculating a flare estimation value of each pixel based on the flare estimation value of the unit; calculating a flare confidence value of each unit based on the degree of imbalance; and calculating the flare correction value based on the flare estimation value of the pixel and the flare confidence value of the unit.

[0011] Optionally, the steps of calculating a flare estimation value of each unit based on the degree of imbalance include: calculating a flare estimation value of each unit of the G channel based on the degree of imbalance of the unit of the G channel; and calculating a flare estimation value of each unit of the B channel and a flare estimation value of each unit of the R channel based on the flare estimation value of the unit of the G channel.

[0012] The steps of calculating a flare estimation value of each pixel based on the flare estimation value of the unit include: calculating the flare estimation value of the pixel based on the flare estimation value of the unit, the average gray value of the unit, and the gray value of the pixel.

[0013] Optionally, the steps of calculating a flare estimation value of each unit of the G channel based on the degree of imbalance of the unit of the G channel include: calculating its own initial estimation value based on the degree of imbalance of each unit of the G channel; and after the initial estimation value of each unit of the G channel is corrected based on the initial estimation value of the unit of the G channel in its neighborhood, obtaining the flare estimation value of the unit of the G channel.

[0014] Optionally, the steps of obtaining the flare estimation value of the unit of the G channel after the initial estimation value of each unit of the G channel is corrected based on the initial estimation value of the unit of the G channel in its neighborhood include: the initial estimation value of each unit of the Gr channel is corrected based on the initial estimation value of the unit of the Gb channel in its neighborhood to obtain the flare estimation value of the unit of the Gr channel; and the initial estimation value of each unit of the Gb channel is corrected based on the initial estimation value of the unit of the Gr channel in its neighborhood to obtain the flare estimation value of the unit of the Gb channel.

[0015] Optionally, the step of calculating the flare estimation value of each unit of the B channel and the flare estimation value of each unit of the R channel based on the flare estimation value of the unit of the G channel includes: each unit of the B channel calculates the flare estimation value of the unit of the B channel based on its own average gray value and the flare estimation value of the unit of the G channel in its neighborhood; and each unit of the R channel calculates the flare estimation value of the unit of the R channel based on its own average gray value and the flare estimation value of the unit of the G channel in its neighborhood.

[0016] Optionally, the step of calculating the flare confidence value of each unit based on the degree of imbalance includes: calculating at least one of an imbalance confidence degree, a gradient confidence degree, and a similarity confidence degree; and obtaining the flare confidence value of the unit based on at least one of the imbalance confidence degree, the gradient confidence degree, and the similarity confidence degree. Wherein, the imbalance confidence degree is mapped based on the degree of imbalance; the gradient confidence degree is remapped based on the gradient amplitude calculated from the average gray value of the unit, and the similarity confidence degree is calculated based on the local similarity of the eigenvalue, and the eigenvalue is calculated based on the gray values of the 4 pixels of the unit.

[0017] Optionally, the step of calculating the flare correction value based on the flare estimation value of the unit and the flare confidence value of the unit includes: calculating the flare correction value (here, the flare correction value belongs to the unit) based on the flare estimation value of the unit and the flare confidence value of the unit in the preset set, where the elements in the preset set are the unit to be calculated and the units in its rectangular neighborhood.

[0018] The step of calculating the flare correction value based on the flare estimation value of the pixel and the flare confidence value of the unit includes: calculating the flare correction value (here, the flare correction value belongs to the pixel) based on the flare estimation value of the pixel included in the unit in the preset set and the flare confidence value of the unit.

[0019] Optionally, when the flare correction value belongs to the unit, the step of obtaining the corrected RAW data based on the flare correction value includes: calculating the corrected RAW data by calculating the average gray value of the unit and the flare correction value; or calculating the corrected RAW data by calculating the gray value of the pixel and the flare correction value.

[0020] When the flare correction value pertains to the pixel, the step of obtaining the corrected RAW data based on the flare correction value includes: calculating the corrected RAW data from the gray value of the pixel and the flare correction value.

[0021] To solve the above technical problem, the present invention further provides a readable storage medium, on which a program is stored, and when the program runs, it executes the above image processing method.

[0022] Compared with the prior art, in an image processing method and a readable storage medium provided by the present invention, the image processing method includes: obtaining RAW data based on a QPD array or an image sensor similar to a QPD array; calculating the non-uniformity of at least a part of the units; obtaining a flare correction value corresponding to each unit / pixel based on the non-uniformity; and correcting the RAW data based on the flare correction value. The inventor found through a large number of experiments and careful observations that flare can cause strong non-uniformity in each 2×2 unit of a QPD or an array structure similar to a QPD. Based on the above phenomenon, flare and ordinary object light can be effectively distinguished, and then the above solution is set. The above solution uses a specific structure and calculates the non-uniformity of each unit, and then corrects the flare of the original image based on the non-uniformity. Finally, the elimination effect of flare in the obtained image is better, and it is not limited to a specific external environment, flare color, and flare mode, solving the problem of poor applicability of the method for eliminating lens flare in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Those of ordinary skill in the art will understand that the provided drawings are used to better understand the present invention and do not constitute any limitation to the scope of the present invention. Among them:

[0024] Figure 1 is a photo affected by flare in the prior art;

[0025] Figure 2 is another photo affected by flare in the prior art;

[0026] Figure 3 is a schematic structural diagram of a QPD array;

[0027] Figure 4 is a schematic diagram of the principle of flare on the imaging of an image sensor in a QPD structure;

[0028] Figure 5 is a gray-scale diagram of the RAW data of a QPD array under a flare condition;

[0029] Figure 6 is a schematic flowchart of the image processing method according to an embodiment of the present invention;

[0030] Figure 7 It is another schematic flowchart of the image processing method according to an embodiment of the present invention;

[0031] Figure 8 It is the effect diagram of the image processed by the image processing method according to an embodiment of the present invention;

[0032] Figure 9 It is the effect diagram of the first output characteristic of the image processing method according to an embodiment of the present invention;

[0033] Figure 10 It is the effect diagram of the second output characteristic of the image processing method according to an embodiment of the present invention.

[0034] In the attached drawings:

[0035] 1 - microlens; 2 - unit; 3 - pixel; 4 - object light; 5 - flare; 6 - sensor;

[0036] 101 - RAW data; 102 - basic flare map; 103 - confidence value map; 104 - corrected flare map; 105 - corrected RAW data. Specific embodiments

[0037] To make the objectives, advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the attached drawings and specific embodiments. It should be noted that the attached drawings are all in very simplified forms and not drawn to scale, and are only used to conveniently and clearly assist in explaining the objectives of the embodiments of the present invention. In addition, the structures shown in the attached drawings are often part of the actual structures. In particular, the attached drawings need to show different emphases and sometimes use different scales.

[0038] As used in the present invention, the singular forms "a", "an" and "the" include plural referents, the term "or" is generally used in the sense of including "and / or", the term "several" is generally used in the sense of including "at least one", the term "at least two" is generally used in the sense of including "two or more", in addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", "third" may explicitly or implicitly include one or at least two of such features. "One end" and "the other end", as well as "proximal end" and "distal end" generally refer to two corresponding parts, which not only include the endpoints. The terms "mounted", "connected", "coupled" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and may be the internal communication of two elements or the interaction relationship between two elements. In addition, as used in the present invention, one element being disposed on another element generally only means that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the two elements may be directly or indirectly connected, coupled, cooperated or transmitted through an intermediate element, and cannot be construed as indicating or implying the spatial position relationship between the two elements, that is, one element may be inside, outside, above, below or on one side of the other element, etc., in any orientation, unless otherwise explicitly specified in the context. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] The core idea of the present invention is to provide an image processing method and a readable storage medium to solve the problem of poor applicability of the method for eliminating lens flare in the prior art.

[0040] The following is a description with reference to the accompanying drawings.

[0041] For the QPD array, it includes a plurality of 2×2 units, and the color channels of the 4 pixels in each of the units are the same. The 4 pixels in the same unit share a microlens 1. Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of the QPD array. Figure 3 In , each small square represents a pixel 3, and 4 pixels 3 form a 2×2 unit 2. The pixels in the same unit 2 have the same color channels, Figure 3 and the circles in represent microlenses 1.

[0042] The inventor found through a large number of experiments and careful observations that glare can cause a strong imbalance in each 2×2 unit of a QPD or an array structure similar to a QPD. Based on the above phenomenon, glare and ordinary object light can be effectively distinguished. Specifically, in a textureless area without glare and with clear focus, the values of these four pixels are almost equal, that is, balanced; when there is glare, since glare is a kind of stray light, it usually enters the microlens 1 at a relatively large angle, resulting in different values among the four pixels, that is, unbalanced, and this kind of imbalance is regular in a large area. Please refer to Figure 4 , Figure 4 FIG. Figure 4 shows the optical paths of object light 4 and glare 5 passing through the microlens 1 and projecting onto the sensor 6. From Figure 4 it can be seen that due to the characteristics of the glare 5 itself, it will always project onto a specific area of the sensor 6, resulting in the imbalance of the four pixels.

[0043] Figure 5 FIG. Figure 5 shows the grayscale image of the RAW data of the QPD array under a glare condition. It can be found that there are the following phenomena: 1. The grayscale value of the pixel in the lower right corner of each 2×2 unit is large; 2. The grayscale value of the G channel is larger than that of the B channel and the R channel.

[0044] Based on the above phenomena, the inventor proposed an image processing method as shown in Figure 6 which includes the following steps:

[0045] S10. Obtain RAW data based on an image sensor of a preset array, where the preset array includes a plurality of 2×2 units, the four pixels in each unit have the same color channels, and the four pixels in each unit share the same microlens.

[0046] S20. Calculate the imbalance degree of the unit, and the imbalance degree is calculated based on the grayscale value difference between the four pixels of the unit itself. The specific calculation method can be set according to different application scenarios. For example, use the maximum grayscale value minus the minimum grayscale value as the imbalance degree, or use variance as the imbalance degree, or other mathematical expressions that can reflect the grayscale value difference size of the four pixels.

[0047] S31. Calculate the glare estimation value of each unit of the G channel based on the imbalance degree of the unit of the G channel.

[0048] S34. Calculate the glare estimation value of each unit of the B channel and the glare estimation value of each unit of the R channel based on the glare estimation value of the unit of the G channel.

[0049] S44. Calculate the imbalance confidence; calculate the gradient confidence and calculate the similarity confidence.

[0050] S45. Obtain the flare confidence value based on the imbalance confidence, the gradient confidence, and the similarity confidence. In one embodiment, the flare confidence value is calculated by multiplying the three. In other embodiments, it can also be calculated by summarizing according to the physical meanings represented by each confidence.

[0051] S50. Calculate the flare correction value for each cell based on the flare estimation value and the flare confidence value of the cell. The flare correction value is an important parameter for correcting the gray value of the original pixel. Based on different precision requirements, the flare correction value of each pixel can be calculated separately, or only the flare correction value of each cell can be calculated. For example, when the size of the original image is 16*16, calculating the flare correction value of each pixel can finally output a RAW image of a 16*16 Bayer array. When the size of the original image is 16*16, calculating the flare correction value of each cell and treating each cell as a pixel can finally output a RAW image of a 4*4 Bayer array. Or, when the size of the original image is 16*16, calculating the flare correction value of each cell and using this flare correction value as the flare correction value of each of the four pixels in the cell, then a RAW image of a 16*16 Bayer array can be finally output, and the correction effect may be reduced but the calculation time is saved. Therefore, the above scheme selection can meet different actual needs. That is to say, the flare correction value can be selected to belong to the cell or to the pixel; in this specification, where necessary, "the flare correction value of the cell" and "the flare correction value of the pixel" are used to show the difference between the two; at the same time, "the flare correction value" is also used to refer to the overall concept including the above two scheme selections. Those skilled in the art can understand when reading "the flare correction value" that at this time, "the flare correction value" refers to "the flare correction value of the cell", "the flare correction value of the pixel" or "the overall concept with two selection possibilities", so such a generalization is appropriate.

[0052] S60. Calculate the corrected RAW data by calculating the average gray value of each cell in the RAW data and the corresponding flare correction value.

[0053] And S70 (not shown), generate the RAW data of the Bayer array based on the corrected RAW data.

[0054] In step S20, based on different subsequent processes, the imbalance degree of each of the units can be calculated, or only the imbalance degree of a part of the units can be calculated. For example, there is such an embodiment where the image processing method includes S10, S20, S32, S33, S34. After step S34, the flare estimation value is directly used as the flare correction value of the unit, and step S60 is executed, and the problem can also be solved under specific working conditions. For example, in the case of a high requirement for calculation speed and a low requirement for the accuracy of flare correction. For such an embodiment, only the imbalance degree of a part of the units needs to be calculated. Or other variants set based on the same inventive concept, using the imbalance degree of a part of the units to estimate other parameter information, and then solving the problem.

[0055] Step S31 and step S34 can be collectively referred to as step S30 for calculating the flare estimation value of each of the units based on the imbalance degree. In different embodiments, step S30 can have different practices. For example, the flare estimation value of each unit can be calculated based on the imbalance degree of each unit. Such a calculation method can simplify the calculation process and should also be regarded as the technical solution of the present invention.

[0056] Implementing step S30 using step S31 and step S34 is a preferred strategy. As mentioned in the previous analysis of Figure 5 , the G channel is more sensitive to flare. Therefore, calculating only the imbalance degree corresponding to the G channel and calculating the flare estimation values of the B and R channels based on the imbalance degree of the G channel can obtain higher accuracy.

[0057] In different embodiments, step S35 may or may not be included. At the same time, the specific details of step S50 and step S60 will also change accordingly. In this specification, the embodiment without step S35 will be introduced first, and then the embodiment including step S35 will be introduced.

[0058] In one embodiment, step S31 further includes step S32 for calculating its own initial estimation value based on the imbalance degree of each unit of the G channel; and step S33, the initial estimation value of each unit of the G channel is corrected based on the initial estimation values of the units of the G channel in its neighborhood to obtain the flare estimation value of the unit of the G channel. That is to say, fully considering the imbalance degree of itself and the neighborhood to obtain a relatively "global" effect. In other embodiments, the initial estimation value can also be directly used as the final flare estimation value without correction.

[0059] "Neighborhood" should be understood as the n units closest to the target unit. Generally, n can take multiples of 4 such as 4, 8, 12, etc. based on symmetry. When approaching the image edge, the number of units in the neighborhood will change, but those skilled in the art can understand which exact units are included in the "neighborhood" at this time.

[0060] In step S32, the mapping relationship between the degree of imbalance and the initial estimate can be set by means such as experimental calibration and neural network training.

[0061] In step S33, after the initial estimate of the unit in each Gr channel is corrected based on the initial estimate of the unit in the Gb channel in its neighborhood, the flare estimate of the unit in the Gr channel is obtained; and, after the initial estimate of the unit in each Gb channel is corrected based on the initial estimate of the unit in the Gr channel in its neighborhood, the flare estimate of the unit in the Gb channel is obtained.

[0062] Specifically,

[0063]

[0064]

[0065] Among them, represents the flare estimate of the unit in the Gb channel, represents the flare estimate of the unit in the Gr channel. represents the initial estimate of the unit in the Gb channel, represents the initial estimate of the unit in the Gr channel. represents the average gray value of the unit in the Gb channel, which is the average of the gray values of the four pixels of this unit, represents the average gray value of the unit in the Gr channel, and is understood in the same way.

[0066] The remaining parameters are calculated based on the following formula:

[0067]

[0068]

[0069]

[0070] Among them, represents the average gray value of the i-th unit in the Gr channel in the neighborhood, represents the average gray value of the i-th unit in the Gb channel in the neighborhood, represents the initial estimate of the i-th unit in the Gr channel in the neighborhood, Represents the initial estimate of the i-th unit of the Gb channel in the neighborhood. The sorting method of the above units can be arbitrary, as long as it satisfies non-omission and non-duplication. n 1 Represents the number of units in the neighborhood in step S33. As described above, n 1 Will change with the change of the position of the calculated unit. In most cases, it is a multiple of 4.

[0071] In step S34, specifically, each unit of the B channel calculates the flare estimate value of the unit of the B channel based on its own average gray value and the flare estimate value of the unit of the G channel in its neighborhood; and each unit of the R channel calculates the flare estimate value of the unit of the R channel based on its own average gray value and the flare estimate value of the unit of the G channel in its neighborhood.

[0072] Specifically,

[0073]

[0074] Among them, Represents the flare estimate value of the unit of the B channel, Represents the flare estimate value of the unit of the R channel. Represents the average gray value of the unit of the B channel, Represents the average gray value of the unit of the R channel. The above average gray value is understood in the same way as Under the same idea.

[0075] The remaining parameters are calculated based on the following formula:

[0076]

[0077] Among them, Represents the average gray value of the i-th unit of the G channel in the neighborhood, Represents the flare estimate value of the i-th unit of the G channel in the neighborhood. The sorting method of the above units can be arbitrary, as long as it satisfies non-omission and non-duplication. In step S32, the G channel is the sum of the Gr channel and the Gb channel and is not distinguished. n 2 Represents the number of units in the neighborhood in step S32. n 2 Will change with the change of the position of the calculated unit. In most cases, it is a multiple of 4. In the same embodiment, n 1 And n 2 Can be the same or different.

[0078] In this embodiment, step S44 includes step S41 for calculating the imbalance confidence, step S42 for calculating the gradient confidence, and step S43 for calculating the similarity confidence. It can be understood that there is no necessary temporal sequence among the above S41 to S43. In other embodiments, step S44 may also only include a part or one of steps S41 to S43. Correspondingly, step S45 may also obtain the flare confidence value based on at least one of the imbalance confidence, the gradient confidence, and the similarity confidence.

[0079] Among them, the imbalance confidence is obtained based on the imbalance mapping; the gradient confidence is remapped based on the gradient amplitude calculated from the average gray value of the unit, and the similarity confidence is calculated based on the local similarity of the eigenvalues, and the eigenvalues are calculated based on the gray values of the 4 pixels of the unit.

[0080] The specific calculation process of the gradient confidence is as follows: use the Sobel filter to calculate the gradient amplitude, and then obtain the gradient confidence according to the gradient amplitude mapping. The operators used are respectively:

[0081] and

[0082] The calculated results are superimposed in the form of a two-dimensional norm or a one-dimensional norm, that is,

[0083]

[0084] where G result represents the final calculation result, represents the calculation result based on the G x operator, represents the calculation result based on the G y operator.

[0085] Of course, the gradient amplitude can also be calculated based on other gradient algorithms, and the mapping relationship between the gradient amplitude and the gradient confidence can still be obtained through methods such as experimental calibration or neural network training.

[0086] The specific calculation process of the similarity confidence is as follows: first calculate the eigenvalue of the unit, for example:

[0087] feature 1 = G l - G 2 , feature 2 = G 3 - G 4 ,

[0088] feature 3 = Gl -G 3 ,feature 4 = G 2 -G 4 。

[0089] Among them, G 1 ~G 4 respectively represent the gray values of the 1st to 4th pixels of the G channel, and the sorting method of 1 to 4 can be referred to Figure 3 for understanding.

[0090] Then, in the field, template matching is performed using this feature, and the matching degree is the similarity degree. The SAD (Sum of Absolute Differences) can be used for judgment during matching.

[0091] The similarity degrees of the B channel and the R channel can be obtained by interpolating the similarity degrees of the G channels around them, and the interpolation method is similar to the calculation method of the flare intensity of the B channel and the R channel. Of course, in different embodiments, other methods can also be selected to calculate the phase velocity.

[0092] After calculating the similarity degree, it is mapped to the similarity confidence degree, and the specific mapping relationship can be obtained by methods such as experimental calibration or neural network training.

[0093] Specifically, in step S50, the step of calculating the flare correction value of each unit based on the flare estimation value and the flare confidence value of the unit includes: calculating the flare correction value based on the flare estimation value and the flare confidence value of the unit in the preset set; the elements in the preset set are the unit to be calculated and the units in its rectangular neighborhood. Calculate according to the following formula:

[0094]

[0095] Among them, f u,c represents the flare correction value of the unit, confidence i represents the flare confidence value of the i-th unit in the preset set, and flare est,i represents the flare estimation value of the i-th unit in the preset set. The sorting method of the above units can be arbitrary, as long as it satisfies no omission and no repetition. n 3 represents the number of units in the rectangular neighborhood in step S50 + 1, that is, n 3 takes values such as 3*3, 5*5, 7*7, 3*5, 5*3, etc. In the same embodiment, n 1 , n 2 and (n 3 -1) can be the same or different.

[0096] In step S60, the corrected average gray value is calculated based on the following formula:

[0097]

[0098] where, represents the average gray value of any unit before correction, represents the average gray value after correction. Combining the of all units can obtain the corrected RAW data.

[0099] It can be understood that since the meanings represented by the flare correction values in different embodiments are different, in step S60, it is not necessarily corrected by subtraction.

[0100] In this embodiment, each unit is processed as a whole. In a better embodiment, the image processing method further includes: step S35, calculating the flare estimation value of each pixel based on the flare estimation value of the unit. Calculate according to the following formula.

[0101]

[0102] where, flare p,est represents the flare estimation value of the pixel, V p represents the gray value of the pixel, represents the average gray value before correction of the unit corresponding to the pixel, flare est represents the flare estimation value of the unit corresponding to the pixel.

[0103] Step S50 is changed to calculate the flare correction value of each pixel based on the flare estimation value of the pixel and the flare confidence value of the unit. Specifically, calculate the flare correction value of the pixel based on the flare estimation value of the pixel included in the unit in the preset set and the flare confidence value of the unit. Calculate according to the following formula.

[0104]

[0105] where, f p,c represents the flare correction value of the pixel,, flar p,est,i represents the flare estimation value of the pixel with the same pixel position as the currently calculated pixel in the i-th unit in the preset set.

[0106] Step S60 is changed to calculate the corrected RAW data from the gray value of the pixel and the flare correction value of the pixel. That is, calculate according to the following formula.

[0107] V new = V - fp,c 。

[0108] Among them, V represents the gray value of any pixel before correction, and V new represents the gray value of the pixel after correction. Combining the Vs of all pixels new can also obtain the corrected RAW data.

[0109] In another feasible embodiment, step S35 may not be included. Step S50 is to calculate the flare correction value of each unit based on the flare estimation value and the flare confidence value of the unit. Step S60 is to calculate the corrected RAW data from the gray value of the pixel and the flare correction value of the unit where the pixel is located. That is, it is calculated according to the following formula.

[0110] V new = V - f u,c 。

[0111] Among them, f u,c represents the flare correction value of the unit corresponding to the pixel. Combining the Vs of all pixels new can also obtain the corrected RAW data.

[0112] The above three variations are all based on the flare correction value to correct the gray value, but some calculate the average gray value of the unit, and some calculate the gray value of the pixel; some are calculated based on the flare correction value of the unit, and some are calculated based on the flare correction value of the pixel. Different variations are for different actual needs.

[0113] Finally, in step S70, the RAW data of the QPD array is converted into the RAW data of the Bayer array to facilitate the subsequent output of a color image. The specific conversion method can be Remosaic (a conversion method, and those skilled in the art can understand the meaning of Remosaic), or, four pixels in the same unit are combined into a large pixel, or other feasible conversion methods.

[0114] The above process steps S44 and S45 can also be collectively referred to as step S40 to calculate the flare confidence value of each unit based on the non-uniformity. In other embodiments, step S40 can also be set in other ways, for example, directly obtaining the flare confidence value of the unit by neural network fitting based on the non-uniformity.

[0115] Step S30 and step S40 may also be collectively referred to as step S100, and the flare correction value is obtained based on the unevenness. In other embodiments, step S100 may also be set to be relatively simple. For example, the flare correction value of each unit is directly obtained based on the unevenness of each unit, that is, the calculation of each flare correction value does not consider the influence of any neighborhood, and the RAW data is directly corrected using the correction value subsequently. Such an embodiment can also solve the technical problem under specific working conditions, and therefore should also be regarded as the protection scope of the technical solution of the present invention.

[0116] The above process may also be combined with Figure 7 for understanding. In the Figure 7 shown process, RAW data 101 is first obtained. Steps S30 and S40 are respectively executed based on the RAW data 101. The basic flare map 102 is obtained through step S30, and each pixel in the basic flare map 102 represents the flare estimation value of one of the units; the confidence value map 103 is obtained through step S40, and each pixel in the confidence value map 103 represents the flare confidence value of one of the units. Then, the basic flare map 102 and the confidence value map 103 are combined (i.e., step S50) to obtain the corrected flare map 104, and each pixel in the corrected flare map 104 represents the flare correction value of one of the units or one of the pixels (specifically determined by the details of different embodiments). Finally, subtraction calculation is performed on the RAW data 101 and the corrected flare map 104 to obtain the corrected RAW data 105.

[0117] The processing effect of the image processing method is as Figure 8 shown. As can be seen from the Figure 8 right side, the flare in the image obtained by executing the image processing method is well eliminated.

[0118] In addition, the image processing method also has some characteristics, and these characteristics can be used to assist in judging whether a flare elimination method adopts a process similar to that of this embodiment. As Figure 9 shown, in some similar angular textures, similar horizontal stripes may appear (as indicated by the arrows). As Figure 10 shown, near some light sources, the edge of the saturation area near the light source is not smooth (as indicated by the arrows).

[0119] In addition, this embodiment also provides a readable storage medium, on which a program is stored. When the program runs, the above image processing method is executed. Since the readable storage medium can run the above image processing method, it can also solve the problems existing in the prior art.

[0120] The readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

[0121] In summary, this embodiment provides an image processing method and a readable storage medium. Among them, the image processing method includes: obtaining RAW data based on a QPD array or an image sensor similar to a QPD array; calculating the non-uniformity of at least some of the units; obtaining a flare correction value for each of the units based on the non-uniformity; and correcting the RAW data based on the flare correction value. The inventor found through a large number of experiments and careful observations that flare can cause strong non-uniformity in each 2×2 unit of a QPD or an array structure similar to a QPD. Based on the above phenomenon, flare and ordinary object light can be effectively distinguished, and then the above solution is set. The above solution uses a specific structure and calculates the non-uniformity of each unit, and then corrects the flare of the original image based on the non-uniformity. Finally, the obtained image has a good effect of eliminating flare, and is not limited to a specific external environment, flare color, and flare mode, solving the problem of poor applicability of the method for eliminating lens flare in the prior art.

[0122] The above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention according to the above disclosure are within the protection scope of the technical solution of the present invention.

Claims

1. An image processing method for reducing or eliminating lens flare, characterized in that, the image processing method includes the following steps: Obtain RAW data based on an image sensor with a preset array, where the preset array includes multiple 2×2 units, the color channels of the 4 pixels in each unit are the same, and the 4 pixels in the same unit share a microlens; Calculate the non-uniformity of at least some of the units, where the non-uniformity is calculated based on the gray value difference between the 4 pixels of the unit itself; Obtain a flare correction value based on the non-uniformity; and, Obtain the corrected RAW data based on the flare correction value; The flare correction value belongs to the unit or belongs to the pixel; When the flare correction value belongs to the unit, the step of obtaining the flare correction value based on the non-uniformity includes: Calculate the flare estimation value of each unit based on the non-uniformity; Calculate the flare confidence value of each unit based on the non-uniformity; and, Calculate the flare correction value based on the flare estimation value and the flare confidence value of the unit; When the flare correction value belongs to the pixel, the step of obtaining the flare correction value based on the non-uniformity includes: Calculate the flare estimation value of each unit based on the non-uniformity; Calculate the flare estimation value of each pixel based on the flare estimation value of the unit; Calculate the flare confidence value of each unit based on the non-uniformity; and, Calculate the flare correction value based on the flare estimation value of the pixel and the flare confidence value of the unit.

2. The image processing method according to claim 1, characterized in that, the image processing method further includes the following steps: Generate RAW data of a Bayer array based on the corrected RAW data.

3. The image processing method according to claim 1, characterized in that, the step of calculating the flare estimation value of each unit based on the non-uniformity includes: Calculate the flare estimation value of each unit of the G channel based on the non-uniformity of the unit of the G channel; and, Calculate the flare estimation value of each unit of the B channel and the flare estimation value of each unit of the R channel based on the flare estimation value of the unit of the G channel; The step of calculating the flare estimation value of each pixel based on the flare estimation value of the unit includes: Calculate the flare estimation value of the pixel based on the flare estimation value of the unit, the average gray value of the unit, and the gray value of the pixel.

4. The image processing method according to claim 3, characterized in that, the step of calculating the flare estimation value of each unit of the G channel based on the non-uniformity of the unit of the G channel includes: Calculate its own initial estimation value based on the non-uniformity of each unit of the G channel; and, After the initial estimation value of each unit of the G channel is corrected based on the initial estimation value of the unit of the G channel in its neighborhood, obtain the flare estimation value of the unit of the G channel.

5. The image processing method according to claim 4, characterized in that, The step of obtaining the flare estimation value of the unit in each G channel after correcting the initial estimation value of the unit in each G channel based on the initial estimation value of the unit in the G channel of its neighborhood includes: The initial estimation value of the unit in each Gr channel is corrected based on the initial estimation value of the unit in the Gb channel of its neighborhood to obtain the flare estimation value of the unit in the Gr channel; and, The initial estimation value of the unit in each Gb channel is corrected based on the initial estimation value of the unit in the Gr channel of its neighborhood to obtain the flare estimation value of the unit in the Gb channel.

6. The image processing method according to claim 3, wherein, The step of calculating the flare estimation value of the unit in each B channel and the flare estimation value of the unit in each R channel based on the flare estimation value of the unit in the G channel includes: The unit in each B channel calculates the flare estimation value of the unit in the B channel based on its own average gray value and the flare estimation value of the unit in the G channel of its neighborhood; and, The unit in each R channel calculates the flare estimation value of the unit in the R channel based on its own average gray value and the flare estimation value of the unit in the G channel of its neighborhood.

7. The image processing method according to claim 1, wherein, The step of calculating the flare confidence value of each unit based on the degree of imbalance includes: Calculating at least one of the imbalance confidence degree, the gradient confidence degree, and the similarity confidence degree; and, Obtaining the flare confidence value of the unit based on at least one of the imbalance confidence degree, the gradient confidence degree, and the similarity confidence degree; wherein, the imbalance confidence degree is mapped based on the degree of imbalance, the gradient confidence degree is remapped based on the gradient amplitude calculated from the average gray value of the unit, the similarity confidence degree is calculated based on the local similarity of the eigenvalue, and the eigenvalue is calculated based on the gray values of the 4 pixels of the unit.

8. The image processing method according to claim 1, wherein, The step of calculating the flare correction value based on the flare estimation value of the unit and the flare confidence value of the unit includes: Calculating the flare correction value based on the flare estimation value of the unit and the flare confidence value of the unit in the preset set, wherein the elements in the preset set are the unit to be calculated and the units in its rectangular neighborhood; The step of calculating the flare correction value based on the flare estimation value of the pixel and the flare confidence value of the unit includes: Calculating the flare correction value based on the flare estimation value of the pixel included in the unit in the preset set and the flare confidence value of the unit.

9. The image processing method according to claim 1, wherein, When the flare correction value belongs to the unit, the step of obtaining the corrected RAW data based on the flare correction value includes: The average gray value of the unit and the flare correction value are used to calculate the corrected RAW data; or, the gray value of the pixel and the flare correction value are used to calculate the corrected RAW data; When the flare correction value belongs to the pixel, the step of obtaining the corrected RAW data based on the flare correction value includes: The gray value of the pixel and the flare correction value are used to calculate the corrected RAW data.

10. A readable storage medium, characterized in that, a program is stored on the readable storage medium, and when the program runs, it executes the image processing method according to any one of claims 1 to 9.

Images