Pixel sensitivity adaptation method, apparatus, image sensor, and program product

Abstract

The application discloses a pixel sensitivity self-adaptive method. The method is applied to an image sensor adopting a Bayer-like pixel array, each image frame collected by the image sensor is divided into M*N first areas through a preset first area division strategy, each first area is not overlapped with each other, and the area of each first area is equal. The method comprises the following steps: for each first area of a current image frame, based on current correction parameters of each color channel in the first area, the sensitivity of the first area is corrected; after the sensitivity correction is completed, through statistical analysis on pixel values of each color channel in the first area, the current correction parameters of each color channel in the first area are updated to obtain target correction parameters, wherein the target correction parameters are used for sensitivity correction of a corresponding first area of a next image frame. Through the application, the effect of pixel sensitivity correction can be improved, and the imaging quality is improved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and particularly relates to a pixel sensitivity adaptive method, a pixel sensitivity adaptive device, an image sensor, and a computer program product. Background Technology

[0002] Modern terminal devices place high demands on image sensors, including but not limited to high resolution, high signal-to-noise ratio, and automatic phase detection autofocus. Based on this, image sensors employing Bayer-like pixel arrays have become widely used. A Bayer-like array distributes pixels of the same color in 2×2 array blocks, so that every four pixels form a small area of ​​the same color, and the data is then pixelated to output the final Bayer array data.

[0003] In actual photoelectric conversion, due to factors such as manufacturing process, materials, pixel structure and / or optical characteristics, the sensitivity of four pixels of the same color may have slight differences. This difference will cause adjacent pixels of the same color to have inconsistent output intensity when receiving light signals, thus affecting the final image quality. Summary of the Invention

[0004] This application provides a pixel sensitivity adaptive method, a pixel sensitivity adaptive device, an image sensor, and a computer program product, which can improve the effect of pixel sensitivity correction and help improve image quality.

[0005] Firstly, this application provides a pixel sensitivity adaptive method, which is applied to an image sensor employing a Bayer-like pixel array. Each image frame acquired by the image sensor is divided into M×N first regions using a preset first region segmentation strategy. These first regions do not overlap and have equal areas. The pixel sensitivity adaptive method includes:

[0006] For each first region of the current image frame, sensitivity correction is performed on the first region based on the current correction parameters of each color channel in the first region;

[0007] After sensitivity correction is completed, the current correction parameters of each color channel in the first region are updated by statistical analysis of the pixel values ​​of each color channel in the first region to obtain the target correction parameters. The target correction parameters are used for sensitivity correction of the corresponding first region in the next image frame.

[0008] Secondly, this application provides a pixel sensitivity adaptive device applied to an image sensor employing a Bayer-like pixel array. Each image frame acquired by the image sensor is divided into M×N first regions using a preset first region segmentation strategy. These first regions do not overlap and have equal areas. The pixel sensitivity adaptive device includes:

[0009] The correction module is used to perform sensitivity correction on each first region of the current image frame based on the current correction parameters of each color channel in the first region.

[0010] The update module is used to update the current correction parameters of each color channel in the first region by statistical analysis of the pixel values ​​of each color channel in the first region after the sensitivity correction is completed, so as to obtain the target correction parameters. The target correction parameters are used for the sensitivity correction of the corresponding first region in the next image frame.

[0011] Thirdly, this application provides an image sensor, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0012] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0013] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0014] The beneficial effects of this application compared to existing technologies are as follows: This application divides each image frame acquired by the image sensor into M×N first regions using a preset first region division strategy. These first regions do not overlap and have equal areas. It can be understood that through this division, this application uses each first region of the current image frame as a correction unit, and performs sensitivity correction on the first region based on the current correction parameters of each color channel within that first region. Furthermore, after sensitivity correction is completed, this application continues to statistically analyze the pixel values ​​of each color channel in the first region, further updating the current correction parameters of each color channel in the first region. The updated target correction parameters can be used when performing sensitivity correction on the next image frame. As can be seen from the above process, this application no longer uses fixed correction parameters during the correction process, but rather iteratively optimizes the correction parameters based on the actual correction effect, thereby maintaining the pixel sensitivity correction effect at a good level and helping to improve subsequent imaging quality.

[0015] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the division of the first region provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of the pixel sensitivity adaptive method provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the second region provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram of the pixel sensitivity adaptive device provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the structure of the image sensor provided in the embodiments of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] Modern terminal devices place high demands on image sensors, including but not limited to high resolution, high signal-to-noise ratio, and automatic phase detection autofocus. Based on this, image sensors employing Bayer-like pixel arrays are widely used. A Bayer-like array distributes pixels of the same color in a 2×2 array block. This way, every four pixels form a small area of ​​the same color, and the data is then pixelated to output Bayer array data. In practical applications, common Bayer-like pixel arrays include the Quad Bayer Color Filter Array (QBC) and a further optimized version sharing a single microlens (Quad Bayer+2×2OCL).

[0029] In actual photoelectric conversion, due to factors such as manufacturing process, materials, pixel structure and / or optical characteristics, the sensitivity of four pixels of the same color in a Bayer-like array (that is, four pixels that constitute a small area of ​​the same color) may have slight differences. This difference will cause adjacent pixels of the same color to have inconsistent output intensity when receiving light signals, thus affecting the final image quality.

[0030] Currently, the common practice is to first calibrate the correction parameters and then perform subsequent sensitivity correction based on these parameters. However, variations in factors such as color temperature, aperture, and motor position can also affect the correction parameters. Establishing correction parameter models for these factors would result in a large number of calibration parameters and high calibration costs. Therefore, existing technologies often use only one set of calibration parameters; or, calibration is performed for different color temperatures, and the calibration parameters obtained at different color temperatures are selected based on the actual situation. Clearly, the correction parameters obtained in this way are fixed and only applicable to the usage environment at the time of calibration. Therefore, once the usage environment changes significantly, the matching accuracy of the correction parameters decreases, leading to poor sensitivity correction results.

[0031] To address the above problems, this application proposes a pixel sensitivity adaptive method, a pixel sensitivity adaptive device, an image sensor, and a computer program product, which can improve the effect of pixel sensitivity correction and help improve image quality. Specific embodiments are described below.

[0032] This application proposes a pixel sensitivity adaptive method, which can be applied to an image processor employing a Bayer-like pixel array; that is, the execution subject of this pixel sensitivity adaptive method is an image processor employing a Bayer-like pixel array. The Bayer-like pixel array has been described above, including but not limited to a four-Bayer color filter array and a further optimized four-Bayer color filter array sharing a single microlens, etc., and will not be elaborated upon here.

[0033] Based on the structural characteristics of Bayer-like pixel arrays, each image frame acquired by the image sensor can be pre-divided to facilitate subsequent sensitivity correction processing. In some examples, the division operation can be based on a preset first region division strategy. This first region division strategy can be: dividing the image frame into M columns and N rows of first regions, where the first regions do not overlap and each first region has an equal area. That is, through the first region division strategy, the image frame can be uniformly divided into several blocks in both the horizontal and vertical directions, resulting in M×N first regions. Please refer to [link / reference]. Figure 1 , Figure 1 The diagram showing the division of the first region is provided, where each small rectangle represents a first region.

[0034] Based on this, please refer to Figure 2 The pixel sensitivity adaptive method provided in this application includes:

[0035] Step 201: For each first region of the current image frame, perform sensitivity correction on the first region based on the current correction parameters of each color channel in the first region.

[0036] The CFA period of a Bayer-like array is 4×4, meaning that the color filter array of the image sensor consists of a 4×4 pixel array, with each period containing 16 pixels. That is, within this 4×4 CFA period, there are 16 pixels: 4 red pixels, 4 blue pixels, and 8 green pixels (the reason for the higher number of green pixels is that the human eye is most sensitive to green, and green pixels are needed to increase image clarity and color perception). Since each pixel carries a filter of a specific color, and the information for each color is captured independently, each pixel within the CFA period can be considered to correspond to an independent color channel. Based on this, for a Bayer-like array, this application embodiment defines it as including 16 color channels.

[0037] After acquiring the current image frame, the image sensor can perform sensitivity correction for each first region of the current image frame. Specifically, it performs sensitivity correction on the first region based on the current correction parameters of each color channel in the first region. These current correction parameters can be read from the image sensor's preset storage space; that is, the image sensor's storage space stores a total of M×N×16 correction parameters, corresponding to the M×N first regions and the 16 color channels in each first region.

[0038] Step 202: After the sensitivity correction is completed, the current correction parameters of each color channel in the first region are updated by statistical analysis of the pixel values ​​of each color channel in the first region, and the target correction parameters are obtained.

[0039] After sensitivity correction, the image sensor performs pixel rearrangement. During this rearrangement, it performs statistical analysis on the pixel values ​​of each color channel in the first region. It's important to note that the analysis focuses on the pixel values ​​obtained after sensitivity correction, not the original pixel values. Through this statistical analysis, the image sensor can predict the differences (i.e., residuals) between four adjacent pixels of the same color after sensitivity correction, and update the current correction parameters for each color channel in the first region based on these residuals, thereby obtaining the target correction parameters.

[0040] It is understandable that, since the image sensor performs the same parameter update operation for each first region, it can ultimately obtain M×N×16 target correction parameters. These target correction parameters can be stored in a preset storage space for sensitivity correction of the next image frame.

[0041] In some embodiments, when the current image frame is the first frame, its current calibration parameters can be obtained through pre-calibration. That is, before the image sensor is put into use, the researchers can calibrate the image sensor to obtain M×N×16 initial calibration parameters, which are then stored in a preset storage space. In this way, after the image sensor is powered on, for the first image frame acquired, the initial calibration parameters can be used as the current calibration parameters, thereby achieving sensitivity calibration for the first image frame.

[0042] Based on this, as can be seen from steps 201 and 202, when the current image frame is not the first frame, its current correction parameters can be obtained through the parameter update operation based on the previous image frame. That is, after the image sensor is powered on, its correction parameters can be continuously updated iteratively with the image frames and stored in a preset storage space. In this way, after the image sensor performs sensitivity correction on the first image frame, it can update to obtain the correction parameters to be used for sensitivity correction on the second image frame; after performing sensitivity correction on the second image frame, it can update to obtain the correction parameters to be used for sensitivity correction on the third image frame; and so on. For the acquired image frame N, the current correction parameters used are actually obtained based on the parameter update operation of image frame N-1, which will not be elaborated here.

[0043] It is understood that the initial calibration parameters obtained from calibration can always be stored in the preset storage space without being overwritten or erased; while during the sensitivity calibration process, the target calibration parameters stored in the storage space can be continuously updated based on the parameter update operations that have been performed, that is, the new target calibration parameters can overwrite the old target calibration parameters in order to save the space occupied by parameter storage, which is not limited here.

[0044] In some embodiments, since the image sensor performs the same parameter update operation on each first region, for ease of understanding, the specific update process of the parameter update operation is described here using any first region as an example:

[0045] Step A1: Calculate the weighted pixel sum value in each color channel of the first region.

[0046] As described earlier, a Bayer-like array is defined with 16 color channels. For each color channel, the image sensor aims to know its sensitivity performance, specifically represented by a weighted sum of pixel values. The calculation method for this weighted sum of pixel values ​​is explained below:

[0047] Step A11: Divide the first region into multiple second regions using a preset second region division strategy.

[0048] The second region division strategy can be as follows: the areas of all second regions obtained from dividing the first region are equal, they do not overlap, and their dimensions are preset. In some examples, this preset size can be 10×10, or other sizes; it is not limited here. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 An example of a second region is given.

[0049] Through this division operation, the first region is divided into multiple second regions. For ease of explanation, let L be the number of second regions obtained from dividing one region. Combining the division process of the first region above, we can see that the entire graph is finally divided into M×N first regions, and each first region has L second regions, that is, the entire graph has a total of M×N×L second regions.

[0050] Step A12: In each second region, calculate the average pixel value in each color channel.

[0051] by Figure 3 Taking the second region shown as an example, for blue, there are four color channels: B11, B12, B21, and B22. The image sensor can calculate the average pixel value for each color channel. For example only, the average pixel value within color channel B11 can be expressed as: meanB11 = ∑B11 / x. Here, B11 represents the pixel value of each pixel belonging to color channel B11 within the second region, and x represents the number of pixels belonging to color channel B11 within that second region (in...). Figure 3 In the illustration, x takes the value of 9. Of course, considering that the denominator will be canceled out in subsequent calculations, the average pixel value in color channel B11 can also be expressed as: meanB11=∑B11 to simplify the calculation process.

[0052] Similarly, the image sensor can calculate the average pixel value in each of the other color channels in the second region, and can use a similar expression. For example, the average pixel value in color channel B12 can be expressed as meanB12, the average pixel value in color channel B21 can be expressed as meanB21, the average pixel value in color channel B22 can be expressed as meanB22, and so on, which will not be elaborated here.

[0053] Step A13: In each second region, the weight of each color channel is calculated based on the preset noise-brightness relationship and the average pixel value in each color channel.

[0054] The noise-luminance relationship can be obtained through pre-statistics and calibration, expressing the relationship between noise and luminance. The luminance described here can be approximately equivalent to pixel values. Based on this noise-luminance relationship and the average pixel values ​​in each color channel, candidate weights for each color channel are calculated. These candidate weights are used to express the flatness confidence of the corresponding color channel, that is, whether the pixel values ​​of each pixel belonging to the corresponding color channel in the second region fluctuate significantly. Taking color channel B11 as an example, the candidate weights for color channel B11 can be calculated as follows:

[0055] The texture weights, calculated first, can be expressed as follows:

[0056] w 11 =min(max(||B 11 -meanB11||1 / NoiseLevel-1,0),1)

[0057] NoiseLevel = Lut(meanB11)

[0058] Then the candidate weights are calculated, which can be expressed as follows:

[0059] w11_=1-w11

[0060] Wherein, Lut() represents the noise-luminance relationship, Lut(mean11) represents the query operation in the noise-luminance relationship with luminance as meanB11, and NoiseLevel represents the noise result obtained through the query operation.

[0061] Among them, B 11 This represents the pixel value of each pixel belonging to the B11 color channel in the second region, and meanB11 represents the average pixel value within the B11 color channel in the second region. Therefore, ||B 11 -meanB11||1 represents the variance of each pixel belonging to the B11 color channel.

[0062] Based on this, since the NoiseLevel result is obtained through a pre-statistical and calibrated noise-luminance relationship, this relationship may be inaccurate. Therefore, the noise ratio ||B| may appear in the calculation. 11 -meanB11||1 / NoiseLevel is less than 1, meaning the variance is less than the noise level. However, physically, since variance is essentially the sum of signal variance and noise variance, this variance should not be less than the noise level. Based on this, we use max(||B 11 The operation `-meanB11||1 / NoiseLevel-1,0)` eliminates calculation errors caused by inaccuracies in the noise-luminance relationship; that is, the noise ratio should be at least 0 (in fact, a noise ratio of 0 indicates that there is no useful signal at that pixel, only noise). Based on this, by using `max(||B11||1 / NoiseLevel-1,0)`... 11 The calculation of -meanB11||1 / NoiseLevel-1,0) can exclude cases that do not exist physically, as described above. Finally, to prevent misjudgment and for robustness, the formula min(max(a,b),1) can be used for comparison to obtain the final texture weight w of the B11 color channel. 11 .

[0063] Among them, w11 _ This represents the candidate weights for color channel B11, also known as flatness weights, used to express flatness confidence. It can be understood that for any color channel, the sum of its flatness weight and texture weight should be 1.

[0064] Similarly, by using the above method, the candidate weights of each color channel in the second region can be obtained. For example, the candidate weight w of color channel B12 can be obtained. 12_ Candidate weights w for color channel B21 21_ and the candidate weights w of color channel B22 22_ Etc., will not be elaborated upon here.

[0065] After obtaining the candidate weights for each color channel, the image sensor can determine the minimum weight among the candidate weights of the color channel and the candidate weights of the target channel as the weight of that color channel.

[0066] In this context, the target channel refers to other color channels that express the same color as the currently considered color channel and, after pixel rearrangement, jointly influence the same pixel. In simpler terms, since a Bayer-like array distributes pixels of the same color in 2×2 array blocks to form small regions of the same color, the target channel here refers to other color channels that form small regions of the same color as the currently considered color channel. This can be understood as the 16 color channels being divided into four groups, each group containing four color channels that express the same color and, after pixel rearrangement, jointly influence the same pixel. For example, color channels B11, B12, B21, and B22 form a group, and for color channel B11, its corresponding target color channels are B12, B21, and B22. Through this operation, each group of color channels shares the same weight.

[0067] For ease of understanding, let's take color channel B11 as an example again. Its weight can be expressed as follows:

[0068] w = min(w11_, w12_, w21_, w22_)

[0069] Here, w is the weight of color channel B11. In fact, the weights of color channels B12, B21, and B22 are also w.

[0070] It is understandable that, through the above steps, although there are 16 color channels in a second region, only 4 weight values ​​need to be calculated in the end (because each group of color channels shares the same weight).

[0071] Step A14: For each color channel, calculate the weighted sum of pixels in the color channel of the first region based on the average pixel value and weight in each color channel of the second region.

[0072] As described earlier, the first region is divided into L second regions. Based on this, for each color channel, the average pixel value and weight within each second region have been obtained through the previous steps. For example, taking color channel B11 as an example, the average pixel value and weight of color channel B11 in the first second region can be obtained, the average pixel value and weight of color channel B11 in the second second region can be obtained, and so on, up to the average pixel value and weight of color channel B11 in the Lth second region; after the image sensor performs weighted summation, the weighted pixel sum value within color channel B11 in the first region can be obtained, which can be expressed by the following formula:

[0073] sumB11=ΣmeanB11*w

[0074] Similarly, the candidate weights of each color channel in the first region can be obtained through the above method. For example, the weighted pixel sum sumB12 of color channel B12, the weighted pixel sum sumB21 of color channel B21, and the weighted pixel sum sumB22 of color channel B22 can be obtained, etc., which will not be elaborated here.

[0075] Step A2: For each color channel in the first region, calculate the average color pixel value based on the weighted pixel sum of the color channel and the weighted pixel sum of the target channel.

[0076] Taking pixel channel B11 as an example, the formula for calculating the average value of its corresponding color pixels can be expressed as:

[0077] Bmean=(sumB11+sumB12+sumB21+sumB22) / 4

[0078] The values ​​sumB11, sumB12, sumB21, and sumB22 have been described previously and will not be repeated here. In fact, a group of color channels in the first region will share the same mean color pixel value. That is, in this step, for a single first region, four mean color pixel values ​​are ultimately calculated: Bmean for the four blue color channels, Grmean for the four green color channels, Gbmean for the other four green color channels, and Rmean for the four red color channels.

[0079] Step A3: Calculate the adjustment coefficient of the color channel based on the weighted pixel sum of the color channel, the average value of the color pixels, and the preset adjustment speed.

[0080] Under ideal conditions, sensitivity differences are completely corrected, so it is desirable that the weighted pixel sums of a set of color channels within a first region are the same or very close. For example, it is desirable that the weighted pixel sums sumB11, sumB12, sumB21, and sumB22 of color channel B11, B12, B21, and B22 are the same or very close. However, in practical applications, complete correction is often impossible, resulting in differences in the weighted pixel sums of a set of color channels within a first region, i.e., residuals. Based on this residual, the adjustment coefficients for each color channel within the first region can be calculated. Taking color channel B11 as an example, the formula for calculating its adjustment coefficient can be expressed as:

[0081] alpha_B11=(Bmean / sumB11)*speed

[0082] Where alpha_B11 represents the adjustment factor for color channel B11, Bmean represents the mean value of the color pixels corresponding to color channel B11 (in reality, the four color channels B11, B12, B21, and B22 correspond to the mean value of the same color pixels), sumB11 represents the weighted pixel sum value of color channel B11, and speed represents the preset adjustment speed. In some examples, to prevent the calculated adjustment factor from overshooting, this adjustment speed is usually limited to less than 1 and greater than 0.

[0083] In some embodiments, to achieve a smooth update of the adjustment coefficient and ensure that the adjustment coefficient is always positive, the image sensor can smooth the ratio of the average color pixel value to the weighted pixel sum of the color channels, and calculate the adjustment coefficient based on the result of the smoothing process and the adjustment speed. Taking color channel B11 as an example, the formula for calculating its adjustment coefficient after smoothing can be expressed as follows:

[0084] alpha_B11=(Bmean / sumB11-1)*speed+1

[0085] Step A4: Calculate the target correction parameters for the color channels based on the current correction parameters and adjustment coefficients of the color channels.

[0086] Image sensors can update the correction parameters of a color channel by multiplying the current correction parameter of that color channel by the adjustment coefficient of that color channel, thereby obtaining the target correction parameter for that color channel. Taking color channel B11 as an example, the update process of its target correction parameter can be specifically expressed as follows:

[0087] Q′_B 11 =Q_B 11 *alpha_B11

[0088] Where alpha_B11 represents the adjustment coefficient corresponding to color channel B11, and Q_B 11 Q′_B represents the current correction factor corresponding to color channel B11. 11 This represents the target correction coefficient corresponding to color channel B11.

[0089] As can be understood, the preceding explanations all used color channel B11 in a specific first region as an example to illustrate its parameter update operation. Since the image frame is divided into M×N first regions and 16 color channels are defined, the image sensor can perform parameter update operations as shown in steps A1-A4 for other color channels in the currently considered first region, as well as for each color channel in each of the other first regions. Ultimately, based on the M×N×16 current correction parameters, M×N×16 target correction parameters can be calculated, which will not be elaborated here.

[0090] In some embodiments, to prevent errors during the parameter update process in extreme cases, the image sensor may also pre-set a first correction parameter threshold and a second correction parameter threshold. The first correction parameter threshold is used to limit the lower limit of the correction parameter; the second correction parameter threshold is used to limit the upper limit of the correction parameter. Then, step A4 can be optimized to: calculate the candidate correction parameters for the color channel based on the current correction parameters and adjustment coefficients of the color channel. This calculation process remains unchanged; that is, it still follows the calculation formula given above, obtaining the calculation result through multiplication. However, this calculation result is not temporarily determined as the final correction parameter for the color channel, but rather as a candidate correction parameter. Then, the candidate correction parameter is compared with the pre-set threshold, with the following three possible outcomes:

[0091] Case 1: The candidate correction parameter is lower than the preset first correction parameter threshold.

[0092] This situation indicates that the currently calculated candidate correction parameters are too low, falling below the lower limit. Therefore, to avoid the influence of outliers, the target correction parameter for the color channel can be determined as the first correction parameter threshold. That is, after the parameter update operation, the final target correction parameter is not the calculated candidate correction parameter, but rather this first correction parameter threshold.

[0093] Case 2: If the candidate correction parameter is higher than the preset second correction parameter threshold, the target correction parameter of the color channel is determined to be the second correction parameter threshold.

[0094] This situation indicates that the currently calculated candidate correction parameter is too high, exceeding the upper limit. Therefore, to avoid the influence of outliers, the target correction parameter for the color channel can be determined as the second correction parameter threshold. That is, after the parameter update operation, the final target correction parameter is not the calculated candidate correction parameter, but rather this second correction parameter threshold.

[0095] Case 3: If the candidate correction parameter is higher than or equal to the first correction parameter threshold and lower than or equal to the second correction parameter threshold, the target correction parameter of the color channel is determined as the candidate correction parameter.

[0096] This indicates that the currently calculated candidate correction parameters are within a reasonable range and can be adopted by the image sensor. In other words, after the parameter update operation, the final target correction parameters are the calculated candidate correction parameters.

[0097] As can be seen from the above, in this embodiment, the image frames acquired by the image sensor are divided into M×N first regions using a preset first region division strategy. These first regions do not overlap and have equal areas. It can be understood that through this division, this application uses each first region of the current image frame as a correction unit, and performs sensitivity correction on the first region based on the current correction parameters of each color channel within that first region. Furthermore, after sensitivity correction is completed, this application continues to statistically analyze the pixel values ​​of each color channel in the first region, further updating the current correction parameters of each color channel in the first region. The updated target correction parameters can be used when performing sensitivity correction on the next image frame. Through the above process, it can be seen that this application's solution no longer uses fixed correction parameters during the correction process, but rather iteratively optimizes the correction parameters based on the actual correction effect, thereby maintaining the pixel sensitivity correction effect at a good level and helping to improve subsequent imaging quality.

[0098] Corresponding to the pixel sensitivity adaptive method provided above, this application also provides a pixel sensitivity adaptive device. This pixel sensitivity adaptive device is applied to an image sensor employing a Bayer-like pixel array. Each image frame acquired by the image sensor is divided into M×N first regions using a preset first region division strategy. The first regions do not overlap and all first regions have equal areas. Figure 4 As shown, the pixel sensitivity adaptive device 4 includes a correction module 401 and an update module 402.

[0099] The correction module 401 is used to perform sensitivity correction on each first region of the current image frame based on the current correction parameters of each color channel in the first region.

[0100] The update module 402 is used to update the current correction parameters of each color channel in the first region by statistical analysis of the pixel values ​​of each color channel in the first region after the sensitivity correction is completed, so as to obtain the target correction parameters. The target correction parameters are used for the sensitivity correction of the corresponding first region in the next image frame.

[0101] In some embodiments, when the current image frame is the first frame, the current correction parameters are obtained through pre-calibration; when the current image frame is not the first frame, the current correction parameters are obtained through parameter update operations based on the previous image frame.

[0102] In some embodiments, the update module 402 includes:

[0103] The first calculation submodule is used to calculate the weighted pixel sum and value in each color channel of the first region respectively;

[0104] The second calculation submodule is used to calculate the average value of color pixels for each color channel in the first region based on the weighted pixel sum of the color channel and the weighted pixel sum of the target channel. The target channel is: other color channels that express the same color as the color channel and affect the same pixels together with the color channel after pixel rearrangement.

[0105] The third calculation submodule is used to calculate the adjustment coefficient of the color channel based on the weighted pixel sum of the color channel, the average value of the color pixels, and the preset adjustment speed.

[0106] The fourth calculation submodule is used to calculate the target correction parameters of the color channels based on the current correction parameters and adjustment coefficients of the color channels.

[0107] In some embodiments, the first computing submodule includes:

[0108] A partitioning unit is used to divide a first region into multiple second regions using a preset second region partitioning strategy;

[0109] The statistical unit is used to calculate the average pixel value in each color channel within each second region;

[0110] The first calculation unit is used to calculate the weight of each color channel in each second region based on the preset noise-brightness relationship and the average pixel value in each color channel.

[0111] The second calculation unit is used to calculate the weighted sum of pixels in the color channels of the first region for each color channel, based on the average pixel value and weight in each color channel of the second region.

[0112] In some embodiments, the first computing unit includes:

[0113] The calculation subunit is used to calculate the candidate weights of each color channel based on the preset noise-brightness relationship and the average pixel value in each color channel.

[0114] A sub-unit is defined to determine the minimum weight among the candidate weights of the color channel and the candidate weights of the target channel for each color channel.

[0115] In some embodiments, the third computing submodule includes:

[0116] The smoothing unit is used to smooth the ratio of the mean value of color pixels to the weighted sum of pixel values ​​of color channels.

[0117] The third calculation unit is used to calculate the adjustment coefficient based on the smoothing results and the adjustment speed.

[0118] In some embodiments, the fourth computing submodule includes:

[0119] The fourth calculation unit is used to calculate the candidate correction parameters for the color channels based on the current correction parameters and adjustment coefficients of the color channels.

[0120] The first determining unit is used to determine the target correction parameter of the color channel as the first correction parameter threshold when the candidate correction parameter is lower than the preset first correction parameter threshold.

[0121] The second determining unit is used to determine the target correction parameter of the color channel as the second correction parameter threshold when the candidate correction parameter is higher than the preset second correction parameter threshold.

[0122] The third determining unit is used to determine the target correction parameter of the color channel as the candidate correction parameter when the candidate correction parameter is higher than or equal to the first correction parameter threshold and lower than or equal to the second correction parameter threshold.

[0123] As can be seen from the above, in this embodiment, the image frames acquired by the image sensor are divided into M×N first regions using a preset first region division strategy. These first regions do not overlap and have equal areas. It can be understood that through this division, this application uses each first region of the current image frame as a correction unit, and performs sensitivity correction on the first region based on the current correction parameters of each color channel within that first region. Furthermore, after sensitivity correction is completed, this application continues to statistically analyze the pixel values ​​of each color channel in the first region, further updating the current correction parameters of each color channel in the first region. The updated target correction parameters can be used when performing sensitivity correction on the next image frame. Through the above process, it can be seen that this application's solution no longer uses fixed correction parameters during the correction process, but rather iteratively optimizes the correction parameters based on the actual correction effect, thereby maintaining the pixel sensitivity correction effect at a good level and helping to improve subsequent imaging quality.

[0124] Corresponding to the pixel sensitivity adaptive method provided above, this application embodiment also provides an image sensor employing a Bayer-like pixel array. Each image frame acquired by this image sensor is divided into M×N first regions using a preset first region segmentation strategy. These first regions do not overlap and have equal areas. Please refer to [link to relevant documentation]. Figure 5 , Figure 5A schematic diagram of the image sensor structure is provided. The electronic device 5 includes: a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor. Specifically, the memory 501 stores the computer program, and the processor 502 executes various functional applications and data processing by running the computer program stored in the memory 501 to acquire resources corresponding to preset events. Specifically, the processor 502 performs the following steps by running the computer program stored in the memory 501:

[0125] For each first region of the current image frame, sensitivity correction is performed on the first region based on the current correction parameters of each color channel in the first region;

[0126] After sensitivity correction is completed, the current correction parameters of each color channel in the first region are updated by statistical analysis of the pixel values ​​of each color channel in the first region to obtain the target correction parameters. The target correction parameters are used for sensitivity correction of the corresponding first region in the next image frame.

[0127] Assuming the above is the first possible implementation, in the second possible implementation provided based on the first possible implementation, when the current image frame is the first frame, the current correction parameter is obtained by pre-calibration, and when the current image frame is not the first frame, the current correction parameter is obtained by parameter update operation based on the previous image frame.

[0128] In a third possible implementation based on the first possible implementation described above, or based on the second possible implementation described above, the current correction parameters of each color channel in the first region are updated by statistical analysis of the pixel values ​​of each color channel in the first region to obtain the target correction parameters, including:

[0129] Calculate the weighted pixel sum value in each color channel of the first region;

[0130] For each color channel in the first region, the average value of the color pixels is calculated based on the weighted sum of the color channels and the weighted sum of the target channels. The target channels are other color channels that express the same color as the color channels and affect the same pixels together with the color channels after pixel rearrangement.

[0131] The adjustment coefficients for the color channels are calculated based on the weighted sum of pixels in the color channels, the average value of the color pixels, and the preset adjustment speed.

[0132] The target correction parameters for the color channels are calculated based on the current correction parameters and adjustment coefficients of the color channels.

[0133] In the fourth possible implementation provided based on the third possible implementation described above, the weighted pixel sum value in each color channel of the first region is calculated, including:

[0134] The first region is divided into multiple second regions using a pre-defined second region division strategy;

[0135] In each second region, the average pixel value within each color channel is calculated.

[0136] In each second region, the weight of each color channel is calculated based on the preset noise-luminance relationship and the average pixel value in each color channel;

[0137] For each color channel, the weighted sum of pixels in the color channel of the first region is calculated based on the average pixel value and weight in each color channel of the second region.

[0138] In the fifth possible implementation provided based on the fourth possible implementation described above, the weight of each color channel is calculated based on a preset noise-luminance relationship and the average pixel value in each color channel, including:

[0139] Based on the preset noise-brightness relationship and the average pixel value in each color channel, the candidate weights of each color channel are calculated.

[0140] For each color channel, the minimum weight among the candidate weights of the color channel and the candidate weights of the target channel is determined as the weight of that color channel.

[0141] In the sixth possible implementation provided based on the third possible implementation described above, the adjustment coefficient of the color channel is calculated according to the weighted pixel sum value of the color channel, the average color pixel value, and the preset adjustment speed, including:

[0142] Smooth the ratio of the average color pixel value to the weighted sum of color channel pixels;

[0143] The adjustment coefficient is calculated based on the results of the smoothing process and the adjustment speed.

[0144] In a seventh possible implementation based on the third possible implementation described above, the target correction parameter of the color channel is calculated according to the current correction parameter and the adjustment coefficient of the color channel, including:

[0145] Based on the current correction parameters and adjustment coefficients of the color channels, the candidate correction parameters of the color channels are calculated.

[0146] If the candidate correction parameter is lower than the preset first correction parameter threshold, the target correction parameter of the color channel is determined to be the first correction parameter threshold.

[0147] If the candidate correction parameter is higher than the preset second correction parameter threshold, the target correction parameter of the color channel is determined as the second correction parameter threshold.

[0148] If the candidate correction parameter is higher than or equal to the first correction parameter threshold and lower than or equal to the second correction parameter threshold, the target correction parameter of the color channel is determined as the candidate correction parameter.

[0149] It should be understood that, in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0150] Memory 501 may include read-only memory and random access memory, and provides instructions and data to processor 502. Some or all of memory 501 may also include non-volatile random access memory. For example, memory 501 may also store device category information.

[0151] As can be seen from the above, in this embodiment, the image frames acquired by the image sensor are divided into M×N first regions using a preset first region division strategy. These first regions do not overlap and have equal areas. It can be understood that through this division, this application uses each first region of the current image frame as a correction unit, and performs sensitivity correction on the first region based on the current correction parameters of each color channel within that first region. Furthermore, after sensitivity correction is completed, this application continues to statistically analyze the pixel values ​​of each color channel in the first region, further updating the current correction parameters of each color channel in the first region. The updated target correction parameters can be used when performing sensitivity correction on the next image frame. Through the above process, it can be seen that this application's solution no longer uses fixed correction parameters during the correction process, but rather iteratively optimizes the correction parameters based on the actual correction effect, thereby maintaining the pixel sensitivity correction effect at a good level and helping to improve subsequent imaging quality.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.

[0158] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A pixel sensitivity adaptive method, characterized in that, The pixel sensitivity adaptive method is applied to an image sensor using a Bayer-like pixel array. Each image frame acquired by the image sensor is divided into M×N first regions by a preset first region division strategy. The first regions do not overlap and the area of ​​each first region is equal. The pixel sensitivity adaptive method includes: For each first region of the current image frame, sensitivity correction is performed on the first region based on the current correction parameters of each color channel in the first region; After the sensitivity correction is completed, the current correction parameters of each color channel in the first region are updated by statistical analysis of the pixel values ​​of each color channel in the first region to obtain the target correction parameters, wherein the target correction parameters are used for the sensitivity correction of the corresponding first region in the next image frame.

2. The pixel sensitivity adaptive method as described in claim 1, characterized in that, When the current image frame is the first frame, the current correction parameter is obtained through pre-calibration; when the current image frame is not the first frame, the current correction parameter is obtained through a parameter update operation based on the previous image frame.

3. The pixel sensitivity adaptive method as described in claim 1 or 2, characterized in that, The step of updating the current correction parameters of each color channel in the first region through statistical analysis of the pixel values ​​of each color channel in the first region to obtain the target correction parameters includes: Calculate the weighted pixel sum value in each of the color channels in the first region; For each color channel in the first region, the average value of the color pixels is calculated based on the weighted sum of pixels of the color channel and the weighted sum of pixels of the target channel. The target channel is: other color channels that express the same color as the color channel and jointly affect the same pixels with the color channel after pixel rearrangement. The adjustment coefficient of the color channel is calculated based on the weighted pixel sum of the color channel, the average value of the color pixels, and the preset adjustment speed. The target correction parameters of the color channel are calculated based on the current correction parameters and the adjustment coefficients of the color channel.

4. The pixel sensitivity adaptive method as described in claim 3, characterized in that, The step of calculating the weighted pixel sum value in each color channel of the first region includes: The first region is divided into multiple second regions using a preset second region division strategy; In each of the second regions, the average pixel value in each color channel is calculated. In each of the second regions, the weights of each color channel are calculated based on a preset noise-luminance relationship and the average pixel value in each color channel. For each color channel, the weighted sum of pixels in the color channel in the first region is calculated based on the average pixel value and weight in each of the second regions.

5. The pixel sensitivity adaptive method as described in claim 4, characterized in that, The calculation of the weight of each color channel based on the preset noise-luminance relationship and the average pixel value in each color channel includes: Based on the preset noise-brightness relationship and the average pixel value in each color channel, the candidate weights of each color channel are calculated. For each color channel, the minimum weight among the candidate weights of the color channel and the candidate weights of the target channel is determined as the weight of the color channel.

6. The pixel sensitivity adaptive method as described in claim 3, characterized in that, The step of calculating the adjustment coefficient of the color channel based on the weighted pixel sum of the color channel, the average value of the color pixels, and the preset adjustment speed includes: The ratio of the average value of the color pixels to the weighted sum of the color channels is smoothed. The adjustment coefficient is calculated based on the smoothing result and the adjustment speed.

7. The pixel sensitivity adaptive method as described in claim 3, characterized in that, The step of calculating the target correction parameters of the color channel based on the current correction parameters and the adjustment coefficient of the color channel includes: Based on the current correction parameters and adjustment coefficients of the color channel, the candidate correction parameters of the color channel are calculated. If the candidate correction parameter is lower than a preset first correction parameter threshold, the target correction parameter of the color channel is determined to be the first correction parameter threshold. If the candidate correction parameter is higher than the preset second correction parameter threshold, the target correction parameter of the color channel is determined to be the second correction parameter threshold. If the candidate correction parameter is higher than or equal to the first correction parameter threshold and lower than or equal to the second correction parameter threshold, the target correction parameter of the color channel is determined as the candidate correction parameter.

8. A pixel sensitivity adaptive device, characterized in that, The pixel sensitivity adaptive device is applied to an image sensor employing a Bayer-like pixel array. Each image frame acquired by the image sensor is divided into M×N first regions by a preset first region division strategy. The first regions do not overlap and the area of ​​each first region is equal. The pixel sensitivity adaptive device includes: The correction module is used to perform sensitivity correction on each first region of the current image frame based on the current correction parameters of each color channel in the first region. The update module is used to update the current correction parameters of each color channel in the first region by statistical analysis of the pixel values ​​of each color channel in the first region after the sensitivity correction is completed, so as to obtain the target correction parameters, wherein the target correction parameters are used for the sensitivity correction of the corresponding first region in the next image frame.

9. An image sensor, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1 to 7.

Images