Non-standard CFA image interpolation method, device and equipment and storage medium

Abstract

The invention provides a non-standard CFA image interpolation method, device and equipment and a storage medium, and relates to the technical field of image processing, and the specific implementation scheme is as follows: respectively calculating gradients in the horizontal direction, the vertical direction and the diagonal direction of each non-standard pixel position in a RAW format graph; a horizontal interpolation estimation value, a vertical interpolation estimation value and a diagonal interpolation estimation value are obtained; calculating corresponding weights in different directions based on the gradients in different directions; performing weighted calculation on the interpolation estimation values in different directions according to the weights in the corresponding directions to obtain a final interpolation; and respectively inserting all the final interpolations into the corresponding non-standard pixel positions to obtain an RAW format graph after interpolation is completed. Through the implementation of the scheme of the invention, the channel relevance is considered, the interpolation is expanded to the RAW domain, the defects that the effect of bilinear interpolation is poor and the calculated amount of bicubic interpolation is too large are avoided, the interpolation efficiency and effect are well balanced, and better hardware friendliness is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to the field of image interpolation technology, and can be applied to non-standard CFA image interpolation scenarios. More specifically, the present application discloses a non-standard CFA image interpolation method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of science and technology, image sensors are widely used in smart phones, digital cameras, security cameras and other product fields. At present, image sensors are usually equipped with a color filter array (CFA) to allow light of a specific wavelength to be incident on the pixel. A typical color filter array is the Bayer array. In order to improve the image acquisition effect of the image sensor in a dark environment, some image sensors are equipped with a Quad Bayer format color filter array, that is, a four-in-one Bayer array, such as Figure 1 The figure shows a schematic diagram of the principle of a standard four-in-one Bayer array provided by the related art. However, in order to improve the performance of image sensors in some special scenes, manufacturers will make non-standard designs for some pixel positions, such as replacing one pixel in each four-in-one pixel area with a non-standard color channel pixel or setting it as a blank pixel, such as Figure 2 The figure shows a principle schematic diagram of a non-standard four-in-one Bayer array provided by the related art, in which the lower right corner pixel position in each four-in-one pixel area is a non-standard pixel position.

[0003] In order to use the standard ISP pipeline to process images, this non-standard CFA needs to be interpolated into Quad Bayer format first, then remosaiced to obtain Bayer format, and then input into the standard ISP for processing. Therefore, the interpolation effect on non-standard pixel positions directly affects the final ISP processing result. The interpolation algorithms provided in related technologies are generally bilinear interpolation, nearest neighbor interpolation, bicubic interpolation and their derivative algorithms, etc. These algorithms are usually interpolated in grayscale or RGB domains, which may result in poor effect or low efficiency.

[0004] It is worth noting that the techniques described in this section are not necessarily techniques that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any technique described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a non-standard CFA image interpolation method, device, equipment and storage medium, which can at least solve the problem of low efficiency or poor effect of the interpolation algorithm for interpolation in the grayscale or RGB domain provided by the related art.

[0006] In a first aspect, the present application provides an image interpolation method for a non-standard CFA, which is applied to an image sensor equipped with a non-standard four-in-one Bayer array, comprising: respectively calculating the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in a RAW format image, and respectively calculating the horizontal interpolation estimate, vertical interpolation estimate and diagonal interpolation estimate of each non-standard pixel position; wherein the RAW format image includes multiple four-in-one image areas, each of the four-in-one image areas includes one non-standard pixel position and three standard pixel positions of the same color channel, and the non-standard pixel position is a blank pixel position or has a color channel different from that of the standard pixel position; respectively calculating the corresponding weights in different directions based on the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position; performing weighted calculation on the horizontal interpolation estimate, the vertical interpolation estimate and the diagonal interpolation estimate of each non-standard pixel position according to the weights of the corresponding directions to obtain a final interpolation; and respectively inserting all the final interpolations into the corresponding non-standard pixel positions to obtain the RAW format image after interpolation.

[0007] The second aspect of the present application provides an image interpolation device of a non-standard CFA, which is applied to an image sensor equipped with a non-standard four-in-one Bayer array, including: a gradient calculation module, which is used to respectively calculate the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in a RAW format image; wherein the RAW format image includes a plurality of four-in-one image areas, each of the four-in-one image areas includes one non-standard pixel position and three standard pixel positions of the same color channel, and the non-standard pixel position is a blank pixel position or has a color channel different from that of the standard pixel position; an estimation module, which is used to respectively calculate each The horizontal interpolation estimation value, vertical interpolation estimation value and diagonal interpolation estimation value of the non-standard pixel position; a weight calculation module, used to calculate the corresponding weights in different directions based on the horizontal direction gradient, vertical direction gradient and diagonal direction gradient of each non-standard pixel position; an interpolation calculation module, used to perform weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position according to the weights in the corresponding directions to obtain a final interpolation; an interpolation module, used to insert all the final interpolations into the corresponding non-standard pixel positions respectively to obtain the RAW format image after the interpolation is completed.

[0008] The third aspect of the present application provides an electronic device, comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the non-standard CFA image interpolation method provided in the first aspect of the embodiment of the present application.

[0009] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the non-standard CFA image interpolation method provided in the first aspect of the embodiment of the present application is implemented.

[0010] As can be seen from the above, according to the non-standard CFA image interpolation method, device, equipment and storage medium provided by the present application, the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image are calculated respectively, and the horizontal interpolation estimate, vertical interpolation estimate and diagonal interpolation estimate of each non-standard pixel position are calculated respectively; the corresponding weights of different directions are calculated based on the gradients of different directions of each non-standard pixel position respectively; the interpolation estimate of different directions of each non-standard pixel position is weightedly calculated according to the weight of the corresponding direction to obtain the final interpolation; all the final interpolations are inserted into the corresponding non-standard pixel positions respectively to obtain the interpolated RAW format image. Through the implementation of the present application scheme, the interpolation is extended to the RAW domain by considering the channel correlation, avoiding the poor effect of bilinear interpolation and the deficiency of excessive bicubic interpolation calculation, so that the interpolation efficiency and effect are well balanced, with better hardware friendliness.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The drawings shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of the principle of a standard four-in-one Bayer array provided for related technologies;

[0014] Figure 2 A schematic diagram of the principle of a non-standard four-in-one Bayer array provided for related technologies;

[0015] Figure 3 A schematic diagram of a basic flow chart of a non-standard CFA image interpolation method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of the principle of horizontal gradient calculation provided in one embodiment of the present application;

[0017] Figure 5 A schematic diagram of the principle of vertical gradient calculation provided by an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the principle of 45° diagonal direction gradient calculation provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of the principle of 135° diagonal direction gradient calculation provided in one embodiment of the present application;

[0020] Figure 8 A schematic diagram of a principle for calculating a horizontal interpolation estimate value provided by an embodiment of the present application;

[0021] Fig. 9 A schematic diagram of a principle for calculating a vertical interpolation estimation value provided by an embodiment of the present application;

[0022] Fig.10 A schematic diagram of a principle for calculating a 45° diagonal interpolation estimate value provided in one embodiment of the present application;

[0023] Fig.11 A schematic diagram of the principle of calculating a 135° diagonal interpolation estimate value provided by an embodiment of the present application;

[0024] Fig.12 A schematic diagram of a detailed process of a non-standard CFA image interpolation method provided in an embodiment of the present application;

[0025] Fig.13 A schematic diagram of functional modules of a non-standard CFA image interpolation device provided in one embodiment of the present application;

[0026] Fig.14 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0028] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the term "plurality" means two or more, unless otherwise clearly and specifically defined.

[0029] In order to solve the problem of low efficiency or poor effect of the interpolation algorithm provided by the related art for interpolation in the grayscale or RGB domain, an embodiment of the present application provides an image interpolation method of non-standard CFA, such as Figure 3 The basic flow chart of the non-standard CFA image interpolation method provided in this embodiment includes the following steps:

[0030] Step 301, respectively calculate the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image, and respectively calculate the horizontal interpolation estimate value, vertical interpolation estimate value and diagonal interpolation estimate value of each non-standard pixel position.

[0031] Specifically, in this embodiment, the RAW format image includes multiple four-in-one image areas, and the four pixel positions in each four-in-one image area are arranged in a 2*2 format, and each four-in-one image area includes a non-standard pixel position and three standard pixel positions of the same color channel, the non-standard pixel position is a blank pixel position or a color channel different from the standard pixel position, and the standard pixel position includes an R channel pixel position, a G channel pixel position, and a B channel pixel position. It should be understood that the diagonal direction mentioned in this embodiment may include a 45° diagonal direction and / or a 135° diagonal direction.

[0032] In an optional implementation of the present embodiment, the above-mentioned steps of respectively calculating the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image include: for each non-standard pixel position in the RAW format image, correspondingly determining the target image area in the overall image area; selecting the pixel value of the target pixel position in the target image area to calculate the horizontal gradient, vertical gradient and diagonal gradient.

[0033] Specifically, this embodiment combines the arrangement characteristics of Quad Bayer and the specificity of non-standard CFA, reasonably selects standard pixel positions related to the non-standard pixel positions to be interpolated in the Raw domain, and extends the interpolation to the Raw domain. First, this embodiment locates a related image area for each non-standard pixel position, such as a 6*6 image area, and then further selects related standard pixel positions in the area for gradient calculation.

[0034] Furthermore, in an optional implementation of the present embodiment, the above-mentioned step of selecting pixel values ​​of target pixel positions in the target image area to calculate horizontal gradients, vertical gradients and diagonal gradients includes: selecting pixel values ​​of different target pixel positions in the target image area according to different selection strategies, and calculating horizontal gradient components of different frequency levels, vertical gradient components of different frequency levels and diagonal gradient components of different frequency levels; and combining the horizontal gradient components of different frequency levels, the vertical gradient components of different frequency levels and the diagonal gradient components of different frequency levels, and calculating the horizontal gradient, vertical gradient and diagonal gradient accordingly.

[0035] Specifically, in this embodiment, when performing gradient calculation for each direction of a non-standard pixel position, this embodiment combines gradient components of multiple different frequency levels in each direction to calculate the final gradient of the component, and the gradient component of each frequency level needs to select the pixel value of a specific standard pixel position according to the corresponding selection strategy for calculation.

[0036] Furthermore, in an optional implementation manner of the present embodiment, the above-mentioned steps of respectively calculating the horizontal gradient, the vertical gradient and the diagonal gradient by combining the horizontal gradient components of different frequency levels, the vertical gradient components of different frequency levels and the diagonal gradient components of different frequency levels, include: assigning corresponding gradient coefficients to the gradient components of different frequency levels in each direction; calculating the horizontal gradient by combining multiple horizontal gradient components of different frequency levels and the corresponding gradient coefficients, calculating the vertical gradient by combining the vertical gradient components of different frequency levels and the corresponding gradient coefficients, and calculating the diagonal gradient by combining the diagonal gradient components of different frequency levels and the corresponding gradient coefficients.

[0037] like Figure 4 FIG. 1 is a schematic diagram of a principle of horizontal gradient calculation provided by the present embodiment. When performing Gb channel interpolation for a non-standard pixel position I(2,2), the present embodiment takes a 6*6 image area for gradient calculation. The horizontal gradient includes low-frequency, high-frequency and other channel high-frequency gradient components. The specific calculation formula is as follows:

[0038]

[0039] Among them, H_grad_low represents the low-frequency gradient component in the horizontal direction, H_grad_high represents the high-frequency gradient component in the horizontal direction, H_grad_other_high1 represents the high-frequency gradient component of the first other channel in the horizontal direction, H_grad_other_high2 represents the high-frequency gradient component of the second other channel in the horizontal direction, H_grad represents the horizontal gradient, abs(·) represents the absolute value calculation, posxy represents the pixel value of the pixel position at the xth row and yth column of the image, and a H , b H 、c H d H It is the gradient coefficient of the gradient components at different frequency levels in the horizontal direction, and usually the sum is 1.

[0040] like Figure 5 FIG. 1 is a schematic diagram showing a principle of vertical gradient calculation provided by this embodiment, and the specific calculation formula is as follows:

[0041]

[0042] Wherein, V_grad_low represents the low-frequency gradient component in the vertical direction, V_grad_high represents the high-frequency gradient component in the vertical direction, V_grad_other_high1 represents the high-frequency gradient component of the first other channel in the vertical direction, V_grad_other_high2 represents the high-frequency gradient component of the second other channel in the vertical direction, V_grad represents the vertical gradient, abs(·) represents the absolute value calculation, posxy represents the pixel value of the pixel position at the xth row and yth column of the image, and a V , b V 、c V d V It is the gradient coefficient of the gradient components at different frequency levels in the vertical direction, and usually the sum is 1.

[0043] like Figure 6 The figure shows a schematic diagram of the principle of 45° diagonal gradient calculation provided by this embodiment. It should be noted that when calculating the diagonal gradient, only the pixel position in the upper left corner is considered. This is because in a flat area, the pixel information in the upper left corner is rich enough for accurate gradient calculation. In an area with more textures, the correlation in the lower right corner is not large, and the calculated gradient cannot accurately reflect the trend of the interpolation point in the diagonal direction, and the amount of calculation is large, so considering the lower right corner may be counterproductive. The specific calculation formula for the 45° diagonal gradient is expressed as follows:

[0044]

[0045] Among them, P_grad_high represents the high-frequency gradient component in the 45° diagonal direction, P_grad_other_high represents the high-frequency gradient component of other channels in the 45° diagonal direction, P_grad represents the 45° diagonal gradient, abs(·) represents the absolute value calculation, posxy represents the pixel value of the pixel position at the xth row and yth column of the image, and a P , b P is the gradient coefficient of the gradient component at different frequency levels in the 45° diagonal direction, a P The value of b P A positive integer multiple of, preferably, a P =8b P .

[0046] like Figure 7 FIG. 1 is a schematic diagram showing a principle of calculating a 135° diagonal gradient provided by this embodiment. The specific calculation formula is as follows:

[0047]

[0048] Among them, Q_grad_high represents the high-frequency gradient component in the 135° diagonal direction, Q_grad_other_high represents the high-frequency gradient component of other channels in the 135° diagonal direction, Q_grad represents the gradient in the 135° diagonal direction, abs(·) represents the absolute value calculation, posxy represents the pixel value of the pixel position at the xth row and yth column of the image, and a Q , b Q is the gradient coefficient of the gradient component at different frequency levels in the 135° diagonal direction, a Q The value of b Q A positive integer multiple of, preferably, a Q =8b Q .

[0049] In an optional implementation manner of the present embodiment, the above-mentioned steps of respectively calculating the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position include: searching leftward and rightward in the horizontal direction for two first associated pixel positions that are the same as and closest to the standard color channel of each non-standard pixel position, and calculating the horizontal interpolation estimation value based on the pixel values ​​of the two first associated pixel positions; wherein the standard color channel is the color channel of the three standard pixel positions in the four-in-one image area where the non-standard pixel position is located; searching upward and downward in the vertical direction for two second associated pixel positions that are the same as and closest to the standard color channel of each non-standard pixel position, and calculating the vertical interpolation estimation value based on the pixel values ​​of the two second associated pixel positions; calculating a 45° diagonal interpolation estimation value based on the pixel values ​​of two standard pixel positions at non-diagonal positions in the four-in-one image area where each non-standard pixel position is located, and / or calculating a 135° diagonal interpolation estimation value based on the pixel value of one standard pixel position at a diagonal position in the four-in-one image area where each non-standard pixel position is located.

[0050] like Figure 8 FIG. 1 is a schematic diagram showing a principle of calculating a horizontal interpolation estimation value provided by this embodiment. The specific calculation formula is as follows:

[0051] H_value=0.75*pos21+0.25*pos25;

[0052] Among them, H_value represents the horizontal interpolation estimate value, and posxy represents the pixel value of the pixel position at the xth row and yth column of the image.

[0053] like Fig. 9 FIG. 1 is a schematic diagram showing a principle of calculating a vertical interpolation estimation value provided by this embodiment. The specific calculation formula is as follows:

[0054] V_value=0.75*pos12+0.25*pos52;

[0055] Among them, V_value represents the vertical interpolation estimate value, and posxy represents the pixel value of the pixel position at the xth row and yth column of the image.

[0056] like Fig.10 FIG. 1 is a schematic diagram showing a principle for calculating a 45° diagonal interpolation estimate value provided by this embodiment. The specific calculation formula is as follows:

[0057] P_value=0.5*pos12+0.5*pos21;

[0058] Among them, P_value represents the 45° diagonal interpolation estimate, and posxy represents the pixel value of the pixel position at the xth row and yth column of the image.

[0059] like Fig.11 FIG. 1 is a schematic diagram showing a principle for calculating a 135° diagonal interpolation estimate value provided by this embodiment. The specific calculation formula is as follows:

[0060] Q_value=0.5*pos11+0.25*pos12+0.25*pos21;

[0061] Among them, Q_value represents the 135° diagonal interpolation estimate, and posxy represents the pixel value of the pixel position at the xth row and yth column of the image.

[0062] Step 302: Calculate corresponding weights in different directions based on the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position.

[0063] Specifically, after obtaining the gradient in each direction, the weight of the corresponding direction can be further calculated according to the gradient in each direction. The specific calculation formula is as follows:

[0064] wH=1.0 / (a*H_grad) 2 ;

[0065] wV=1.0 / (a*V_grad) 2 ;

[0066] wP=1.0 / (b*P_grad) 2 ;

[0067] wQ=1.0 / (b*Q_grad) 2 ;

[0068] Among them, wH represents the weight in the horizontal direction, H_grad represents the gradient in the horizontal direction, wV represents the weight in the vertical direction, V_grad represents the gradient in the vertical direction, wP represents the weight in the 45° diagonal direction, P_grad represents the gradient in the 45° diagonal direction, wQ represents the weight in the 135° diagonal direction, Q_grad represents the gradient in the 135° diagonal direction, a represents the first gradient proportional coefficient, and b represents the second gradient proportional coefficient.

[0069] Furthermore, in an optional implementation of the present embodiment, the gradient coefficients corresponding to the gradient components of the gradients in different directions and at different frequency levels may be obtained; then, the ratio of the first gradient proportional coefficient to the second gradient proportional coefficient may be determined based on the multiple gradient coefficients; and finally, the first gradient proportional coefficient and the second gradient proportional coefficient may be set based on the ratio.

[0070] Specifically, the gradient proportionality coefficient of this embodiment is used to balance the gradient magnitudes in various directions. The sum of multiple gradient proportionality coefficients can usually be 1. Preferably, the gradient proportionality coefficient can be based on the above-mentioned H、a V 、a P 、a Q The value of determines the ratio of b to a.

[0071] Step 303: Perform weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position according to the weights of the corresponding directions to obtain a final interpolation value.

[0072] Specifically, the weighted calculation method of this embodiment includes weighted average calculation or weighted sum calculation. Preferably, the weighted calculation formula of this embodiment can be expressed as follows:

[0073] Estimate=wH*H_value+wV*V_value+wP*P_value+wQ*Q_value;

[0074] Among them, Estimate represents the final interpolation, wH represents the weight in the horizontal direction, wV represents the weight in the horizontal direction, wP represents the weight in the 45° diagonal direction, wQ represents the weight in the 135° diagonal direction, H_value represents the horizontal interpolation estimate, V_value represents the vertical interpolation estimate, P_value represents the 45° diagonal interpolation estimate, and Q_value represents the 135° diagonal interpolation estimate.

[0075] Step 304: insert all final interpolation values ​​into corresponding non-standard pixel positions respectively to obtain a RAW format image after interpolation.

[0076] Specifically, the above calculation process can be executed for each non-standard pixel position, and the final interpolation values ​​of all non-standard pixel positions are calculated respectively. Then, the interpolation operation is performed based on the final interpolation values, and the interpolation of the entire Raw image can be completed.

[0077] It should be noted that, on the one hand, this embodiment combines the arrangement characteristics of Quad Bayer and the specificity of non-standard CFA, reasonably selects interpolation-related pixel points in the Raw domain, and extends the interpolation to the Raw domain, taking into account both low power consumption and effect; on the other hand, this embodiment uses other channel components except the interpolation component to improve the accuracy of gradient calculation; on the other hand, this embodiment performs gradient calculation from multiple directions such as horizontal, vertical and diagonal directions to obtain the final weighted estimate, which can still have a good interpolation effect on complex edge textures.

[0078] Fig.12 The method in is a refined non-standard CFA image interpolation method provided in an embodiment of the present application, which specifically includes the following process:

[0079] Step 1201: for each non-standard pixel position in the RAW format image, a target image area is correspondingly determined in the overall image area;

[0080] Step 1202: Select pixel values ​​at different target pixel positions in the target image region according to different selection strategies, and calculate horizontal gradient components at different frequency levels, vertical gradient components at different frequency levels, and diagonal gradient components at different frequency levels;

[0081] Step 1203, assigning corresponding gradient coefficients to the gradient components of different frequency levels in each direction;

[0082] Step 1204: Calculate the horizontal gradient by combining multiple horizontal gradient components of different frequency levels and corresponding gradient coefficients, calculate the vertical gradient by combining vertical gradient components of different frequency levels and corresponding gradient coefficients, and calculate the diagonal gradient by combining diagonal gradient components of different frequency levels and corresponding gradient coefficients;

[0083] Step 1205: Search leftward and rightward in the horizontal direction for two first associated pixel positions that have the same standard color channel as each non-standard pixel position and are closest to each other, and calculate a horizontal interpolation estimate based on the pixel values ​​of the two first associated pixel positions;

[0084] Step 1206: Search upward and downward in the vertical direction for two second associated pixel positions that have the same standard color channel as each non-standard pixel position and are closest to each other, and calculate a vertical interpolation estimate based on the pixel values ​​of the two second associated pixel positions;

[0085] Step 1207: Calculate a 45° diagonal interpolation estimate based on the pixel values ​​of two standard pixel positions at non-diagonal positions in the four-in-one image region where each non-standard pixel position is located, and / or calculate a 135° diagonal interpolation estimate based on the pixel value of one standard pixel position at a diagonal position in the four-in-one image region where each non-standard pixel position is located;

[0086] Step 1208, calculating corresponding weights in different directions based on the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position;

[0087] Step 1209: performing weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position according to the weights of the corresponding directions to obtain a final interpolation value;

[0088] Step 1210: insert all final interpolation values ​​into corresponding non-standard pixel positions respectively to obtain a RAW format image after interpolation.

[0089] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.

[0090] Fig.13 An image interpolation device for a non-standard CFA is provided in one embodiment of the present application. The image interpolation device for a non-standard CFA can be used to implement the image interpolation method for a non-standard CFA in the aforementioned embodiment, and mainly includes:

[0091] A gradient calculation module 1301 is used to respectively calculate the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image; wherein the RAW format image includes a plurality of four-in-one image regions, each of which includes a non-standard pixel position and three standard pixel positions of the same color channel, and the non-standard pixel position is a blank pixel position or has a color channel different from that of the standard pixel position;

[0092] An estimation module 1302, used to respectively calculate a horizontal interpolation estimation value, a vertical interpolation estimation value and a diagonal interpolation estimation value of each non-standard pixel position;

[0093] A weight calculation module 1303 is used to calculate the corresponding weights in different directions based on the horizontal direction gradient, vertical direction gradient and diagonal direction gradient of each non-standard pixel position;

[0094] An interpolation calculation module 1304 is used to perform weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position according to the weights of the corresponding directions to obtain a final interpolation value;

[0095] The interpolation module 1305 is used to insert all the final interpolation values ​​into the corresponding non-standard pixel positions to obtain a RAW format image after the interpolation is completed.

[0096] It should be noted that the non-standard CFA image interpolation methods in the aforementioned embodiments can all be implemented based on the non-standard CFA image interpolation device provided in this embodiment. Ordinary technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the non-standard CFA image interpolation device described in this embodiment can be implemented by referring to the corresponding working process in the aforementioned method embodiment, and will not be repeated here.

[0097] Based on the technical solution of the embodiment of the present application, the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image are calculated respectively, and the horizontal interpolation estimate, vertical interpolation estimate and diagonal interpolation estimate of each non-standard pixel position are calculated respectively; the corresponding weights of different directions are calculated based on the gradients of different directions of each non-standard pixel position respectively; the interpolation estimate of different directions of each non-standard pixel position is weightedly calculated according to the weight of the corresponding direction to obtain the final interpolation; all the final interpolations are inserted into the corresponding non-standard pixel positions respectively to obtain the interpolated RAW format image. Through the implementation of the solution of the present application, the interpolation is extended to the RAW domain taking into account the channel correlation, avoiding the poor effect of bilinear interpolation and the deficiency of excessive bicubic interpolation calculation, so that the interpolation efficiency and effect are well balanced, with better hardware friendliness.

[0098] Fig.14 An electronic device is provided for an embodiment of the present application. The electronic device can be used to implement the non-standard CFA image interpolation method in the aforementioned embodiment, and mainly includes: a memory 1401 and a processor 1402. The memory 1401 stores a computer program 1403 that can be run on the processor 1402. The memory 1401 and the processor 1402 are in communication connection. When the processor 1402 executes the computer program 1403, the non-standard CFA image interpolation method in the aforementioned embodiment is implemented. The number of processors 1402 can be one or more.

[0099] The memory 1401 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as a disk memory. The memory 1401 is used to store executable program codes, and the processor 1402 is coupled to the memory 1401 .

[0100] Furthermore, the present application also provides a computer-readable storage medium, which may be provided in the electronic device in the above embodiments. Fig.14 Memory in the illustrated embodiment.

[0101] The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the non-standard CFA image interpolation method in the aforementioned embodiment is implemented. Further, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0102] It should be understood that the devices and methods disclosed in the embodiments provided in the present application can also be implemented in any other equivalent manner. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0103] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0105] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0106] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0107] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] The above is a description of the non-standard CFA image interpolation method, device, equipment and storage medium provided in the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A non-standard CFA image interpolation method, characterized in that: Applied to an image sensor configured with a non-standard four-in-one Bayer array, the image interpolation method comprises: Calculating the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image respectively, and calculating the horizontal interpolation estimation value, vertical interpolation estimation value and diagonal interpolation estimation value of each non-standard pixel position respectively; wherein the RAW format image includes a plurality of four-in-one image areas, each of the four-in-one image areas includes one non-standard pixel position and three standard pixel positions of the same color channel, and the non-standard pixel position is a blank pixel position or has a color channel different from that of the standard pixel position; Calculating corresponding weights in different directions based on the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position respectively; Performing weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each of the non-standard pixel positions according to the weights in the corresponding directions to obtain a final interpolation value; All the final interpolation values ​​are respectively inserted into the corresponding non-standard pixel positions to obtain the RAW format image after interpolation.

2. The image interpolation method according to claim 1, characterized in that: The step of respectively calculating the horizontal gradient, the vertical gradient and the diagonal gradient of each non-standard pixel position in the RAW format image comprises: For each non-standard pixel position in the RAW format image, a target image area is correspondingly determined in the overall image area; The pixel value of the target pixel position is selected in the target image area to calculate the horizontal gradient, vertical gradient and diagonal gradient.

3. The image interpolation method according to claim 2, characterized in that: The step of selecting the pixel value of the target pixel position in the target image area to calculate the horizontal gradient, the vertical gradient and the diagonal gradient comprises: Select pixel values ​​of different target pixel positions in the target image area according to different selection strategies, and calculate horizontal gradient components of different frequency levels, vertical gradient components of different frequency levels, and diagonal gradient components of different frequency levels; The horizontal direction gradient components at different frequency levels, the vertical direction gradient components at different frequency levels, and the diagonal direction gradient components at different frequency levels are respectively combined to calculate the horizontal direction gradient, the vertical direction gradient, and the diagonal direction gradient accordingly.

4. The image interpolation method according to claim 3, characterized in that: The step of respectively combining the horizontal direction gradient components at different frequency levels, the vertical direction gradient components at different frequency levels, and the diagonal direction gradient components at different frequency levels to calculate the horizontal direction gradient, the vertical direction gradient, and the diagonal direction gradient accordingly comprises: The corresponding gradient coefficients are assigned to the gradient components of different frequency levels in each direction; The horizontal gradient is calculated by combining multiple horizontal gradient components of different frequency levels and the corresponding gradient coefficients, the vertical gradient is calculated by combining the vertical gradient components of different frequency levels and the corresponding gradient coefficients, and the diagonal gradient is calculated by combining the diagonal gradient components of different frequency levels and the corresponding gradient coefficients.

5. The image interpolation method according to claim 1, characterized in that: The step of respectively calculating the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each of the non-standard pixel positions comprises: Searching leftward and rightward in the horizontal direction for two first associated pixel positions that are the same as the standard color channel of each of the non-standard pixel positions and are closest to each other, and calculating a horizontal interpolation estimate based on the pixel values ​​of the two first associated pixel positions; wherein the standard color channel is the color channel of the three standard pixel positions in the four-in-one image area where the non-standard pixel position is located; Searching upward and downward in the vertical direction for two second associated pixel positions that have the same standard color channel as the non-standard pixel positions and are closest to each other, and calculating a vertical interpolation estimate based on the pixel values ​​of the two second associated pixel positions; A 45° diagonal interpolation estimate is calculated based on the pixel values ​​of two standard pixel positions at non-diagonal positions in the four-in-one image region where each of the non-standard pixel positions is located, and / or a 135° diagonal interpolation estimate is calculated based on the pixel value of one standard pixel position at a diagonal position in the four-in-one image region where each of the non-standard pixel positions is located.

6. The image interpolation method according to claim 1, characterized in that: The calculation formula of the weight is expressed as: wH=1.0 / (a*H_grad) 2 ; wV=1.0 / (a*V_grad) 2 ; wP=1.0 / (b*P_grad) 2 ; wQ=1.0 / (b*Q_grad) 2 ; Among them, wH represents the weight in the horizontal direction, H_grad represents the horizontal gradient, wV represents the weight in the vertical direction, V_grad represents the vertical gradient, wP represents the weight in the 45° diagonal direction, P_grad represents the 45° diagonal gradient, wQ represents the weight in the 135° diagonal direction, Q_grad represents the 135° diagonal gradient, a represents the first gradient proportional coefficient, and b represents the second gradient proportional coefficient.

7. The image interpolation method according to claim 6, characterized in that: Also includes: Obtaining gradient coefficients corresponding to gradient components of gradients in different directions and at different frequency levels; Determining a ratio of the first gradient proportionality coefficient to the second gradient proportionality coefficient based on the plurality of gradient coefficients; The first gradient scale factor and the second gradient scale factor are set based on the ratio.

8. An image interpolation device for non-standard CFA, characterized in that: Applicable to an image sensor configured with a non-standard four-in-one Bayer array, the image interpolation device comprises: A gradient calculation module, used to respectively calculate the horizontal gradient, vertical gradient and diagonal gradient of each non-standard pixel position in the RAW format image; wherein the RAW format image includes a plurality of four-in-one image areas, each of the four-in-one image areas includes one non-standard pixel position and three standard pixel positions of the same color channel, and the non-standard pixel position is a blank pixel position or has a color channel different from that of the standard pixel position; An estimation module, used for respectively calculating a horizontal interpolation estimation value, a vertical interpolation estimation value and a diagonal interpolation estimation value of each of the non-standard pixel positions; A weight calculation module, used for calculating corresponding weights in different directions based on the horizontal direction gradient, vertical direction gradient and diagonal direction gradient of each non-standard pixel position; An interpolation calculation module, used for performing weighted calculation on the horizontal interpolation estimation value, the vertical interpolation estimation value and the diagonal interpolation estimation value of each non-standard pixel position according to the weights in corresponding directions to obtain a final interpolation value; The interpolation module is used to insert all the final interpolations into the corresponding non-standard pixel positions to obtain the RAW format image after interpolation.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the non-standard CFA image interpolation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the non-standard CFA image interpolation method according to any one of claims 1 to 7 are implemented.

Images