Bayer image interpolation method, device, equipment and medium
By combining bilinear interpolation and chromatic aberration interpolation algorithms, and using horizontal and vertical gradients to determine interpolation weights, the interpolation anomaly problem in rapidly changing regions of the red and blue channels is solved, thereby improving image quality and detail.
Patent Information
- Application Number
- CN202610129238.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-15
AI Technical Summary
Existing techniques often fail to satisfy the assumption of constant color difference when interpolating image regions where the red and blue channels change rapidly, leading to abnormal interpolation results and affecting image quality and detail.
A combination of bilinear interpolation and color difference interpolation algorithms is adopted. The interpolation weights are determined based on the horizontal and vertical gradients. The initial green interpolation is weighted and fused to calculate the target green interpolation for red and blue pixels. The red and blue interpolation is determined based on the difference between the green pixels and the neighboring pixels.
It improves the accuracy of green interpolation calculated from red and blue pixels, avoids abnormal interpolation results, and enhances image quality and detail.
Smart Images
Figure CN122048638A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a Bayer image interpolation method, apparatus, device and medium. Background Technology
[0002] Most commercially available image sensors use the Bayer pattern format to acquire corresponding Bayer images. A key feature of Bayer images is that each pixel records only one of the three channels: red, green, and blue, arranged in a specific order. Red and blue information each account for 25%, and green information accounts for 50%. Interpolation (or de-mosaicing) aims to calculate the missing two color information for each pixel in the Bayer image, thus converting each pixel into complete red, green, and blue three-channel data.
[0003] Traditional interpolation methods assume constant color difference. However, for image regions where the red and blue channels change rapidly, this assumption is often not met, which may lead to abnormal interpolation results. These abnormalities can be easily amplified by subsequent processing steps in the Image Signal Processor (ISP), severely affecting the final image quality and detail. Summary of the Invention
[0004] This application provides a Bayer image interpolation method, apparatus, device, and medium that can interpolate Bayer images more accurately, thereby improving the image quality and detail of the final image.
[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, an interpolation method for Bayer images is provided. This method may include: for each first pixel in the Bayer image, weighted fusion of the initial green interpolation values of the first pixel calculated using bilinear interpolation and color difference interpolation algorithms, respectively, based on the interpolation weights of the first pixel, to obtain a target green interpolation value for the first pixel. The first pixel is a red pixel and a blue pixel in the Bayer image, and the interpolation weights are determined based on the horizontal and vertical gradients corresponding to the first pixel; determining another color interpolation value for the first pixel based on the target green interpolation value; and for each green pixel in the Bayer image, determining a red interpolation value and a blue interpolation value for the green pixel based on the pixel value of the green pixel, the difference between the pixel values of the green pixel and the pixel values of the neighboring pixels of the green pixel, and the corresponding target green interpolation value.
[0006] In one possible implementation, for each first pixel in the Bayer image, the initial green interpolation values of the first pixel calculated using bilinear interpolation and chromatic aberration interpolation algorithms are weighted and fused based on the interpolation weights of the first pixel to obtain the target green interpolation value of the first pixel. This includes: determining the horizontal and vertical gradients of the first pixel within a preset pixel range centered on the first pixel; determining the interpolation weights based on the horizontal and vertical gradients; calculating a first initial green interpolation value of the first pixel using bilinear interpolation and a second initial green interpolation value of the first pixel using chromatic aberration interpolation; and weighting and fusing the first and second initial green interpolation values based on the interpolation weights to obtain the target green interpolation value of the first pixel.
[0007] In another possible implementation, within a preset pixel range centered on the first pixel, the horizontal gradient and vertical gradient of the first pixel are determined, including: within the preset pixel range, determining the horizontal gradient of the first pixel based on the green horizontal gradients corresponding to each pixel in the same column as the first pixel in the horizontal neighborhood; and within the preset pixel range, determining the vertical gradient of the first pixel based on the green vertical gradients corresponding to each pixel in the same row as the first pixel in the vertical neighborhood.
[0008] In another possible implementation, within a preset pixel range centered on the first pixel, determining the horizontal and vertical gradients of the first pixel includes: within the preset pixel range, determining the horizontal gradient of the first pixel based on the pixel horizontal gradients between the horizontal neighboring pixels corresponding to each pixel in the same column as the first pixel; and within the preset pixel range, determining the vertical gradient of the first pixel based on the pixel vertical gradients between the vertical neighboring pixels corresponding to each pixel in the same row as the first pixel.
[0009] In another possible implementation, the interpolation weights are determined based on the horizontal and vertical gradients, including: determining the ratio between the difference between the horizontal and vertical gradients and the sum of the horizontal and vertical gradients as the interpolation weights.
[0010] In another possible implementation, the first initial green interpolation value of the first pixel is calculated based on the bilinear interpolation algorithm, including: obtaining the first horizontal initial green interpolation value based on the average pixel value of the horizontal neighboring pixels of the first pixel; and obtaining the first vertical initial green interpolation value based on the average pixel value of the vertical neighboring pixels of the first pixel.
[0011] In another possible implementation, the second initial green interpolation of the first pixel is calculated based on the color difference interpolation algorithm, including: obtaining the second horizontal initial green interpolation based on the average pixel value of the first pixel's horizontal neighboring pixels and the second gradient of the first pixel in the horizontal direction; and obtaining the second vertical initial green interpolation based on the average pixel value of the first pixel's vertical neighboring pixels and the second gradient of the first pixel in the vertical direction.
[0012] In another possible implementation, the first initial green interpolation and the second initial green interpolation are weighted and fused based on interpolation weights to obtain the target green interpolation of the first pixel. This includes: weighting and fusing the first initial green interpolation and the second initial green interpolation based on interpolation weights; verifying whether the fusion result is located between the maximum and minimum values in each neighboring green pixel of the first pixel; and determining the fusion result as the target green interpolation of the first pixel when the verification passes.
[0013] Secondly, an interpolation apparatus for a Bayer image is provided. The apparatus may include: a processing module, configured to, for each first pixel in the Bayer image, perform weighted fusion of the initial green interpolation values of the first pixel calculated based on bilinear interpolation and color difference interpolation algorithms respectively, based on the interpolation weights of the first pixel, to obtain a target green interpolation value for the first pixel, wherein the first pixel is a red pixel and a blue pixel in the Bayer image, and the interpolation weights are determined based on the horizontal and vertical gradients corresponding to the first pixel; determine another color interpolation value for the first pixel based on the target green interpolation value; and for each green pixel in the Bayer image, determine a red interpolation value and a blue interpolation value for the green pixel based on the pixel value of the green pixel and the difference between the pixel values of the green pixel and the pixel values of the neighboring pixels of the green pixel and the corresponding target green interpolation value.
[0014] In one possible implementation, the processing module is specifically used to determine the horizontal and vertical gradients of the first pixel within a preset pixel range centered on the first pixel; determine interpolation weights based on the horizontal and vertical gradients; calculate a first initial green interpolation value for the first pixel based on a bilinear interpolation algorithm, and calculate a second initial green interpolation value for the first pixel based on a color difference interpolation algorithm; and perform weighted fusion of the first initial green interpolation value and the second initial green interpolation value based on the interpolation weights to obtain the target green interpolation value for the first pixel.
[0015] In another possible implementation, the processing module is specifically used to determine the horizontal gradient of the first pixel within a preset pixel range based on the green horizontal gradients corresponding to each pixel in the same column as the first pixel in the horizontal neighborhood; and to determine the vertical gradient of the first pixel within a preset pixel range based on the green vertical gradients corresponding to each pixel in the same row as the first pixel in the vertical neighborhood.
[0016] In another possible implementation, the processing module is specifically used to determine the horizontal gradient of the first pixel within a preset pixel range based on the pixel horizontal gradient between the horizontal neighboring pixels corresponding to each pixel in the same column as the first pixel; and to determine the vertical gradient of the first pixel within a preset pixel range based on the pixel vertical gradient between the vertical neighboring pixels corresponding to each pixel in the same row as the first pixel.
[0017] In another possible implementation, the processing module is specifically used to determine the interpolation weights as the ratio between the difference between the horizontal and vertical gradients and the sum of the horizontal and vertical gradients.
[0018] In another possible implementation, the processing module is specifically used to obtain a first horizontal initial green interpolation based on the average pixel value of the horizontal neighboring pixels of the first pixel; and to obtain a first vertical initial green interpolation based on the average pixel value of the vertical neighboring pixels of the first pixel.
[0019] In another possible implementation, the processing module is specifically used to obtain a second horizontal initial green interpolation based on the average pixel value of the horizontal neighboring pixels of the first pixel and the second gradient of the first pixel in the horizontal direction; and to obtain a second vertical initial green interpolation based on the average pixel value of the vertical neighboring pixels of the first pixel and the second gradient of the first pixel in the vertical direction.
[0020] In another possible implementation, the processing module is specifically used to perform weighted fusion of the first initial green interpolation and the second initial green interpolation based on the interpolation weight; to verify whether the fusion result is located between the maximum value and the minimum value in each neighboring green pixel of the first pixel; and to determine that the fusion result is the target green interpolation of the first pixel when the verification passes.
[0021] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a communication interface. The memory and the communication interface are coupled to the processor, and the memory stores computer program code, including computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in any one of the first aspects above.
[0022] Fourthly, a computer-readable storage medium is provided that stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of the first aspects above.
[0023] Fifthly, a computer program product comprising computer program instructions is provided, which, when executed by a processor, cause the processor to perform the method as described in any one of the first aspects above.
[0024] In a sixth aspect, an apparatus (e.g., a system-on-a-chip) is provided, comprising a processor for supporting an electronic device in implementing the methods described in the first aspect above. In one possible design, the apparatus further comprises a memory for storing program instructions and data necessary for the electronic device. When the apparatus is a system-on-a-chip, it may be composed of chips or may include chips and other discrete devices.
[0025] In this embodiment, when interpolating red and blue pixels (i.e., the first pixel) in a Bayer image, initial green interpolation values can be calculated using bilinear interpolation and color difference interpolation algorithms, respectively. Then, based on interpolation weights determined by the horizontal and vertical gradients corresponding to the first pixel, the initial green interpolation values obtained by the two algorithms are weighted and fused to obtain the final target green interpolation value for the first pixel. Furthermore, another color interpolation value for the first pixel can be determined based on the target green interpolation value. For each green pixel in the Bayer image, the red and blue interpolation values of that green pixel can be determined based on the pixel value of the green pixel, the difference between the pixel values of its neighboring pixels and the corresponding target green interpolation value. In this way, by using interpolation weights determined by the horizontal and vertical gradients corresponding to the first pixel, the influence of the constant color difference assumption on the green interpolation calculation results can be adaptively adjusted, thereby improving the accuracy of the green interpolation values calculated for red and blue pixels. This avoids abnormal interpolation results when interpolating image regions where the red and blue channels change rapidly, thus improving the image quality and detail of the final image. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating a Bayer image interpolation method provided in an embodiment of this application.
[0027] Figure 2 This is a schematic diagram of an image block centered on a first pixel, provided in an embodiment of this application.
[0028] Figure 3 This is a schematic diagram of the structure of a Bayer image interpolation device provided in an embodiment of this application.
[0029] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that “ / ” means “or,” for example, A / B can mean A or B; “and / or” in the text is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can mean: A alone, A and B simultaneously, and B alone.
[0031] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0032] The terms "first" and "second" in the following embodiments of this application are for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0033] Most commercially available image sensors use the Bayer pattern format to acquire corresponding Bayer images. A key feature of Bayer images is that each pixel records only one of the three channels: red, green, and blue, arranged in a specific order. Red and blue information each account for 25%, and green information accounts for 50%. Interpolation (or de-mosaicing) aims to calculate the missing two color information for each pixel in the Bayer image, thus converting each pixel into complete red, green, and blue three-channel data.
[0034] Traditional interpolation methods assume constant color difference. However, for image regions where the red and blue channels change rapidly, this assumption is often not met, which may lead to abnormal interpolation results. These abnormalities can be easily amplified by subsequent processing steps in the Image Signal Processor (ISP), severely affecting the final image quality and detail.
[0035] To address this, this application provides an interpolation method for Bayer images. When performing interpolation calculations for red and blue pixels (i.e., the first pixel) in a Bayer image, the method calculates initial green interpolation values for the first pixel using both bilinear and color difference interpolation algorithms. Then, based on interpolation weights determined by the horizontal and vertical gradients corresponding to the first pixel, the initial green interpolation values obtained by the two algorithms are weighted and fused to obtain the final target green interpolation value for the first pixel. Furthermore, another color interpolation value for the first pixel can be determined based on the target green interpolation value. For each green pixel in the Bayer image, the red and blue interpolation values for that green pixel can be determined based on the pixel value of the green pixel, the difference between the pixel values of its neighboring pixels and the corresponding target green interpolation value. In this way, by using the interpolation weights determined based on the horizontal and vertical gradients corresponding to the first pixel, the influence of the constant color difference assumption on the green interpolation calculation results can be adaptively adjusted, thereby improving the accuracy of the green interpolation calculated for red and blue pixels. This avoids the occurrence of abnormal interpolation results when interpolating image regions where the red and blue channels change rapidly, and thus improves the image quality and detail of the final image.
[0036] The method provided in this application will now be described in detail with reference to the accompanying drawings.
[0037] like Figure 1 As shown, the interpolation method for the Bayer image may specifically include the following S101-S103.
[0038] S101. For each first pixel in the Bayer image, the initial green interpolation of the first pixel calculated based on the bilinear interpolation algorithm and the color difference interpolation algorithm is weighted and fused to obtain the target green interpolation of the first pixel.
[0039] In this embodiment, the first pixel refers to the red and blue pixels in the Bayer image. Specifically, for each red and blue pixel in the Bayer image, the corresponding target green interpolation can be calculated based on this step. In this embodiment, the interpolation weight of the first pixel is determined based on the horizontal and vertical gradients corresponding to the first pixel. It should be noted that in some possible implementations of this embodiment, before S101, the Bayer image can be acquired first, and preprocessing such as preprocessing based on ISP-related technologies can be performed on the Bayer image before executing the Bayer image interpolation method provided in this application.
[0040] For example, for each first pixel, the horizontal and vertical gradients of the first pixel can be determined within a preset pixel range centered on the first pixel. Then, the interpolation weights of the first pixel are determined based on the horizontal and vertical gradients. A first initial green interpolation value for the first pixel is calculated based on a bilinear interpolation algorithm, and a second initial green interpolation value for the first pixel is calculated based on a color difference interpolation algorithm. The first and second initial green interpolations can then be weighted and fused based on the interpolation weights to obtain the target green interpolation value for the first pixel. In this embodiment, the order in which the interpolation weights of the first pixel are determined, and the first and second initial green interpolations are calculated, is not restricted and can be configured according to actual conditions, as long as the interpolation weights, the first initial green interpolation value, and the second initial green interpolation value of the first pixel can be obtained separately, so that the first and second initial green interpolations can be weighted and fused based on the interpolation weights to obtain the target green interpolation value.
[0041] As an example, when determining the horizontal and vertical gradients of a first pixel within a preset pixel range centered on the first pixel, the horizontal gradient of the first pixel can be determined within this preset pixel range based on the green horizontal gradients of each pixel in the same column as the first pixel in its horizontal neighborhood. Similarly, within the preset pixel range, the vertical gradient of the first pixel can be determined based on the green vertical gradients of each pixel in the same row as the first pixel in its vertical neighborhood. In this way, the horizontal and vertical gradients of the first pixel can be accurately determined based on the green horizontal gradients of pixels in the same column and the green vertical gradients of pixels in the same row, ensuring that the obtained horizontal and vertical gradients accurately and comprehensively represent the magnitude of green differences in the horizontal and vertical directions within the neighborhood of the first pixel. This facilitates obtaining more accurate interpolation weights based on these horizontal and vertical gradients.
[0042] Specifically, among the pixels in the same row as the first pixel, if the pixel is a red or blue pixel, the horizontal gradient between the green pixels in its horizontal neighborhood can be directly used as the corresponding green horizontal gradient of that pixel. Conversely, if the pixel is green, the average of the green horizontal gradients of its horizontal neighborhood pixels (i.e., red or blue pixels) can be used as the corresponding green horizontal gradient of that pixel. For example, a weighted average of each green horizontal gradient can be calculated based on preset weights to obtain the horizontal gradient of the first pixel, and a weighted average of each green vertical gradient can be calculated based on preset weights to obtain the vertical gradient of the first pixel.
[0043] For example, assuming the first pixel is the blue pixel, and the Bayer image is arranged in a red-green-green-blue pattern, with a preset pixel range of 5×5 pixels, and using the top-left pixel of this preset pixel range as the origin (0,0), the row sequence as the x-axis, and the column sequence as the y-axis, a coordinate system is constructed as follows: Then, the first pixel is the center pixel with coordinates (2,2). Therefore, the green horizontal gradients corresponding to the red and blue pixels in the same column as the first pixel can be: , , Accordingly, the green horizontal gradients corresponding to the green pixels in each pixel located in the same column as the first pixel can be: , Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates. This is the green horizontal gradient corresponding to the corresponding pixel in the same column as the first pixel. Therefore, based on preset weights, a weighted average of the green horizontal gradients corresponding to each pixel can be calculated, thus obtaining the horizontal gradient of the first pixel. .like, .
[0044] Similarly, the green vertical gradients corresponding to the red and blue pixels in the same row as the first pixel can be: , , Accordingly, the green vertical gradients corresponding to the green pixels in each pixel located in the same row as the first pixel can be: , Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates. This is the green vertical gradient corresponding to the corresponding pixel in the same row as the first pixel. Therefore, based on preset weights, a weighted average of the green vertical gradients corresponding to each pixel can be calculated, thus obtaining the vertical gradient of the first pixel. .like, .
[0045] As another example, when determining the horizontal and vertical gradients of a first pixel within a preset pixel range centered on the first pixel, the horizontal gradient of the first pixel can be determined within this preset pixel range based on the pixel horizontal gradients between the horizontal neighboring pixels corresponding to each pixel in the same column as the first pixel, and the vertical gradient of the first pixel can be determined based on the pixel vertical gradients between the vertical neighboring pixels corresponding to each pixel in the same row as the first pixel. In this way, the horizontal and vertical gradients of the first pixel can be determined based on the pixel horizontal gradients of pixels in the same column and the pixel vertical gradients of pixels in the same row as the first pixel. This calculation method involves fewer pixels, thus reducing computational complexity and improving the computational efficiency of obtaining the horizontal and vertical gradients of the first pixel. This, in turn, improves the execution efficiency of the Bayer image interpolation method provided in this application, thereby reducing the latency of the system processing.
[0046] For example, the maximum value of the horizontal gradient of each pixel can be used as the horizontal gradient of the first pixel, and the maximum value of the vertical gradient of each pixel can be used as the vertical gradient of the first pixel.
[0047] For example, assuming the first pixel is the blue pixel, and the Bayer image is arranged in a red-green-green-blue pattern, with a preset pixel range of 3×3 pixels, and using the top-left pixel of this preset pixel range as the origin (0,0), the row sequence as the x-axis, and the column sequence as the y-axis, a coordinate system is constructed. Then, the first pixel, as the center pixel, has coordinates (2,2). Therefore, the horizontal gradients of each pixel in the same column as the first pixel can be: , , Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates. This is the horizontal gradient of the pixel corresponding to the first pixel in the same column. Therefore, the maximum value among the horizontal gradients of all pixels can be used as the horizontal gradient of the first pixel. .Right now, .
[0048] Similarly, the vertical gradient of each pixel in the same row as the first pixel can be: , , Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates. This is the vertical gradient of the pixel corresponding to the first pixel in the same row. Therefore, the maximum value among the vertical gradients of all pixels can be used as the vertical gradient of the first pixel. .Right now, .
[0049] Based on the foregoing example, in some possible implementations, the interpolation weight of the first pixel can be determined based on the ratio of the difference between the horizontal and vertical gradients of the first pixel to their sum. That is, the interpolation weight... The formula can be satisfied: .in, The horizontal gradient of the first pixel. Let be the vertical gradient of the first pixel. Thus, a larger interpolation weight indicates a greater difference between the horizontal and vertical directions at the first pixel's position, making horizontal or vertical lines more likely to exist; conversely, a smaller interpolation weight has the opposite effect. Therefore, the interpolation weight obtained in this way can accurately characterize the impact of the constant color difference assumption on the green interpolation calculation, facilitating the subsequent obtaining of a more accurate green interpolation based on this weight.
[0050] In this embodiment of the application, the method of calculating the first initial green interpolation value of the first pixel based on the bilinear interpolation algorithm can be specifically to obtain the first horizontal initial green interpolation value based on the average pixel value of the horizontal neighboring pixels of the first pixel, and to obtain the first vertical initial green interpolation value based on the average pixel value of the vertical neighboring pixels of the first pixel.
[0051] For example, assuming the first pixel is the blue pixel, and the Bayer image is arranged in a red-green-green-blue pattern, then... Figure 2 The image shown is a 5×5 pixel patch centered on the first pixel (pixel B). Taking the top-left pixel of this patch as the origin (0,0), the row sequence as the x-axis, and the column sequence as the y-axis, the first pixel as the center pixel has coordinates (2,2). In this image patch, the neighboring green pixels of the first pixel include pixels G1, G2, G3, and G4; the neighboring red pixels of the first pixel include pixels R1, R2, R3, and R4; and the neighboring blue pixels of the first pixel include pixels B1, B2, B3, and B4.
[0052] based on Figure 2 The image patch shown has a first horizontal initial green interpolation for the first pixel. The formula can be satisfied: ,Right now Accordingly, the first vertical initial green interpolation of the first pixel. The formula can be satisfied: ,Right now Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates.
[0053] In this embodiment, the method of calculating the second initial green interpolation value of the first pixel based on the color difference interpolation algorithm can specifically be as follows: A second horizontal initial green interpolation value is obtained based on the average pixel value of the first pixel's horizontal neighboring pixels and the second-order gradient of the first pixel in the horizontal direction. Similarly, a second vertical initial green interpolation value is obtained based on the average pixel value of the first pixel's vertical neighboring pixels and the second-order gradient of the first pixel in the vertical direction. This allows for a relatively accurate estimation of the initial green interpolation value of the first pixel based on its neighboring green pixel values and second-order gradient, improving the accuracy of the final target green interpolation value.
[0054] For example, continue with, as Figure 2 Taking the image patch shown as an example, then based on... Figure 2 The image patch shown has a second-level initial green interpolation for the first pixel. The formula can be satisfied: Accordingly, the first vertical initial green interpolation of the first pixel. The formula can be satisfied: .in, Let be the second-order gradient of the first pixel in the horizontal direction. , Let be the second-order gradient of the first pixel in the vertical direction. Therefore, taking the top-left pixel of the image patch as the origin (0,0), the row sequence as the x-axis, and the column sequence as the y-axis as an example, then... , Among them, the above-mentioned This represents the pixel value corresponding to the pixel at the given coordinates.
[0055] Based on the aforementioned example, after calculating the interpolation weights and the first and second initial green interpolations of the first pixel, the first and second initial green interpolations can be weighted and fused based on the interpolation weights to obtain the final target green interpolation of the first pixel.
[0056] In some possible implementations, the first horizontal initial green interpolation and the second horizontal initial green interpolation can be weighted and fused based on interpolation weights to obtain the horizontal target green interpolation, and the first vertical initial green interpolation and the second vertical initial green interpolation can be weighted and fused to obtain the vertical target green interpolation. Then, the average of the horizontal and vertical target green interpolations is used as the final target green interpolation for the first pixel. That is, the horizontal target green interpolation... The formula can be satisfied: For example, based on the aforementioned example... and ,but Vertical target green interpolation The formula can be satisfied: For example, based on the aforementioned example... and ,but Therefore, the final target green interpolation can be obtained. .
[0057] Of course, in some other possible implementations, the average of the first horizontal initial green interpolation and the first vertical initial green interpolation, as well as the average of the second horizontal initial green interpolation and the second vertical initial green interpolation, can be calculated first. Then, based on the aforementioned weighted fusion method, the two averages can be weighted and fused according to the interpolation weight to obtain the target green interpolation. No limitation is made here.
[0058] In this embodiment of the application, as a possible implementation, after obtaining the fusion result through the aforementioned weighted fusion, the fusion result can be verified to be between the maximum and minimum values of the green pixels in each neighboring area of the first pixel. If the verification passes, the fusion result is then determined to be the target green interpolation of the first pixel. That is, the target green interpolation. The formula can be satisfied: ,in, The minimum value among all neighboring green pixels of the first pixel, for example, using Figure 2 Taking the image block shown as an example... , The maximum value of the green pixels in the neighborhood of the first pixel, for example, using Figure 2 Taking the image block shown as an example... .
[0059] In this way, the obtained target green interpolation can be limited to the pixel value range of the neighboring green pixels of the first pixel through verification, thus avoiding obtaining abnormal target green interpolation.
[0060] S102. Based on the target green interpolation, determine another color interpolation for the first pixel.
[0061] The other color interpolation for the first pixel is the interpolation corresponding to the other color (i.e., the other color to be interpolated) among the two missing color information of the first pixel, excluding green. For example, when the first pixel is a red pixel, the other color interpolation for the first pixel is the blue interpolation; when the first pixel is a blue pixel, the other color interpolation for the first pixel is the red interpolation.
[0062] In this embodiment, as an example, another color interpolation for the first pixel can be calculated based on the target green interpolation and the pixel values of the neighboring color pixels to be interpolated. For example, within a 3×3 pixel region centered on the first pixel, a coordinate system is constructed with the top-left pixel of this region as the origin (0,0), the row sequence as the horizontal coordinate, and the column sequence as the vertical coordinate. The first pixel, as the center pixel, has coordinates (1,1). Based on the arrangement characteristics of the Bayer array, the coordinates of the neighboring color pixels to be interpolated for the first pixel are (0,0), (0,2), (2,0), and (2,2). Therefore, the other color interpolation C for the first pixel can satisfy the formula: Where D is the difference between the pixel value of the pixel at the corresponding coordinate and the target green interpolation corresponding to that coordinate, i.e., D0 , This refers to the pixel value corresponding to the corresponding coordinates. Interpolate the target green color for the pixel at the corresponding coordinates.
[0063] Of course, in the embodiments of this application, the above is only an example of a specific implementation of determining another color interpolation of the first pixel based on the target green interpolation (i.e., the green interpolation of the first pixel). In practical applications, any related method for calculating the interpolation of another color other than green corresponding to the red or blue pixel based on the assumption of constant color difference can also be used, and no limitation is made here.
[0064] S103. For each green pixel in the Bayer image, determine the red and blue interpolations of the green pixel based on the pixel value of the green pixel and the difference between the pixel values of the neighboring pixels of the green pixel and the corresponding target green interpolation.
[0065] For example, the red interpolation of a green pixel can be calculated based on the pixel value of the green pixel and the difference between the pixel values of its neighboring red pixels and their corresponding target green interpolation. Similarly, the blue interpolation of a green pixel can be calculated based on the pixel value of the green pixel and the difference between the pixel values of its neighboring blue pixels and their corresponding target green interpolation.
[0066] For example, within a 3×3 pixel region centered on the green pixel to be interpolated, a coordinate system is constructed with the top-left pixel of this region as the origin (0,0), the row sequence as the x-coordinate, and the column sequence as the y-coordinate. In this case, the green pixel, as the center pixel, has coordinates (1,1). Based on the arrangement characteristics of the Bayer array, the neighboring red pixels of this green pixel can be located horizontally, and the corresponding neighboring blue pixels can be located vertically; or, the neighboring red pixels of this green pixel can be located vertically, and the corresponding neighboring blue pixels can be located horizontally. Therefore, when the green pixel is adjacent to a red pixel horizontally and to a blue pixel vertically, its red interpolation R can satisfy the formula: Correspondingly, its blue interpolation B satisfies the formula: When the green pixel is adjacent to a blue pixel horizontally and to a red pixel vertically, its red interpolation R satisfies the formula: Correspondingly, its blue interpolation B satisfies the formula: Where G is the pixel value of the green pixel at the corresponding coordinate (i.e., the green pixel to be interpolated), and D is the difference between the pixel value of the pixel at the corresponding coordinate and the target green interpolation corresponding to that coordinate, i.e., D0. , This refers to the pixel value corresponding to the corresponding coordinates. Interpolate the target green color for the pixel at the corresponding coordinates.
[0067] Of course, in the embodiments of this application, the above is only an example of a specific implementation method for determining the red interpolation and blue interpolation of a green pixel based on the pixel value of the green pixel (i.e. the pixel value of the green pixel to be interpolated) and the pixel values of its neighboring pixels. In practical applications, any related methods for calculating the red interpolation and blue interpolation corresponding to the green pixel can also be used, and no restrictions are imposed here.
[0068] It should be noted that, based on the method provided in the embodiments of this application, after obtaining the target green interpolation and blue interpolation corresponding to each red pixel, the target green interpolation and red interpolation corresponding to each blue pixel, and the red interpolation and blue interpolation corresponding to each green pixel in the Bayer image, the Bayer image can be reconstructed into a full-color image through subsequent processing.
[0069] In this embodiment, the execution order of S102 and S103 is not specifically limited. In practical applications, it can be configured according to the actual situation. For example, the interpolation values of the two colors for the missing non-green pixels can be obtained first based on S101 and S102, and then the red and blue interpolation values for each green pixel can be obtained based on S103. Alternatively, after obtaining the green interpolation value (i.e., the target green interpolation value) for each non-green pixel based on S101, the red and blue interpolation values for each green pixel can be obtained first based on S103, and then the interpolation value of the other color for the missing non-green pixel can be obtained based on S102. Alternatively, the related processing of S102 and S103 can be executed in parallel according to the execution logic.
[0070] In summary, the Bayer image interpolation method provided in this application can adaptively adjust the influence of the constant color difference assumption on the green interpolation calculation result by using interpolation weights determined based on the horizontal and vertical gradients corresponding to the first pixel. This improves the accuracy of the green interpolation calculated for red and blue pixels, avoids abnormal interpolation results when interpolating image regions where the red and blue channels change rapidly, and ultimately improves the image quality and detail of the final image.
[0071] Based on the examples given above, those skilled in the art will obviously be able to make various equivalent modifications or changes, which also fall within the scope of the embodiments of this application.
[0072] It is understood that, in order to achieve the above-mentioned functions, electronic devices may include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0073] This application embodiment can divide the above method examples into functional modules to obtain an apparatus capable of implementing the above method examples. For example, Figure 3 A schematic diagram of a Bayer image interpolation device provided in an embodiment of this application is shown. Figure 3As shown, the device may include: a processing module 301, configured to, for each first pixel in the Bayer image, perform weighted fusion of the initial green interpolation of the first pixel calculated based on the bilinear interpolation algorithm and the color difference interpolation algorithm respectively, based on the interpolation weight of the first pixel, to obtain the target green interpolation of the first pixel, wherein the first pixel is a red pixel and a blue pixel in the Bayer image, and the interpolation weight is determined based on the horizontal gradient and vertical gradient corresponding to the first pixel; determine another color interpolation of the first pixel based on the target green interpolation; and for each green pixel in the Bayer image, determine the red interpolation and blue interpolation of the green pixel based on the pixel value of the green pixel and the difference between the pixel value of the green pixel and the pixel value of the neighboring pixels of the green pixel and the corresponding target green interpolation.
[0074] In one possible implementation, the processing module 301 is specifically used to determine the horizontal gradient and vertical gradient of the first pixel within a preset pixel range centered on the first pixel; determine the interpolation weights based on the horizontal gradient and vertical gradient; calculate the first initial green interpolation value of the first pixel based on the bilinear interpolation algorithm, and calculate the second initial green interpolation value of the first pixel based on the color difference interpolation algorithm; and perform weighted fusion of the first initial green interpolation value and the second initial green interpolation value based on the interpolation weights to obtain the target green interpolation value of the first pixel.
[0075] In another possible implementation, the processing module 301 is specifically used to determine the horizontal gradient of the first pixel within a preset pixel range based on the green horizontal gradients corresponding to each pixel in the same column as the first pixel in the horizontal neighborhood; and to determine the vertical gradient of the first pixel within a preset pixel range based on the green vertical gradients corresponding to each pixel in the same row as the first pixel in the vertical neighborhood.
[0076] In another possible implementation, the processing module 301 is specifically used to determine the horizontal gradient of the first pixel within a preset pixel range based on the pixel horizontal gradient between the horizontal neighboring pixels corresponding to each pixel in the same column as the first pixel; and to determine the vertical gradient of the first pixel within a preset pixel range based on the pixel vertical gradient between the vertical neighboring pixels corresponding to each pixel in the same row as the first pixel.
[0077] In another possible implementation, the processing module 301 is specifically used to determine the interpolation weight as the ratio between the difference between the horizontal gradient and the vertical gradient and the sum of the horizontal gradient and the vertical gradient.
[0078] In another possible implementation, the processing module 301 is specifically used to obtain a first horizontal initial green interpolation based on the average pixel value of the horizontal neighboring pixels of the first pixel; and to obtain a first vertical initial green interpolation based on the average pixel value of the vertical neighboring pixels of the first pixel.
[0079] In another possible implementation, the processing module 301 is specifically used to obtain a second horizontal initial green interpolation based on the average pixel value of the horizontal neighboring pixels of the first pixel and the second gradient of the first pixel in the horizontal direction; and to obtain a second vertical initial green interpolation based on the average pixel value of the vertical neighboring pixels of the first pixel and the second gradient of the first pixel in the vertical direction.
[0080] In another possible implementation, the processing module 301 is specifically used to perform weighted fusion of the first initial green interpolation and the second initial green interpolation based on the interpolation weight; to verify whether the fusion result is located between the maximum value and the minimum value in each neighboring green pixel of the first pixel; and to determine that the fusion result is the target green interpolation of the first pixel when the verification passes.
[0081] This application also provides an electronic device, which can be any electronic device equipped with an image sensor. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Figure 4 As shown, the electronic device includes a bus 401, a processor 402, a memory 403, and a communication interface 404. The processor 402, memory 403, and communication interface 404 communicate via the bus 401. The memory 403 stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor 402, the electronic device performs the method provided in the foregoing embodiments. It should be understood that this application does not limit the number of processors 402 and memories 403 in the electronic device.
[0082] Bus 401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 401 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus 401 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 401 may include a path for transmitting information between various components of an electronic device (e.g., memory 403, processor 402, communication interface 404).
[0083] Processor 402 may include any one or more processors such as a central processing unit, a graphics processing unit, a microprocessor (MP), or a digital signal processor (DSP).
[0084] Memory 403 may include volatile memory, such as random access memory (RAM). Memory 403 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0085] Communication interface 404 uses a command distribution module, such as, but not limited to, a network interface card or transceiver, to enable communication between electronic devices and other devices or communication networks.
[0086] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, perform the method provided in the foregoing embodiments.
[0087] Furthermore, in the embodiments of this application, the functional units or modules of the apparatus used to implement the aforementioned method example can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0089] Therefore, this application also provides a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the method provided in the foregoing embodiments.
[0090] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor).
[0091] For example, the interface circuit can read instructions stored in the memory and send those instructions to the processor. When the processor executes the instructions, it causes the chip system to perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.
[0092] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A Bayer image interpolation method, characterized in that, include: For each first pixel in the Bayer image, the initial green interpolation of the first pixel calculated using the bilinear interpolation algorithm and the color difference interpolation algorithm are weighted and fused based on the interpolation weight of the first pixel to obtain the target green interpolation of the first pixel. The first pixel is the red pixel and the blue pixel in the Bayer image. The interpolation weight is determined based on the horizontal gradient and the vertical gradient corresponding to the first pixel. Based on the target green interpolation, determine another color interpolation for the first pixel; For each green pixel in the Bayer image, the red and blue interpolations of the green pixel are determined based on the pixel value of the green pixel and the difference between the pixel values of the neighboring pixels of the green pixel and the corresponding target green interpolation.
2. The method according to claim 1, characterized in that, For each first pixel in the Bayer image, the initial green interpolation values of the first pixel calculated using bilinear interpolation and chromatic aberration interpolation algorithms are weighted and fused based on the interpolation weights of the first pixel to obtain the target green interpolation value of the first pixel, including: Within a preset pixel range centered on the first pixel, the horizontal gradient and the vertical gradient of the first pixel are determined; The interpolation weights are determined based on the horizontal gradient and the vertical gradient; The first initial green interpolation value of the first pixel is calculated based on the bilinear interpolation algorithm, and the second initial green interpolation value of the first pixel is calculated based on the color difference interpolation algorithm. The first initial green interpolation and the second initial green interpolation are weighted and fused based on the interpolation weights to obtain the target green interpolation of the first pixel.
3. The method according to claim 2, characterized in that, Determining the horizontal gradient and the vertical gradient of the first pixel within a preset pixel range centered on the first pixel includes: Within the preset pixel range, the horizontal gradient of the first pixel is determined based on the green horizontal gradient corresponding to each pixel in the same column as the first pixel in the horizontal neighborhood. Within the preset pixel range, the vertical gradient of the first pixel is determined based on the green vertical gradient corresponding to each pixel in the vertical neighborhood of the first pixel.
4. The method according to claim 2, characterized in that, Determining the horizontal gradient and the vertical gradient of the first pixel within a preset pixel range centered on the first pixel includes: Within the preset pixel range, the horizontal gradient of the first pixel is determined based on the pixel horizontal gradient between the horizontal neighboring pixels corresponding to each pixel in the same column as the first pixel. Within the preset pixel range, the vertical gradient of the first pixel is determined based on the vertical gradient between the vertical neighboring pixels corresponding to each pixel in the same row as the first pixel.
5. The method according to claim 2, characterized in that, Determining the interpolation weights based on the horizontal gradient and the vertical gradient includes: The interpolation weight is determined by the ratio between the difference between the horizontal gradient and the vertical gradient and the sum of the horizontal gradient and the vertical gradient.
6. The method according to claim 2, characterized in that, The calculation of the first initial green interpolation value of the first pixel based on the bilinear interpolation algorithm includes: The first horizontal initial green interpolation is obtained based on the average pixel value of the horizontal neighboring pixels of the first pixel; The first vertical initial green interpolation is obtained based on the average pixel value of the vertical neighboring pixels of the first pixel.
7. The method according to claim 2, characterized in that, The calculation of the second initial green interpolation value for the first pixel based on the color difference interpolation algorithm includes: The second horizontal initial green interpolation is obtained based on the average pixel value of the horizontal neighboring pixels of the first pixel and the second gradient of the first pixel in the horizontal direction. The second vertical initial green interpolation is obtained based on the average pixel value of the vertical neighboring pixels of the first pixel and the second-order gradient of the first pixel in the vertical direction.
8. The method according to claim 2, characterized in that, The step of weightedly fusing the first initial green interpolation and the second initial green interpolation based on the interpolation weights to obtain the target green interpolation of the first pixel includes: The first initial green interpolation and the second initial green interpolation are weighted and fused based on the interpolation weights; Based on the maximum and minimum values of the green pixels in each neighboring area of the first pixel, the fusion result is checked to see if it is between the maximum and minimum values. If the verification passes, the fusion result is determined to be the target green interpolation of the first pixel.
9. An interpolation device for Bayer images, characterized in that, include: The processing module is configured to, for each first pixel in the Bayer image, perform weighted fusion of the initial green interpolation values of the first pixel calculated using bilinear interpolation and color difference interpolation algorithms, respectively, based on the interpolation weights of the first pixel, to obtain a target green interpolation value for the first pixel, wherein the first pixel is a red pixel and a blue pixel in the Bayer image, and the interpolation weights are determined based on the horizontal and vertical gradients corresponding to the first pixel; determine another color interpolation value for the first pixel based on the target green interpolation value; and for each green pixel in the Bayer image, determine the red and blue interpolation values of the green pixel based on the pixel value of the green pixel and the difference between the pixel value of the green pixel and the pixel value of the neighboring pixels of the green pixel and the corresponding target green interpolation value.
10. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a communication interface; the memory and the communication interface are coupled to the processor, the memory is used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the method as described in any one of claims 1 to 8.