An image array format conversion method, device, electronic device and storage medium

Abstract

The present application discloses an image array format conversion method, apparatus, an electronic device, and a computer-readable storage medium. The method includes: obtaining a CFA array mixed by a first image sensor and a second image sensor as an image array to be converted; constructing an intermediate Bayer array based on the distribution of target pixel points in the image array to be converted; respectively calculating the other two color components of each target pixel point based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array to obtain a first target array; calculating the three color components of each D pixel point based on the three color components of the adjacent pixel points in the first target array to obtain a second target array; and obtaining a target Bayer array based on the second target array. The present application realizes the conversion of the mixed CFA array into a Bayer array with the same resolution size as the original image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. More specifically, it relates to an image array format conversion method, an apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] APS (Active Pixel Sensor) is a traditional image sensor, and DVS (Dynamic Vision Sensor) is a new type of image sensor. Different from traditional image sensors, DVS is an event-based camera. When the light intensity change exceeds a certain threshold, events of getting stronger or weaker will be generated. DVS has the advantages of low illuminance, wide dynamic range, and fast response.

[0003] Due to the increasing real-world demands, there are more and more application scenarios that combine the two image sensors. Alpix003 designs DVS and APS in the same CFA (Color Filter Array) array, making the same-color pixels of the two sensors closer, which is more conducive to super-resolution, deblurring, and high-speed imaging, and can better solve the problems of image reconstruction and synchronization.

[0004] In order to make Alpix003 a general image sensor, it is a technical problem that those skilled in the art need to solve how to convert the CFA array mixed with APS and DVS into a Bayer array (Bayer pattern) with the same resolution size as the original image. Summary of the Invention

[0005] The purpose of the present application is to provide an image array format conversion method, an apparatus, an electronic device, and a computer-readable storage medium, which realizes the conversion of the mixed CFA array into a Bayer array with the same resolution size as the original image.

[0006] To achieve the above purpose, the present application provides an image array format conversion method, including:

[0007] Obtain a CFA array mixed with a first image sensor and a second image sensor as an image array to be converted; wherein, the image array to be converted includes target pixel points with one color component among R, G, and B and D pixel points;

[0008] Without considering the D pixel points in the conversion image array, construct an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted;

[0009] Based on the color components of the adjacent pixels of each of the target pixel points in the intermediate Bayer array, calculate the other two color components of each of the target pixel points to obtain a first target array;

[0010] According to the three color components of the adjacent pixels of each D pixel point in the first target array, calculate the three color components of each D pixel point to obtain a second target array;

[0011] Based on the second target array to obtain a target Bayer array.

[0012] Wherein, the constructing the intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted includes:

[0013] Construct the i-th intermediate Bayer array based on the distribution of the target pixel points in the (i + 2n)-th column of the array to be converted; wherein, 1 ≤ i ≤ m, m is the total number of intermediate Bayer arrays, and n is an integer greater than or equal to 0.

[0014] Wherein, the calculating the other two color components of each of the target pixel points based on the color components of the adjacent pixels of each of the target pixel points in the intermediate Bayer array includes:

[0015] Perform linear interpolation on the intermediate Bayer array in the vertical and horizontal directions respectively to calculate the R component and B component of the target pixel point;

[0016] Perform bilinear interpolation on the intermediate Bayer array to calculate the G component of the target pixel point.

[0017] Wherein, obtaining the first target array includes:

[0018] According to the other two color components of each of the target pixel points calculated, fill the intermediate Bayer array to obtain the first target array.

[0019] Wherein, after obtaining the first target array, it further includes:

[0020] Use the median filtering algorithm to correct the color components of the target pixel points in the first target array.

[0021] Wherein, the calculating the three color components of each D pixel point according to the three color components of the adjacent pixels in the first target array includes:

[0022] According to the other two color components of each of the target pixel points calculated in the first target array, fill the image array to be converted to obtain a third target array;

[0023] Perform bilinear interpolation on the third target array to calculate the three color components of each D pixel point.

[0024] Among them, obtaining the second target array includes:

[0025] Fill the image array to be converted based on the three color components of each calculated D pixel point to obtain the second target array.

[0026] Among them, after obtaining the second target array, it further includes:

[0027] Use the median filtering algorithm to correct the color components of the target pixel points in the second target array.

[0028] Among them, the use of the median filtering algorithm to correct the color components of the target pixel points in the second target array includes:

[0029] Perform bilinear interpolation on the corrected second target array to correct the color components of the D pixel points.

[0030] Among them, obtaining the target Bayer array based on the second target array includes:

[0031] Determine the target arrangement of the target Bayer array;

[0032] Extract the required color components of each pixel point in the second target array based on the target arrangement to obtain the target Bayer array.

[0033] To achieve the above object, the present application provides an image array format conversion device, including:

[0034] An acquisition module for acquiring a CFA array mixed with a first image sensor and a second image sensor as an image array to be converted; wherein, the image array to be converted includes target pixel points and D pixel points with one color component among R, G, and B;

[0035] A construction module for constructing an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted;

[0036] A calculation module, based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array, calculates the other two color components of each target pixel point respectively to obtain a first target array, and calculates the three color components of each D pixel point according to the three color components of the adjacent pixel points in the first target array to obtain a second target array;

[0037] A conversion module for obtaining a target Bayer array based on the second target array.

[0038] Among them, the first image sensor is APS, and the second image sensor is DVS.

[0039] To achieve the above object, the present application provides an electronic device, including:

[0040] A memory for storing a computer program;

[0041] A processor for implementing the steps of the above image array format conversion method when executing the computer program.

[0042] To achieve the above object, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above image array format conversion method are implemented.

[0043] As can be seen from the above solution, an image array format conversion method provided by the present application includes: obtaining a CFA array mixed with a first image sensor and a second image sensor as an image array to be converted; among them, the image array to be converted includes target pixel points and D pixel points having one color component among R, G, and B; constructing an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted; respectively calculating the other two color components of each target pixel point based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array to obtain a first target array; calculating the three color components of each D pixel point according to the three color components of the adjacent pixel points of each D pixel point in the first target array to obtain a second target array; obtaining a target Bayer array based on the second target array.

[0044] The image array format conversion method provided by this application first splits the target pixel points with one color component among R, G, and B in the image array to be converted into two intermediate Bayer arrays, calculates the other two color components of each target pixel point in the intermediate Bayer arrays using the linear interpolation algorithm, and then calculates the three color components of the D pixel points again using linear interpolation based on the three color components of the target pixel points, thereby obtaining the three color components of each pixel point in the image array to be converted. Finally, based on the three color components of each pixel point, the image array to be converted is converted into a target Bayer array. Since the three color components of each pixel point in the image array to be converted are known, the image resolution of the target Bayer array can be the same as that of the image array to be converted. It can be seen that this application realizes the conversion of a mixed CFA array into a Bayer array with the same size as the original image resolution, making the Alpix003 with a mixed APS and DVS CFA array more versatile. This application also discloses an image array format conversion device, an electronic device, and a computer-readable storage medium, which can also achieve the above technical effects.

[0045] It should be understood that the above general description and the following detailed description are only exemplary and do not limit this application. Description of the Drawings

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification, together with the following specific embodiments to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0047] Figure 1 It is a flowchart of an image array format conversion method shown according to an exemplary embodiment;

[0048] Figures 2a - 2h They are eight arrangement ways of a CFA array with a mixed APS and DVS;

[0049] Figure 3a It is an image array to be converted shown according to an exemplary embodiment;

[0050] Figure 3b and Figure 3c are Figure 3a the corresponding first intermediate Bayer array and second intermediate Bayer array;

[0051] Figure 4An intermediate Bayer array shown according to an exemplary embodiment;

[0052] Figure 5 A third target array shown according to an exemplary embodiment;

[0053] Figure 6 A second target array shown according to an exemplary embodiment;

[0054] Figures 7a - 7d Four arrangement modes of the Bayer array;

[0055] Figure 8 A structural diagram of an image array format conversion device shown according to an exemplary embodiment;

[0056] Figure 9 A structural diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence.

[0058] The embodiments of the present application disclose an image array format conversion method, which realizes converting a mixed CFA array into a Bayer array with the same resolution size as the original image.

[0059] See Figure 1 , a flowchart of an image array format conversion method shown according to an exemplary embodiment, as Figure 1 shown, includes:

[0060] S101: Obtain a CFA array mixed with a first image sensor and a second image sensor as an image array to be converted; wherein, the image array to be converted includes target pixel points and D pixel points with one color component among R, G, and B;

[0061] The purpose of this embodiment is to convert the CFA array mixing the first image sensor and the second image sensor into a Bayer array with the same resolution as the original image. It can be understood that the first image sensor can be an APS, the second image sensor can be a DVS, or other image sensors, such as an EVS (Event-based vision sensor, event camera), which is not limited herein. In this embodiment, the case where the first image sensor is an APS and the second image sensor is a DVS is taken as an example for illustration. The Alpix003 pattern is designed based on the pixel interleaving of the APS and the DVS. In order to better recover the missing RGB color values of the DVS pixels and reconstruct a complete image, the APS pixels adopt an arrangement similar to that of the Bayer array. The eight arrangements of Alpix003 (i.e., the CFA array mixing the APS and the DVS) are as Figures 2a - 2h shown.

[0062] S102: Without considering the D pixel points in the converted image array, construct an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted;

[0063] In this step, without considering the D pixel points (i.e., DVS pixel points, simply referred to as D pixels in this article) in the converted image array, the image array to be converted is split into multiple intermediate Bayer images, and the i-th intermediate Bayer array is constructed based on the distribution of the target pixel points in the (i + 2n)-th column of the image array to be converted; where 1 ≤ i ≤ m, m is the total number of intermediate Bayer arrays, and n is an integer greater than or equal to 0. For example, the first intermediate Bayer array is constructed based on the distribution of the target pixel points in the odd-numbered columns of the image array to be converted, and the second intermediate Bayer array is constructed based on the distribution of the target pixel points in the even-numbered columns of the image array to be converted, or the first intermediate Bayer array is constructed based on the distribution of the target pixel points in the odd-numbered rows of the image array to be converted, and the second intermediate Bayer array is constructed based on the distribution of the target pixel points in the even-numbered rows of the image array to be converted, so as to split the image to be converted into two intermediate Bayer arrays, whose resolution is 1 / 4 of the resolution of the image array to be converted. The size of the image array to be converted is Width × Height, and the size of the intermediate Bayer image is Width / 2 × Height / 2, where the intermediate Bayer image only includes the target pixel points with one color component among R, G, and B. Taking the first arrangement in Figure 2 as an example, the image array to be converted is as Figure 3a shown, and the first intermediate Bayer array and the second intermediate Bayer array after splitting are as Figure 3b and 3c shown.

[0064] S103: Based on the color components of the adjacent pixel points of each of the target pixel points in the intermediate Bayer array, calculate the other two color components of each of the target pixel points respectively to obtain a first target array;

[0065] In this step, in each intermediate Bayer array, use a linear interpolation algorithm or a weighted algorithm, etc. to calculate the other two color components of each target pixel point to obtain the first target array. As a feasible implementation manner, obtaining the first target array includes: filling the intermediate Bayer array according to the other two color components of each of the calculated target pixel points to obtain the first target array. In a specific implementation, fill the other two color components calculated for each target pixel point into the corresponding position of the target pixel point in the intermediate Bayer array to obtain the first target array.

[0066] As a feasible implementation manner, the calculating the other two color components of each of the target pixel points based on the color components of the adjacent pixel points of each of the target pixel points in the intermediate Bayer array includes: performing linear interpolation on the intermediate Bayer array in the vertical direction and the horizontal direction respectively to calculate the R component and the B component of the target pixel point; performing bilinear interpolation on the intermediate Bayer array to calculate the G component of the target pixel point. In a specific implementation, for the R component and the B component that need to be calculated, first perform linear interpolation in the vertical direction, and then perform linear interpolation in the horizontal direction. Figure 4Taking the middle Bayer array shown as an example, when calculating the R component, first perform linear interpolation in the vertical direction to calculate the R component of the target pixel points in the odd-numbered columns (columns with even ordinates), such as (1,0), (3,0), (5,0), (7,0), (1,2), (3,2), (5,2), (7,2), etc. Taking the calculation of R(3,2) as an example, R(3,2) = (R(2,2) + R(4,2)) / 2. Then perform linear interpolation in the horizontal direction to calculate the R component of the target pixel points in the even-numbered columns (columns with odd ordinates), such as (0,1), (1,1), (2,1), (3,1), (4,1), (5,1), (6,1), (7,1), (0,3), (1,3), (2,3), (3,3), (4,3), (5,3), (6,3), (7,3), etc. Taking the calculation of R(2,3) as an example, R(2,3) = (R(2,2) + R(2,4)) / 2. When calculating the B component, first perform linear interpolation in the vertical direction to calculate the B component of the target pixel points in the even-numbered columns (columns with odd ordinates), such as (0,1), (2,1), (4,1), (6,1), (0,3), (2,3), (4,3), (6,3), etc. Then perform linear interpolation in the horizontal direction to calculate the B component of the target pixel points in the odd-numbered columns (columns with even ordinates), such as (0,2), (1,2), (2,2), (3,2), (4,2), (5,2), (6,2), (7,2), (0,4), (1,4), (2,4), (3,4), (4,4), (5,4), (6,4), (7,4), etc. The calculation method is similar to that of the R component and will not be elaborated here. For the G component to be calculated, use the bilinear interpolation algorithm for calculation. Taking the calculation of G(3,3) as an example, if |G(2,3) - G(4,3)| < |G(3,2) - G(3,4)|, then G(3,3) = (G(2,3) + G(4,3)) / 2; if |G(2,3) - G(4,3)| > |G(3,2) - G(3,4)|, then G(3,3) = (G(3,2) + G(3,4)) / 2; if |G(2,3) - G(4,3)| = |G(3,2) - G(3,4)|, then G(3,3) = (G(2,3) + G(4,3) + G(3,2) + G(3,4)) / 4.

[0067] As a preferred implementation, after this step, it further includes: using a median filtering algorithm to correct the color components of the target pixel points in the first target array. In a specific implementation, the median filtering algorithm is used to correct the color components of the target pixel points in the first target array. First, the other two color components of each calculated target pixel point are filled into the corresponding intermediate Bayer array, then two color difference arrays GR and GB of G - R and G - B are generated based on the filled intermediate Bayer array, then median filtering is performed on the two color difference arrays, and finally, the color components of the target pixel points are corrected based on the filtering result. For example, R(2,3) = G(2,3) - GR(2,3), B(2,3) = G(2,3) - GB(2,3), G(3,3) = B(3,3) + GB(3,3), R(3,3) = G(3,3) - GR(3,3), G(2,2) = R(2,3) + GR(2,2), B(2,2) = G(2,2) - GB(2,2). Preferably, only the other two calculated color components of the target pixel points can be corrected, which can not only improve the correction efficiency, but also retain the original pixels in the image array to be converted, and avoid distortion of the corrected first target array.

[0068] S104: Calculate the three color components of each D pixel point in the first target array based on the three color components of the adjacent pixel points of each D pixel point in the first target array, to obtain a second target array;

[0069] In this step, determine the adjacent pixel points of each D pixel point, determine the three color components of the adjacent pixel points in the first target array, and calculate the three color components of each D pixel point based on the three color components of the adjacent pixel points of each D pixel point, to obtain a second target array. As a feasible implementation, obtaining the second target array includes: filling the image array to be converted according to the three color components of each calculated D pixel point to obtain the second target array. In a specific implementation, the three color components calculated for each D pixel point are filled into the corresponding position of this D pixel point in the image array to be converted, to obtain the second target array.

[0070] As a feasible implementation manner, calculating the three color components of each of the D pixel points according to the three color components of adjacent pixel points in the first target array includes: filling the to-be-converted image array with the other two color components of each of the target pixel points calculated in the first target array to obtain a third target array; performing bilinear interpolation on the third target array to calculate the three color components of each of the D pixel points. In a specific implementation, first, fill the other two color components calculated for each target pixel point to the corresponding position of the target pixel point in the to-be-converted image array to obtain a third target array, as Figure 5 shown. Then perform bilinear interpolation on the third target array to calculate the three color components of each D pixel point. In a specific implementation, taking the calculation of R(1,1), G(1,1), and B(1,1) as an example, if |R(0,1) - R(2,1)| < |R(1,0) - R(1,2)|, then R(1,1) = (R(0,1) + R(2,1)) / 2; if |R(0,1) - R(2,1)| > |R(1,0) - R(1,2)|, then R(1,1) = (R(1,0) + R(1,2)) / 2; if |R(0,1) - R(2,1)| = |R(1,0) - R(1,2)|, then R(1,1) = (R(0,1) + R(2,1) + R(1,0) + R(1,2)) / 4; if |G(0,1) - G(2,1)| < |G(1,0) - G(1,2)|, then G(1,1) = (G(0,1) + G(2,1)) / 2; if |G(0,1) - G(2,1)| > |G(1,0) - G(1,2)|, then G(1,1) = (G(1,0) + G(1,2)) / 2; if |G(0,1) - G(2,1)| = |G(1,0) - G(1,2)|, then G(1,1) = (G(0,1) + G(2,1) + G(1,0) + G(1,2)) / 4; if |B(0,1) - B(2,1)| < |B(1,0) - B(1,2)|, then B(1,1) = (B(0,1) + B(2,1)) / 2; if |B(0,1) - B(2,1)| > |B(1,0) - B(1,2)|, then B(1,1) = (B(1,0) + B(1,2)) / 2; if |B(0,1) - B(2,1)| = |B(1,0) - B(1,2)|, then B(1,1) = (B(0,1) + B(2,1) + B(1,0) + B(1,2)) / 4. Fill the three color components of each calculated D pixel point to the position of each D pixel point in the third target array to obtain a second target array, as Figure 6 shown.

[0071] As a preferred implementation manner, after this step, it further includes: correcting the color components of the target pixel points in the second target array by using a median filtering algorithm. In a specific implementation, the specific correction process is similar to the correction process described in step S103, and will not be elaborated here. Preferably, only the other two calculated color components of the target pixel points can be corrected.

[0072] As a preferred implementation manner, the correcting the color components of the target pixel points in the second target array by using a median filtering algorithm includes: performing bilinear interpolation on the corrected second target array to correct the color components of the D pixel points. In a specific implementation, bilinear interpolation is performed again on the second target array corrected by median filtering to correct the three color components of each D pixel point. The specific bilinear interpolation process is similar to the above, and will not be elaborated here.

[0073] S105: Obtain a target Bayer array based on the second target array.

[0074] In this step, first determine the target arrangement mode of the target Bayer array to be converted, and then extract the required color components of each pixel point in the second target array based on the target arrangement mode by using algorithms such as remosaic to obtain the target Bayer array. The four arrangement modes of the Bayer array are as Figures 7a - 7d shown.

[0075] The image array format conversion method provided by the embodiments of the present application first splits the target pixel points with one color component among R, G, and B in the image array to be converted into two intermediate Bayer arrays, calculates the other two color components of each target pixel point in the intermediate Bayer array by using a linear interpolation algorithm, and then calculates the three color components of the D pixel points again according to the three color components of the target pixel points by using linear interpolation, so as to obtain the three color components of each pixel point in the image array to be converted. Finally, based on the three color components of each pixel point, the image array to be converted is converted into a target Bayer array. Since the three color components of each pixel point in the image array to be converted are known, the image resolution of the target Bayer array can be the same as that of the image array to be converted. It can be seen that the embodiments of the present application realize the conversion of the mixed CFA array into a Bayer array with the same image resolution as the original image, making the Alpix003 using the mixed CFA array of APS and DVS more versatile.

[0076] Next, an image array format conversion device provided by the embodiments of the present application will be introduced. The image array format conversion device described below can be referred to each other with the image array format conversion method described above.

[0077] See Figure 8, A structural diagram of an image array format conversion device shown according to an exemplary embodiment is as follows Figure 8 shown, including:

[0078] An acquisition module 801, configured to acquire a CFA array mixed by a first image sensor and a second image sensor as an image array to be converted; wherein, the image array to be converted includes target pixel points with one color component among R, G, and B and D pixel points;

[0079] A construction module 802, configured to construct an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted without considering the D pixel points in the converted image array;

[0080] A calculation module 803, configured to calculate the other two color components of each target pixel point respectively based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array to obtain a first target array, and calculate the three color components of each D pixel point according to the three color components of the adjacent pixel points of each D pixel point in the first target array to obtain a second target array;

[0081] A conversion module 804, configured to obtain a target Bayer array based on the second target array.

[0082] The image array format conversion device provided by the embodiments of the present application first splits the target pixel points with one color component among R, G, and B in the image array to be converted into two intermediate Bayer arrays, uses the linear interpolation algorithm to calculate the other two color components of each target pixel point in the intermediate Bayer array, and then uses linear interpolation again according to the three color components of the target pixel points to calculate the three color components of each D pixel point, so as to obtain the three color components of each pixel point in the image array to be converted. Finally, based on the three color components of each pixel point, the image array to be converted is converted into a target Bayer array. Since the three color components of each pixel point in the image array to be converted are known, the image resolution of the target Bayer array can be the same as that of the image array to be converted. It can be seen that the embodiments of the present application realize the conversion of the mixed CFA array into a Bayer array with the same image resolution as the original image, making the Alpix003 using the CFA array mixed with APS and DVS more versatile.

[0083] On the basis of the above embodiment, as a preferred implementation manner, the construction module 802 is specifically a module that constructs the i-th intermediate Bayer array based on the distribution of the target pixel points in the (i + 2n)-th column of the array to be converted; where 1 ≤ i ≤ m, m is the total number of intermediate Bayer arrays, and n is an integer greater than or equal to 0.

[0084] Based on the above embodiments, as a preferred embodiment, the calculation module 803 includes:

[0085] A first calculation unit, configured to perform linear interpolation on the intermediate Bayer array in both the vertical and horizontal directions to calculate the R component and the B component of the target pixel point;

[0086] A second calculation unit, configured to perform bilinear interpolation on the intermediate Bayer array to calculate the G component of the target pixel point.

[0087] Based on the above embodiments, as a preferred embodiment, the calculation module 803 includes:

[0088] A first filling unit, configured to fill the intermediate Bayer array according to the other two color components of each calculated target pixel point to obtain the first target array.

[0089] Based on the above embodiments, as a preferred embodiment, it further includes:

[0090] A first correction module, configured to correct the color components of the target pixel points in the first target array by using a median filtering algorithm.

[0091] Based on the above embodiments, as a preferred embodiment, the calculation module 803 includes:

[0092] A second filling unit, configured to fill the image array to be converted according to the other two color components of each calculated target pixel point in the first target array to obtain a third target array;

[0093] A third calculation unit, configured to perform bilinear interpolation on the third target array to calculate the three color components of each D pixel point.

[0094] Based on the above embodiments, as a preferred embodiment, the calculation module 803 includes:

[0095] A second filling unit, configured to fill the image array to be converted according to the three color components of each calculated D pixel point to obtain the second target array.

[0096] Based on the above embodiments, as a preferred embodiment, it further includes:

[0097] A second correction module, configured to correct the color components of the target pixel points in the second target array by using a median filtering algorithm.

[0098] Based on the above embodiments, as a preferred embodiment, it further includes:

[0099] A third correction module, configured to perform bilinear interpolation on the corrected second target array to correct the color components of the D pixel points.

[0100] Based on the above embodiments, as a preferred implementation manner, the conversion module 804 includes:

[0101] A determination unit, configured to determine the target arrangement mode of the target Bayer array;

[0102] A conversion unit, configured to extract the required color components of each pixel point in the second target array based on the target arrangement mode to obtain the target Bayer array.

[0103] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0104] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application further provide an electronic device. Figure 9 Shown is a structural diagram of an electronic device according to an exemplary embodiment, as Figure 9 shown, the electronic device includes:

[0105] A communication interface 1, capable of interacting with other devices such as network devices for information.

[0106] A processor 2, connected to the communication interface 1 to implement information interaction with other devices, and when running a computer program, configured to execute the image array format conversion method provided by the above one or more technical solutions. And the computer program is stored on the memory 3.

[0107] Of course, in actual application, each component in the electronic device is coupled together through a bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. The bus system 4 includes, in addition to the data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 9 all kinds of buses are labeled as the bus system 4.

[0108] The memory 3 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer program for operating on the electronic device.

[0109] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), a ferromagnetic random access memory (FRAM, ferromagnetic random access memory), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM, Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as a static random access memory (SRAM, Static Random Access Memory), a synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory), a dynamic random access memory (DRAM, Dynamic Random Access Memory), a synchronous dynamic random access memory (SDRAM, Synchronous Dynamic Random Access Memory), a double data rate synchronous dynamic random access memory (DDR SDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), an enhanced synchronous dynamic random access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), a sync link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), a direct rambus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 3 described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memories.

[0110] The method disclosed in the embodiments of the present application above can be applied to the processor 2 or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 2 or instructions in the form of software. The above-mentioned processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiments of the present application, it can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and combines its hardware to complete the steps of the foregoing method.

[0111] When the processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of the present application. For the sake of brevity, they will not be described in detail here.

[0112] In an exemplary embodiment, the embodiments of the present application further provide a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 including a stored computer program. The above computer program can be executed by the processor 2 to complete the steps of the foregoing method. The computer-readable storage medium may be a FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0113] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: mobile storage devices, ROM, RAM, magnetic disks, or optical discs, etc., which can store program codes.

[0114] Alternatively, if the above integrated units of the present application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as removable storage devices, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0115] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image array format conversion method, characterized in that, Including: Obtaining a CFA array mixed with a first image sensor and a second image sensor as an image array to be converted; wherein, the image array to be converted includes target pixel points having one color component among R, G, and B and D pixel points. Without considering the D pixel points in the converted image array, constructing an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted. Based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array, respectively calculating the other two color components of each target pixel point to obtain a first target array. According to the three color components of the adjacent pixel points of each D pixel point in the first target array, calculating the three color components of each D pixel point to obtain a second target array. Based on the second target array to obtain a target Bayer array.

2. The image array format conversion method according to claim 1, characterized in that, The constructing an intermediate Bayer array based on the distribution of the target pixel points in the image array to be converted includes: Constructing the i-th intermediate Bayer array based on the distribution of the target pixel points in the (i + 2n)-th column of the array to be converted; wherein, 1 ≤ i ≤ m, m is the total number of intermediate Bayer arrays, and n is an integer greater than or equal to 0.

3. The image array format conversion method according to claim 1, characterized in that, The calculating the other two color components of each target pixel point based on the color components of the adjacent pixel points of each target pixel point in the intermediate Bayer array includes: Performing linear interpolation on the intermediate Bayer array in the vertical direction and the horizontal direction respectively to calculate the R component and the B component of the target pixel point. Performing bilinear interpolation on the intermediate Bayer array to calculate the G component of the target pixel point.

4. The image array format conversion method according to claim 1, characterized in that, Obtaining the first target array includes: Filling the intermediate Bayer array according to the other two color components of each calculated target pixel point to obtain the first target array.

5. The image array format conversion method according to claim 1, characterized in that, After obtaining the first target array, it further includes: Using a median filtering algorithm to correct the color components of the target pixel points in the first target array.

6. The image array format conversion method according to claim 1, characterized in that, The calculating the three color components of each D pixel point according to the three color components of the adjacent pixel points of each D pixel point in the first target array includes: Filling the image array to be converted according to the other two color components of each calculated target pixel point in the first target array to obtain a third target array. Performing bilinear interpolation on the third target array to calculate the three color components of each D pixel point.

7. The image array format conversion method according to claim 1, characterized in that, Obtaining the second target array includes: Filling the image array to be converted according to the three color components of each calculated D pixel point to obtain the second target array.

8. The image array format conversion method according to claim 1, characterized in that, After obtaining the second target array, it further includes: Using a median filtering algorithm to correct the color components of the target pixel points in the second target array.

9. The image array format conversion method according to claim 8, characterized in that, The using a median filtering algorithm to correct the color components of the target pixel points in the second target array includes: Performing bilinear interpolation on the corrected second target array to correct the color components of the D pixel points.

10. The image array format conversion method according to claim 1, characterized in that, The obtaining of the target Bayer array based on the second target array includes: Determining the target arrangement of the target Bayer array; Extracting the color components required for each pixel in the second target array based on the target arrangement to obtain the target Bayer array.

11. An image array format conversion device, characterized in that, It includes: An acquisition module, configured to acquire the CFA array of the hybrid of the first image sensor and the second image sensor as the image array to be converted; wherein, the image array to be converted includes target pixels and D pixels each having one color component among R, G, and B; A construction module, configured to construct an intermediate Bayer array based on the distribution of the target pixels in the image array to be converted without considering the D pixels in the converted image array; A calculation module, calculating the other two color components of each target pixel respectively based on the color components of the adjacent pixels of each target pixel in the intermediate Bayer array to obtain the first target array, and calculating the three color components of each D pixel according to the three color components of the adjacent pixels of each D pixel in the first target array to obtain the second target array; A conversion module, configured to obtain the target Bayer array based on the second target array.

12. The image array format conversion device according to claim 11, characterized in that, The first image sensor is an APS, and the second image sensor is a DVS.

13. An electronic device, characterized in that, It includes: A memory, configured to store a computer program; A processor, configured to implement the steps of the image array format conversion method according to any one of claims 1 to 10 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the image array format conversion method according to any one of claims 1 to 10 are implemented.

Images