Method and apparatus for processing non-standard sparse color filter array images

Abstract

The application relates to a processing method and device of a non-standard sparse color filter array image, wherein the method comprises the following steps: acquiring a non-standard sparse color filter array image; using an interpolation algorithm, reconstructing three-channel color information of an APS pixel position based on single-channel color information of the APS pixel position in the color filter array image, filling the three-channel color information to a corresponding position of the APS pixel in the color filter array image, and obtaining at least three half-color array images; determining an interpolation direction map of an interpolated APS pixel at a DVS pixel based on three-channel color information of an APS pixel position of the half-color array image, so as to reconstruct three-channel color information of a DVS pixel position; and filling the three-channel color information to a corresponding position of the DVS pixel in the color filter array image, and obtaining a full-color image. The application solves the problem that a non-standard sparse color filter array image cannot be reconstructed in the related art, realizes the reconstruction of the non-standard sparse color filter array image, and can be applied to a conventional ISP algorithm.

Description

[0001] Technology Neighborhood

[0002] This application relates to the field of image processing technology, and in particular to methods and apparatus for processing non-standard sparse color filter array images. Background Technology

[0003] Dynamic Vision Sensor (DVS) is a novel type of vision sensor with advantages such as high-speed response (up to tens of thousands of hertz) and large dynamic range imaging. It achieves low data volume and low power consumption data output, solving problems such as insufficient frame rate, small dynamic range response, frame loss, and bloated algorithms caused by excessive data volume in traditional camera image sensors (CIS). Since DVS cannot capture the fine textures of natural scenes, an image chip vision sensor was proposed. This sensor interleaves traditional APS (Active Pixel Vision Sensor) pixels with novel DVS pixels, simultaneously outputting color CFA images and DVS data to achieve high-resolution RGB images and event data while ensuring high signal-to-noise ratio. This image chip vision sensor is covered with a color filter array arranged in a Bayer pattern. This array ensures that each pixel can only sample one of the three primary colors: red (R), green (G), and blue (B). The other two color values ​​are interpolated based on their neighborhood information to obtain the desired full-color image; this process is called de-mosaic.

[0004] Existing methods for demosaicing color filter array images generally employ directional linear minimum mean square error (LMMSE). This method first estimates the color differences in the horizontal and vertical directions using LMSE, then obtains the final color difference signal through directional weighting, and finally estimates the missing pixel values ​​for each channel. However, because DVS and APS are distributed at intervals, the binary data output by DVS pixels cannot provide intensity information for APS pixels, resulting in a discontinuous arrangement of APS pixels. Consequently, the above method cannot estimate the three-channel colors of DVS pixels, and therefore cannot reconstruct non-standard sparse color filter array images.

[0005] There is currently no effective solution to the problem that related technologies cannot achieve image reconstruction of non-standard sparse color filter arrays. Summary of the Invention

[0006] This embodiment provides a method and apparatus for processing non-standard sparse color filter array images to solve the problem that non-standard sparse color filter array images cannot be reconstructed in related technologies.

[0007] Firstly, this embodiment provides a method for processing non-standard sparse color filter array images, including:

[0008] Obtain a non-standard sparse color filter array image, the color filter array image including APS pixels and DVS pixels intersecting with the APS pixels;

[0009] Using an interpolation algorithm, the three-channel color information of the APS pixel position is reconstructed based on the single-channel color information of the APS pixel position in the color filter array image.

[0010] The three-channel color information of the reconstructed APS pixel position is filled into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images;

[0011] Based on the three-channel color information of the APS pixel position in the half-color array image, the interpolation direction pattern of the interpolated APS pixel at the DVS pixel is determined to reconstruct the three-channel color information of the DVS pixel position.

[0012] The reconstructed three-channel color information of the DVS pixel position is filled into the corresponding position of the DVS pixel in the color filter array image to obtain a full-color image.

[0013] In some embodiments, after obtaining the full-color image, the method further includes:

[0014] The full-color image is downsampled to obtain a standard Bayer array image; the standard Bayer array image is suitable for the ISP algorithm.

[0015] In some embodiments, the step of using an interpolation algorithm to reconstruct the three-channel color information of the APS pixel location based on the single-channel color information of the APS pixel location in the color filter array image includes:

[0016] The color filter array image is downsampled to obtain the first Bayer array sub-image of APS pixels;

[0017] Using an interpolation algorithm, based on the single-channel color information of the APS pixel position, the pixels missing in the green channel in the first Bayer array sub-image are evaluated to obtain the first green channel image;

[0018] Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the red channel of the first Bayer array sub-image based on the first green channel image to obtain the first red channel image;

[0019] Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the blue channel of the first Bayer array sub-image based on the first green channel image to obtain the first blue channel image;

[0020] Based on the first green channel image, the first red channel image, and the first blue channel image, the three-channel color information of the APS pixel location is reconstructed.

[0021] In some embodiments, the step of using an interpolation algorithm to evaluate pixels missing in the green channel in the first Bayer array sub-image based on the single-channel color information of the APS pixel location to obtain a first green channel image includes:

[0022] Based on the single-channel color information of the APS pixel position, the weight direction map of the missing green channel pixels in the first Bayer array sub-image is determined.

[0023] Using an interpolation formula, based on the weighted direction map of the missing green channel pixels in the first Bayer array sub-image, interpolation is performed on the missing green channel pixels in the first Bayer array sub-image to obtain the first green channel image.

[0024] In some of these embodiments, the interpolation formula is:

[0025] g=ω V g V +(1-ω V )g H ;

[0026] In the formula, g represents the interpolation at R / B; g V Indicates interpolation in the vertical direction; g H ω represents the interpolation in the horizontal direction. V Weights in the vertical direction.

[0027] In some embodiments, determining the weighted orientation map of the missing green channel pixels in the first Bayer array sub-image based on the single-channel color information of the APS pixel location includes:

[0028] Determine the gradient of the missing green channel pixels in the first Bayer array sub-image, and use the gradient to determine the initial weights of the APS pixels in the four directions of up, down, left, and right.

[0029] The first weights in the horizontal and vertical directions are determined using the gradient variance of the interpolation plot.

[0030] The second weights in the horizontal and vertical directions are determined by using the color difference variance of the interpolation graph;

[0031] Based on the first weight and the second weight, noise is judged on the direction of the initial weight to determine the weight direction map of the missing pixels in the green channel of the first Bayer array sub-image.

[0032] In some embodiments, determining the interpolation direction map of the interpolated APS pixel at the DVS pixel location based on the three-channel color information of the APS pixel location in the half-color array image, in order to reconstruct the three-channel color information at the DVS pixel location, includes:

[0033] The half-color array image is downsampled to obtain the second Bayer array sub-image of DVS pixels;

[0034] Based on the three-channel color information of the APS pixel location, the weight orientation map of each missing pixel in the second Bayer array sub-image of the DVS pixel is determined to reconstruct the three-channel color information of the DVS pixel location.

[0035] In some embodiments, reconstructing the three-channel color information of the DVS pixel location includes:

[0036] Using the first weighted interpolation formula, based on the weighted direction map of the missing green channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing green channel pixels in the second Bayer array sub-image to obtain the second green channel image;

[0037] Using the second weighted interpolation formula, based on the weighted direction map of the missing red channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing red channel pixels in the second Bayer array sub-image to obtain the second red channel image;

[0038] Using the third weighted interpolation formula, based on the weighted direction map of the missing blue channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing blue channel pixels in the second Bayer array sub-image to obtain the second blue channel image;

[0039] The three-channel color information of the DVS pixel location is reconstructed based on the second green channel image, the second red channel image, and the second blue channel image.

[0040] Secondly, this embodiment provides a processing device for a non-standard sparse color filter array image, including: an acquisition module, a first reconstruction module, a first filling module, a second reconstruction module, and a second filling module;

[0041] The acquisition module is used to acquire a non-standard sparse color filter array image, the color filter array image including APS pixels and DVS pixels arranged intersecting with the APS pixels;

[0042] The first reconstruction module is used to reconstruct the three-channel color information of the APS pixel position based on the single-channel color information of the APS pixel position in the color filter array image using an interpolation algorithm.

[0043] The first filling module is used to fill the three-channel color information of the reconstructed APS pixel position into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images;

[0044] The second reconstruction module is used to determine the interpolation direction pattern of the interpolated APS pixel at the DVS pixel based on the three-channel color information of the APS pixel position in the half-color array image, so as to reconstruct the three-channel color information of the DVS pixel position.

[0045] The second filling module is used to fill the corresponding positions of the DVS pixels in the color filter array image with the three-channel color information of the reconstructed DVS pixel positions to obtain a full-color image.

[0046] Thirdly, this embodiment provides a sensor including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the non-standard sparse color filter array image processing method described in the first aspect above.

[0047] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the processing method for non-standard sparse color filter array images described in the first aspect above.

[0048] Compared with related technologies, the processing method and apparatus for non-standard sparse color filter array images provided in this embodiment acquire a non-standard sparse color filter array image, which includes APS pixels and DVS pixels intersecting with the APS pixels; using an interpolation algorithm, reconstruct the three-channel color information of the APS pixel positions based on the single-channel color information of the APS pixel positions in the color filter array image; fill the corresponding positions of the APS pixels in the color filter array image with the reconstructed three-channel color information of the APS pixel positions to obtain at least three half-color array images; determine the interpolation direction map of the interpolated APS pixels at the DVS pixels based on the three-channel color information of the APS pixel positions in the half-color array images to reconstruct the three-channel color information of the DVS pixel positions; fill the corresponding positions of the DVS pixels in the color filter array image with the reconstructed three-channel color information of the DVS pixels to obtain a full-color image. This invention solves the problem that related technologies cannot reconstruct images from non-standard sparse color filter arrays, and enables the reconstruction of images from non-standard sparse color filter arrays, making it applicable to conventional ISP algorithms.

[0049] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a hardware structure block diagram of a terminal device for a non-standard sparse color filter array image processing method provided in an embodiment of this application.

[0052] Figure 2 This is a flowchart of a method for processing non-standard sparse color filter array images according to an embodiment of this application;

[0053] Figure 3 yes Figure 2 A schematic diagram of step S220 in the middle section;

[0054] Figure 4 yes Figure 2 Flowchart of step S220;

[0055] Figure 5 This is a flowchart of a method for processing non-standard sparse color filter array images according to another embodiment of this application;

[0056] Figure 6 This is a schematic diagram of a one-dimensional horizontal Bayer array sub-diagram provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of a one-dimensional vertical Bayer array sub-diagram provided in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of a process for determining the first weight using the gradient variance of an interpolation graph, provided in an embodiment of this application.

[0059] Figure 9 This is a schematic diagram of a process for determining the second weight using the color difference variance of an interpolation graph, provided in an embodiment of this application.

[0060] Figure 10 This is a schematic diagram of Demosiac provided in an embodiment of this application;

[0061] Figure 11 This is a schematic diagram of the process for determining the interpolation direction pattern of the interpolated APS pixel at the DVS pixel, provided in an embodiment of this application.

[0062] Figure 12 This is a structural block diagram of a color mapping device for digital images provided in an embodiment of this application.

[0063] In the diagram: 210, Acquisition module; 220, First reconstruction module; 230, First filling module; 240, Second reconstruction module; 250, Second filling module. Detailed Implementation

[0064] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0065] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by a person skilled in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0066] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for processing non-standard sparse color filter array images according to this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0067] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the non-standard sparse color filter array image processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0069] This embodiment provides a method for processing non-standard sparse color filter array images. Figure 2 This is a flowchart of the processing method for non-standard sparse color filter array images in this embodiment, as shown below. Figure 2 As shown, the process includes the following steps:

[0070] Step S210: Obtain a non-standard sparse color filter array image, which includes APS pixels and DVS pixels that are interleaved with the APS pixels.

[0071] Step S220: Using an interpolation algorithm, the three-channel color information of the APS pixel position is reconstructed based on the single-channel color information of the APS pixel position in the color filter array image.

[0072] Step S230: Fill the corresponding positions of the APS pixels in the color filter array image with the three-channel color information of the reconstructed APS pixel positions to obtain at least three half-color array images.

[0073] Step S240: Based on the three-channel color information of the APS pixel position of the half-color array image, determine the interpolation direction pattern of the interpolated APS pixel at the DVS pixel position, so as to reconstruct the three-channel color information of the DVS pixel position.

[0074] Step S250: Fill the corresponding positions of the DVS pixels in the color filter array image with the three-channel color information of the reconstructed DVS pixel positions to obtain a full-color image.

[0075] It should be noted that a non-standard sparse color filter array (CFA) image includes APS pixels and DVS pixels; APS pixels and DVS pixels are arranged alternately. The blank areas are DVS pixels; the RGB areas are APS pixels. First, an interpolation algorithm is used to reconstruct the three-channel color information (pic1) of the APS pixel location based on the single-channel color information at that location, thereby filling the corresponding positions of the APS pixels in the color filter array image, such as... Figure 3 As shown. Then, based on the three-channel color information of the reconstructed APS pixel position, the reconstruction of the three-channel color information of the DVS pixel position is guided. Finally, the reconstructed three-channel color information of the DVS pixel position is filled into the corresponding position of the DVS pixel in the color filter array image to obtain a full-color image.

[0076] The interpolation algorithm is not limited; for example, it can be nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. In other embodiments, algorithms based on Neighbor Embedding, deep learning, etc., can also be used without restriction. The half-color array image is an image in which the APS pixels in the color filter array image are completely filled. Since the reconstructed APS pixel positions have complete three-channel color information, that is, including the green channel image, red channel image, and blue channel image, each filling will result in a half-color array image; therefore, it is at least a half-color array image. In other embodiments, these three half-color array images can also be merged into one.

[0077] Through the above steps, the three-channel color information of the APS pixel position is reconstructed first, and the three-channel color information of the DVS pixel position is reconstructed based on the three-channel color information of the APS pixel position. This enables the reconstruction of non-standard sparse color filter array images, which is applicable to conventional ISP algorithms and solves the problem that related technologies cannot reconstruct non-standard sparse color filter array images.

[0078] The above steps are explained in detail below:

[0079] In some of these embodiments, such as Figure 4 As shown, step S220 includes the following steps;

[0080] Step S221: Downsample the color filter array image to obtain the first Bayer array sub-image of APS pixels;

[0081] Step S222: Using an interpolation algorithm, based on the single-channel color information of the APS pixel position, evaluate the pixels missing in the green channel in the first Bayer array sub-image to obtain the first green channel image.

[0082] Step S223: Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the red channel of the first Bayer array sub-image based on the first green channel image to obtain the first red channel image;

[0083] Step S224: Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the blue channel of the first Bayer array sub-image based on the first green channel image to obtain the first blue channel image.

[0084] Step S225: Reconstruct the three-channel color information of the APS pixel location based on the first green channel image, the first red channel image, and the first blue channel image.

[0085] Specifically, based on the specific structure of the alternating arrangement of APS and DVS pixels in the color filter array image, a first Bayer array sub-image composed of APS pixels is obtained through downsampling. Downsampling can be performed using a preset window size, such as 2x2, 3x3, 4x4, or 5x5. The downsampled first Bayer array sub-image can be the RGGB first Bayer array sub-image Raw1. In other embodiments, downsampling can simultaneously obtain two or more first Bayer array sub-images, allowing for parallel processing to improve processing efficiency.

[0086] In this embodiment, the interpolation algorithm is the Demosaic algorithm. It calculates the weight of the interpolation g at the R / B position in the first Bayer array sub-image and adaptively completes the interpolation of g using weights in different directions such as horizontal and vertical, thereby reconstructing high-quality pixels missing from the green channel. Taking advantage of the clearer and more complete nature of the reconstructed green channel pixels, the interpolation algorithm is used to interpolate the missing red and blue channels in the color difference plane in the first Bayer array sub-image, thereby recovering the first red channel image and the first blue channel image. Finally, the corresponding three-channel color information from the reconstructed three-channel image is filled into the corresponding positions of the APS pixels in the color filter array image. The process is as follows: Figure 4 As shown, this recovers the missing color pixels of the APS pixels in the color filter array image. In other embodiments, the preferred interpolation algorithm for steps S223 and S224 is bilinear interpolation. Of course, bicubic interpolation, nearest neighbor interpolation, and other algorithms can also be used.

[0087] In some of these embodiments, such as Figure 5 As shown, in Figure 2Based on the embodiment, after obtaining the full-color image, the following steps are also included:

[0088] Step S260: Downsample the full-color image to obtain a standard Bayer array image; the standard Bayer array image is suitable for the ISP algorithm.

[0089] The standard Bayer array image obtained through downsampling can include four types: BGGR Bayer array image; GBRG Bayer array image; GRBG Bayer array image; and RGGB Bayer array image. The ISP (Image Signal Processor) algorithm performs post-processing on the obtained standard Bayer array image, mainly including linear correction, noise removal, bad pixel removal, interpolation, white balance, and automatic exposure control. This application can reconstruct non-standard sparse color filter array images, thus being compatible with conventional ISP algorithms. Therefore, the sensor simultaneously outputs RGB images and event data, possessing the advantages of low latency, high temporal resolution, and high dynamic range of event cameras, while also outputting high-resolution RGB images and recording texture maps of static scenes, compatible with traditional processing algorithms. This ensures the sensor's application prospects are broad, including low-power detection and eye tracking in laptops, deblurring, high-speed photography, and high dynamic range photography in mobile phones, privacy masking, fast response, and high dynamic range video in security and autonomous driving, and localization and mapping, and human-computer interaction in robotics.

[0090] In some embodiments, step S222 includes the following steps:

[0091] Based on the single-channel color information of the APS pixel position, the weight orientation map of the missing green channel pixels in the first Bayer array sub-image is determined.

[0092] Using an interpolation formula, based on the weighted orientation map of the missing green channel pixels in the first Bayer array sub-image, interpolation is performed on the missing green channel pixels in the first Bayer array sub-image to obtain the first green channel image.

[0093] The interpolation formula is as follows:

[0094] g=ω V g V +(1-ω V )g H ;

[0095] In the formula, g represents the interpolation at R / B; g V Indicates interpolation in the vertical direction; g H ω represents the interpolation in the horizontal direction. V Weights in the vertical direction.

[0096] The determination of the weighted orientation map of missing green channel pixels in the first Bayer array sub-image based on single-channel color information of APS pixel location includes the following steps:

[0097] Determine the gradient of the missing green channel pixels in the first Bayer array sub-image, and use the gradient to determine the initial weights of the APS pixels in the four directions of up, down, left, and right.

[0098] The first weights in the horizontal and vertical directions are determined using the gradient variance of the interpolation plot.

[0099] The second weights in the horizontal and vertical directions are determined by using the color difference variance of the interpolation graph;

[0100] Based on the first and second weights, noise is judged on the direction of the initial weights to determine the weight direction map of the missing pixels in the green channel of the first Bayer array sub-image.

[0101] Specifically, the gradient of the same color reflects the correlation between the center pixel and pixels in different directions; the gradient is determined in ways such as: Figure 6 As shown, this is a schematic diagram of a Bayer array subgraph with a horizontal direction and one-dimensional coordinates as an example. The horizontal gradient of R0 can be: δ0=|G -1 -G1|; The gradient sum in the left direction is: δ L =δ -2 +δ -1 +δ0; The right-hand gradient sum is: δ R =δ0 + δ1 + δ2; then the initial weights of the left and right sides of R0 can be determined by the horizontal gradient of R0; taking the left direction as an example, the initial weights of the left direction at R0 are: The initial weights in the right direction at R0 are: For the same reason, such as Figure 7 As shown, a schematic diagram of the Bayer array subgraph with a vertical direction and one-dimensional coordinates is given. The vertical gradient of R0 can be expressed as: δ0=|G -1 -G1|; where G -a The pixel below R0; G a For the pixel above R0, the initial weight ω in the upward direction at R0 can be determined. U ; and the initial weight ω in the upward direction at R0 U Furthermore, the smaller the gradient, the more similar the pixels, and the greater the likelihood of using it for interpolation.

[0102] Specifically, the interpolation gradient is defined as the difference between interpolations in different directions. The horizontal interpolation gradient at R0 can be obtained by subtracting the left and right interpolations. Then, the gradient variances of the interpolation maps in the four directions are calculated. Finally, the directional weights on the CFA are combined to obtain the gradient variances of the horizontal and vertical interpolation maps, which are then normalized to obtain the first weights in the horizontal and vertical directions. For example: Figure 8 As shown, interpolation is first performed based on the neighborhood in different directions, and then the interpolation values ​​in different directions are subtracted to obtain the interpolation gradient. Wherein, from Determine id0 = gL0 - gR0 = G -1 -G1+0.5R2-0.5R -2 The variance of the interpolation gradient in the left direction at R0 can be defined as follows: For the same reason, the gradient variance of the interpolation plot in the right direction at R0 can be determined. Gradient variance of the interpolation plot in the upward direction at R0 Gradient variance of the interpolation plot to the right at R0 For the first weight, the gradient variance of the interpolation plot in the horizontal direction at R0 can be: The gradient variance of the interpolation plot in the vertical direction at R0 can be: After normalization, the first weight in the vertical direction is: The first weight in the horizontal direction is:

[0103] Specifically, such as Figure 9 As shown, horizontal interpolation is performed first; then color difference interpolation is performed. The variance of the color difference in the interpolated image to the left of R0 is defined as: in, For the same reason, the variance of the interpolation plot to the right at R0 can be determined. Interpolation plot color difference variance at R0 Interpolation plot color difference variance at R0 in the downward direction For the second weight; the interpolation color difference variance in the horizontal direction at R0 can be: The gradient variance of the interpolation plot in the vertical direction at R0 can be: After normalization, the first weight in the vertical direction is: The first weight in the horizontal direction is:

[0104] Specifically, to determine the presence of noise, the direction of the initial weights is analyzed based on the first and second weights. If the directions of the first and second weights differ from those of the initial weights, the pixels are identified as noise or have incorrect orientations, and this is corrected using a statistical formula. The corrected vertical weight ω is then used. V The statistical formula is:

[0105]

[0106] In the formula, ω 0，0It is the central weight; ω -1，-1 It is the weight to the upper left of the center weight; ω -1，1 It is the weight to the upper right of the center weight; ω 1，-1 It is the weight to the lower left of the center weight; ω 1，1 It is the weight to the lower right of the center weight.

[0107] Based on this, the weighted orientation map of the missing green channel pixels in the first Bayer array sub-image is determined. Then, with the accurate weighted orientation map determined, the complete green component is obtained for the missing green channel pixels in the first Bayer array sub-image (interpolation g at R / B), thus obtaining the first green channel image.

[0108] Finally, using bilinear interpolation, the pixels missing in the red and blue channels of the first Bayer array sub-image are interpolated based on the first green channel image to obtain the first red channel image and the first blue channel image, respectively; thus, the three-channel color information of the APS pixel location is reconstructed. For R / B interpolation b / r; for example: for R... 0，0 It is the center pixel R; B -1，-1 It is the pixel B to the upper left of the center pixel R; B -1，1 It is the pixel B to the upper right of the center pixel R; B 1，-1 It is the pixel B to the lower left of the center pixel R; B 1，1 It is pixel B to the lower right of the center pixel R. R 0，0 b is obtained using bilinear interpolation within the color difference plane. 0，0 ;

[0109]

[0110] For interpolating b / r in G; for example: for G 0，0 It is the center pixel G; B -1，0 It is pixel B directly above the center pixel G; R 0，1 It is the pixel R to the right of the center pixel G; B 1，0 It is pixel B directly below the center pixel G; R 0，-1 It is the pixel R to the left of the center pixel G. G 0，0 b is obtained using the bilinear interpolation algorithm. 0，0 ;

[0111]

[0112] G 0，0 r is obtained using the bilinear interpolation algorithm. 0，0 ;

[0113]

[0114] As mentioned earlier, this application processes the spatial and spectral correlation of an image to estimate the color missing from the center pixel using a neighborhood color filter array. Spatial correlation refers to the correlation within a channel, which is interpolated using information from the original color filter array. Spectral correlation can be understood as the correlation between channels, i.e., color correlation. Color correlation is mainly divided into two types: color difference map (RG, BG) and color ratio map (R / G, B / G). This embodiment uses the color difference map (RG, BG). In other embodiments, the color ratio map (R / G, B / G) can also be used.

[0115] Since the green component accounts for half of the image in the first Bayer array sub-image, interpolation of the green component is prioritized to obtain the complete first green channel image. Then, bilinear interpolation of the red and blue components is performed using the chromaticity plane, which is accurate and easy to implement. Interpolating r and b on the chromaticity plane is equivalent to referencing the texture direction of the green component (g). Accurate r and b are interpolated using accurate g, eliminating the need for additional direction judgment and effectively reducing algorithm complexity. Therefore, the accuracy of the green component reconstruction plays a decisive role in the subsequent results.

[0116] The key to green component (g) interpolation is the accuracy of the interpolation direction. Pre-interpolation is first performed based on assumptions, and then different weights are assigned to each direction according to a certain criterion. Using a weighted interpolation method, while accurately reconstructing the texture, it effectively reduces interpolation artifacts and further improves the quality of the interpolated image. Compared to existing interpolation methods that select multiple directions, this application achieves smoother interpolation results, especially in flat areas, avoiding the appearance of horizontal and vertical false stripes, which is beneficial for subsequent processing in the image system. Figure 10 As shown, at each R / B, interpolation is performed using the horizontal direction g and the vertical direction g. v The interpolation method can be ACPI, then the weights for the two directions are obtained through gradients, and finally the candidate interpolations in the horizontal and vertical directions are weighted to obtain the interpolation result g = ω. V *g v +ω H *g. For example... Figure 6 As shown, taking the current center pixel as R as an example, the horizontal interpolation g at R0 is defined as: g H =-0.25R -2 +0.5G -1 +0.5R0+0.5G1-0.25R2; where: R and G are the pixel values ​​at the corresponding positions in the non-standard sparse color filter array image; g represents the pixel value calculated by interpolation using the above formula; the subscript number indicates the current center position, and positive and negative indicate the left and right positions; for example: G -1 G represents the pixel value G at the leftmost position in a non-standard sparse color filter array image. For example... Figure 7As shown, the vertical interpolation g at R0 v Defined as: g v =-0.25R -2 +0.5G -1 +0.5R0+0.5G1-0.23R2; where: R and G are the pixel values ​​at the corresponding positions in the non-standard sparse color filter array image; g represents the pixel value calculated by interpolation using the above formula; the subscript number indicates the current center position, and positive and negative indicate the vertical position; for example: G -1 G represents the pixel value G at position 1 in a non-standard sparse color filter array image.

[0117] Since DVS pixels do not provide any color information, the three-channel color information of APS pixels is needed to guide the color interpolation of DVS pixels. In one embodiment, step S240 includes the following steps:

[0118] Step S241: Downsample the half-color array image to obtain the second Bayer array sub-image of DVS pixels;

[0119] Step S242: Based on the three-channel color information of the APS pixel position, determine the weight orientation map of the missing pixels in each channel of the second Bayer array sub-image of the DVS pixel, so as to reconstruct the three-channel color information of the DVS pixel position.

[0120] Specifically, such as Figure 11 As shown, downsampling of half-color array images can be achieved using a binning operation. For example, within a 2x2 window, binning is performed on three half-color array images PIC3 to obtain two types of second Bayer array sub-images: one of type RGGB and the other of type GRBG, which can be denoted as second Bayer array sub-image raw3 and second Bayer array sub-image raw4, respectively. The direction maps of the interpolation g at the R and B channel positions in images raw3 and raw4 are calculated and denoted as DMap1 and DMap2. The steps for calculating the weighted direction maps are the same as those for calculating the weighted direction map of interpolation g at R / B using the Demosaic algorithm, and the specific process will not be elaborated here. In other embodiments, the size of the window can be adjusted, which will not be described in detail here.

[0121] In one embodiment, reconstructing the three-channel color information of the DVS pixel location includes:

[0122] Using the first weighted interpolation formula, based on the weighted direction map of the missing green channel pixels in the second Bayer array sub-image of DVS pixels, interpolation is performed on the missing green channel pixels in the second Bayer array sub-image to obtain the second green channel image;

[0123] Using the second weighted interpolation formula, based on the weighted direction map of the missing red channel pixels in the second Bayer array sub-image of DVS pixels, interpolation is performed on the missing red channel pixels in the second Bayer array sub-image to obtain the second red channel image.

[0124] Using the third weighted interpolation formula, based on the weighted direction map of the missing blue channel pixels in the second Bayer array sub-image of DVS pixels, interpolation is performed on the missing blue channel pixels in the second Bayer array sub-image to obtain the second blue channel image;

[0125] The three-channel color information of the DVS pixel location is reconstructed based on the second green channel image, the second red channel image, and the second blue channel image.

[0126] The DVS pixel is weighted and interpolated with four APS pixels in the horizontal and vertical directions of its neighborhood through a weighted direction map to obtain the reconstructed three channels of the DVS. Specifically, the second Bayer array sub-image raw3 corresponding to the DVS pixel after the Binning operation is found, and the interpolation direction weights at the R / B positions in the image are determined. Figure 11 As shown, within the 2x2 first window, the interpolation direction weights at the R / B positions are only present in the corresponding second Bayer array sub-image raw3. Therefore, the direction map corresponding to the second Bayer array sub-image raw3 is used to guide the interpolation of DVS pixels a and b within the first window. Similarly, within the second window, the direction map corresponding to the second Bayer array sub-image raw4 is used to guide the interpolation of DVS pixels c and d within its window.

[0127] The r / g / b reconstructed values ​​at DVS pixels are obtained using a weighted interpolation formula. For example, for g... 0，0 G is the center pixel to be reconstructed; -1，0 It is the pixel G to the left of the center pixel g; G 0，1 It is the pixel G directly below the center pixel g; G 1，0 It is the pixel G to the right of the center pixel g; G 0，-1 It is the pixel G directly above the center pixel g.

[0128] The first weighted interpolation formula is:

[0129]

[0130] Where, ω h and ω v These are the horizontal and vertical weights obtained from the second Bayer array subgraph.

[0131] For r 0，0 R is the center pixel to be reconstructed; -1，0 It is the pixel R to the left of the center pixel r; R 0，1 It is the pixel R directly below the center pixel r; R1，0 It is the pixel R to the right of the center pixel r; R 0，-1 It is the pixel R directly above the center pixel r. The second weighted interpolation formula is:

[0132] For b 0，0 B is the center pixel to be reconstructed; -1，0 B0 and B1 are the pixels B to the left of the center pixel B; B0 and B1 are the pixels B directly below the center pixel B; B 1，0 It is the pixel B to the right of the center pixel b; B 0，-1 It is the pixel B directly above the center pixel b.

[0133] The third weighted interpolation formula is:

[0134] R / G / B represent the pixel values ​​of the three channels.

[0135] Finally, the reconstructed three-channel color information of the DVS pixel position is filled into the corresponding position of the DVS pixel in the color filter array image to obtain a full-color image.

[0136] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0137] This embodiment also provides a non-standard sparse color filter array image de-mosaic device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described previously. The terms "module," "unit," "subunit," etc., used below can refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0138] Figure 12 This is a structural block diagram of the non-standard sparse color filter array image processing device in this embodiment, as shown below. Figure 12 As shown, the device includes: an acquisition module 210, a first reconstruction module 220, a first filling module 230, a second reconstruction module 240, and a second filling module 250;

[0139] The acquisition module 210 is used to acquire a non-standard sparse color filter array image, which includes APS pixels and DVS pixels intersecting with the APS pixels.

[0140] The first reconstruction module 220 is used to reconstruct the three-channel color information of the APS pixel position based on the single-channel color information of the APS pixel position in the color filter array image using an interpolation algorithm.

[0141] The first filling module 230 is used to fill the three-channel color information of the reconstructed APS pixel position into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images.

[0142] The second reconstruction module 240 is used to determine the interpolation direction pattern of the interpolated APS pixel at the DVS pixel based on the three-channel color information of the APS pixel position of the half-color array image, so as to reconstruct the three-channel color information of the DVS pixel position.

[0143] The second filling module 250 is used to fill the three-channel color information of the reconstructed DVS pixel position into the corresponding position of the DVS pixel in the color filter array image to obtain a full-color image.

[0144] The above-mentioned device solves the problem that the reconstruction of non-standard sparse color filter array images cannot be achieved in related technologies, and realizes the reconstruction of non-standard sparse color filter array images, which can be applied to conventional ISP algorithms.

[0145] In one embodiment, Figure 12 In addition to this, it also includes a downsampling module;

[0146] The downsampling module is used to downsample the full-color image to obtain a standard Bayer array image; the standard Bayer array image is suitable for the ISP algorithm.

[0147] In one embodiment, the first reconstruction module 220 includes a first downsampling unit, an evaluation unit, a first bilinear interpolation unit, a second bilinear interpolation unit, and a first reconstruction unit;

[0148] The first downsampling unit is used to downsample the color filter array image to obtain the first Bayer array sub-image of APS pixels;

[0149] An evaluation unit is used to evaluate pixels missing in the green channel in the first Bayer array sub-image using an interpolation algorithm based on the single-channel color information of the APS pixel position, so as to obtain the first green channel image.

[0150] The first bilinear interpolation unit is used to perform bilinear interpolation on the pixels missing in the red channel of the first Bayer array sub-image based on the first green channel image to obtain the first red channel image.

[0151] The second bilinear interpolation unit is used to perform bilinear interpolation on the pixels missing in the blue channel of the first Bayer array sub-image based on the first green channel image to obtain the first blue channel image.

[0152] The first reconstruction unit is used to reconstruct the three-channel color information of the APS pixel position based on the first green channel image, the first red channel image, and the first blue channel image.

[0153] In one embodiment, the evaluation unit is further configured to determine the weighted orientation map of the missing green channel pixels in the first Bayer array sub-image based on the single-channel color information of the APS pixel position.

[0154] Using an interpolation formula, based on the weighted direction map of the missing green channel pixels in the first Bayer array sub-image, interpolation is performed on the missing green channel pixels in the first Bayer array sub-image to obtain the first green channel image.

[0155] In one embodiment, the interpolation formula is:

[0156] g=ω V g V +(1-ω V )g H ;

[0157] In the formula, g represents the interpolation at R / B; g V Indicates interpolation in the vertical direction; g H ω represents the interpolation in the horizontal direction. V Weights in the vertical direction.

[0158] In one embodiment, the evaluation unit is further configured to determine the gradient of the missing green channel pixel in the first Bayer array sub-image, and use the gradient to determine the initial weights of the APS pixel in the four directions of up, down, left, and right.

[0159] The first weights in the horizontal and vertical directions are determined using the gradient variance of the interpolation plot.

[0160] The second weights in the horizontal and vertical directions are determined by using the color difference variance of the interpolation graph;

[0161] Based on the first weight and the second weight, noise is judged on the direction of the initial weight to determine the weight direction map of the missing pixels in the green channel of the first Bayer array sub-image.

[0162] In one embodiment, the second reconstruction module 240 includes a second downsampling unit and a second reconstruction unit;

[0163] The second downsampling unit is used to downsample the half-color array image to obtain the second Bayer array sub-image of DVS pixels;

[0164] The second reconstruction unit is used to determine the weight direction map of each missing pixel in the second Bayer array sub-image of the DVS pixel based on the three-channel color information of the APS pixel position, so as to reconstruct the three-channel color information of the DVS pixel position.

[0165] In one embodiment, the second reconstruction unit is further configured to use a first weighted interpolation formula to interpolate the pixels missing in the green channel of the second Bayer array sub-image based on the weighted direction map of the missing green channel pixels in the second Bayer array sub-image of the DVS pixels, thereby obtaining a second green channel image.

[0166] Using the second weighted interpolation formula, based on the weighted direction map of the missing red channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing red channel pixels in the second Bayer array sub-image to obtain the second red channel image;

[0167] Using the third weighted interpolation formula, based on the weighted direction map of the missing blue channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing blue channel pixels in the second Bayer array sub-image to obtain the second blue channel image;

[0168] The three-channel color information of the DVS pixel location is reconstructed based on the second green channel image, the second red channel image, and the second blue channel image.

[0169] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0170] This embodiment also provides a sensor, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0171] Optionally, the sensor may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0172] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0173] S1, acquire a non-standard sparse color filter array image, which includes APS pixels and DVS pixels interleaved with the APS pixels;

[0174] S2, using an interpolation algorithm, reconstructs the three-channel color information of the APS pixel position based on the single-channel color information of the APS pixel position in the color filter array image;

[0175] S3, fill the three-channel color information of the reconstructed APS pixel position into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images;

[0176] S4. Based on the three-channel color information of the APS pixel position in the half-color array image, determine the interpolation direction pattern of the interpolated APS pixel at the DVS pixel position in order to reconstruct the three-channel color information of the DVS pixel position.

[0177] S5. Fill the corresponding positions of the DVS pixels in the color filter array image with the three-channel color information of the reconstructed DVS pixel positions to obtain a full-color image.

[0178] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0179] Furthermore, in conjunction with the processing method for non-standard sparse color filter array images provided in the above embodiments, this embodiment can also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the non-standard sparse color filter array image processing methods described in the above embodiments.

[0180] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0181] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0182] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0183] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for processing non-standard sparse color filter array images, characterized in that, include: Obtain a non-standard sparse color filter array image, the color filter array image including APS pixels and DVS pixels intersecting with the APS pixels; Using an interpolation algorithm, the three-channel color information of the APS pixel position is reconstructed based on the single-channel color information of the APS pixel position in the color filter array image. The three-channel color information of the reconstructed APS pixel position is filled into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images; Based on the three-channel color information of the APS pixel position in the half-color array image, the interpolation direction map of the interpolated APS pixel at the DVS pixel is determined to reconstruct the three-channel color information of the DVS pixel position. Specifically, this includes: downsampling the half-color array image to obtain a second Bayer array sub-image of the DVS pixel; and based on the three-channel color information of the APS pixel position, determining the weight direction map of each missing pixel in the second Bayer array sub-image of the DVS pixel to reconstruct the three-channel color information of the DVS pixel position. The reconstructed three-channel color information of the DVS pixel position is filled into the corresponding position of the DVS pixel in the color filter array image to obtain a full-color image.

2. The method for processing non-standard sparse color filter array images according to claim 1, characterized in that, After obtaining the full-color image, the following is also included: The full-color image is downsampled to obtain a standard Bayer array image; the standard Bayer array image is suitable for the ISP algorithm.

3. The method for processing non-standard sparse color filter array images according to claim 1, characterized in that, The method of using an interpolation algorithm to reconstruct the three-channel color information of the APS pixel position based on the single-channel color information of the APS pixel position in the color filter array image includes: The color filter array image is downsampled to obtain the first Bayer array sub-image of APS pixels; Using an interpolation algorithm, based on the single-channel color information of the APS pixel position, the pixels missing in the green channel in the first Bayer array sub-image are evaluated to obtain the first green channel image; Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the red channel of the first Bayer array sub-image based on the first green channel image to obtain the first red channel image; Using an interpolation algorithm, bilinear interpolation is performed on the pixels missing in the blue channel of the first Bayer array sub-image based on the first green channel image to obtain the first blue channel image; Based on the first green channel image, the first red channel image, and the first blue channel image, the three-channel color information of the APS pixel location is reconstructed.

4. The method for processing non-standard sparse color filter array images according to claim 3, characterized in that, The method of using an interpolation algorithm to evaluate pixels missing in the green channel in the first Bayer array sub-image based on the single-channel color information of the APS pixel position to obtain the first green channel image includes: Based on the single-channel color information of the APS pixel position, the weight direction map of the missing green channel pixels in the first Bayer array sub-image is determined. Using an interpolation formula, based on the weighted direction map of the missing green channel pixels in the first Bayer array sub-image, interpolation is performed on the missing green channel pixels in the first Bayer array sub-image to obtain the first green channel image.

5. The method for processing non-standard sparse color filter array images according to claim 4, characterized in that, The interpolation formula is: ； In the formula, Indicates the interpolation at R / B; Indicates interpolation in the vertical direction; Indicates interpolation in the horizontal direction; Weights in the vertical direction.

6. The method for processing non-standard sparse color filter array images according to claim 4, characterized in that, The determination of the weighted orientation map of the missing green channel pixels in the first Bayer array sub-image based on the single-channel color information of the APS pixel position includes: Determine the gradient of the missing green channel pixels in the first Bayer array sub-image, and use the gradient to determine the initial weights of the APS pixels in the four directions of up, down, left, and right. The first weights in the horizontal and vertical directions are determined using the gradient variance of the interpolation plot. The second weights in the horizontal and vertical directions are determined by using the color difference variance of the interpolation graph; Based on the first weight and the second weight, noise is judged on the direction of the initial weight to determine the weight direction map of the missing pixels in the green channel of the first Bayer array sub-image.

7. The method for processing non-standard sparse color filter array images according to claim 1, characterized in that, The reconstruction of the three-channel color information of the DVS pixel location includes: Using the first weighted interpolation formula, based on the weighted direction map of the missing green channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing green channel pixels in the second Bayer array sub-image to obtain the second green channel image; Using the second weighted interpolation formula, based on the weighted direction map of the missing red channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing red channel pixels in the second Bayer array sub-image to obtain the second red channel image; Using the third weighted interpolation formula, based on the weighted direction map of the missing blue channel pixels in the second Bayer array sub-image of the DVS pixels, interpolation is performed on the missing blue channel pixels in the second Bayer array sub-image to obtain the second blue channel image; The three-channel color information of the DVS pixel location is reconstructed based on the second green channel image, the second red channel image, and the second blue channel image.

8. A processing apparatus for a non-standard sparse color filter array image, characterized in that, include: The module includes an acquisition module, a first reconstruction module, a first filling module, a second reconstruction module, and a second filling module. The acquisition module is used to acquire a non-standard sparse color filter array image, the color filter array image including APS pixels and DVS pixels arranged intersecting with the APS pixels; The first reconstruction module is used to reconstruct the three-channel color information of the APS pixel position based on the single-channel color information of the APS pixel position in the color filter array image using an interpolation algorithm. The first filling module is used to fill the three-channel color information of the reconstructed APS pixel position into the corresponding position of the APS pixel in the color filter array image to obtain at least three half-color array images; The second reconstruction module is used to determine the interpolation direction map of the interpolated APS pixel at the DVS pixel based on the three-channel color information of the APS pixel position in the half-color array image, so as to reconstruct the three-channel color information of the DVS pixel position. Specifically, it includes: downsampling the half-color array image to obtain the second Bayer array sub-image of the DVS pixel. Based on the three-channel color information of the APS pixel position, the weight direction map of each missing pixel in the second Bayer array sub-image of the DVS pixel is determined to reconstruct the three-channel color information of the DVS pixel position. The second filling module is used to fill the corresponding positions of the DVS pixels in the color filter array image with the three-channel color information of the reconstructed DVS pixel positions to obtain a full-color image.

9. A sensor, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a method for processing a non-standard sparse color filter array image according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the processing method for the non-standard sparse color filter array image according to any one of claims 1-7.

Images