RGBIR image processing method and system, terminal and medium

By performing pre-interpolation and color correction on RGBIR images, the problem of poor performance in traditional ISP pipelines for processing RGBIR images is solved, generating high-quality Bayer and IR images, simplifying the design and reducing computational costs.

CN118984373APending Publication Date: 2024-11-19BEIJING HUIXI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310518915.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Traditional ISP pipelines struggle to effectively process RGBIR images, resulting in poor RGB image interpolation and an inability to fully utilize the processing capabilities of the ISP pipeline, particularly in well-lit and nighttime scenes where image quality is subpar.

Method used

By calculating the R channel pre-interpolation at the B pixel location and the B channel pre-interpolation at the R pixel location, the RGBIR image is converted into Bayer format images of RGGIR and BGGIR. Combined with G channel interpolation, the R, B, and IR channel interpolations for each pixel are calculated, and color correction is performed to remove the IR component, generating high-quality Bayer and IR images.

Benefits of technology

Without altering the existing ISP pipeline, high-quality Bayer and IR images are generated, improving the quality of interpolated images, simplifying the design, reducing computational costs, and minimizing color correction errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984373A_ABST
    Figure CN118984373A_ABST
Patent Text Reader

Abstract

The invention provides an RGBIR image processing method and system, and the method comprises the steps: calculating an R channel pre-interpolation at a B pixel point position, calculating a B channel pre-interpolation at an R pixel point position, and converting an RGBIR image into a Bayer format image; calculating a G channel interpolation based on the Bayer format image to obtain a G channel image; and calculating R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point by using the G channel image, and performing color correction on R, G and / or B channel values to be output to remove IR components to obtain a final Bayer image and an IR image. According to the method, the Bayer image and the IR image which are high in quality can be generated at the same time aiming at the RGBIR image, various application scenes are met, and the generated Bayer image can be directly processed by using the existing ISP pipeline based on the Bayer image format.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an RGBIR image processing method, system, terminal and medium. Background Art

[0002] RGBIR image sensors can collect visible light images and infrared images at the same time, providing clear images both during the day and at night, making up for the shortcomings of visible light sensors in shooting at night or in poor lighting conditions. At the same time, in the case of large changes in the lighting range of the vehicle environment and uneven lighting, infrared light can provide more stable and reliable image acquisition, better meeting the application requirements of driver status monitoring, autonomous driving perception, etc.

[0003] A commonly used 4x4 format RGBIR image sensor array is shown in Figure 1 As shown. Since the traditional ISP pipeline is usually designed to process Bayer format images. In order for the traditional ISP pipeline to support RGBIR image processing and support applications in well-lit and night scenes, the RGBIR image needs to be pre-processed and converted into the Bayer format. The RGB pixel filter of the RGBIR image sensor does not filter out the IR component, and the IR component is superimposed on the RGB pixel signal. In application scenarios that require visible light images, in order to obtain color-corrected RGB images, the RGB pixel values ​​need to be color corrected to remove the IR component. At the same time, in the application scenarios of infrared images, the ISP pipeline must also be able to output high-quality IR images.

[0004] In traditional solutions, full-size RGB and IR images are usually interpolated by upsampling based on the 4x4 pixel arrangement format of RGBIR, and then the IR component in the RGB channel is separated by a color correction matrix to generate a corrected RGB image. On the one hand, since some R and B pixels are replaced by IR pixels, the number of R and B pixels is reduced, and the interpolation image effect is usually poor. On the other hand, the directly output RGB image does not fully utilize the processing of the ISP pipeline Bayer domain, and it is difficult to achieve the best image effect. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a RGBIR image processing method, system, terminal and medium.

[0006] According to one aspect of the present invention, there is provided a RGBIR image processing method, comprising:

[0007] The R channel pre-interpolation is calculated at the B pixel position, the B channel pre-interpolation is calculated at the R pixel position, and the RGBIR image is transformed into a Bayer format image of RGGIR and a Bayer format image of BGGIR;

[0008] Based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR, a G channel interpolation is calculated to obtain a G channel image;

[0009] Using the G channel image, calculate the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel;

[0010] According to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel, the R, G and / or B channel values ​​to be output are color corrected to remove the IR component, and the final Bayer image and IR image are obtained.

[0011] Preferably, calculating the R channel pre-interpolation value at the B pixel point position, calculating the B channel pre-interpolation value at the R pixel point position, and transforming the RGBIR image into a Bayer format image based on the R channel pre-interpolation value and the B channel pre-interpolation value, comprises:

[0012] Based on the RGBIR array image, the R channel pre-interpolation is calculated at the B pixel position, and the original B channel value is replaced to generate an array image in RGGIR format. The B channel pre-interpolation is calculated at the R pixel position, and the original R channel value is replaced to generate an array image in BGGIR format. The RGGIR format array image and the BGGIR format array image are the corresponding Bayer format images after transformation.

[0013] Preferably, calculating the R channel pre-interpolation at the B pixel point position includes:

[0014] Determine the current B pixel position, where the position includes: continuous in the horizontal direction, continuous in the vertical direction, or no obvious continuous direction;

[0015] When the positions are continuous in the horizontal direction, the R channel pre-interpolation result is calculated using the R pixel values ​​adjacent in the horizontal direction, then:

[0016] R=(Ra+Rb) / 2+Δ H

[0017] Among them, R is the R channel pre-interpolation result, Ra and Rb are the R pixel values ​​adjacent to the B pixel in the horizontal direction, Δ H It is the horizontal gradient estimation of pixel B;

[0018] When the positions are continuous in the vertical direction, the R channel pre-interpolation result is calculated using the R pixel values ​​adjacent in the horizontal direction, then:

[0019] R=(Rc+Rd) / 2+Δ V

[0020] Among them, Rc and Rd are the R pixel values ​​adjacent to the B pixel in the vertical direction, Δ V It is the vertical gradient estimation of pixel B;

[0021] When the position has no obvious continuous direction, the R channel pre-interpolation result is calculated as:

[0022] R=(Ra+Rb+Rc+Rd) / 4+Δ

[0023] Among them, Δ is the undirected gradient estimate;

[0024] Using the above method, the B channel interpolation is calculated at the R pixel position.

[0025] Preferably, the calculating of the G channel interpolation based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR comprises:

[0026] Based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR, respectively calculating corresponding G channel values;

[0027] The G channel values ​​of the Bayer format image of the RGGIR and the Bayer format image of the BGGIR are averaged to obtain a G channel interpolation.

[0028] Preferably, the step of calculating the R channel interpolation, the B channel interpolation and the IR channel interpolation of each pixel using the G channel image comprises:

[0029] Calculate the B channel interpolation of the R pixel, including:

[0030] Get four B pixels that are adjacent to the R pixel in the horizontal and vertical directions;

[0031] The GB color difference D of the four B pixels is calculated as:

[0032] D=GB

[0033] Among them, G is the G channel interpolation of the B pixel point, and B is the pixel value of the original B pixel point of the RGBIR image;

[0034] Calculate the directional gradient Δ from the R pixel to the four B pixels d for:

[0035] Δd =∑|X i -X j |

[0036] Where X is the original pixel value of the X pixel in the RGBIR image, X is R, G, B or IR, and the subscripts i and j represent the position of the X pixel;

[0037] The weight w of the pixel R is calculated based on the gradient:

[0038] w=1 / (Δ d ) 2

[0039] According to the weighted color difference, the B channel interpolation B of the R pixel is calculated as:

[0040]

[0041] Calculate the R channel interpolation of the IR pixel, including:

[0042] Using the above method, first calculate the R channel values ​​of the four pixels at the diagonal positions adjacent to the IR pixel, then calculate the GR color difference of the four pixels, and finally calculate the weighted color difference of the IR pixel to obtain the R channel interpolation of the IR pixel;

[0043] Calculate the B channel interpolation of other pixels, including:

[0044] According to the original pixel value of the B pixel of the RGBIR image and the B channel interpolation, the color difference map D between the G channel and the B channel is calculated. G-B for:

[0045] D G-B =GB

[0046] Among them, G is the G channel interpolation, B is the pixel value of the original B pixel point of the RGBIR image, the B channel interpolation of the R pixel point, and the B channel interpolation of the IR pixel point;

[0047] The GB color difference at the position without B channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GB color difference map D' of other pixels G-B ;

[0048] From the G channel image and the GB color difference map of other pixels, the B channel interpolation B′ of other pixels is obtained as follows:

[0049] B′=G-D' G-B

[0050] Calculate the R channel interpolation of other pixels, including:

[0051] Calculate the color difference map D between the G channel and the R channelG-R for:

[0052] D G-R =GR

[0053] Among them, G is the G channel interpolation, R is the pixel value of the original R pixel point of the RGBIR image, the R channel interpolation of the B pixel point, and the R channel interpolation of the IR pixel point;

[0054] The GR color difference at the position without R channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GR color difference map D' of other pixels G-R ;

[0055] From the G channel image and the GR color difference map of other pixels, the R channel interpolation R′ of other pixels is obtained as follows:

[0056] R′=G-D' G-R

[0057] Among them, G is the G channel interpolation;

[0058] Calculate the IR channel interpolation of each pixel, including:

[0059] Calculate the color difference map D between the G channel and the IR channel G-IR for:

[0060] D G-IR =G-IR

[0061] Among them, G is the G channel interpolation, and IR is the pixel value of the original IR pixel point of the RGBIR image;

[0062] The G-IR color difference at the position where there is no IR pixel value is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the G-IR color difference map D' of other pixels G-IR ;

[0063] From the G channel image and the G-IR color difference map of other pixels, the IR channel interpolation Ir′ of other pixels is obtained as:

[0064] IR′=G-D′ G-IR .

[0065] Preferably, the color correction of the R, G and / or B channel values ​​to be output to remove the IR component according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point comprises:

[0066] Perform low-pass filtering on the G image to separate the high-frequency G of the G image H for:

[0067] G H=Gf l (G)

[0068] Among them, f l (·) represents the low-pass filter function;

[0069] Calculate the low frequency X of the current output channel value L for:

[0070] X L =XG H

[0071] Where X represents R, G or B channel;

[0072] Perform color correction on the color channels of the output Bayer image, including:

[0073] X' L =M X p

[0074] Where X represents the output channel value; p = [R, G, B, IR] T is the input value, which is the four-channel value of the current pixel; M X Represents the mapping relationship between the output channel X and the input value p;

[0075] The color-corrected value is added back to the high frequency for detail enhancement to obtain the final output value X output for:

[0076] X output =X' L +wG H

[0077] Where w is the enhancement coefficient;

[0078] Perform IR channel image enhancement on the output IR image to obtain the final IR image IR output for:

[0079] IR output =IR+wG H

[0080] Among them, IR is an IR image composed of IR channel interpolation, and the number of pixels of the IR image is consistent with the number of pixels of the original RGBIR image.

[0081] According to another aspect of the present invention, there is provided an RGBIR image processing system, comprising:

[0082] A format conversion module, which is used to calculate the R channel pre-interpolation at the B pixel point position, calculate the B channel pre-interpolation at the R pixel point position, and convert the RGBIR image into a Bayer format image of RGGIR and a Bayer format image of BGGIR;

[0083] A G image generation module, which calculates G channel interpolation based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR to obtain a G channel image;

[0084] A channel interpolation calculation module, which uses the G channel image to calculate the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel;

[0085] The image processing module performs color correction on the R, G and / or B channel values ​​to be output and removes the IR component according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point to obtain the final Bayer image and IR image.

[0086] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-described methods or to run the above-described system when executing the program.

[0087] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the above-mentioned methods, or to run the above-mentioned system.

[0088] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0089] The RGBIR image processing method, system, terminal and medium provided by the present invention can simultaneously generate high-quality Bayer images and IR images for RGBIR images obtained by an RGBIR sensor, meeting various application scenarios.

[0090] The RGBIR image processing method, system, terminal and medium provided by the present invention can generate a Bayer image that can be directly processed using an existing ISPpipeline based on the Bayer image format without changing the original ISPpipeline.

[0091] The RGBIR image processing method, system, terminal and medium provided by the present invention only need to perform fine interpolation on the channel values ​​output in the Bayer format, without adding excessive calculations, thereby improving calculation efficiency and reducing calculation costs.

[0092] The RGBIR image processing method, system, terminal and medium provided by the present invention pre-interpolate the RGBIR pixel arrangement format into a format similar to Bayer, and then interpolate based on the Bayer format method to improve the quality of the interpolated image.

[0093] The RGBIR image processing method, system, terminal and medium provided by the present invention fully consider the compatibility with the traditional ISP pipeline, simplify the design of RGBIR imaging, and output images with better quality.

[0094] The RGBIR image processing method, system, terminal and medium provided by the present invention perform low-pass filtering on the image before removing the IR component, thereby reducing the edge difference caused by interpolation and reducing the color correction error. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0096] Figure 1 Schematic diagram of an RGBIR image sensor array in the prior art.

[0097] Figure 2 The figure is a workflow diagram of an RGBIR image processing method in a preferred embodiment of the present invention.

[0098] Figure 3 Schematic diagram of converting a 4x4 RGBIR array into a 2x2 array similar to the Bayer format in an embodiment of the present invention; wherein (a) is a 2x2 array in the RGGIR format, and (b) is a 2x2 array in the BGGIR format.

[0099] Figure 4 This is an example diagram of pre-interpolation data in a preferred embodiment of the present invention.

[0100] Figure 5 This is an example diagram of R / B channel interpolation data in a preferred embodiment of the present invention.

[0101] Figure 6 The figure is a schematic diagram of conversion from RGBIR to Bayer format in a preferred embodiment of the present invention.

[0102] Figure 7 It is a schematic diagram of the component modules of the RGBIR image processing system in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0103] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0104] An embodiment of the present invention provides an RGBIR image processing method, which first pre-interpolates and calculates the R channel interpolation at the B pixel position, pre-interpolates and calculates the B channel interpolation at the R pixel position, transforms the 4x4 RGBIR array image into a 2x2 arrangement similar to the Bayer format, calculates the G channel interpolation of each pixel based on the 2x2 arrangement interpolation method, reconstructs a high-definition G channel image, and then uses the G channel image as a guide image to calculate the R and B channel interpolation of each pixel. According to the R, G, B and IR channel interpolation of each pixel, the R, G or B channel value to be output is color corrected to remove the IR component.

[0105] like Figure 2 As shown, the RGBIR image processing method provided by the embodiment of the present invention may include:

[0106] S1, calculate the R channel pre-interpolation at the B pixel position, calculate the B channel pre-interpolation at the R pixel position, and transform the RGBIR image into a Bayer format image of RGGIR and a Bayer format image of BGGIR;

[0107] S2, based on the Bayer format image of RGGIR and the Bayer format image of BGGIR, calculate the G channel interpolation to obtain the G channel image;

[0108] S3, using the G channel image, calculates the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel;

[0109] S4, according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point, color correction is performed on the R, G and / or B channel values ​​to be output to remove the IR component, so as to obtain the final Bayer image and IR image.

[0110] In a preferred embodiment of S1, an R channel pre-interpolation value is calculated at a B pixel point position, a B channel pre-interpolation value is calculated at an R pixel point position, and based on the R channel pre-interpolation value and the B channel pre-interpolation value, an RGBIR image is transformed into a Bayer format image, including:

[0111] Based on the RGBIR array image, the R channel pre-interpolation is calculated at the B pixel position, and the original B channel value is replaced to generate an array image in RGGIR format. The B channel pre-interpolation is calculated at the R pixel position, and the original R channel value is replaced to generate an array image in BGGIR format. The array image in RGGIR format and the array image in BGGIR format are the corresponding Bayer format images after transformation.

[0112] In a preferred embodiment of S1, calculating the R channel pre-interpolation at the B pixel point position includes:

[0113] Determine the current position of pixel B, where the position includes: horizontally continuous, vertically continuous, or no obvious continuous direction;

[0114] When the positions are continuous in the horizontal direction, the R channel pre-interpolation result is calculated using the horizontally adjacent R pixel values, then:

[0115] R=(Ra+Rb) / 2+Δ H

[0116] Among them, R is the R channel pre-interpolation result, Ra and Rb are the R pixel values ​​adjacent to the B pixel in the horizontal direction, Δ H It is the horizontal gradient estimation of pixel B;

[0117] When the positions are continuous in the vertical direction, the R channel pre-interpolation result is calculated using the adjacent R pixel values ​​in the horizontal direction, then:

[0118] R=(Rc+Rd) / 2+Δ V

[0119] Among them, Rc and Rd are the R pixel values ​​adjacent to the B pixel in the vertical direction, Δ V It is the vertical gradient estimation of pixel B;

[0120] When the position has no obvious continuous direction, the R channel pre-interpolation result is calculated as:

[0121] R=(Ra+Rb+Rc+Rd) / 4+Δ

[0122] Among them, Δ is the undirected gradient estimate;

[0123] Using the above method, the B channel interpolation is calculated at the R pixel position.

[0124] In a preferred embodiment of S2, based on the Bayer format image of RGGIR and the Bayer format image of BGGIR, calculating the G channel interpolation includes:

[0125] Based on the Bayer format image of RGGIR and the Bayer format image of BGGIR, the corresponding G channel values ​​are calculated respectively;

[0126] The G channel values ​​of the Bayer format image of RGGIR and the Bayer format image of BGGIR are averaged to obtain the G channel interpolation.

[0127] In a preferred embodiment of S3, using the G channel image, calculating the R channel interpolation, the B channel interpolation and the IR channel interpolation of each pixel point includes:

[0128] Calculate the B channel interpolation of the R pixel, including:

[0129] Get four B pixels that are adjacent to the R pixel in the horizontal and vertical directions;

[0130] The GB color difference D of the four B pixels is calculated as:

[0131] D=GB

[0132] Among them, G is the G channel interpolation of the B pixel point, and B is the pixel value of the original B pixel point of the RGBIR image;

[0133] Calculate the directional gradient Δ from the R pixel to the four B pixels d for:

[0134] Δ d =∑|X i -X j |

[0135] Where X is the original pixel value of the X pixel in the RGBIR image, X is R, G, B or IR, and the subscripts i and j represent the position of the X pixel;

[0136] The weight w of the pixel R is calculated based on the gradient:

[0137] w=1 / (Δ d ) 2

[0138] According to the weighted color difference, the B channel interpolation B of the R pixel is calculated as:

[0139]

[0140] Calculate the R channel interpolation of the IR pixel, including:

[0141] Using the above method, first calculate the R channel values ​​of the four pixels at the diagonal positions adjacent to the IR pixel, then calculate the GR color difference of the four pixels, and finally calculate the weighted color difference of the IR pixel to obtain the R channel interpolation of the IR pixel;

[0142] Calculate the B channel interpolation of other pixels, including:

[0143] According to the original pixel value of the B pixel of the RGBIR image and the B channel interpolation, the color difference map D between the G channel and the B channel is calculated. G-B for:

[0144] D G-B =GB

[0145] Among them, G is the G channel interpolation, B is the pixel value of the original B pixel point of the RGBIR image, the B channel interpolation of the R pixel point, and the B channel interpolation of the IR pixel point;

[0146] The GB color difference at the position without B channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GB color difference map D' of other pixels G-B ;

[0147] From the G channel image and the GB color difference map of other pixels, the B channel interpolation B′ of other pixels is obtained as follows:

[0148] B′=G-D' G-B

[0149] Calculate the R channel interpolation of other pixels, including:

[0150] Calculate the color difference map D between the G channel and the R channel G-R for:

[0151] D G-R =GR

[0152] Among them, G is the G channel interpolation, R is the pixel value of the original R pixel point of the RGBIR image, the R channel interpolation of the B pixel point, and the R channel interpolation of the IR pixel point;

[0153] The GR color difference at the position without R channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GR color difference map D' of other pixels G-R ;

[0154] From the G channel image and the GR color difference map of other pixels, the R channel interpolation R′ of other pixels is obtained as follows:

[0155] R′=G-D' G-R

[0156] Among them, G is the G channel interpolation;

[0157] Calculate the IR channel interpolation of each pixel, including:

[0158] Calculate the color difference map D between the G channel and the IR channelG-IR for:

[0159] D G-IR =G-IR

[0160] Among them, G is the G channel interpolation, and IR is the pixel value of the original IR pixel point of the RGBIR image;

[0161] The G-IR color difference at the position where there is no IR pixel value is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the G-IR color difference map D' of other pixels G-IR ;

[0162] From the G channel image and the G-IR color difference map of other pixels, the IR channel interpolation IR′ of other pixels is obtained as follows:

[0163] IR′=G-D′ G-IR .

[0164] In a preferred embodiment of S4, according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point, color correction is performed on the R, G and / or B channel values ​​to be output to remove the IR component, including:

[0165] Perform low-pass filtering on the G image to separate the high-frequency G of the G image H for:

[0166] G H =Gf l (G)

[0167] Among them, F l (·) represents the low-pass filter function;

[0168] Calculate the low frequency X of the current output channel value L for:

[0169] X L =XG H

[0170] Where X represents R, G or B channel;

[0171] Perform color correction on the color channels of the output Bayer image, including:

[0172] X' L =M X p

[0173] Where X represents the output channel value; p = [R, G, B, IR] T is the input value, which is the four-channel value of the current pixel; M X Represents the mapping relationship between the output channel X and the input value p;

[0174] The color-corrected value is added back to the high frequency for detail enhancement to obtain the final output value X output for:

[0175] X output =X' L +wG H

[0176] Where w is the enhancement coefficient;

[0177] Perform IR channel image enhancement on the output IR image to obtain the final IR image IR output for:

[0178] IR output =IR+wG H

[0179] Among them, IR is the IR image composed of IR channel interpolation, and the number of pixels of the IR image is consistent with the number of pixels of the original RGBIR image.

[0180] The technical solution and working principle of the RGBIR image processing method provided by the above embodiment of the present invention are further described below in conjunction with the accompanying drawings and specific application examples of the present invention.

[0181] S1, calculate the G channel interpolation of each pixel.

[0182] In image demosaicing, the G channel with the largest number of pixels is usually interpolated first, and then the R and B images are interpolated using the better image resolution information of the G image. A variety of methods based on the Bayer image format to improve the interpolation quality of the G image have been proposed, such as gradient weighted methods, residual interpolation, channel correlation interpolation, etc. Since the RGBIR pixel arrangement format is different from the Bayer image, the embodiment of the present invention first pre-interpolates the R and B pixels into a 2x2 arrangement format, and then uses the above method to improve the interpolation quality of the G image to interpolate the G channel.

[0183] In a preferred embodiment of S1, the step may further include:

[0184] S1.1, calculate the pre-interpolation values ​​of the R and B pixel positions.

[0185] For a 4x4 RGBIR array, the R channel pre-interpolation is calculated at the B pixel position and the original B channel value is replaced. Then a 2x2 array in RGGIR format can be generated, such as Figure 3 As shown in (a) in the figure. Calculate the B channel pre-interpolation at the R pixel position and replace the original R channel value to generate a 2x2 array in BGGIR format, as shown in Figure 3As shown in (b). The array image in RGGIR format and the array image in BGGIR format are the corresponding Bayer format images after transformation. Further, the R channel pre-interpolation is calculated at the B pixel position, including:

[0186] Determine the current position of pixel B, where the position includes: horizontally continuous, vertically continuous, or no obvious continuous direction;

[0187] When the positions are continuous in the horizontal direction, the R channel pre-interpolation result is calculated using the horizontally adjacent R pixel values, then:

[0188] R=(Ra+Rb) / 2+Δ H

[0189] Among them, R is the R channel pre-interpolation result, Ra and Rb are the R pixel values ​​adjacent to the B pixel in the horizontal direction, Δ H It is the horizontal gradient estimation of pixel B;

[0190] When the positions are continuous in the vertical direction, the R channel pre-interpolation result is calculated using the adjacent R pixel values ​​in the horizontal direction, then:

[0191] R=(Rc+Rd) / 2+Δ V

[0192] Among them, Rc and Rd are the R pixel values ​​adjacent to the B pixel in the vertical direction, Δ v It is the vertical gradient estimation of pixel B;

[0193] When the position has no obvious continuous direction, the R channel pre-interpolation result is calculated as:

[0194] R=(Ra+Rb+Rc+Rd) / 4+Δ

[0195] Where Δ is the directionless gradient estimate, which is calculated by the pixel value difference between the central B pixel and the adjacent B pixels above, below, left and right.

[0196] Using the above method, the B channel pre-interpolation is calculated at the R pixel position.

[0197] In a specific application example, taking the calculation of the R channel pre-interpolation of the B pixel position as an example, Figure 4 As shown, first determine whether the current B pixel position is continuous in the horizontal direction, continuous in the vertical direction, or has no obvious continuous direction. Direction determination can use the existing edge direction determination method. If it is continuous in the horizontal direction, use the R pixel values ​​adjacent to the horizontal direction to calculate the pre-interpolation result, and the R channel pre-interpolation R22 at the B22 position is:

[0198] R22=(R20+R24) / 2+Δ22 H

[0199] Among them, R20 and R24 are the R pixel values ​​adjacent to B22 in the horizontal direction, Δ22 H is the horizontal gradient estimate at position B22, which can be calculated from the adjacent G channel values.

[0200] If it is vertically continuous, the R channel pre-interpolation R22 at position B22 is:

[0201] R22=(R02+R42) / 2+Δ22 V

[0202] Among them, R02 and R42 are the R pixel values ​​adjacent to B22 in the vertical direction, Δ22 V is the vertical gradient estimate at position B22.

[0203] Otherwise, the R channel pre-interpolation R22 at position B22 is:

[0204] R22=(R20+R24+R02+R42) / 4+Δ22

[0205] Where Δ22 is the undirected gradient estimate of the B22 position.

[0206] S1.2, use a similar method to calculate the B channel pre-interpolation value of the R pixel position.

[0207] S2, calculate the G channel interpolation.

[0208] The G channel interpolation result is needed to assist the interpolation calculation of the R and B channels. Improving the interpolation accuracy of the G channel is the key to improving image quality. You can use a variety of methods to improve the interpolation quality of the G image as mentioned above, which will not be described in detail here. Since S1.1 pre-interpolates two images in a 2x2 format, in this step, each 2x2 format image will calculate a G channel interpolation map, and the two G channel interpolation maps are averaged as the final G interpolation result:

[0209] G=(G RGGIR +G BGGIR ) / 2

[0210] Among them, G is the final G interpolation result, G RGRIR G is the G channel interpolation calculated by the Bayer format G interpolation method for the RGGIR image. BGGIR G channel interpolation calculated for BGGIR image based on Bayer format G interpolation method.

[0211] S3, calculates R, B channel interpolation and IR channel interpolation.

[0212] The interpolation of R and B channels is divided into two steps. The complex interpolation of the R and B channel values ​​to be output (i.e., the interpolation method based on gradient weighting in S1) is performed to make the direction judgment more accurate, and the interpolation result retains more details to ensure the quality of the output image; the simple interpolation of the R and B channel values ​​only used for color correction (i.e., the interpolation method of upsampling the color difference map to complete the missing color difference values ​​in S3.3) is performed to reduce the complexity of the processing. Figure 6 For example, Figure 6 The RGBIR format shown in (a) is converted to Figure 6 In the BGGR Bayer format shown in (b), the G pixel output is still the G pixel value, the R pixel output is the B pixel value, and the IR pixel output is the R pixel value. If the R and B pixel positions in the output image format are swapped, the interpolation method is similar.

[0213] In a preferred embodiment of S3, the step may further include:

[0214] S3.1, calculate the B channel interpolation of the R pixel point. It includes:

[0215] Get four B pixels that are adjacent to the R pixel in the horizontal and vertical directions;

[0216] The GB color difference D of the four B pixels is calculated as:

[0217] D=GB

[0218] Among them, G is the G channel interpolation of the B pixel point, and B is the pixel value of the original B pixel point of the RGBIR image;

[0219] Calculate the directional gradient Δ from the R pixel to the four B pixels d for:

[0220] Δ d =∑|X i -X j |

[0221] Where X is the original pixel value of the X pixel in the RGBIR image, X is R, G, B or IR, and the subscripts i and j represent the position of the X pixel;

[0222] The weight w of the pixel R is calculated based on the gradient:

[0223] w=1 / (Δ d ) 2

[0224] According to the weighted color difference, the B channel interpolation B of the R pixel is calculated as:

[0225]

[0226] In a specific application example, the G channel interpolation has been calculated at the B pixel position, such as Figure 5 As shown, the GB color difference of the four pixels B02, B20, B24, and B42 is calculated as:

[0227] D 02 =G02-B02

[0228] D 20 =G20-B20

[0229] D 24 =G24-B24

[0230] D 42 =G42-B42

[0231] Among them, D * is the GB color difference of the B pixel at position *, and G* is the G channel interpolation of the B pixel at position *;

[0232] Calculate the gradient along the directions of the four points. An example of gradient calculation is as follows (i.e., the difference of the same channel values ​​along the direction is calculated as the gradient):

[0233] Δ 02 =|G01-G21|+|G12-G32|+|G03-G23|

[0234] Δ 20 =|G10-G12|+|G21-G23|+|G30-G32|

[0235] Δ 24 =|G12-G14|+|G21-G23|+|G32-G34|

[0236] Δ 42 =|G21-G41|+|G12-G32|+|G23-G43|

[0237] Among them, Δ * is the gradient of the pixel B at position *;

[0238] Calculate weights based on gradients:

[0239] w 02 =1 / (Δ 02 ) 2

[0240] w 20 =1 / (Δ 20 ) 2

[0241] w 24 =1 / (Δ 24 )2

[0242] w 42 =1 / (Δ 42 ) 2

[0243] Among them, w * is the weight of the B pixel at position *;

[0244] The B channel interpolation B22 of the R22 pixel is calculated based on the weighted color difference:

[0245]

[0246] S3.2, calculate the R channel interpolation of the IR pixel point. It includes:

[0247] Using the above method, first calculate the R channel values ​​of the four pixels at the diagonal positions adjacent to the IR pixel, then calculate the GR color difference of the four pixels, and finally calculate the weighted color difference of the IR pixel to obtain the R channel interpolation of the IR pixel.

[0248] In a specific application example, Figure 5 As shown, the R interpolation at positions B24 and B42 can be calculated according to the method of S3.1 to obtain the four R channel values ​​at the adjacent diagonal positions of IR33. To calculate the GR color difference, a method similar to S3.1 is used to calculate the weighted color difference at the position of IR33 and calculate the R channel interpolation as:

[0249]

[0250] S3.3, calculate the B channel interpolation of other pixels. Including:

[0251] According to the original pixel value of the B pixel of the RGBIR image and the interpolated B channel interpolation (that is, the B channel interpolation of the R pixel and the B channel interpolation of the IR pixel), the color difference map D between the G channel and the B channel is calculated. G-B for:

[0252] D G-B =GB

[0253] Among them, G is the G channel interpolation, and B is the B channel interpolation;

[0254] The GB color difference at the position without B channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GB color difference map D' of other pixels G-B ;

[0255] From the G channel image and the GB color difference map of other pixels, the B channel interpolation B′ of other pixels is obtained as follows:

[0256] B′=G-D' G-B

[0257] In a specific application example, the color difference D between the G and B channels is calculated based on the B pixel point and the interpolated B channel. G-B for:

[0258] D G-B =FB

[0259] Among them, the GB color difference at the position without B channel interpolation is 0. By interpolating the color difference image, the color difference values ​​of other pixels are supplemented. The interpolation method can use bilinear interpolation, bicubic interpolation, Gaussian filter upsampling and other methods. Taking Gaussian filter upsampling as an example, the GB color difference D' of other pixels G-B for:

[0260] D' G-B = up(D G-B ).*gauss_kernel

[0261] From the G channel image and the GB color difference image of other pixels, the B channel interpolation B′ of other pixels can be obtained as:

[0262] B′=G-D' G-B

[0263] S3.4, calculate the R channel interpolation of other pixels.

[0264] The same color difference calculation and interpolation method as S3.3 is used to calculate the R channel interpolation of other pixels, which will not be repeated here.

[0265] S3.5, calculating the IR channel interpolation value of each pixel.

[0266] The IR interpolation of each pixel is calculated using the same color difference calculation and interpolation method as S3.3, which will not be repeated here.

[0267] S4, color correction and output.

[0268] In order to balance the difference in clarity between different interpolation methods and avoid the introduction of color correction errors at the edge of the image due to interpolation errors, low-pass filtering is first performed to separate the low-frequency basic image and the high-frequency detail image, color correction is performed on the basic image value, and then the details are added back to enhance the output image.

[0269] Perform low-pass filtering on the G channel image to separate the high-frequency G H for:

[0270] G H =Gf l (G)

[0271] Among them, f l (·) represents a low-pass filter function.

[0272] Calculate the low frequency X of the current output channel value L for:

[0273] X L =XG H

[0274] Where X represents the R, G, or B channel.

[0275] If the output is a Bayer format image, color correction is performed on the color channel to be output. Usually, a color correction matrix is ​​used to correct the RGB channel values. The correction relationship is as follows:

[0276] X' L =M X p

[0277] Where X represents the output channel value; p = [R, G, B, IR] T is the input value, which is the four-channel value of the current pixel; M X Represents the mapping relationship between the output channel X and the input value p. Specifically, M X It can be calculated by shooting a color card and giving target values ​​for each color block using the least squares method.

[0278] The color-corrected value is added back to the high frequency for detail enhancement to obtain the final output value X output for:

[0279] X output =X' L +wG H

[0280] Where w is the enhancement factor.

[0281] If the output is an IR image, then the enhanced IR image IR is output output for:

[0282] IR output =IR+wG H

[0283] Among them, IR is an IR image composed of IR channel interpolation, and the size (number of pixels) of the IR image is consistent with the size (number of pixels) of the original RGBIR image.

[0284] The solution provided by the above embodiment of the present invention is further described in detail below in conjunction with the accompanying drawings and specific application implementations.

[0285] Specific application example 1:

[0286] First, calculate the R channel value at the B pixel position, replace the original B channel value, and generate a 2x2 array in RGGIR format, such as Figure 3 As shown in (a) in the figure. Calculate the B channel value at the R pixel position, replace the original R channel value, and generate a 2x2 array in BGGIR format, as shown in Figure 3 As shown in (b).

[0287] Taking the R pixel as the center, calculate the color difference D = GR between the G pixel and the R pixel in the four directions, as well as the gradient Δ in the four directions. T , Δ B , Δ L , Δ R . Calculate the weight w according to the gradient T =1 / (Δ T ) 2 、w B =1 / (Δ B ) 2 、w L =1 / (Δ L ) 2 、w R =1 / (Δ R ) 2 , then the G channel interpolation of the R pixel position is calculated as:

[0288]

[0289] Similarly, the G channel interpolation of the B pixel and the IR pixel is calculated to generate a complete G plane image.

[0290] Calculate the R channel interpolation of the B pixel position. With the B pixel as the center, calculate the color difference D = GR between the G pixel and the R pixel in the four directions of up, down, left, and right, as well as the gradient Δ in the four directions. T , Δ B , Δ L , Δ R and weight w T 、w B 、w L 、w R , then the R channel interpolation of the B pixel position is calculated as:

[0291]

[0292] Generate color difference map D based on known R channel value G-R , using Gaussian filtering upsampling to generate a full-size color difference map:

[0293]

[0294] Calculate the R channel values ​​of other pixels:

[0295]

[0296] Similarly, the B and IR channel interpolations are calculated.

[0297] like Figure 6 , perform color correction on the output channel values.

[0298] Perform low-pass filtering on the G image to separate the high-frequency g of the G image H for:

[0299] G H =Gf l (G)

[0300] Calculate the low frequency X of the current output channel value L for:

[0301] X L =XG H

[0302] Where X represents the R, G, or B channel.

[0303] Use the color correction matrix to correct the RGB channel values. The correction relationship is as follows:

[0304] X' L =M X p

[0305] The color corrected values ​​are added back to the high frequencies for detail enhancement, resulting in the final output values.

[0306] X output =X' L +wG H .

[0307] Specific application example 2:

[0308] First, calculate the R channel value at the B pixel position, replace the original B channel value, and generate a 2x2 array in RGGIR format, such as Figure 3 As shown in (a) in the figure. Calculate the B channel value at the R pixel position, replace the original R channel value, and generate a 2x2 array in BGGIR format, as shown in Figure 3 As shown in (b).

[0309] Taking the R pixel as the center, calculate the color difference D = GR between the G pixel and the R pixel in the four directions, as well as the gradient Δ in the four directions. T , Δ B , Δ L , Δ R . Calculate the weight w according to the gradient T =1 / (ΔT ) 2 、w B =1 / (Δ B ) 2 、w L =1 / (Δ L ) 2 、w R =1 / (Δ R ) 2 . Then the G channel interpolation of the R pixel position is calculated as:

[0310]

[0311] Similarly, the G channel interpolation of the B pixel and the IR pixel is calculated to generate a complete G plane image.

[0312] Calculate the R channel interpolation of the B pixel position. With the B pixel as the center, calculate the color difference D = GR between the G pixel and the R pixel in the four directions of up, down, left, and right, as well as the gradient Δ in the four directions T , Δ B , Δ L , Δ R and weight w T 、w B 、w L 、w R . Calculate the R channel interpolation of the B pixel position as:

[0313]

[0314] Generate color difference map D based on known R channel value G-R . The full-size color difference image generated by upsampling with Gaussian filtering is:

[0315]

[0316] Calculate the R channel interpolation of other pixels:

[0317]

[0318] Similarly, the B channel interpolation and the IR channel interpolation are calculated.

[0319] Perform low-pass filtering on the G image to separate the high-frequency G of the G image H for:

[0320] G H =Gf l (G)

[0321] Add high frequencies back to the IR channel value to enhance the details and get the final output IR value IR output for:

[0322] IR output =IR+wG H .

[0323] An embodiment of the present invention provides an RGBIR image processing system.

[0324] like Figure 7 As shown, the RGBIR image processing system provided by this embodiment may include:

[0325] A format conversion module, which is used to calculate the R channel pre-interpolation at the B pixel point position, calculate the B channel pre-interpolation at the R pixel point position, and convert the RGBIR image into a Bayer format image of RGGIR and a Bayer format image of BGGIR;

[0326] A G image generation module, which calculates the G channel interpolation based on the Bayer format image of RGGIR and the Bayer format image of BGGIR to obtain a G channel image;

[0327] A channel interpolation calculation module, which uses the G channel image to calculate the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel;

[0328] The image processing module performs color correction on the R, G and / or B channel values ​​to be output and removes the IR component according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point to obtain the final Bayer image and IR image.

[0329] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0330] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above methods or to run the above system when executing the program.

[0331] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0332] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0333] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0334] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0335] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the above methods, or to run the above system.

[0336] The RGBIR image processing method, system, terminal and medium provided by the above embodiments of the present invention can simultaneously generate high-quality Bayer images and IR images for RGBIR images obtained by RGBIR sensors, meeting various application scenarios. Without changing the original ISP pipeline, the generated Bayer image can be directly processed using the existing ISP pipeline based on the Bayer image format. It is only necessary to perform fine interpolation on the channel values ​​output in the Bayer format, which will not increase excessive calculations, improve the calculation efficiency, and reduce the calculation cost. The RGBIR pixel arrangement format is pre-interpolated into a format similar to Bayer, and then interpolated based on the Bayer format method to improve the quality of the interpolated image. Fully considering the compatibility with the traditional ISP pipeline, only the part of the RGBIR image to the Bayer image conversion needs to be designed, and the other image processing parts can reuse the existing ISP pipeline, and there is no need to specially design image processing for RGBIR format images, simplifying the design of RGBIR imaging and outputting images with better quality. The image is low-pass filtered before removing the IR component to reduce the edge difference caused by interpolation and reduce color correction errors.

[0337] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0338] All matters not covered in the above embodiments of the present invention are well known in the art.

[0339] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A RGBIR image processing method, characterized in that: include: The R channel pre-interpolation is calculated at the B pixel position, the B channel pre-interpolation is calculated at the R pixel position, and the RGBIR image is transformed into a Bayer format image of RGGIR and a Bayer format image of BGGIR; Based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR, a G channel interpolation is calculated to obtain a G channel image; Using the G channel image, calculate the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel; According to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel, the R, G and / or B channel values ​​to be output are color corrected to remove the IR component, and the final Bayer image and IR image are obtained.

2. The RGBIR image processing method according to claim 1, characterized in that: The method comprises: calculating an R channel pre-interpolation value at a B pixel point position, calculating a B channel pre-interpolation value at an R pixel point position, and transforming an RGBIR image into a Bayer format image based on the R channel pre-interpolation value and the B channel pre-interpolation value. Based on the RGBIR array image, the R channel pre-interpolation is calculated at the B pixel position, and the original B channel value is replaced to generate an array image in RGGIR format. The B channel pre-interpolation is calculated at the R pixel position, and the original R channel value is replaced to generate an array image in BGGIR format. The RGGIR format array image and the BGGIR format array image are the corresponding Bayer format images after transformation.

3. The RGBIR image processing method according to claim 2, characterized in that: The calculating the R channel pre-interpolation at the B pixel point position includes: Determine the current B pixel position, where the position includes: continuous in the horizontal direction, continuous in the vertical direction, or no obvious continuous direction; When the positions are continuous in the horizontal direction, the R channel pre-interpolation result is calculated using the R pixel values ​​adjacent in the horizontal direction, then: R=(Ra+Rb) / 2+Δ H Among them, R is the R channel pre-interpolation result, Ra and Rb are the R pixel values ​​adjacent to the B pixel in the horizontal direction, Δ H It is the horizontal gradient estimation of pixel B; When the positions are continuous in the vertical direction, the R channel pre-interpolation result is calculated using the R pixel values ​​adjacent in the horizontal direction, then: R=(Rc+Rd) / 2+Δ V Among them, Rc and Rd are the R pixel values ​​adjacent to the B pixel in the vertical direction, Δ V It is the vertical gradient estimation of pixel B; When the position has no obvious continuous direction, the R channel pre-interpolation result is calculated as: R=(Ra+Rb+Rc+Rd) / 4+Δ Among them, Δ is the undirected gradient estimate; Using the above method, the B channel interpolation is calculated at the R pixel position.

4. The RGBIR image processing method according to claim 1, characterized in that: The calculating of the G channel interpolation based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR comprises: Based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR, respectively calculating corresponding G channel values; The G channel values ​​of the Bayer format image of the RGGIR and the Bayer format image of the BGGIR are averaged to obtain a G channel interpolation.

5. The RGBIR image processing method according to claim 1, characterized in that: The method of calculating the R channel interpolation, the B channel interpolation and the IR channel interpolation of each pixel using the G channel image includes: Calculate the B channel interpolation of the R pixel, including: Get four B pixels that are adjacent to the R pixel in the horizontal and vertical directions; Calculate the GB color difference D of four B pixels as: D=GB Among them, G is the G channel interpolation of the B pixel point, and B is the pixel value of the original B pixel point of the RGBIR image; Calculate the directional gradient Δ from the R pixel to the four B pixels d for: Δ d =∑|X i -X j | Where X is the original pixel value of the X pixel in the RGBIR image, X is R, G, B or IR, and the subscripts i and j represent the position of the X pixel; The weight w of the pixel R is calculated based on the gradient: w=1 / (D d ) 2 According to the weighted color difference, the B channel interpolation B of the R pixel is calculated as: Calculate the R channel interpolation of the IR pixel, including: Using the above method, first calculate the R channel values ​​of the four pixels at the diagonal positions adjacent to the IR pixel, then calculate the GR color difference of the four pixels, and finally calculate the weighted color difference of the IR pixel to obtain the R channel interpolation of the IR pixel; Calculate the B channel interpolation of other pixels, including: According to the original pixel value of B pixel of RGBIR image, the B channel interpolation of R pixel and the B channel interpolation of IR pixel, the color difference map D between G channel and B channel is calculated. G-B for: D G-B =G-B Among them, G is the G channel interpolation, B is the pixel value of the original B pixel point of the RGBIR image, the B channel interpolation of the R pixel point, and the B channel interpolation of the IR pixel point; The GB color difference at the position without B channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GB color difference map D' of other pixels G-B ; From the G channel image and the GB color difference map of other pixels, the B channel interpolation B′ of other pixels is obtained as follows: B′=G-D’ G-B Calculate the R channel interpolation of other pixels, including: Calculate the color difference map D between the G channel and the R channel G-R for: D G-R =G-R Among them, G is the G channel interpolation, R is the pixel value of the original R pixel of the RGBIR image, the R channel interpolation of the B pixel, and the R channel interpolation of the IR pixel; The GR color difference at the position without R channel interpolation is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the GR color difference map D' of other pixels G-R ; From the G channel image and the GR color difference map of other pixels, the R channel interpolation R′ of other pixels is obtained as follows: R′=G-D’ G-R Among them, G is the G channel interpolation; Calculate the IR channel interpolation of each pixel, including: Calculate the color difference map D between the G channel and the IR channel G-IR for: D G-IR =G-IR Among them, G is the G channel interpolation, and IR is the pixel value of the original IR pixel point of the RGBIR image; The G-IR color difference at the position where there is no IR pixel value is 0; by interpolating the color difference image, the color difference values ​​of other pixels are supplemented to obtain the G-IR color difference map D' of other pixels G-IR ; From the G channel image and the G-IR color difference map of other pixels, the IR channel interpolation IR′ of other pixels is obtained as follows: IR′=G-D' G-IR 。 6. The RGBIR image processing method according to claim 1, characterized in that: The method of performing color correction on the R, G and / or B channel values ​​to be output to remove the IR component according to the G channel interpolation, the R channel interpolation, the B channel interpolation and the IR channel interpolation of each pixel point includes: Perform low-pass filtering on the G image to separate the high-frequency G of the G image H for: G H =G-f l (G) Among them, f l (·) represents the low-pass filter function; Calculate the low frequency x of the current output channel value L for: x L =X-G H Where X represents R, G or B channel; Perform color correction on the color channels of the output Bayer image, including: X′ L =M X p Where X represents the output channel value; p = [R, G, B, IR] T is the input value, which is the four-channel value of the current pixel; M X Represents the mapping relationship between the output channel X and the input value p; The color-corrected value is added back to the high frequency for detail enhancement to obtain the final output value X output for: X output =X′ L +wG H Where w is the enhancement coefficient; Perform IR channel image enhancement on the output IR image to obtain the final IR image IR output for: AND output =IR+wG H Among them, IR is an IR image composed of IR channel interpolation, and the number of pixels of the IR image is consistent with the number of pixels of the original RGBIR image.

7. An RGBIR image processing system, characterized in that: include: A format conversion module, which is used to calculate the R channel pre-interpolation at the B pixel point position, calculate the B channel pre-interpolation at the R pixel point position, and convert the RGBIR image into a Bayer format image of RGGIR and a Bayer format image of BGGIR; A G image generation module, which calculates G channel interpolation based on the Bayer format image of the RGGIR and the Bayer format image of the BGGIR to obtain a G channel image; A channel interpolation calculation module, which uses the G channel image to calculate the R channel interpolation, B channel interpolation and IR channel interpolation of each pixel; The image processing module performs color correction on the R, G and / or B channel values ​​to be output and removes the IR component according to the G channel interpolation, R channel interpolation, B channel interpolation and IR channel interpolation of each pixel point to obtain the final Bayer image and IR image.

8. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 6, or to run the system described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 6, or to run the system described in claim 7.