Method for demosaicing image with near-infrared channel, image processing system and electronic equipment

Abstract

The invention provides an image demosaicing method with a near-infrared channel, an image processing system and electronic equipment, and the method comprises the steps: carrying out the resampling of G, B and R of an original image, and obtaining a first G channel image, a first B channel image and a first R channel image; further obtaining a BG color difference image and an RG color difference image; performing preliminary demosaicing interpolation processing by using the G channel value and the NIR channel value to obtain a second G channel image; respectively reconstructing the BG color difference image and the RG color difference image with the second G channel image to obtain a second B channel image and a second R channel image; further obtaining an RGB image; and according to the RGB image and the NIR channel value, performing interpolation calculation to obtain an NIR image. Through the method, some information which cannot be acquired by RGB can be acquired in the NIR channel, so that the negative influence caused by the lack of the RGB sampling rate is made up while the near-infrared advantage is reflected, the image with higher quality is obtained, and the problem of how to perform demosaicing processing on the image with the NIR channel is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image de-mosaicing method with a near-infrared channel, an image processing system and an electronic device. Background Art

[0002] Usually, image sensors can only capture grayscale images. In order to output color images, it is necessary to use the Bayer filter to separate the light into three color channels: R (red), G (green), and B (blue), and arrange each pixel in the Bayer pattern. The image output by the Bayer filter is actually a mosaic image, in which each pixel contains only one of the three color channels of red, green, and blue. For example, in the RGGB Bayer pattern, each image block contains 2×2 pixels, the upper left pixel is the R channel value, the upper right and lower left pixels are the G channel values, and the lower right pixel is the B channel value. Therefore, the resolution of the G channel in the mosaic image is only half of that of the final image, and the resolution of the B and R channels is only one-quarter of that of the final image. Later, the demosaicing algorithm is needed to restore the low-resolution channel to the high-resolution channel value, and finally output the RGB image.

[0003] With the development of science and technology, people have found that near-infrared (NIR) images have unique advantages over RGB images. For example, near-infrared light has high penetration, so near-infrared images can show the internal structure of certain materials. Another example is that near-infrared images are insensitive to lighting conditions and are less affected by weather, so the imaging effect is better than RGB images under low light or shadow conditions. As a result, people have developed new image sensors that can collect data from four channels: R, G, B, and NIR, in order to use the imaging advantages of near-infrared to enhance the weaker features of the original RGB channel and obtain better quality color images.

[0004] However, since the raw data sampled by the new image sensor has a new NIR channel, and the R and B channels have the characteristics of staggered distribution of the Bayer arrangement, the sampling rate of the R and B channels is lower than that of the traditional image sensor; at the same time, the NIR channel cannot be used to generate RGB values, but can only be used for weight calculation. Therefore, the traditional de-mosaic interpolation method is not applicable to the new image sensor. If the traditional de-mosaic interpolation method is used, it will not only fail to reflect the advantages of the new image sensor, but also cause problems such as color confusion and fail to output images normally. Summary of the invention

[0005] The object of the present invention is to provide an image demosaicing method, an image processing system and an electronic device with a near-infrared channel, so as to solve the problem of how to demosaic an image with a NIR channel.

[0006] In order to solve the above technical problems, the present invention provides an image demosaicing method with a near-infrared channel, comprising: Acquire an original image, wherein the original image has a G channel, a B channel, an R channel, and a NIR channel; Resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image with the same resolution; Subtracting the first B channel image and the first R channel image from the first G channel image to obtain a first BG color difference image and a first RG color difference image, and upsampling the first BG color difference image and the first RG color difference image back to full resolution to obtain a second BG color difference image and a second RG color difference image; Performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image; Reconstructing the second BG color difference image and the second RG color difference image with the second G channel image respectively to obtain a second B channel image and a second R channel image; combining the second B channel image, the second R channel image and the second G channel image to obtain a full-resolution RGB image; According to the NIR channel values ​​of the RGB image and the original image, the full-resolution NIR image is obtained by interpolation.

[0007] Optionally, in the image demosaicing method with a near-infrared channel, the method of resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image of the same resolution includes: According to the sampling rates of the G channel, B channel and R channel of the original image, confirm the target sampling rate; For the G channel, a sampling window of the current pixel is determined according to the target sampling rate, and the average of the G channel values ​​of all G pixels in the sampling window is calculated as the G channel value after resampling of the current pixel, thereby obtaining a first G channel image; For the B channel or the R channel, the multi-directional gradient of the current pixel is calculated according to the target sampling rate, and the B channel value or the R channel value of the current pixel after resampling is obtained by using the multi-directional gradient calculation according to the demosaic interpolation method; For the NIR channel, the average of the B channel values ​​of the two B pixels adjacent to the current pixel is used as the B channel value of the current pixel after resampling, and the average of the R channel values ​​of the two R pixels adjacent to the current pixel is used as the R channel value of the current pixel after resampling; The resampled B channel value is combined with the original B channel value to obtain a first B channel image; the resampled R channel value is combined with the original R channel value to obtain a first R channel image.

[0008] Optionally, in the image demosaicing method with a near infrared channel, the method of performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image includes: Calculate the multi-directional first gradient information of the G channel; Calculate multi-directional second gradient information of the NIR channel; The first gradient information and the second gradient information are weighted and fused to obtain a second G channel image.

[0009] Optionally, in the image demosaicing method with a near-infrared channel, the method for calculating multi-directional first gradient information of the G channel includes: Calculate the first vertical gradient of the G channel , expressed as:

[0010] Calculate the first horizontal gradient of the G channel , expressed as:

[0011] in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel.

[0012] Optionally, in the image demosaicing method with a near infrared channel, the method for calculating multi-directional second gradient information of the NIR channel includes: If the current pixel is an R pixel or a B pixel, the second gradient in the vertical direction of the NIR channel is calculated , expressed as: / 2 Calculate the second gradient in the horizontal direction of the NIR channel , expressed as: / 2 If the current pixel is a NIR pixel, calculate the second gradient in the vertical direction of the NIR channel , expressed as:

[0013] Calculate the second gradient in the horizontal direction of the NIR channel , expressed as:

[0014] in, Indicates the coordinate position of the current pixel. Indicates the NIR channel value of the current pixel.

[0015] Optionally, in the image demosaicing method with a near-infrared channel, the method of weighted fusion of the first gradient information and the second gradient information to obtain the second G channel image includes: Obtaining a comprehensive gradient according to the first gradient information and the second gradient information, wherein the comprehensive gradient includes a vertical comprehensive gradient and a horizontal comprehensive gradient; According to the vertical integrated gradient and the horizontal integrated gradient, the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction are determined; Calculate the vertical interpolation of the current pixel and horizontal interpolation , expressed as:

[0016]

[0017] in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel; According to the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction, a weighted sum calculation is performed on the vertical interpolation and the horizontal interpolation of the current pixel to obtain a second G channel image.

[0018] Optionally, in the image demosaicing method with a near-infrared channel, the method of obtaining a full-resolution NIR image by interpolation calculation based on the NIR channel values ​​of the RGB image and the original image includes: Calculate the multi-directional gradient information, brightness information and texture information of RGB images; Calculate the multi-directional gradient information and texture information of the original NIR image; According to the brightness information and texture information of the RGB image, the multi-directional gradient information of the RGB image and the multi-directional gradient information of the original NIR image are weighted and fused to obtain the final gradient information; According to the final gradient information, confirm the interpolation weight of the NIR channel; According to the interpolation weight of the NIR channel, the original NIR channel value is interpolated to obtain a full-resolution NIR image.

[0019] Optionally, in the image demosaicing method with a near-infrared channel, the image demosaicing method with a near-infrared channel further includes: Performing texture judgment and optimization on the second G channel image to obtain a third G channel image; The second BG color difference image and the second RG color difference image are reconstructed with the third G channel image respectively to obtain a second B channel image and a second R channel image.

[0020] Optionally, in the image demosaicing method with a near-infrared channel, the method of performing texture judgment and optimization on the second G channel image to obtain a third G channel image includes: In a preset window of the second G channel image, the difference between the texture values ​​of adjacent pixels and the texture values ​​of every other pixel is calculated to obtain a texture difference; If the texture difference is greater than the texture threshold, the pixels in the current preset window are filtered; The second G channel image is traversed to obtain a third G channel image.

[0021] In order to solve the above technical problems, the present invention further provides an image processing system for implementing the image demosaicing method with a near-infrared channel as described in any one of the above items, wherein the image processing system comprises: An image acquisition module, used for acquiring original images; A demosaic processing module is used to perform a demosaic processing using the G channel value, the B channel value, the R channel value and the NIR channel value of the original image to obtain a full-resolution RGB image and a NIR image; Image output module, used to output full-resolution RGB images and NIR images.

[0022] In order to solve the above technical problems, the present invention also provides an electronic device, including a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image de-mosaicing method with a near-infrared channel as described in any of the above items is executed.

[0023] The present invention provides an image de-mosaicing method with a near-infrared channel, an image processing system and an electronic device, comprising: obtaining an original image, wherein the original image has a G channel, a B channel, an R channel and a NIR channel; resampling the G channel, the B channel and the R channel of the original image to obtain a first G channel image, a first B channel image and a first R channel image with the same resolution; subtracting the first B channel image and the first R channel image from the third G channel image respectively to obtain a first BG color difference image and a first RG color difference image, and upsampling the first BG color difference image and the first RG color difference image Return to full resolution to obtain the second BG color difference image and the second RG color difference image; perform preliminary de-mosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain the second G channel image; reconstruct the second BG color difference image and the second RG color difference image with the second G channel image respectively to obtain the second B channel image and the second R channel image; combine the second B channel image, the second R channel image and the second G channel image to obtain a full-resolution RGB image; interpolate and calculate the full-resolution NIR image according to the RGB image and the NIR channel value of the original image. Through this method, some information that cannot be collected by RGB can be obtained in the NIR channel, so that while reflecting the advantages of near-infrared, the negative impact caused by the lack of RGB sampling rate is compensated, and a higher quality image is obtained, solving the problem of how to de-mosaic images with NIR channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of the image demosaicing method with a near-infrared channel provided in this embodiment; Figure 2 A schematic diagram of pixel arrangement with a G channel, a B channel, an R channel and an NIR channel provided in this embodiment; Figure 3 This is a schematic diagram of the structure of the image processing system provided in this embodiment. DETAILED DESCRIPTION

[0025] The image de-mosaicing method, image processing system and electronic device with a near-infrared channel proposed in the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the accompanying drawings is often part of the actual structure. In particular, the emphasis that each drawing needs to show is different, and sometimes different proportions are used.

[0026] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects in order to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that the structures used in this way can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] This embodiment provides an image demosaicing method with a near infrared channel. Figure 1 As shown, including: S1, acquiring an original image, wherein the original image has a G channel, a B channel, an R channel and a NIR channel; S2, resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image of the same resolution; S3, respectively subtracting the first B channel image and the first R channel image from the first G channel image to obtain a first BG color difference image and a first RG color difference image, and upsampling the first BG color difference image and the first RG color difference image back to full resolution to obtain a second BG color difference image and a second RG color difference image; S4, performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image; S5, reconstructing the second BG color difference image and the second RG color difference image with the second G channel image respectively to obtain a second B channel image and a second R channel image; S6, combining the second B channel image, the second R channel image and the second G channel image to obtain a full-resolution RGB image; S7, according to the NIR channel values ​​of the RGB image and the original image, interpolation calculation is performed to obtain a full-resolution NIR image.

[0028] The image demosaicing method with a near-infrared channel provided in this embodiment can obtain some information that cannot be collected by RGB in the NIR channel, so that while reflecting the advantages of near-infrared, the negative impact caused by the lack of RGB sampling rate can be compensated, and higher quality images can be obtained, solving the problem of how to demosaic images with NIR channels.

[0029] In practical applications, the implementation order of step S4 can be placed before step S2, before S3, or performed simultaneously with step S2 or S3, and this application does not limit this. Technical solutions after the order of steps is changed without violating the purpose of this application should also fall within the scope of protection of this application.

[0030] Specifically, in this embodiment, step S1, obtaining an original image. The original image obtained in this step should be a grayscale image having a G channel, a B channel, an R channel, and an NIR channel, so that in the process of converting the grayscale image to a color image, the image demosaicing process can be used to obtain a color image and a NIR image, and the advantages of visible light and invisible light can be combined to obtain a better demosaicing effect.

[0031] In practical applications, a new image sensor can be used to obtain an original image, so that the original image has a G channel, a B channel, an R channel, and an NIR channel.

[0032] The pixel arrangement with G channel, B channel, R channel and NIR channel is shown as follows Figure 2 As shown, G pixels are arranged alternately in rows and columns, R pixels and B pixels are arranged alternately in the same row and column of even rows and even columns, and NIR pixels are arranged alternately in odd rows and odd columns. In this way, the sampling rate of the G channel is 1 / 2, the sampling rate of the NIR channel is 1 / 4, and the sampling rate of the R channel and the B channel is only 1 / 8.

[0033] Further, in this embodiment, step S2, the method of resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image of the same resolution includes: S21, confirming a target sampling rate according to the sampling rates of the G channel, the B channel, and the R channel of the original image.

[0034] In this embodiment, considering that in a data mode with an NIR channel, the data of these channels may not necessarily be directly extracted to form an image with geometric scaling of the original resolution, and the distribution law or sampling rate of each channel in different data modes may be different. In order to facilitate the subsequent color difference calculation, the data of each channel needs to be sampled proportionally according to the original resolution, wherein, for channels with high sampling rates, downsampling is generally performed; and for channels with low sampling rates, upsampling or demosaicing is generally performed to achieve the purpose.

[0035] by Figure 2 Taking the pixel arrangement shown as an example, the sampling rate of the G channel is 1 / 2, and the sampling rates of the R channel and the B channel are 1 / 8. Therefore, an intermediate ratio can be selected, that is, they are all resampled to 1 / 4 (1 / 2 in length and width). That is, in this embodiment, the target sampling rate is 1 / 4.

[0036] S22, for the G channel, confirm the sampling window of the current pixel according to the target sampling rate, and calculate the average of the G channel values ​​of all G pixels in the sampling window as the G channel value after resampling of the current pixel, to obtain a first G channel image.

[0037] In this embodiment, a 2×2 sampling window may be used, and the average value of the pixel values ​​of two G pixels in the sampling window is used as the G channel value after downsampling. In this way, the sampling accuracy can be guaranteed while reducing the computational complexity.

[0038] In this way, the G channel values ​​of the two G pixels are merged into one G channel value, so that the sampling rate is reduced from 1 / 2 to 1 / 4, reaching the target sampling rate.

[0039] S23, for the B channel or the R channel, the multi-directional gradient of the current pixel is calculated according to the target sampling rate, and the B channel value or the R channel value of the current pixel after resampling is obtained by using the multi-directional gradient calculation according to the demosaic interpolation method.

[0040] Specifically, Figure 2 Taking the pixel arrangement shown as an example, the current pixel is the R pixel with coordinates (4,3), then the process of resampling it in the B channel can be expressed as:

[0041]

[0042]

[0043] When performing interpolation calculations, the four B pixels closest to the R pixel are selected for calculation, so as to reduce the calculation complexity while ensuring the resampling accuracy. In this way, the resampled values ​​of all R pixels in the B channel can be obtained.

[0044] The process of resampling B pixels in the R channel can refer to the above process, and this application will not go into details.

[0045] In this way, the R channel includes the R channel value of the B pixel, so that the sampling rate of the R channel is increased from 1 / 8 to 1 / 4, reaching the target sampling rate; similarly, the B channel includes the B channel value of the R pixel, so that the sampling rate of the B channel is increased from 1 / 8 to 1 / 4, reaching the target sampling rate.

[0046] S24, for the NIR channel, the average of the B channel values ​​of the two B pixels adjacent to the current pixel is used as the B channel value of the current pixel after resampling, and the average of the R channel values ​​of the two R pixels adjacent to the current pixel is used as the R channel value of the current pixel after resampling.

[0047] by Figure 2 Taking the pixel arrangement shown as an example, the current pixel is an NIR pixel with coordinates (3,4). Considering that the number of B pixels and R pixels near the NIR pixel is small, in order to ensure image processing efficiency, this embodiment directly uses the average of the B channel values ​​of two adjacent B pixels as the B channel value of the current pixel after resampling, that is, the interpolated B channel value of the NIR pixel at the position (3,4) can be expressed as:

[0048] Similarly, when performing R channel interpolation, the two R pixels around the NIR pixel can be used to perform mean calculation to obtain the interpolated R channel value corresponding to the NIR pixel.

[0049] S25, combining the resampled B channel value with the original B channel value to obtain a first B channel image; combining the resampled R channel value with the original R channel value to obtain a first R channel image.

[0050] In this way, the resampled B channel value includes the interpolation of the R pixel and the NIR pixel in the B channel and the original B pixel value; the resampled R channel value includes the interpolation of the B pixel and the NIR pixel in the R channel and the original R pixel value, so that the sampling rate of the B channel and the R channel becomes 1 / 2.

[0051] At this time, a method similar to step S22 is used again, using a 2×2 sampling window, and taking the average pixel value of two B pixels in the sampling window as the down-sampled B channel value, so that the two B channel values ​​are merged into one B channel value, so that the sampling rate is reduced from 1 / 2 to 1 / 4, achieving the target sampling rate, and obtaining the first B channel image.

[0052] Similarly, the average pixel value of the two R pixels in the sampling window is used as the R channel value after downsampling, so that the two R channel values ​​are merged into one R channel value, so that the sampling rate is reduced from 1 / 2 to 1 / 4, reaching the target sampling rate, and obtaining the first R channel image.

[0053] Furthermore, in this embodiment, in step S3, the first B channel image and the first R channel image are respectively subtracted from the first G channel image to obtain a first BG color difference image and a first RG color difference image, and the first BG color difference image and the first RG color difference image are upsampled back to full resolution to obtain a second BG color difference image and a second RG color difference image.

[0054] In this embodiment, the process of subtracting the first B channel image and the first R channel image from the first G channel image to obtain the first BG color difference image and the first RG color difference image can be expressed as:

[0055]

[0056] in, represents the first R channel image, represents the first B channel image, represents the first G channel image, represents the first RG color difference image, Represents the first BG color difference image.

[0057] And, in this embodiment, the process of upsampling the first BG color difference image and the first RG color difference image back to full resolution to obtain the second BG color difference image and the second RG color difference image can be expressed as:

[0058]

[0059] in, represents the second RG color difference image, represents the second BG color difference image, Indicates upsampling.

[0060] It should be noted that the method of subtracting the first B channel image and the first R channel image from the first G channel image respectively to obtain the first BG color difference image and the first RG color difference image is well known to those skilled in the art, and the method of upsampling the first BG color difference image and the first RG color difference image back to full resolution is well known to those skilled in the art, and this application will not go into details therein.

[0061] Furthermore, in this embodiment, in step S4, the method of performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image includes: S41, calculating multi-directional first gradient information of the G channel.

[0062] In this embodiment, the first gradient information includes a first gradient in the vertical direction and a first gradient in the horizontal direction. The calculation formula can be expressed as:

[0063] The first horizontal gradient of the G channel The calculation formula can be expressed as:

[0064] in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel.

[0065] In practical applications, when calculating the multi-directional first gradient information of the G channel, the pixel channels are not distinguished, that is, the B pixels, R pixels and NIR pixels can all be calculated using the above formula. Figure 2 Taking the pixel arrangement shown in the figure as an example, the R pixel at the coordinate position (4,3) is calculated. The first vertical gradient of the G channel of the R pixel is It is expressed as:

[0066] And, the first horizontal gradient of the G channel of the R pixel It is expressed as:

[0067] Since there are G pixels adjacent to the B pixels, R pixels and NIR pixels on both sides in the vertical direction and on both sides in the horizontal direction, this embodiment uses the pixel values ​​of the adjacent G pixels to calculate the gradient, which can reduce the calculation complexity as much as possible while obtaining more accurate gradient information, thereby reducing system power consumption and improving system processing efficiency.

[0068] Of course, in practical applications, the first gradient information may also include a first gradient in a diagonal direction and a first gradient in an anti-diagonal direction. The specific calculation method of the gradient is well known to those skilled in the art and will not be described in detail in this application.

[0069] S42, calculating multi-directional second gradient information of the NIR channel.

[0070] In this embodiment, the second gradient information includes a second gradient in the vertical direction and a second gradient in the horizontal direction. Since the B pixels and R pixels are arranged differently from the NIR pixels, they need to be calculated separately: If the current pixel is an R pixel or a B pixel, the second gradient in the vertical direction of the NIR channel is calculated , expressed as: / 2 Calculate the second gradient in the horizontal direction of the NIR channel , expressed as: / 2 in, Indicates the coordinate position of the current pixel. Indicates the NIR channel value of the current pixel.

[0071] Still Figure 2 Taking the pixel arrangement shown in the figure as an example, the R pixel at the coordinate position (4,3) is taken for calculation. The second gradient in the vertical direction of the NIR channel of the R pixel is It is expressed as: / 2 And, the second gradient in the horizontal direction of the NIR channel of the R pixel It is expressed as: / 2 When calculating the second gradient information of the NIR channel of the R pixel and the B pixel, this embodiment uses the pixel values ​​of the four adjacent NIR pixels to calculate the gradient. This can reduce the calculation complexity as much as possible while obtaining more accurate gradient information, thereby reducing system power consumption and improving system processing efficiency.

[0072] If the current pixel is a NIR pixel, calculate the second gradient in the vertical direction of the NIR channel , expressed as:

[0073] Calculate the second gradient in the horizontal direction of the NIR channel , expressed as:

[0074] in, Indicates the coordinate position of the current pixel. Indicates the NIR channel value of the current pixel.

[0075] Still Figure 2 Taking the pixel arrangement shown in the figure as an example, the NIR pixel at the coordinate position (3,4) is taken for calculation. The second gradient in the vertical direction of the NIR channel of the NIR pixel is It is expressed as:

[0076] And, the second gradient in the horizontal direction of the NIR channel of the NIR pixel It is expressed as:

[0077] When calculating the second gradient information of the NIR channel of the NIR pixel, this embodiment uses the pixel values ​​of two NIR pixels that are adjacent in the horizontal and vertical directions to calculate the gradient. This can reduce the calculation complexity as much as possible while obtaining more accurate gradient information, thereby reducing system power consumption and improving system processing efficiency.

[0078] S43, weighted fusion of the first gradient information and the second gradient information to obtain a second G channel image.

[0079] Specifically, in this embodiment, first, a comprehensive gradient is obtained according to the first gradient information and the second gradient information, wherein the comprehensive gradient includes a vertical comprehensive gradient and a horizontal comprehensive gradient. In practical applications, the method of fusing to obtain the comprehensive gradient can be determined according to the characteristics of the image sensor, the characteristics of the image processing system, etc. For example, the first gradient information and the second gradient information can be linearly combined to obtain the comprehensive gradient, or the square root average of the first gradient information and the second gradient information can be calculated to obtain the comprehensive gradient, or the first gradient information and the second gradient information can be nonlinearly operated to obtain the comprehensive gradient, etc. This application does not limit this. The vertical comprehensive gradient It is expressed as:

[0080] And, the horizontal integrated gradient It is expressed as:

[0081] Here, f represents the method used to combine the first gradient information and the second gradient information.

[0082] Then, according to the vertical integrated gradient and the horizontal integrated gradient, the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction are determined. This embodiment provides an implementation as follows: Interpolation weight in vertical direction It is expressed as:

[0083] Interpolation weight in horizontal direction It is expressed as:

[0084] in, and is a constant and can be calibrated and determined according to the application conditions of the image sensor and image processing system in practical applications.

[0085] In the calculation method of the interpolation weights in the vertical direction and the interpolation weights in the horizontal direction given in this embodiment, a smaller interpolation weight can be assigned to the direction with a larger gradient value, and a larger interpolation weight can be assigned to the direction with a smaller gradient value, thereby ensuring the balance of each direction during the interpolation calculation.

[0086] Next, calculate the vertical interpolation of the current pixel and horizontal interpolation , which can be specifically expressed as:

[0087]

[0088] in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel.

[0089] Calculate the vertical interpolation of the current pixel and horizontal interpolation When R pixels, B pixels and NIR pixels are not distinguished, the above formula can be used for calculation. Figure 2 Take the pixel arrangement shown in the figure as an example, and take the R pixel at the coordinate position (4,3) for calculation. The vertical interpolation of the R pixel is It is expressed as:

[0090] And, the horizontal interpolation of the R pixel Shown as:

[0091] Finally, according to the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction, the vertical interpolation and the horizontal interpolation of the current pixel are weighted and summed to obtain the second G channel image. That is, the G channel values ​​of the R pixel, the B pixel, and the NIR pixel in the second G channel image can be expressed as:

[0092] The pixel value of the original G pixel is retained and combined with the G channel value obtained by interpolation calculation of the R pixel, the B pixel and the NIR pixel, so as to obtain a second G channel image.

[0093] In the process of obtaining the second G channel image in step S4 of this embodiment, the NIR channel only participates in the texture and direction calculation to obtain the interpolation weight of the G channel, but does not participate in the generation of the G channel value. By using the original G channel value and the original NIR image to calculate the multi-dimensional and multi-level image gradient, by calculating the de-mosaic interpolation weights in the horizontal and vertical directions, the original G channel value is interpolated from the horizontal and vertical directions respectively and weighted fused according to the weights. Due to the difference in the photosensitivity characteristics of NIR pixels, in some areas with defective G channel texture, NIR has a probability of performing better. At this time, the gradient can be calculated more accurately by combining the NIR image gradient.

[0094] In practical applications, the specific calculation method of step S23 can be similar to the calculation method of the preliminary demosaicing interpolation processing in step S4, to calculate the multi-directional gradients of the R channel and the B channel, and then use the multi-directional gradients to obtain multi-directional interpolation weights. In order to reduce the computational complexity as much as possible, in practical applications, step S23 can directly use the first gradient information and difference weights calculated in step S4.

[0095] Further, in this embodiment, in step S5, the second BG color difference image and the second RG color difference image are reconstructed with the second G channel image respectively to obtain a second B channel image and a second R channel image; in this embodiment, the process can be expressed as:

[0096]

[0097] in, Represents the second R channel image, Represents the second B channel image.

[0098] It should be noted that the method of reconstructing the second BG color difference image and the second RG color difference image respectively with the first G channel image is well known to those skilled in the art, and will not be described in detail in this application.

[0099] Preferably, in this embodiment, step S5 may include: S51, performing texture judgment and optimization on the second G channel image to obtain a third G channel image.

[0100] Specifically, in this embodiment, the step includes: First, in a preset window of the second G channel image, the difference between the texture value of the adjacent pixel and the texture value of the alternate pixel is calculated to obtain a texture difference, wherein the adjacent pixel refers to the pixel immediately adjacent to the current pixel, and the alternate pixel refers to the pixel one pixel away from the current pixel.

[0101] In practical applications, different scales and calculation methods can be used to judge or eliminate different textures. Common calculation methods include adjacent point difference, single point difference, two point difference, directional gradient operator, etc. For the direction, you can choose horizontal direction, vertical direction, diagonal direction (45° oblique), anti-diagonal direction (135° oblique), etc.

[0102] For example, for maze patterns, we can first calculate the maximum value of multi-scale details in different directions within a window, and then take the difference between the maximum value of the details of adjacent pixels and every other pixel. The larger the difference, the stronger the maze pattern judgment, and the greater the weight for texture restoration.

[0103] by Figure 2 Take the pixel arrangement shown as an example. The current pixel coordinate is (4,3). The current pixel is placed at the center of the 5×5 window. The original G channel data is used to calculate the anti-diagonal direction (135° oblique) texture, where: The texture values ​​of adjacent pixels can be expressed as:

[0104] And, the texture value of every other pixel can be expressed as:

[0105] Then, the texture difference in the anti-diagonal direction (135° oblique) can be expressed as:

[0106] Similarly, those skilled in the art can obtain texture differences in other directions, which will not be elaborated in this application.

[0107] If the texture difference is greater than the texture threshold, it is considered that there is an abnormal texture in the window, so the current pixel needs to be filtered. This embodiment provides a filtering method as follows:

[0108]

[0109] Among them, C and B are constants, which can be calibrated and determined according to the application conditions of the image sensor and image processing system in practical applications; The first G channel image is filtered. In practical applications, a filtering method can be reasonably selected according to actual needs.

[0110] If the texture difference is less than the texture threshold, it is considered that there is no abnormal texture in the window, so there is no need to filter the current pixel. That is, the G channel value of the current pixel in the second G channel image retains the G channel value in the first G channel image, which is simply expressed as G2=G1.

[0111] In practical applications, the texture threshold can be set according to actual conditions. Also, different optimization methods can be selected for different textures, which is not limited in this application.

[0112] The second G channel image is traversed to obtain a third G channel image.

[0113] In step S51 of this embodiment, considering that the NIR channel needs to be photosensitized, the lens lacks a near-infrared filter. The near-infrared light passing through the lens will have a certain impact on the RGB channels, and there is a risk of abnormal texture. The sampling rate of the G channel is usually the highest, so the texture judgment is performed on the G channel (the second G channel image). If abnormal texture exists, the texture is eliminated, so that the third G channel image after the abnormal texture is eliminated can be used to guide the demosaicing process of the R channel, B channel, and NIR channel, thereby suppressing the abnormal texture of the final image and effectively improving the image quality.

[0114] S52: Reconstruct the second BG color difference image and the second RG color difference image with the third G channel image respectively to obtain a second B channel image and a second R channel image.

[0115] In this way, by performing texture optimization on the second G channel image obtained by the preliminary demosaicing interpolation process, the texture can be removed and the image quality can be improved while ensuring the accuracy of gradient calculation in each area of ​​the image.

[0116] Furthermore, in this embodiment, in step S6, the second B channel image, the second R channel image and the second G channel image are combined to obtain a full-resolution RGB image.

[0117] In this embodiment, the process can be expressed as:

[0118] in, It indicates that the second B channel image, the second R channel image and the second G channel image are combined.

[0119] It should be noted that the method of combining the second B channel image, the second R channel image and the second G channel image is well known to those skilled in the art, and will not be described in detail in this application.

[0120] Furthermore, in this embodiment, in step S7, the method of obtaining a full-resolution NIR image by interpolation calculation according to the NIR channel values ​​of the RGB image and the original image includes: S71, calculating multi-directional gradient information gradRGB, brightness information Y and texture information detailRGB of the RGB image.

[0121] Among them, the method for calculating multi-directional gradient information can refer to step S4, the method for calculating texture information can refer to step S5, and the method for calculating brightness information is well known to those skilled in the art and will not be repeated in this application.

[0122] S72, calculating multi-directional gradient information gradN and texture information detailN of the original NIR image.

[0123] Among them, the method for calculating multi-directional gradient information can refer to step S4, and the method for calculating texture information can refer to step S5, which will not be repeated in this application.

[0124] S73, according to the brightness information and texture information of the RGB image, weighted fusion of the multi-directional gradient information of the RGB image and the multi-directional gradient information of the original NIR image to obtain the final gradient information .

[0125] This embodiment provides a method for calculating the final gradient information as follows:

[0126]

[0127]

[0128] in, The function h is monotonically decreasing with respect to Y, and It is monotonically increasing. In practical applications, different data models can be used to calibrate the function h.

[0129] S74, confirming the interpolation weight of the NIR channel according to the final gradient information.

[0130] Specifically, reference may be made to step S2 to obtain the interpolation weight according to the gradient information, which will not be described in detail in this application.

[0131] S75 , performing interpolation calculation on the original NIR channel value according to the interpolation weight of the NIR channel to obtain a full-resolution NIR image.

[0132] Specifically, the full-resolution NIR channel value may be calculated by weighted summation according to the interpolation weights, which will not be described in detail in this application.

[0133] The image demosaicing method with a near-infrared channel provided in this embodiment adds near-infrared (NIR) channel data to the processed raw data compared to traditional RGB raw data. The sampling rates of the R channel and the B channel are lower than those of traditional image blocks, and color restoration is more difficult. However, the NIR channel obtained by demosaicing interpolation can obtain some information that cannot be collected by RGB. Therefore, this embodiment can not only generate a normal visible image by performing demosaicing interpolation processing on the collected raw data, but also compensate for the negative impact of the lack of RGB sampling rate while reflecting the advantages of near-infrared (NIR) pixels, thereby obtaining a better image.

[0134] The image de-mosaicing method with a near-infrared channel provided in this embodiment can provide more image detail information for the subsequent processing of the full-resolution RGB image, thereby guiding the subsequent image processing process to perform denoising, dehazing, and enhancement on the RGB image, so that the original image can be output as a clearer image after processing, thereby improving the image quality.

[0135] This embodiment also provides an image processing system for implementing the above-mentioned image demosaicing method with a near-infrared channel, such as Figure 3 As shown, the image processing system comprises: An image acquisition module, used for acquiring original images; A demosaic processing module is used to perform a demosaic processing using the G channel value, the B channel value, the R channel value and the NIR channel value of the original image to obtain a full-resolution RGB image and a NIR image; Image output module, used to output full-resolution RGB images and NIR images.

[0136] The image processing system provided in this embodiment can obtain some information that cannot be collected by RGB in the NIR channel through the processing of the de-mosaic processing module, so that while reflecting the advantages of near-infrared, the negative impact caused by the lack of RGB sampling rate can be compensated, and higher quality images can be obtained, solving the problem of how to de-mosaic images with NIR channels.

[0137] Furthermore, this embodiment further provides an electronic device, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image de-mosaicing method with a near-infrared channel as described above is executed.

[0138] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this.

[0139] The present embodiment provides an image demosaicing method with a near-infrared channel, an image processing system, and an electronic device, including: obtaining an original image, wherein the original image has a G channel, a B channel, an R channel, and a NIR channel; resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image with the same resolution; subtracting the first B channel image and the first R channel image from the third G channel image to obtain a first BG color difference image and a first RG color difference image, and upsampling the first BG color difference image and the first RG color difference image. The method is to sample back to full resolution to obtain a second BG color difference image and a second RG color difference image; perform preliminary de-mosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image; reconstruct the second BG color difference image and the second RG color difference image with the second G channel image respectively to obtain a second B channel image and a second R channel image; combine the second B channel image, the second R channel image and the second G channel image to obtain a full-resolution RGB image; interpolate and calculate the full-resolution NIR image according to the NIR channel value of the RGB image and the original image. Through this method, some information that cannot be collected by RGB can be obtained in the NIR channel, so that while reflecting the advantages of near-infrared, the negative effects caused by the lack of RGB sampling rate are compensated, and higher-quality images are obtained, solving the problem of how to de-mosaic images with NIR channels.

[0140] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. An image demosaicing method with a near-infrared channel, characterized in that: include: Acquire an original image, wherein the original image has a G channel, a B channel, an R channel, and a NIR channel; Resampling the G channel, the B channel, and the R channel of the original image to obtain a first G channel image, a first B channel image, and a first R channel image with the same resolution; Subtracting the first B channel image and the first R channel image from the first G channel image to obtain a first BG color difference image and a first RG color difference image, and upsampling the first BG color difference image and the first RG color difference image back to full resolution to obtain a second BG color difference image and a second RG color difference image; Performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image; Reconstructing the second BG color difference image and the second RG color difference image with the second G channel image respectively to obtain a second B channel image and a second R channel image; combining the second B channel image, the second R channel image and the second G channel image to obtain a full-resolution RGB image; According to the NIR channel values ​​of the RGB image and the original image, the full-resolution NIR image is obtained by interpolation.

2. The image demosaicing method with a near-infrared channel according to claim 1, characterized in that: The method of resampling the G channel, the B channel and the R channel of the original image to obtain a first G channel image, a first B channel image and a first R channel image of the same resolution includes: According to the sampling rates of the G channel, B channel and R channel of the original image, confirm the target sampling rate; For the G channel, a sampling window of the current pixel is determined according to the target sampling rate, and the average of the G channel values ​​of all G pixels in the sampling window is calculated as the G channel value after resampling of the current pixel, thereby obtaining a first G channel image; For the B channel or the R channel, the multi-directional gradient of the current pixel is calculated according to the target sampling rate, and the B channel value or the R channel value of the current pixel after resampling is obtained by using the multi-directional gradient calculation according to the demosaic interpolation method; For the NIR channel, the average of the B channel values ​​of the two B pixels adjacent to the current pixel is used as the B channel value of the current pixel after resampling, and the average of the R channel values ​​of the two R pixels adjacent to the current pixel is used as the R channel value of the current pixel after resampling; The resampled B channel value is combined with the original B channel value to obtain a first B channel image; the resampled R channel value is combined with the original R channel value to obtain a first R channel image.

3. The image demosaicing method with a near-infrared channel according to claim 1, characterized in that: The method of performing preliminary demosaicing interpolation processing using the G channel value and the NIR channel value of the original image to obtain a second G channel image includes: Calculate the multi-directional first gradient information of the G channel; Calculate multi-directional second gradient information of the NIR channel; The first gradient information and the second gradient information are weighted and fused to obtain a second G channel image.

4. The image demosaicing method with a near-infrared channel according to claim 3, characterized in that: The method for calculating multi-directional first gradient information of the G channel includes: Calculate the first vertical gradient of the G channel , expressed as: Calculate the first horizontal gradient of the G channel , expressed as: in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel.

5. The image demosaicing method with a near-infrared channel according to claim 3, characterized in that: The method for calculating multi-directional second gradient information of the NIR channel comprises: If the current pixel is an R pixel or a B pixel, the second gradient in the vertical direction of the NIR channel is calculated , expressed as: / 2 Calculate the second gradient in the horizontal direction of the NIR channel , expressed as: / 2 If the current pixel is a NIR pixel, calculate the second gradient in the vertical direction of the NIR channel , expressed as: Calculate the second gradient in the horizontal direction of the NIR channel , expressed as: in, Indicates the coordinate position of the current pixel. Indicates the NIR channel value of the current pixel.

6. The image demosaicing method with a near infrared channel according to claim 3, characterized in that: The method for weightedly fusing the first gradient information and the second gradient information to obtain a second G channel image includes: Obtaining a comprehensive gradient according to the first gradient information and the second gradient information, wherein the comprehensive gradient includes a vertical comprehensive gradient and a horizontal comprehensive gradient; According to the vertical integrated gradient and the horizontal integrated gradient, the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction are determined; Calculate the vertical interpolation of the current pixel and horizontal interpolation , expressed as: in, Indicates the coordinate position of the current pixel. Indicates the G channel value of the current pixel; According to the interpolation weight in the vertical direction and the interpolation weight in the horizontal direction, a weighted sum calculation is performed on the vertical interpolation and the horizontal interpolation of the current pixel to obtain a second G channel image.

7. The image demosaicing method with a near infrared channel according to claim 1, characterized in that: The method for obtaining a full-resolution NIR image by interpolation calculation based on the NIR channel values ​​of the RGB image and the original image includes: Calculate the multi-directional gradient information, brightness information and texture information of RGB images; Calculate the multi-directional gradient information and texture information of the original NIR image; According to the brightness information and texture information of the RGB image, the multi-directional gradient information of the RGB image and the multi-directional gradient information of the original NIR image are weighted and fused to obtain the final gradient information; According to the final gradient information, confirm the interpolation weight of the NIR channel; According to the interpolation weight of the NIR channel, the original NIR channel value is interpolated to obtain a full-resolution NIR image.

8. The image demosaicing method with a near infrared channel according to claim 1, characterized in that: The image demosaicing method with a near infrared channel also includes: Performing texture judgment and optimization on the second G channel image to obtain a third G channel image; The second BG color difference image and the second RG color difference image are reconstructed with the third G channel image respectively to obtain a second B channel image and a second R channel image.

9. The image demosaicing method with a near infrared channel according to claim 8, characterized in that: The method of performing texture judgment and optimization on the second G channel image to obtain a third G channel image includes: In a preset window of the second G channel image, the difference between the texture values ​​of adjacent pixels and the texture values ​​of every other pixel is calculated to obtain a texture difference; If the texture difference is greater than the texture threshold, the pixels in the current preset window are filtered; The second G channel image is traversed to obtain a third G channel image.

10. An image processing system, used to implement the image demosaicing method with a near-infrared channel as claimed in any one of claims 1 to 9, characterized in that: The image processing system comprises: An image acquisition module, used for acquiring original images; A demosaic processing module is used to perform a demosaic processing using the G channel value, the B channel value, the R channel value and the NIR channel value of the original image to obtain a full-resolution RGB image and a NIR image; Image output module, used to output full-resolution RGB images and NIR images.

11. An electronic device, characterized in that: The method comprises a memory, a processor and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image de-mosaicing method with a near-infrared channel as described in any one of claims 1 to 9 is executed.

Images