A single-lens visible-near infrared imaging system based on multi-channel color filters and algorithms

By using multi-channel color filters and efficient imaging processing algorithms in the multi-spectral imaging system, the color deviation problem caused by near-infrared imaging is solved, and high-quality visible and near-infrared imaging is achieved, which simplifies the device structure and reduces the cost.

CN118828167BActive Publication Date: 2024-11-29UNIV OF SCI & TECH BEIJING
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411296779.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-29
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

When the existing multi-spectral imaging system acquires near-infrared imaging of 700-800nm, it will cause color deviations in the red, green and blue channels, affecting the quality of visible light imaging. The equipment is complex, high cost, slow imaging speed and difficult data alignment. .

Method used

Using a single-lens visible light-near-infrared light imaging system based on multi-channel color filters and algorithms, the high level of visible light and near-infrared light is achieved through the four-channel color filter module, pre-filter, image sensor and imaging processing module. Quality imaging. The system uses a full-band spectrometer to obtain ambient spectral information, combining the separation matrix and the bias matrix for signal separation and correction.

Benefits of technology

It realizes high-quality visible and near-infrared imaging by a single shot, avoids color deviation, simplifies the equipment structure, reduces costs, improves imaging speed and maintenance convenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118828167B_ABST
    Figure CN118828167B_ABST
Patent Text Reader

Abstract

The present invention discloses a single-lens visible-near-infrared imaging system based on a multi-channel color filter and an algorithm, belonging to the field of image capture and processing. The system comprises: a four-channel color filter module, a front filter, an image sensor and an imaging processing module; wherein the four-channel color filter module is used to change the color filter of a traditional camera to obtain a four-channel color filter; the front filter is installed in front of the camera, and the front filter is used to filter ultraviolet rays; the image sensor is used to obtain raw imaging data taken by the camera; the imaging processing module is used to perform imaging processing based on a matrix algorithm to obtain an RGB image and a near-infrared image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image capturing and processing, and in particular relates to a single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms. Background Art

[0002] Multispectral imaging technology has a wide range of applications in remote sensing, agriculture, environmental monitoring, medical diagnosis and other fields. Traditional multispectral imaging systems usually rely on multiple cameras or scanning imaging devices to obtain images in different spectral ranges through multiple shots or line-by-line scanning. Although these methods can provide rich spectral information, they have problems such as complex equipment, high cost, slow imaging speed and difficulty in data alignment.

[0003] The multispectral four-channel camera is an emerging imaging technology that can acquire multiple independent channels of visible light and near-infrared light in a single shot through a single lens. The RGBN cameras currently on the market have an additional NIR channel compared to traditional RGB cameras, which can provide clearer and more detailed images under more diverse environmental conditions. However, since the response difference of 700-800nm ​​near-infrared light in the red, green, and blue channels of the sensor has not been properly processed, these RGBN cameras usually use a notch filter design to filter out the near-infrared light in the 700-800nm ​​band. The near-infrared light in the 700-800nm ​​band is the highest intensity near-infrared light in sunlight. The environmental characteristics of many plants and organisms have specific spectral responses in this band. The lack of this band will affect the monitoring and analysis of related characteristics. In addition, the absorption and scattering of infrared bands above 800nm ​​by water mist in the air increases sharply, which greatly reduces the effect of near-infrared imaging in foggy weather and limits its application prospects.

[0004] In the existing technology, image sensors have the strongest near-infrared response in the 700-800nm ​​band. When collecting near-infrared images in this band, serious color cast problems will occur in the red, green, and blue channels, resulting in a decrease in the quality of visible light imaging. There is currently no mature solution to effectively solve this problem internationally. Some existing studies and attempts include:

[0005] Conventional Color Matrix Correction: Attempts to resolve the color deviation caused by the 700-800nm ​​band by adjusting the conventional color matrix (CCM), but the actual effect is limited, especially in complex natural environments.

[0006] Segmentation based on plant shape: Use image processing technology to segment the plant shape first, and then perform color correction. Although this method can improve the accuracy of color correction, the algorithm is complex and the processing time is long, which is not suitable for real-time applications.

[0007] Therefore, the present invention proposes a single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms, which is generally used for collecting near-infrared imaging above 700nm. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm to solve the problems existing in the above-mentioned prior art.

[0009] To achieve the above object, the present invention provides a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm, comprising:

[0010] Four-channel color filter module, front filter, image sensor and imaging processing module;

[0011] Wherein, the four-channel color filter module is used to change the color filter of a traditional camera to obtain a four-channel color filter;

[0012] The front filter is installed in front of the camera, and the front filter is used to filter ultraviolet rays;

[0013] The image sensor is used to obtain raw imaging data captured by the camera;

[0014] The imaging processing module is used to perform imaging processing based on a matrix algorithm to obtain an RGB image and a near-infrared image.

[0015] Optionally, the front filter includes but is not limited to a bandpass filter of a specific wavelength band, a trap filter, a multi-layer dielectric film coating, a dynamic filter combination and a filter combination.

[0016] Optionally, the image sensor includes: a silicon-based CMOS sensor and an InGaAs sensor.

[0017] Optionally, the single-lens visible light-near infrared light imaging system further comprises a full-band spectrometer, and the full-band spectrometer is used to detect ambient light;

[0018] The full-band spectrometer includes a spectrum information acquisition unit, an image adjustment unit and a matrix calculation unit;

[0019] The spectral information acquisition unit is used to measure the spectral distribution of ambient light in real time using an external spectrometer to obtain spectral data;

[0020] The image adjustment unit is used to adjust the white balance of the RGB channels in the image;

[0021] The matrix calculation unit is used to select a separation matrix and a bias matrix under current lighting conditions after identification based on the spectral data.

[0022] Optionally, the imaging processing module includes a RAW data processing unit, a visible light-near infrared light imaging unit and a color conversion and correction unit;

[0023] The RAW data processing unit is used to sequentially perform black level correction, demosaicing and interpolation processing on the raw imaging data obtained by the image sensor to obtain processed imaging data;

[0024] The visible light-near infrared light imaging unit is used to perform signal separation on the processed imaging data based on the separation matrix and the bias matrix to obtain a near infrared image and an initial visible light image;

[0025] The color conversion and correction unit is used for performing color space conversion on the initial visible light image to obtain a visible light image.

[0026] Optionally, the expression for performing signal separation on the processed imaging data based on the separation matrix and the bias matrix is:

[0027]

[0028] Wherein, Output is the RGB value obtained after separation correction and the corresponding near-infrared channel n value, A represents the separation matrix, C represents the bias matrix, and Input represents the input vector.

[0029] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement various modules of a single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms.

[0030] The present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, each module of a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm is implemented.

[0031] Compared with the prior art, the present invention has the following advantages and technical effects:

[0032] The present invention is based on the RGBN color filter design. By optimizing the filter and algorithm, it is possible to obtain high-quality imaging of visible light (400-700nm) and near-infrared light (above 700nm) in a single shot, providing an important innovation for the advancement of multispectral imaging technology.

[0033] The advantages of multispectral four-channel cameras include: ① Multispectral imaging: By capturing light in multiple spectral bands, multispectral four-channel cameras can obtain more dimensional information and enhance image analysis and processing capabilities. ② Enhanced environmental perception: Near-infrared light can provide additional environmental details, which helps to improve image quality and recognition accuracy under low light, complex weather and other conditions. ③ Wide application: As an advanced extension of three-channel RGB cameras, multispectral four-channel cameras have broad application potential in agriculture, remote sensing, environmental monitoring, intelligent driving and medical fields. In the agricultural field, multispectral images can be used to analyze the health of crops; in remote sensing applications, they can be used for surface cover classification and environmental monitoring; in intelligent driving, multispectral four-channel cameras can improve the perception of vehicles under various lighting conditions, provide more spectral information, thereby improving the performance of the perception system under complex lighting conditions, enhancing environmental perception, assisting vehicles in making smarter decisions and path planning, and improving the safety and stability of the system; in the medical field, multispectral four-channel cameras can be used to enhance lesion detection and surgical navigation, by capturing images of different spectral bands to reveal the reflection and absorption characteristics of tissues under different spectra, improving the accuracy of lesion detection, and providing real-time tissue imaging during surgery, helping doctors to more accurately identify and distinguish different tissue types and lesion areas, improving the success rate of surgery and the speed of patient recovery. ④ Cost-effectiveness: The single-lens design reduces the complexity and cost of the equipment, and has significant cost-effectiveness and maintenance convenience compared with multiple-shot or scanning imaging equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 This is a flow chart of a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of distinguishing light source types according to an embodiment of the present invention;

[0037] Figure 3 A and C matrix calculation flow chart of an embodiment of the present invention;

[0038] Figure 4 is a typical sunlight spectrum of an embodiment of the present invention;

[0039] Figure 5 is the spectrum of the fluorescent lamp in the embodiment of the present invention;

[0040] Figure 6 Spectrum of an incandescent lamp according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Embodiment 1

[0044] In international cutting-edge research, adding near-infrared light in the 700-800nm ​​band usually faces serious color deviation problems, because traditional RGBN camera filter design and image processing algorithms are difficult to effectively process the spectral information of this band. However, the single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms of the present invention successfully overcomes the color cast problem in 700-800nm ​​band imaging through precise separation algorithms. The system can not only obtain high-brightness color, shape, contour and texture information under natural light and artificial light, but also adapt to the needs of diverse application scenarios, such as low light, severe weather conditions, etc.

[0045] The existing technology has the following technical defects: ① Multiple shooting problems: Traditional multispectral cameras often require multiple shooting to obtain images in different bands, which not only increases the imaging time, but also may cause image misalignment due to the shooting interval, affecting the accuracy and consistency of the data. ② Equipment complexity: Scanning multispectral cameras require complex mechanical and optical systems to scan and acquire images line by line, which increases the cost and maintenance difficulty of the system, and also has high requirements for the operating environment. ③ Synchronization problem: In a multi-camera system, it is a challenge to achieve synchronous acquisition of images of each channel, especially in dynamic scenes, where image misalignment and time inconsistency are prone to occur. ④ High cost: Existing multispectral imaging systems are usually expensive and not suitable for large-scale promotion and application, especially in application fields with limited resources. ⑤ Band limitation: The band coverage of existing multispectral cameras is limited and cannot cover all bands of visible light and near-infrared light at the same time. In particular, cameras based on RGBN color filters lack coverage in the 700-800nm ​​band, which limits their effectiveness in certain specific applications.

[0046] In contrast, the present invention adopts a new filter design that filters out ultraviolet rays, and uses a new visible light-near infrared light signal separation algorithm in combination with environmental spectrum information. This algorithm is not only applicable to traditional RGBN color filters, but can also be seamlessly applied to other color filters such as CMYG or RGBW image systems. Through the new filter design and high-precision light signal separation algorithm, a wider band of near-infrared light capture and precise imaging are achieved.

[0047] The present invention aims to obtain high-quality images of visible light and near-infrared light through a single-shot single-lens shooting to meet the needs of color, shape, contour and texture information in various application scenarios, especially under low illumination and adverse weather conditions, bringing new application prospects and technological advances to the field of machine vision.

[0048] like Figure 1 As shown, in this embodiment, a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm is provided, the system comprising: a four-channel color filter module, a front filter, an image sensor, a full-band spectrometer and an imaging processing module; wherein the four-channel color filter module is used to change the color filter of a traditional camera to obtain a four-channel color filter; the front filter is installed in front of the camera, and the front filter is used to filter ultraviolet rays; the image sensor is used to obtain the original imaging data taken by the camera; the full-band spectrometer is used to detect ambient light; the imaging processing module is used to perform imaging processing based on a matrix algorithm to obtain an RGB image and a near-infrared image.

[0049] Furthermore, through optimized design and advanced algorithms, high-quality images of visible light (400-700nm) and near-infrared light (above 700nm) can be captured in a single shot. The system is based on four-channel color filters with different responses, such as RGBN (red, green, blue, near-infrared), CMYG (cyan, magenta, yellow, green), RGBW (red, green, blue, white light), and is optimized for various application scenarios, including the following steps:

[0050] Four-channel color filter design: By changing the color filter (RGGB) of the traditional camera, four-channel color filters such as RGBN, RGBW, CMYG, etc. are designed. These filters can be customized according to specific application requirements to optimize the response of visible light and near-infrared light. The present invention adopts an innovative four-channel color filter system that can capture visible light above 400nm and near-infrared light at the same time. This design covers the traditional RGB channels (red, green, blue) and a near-infrared (NIR) channel, or other combinations such as RGBW, CMYG, etc., which are configured according to application requirements.

[0051] Pre-filter: Install a UV filter in front of the camera to ensure that only light above 400nm passes through the filter, thereby eliminating the interference of UV on imaging and improving imaging quality. Add a filter to filter out UV in the imaging system to ensure that the camera only collects light intensity above 400nm, thereby improving the accuracy and reliability of imaging.

[0052] Full-band spectrometer: An optional full-band spectrometer is used to detect ambient light. The spectrometer outputs the white balance information, separation matrix, and bias matrix required for color separation, ensuring that the system can achieve accurate color separation under various lighting conditions.

[0053] Imaging processing: The four-channel intensity information obtained by the camera hardware is calculated by applying the corresponding separation matrix and bias matrix under different lighting environments. The specific steps are as follows: The collected four-channel intensity information is multiplied by the separation matrix to separate the spectral information of each channel. The color deviation is corrected by the bias matrix, and finally a high-precision RGB image and near-infrared image are obtained.

[0054] Fast image processing based on matrix algorithm: ① Efficient image processing: This technology uses matrix algorithm to process four-channel intensity information. Compared with the image processing method that relies on neural network, it significantly reduces the demand for hardware resources, allowing the system to complete image processing in a shorter time. ② Real-time application: Due to the use of efficient matrix operations, this system can be quickly applied to scenes that require real-time processing such as video surveillance, providing instant visible light and near-infrared spectrum images. ③ Separation matrix and bias matrix: The intensity value of each channel at the spatial point position is obtained based on RAW graphics processing technology, and then the four-channel intensity information is converted into high-precision RGB and NIR signals using separation matrix and bias matrix. The separation matrix and bias matrix have a corresponding relationship with the white balance scene, and the corresponding matrix is ​​automatically selected according to different lighting conditions (such as natural light, incandescent lamp, halogen lamp, fluorescent lamp, etc.). ④ Final image output: Through color space conversion and gamma correction, the system finally outputs high-quality images and near-infrared images that meet the sRGB color gamut standard.

[0055] The specific implementation of this embodiment includes:

[0056] Color filter selection and hardware configuration: This system uses a four-channel color filter, usually using the RGBN configuration, where the RGB channel: red, green, and blue color filters are used to capture visible light (400-700nm) signals; the N channel: a near-infrared color filter that captures near-infrared light signals above 700nm; and an ultraviolet filter: filters out ultraviolet light (below 400nm) to ensure that the system only collects light intensity above 400nm.

[0057] Traditional multispectral cameras usually perform imaging in the near-infrared band above 800nm, while this design can capture near-infrared light in the 700-800nm ​​band, which is the highest intensity near-infrared light region in sunlight, and can significantly enhance the monitoring and analysis capabilities of plants and biological features.

[0058] Furthermore, without the spectrometer to assist in photometry, the RAW data processing flow is as follows:

[0059] 1. Black level correction: Adjust the black level value of each pixel to ensure that the image data accurately reflects the brightness in the scene.

[0060] 2. White balance adjustment: Automatically measure and adjust the white balance of the image's RGB channels based on preset white balance scenes (such as natural light, incandescent light, fluorescent light, etc.).

[0061] 3. Separation matrix and bias matrix calculation: Based on the selected white balance scene, the pre-calculated separation matrix and bias matrix are applied. These matrices are used to convert the four-channel intensity information into accurate RGB and NIR signal intensities.

[0062] 4. De-mosaicing and interpolation: Convert the monochrome pixel values ​​of the Bayer filter array into complete RGB and N channel intensity values, and fill the gaps between pixels through linear interpolation.

[0063] 5. Application of separation matrix and bias matrix: Multiply the processed four-channel intensity information by the corresponding separation matrix and add the preset bias matrix to obtain accurate RGB and NIR signal intensities.

[0064] 6. Color space conversion: Convert the processed RGB signal to sRGB color space for subsequent display and processing.

[0065] 7. Gamma correction: Apply the gamma curve to adjust the brightness and contrast of the image, and finally output high-quality multispectral images.

[0066] Furthermore, the full-band spectrometer includes a spectral information acquisition unit, an image adjustment unit and a matrix calculation unit; wherein the spectral information acquisition unit is used to use an external spectrometer to measure the spectral distribution of ambient light in real time to obtain spectral data, and by analyzing the spectral data and identifying the type of light source (such as sunlight, fluorescent lamp, incandescent lamp, etc.), the system can dynamically select the separation matrix and bias matrix corresponding to the light source, so as to accurately correct and optimize the image; the image adjustment unit is used to adjust the white balance of the RGB channels in the image; the matrix calculation unit is used to calculate the separation matrix and bias matrix under the current lighting conditions based on the spectral data.

[0067] Furthermore, the imaging processing module includes a RAW data processing unit, a visible light-near infrared light imaging unit and a color conversion and correction unit; wherein the RAW data processing unit is used to sequentially perform black level correction, demosaicing and interpolation processing on the raw imaging data obtained by the image sensor to obtain processed imaging data; the visible light-near infrared light imaging unit is used to perform signal separation on the processed imaging data based on the separation matrix and the bias matrix to obtain a near infrared image and an initial visible light image; the color conversion and correction unit is used to perform color space conversion on the initial visible light image to obtain a visible light image. In the case of spectrometer-assisted photometry, the spectral photometry process includes:

[0068] 1. Spectral information acquisition: Use an external spectrometer to measure the spectral distribution of ambient light in real time and obtain detailed wavelength and intensity data.

[0069] 2. White balance adjustment: Based on the spectral data provided by the spectrometer, accurately measure and adjust the white balance of the image's RGB channels to ensure color accuracy and consistency.

[0070] 3. Separation matrix and bias matrix calculation: Based on the spectral measurement data, accurately calculate the separation matrix and bias matrix under the current lighting conditions.

[0071] 4.RAW data processing flow: including black level correction, demosaicing and interpolation steps.

[0072] 5. Application of separation matrix and bias matrix: According to the data of spectral measurement, the accurately calculated separation matrix and bias matrix are applied to obtain accurate RGB and NIR signal intensity.

[0073] 6. Color space conversion: Convert the processed RGB signal to sRGB color space.

[0074] 7. Gamma correction: Apply the gamma curve to the image to adjust the brightness and contrast of the final output image.

[0075] Furthermore, as a specific implementation of this embodiment, in the present invention, the white balance processing of the RGBN image is implemented by a global white balance algorithm that integrates N channels. This method not only considers the white balance of the traditional RGB channels, but also combines the information of the N channel (i.e., the near-infrared channel), making the overall white balance processing more comprehensive and accurate. Figure 2 shown.

[0076] Global white balance adjustment: For the acquired RGBN image RAW data, black level correction is performed first, and then global white balance adjustment is performed. The white balance adjustment takes into account the traditional RGB channel white balance, and integrates the information of the N channel (near infrared channel) to make the white balance processing more reasonable. Since infrared light affects the color temperature, the traditional white balance method may not accurately reflect the actual lighting environment. By combining the intensity information of the N channel, the large fluctuations in the gain coefficient caused by infrared light can be effectively corrected to obtain more reasonable white balance parameters. For different lighting environments, the white balance gain coefficient is calculated as follows:

[0077] (1) Red channel gain R gain :

[0078]

[0079] in, G avg is the average intensity value of the green channel, N avg is the average intensity value of the near-infrared channel, R avg is the average intensity value of the red channel, and k is a coefficient related to the sensor. In a specific embodiment, k is 0.5.

[0080] (2) Green channel gain G gain , the green channel gain coefficient is usually set to 1 because it is usually used as the reference channel.

[0081] (3) Blue channel gain B gain :

[0082]

[0083] in, B avg is the average intensity value of the blue channel, and m is another coefficient related to the sensor. In a specific embodiment, the values ​​of k and m are 0.5 and 0.3 respectively.

[0084] (4) Infrared channel gain N gain :

[0085]

[0086] The gain of the near-infrared channel is equal to the gain of the red channel to ensure consistency in image processing when infrared light is strong.

[0087] Classification of illumination without the aid of a spectrometer:

[0088] (1) After global white balance adjustment, calculate the intensity ratio of N channels N ratio , to assist in lighting classification. N ratio The calculation method is:

[0089]

[0090] (2) By analyzing the white balance gain coefficient ( R gain , G gain , B gain )as well as N ratio , can more accurately identify lighting conditions and thus classify lighting. This classification method can better cope with different light sources, especially in high infrared environments, and improve the accuracy of lighting classification.

[0091] Light classification with the aid of a micro-spectrometer:

[0092] The spectrometer obtains the spectral characteristics of the light source by decomposing the ambient light into a spectrum of individual wavelengths and measuring the light intensity of each band. These characteristics include the relative intensity distribution of visible light and near-infrared light. Based on these spectral characteristics, the spectrometer is able to classify the light into different types, such as natural light, artificial light, or light with high infrared content. Through pre-set classification criteria, the spectrometer automatically classifies the light collected in real time.

[0093] By classifying the illumination, the system can apply the corresponding A and C matrices to different illumination types to obtain highly accurate infrared signal separation and visible light color correction, thus achieving high-quality imaging effects.

[0094] Furthermore, if Figure 3 As shown, in the present invention, the A matrix and the C matrix are calculated by the sample data collected under the illumination of the selected light source. The specific process is as follows:

[0095] (1) Sample preparation: A standard 140 color sample card (based on the X-Rite standard) and plant samples of various colors (such as white, brown, red, orange, yellow, green, blue, cyan, purple, etc.) were selected as data collection objects.

[0096] (2) Light source selection and filtering: Use the selected D light source to illuminate the sample, and mount a filter in the 400-700nm band to collect red, green and blue three-channel images. Use a filter that only passes wavelengths above 400nm (to remove ultraviolet interference) to collect four-channel images.

[0097] (3) Image acquisition and preprocessing: The acquired images are preprocessed to remove black level and white balance. The RGB and RGBN intensity values ​​of each sample point are extracted, n=N is taken, and these data are integrated into the database.

[0098] (4) Matrix calculation: Through the linear regression method, based on the collected RGBN data, the A matrix and C matrix under the D light source are calculated. When the light source does not contain 700-800nm ​​infrared light, the C matrix is ​​zero, and only the A matrix is ​​needed to complete the separation of infrared signals and the correction of RGB colors.

[0099] Assuming that the RAW information collected by the four-channel sensor is R, G, B, and N after black level and white balance correction, there is a separation matrix A corresponding to the lighting condition:

[0100]

[0101] The corresponding bias matrix C is:

[0102]

[0103]

[0104]

[0105]

[0106] The output Output[r; g; b; n] is the RGB value obtained after separation correction and the corresponding near-infrared channel n value.

[0107]

[0108] If the sensor is arranged as CMYG, then the A and C matrices corresponding to the corresponding illumination of the sensor are substituted into the above formula to obtain:

[0109]

[0110] Through the above implementation, the system can efficiently process RAW data, accurately extract multispectral information, and generate clear and accurate multispectral images according to different lighting conditions. It is suitable for a wide range of application scenarios, including video surveillance and other fields that require fast response and high-quality imaging.

[0111] The present invention adopts a four-channel color filter design for continuous spectrum acquisition, which uses an innovative four-channel color filter system that can simultaneously capture visible light and near-infrared light above 400nm. This design covers traditional RGB channels (red, green, blue) and a near-infrared (NIR) channel, or other combinations such as RGBW, CMYG, etc., which are configured according to application requirements.

[0112] Traditional multispectral cameras usually perform imaging in the near-infrared band above 800nm, while this design can capture near-infrared light in the 700-800nm ​​band, which is the near-infrared light region with the highest intensity in sunlight, and can significantly improve the monitoring and analysis capabilities of plant and biological characteristics. At the same time, the present invention adds a filter to filter out ultraviolet rays in the imaging system to ensure that the camera only collects light intensity above 400nm, thereby improving the accuracy and reliability of imaging.

[0113] The present invention adopts a fast image processing technology based on a matrix algorithm. This technology uses a matrix algorithm to process four-channel intensity information. Compared with the image processing method that relies on a neural network, it significantly reduces the demand for hardware resources, allowing the system to complete image processing in a shorter time. Due to the use of efficient matrix operations, this system can be quickly applied to scenes that require real-time processing, such as video surveillance, to provide instant visible light and near-infrared spectral images.

[0114] The present invention obtains the intensity value of each channel at the spatial point position based on RAW graphics processing technology, and then uses the separation matrix and bias matrix to convert the four-channel intensity information into high-precision RGB and NIR signals. The separation matrix and bias matrix have a corresponding relationship with the white balance scene, and the corresponding matrix is ​​automatically selected according to different lighting conditions (such as natural light, incandescent lamp, halogen lamp, fluorescent lamp, etc.).

[0115] Through color space conversion and gamma correction, the system of the present invention finally outputs high-quality images and near-infrared images that meet the sRGB color gamut standard.

[0116] Furthermore, this embodiment also includes a filter design and optimization scheme: in addition to using an ultraviolet filter to filter out light below 400nm, the multi-spectral camera of the present invention solves the color cast problem of 700-800nm ​​near-infrared light on imaging through an innovative algorithm. On this basis, a variety of different filter combinations can also be designed to further optimize the system's response and imaging quality in specific bands. For example:

[0117] 1. Bandpass filter for a specific band: A bandpass filter of 400-800nm ​​can be designed to ensure that light within this range can be effectively captured, thereby improving the signal strength and signal-to-noise ratio of the system in this band.

[0118] 2. Trap filter design: Adding a 650-700nm trap filter highlights the bandpass of 400-650nm and above 700nm. This design can significantly enhance the imaging capability of the system in these specific bands, providing clearer images and more accurate spectral data.

[0119] 3. Multilayer dielectric film coating technology: Using multilayer dielectric film coating technology, a spectral filter with high selectivity and high transmittance can be achieved. This filter can not only provide excellent light transmission performance in a specific band, but also effectively reduce noise and improve imaging quality.

[0120] 4. Dynamic filter combination: According to the needs of different application scenarios, different filters can be flexibly replaced or combined to adapt to various complex lighting environments and imaging requirements. The design and combination of these filters will be included in the patent protection scope of this invention to ensure the excellent performance of the system in a variety of application scenarios.

[0121] 5. Multiple filter combinations: In addition to the RGBN (red, green, blue, near infrared) four-channel filter combination, other combinations can also be used, such as RGG*B (red, green, light green, blue), RGBW (red, green, blue, white), CMYG (cyan, magenta, yellow, green), etc. These combinations can be designed according to the needs of different application scenarios to optimize the imaging effect. For example, RGBW filters can provide higher brightness in low light conditions, while CMYG filters can be used for more accurate color reproduction.

[0122] Furthermore, this embodiment also includes an image sensor optimization solution: although the present invention uses silicon-based CMOS technology (covering 400-1100nm), other sensor substrate technologies can also be substituted, such as InGaAs sensors (covering 400-1700nm) can achieve visible light near-infrared imaging, but the filter design and algorithm innovation of the present invention still need to be protected. As long as the multi-spectral imaging technology of visible light RGB color is realized through color microlenses, the visible light / infrared light separation method of the present invention can be used to obtain color-accurate RGB images.

[0123] Furthermore, this embodiment also includes algorithm innovation: the present invention successfully solves the serious color cast problem encountered by traditional multi-spectral cameras in 700-800nm ​​band imaging through unique algorithm design. Through precise color separation matrix and bias matrix operations, the present invention can realize fast processing of RGBN underlying intensity information without relying on complex hardware resources. This algorithm not only improves the system's operational efficiency, but can also be quickly applied to video surveillance and real-time imaging fields.

[0124] In addition to using separation matrix and bias matrix to convert from RGBN to RGB and NIR images, LUT table lookup technology can also be used to bypass the matrix algorithm. Different LUT tables are established under different lighting conditions to achieve mapping. Although this method does not seem to use the matrix algorithm of separation matrix or bias matrix, the LUT table itself can be made by separation matrix and bias matrix. Therefore, the use of LUT table to obtain separation results should also fall within the scope of protection of this patent.

[0125] As a specific implementation of this embodiment, this embodiment takes a 24-color block image as an example, and the RGB standard RGB values ​​of the color blocks are shown in Table 1.

[0126] Table 1

[0127]

[0128] When there is no spectrometer, the white balance calculation adopts the grayscale world white balance algorithm. However, due to the existence of the near-infrared channel, the weight of the near-infrared channel is considered for optimization. The average value of the four channels in the image after removing the black level is first calculated, as shown in Table 2.

[0129] Table 2

[0130]

[0131] Calculate the red channel gain R gain =( G avg +0.5* N avg ) / R avg =1.2

[0132] Green channel gain G gain =1

[0133] Calculate blue channel gain B gain =( G avg +0.5* N avg ) / ( B avg +0.3* N avg )=2

[0134] For near infrared channel selection N gain = R gain =1.2

[0135] calculate N ratio = N avg / ( R avg + G avg + B avg +N avg )=0.25

[0136] The scene light collected by the spectrometer is as follows Figure 4 As shown, Figure 4 This is a typical sunlight spectrum.

[0137] After initially determining that the light source is sunlight through white balance calculation or spectrometer, separate bias matrices A and C corresponding to the ambient light are selected:

[0138] a 11 = 1.00000000e+00; a 12 = 0.00000000e+00; a 13 = -1.11022302e-16; a 14 = -1.00000000e+00; a 21 = 3.72809628e-03; a 22 = 9.99684066e-01; a 23 = -2.10391532e-04; a 24 = -5.47014778e-01; a 31 = 1.34482260e-02; a 32 = -1.74677256e-03; a 33 =1.00012521e+00; a 34 = -5.76923692e-01; a 41 = 0; a 42 = 0; a 43 = 0; a 44 = 1;

[0139] c1= 2.93312041e-11; c2= -1.60817654e+01; c3= -5.22393206e+01; c4= 0;

[0140] For the RGBN arrangement, the effect of visible light on the infrared channel is ignored in the matrix.

[0141] The initial values ​​of the four channels collected in the 24-color block experiment are shown in Table 3:

[0142] Table 3

[0143]

[0144] The base black level is 128.

[0145] The calculation of black level correction and white balance correction is: R'=(R-128)*R gain ; G'=(G-128)*G gain ; B'=(B-128)*B gain ; N'=(N-128)*N gain

[0146] The black level and white balance after correction are shown in Table 4:

[0147] Table 4

[0148]

[0149] After separation bias matrix processing, the process formula is: output = A * input + C

[0150] The results are shown in Table 5:

[0151] Table 5

[0152]

[0153] The n' channel can be directly corrected for brightness and output as a near-infrared image.

[0154] And r'g'b' needs to be further converted to color space through the Cam2SRGB matrix, which mainly converts the color sensed by the sensor into the color of the standard SRGB space.

[0155] Cam2sRGB =[1.21904231, 0.1110202, -0.33006252; -0.24073172, 1.50120797, -0.26047625; 0.0931398, -0.4144, 1.41851559]

[0156] For example, r=1.21904231*r'+0.1110202*g'-0.33006252*b'

[0157] g=-0.24073172*r'+1.50120797*g'-0.26047625*b'

[0158] b=0.0931398*r'-0.4144*g'+1.41851559*b'

[0159] After color space conversion, the results are shown in Table 6:

[0160] Table 6

[0161]

[0162] For brightness adjustment, each channel is multiplied by a gain factor of 1.6668 as shown in Table 7:

[0163] Table 7

[0164]

[0165] Each channel is divided by 3967, and negative values ​​are zeroed, so that each data is normalized to be within the range of 0-1, as shown in Table 8:

[0166] Table 8

[0167]

[0168] Perform gamma transformation according to gamma=2.2, that is, output=input^(1 / 2.2), as shown in Table 9:

[0169] Table 9

[0170]

[0171] Finally, the intensity of each channel is multiplied by 255 to obtain an 8-bit color value, as shown in Table 10:

[0172] Table 10

[0173]

[0174] This embodiment obtains the corresponding data by using filters that pass 400-700nm and above 400nm under the same light source. After removing the black level of the metadata and adjusting the white balance, the RGBN and corresponding RGBN images are obtained. By collecting four-channel brightness data of a large number of sample points on the standard color card and green plants, and summarizing these data and saving them in a list. In the Python environment, the separation and bias matrix of RGBN to RGBN is calculated by linear regression. The coefficients and intercepts in the final result correspond to matrix A and matrix C, respectively.

[0175] This embodiment also discloses a spectrometer, which can identify the type of light source through spectral data, so as to select matrix A and matrix C under the corresponding light source. Figure 5 and Figure 6The spectra of fluorescent lamps and incandescent lamps are shown respectively. Compared with the algorithm that identifies the light source based on the white balance calculation of the whole image, the spectrometer is more accurate in identifying the light source.

[0176] As a specific implementation of this embodiment, the source code for calculating the matrix A and the matrix C is included:

[0177] import pandas as pd

[0178] from sklearn.model_selection import train_test_split

[0179] from sklearn.linear_model import LinearRegression

[0180] from sklearn.metrics import mean_squared_error, mean_absolute_error,r2_score

[0181] import pickle

[0182] from scipy.io import savemat

[0183] # Read the Excel file containing data

[0184] data = pd.read_excel('rgbnRGBN.xlsx')

[0185] # Extract features and target variables

[0186] X = data[['R', 'G', 'B', 'N']]

[0187] y = data[['r', 'g', 'b','n']]

[0188] # Split the data into training and testing sets

[0189] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

[0190] # Create a linear regression model

[0191] model = LinearRegression()

[0192] # Train the model

[0193] model.fit(X_train, y_train)

[0194] # Get the separation matrix and bias term

[0195] coefficients = model.coef_

[0196] intercept = model.intercept_

[0197] # Output separation matrix and bias term

[0198] print("Coefficients (Weights):", coefficients)

[0199] print("Intercept (Bias):", intercept)

[0200] # Save model parameters to file

[0201] with open('model_parameters.pkl', 'wb') as f: # 'wb' means writing binary data

[0202] pickle.dump({'coefficients': coefficients, 'intercept': intercept},f)

[0203] # Save model parameters to a .mat file

[0204] mat_data = {'coefficients': coefficients, 'intercept': intercept}

[0205] savemat('model_parameters.mat', mat_data)

[0206] # Predict the values ​​of the test set

[0207] y_pred = model.predict(X_test)

[0208] # Calculate performance evaluation indicators

[0209] mse = mean_squared_error(y_test, y_pred)

[0210] rmse = (mse ** 0.5)

[0211] mae = mean_absolute_error(y_test, y_pred)

[0212] r2 = r2_score(y_test, y_pred)

[0213] print("Mean Squared Error (MSE):", mse)

[0214] print("Root Mean Squared Error (RMSE):", rmse)

[0215] print("Mean Absolute Error (MAE):", mae)

[0216] print("R-squared (R²):", r2)

[0217] # Save model parameters and performance evaluation indicators to files

[0218]

[0219] The multi-spectral four-channel imaging system of the present invention achieves the following remarkable effects through a series of technical solutions:

[0220] Efficient multispectral imaging: Through the innovative four-channel color filter design (such as RGBN, RGBW, CMYG, etc.), images of visible light (400-700nm) and near-infrared light (above 700nm) are simultaneously acquired in a single shot, significantly improving the efficiency and quality of multispectral imaging. Traditional multispectral imaging systems rely on multiple cameras or scanning devices to acquire images through multiple shots or line-by-line scanning, while the present invention adopts a single-lens design, which simplifies the device structure, reduces imaging time and image misalignment problems. The imaging algorithm is fast and efficient, and directly separates and corrects the intensity information of the underlying imaging layer to obtain the intensity information corresponding to each channel.

[0221] Extended spectral response: This invention covers near-infrared light in the 700-800nm ​​band, filling the common response gap in existing RGBN cameras. The 700-800nm ​​band has the highest intensity in sunlight and has important spectral characteristic responses in agriculture and biological monitoring, which improves the application effect of the system in these fields.

[0222] Avoid color deviation: Since traditional RGBN cameras will have serious color cast problems when adding near-infrared light in the 700-800nm ​​band, the present invention effectively avoids color deviation through a new color filter design and separation algorithm, ensuring the acquisition of high-precision RGB images and NIR images.

[0223] Adapt to various lighting conditions: The system is equipped with a front UV filter (passing above 400nm) and a full-band spectrometer to detect ambient light, output the separation matrix and bias matrix required for color separation, so that the camera can obtain high-brightness color, shape, contour and texture information under both natural and artificial light. It is particularly suitable for low-light environments, bad weather and other conditions, enhancing the system's adaptability in a variety of application scenarios.

[0224] The present invention has a wide range of application potentials: the multispectral four-channel imaging system of the present invention can design four-channel color filters according to different application requirements. If near-infrared intensity is required, RGBN color filters can be selected. If a high signal-to-noise ratio of visible light signal is required, RGBW, CMYG, etc. can be selected. It is widely used in agriculture, remote sensing, environmental monitoring, medical diagnosis and other fields. For example, in agriculture, multispectral images are used to analyze crop health; in remote sensing applications, they are used for land cover classification and environmental monitoring, which improves the accuracy and practicality of data analysis.

[0225] The present invention discloses a fast and efficient algorithm: the present invention uses a matrix algorithm to directly perform a matrix algorithm on the underlying intensity information of the four channels. Compared with the system relying on the neural network algorithm, it does not require additional hardware resources and has fast and efficient calculation. It can be directly applied to fields that require real-time processing, such as video surveillance, and can provide clear and accurate multispectral images under various lighting conditions.

[0226] The present invention has the advantages of cost-effectiveness and system simplification: compared with the existing multi-spectral imaging system, the single-lens design of the present invention reduces the equipment complexity and cost, avoids the system defects of multiple shots or line-by-line scanning, improves the imaging speed and maintenance convenience, and has significant cost-effectiveness.

[0227] In summary, the multispectral four-channel imaging system of the present invention has achieved remarkable results in improving imaging efficiency, expanding spectral response, avoiding color deviation, adapting to diverse lighting conditions, wide application, fast and efficient algorithms and cost-effectiveness through innovative technical solutions, providing strong support for the development and application of multispectral imaging technology.

[0228] Embodiment 2

[0229] An embodiment of the present invention also provides an electronic device, including: a memory and a processor; the memory is used to store programs; the processor is used to execute programs to implement various modules of a single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms.

[0230] Embodiment 3

[0231] An embodiment of the present invention further provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, various modules of a single-lens visible light-near infrared light imaging system based on a multi-channel color filter and an algorithm are implemented.

[0232] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms, characterized in that: include: Four-channel color filter module, front filter, image sensor and imaging processing module; Wherein, the four-channel color filter module is used to change the color filter of a traditional camera to obtain a four-channel color filter; The front filter is installed in front of the camera, and the front filter is used to filter ultraviolet rays; The image sensor is used to obtain raw imaging data captured by the camera; The imaging processing module is used to perform imaging processing based on a matrix algorithm to obtain an RGB image and a near-infrared image; The single-lens visible light-near infrared light imaging system further comprises a full-band spectrometer, and the full-band spectrometer is used to detect ambient light; The full-band spectrometer includes a spectrum information acquisition unit, an image adjustment unit and a matrix calculation unit; The spectral information acquisition unit is used to measure the spectral distribution of ambient light in real time using an external spectrometer to obtain spectral data; The image adjustment unit is used to adjust the white balance of the RGB channels in the image; The matrix calculation unit is used to select a separation matrix and a bias matrix under current illumination conditions after identification based on the spectral data; The imaging processing module includes a RAW data processing unit, a visible light-near infrared light imaging unit and a color conversion and correction unit; The RAW data processing unit is used to sequentially perform black level correction, demosaicing and interpolation processing on the raw imaging data obtained by the image sensor to obtain processed imaging data; The visible light-near infrared light imaging unit is used to perform signal separation on the processed imaging data based on the separation matrix and the bias matrix to obtain a near infrared image and an initial visible light image; The color conversion and correction unit is used to perform color space conversion on the initial visible light image to obtain a visible light image; The expression for performing signal separation on the processed imaging data based on the separation matrix and the bias matrix is: Wherein, Output is the RGB value obtained after separation correction and the corresponding near-infrared channel n value, A represents the separation matrix, C represents the bias matrix, and Input represents the input vector.

2. The single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms according to claim 1, characterized in that: The pre-filter includes but is not limited to a 400-800 nm bandpass filter, a trap filter, a multi-layer dielectric film coating, a dynamic filter combination and a filter combination.

3. The single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms according to claim 1, characterized in that: The image sensor includes: a silicon-based CMOS sensor and an InGaAs sensor.

4. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each module of the single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms as described in any one of claims 1 to 3.

5. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each module of the single-lens visible light-near infrared light imaging system based on multi-channel color filters and algorithms as described in any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Image processing method and related equipment

    CN115701128A

  • Color filter array, image acquisition method, image sensor and acquisition device

    CN118483856A