Multi-sensor fused full-color low-light night vision device based on visible light up-conversion

Abstract

The invention relates to a multi-sensor fused full-color low-light night vision device based on visible light up-conversion. The full-color low-light night vision device comprises an objective lens assembly, an infrared spectroscope, a visible light up-converter, a main sensor assembly, a controller, a gray level sensor assembly, an image processing module and a display device. The objective lens assembly is used for collecting ambient light and transmitting the ambient light to the infrared spectroscope, the infrared spectroscope divides the light into two paths, one path is visible light, the other path is near-infrared light, the visible light is transmitted to the main sensor assembly, the main sensor assembly outputs a first full-color image, the near-infrared light is transmitted to the visible light up-converter, and the visible light up-converter outputs a second full-color image. The visible light up-converter is used for converting the near-infrared light into a second full-color image, the gray level sensor assembly outputs a gray level image, and the first full-color image, the second full-color image and the gray level image are fused through the image processing module and then output to the display device.

Description

Technical Field

[0001] This invention relates to the field of night vision imaging technology, specifically to a full-color low-light night vision device based on visible light upconversion and multi-sensor fusion. Background Technology

[0002] Full-color low-light night vision devices are a new type of night vision equipment that integrates low-light enhancement and multispectral image fusion technologies. They overcome the limitations of traditional low-light night vision devices with monochrome displays and thermal imagers with low contrast, and are widely used in military operations, security monitoring, and individual soldier reconnaissance. Existing technology 1 (CN208273080U) discloses a low-light night vision device based on dual-optical-path imaging. External light, after passing through objective lens 1, is split into visible light and near-infrared light by beam splitter prism 9. The visible light is processed by first imaging chip 5 to form a color image signal, while the near-infrared light is processed by second imaging chip 6 to form a black-and-white image signal. These signals are then synthesized by image processor 7 to obtain a color image. However, in extremely low-light environments, especially in environments with 0.0001 lux (near complete darkness), the visible light signal is very weak and insufficient to provide enough color information, leading to color distortion or loss of detail in the final image. Prior art 2 (CN111458890B) proposes that after light passes through infrared beam splitter 101, visible light passes through blue band beam splitter 301 and green band beam splitter 302 respectively, and after being enhanced, is combined with infrared light to form a true-color image. This prior art performs beam splitting and enhancement processing on extremely weak visible light. However, it still focuses on the processing of visible light, and the effect is not ideal in extremely low light environments.

[0003] Visible upconversion is an optical process that converts low-energy photons into high-energy visible light photons. The core of this process is to achieve the conversion from "low-energy photon input to high-energy photon output" through the nonlinear optical effects and quantum control of materials or devices. This breaks through the wavelength and intensity limitations of traditional optical detection. Using this technology, infrared light can be converted into visible light (e.g., patent CN103165727A), which provides a new approach for the improvement of full-color low-light night vision devices. Summary of the Invention

[0004] The purpose of this invention is to provide a full-color low-light night vision device based on visible light upconversion and multi-sensor fusion. By integrating a visible light upconversion module and a multi-sensor fusion system, it achieves high-fidelity full-color imaging in extremely low-light environments.

[0005] To address this, the present invention provides a full-color low-light night vision device based on visible light up-conversion multi-sensor fusion, comprising an objective lens assembly, an infrared beam splitter, a visible light up-converter, a main sensor assembly, a controller, a grayscale sensor assembly, an image processing module, and a display device. The objective lens assembly collects ambient light and transmits it to the infrared beam splitter, which splits the light into two paths: one for visible light and one for near-infrared light. The visible light is transmitted to the main sensor assembly, which outputs a first full-color image. The near-infrared light is transmitted to the visible light up-converter, which converts the near-infrared light into a second full-color image. The grayscale sensor assembly outputs a grayscale image. The first full-color image, the second full-color image, and the grayscale image are fused by the image processing module and then output to the display device.

[0006] Furthermore, the visible light up-converter includes a sensor array comprising multiple sensor units representing different pixels. Each sensor unit includes an R-conversion layer, a G-conversion layer, a B-conversion layer, and a pixel combining unit. The R-conversion layer, G-conversion layer, and B-conversion layer respectively include wavelength conversion materials, optical fibers, and optical sensors, thereby converting near-infrared light into red, green, and blue light signals, respectively. The pixel combining unit amplifies and performs analog-to-digital conversion on the aforementioned red, green, and blue light signals to obtain three RGB brightness values. The brightness values ​​are then fused to obtain the color pixels corresponding to the sensor unit (32), and finally the second full-color image is obtained.

[0007] Furthermore, the controller transmits the pixel color information of the first full-color image to the pixel synthesis unit, and the pixel synthesis unit obtains the RGB values ​​of the pixels at the corresponding positions of the sensor units in the pixel color information of the first full-color image: and regarding the above Make corrections: ,Will , , The color pixels corresponding to the sensor units are fused together to obtain the second full-color image.

[0008] Furthermore, the grayscale sensor assembly is located on one side of the objective lens assembly to ensure that it can acquire grayscale signals from the same field of view and obtain a grayscale image.

[0009] Furthermore, the image processing module includes an image fusion unit, an image optimization unit, and an image output unit; the image fusion unit performs the following steps: S1: Select the first full-color image as the reference image, perform grayscale processing on the second full-color image and the reference image to obtain the second grayscale image and the first grayscale image respectively, and perform normalization preprocessing on the first grayscale image, the second grayscale image and the grayscale image output by the grayscale sensor component to obtain the first grayscale normalized image, the second grayscale normalized image and the third grayscale normalized image. S2: Feature points are extracted from the first, second, and third gray-level normalized images using the ORB algorithm. A valid feature point matching set is obtained through FLANN matching and Ratio Test filtering. The spatial transformation matrix of the second full-color image and the gray-level image output by the gray-level sensor component relative to the reference image is solved based on the RANSAC algorithm. Based on the spatial transformation matrix, spatial transformations are performed on the second full-color image and the gray-level image output by the gray-level sensor component to obtain the registered second full-color image and the registered gray-level image. S3: Perform pixel-level weighted fusion of the reference image and the registered second full-color image to obtain the full-color base image; S4: Perform pixel-level weighted fusion of the full-color base image and the registered grayscale image to obtain a full-color fused image.

[0010] Furthermore, in step S3 above, the weight of the reference image is... The weights of the registered second full-color image are: The pixel values ​​of the RGB channels of a full-color base image are obtained using the following calculation method: , , , These are the RGB channel pixel values ​​of the reference image, , , These are the RGB channel pixel values ​​of the second full-color image; in step S4 above, the weight of the full-color base image is... The weights of the registered grayscale images are 1- ,0< <1.

[0011] Furthermore, it also includes a light sensor or a location sensor, as described in step S3 above: ,in L Given the current ambient light intensity, The preset maximum light intensity, This is the weighting adjustment coefficient. Basic weights; The current ambient light intensity is obtained from the light sensor, or from the latitude and longitude coordinates of the location obtained from the geographic location sensor and combined with the current time.

[0012] Furthermore, the image optimization unit includes a color correction submodule and a CLAHE detail enhancement submodule, used to perform color correction and detail enhancement on the full-color fused image.

[0013] In this invention, a visible light up-converter is first introduced into the low-light night vision device. After being split by an infrared beam splitter, the near-infrared light is converted into visible light, making full use of the information of the near-infrared light to obtain a second full-color image. Combined with the first full-color image of visible light, the color information of the final image is enriched. Compared with the prior art that only uses visible light to obtain color information, this avoids the defects of color distortion and loss of detail in low-light environments, especially in extremely low-light environments. Secondly, the information of the first full-color image can be used to correct the generation of the second full-color image, giving full play to the realistic color characteristics of visible light, so that the second full-color image is closer to the real color. Thirdly, the image fusion unit adjusts the proportion of the first and second full-color images in the final synthesized image according to the ambient light intensity, which has strong adaptability to changes in ambient light, so that the colors of the images obtained in different low-light environments are basically consistent. Finally, the grayscale image information is fused to improve the target recognition and scene perception capabilities in complex environments, provide accurate edge information for the image enhancement algorithm of the night vision device, and improve the stereoscopic effect of the imaging. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of a full-color low-light night vision device; Figure 2 This is a schematic diagram of the visible light up-converter structure of the present invention; Figure 3 This is a schematic diagram of the sensor unit structure. Detailed Implementation

[0015] See Figure 1 The full-color low-light night vision device of the present invention includes an objective lens assembly 1, an infrared beam splitter 2, a visible light up-converter 3, a main sensor assembly 4, a controller 5, a grayscale sensor assembly 6, an image processing module 7, and a display device 8.

[0016] The objective lens assembly 1 includes an objective lens with a large aperture of F1.0 and is equipped with a multi-layer infrared anti-reflective coating and a diamond protective coating, which can efficiently collect weak visible light, starlight, moonlight and near-infrared light in the environment.

[0017] The light passing through the objective lens assembly is split into two paths after passing through the infrared beam splitter: one is visible light and the other is near-infrared light. The visible light is output to the main sensor assembly 4, and after processing, a first full-color image is obtained. The near-infrared light is output to the visible light up-converter 3, which is used to convert the near-infrared light into a visible light image signal, and after processing, a second full-color image is obtained.

[0018] The grayscale sensor assembly 6 is located on one side of the objective lens assembly 1, ensuring that it can acquire grayscale signals from the same field of view. This is used to capture grayscale details of object textures (such as clothing wrinkles and architectural patterns) in low-light scenes, compensating for detail loss caused by the visible light up-converter in low light, and obtaining a grayscale image. The grayscale sensor assembly captures subtle grayscale gradients between the target and the background (such as the transition area between a person's outline and the night sky), providing accurate edge information for the night vision device's image enhancement algorithm and improving the stereoscopic effect of the image.

[0019] For visible light up-converters, wavelength conversion materials are used to convert near-infrared light into visible light. By using different materials, near-infrared light can be converted into red, green, and blue light (CN103165727A discloses that by changing the material selection of the OLED, the full spectrum of visible light output can be achieved as required). Other materials can also be used for wavelength conversion, such as NaYF4 as the matrix material, doped with Yb. 3+ / Er 3+ Red light can be obtained, and it is doped with Yb. 3+ / Ho 3+ Green light can be obtained by doping with Yb. 3+ / Tm 3+ Blue light can be obtained, and each material responds to infrared light and outputs a single primary color light.

[0020] To achieve the output of the second full-color image, see [link / reference]. Figure 2-3 The visible light up-converter 3 of the present invention includes a sensor array 31, which includes a plurality of sensor units 32, each sensor unit 32 representing a pixel (e.g., 256 pixels). 256 sensor units indicate that it can output 256 Each sensor unit 32 (with a 256-pixel image) includes: an R-conversion layer 321, a G-conversion layer 322, a B-conversion layer 323, and a pixel synthesis unit 324.

[0021] The R conversion layer 321 includes an R wavelength conversion material 3211, an optical fiber 3212, and an R photoelectric sensor 3213, thereby converting near-infrared light into red photoelectric signals. Similarly, the G conversion layer 322 and the B conversion layer 323 convert near-infrared light into green photoelectric signals and blue photoelectric signals, respectively.

[0022] Pixel combining unit 324 is used to amplify and perform analog-to-digital conversion on the aforementioned red, green, and blue light signals to obtain three RGB brightness values. The RGB brightness values ​​are fused to obtain colored pixels. Finally, the visible light up-converter 3 converts near-infrared light into a second full-color image.

[0023] As one aspect of the present invention, since the second full-color image is obtained based on near-infrared light, its colors have a certain deviation from the true colors. Therefore, after the pixel synthesis unit 324 obtains the three RGB brightness values, the present invention uses the first full-color image to correct them. Specifically: Based on the location of sensor unit 32, the RGB value of the corresponding pixel is found in the first full-color image (preferably, the resolution of the first full-color image is consistent with the number of rows and columns of the sensor array): Regarding the above Make corrections: ,Will , , The images are then fused to obtain color pixels. This is equivalent to introducing visible light information into the near-infrared light processing, making the second full-color image obtained from near-infrared light processing closer to the actual colors.

[0024] The image processing module is described in detail below: It includes an image fusion unit, an image optimization unit, and an image output unit.

[0025] The image fusion unit is used to fuse the first full-color image, the second full-color image, and the grayscale image. Its processing includes the following steps: S1: Select the first full-color image as the reference image, perform grayscale processing on the second full-color image and the reference image to obtain the second grayscale image and the first grayscale image respectively, and perform normalization preprocessing on the first grayscale image, the second grayscale image and the grayscale image output by the grayscale sensor component 6 to obtain the first grayscale normalized image, the second grayscale normalized image and the third grayscale normalized image. S2: Feature points are extracted from the first, second, and third gray-level normalized images using the ORB algorithm. A valid feature point matching set is obtained through FLANN matching and Ratio Test filtering. The spatial transformation matrix of the second full-color image and the gray-level image output by the gray-level sensor component 6 relative to the reference image is solved based on the RANSAC algorithm. Spatial transformations are then performed on the second full-color image and the gray-level image output by the gray-level sensor component 6 based on the transformation matrix to obtain the registered second full-color image and the registered gray-level image. S3: Perform pixel-level weighted fusion of the reference image and the registered second full-color image to obtain the full-color base image; S4: Perform pixel-level weighted fusion of the full-color base image and the registered grayscale image to obtain a full-color fused image.

[0026] In step S3 above, the weight of the reference image (first full-color image) is: The weights of the registered second full-color image are: The pixel values ​​of the RGB channels of a full-color base image are obtained using the following calculation method: ,in, , , These are the RGB channel pixel values ​​of the full-color base image. , , These are the RGB channel pixel values ​​of the reference image, , , These are the RGB channel pixel values ​​of the second full-color image, respectively. In step S4 above, the weights of the full-color base image are... The weights of the registered grayscale images are 1- ,0< <1.

[0027] By adjusting and It can adjust the visible light portion, infrared light portion, and grayscale image portion to form complementarity, thereby creating the final full-color fused image. The image's color is primarily determined by the first and second full-color images, while grayscale image data compensates for detail loss in extremely low-light conditions.

[0028] By combining the first full-color image with visible light, the color information of the final image is enriched. Compared with the existing technology that only uses visible light to obtain color information, it avoids the defects of color distortion and loss of detail in low-light environment, especially in extremely low-light environment. Secondly, the information of the first full-color image can be used to correct the generation of the second full-color image, giving full play to the realistic color characteristics of visible light, so that the second full-color image is closer to the real color.

[0029] As another aspect of the present invention, considering that ambient lighting conditions vary in actual use, and that the visible light portion can provide more accurate color information under good ambient lighting conditions, the weights in step S3 above are adjusted accordingly. The following optimization scheme is proposed: Add a light sensor or a geolocation sensor to the night vision device. The light sensor can directly obtain the ambient light intensity, while the geolocation sensor obtains the current geographical coordinates of the environment. Combined with local time information, the ambient light intensity can also be calculated.

[0030] Weights of the reference image ,in L Given the current ambient light intensity, The preset maximum light intensity, This is the weighting adjustment coefficient. The base weights are used; the weights of the second full-color image after registration are... ; Therefore, the weights of the reference image can be automatically adjusted according to environmental conditions, thereby automatically obtaining a high-quality full-color fused image. Preferably... =1000 lux =0.5, =0.4.

[0031] The proportions of the first and second full-color images in the final composite image are adjusted according to the ambient light intensity, which makes the image highly adaptable to changes in ambient light and ensures that the colors of the images obtained under different low-light conditions are basically consistent.

[0032] The image optimization unit includes a color correction submodule and a CLAHE detail enhancement submodule. The color correction submodule, based on a 3D Color Enhancement Engine, establishes a color lookup table (LUT). This LUT is trained using a massive amount of low-light scene samples and contains color calibration parameters under different light intensities. Simultaneously, it employs an AI semantic partitioning algorithm to perform semantic segmentation on the fused image, identifying different regions such as sky, vegetation, human figures, and buildings. Personalized calibration is then applied to the color characteristics of each region; for example, green gain is increased in vegetation areas, and skin saturation is adjusted in human figure areas to ensure color fidelity ≥90%.

[0033] The CLAHE detail enhancement submodule employs the Adaptive Histogram Equalization (CLAHE) algorithm, which divides the grayscale range of the image into multiple sub-regions and performs histogram equalization on each sub-region to enhance the edge details and contrast of the image. At the same time, a contrast limit parameter is set (value range: 2-4) to avoid overexposure. In addition, an edge detection algorithm (Canny algorithm) is used to extract the edge information of the image, and edge enhancement filtering is used to strengthen the edge regions, thereby improving the detail clarity of the image.

[0034] The image output unit is used to transmit the processed image signal to the display device for display.

[0035] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A full-color low-light night vision device based on visible light upconversion and multi-sensor fusion, characterized in that, The system includes an objective lens assembly (1), an infrared beam splitter (2), a visible light up-converter (3), a main sensor assembly (4), a controller (5), a grayscale sensor assembly (6), an image processing module (7), and a display device (8). The objective lens assembly (1) is used to collect ambient light and transmit it to the infrared beam splitter (2). The infrared beam splitter (2) splits the light into two paths, one being visible light and the other being near-infrared light. The visible light is transmitted to the main sensor assembly (4), which outputs a first full-color image. The near-infrared light is transmitted to the visible light up-converter (3), which converts the near-infrared light into a second full-color image. The grayscale sensor assembly (6) outputs a grayscale image. The first full-color image, the second full-color image, and the grayscale image are fused by the image processing module (7) and then output to the display device (8).

2. The full-color low-light night vision device according to claim 1, characterized in that, The visible light up-converter (3) includes a sensor array (31), which includes multiple sensor units (32) representing different pixels. Each sensor unit (32) includes an R-conversion layer (321), a G-conversion layer (322), a B-conversion layer (323), and a pixel synthesis unit (324). The R-conversion layer (321), G-conversion layer (322), and B-conversion layer (323) respectively include wavelength conversion materials, optical fibers, and optical sensors, thereby converting near-infrared light into red, green, and blue light signals, respectively. The pixel synthesis unit (324) is used to amplify and perform analog-to-digital conversion on the red, green, and blue light signals to obtain three RGB brightness values. The brightness values ​​are then fused to obtain the color pixels corresponding to the sensor unit (32), and finally the second full-color image is obtained.

3. The full-color low-light night vision device according to claim 2, characterized in that, The controller (5) transmits the pixel color information of the first full-color image to the pixel synthesis unit (324), and the pixel synthesis unit (324) obtains the RGB values ​​of the pixels at the corresponding positions of the sensor unit (32) in the pixel color information of the first full-color image. and regarding the above Make corrections: ,Will , , The color pixels corresponding to the sensor unit (32) are obtained by fusion, and finally the second full-color image is obtained.

4. The full-color low-light night vision device according to claim 2 or 3, characterized in that, The grayscale sensor assembly is located on one side of the objective lens assembly to ensure that it can acquire grayscale signals from the same field of view and obtain a grayscale image.

5. The full-color low-light night vision device according to claim 4, characterized in that, The image processing module includes an image fusion unit, an image optimization unit, and an image output unit; the image fusion unit performs the following steps: S1: Select the first full-color image as the reference image, perform grayscale processing on the second full-color image and the reference image to obtain the second grayscale image and the first grayscale image respectively, and perform normalization preprocessing on the first grayscale image, the second grayscale image and the grayscale image output by the grayscale sensor component to obtain the first grayscale normalized image, the second grayscale normalized image and the third grayscale normalized image. S2: Feature points are extracted from the first, second, and third gray-level normalized images using the ORB algorithm. A valid feature point matching set is obtained through FLANN matching and Ratio Test filtering. The spatial transformation matrix of the second full-color image and the gray-level image output by the gray-level sensor component relative to the reference image is solved based on the RANSAC algorithm. Based on the spatial transformation matrix, spatial transformations are performed on the second full-color image and the gray-level image output by the gray-level sensor component to obtain the registered second full-color image and the registered gray-level image. S3: Perform pixel-level weighted fusion of the reference image and the registered second full-color image to obtain the full-color base image; S4: Perform pixel-level weighted fusion of the full-color base image and the registered grayscale image to obtain a full-color fused image.

6. The full-color low-light night vision device according to claim 5, characterized in that, In step S3 above, the weight of the reference image is: The weights of the registered second full-color image are: The pixel values ​​of the RGB channels of a full-color base image are obtained using the following calculation method: , , , These are the RGB channel pixel values ​​of the reference image, , , These are the RGB channel pixel values ​​of the second full-color image, respectively; In step S4 above, the weight of the full-color base image is: The weights of the registered grayscale images are 1- ,0< <1.

7. The full-color low-light night vision device according to claim 6, characterized in that, It also includes a light sensor or a location sensor, in step S3 above: ,in L Given the current ambient light intensity, The preset maximum light intensity, This is the weighting adjustment coefficient. Basic weights; The current ambient light intensity is obtained from the light sensor, or from the latitude and longitude coordinates of the location obtained from the geographic location sensor and combined with the current time.

8. The full-color low-light night vision device according to claim 5, characterized in that, The image optimization unit includes a color correction submodule and a CLAHE detail enhancement submodule, which are used to perform color correction and detail enhancement on the full-color fused image.

Images