A low-light imaging method based on multi-spectral recognition

Abstract

The present application relates to the technical field of micro-light imaging, and discloses a micro-light imaging method based on multispectral recognition, and a micro-light imaging system based on multispectral recognition, which comprises a shell, an objective lens, a multispectral detector, and an image acquisition module; the objective lens and the shell are rotationally connected, the multispectral detector and the image acquisition module are electrically connected, and the multispectral detector, the image acquisition module, and the shell are fixedly connected; the multispectral detector comprises a detector body and a micro-nano structure film layer, the micro-nano structure film layer is arranged on one side of the detector body facing the objective lens, and the micro-nano structure film layer is used for modulating a projection spectrum. The micro-nano structure film layer is used for modulating the projection spectrum, so that the detector body can simultaneously receive spectral modulation data of multiple wave bands, the target can be found and recognized through spectral characteristics under weak light conditions, the problem that the target is difficult to find and recognize due to complex background and large image noise at night is effectively solved, the night observation effect is improved, and the night operation capability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of low-light imaging technology, and particularly relates to a low-light imaging method based on multi-spectral recognition. BACKGROUND

[0002] A low-light imaging system is an instrument that works in a night or weak light environment. The basic working principle is that the light radiated or reflected by a night scene target is imaged on a low-light detector target surface through an objective lens, and the low-light detector processes the received image through photoelectric conversion and processing circuit to output a visible target image. As a core device of the low-light imaging system, the low-light detector has the remarkable characteristics of day and night use, low cost, small power consumption, high sensitivity, and has been widely used in security, industry, medical treatment, military and other fields. However, although the existing low-light detector has high sensitivity and can realize night observation, it also has the following disadvantages: high night noise, difficulty in finding and identifying targets in a dark complex background, poor night observation effect, and poor night operation ability.

[0003] Therefore, it is necessary to provide a low-light imaging system capable of meeting the night target recognition requirement. SUMMARY

[0004] In view of this, the present application provides a low-light imaging method based on multi-spectral recognition.

[0005] Specifically, the technical scheme comprises the following:

[0006] In a first aspect, the present application provides a low-light imaging system based on multi-spectral recognition, comprising:

[0007] a shell, an objective lens, a multi-spectral detector, and an image acquisition module;

[0008] The objective lens and the shell are rotationally connected, the multi-spectral detector and the image acquisition module are electrically connected, and the multi-spectral detector, the image acquisition module and the shell are fixedly connected.

[0009] The multi-spectral detector comprises a detector body and a micro-nano structure film layer, the micro-nano structure film layer is arranged on a side of the detector body facing the objective lens, and the micro-nano structure film layer is used for modulating a projection spectrum.

[0010] Illustratively, the micro-nano structure film layer comprises a substrate and a metamaterial micro-nano array.

[0011] The metamaterial micro-nano array is located on a side of the detector body facing the objective lens, and the substrate is located on a side of the metamaterial micro-nano array facing the objective lens.

[0012] Illustratively, the metamaterial micro-nano array comprises a plurality of channel units.

[0013] Each of the channel units comprises a plurality of spectral channels;

[0014] Each of the spectral channels comprises a micro-nano structure array;

[0015] One of the spectral channels corresponds to one of the micro-nano structure arrays, and the micro-nano structure arrays in the plurality of spectral channels are different from each other.

[0016] Illustratively, the number of the spectral channels arranged in the micro-nano array of the metamaterial is a first number;

[0017] The number of the pixels arranged in the detector body is a second number;

[0018] The first number and the second number are the same;

[0019] The spectral channels and the pixels are arranged one by one.

[0020] Illustratively, the size of the spectral channels arranged in the micro-nano array of the metamaterial is a first size;

[0021] The size of the pixels arranged in the detector body is a second size;

[0022] The first size and the second size are the same.

[0023] Illustratively, the objective lens comprises a first lens sheet, a second lens sheet, a third lens sheet, a fourth lens sheet, a fifth lens sheet, and a sixth lens sheet, and the first lens sheet, the second lens sheet, the third lens sheet, the fourth lens sheet, the fifth lens sheet, and the sixth lens sheet are arranged in turn along the optical axis of the objective lens in the direction gradually approaching the multi-spectral detector;

[0024] The first lens sheet, the second lens sheet, the fourth lens sheet, the fifth lens sheet, and the sixth lens sheet are all meniscus lenses, the first lens sheet, the second lens sheet, the fourth lens sheet, and the fifth lens sheet are curved away from the multi-spectral detector, and the sixth lens sheet is curved toward the multi-spectral detector;

[0025] The third lens sheet is a double-concave lens.

[0026] In a second aspect, a micro-light imaging method based on multi-spectral recognition is provided, which adopts the multi-spectral imaging system based on multi-spectral recognition according to the first aspect;

[0027] The method comprises:

[0028] Obtaining spectral modulation data;

[0029] Preprocess the spectral modulation data to obtain preprocessed spectral modulation data;

[0030] Based on a pre-set multi-spectral calculation and color synthesis model, obtain multi-spectral color image data according to the preprocessed spectral modulation data, the multi-spectral color image data including material information of each pixel;

[0031] Based on a pre-set feature extraction and target recognition model, obtain a target region and position coordinates of the target region and generate a target frame according to the multi-spectral color image data;

[0032] Output the multi-spectral color image data and display the target frame in the multi-spectral color image data.

[0033] Illustratively, based on a pre-set multi-spectral calculation and color synthesis model, obtain multi-spectral color image data according to the preprocessed spectral modulation data, the multi-spectral color image data including material information of each pixel, including:

[0034] Obtain spectral modulation data of each pixel in the preprocessed spectral modulation data, and reconstruct the spectral modulation data of each pixel, the spectral modulation data of each pixel after reconstruction including multi-channel spectral modulation data;

[0035] Based on a pre-set multi-element multi-target optimization model, establish a mapping relationship between spectral modulation data and RGB according to the spectral modulation data of each pixel after reconstruction;

[0036] According to the mapping relationship between the spectral modulation data and RGB, perform color restoration on each pixel data to generate color image data;

[0037] According to a pre-set fusion model, fuse the spectral modulation data of each pixel after reconstruction and the color image data to obtain fused image data;

[0038] According to a pre-set calculation model, calculate the spectral modulation data of each pixel after reconstruction in the fused image data to obtain material information of each pixel and the multi-spectral color image data.

[0039] Illustratively, according to a pre-set calculation model, calculate the spectral modulation data of each pixel after reconstruction in the fused image data to obtain material information of each pixel and the multi-spectral color image data, including:

[0040] Based on a pre-set waveband partition rule, partition the spectral modulation data of each pixel after reconstruction by waveband to obtain spectral modulation data of multiple wavebands;

[0041] Spectral inversion is performed on the spectral modulation data of each wave band to obtain a spectral curve corresponding to the spectral modulation data of each wave band;

[0042] Based on the pre-set ground object standard spectral curve, the spectral curve corresponding to the spectral modulation data of each wave band is compared with the ground object standard spectral curve, and the material information corresponding to the spectral modulation data of each wave band is obtained according to the comparison result;

[0043] According to the material information corresponding to the spectral modulation data of each wave band, the material information of each pixel is obtained.

[0044] For example, based on the pre-set feature extraction and target recognition model, the target region and the position coordinates of the target region are obtained according to the multi-spectral color image data, and a target box is generated, including:

[0045] Based on the pre-set deep learning convolutional neural network, the shallow low-level features and deep high-level features in the spectral modulation data of each spectral channel are extracted according to the spectral modulation data of each pixel after reconstruction in the multi-spectral color image data;

[0046] Based on the pre-set dynamic weighting model, the shallow low-level features and the deep high-level features of different spectral channels are weighted and fused to obtain a fused feature map, wherein the weight of each spectral channel is dynamically adjusted according to the environmental characteristics;

[0047] According to the fused feature map and the material information of each pixel, the target region and the corresponding position information thereof are determined in the multi-spectral color image data, and a target box is generated.

[0048] The technical scheme provided by the present application has at least the following beneficial effects:

[0049] The multi-spectral detector used in the low-light imaging system in the present application includes a detector body and a micro-nano structure film layer, the micro-nano structure film layer is used for modulating the projected spectrum, so that the detector body can simultaneously receive spectral modulation data of multiple wave bands, and the target can be found and recognized through spectral characteristics in weak light conditions or atmospheric glow environment, effectively solving the problem that it is difficult to find and recognize the target due to complex target background and large night image noise, improving the night observation effect and enhancing the night operation ability. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 Structure diagram of a micro-light imaging system in an embodiment of the present application;

[0052] Figure 2 Structure diagram of an optical system in an objective lens in an embodiment of the present application;

[0053] Figure 3 Structure diagram of a multi-spectrum detector and an image acquisition module in an embodiment of the present application;

[0054] Figure 4 Structure diagram of a micro-nano structure film layer in an embodiment of the present application;

[0055] Figure 5 Structure diagram of a micro-nano structure film layer in an embodiment of the present application; Figure 4 Structure diagram of a micro-nano structure film layer in an embodiment of the present application;

[0056] Figure 6 Test result diagram of a micro-light imaging system in an embodiment of the present application, Figure 6 (a) is a test result diagram when a test distance is 10m, Figure 6 (b) is a test result when a test distance is 20m, Figure 6 (c) is a test result when a test distance is 30m, Figure 6 (d) is a test result when a test distance is 50m, Figure 6 (e) is a test result when a test distance is 80m, Figure 6 (f) is a test result when a test distance is 100m;

[0057] Figure 7 Flow diagram of a micro-light imaging method in an embodiment of the present application;

[0058] Figure 8 Interface diagram of an image acquisition module in an embodiment of the present application;

[0059] Figure 9 Flow diagram of a multi-spectrum calculation and color synthesis model in an embodiment of the present application;

[0060] Figure 10 Flow diagram of a feature extraction and target identification model in an embodiment of the present application.

[0061] The reference signs in the drawings respectively represent:

[0062] 1 - objective; 11 - first lens; 12 - second lens; 13 - third lens; 14 - fourth lens; 15 - fifth lens; 16 - sixth lens; 2 - multi-spectral detector; 200 - micro-nano structure film layer; 201 - substrate; 202 - micro-nano array of metamaterial; 203 - channel unit; 204 - spectral channel; 3 - image acquisition module; 4 - shell; 5 - limit screw.

[0063] The specific embodiments of the present application have been shown by the above drawings, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0065] Before the embodiments of the present application are described in further detail, the orientation terms such as “upper”, “lower”, “side” involved in the embodiments of the present application are taken as the reference of the orientation shown in the drawings, and do not have the meaning of limiting the scope of protection of the present application. Figure 1

[0066] In order to make the technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below in combination with the drawings.

[0067] As described above, the existing low-light detector has the problems of large night noise, difficulty in finding and identifying targets under complex dark background, poor night observation effect, and poor night operation ability. Therefore, the present application provides a low-light imaging system based on multi-spectral recognition.

[0068] As shown in Figures 1 to 5 A low-light imaging system based on multi-spectral recognition includes a shell 4, an objective 1, a multi-spectral detector 2, an image acquisition module 3, and a limit screw 5. As shown in Figure 1 ​As shown, the objective 1 and the shell 4 are rotationally connected, the multi-spectral detector 2 and the image acquisition module 3 are electrically connected, and the multi-spectral detector 2, the image acquisition module 3 and the shell 4 are fixedly connected; the multi-spectral detector 2 comprises a detector body and a micro-nano structure film layer 200, the micro-nano structure film layer 200 is arranged on a side of the detector body facing the objective 1, and the micro-nano structure film layer 200 is used for modulating a projection spectrum. The multi-spectral detector 2 used in the micro-light imaging system in the application comprises a detector body and a micro-nano structure film layer 200, the micro-nano structure film layer 200 is used for modulating a projection spectrum, so that the detector body can simultaneously receive spectrum modulation data of multiple wave bands, and the target can be found and identified through spectrum characteristics in a weak light condition or an atmospheric glow environment, the problem that the target is difficult to find and identify due to a complex target background and large night image noise is effectively solved, the night observation effect is improved, and the night operation ability is improved.

[0069] As shown in the figure, Figure 1 As shown, the objective 1 is rotationally connected with the shell 4 through a threaded structure outside a lens barrel, a limiting screw 5 is screwed into a mounting hole in the shell 4, and the multi-spectral detector 2 and the image acquisition module 3 are fixedly connected with the shell 4 through screws. After installation, a theoretical position of a rear section of the objective 1 is located on a focal plane of the multi-spectral detector 2, and an imaging distance of the micro-light imaging system at this position is infinity. The objective 1 can be screwed to change the position of the objective 1 relative to the multi-spectral detector 2, and when an outer contour of the lens barrel of the objective 1 touches the limiting screw 5, the imaging distance of the micro-light imaging system is 250 mm, and continuous clear imaging change in a range of 250 mm to infinity can be realized by screwing the objective 1.

[0070] As shown in the figure,

[0071] As shown in the figure,

[0072] As shown in the figure, Figure 2The objective lens 1 comprises a first lens 11, a second lens 12, a third lens 13, a fourth lens 14, a fifth lens 15, and a sixth lens 16. The first lens 11, the second lens 12, the third lens 13, the fourth lens 14, the fifth lens 15, and the sixth lens 16 are sequentially arranged along the optical axis of the objective lens 1 in the direction of approaching the multi-spectral detector 2. The first lens 11, the second lens 12, the fourth lens 14, the fifth lens 15, and the sixth lens 16 are all meniscus lenses. The central part of the first lens 11, the second lens 12, the fourth lens 14, and the fifth lens 15 is curved away from the multi-spectral detector 2, and the central part of the sixth lens 16 is curved towards the multi-spectral detector 2. The third lens 13 is a double-concave lens.

[0073] The first lens 11, the second lens 12, the third lens 13, the fourth lens 14, the fifth lens 15, and the sixth lens 16 constitute an optical system in the objective lens 1, which is used to transmit the light radiated or reflected by the night scene to the focal plane of the multi-spectral detector 2. After the multi-spectral detector 2 acquires the spectral modulation data, the spectral modulation data is sent to the image acquisition module 3 for signal conversion and image processing. The spectral modulation data is acquired by the micro-nano structure film layer 200.

[0074] The optical system in the objective lens 1 cooperatively realizes aberration correction and light focusing optimization through multiple lenses: the meniscus lenses correct field curvature, and the double-concave lens balances the optical power to correct spherical aberration, which together ensures that different light rays are accurately focused on the detector focal plane, thereby improving the imaging clarity. After the light is initially converged by the meniscus first lens 11 and the second lens 12 and adjusted by the double-concave third lens 13, it is further converged by the subsequent meniscus fourth lens 14, fifth lens 15, and sixth lens 16, and is efficiently focused on the focal plane of the multi-spectral detector 2, thereby providing high-quality images for signal conversion and processing and ensuring clear and high-quality imaging under low light conditions at night.

[0075] As shown in Figure 5 The micro-nano structure film layer 200 comprises a substrate 201 and a micro-nano array of metamaterials 202. Figure 1 、 Figure 3 、 Figure 5 The micro-nano array of metamaterials 202 is located on the side of the detector body facing the objective lens 1, and the substrate 201 is located on the side of the micro-nano array of metamaterials 202 facing the objective lens 1.

[0076] The substrate 201 is made of quartz glass.

[0077] For example, such as Figure 5 As shown, the metamaterial micro / nano array 202 includes multiple channel units 203; each channel unit 203 includes multiple spectral channels 204; each spectral channel 204 includes a micro / nano structure array; one spectral channel 204 corresponds to one type of micro / nano structure array, and the micro / nano structure arrays in the multiple spectral channels 204 are all different from each other.

[0078] For example, such as Figure 4 and Figure 5 As shown, the metamaterial micro / nano array 202 is composed of M×N channel units 203. Figure 4 Enlarged structural diagram of the structure within the local frame as shown below Figure 5 As shown, each channel unit 203 is divided into a nine-square grid, with each square representing a spectral channel 204. In this application, eight of the nine squares contain micro-nano structure arrays, meaning each channel unit 203 includes eight spectral channels 204. In this embodiment, each spectral channel 204 is modulated by the micro-nano structure array to allow only light of a preset spectral band to pass through. By changing the structure of the micro-nano structure array, the micro-nano structure arrays in the eight spectral channels 204 are made different from each other. Thus, each channel unit 203 is modulated by the eight spectral channels 204 to allow light of eight different preset spectral bands to pass through.

[0079] For example, such as Figure 4 and Figure 5 As shown, multiple channel units 203 on the metamaterial micro / nano array 202 are repeatedly arranged, and the placement direction of the multiple channel units 203 is consistent. Thus, through the arrangement of the metamaterial micro / nano array 202, the multispectral detector 2 has the capability of eight modulated transmission spectra. The transmission performance of the eight modulated spectra is different. To facilitate subsequent inversion calculations, and because all eight modulated spectra are broadband, high energy utilization can be achieved to meet the minimum requirements for use in low-light environments.

[0080] For example, in the metamaterial micro / nano array 202 described in this application, the number of spectral channels 204 is a first value; the number of pixels in the detector body is a second value; the first value and the second value are the same; the spectral channels 204 and the pixels are configured in a one-to-one correspondence.

[0081] For example, the size of the spectral channel 204 in the metamaterial micro / nano array 202 is set as a first size; the size of the pixel in the detector body is set as a second size; the first size and the second size are the same.

[0082] For example, the design of the metamaterial micro / nano array 202 is described below:

[0083] For example, the metamaterial micro / nano array 202 is the basis for achieving broadband, modulation and high energy utilization. It involves multi-faceted research and optimization of parameters such as materials used, appearance, array arrangement and structural size. The materials used include, but are not limited to, Au, Ag, Cu, TiO2, etc.

[0084] For example, the metal layer in the metamaterial micro / nano array 202 can be equivalent to an inductor in a circuit. L m and L e The SiO2 dielectric layer (substrate 201) is equivalent to a capacitor in a circuit. C m The surrounding environment medium is equivalent to the capacitance in the circuit. C e In an LC series circuit, when the circuit is in a resonant state, the resonant frequency can be determined using formula (1). f for:

[0085]

[0086] Based on the above plasmonic equivalent circuit principle, the metamaterial micro-nano array 202 was finally designed by modulating the resonant frequency of the micro-nano structure array.

[0087] For example, the fabrication method of the multispectral detector 2 is described below: and

[0088] The fabrication method of the multispectral detector 2 in this application includes: the packaging process uses a wafer-level flip-chip process to encapsulate the micro-nano structure film layer 200 on the photosensitive surface of the detector body; the method includes five steps: preparation, alignment, fixation, curing, and detection.

[0089] Step 1: Preparation. Place the micro / nano structure film 200 and the detector body in a dust-free environment to prevent impurities from existing on the photosensitive surfaces of the micro / nano structure film 200 and the detector body, which could affect the multispectral imaging effect.

[0090] Step 2: Alignment. Alignment refers to achieving pixel alignment under the guidance of marker bits, reducing crosstalk between adjacent pixels. Specifically, the micro / nano structure film layer 200 is flip-chipped, with the side of the micro / nano structure film layer 200 without the metamaterial micro / nano array 202 facing upwards and the side containing the metamaterial micro / nano array 202 facing downwards. Guided by the marker bits, the metamaterial micro / nano array 202 on the substrate 201 (quartz glass substrate) is ensured to be precisely aligned with the photosensitive surface pixels of the detector body, thereby reducing crosstalk between adjacent pixels and improving imaging quality.

[0091] Third step: fixing. Fixing refers to fixing the substrate 201 and the detector body by applying silver glue. Specifically, silver glue is applied to the contact surface between the edge of the substrate 201 (quartz glass substrate) and the detector body, and the two are fixed together by the conductivity and adhesion of the silver glue.

[0092] Fourth step: curing. Curing refers to waiting for the silver glue to cure, completing the packaging process. Specifically, the substrate 201 (quartz glass substrate) after applying silver glue and the detector body are placed in a curing device, and cured under preset temperature and time conditions, so that the silver glue is completely cured, and the stable connection between the substrate 201 (quartz glass substrate) and the detector body is ensured.

[0093] Fifth step: testing. Testing refers to performance testing of the packaged multispectral detector 2, which requires bad pixel testing and noise testing of the multispectral detector 2. The bad pixel testing is because the bad pixel is a white point in the output image in a full black environment, or a black point in the output image in a high light environment, so the multispectral detector 2 needs to be placed in a full black environment and a high light environment for imaging detection respectively. If there is no corresponding white point or black point in the test results of the two tests, the test evaluation is qualified. The noise test is to collect images of the multispectral detector 2 that has passed the bad pixel test in different brightness environments, and analyze the gray scale image gray scale value output by the multispectral detector 2 using the comparison method, to study the noise characteristics of the output spectral image.

[0094] Specific use test process:

[0095] Under low light conditions (environmental light intensity value is not greater than 10-3Lux), the tester is located in a green environment vegetation, and the multispectral detector 2 is erected at a distance of 10 meters from the tester, ensuring that the tester is within the field of view of the multispectral detector 2.

[0096] Further, the objective lens 1 collects and focuses the weak light radiated or reflected by the night scene through the meniscus lens.

[0097] Further, the multispectral detector 2 is debugged to check whether the gain, exposure and other parameters of the low-light imaging system are normal. By adjusting the gain and exposure, the recognition effect is observed and identified, and the data is repeatedly collected and saved to ensure that clear images can be obtained under extremely low light conditions.

[0098] Further, the collected spectral modulation data is then sent to the image acquisition module 3 for signal conversion and image processing.

[0099] Further, the test distance is changed to 10m, 20m, 30m, 50m, 80m, and 100m, and multispectral color image data is output, as shown in Figure 6 Figure 6 ​Fig. 2(a) is a schematic diagram of the test results when the test distance is 10 m, Figure 6 Fig. 2(b) is a schematic diagram of the test results when the test distance is 20 m, Figure 6 Fig. 2(c) is a schematic diagram of the test results when the test distance is 30 m, Figure 6 Fig. 2(d) is a schematic diagram of the test results when the test distance is 50 m, Figure 6 Fig. 2(e) is a schematic diagram of the test results when the test distance is 80 m, Figure 6 Fig. 2(f) is a schematic diagram of the test results when the test distance is 100 m.

[0100] The micro-light imaging system of the present application has significantly improved imaging quality under micro-light conditions. The multi-channel multi-band acquisition capability of the multi-spectral detector 2, combined with the efficient processing of the image acquisition module 3, enables the micro-light imaging system to provide more abundant and accurate spectral data. In addition, the micro-light imaging system realizes the spectral adjustment mechanism through the micro-nano structure film layer 200, so that high-quality multi-spectral color images can be obtained under different imaging distances, which has important application value in the fields of night surveillance, astronomical observation, and biomedical imaging, etc.

[0101] The working illuminance range, sensitivity, wide dynamic range, spectral resolution, channel unit 203, response time, environmental adaptability, size and weight of the micro-light imaging system of the present application all meet the requirements of micro-light night vision. The working illuminance range is from 0.001 Lux to 100000 Lux. This wide illuminance range enables the multi-spectral detector 2 to operate stably under indoor, outdoor and various lighting conditions, including but not limited to sunlight, fluorescent light, LED light, etc. The sensitivity is defined as the minimum light signal intensity that can be detected under a certain illuminance, and the specific value can reach 0.001 Lux, ensuring that high-quality images can also be obtained under weak light sources. The wide dynamic range reaches 72 dB, which can maintain the authenticity of image details and colors in high-contrast scenes. The spectral resolution is 30-100 nm, which can distinguish close spectral lines, thereby providing more accurate color information and more abundant details in multi-spectral imaging. The channel unit 203 contains eight independent spectral channels 204, each corresponding to a specific wavelength range. The multi-spectral detector 2 can capture a wide range of spectral information from ultraviolet to infrared, providing abundant data for analysis and processing. The response time is 40 ms, which enables the micro-light imaging system to quickly capture dynamic scenes and is suitable for applications that require fast imaging. Considering portability and integration, the weight of the micro-light imaging system is 0.5 Kg, making it easy to integrate into various devices.

[0102] For example, the present application also provides a micro-light imaging method based on multi-spectral recognition, which adopts a micro-light imaging system based on multi-spectral recognition as described above;

[0103] As shown in Figure 7 , the method comprises:

[0104] S10: Obtain spectral modulation data.

[0105] As shown in Figure 8 , each module in the data acquisition process forms an organic whole with the algorithm process through the hardware interface, realizing the transmission link based on the MIPI (Mobile Industry Processor Interface) interface.

[0106] Exemplarily, as shown in Figure 8 , the objective lens 1 converges the scene light to the focal plane of the multispectral detector 2, the spectral detector supports independent imaging of multiple spectral channels 204, and the output RAW data (original spectral modulation data) is transmitted to the image acquisition module 3 through the compatible MIPI-CSI interface. The interface circuit adapts to the voltage swing and data rate of the multispectral detector 2, ensuring synchronous acquisition of multiple spectral channel 204 signals. The high-speed ADC converts the analog signal into digital RAW data (spectral modulation data), which meets the output requirements of the multispectral detector 2 in terms of sampling rate and resolution, and integrates the power management function to reduce power consumption. The FPGA control unit generates timing and processes signals, generates timing signals required by the multispectral detector 2, realizes data buffering, format conversion and preliminary processing, and transmits structured data frames to subsequent units through the bus.

[0107] Exemplarily, the spectral modulation data refers to the spectral data obtained by modulating through multiple spectral channels 204. The spectral modulation is realized through the micro-nano structure array on each spectral channel 204.

[0108] Exemplarily, the spectral modulation data includes spectral data and image data. The spectral data refers to the spectral response information of the ground object at different wavebands. Different ground objects have different spectral data due to different absorption and reflection characteristics. The image data refers to the spatial information of the ground object. The spectral modulation data combines spectral data and image data, which can more comprehensively and accurately describe the characteristics and attributes of the ground object. Through the analysis of the spectral data, the types and components of the ground object can be identified; and the image data can provide the spatial distribution and morphological information of the ground object.

[0109] S20: Preprocess the spectral modulation data to obtain preprocessed spectral modulation data.

[0110] Exemplarily, the preprocessing includes denoising, color correction and white balance adjustment of the spectral modulation data. The preprocessing also includes data enhancement of the spectral modulation data. The spectral modulation data is normalized.

[0111] Exemplarily, as shown in Figure 8As shown, preprocessing is primarily led by the image processing unit (ISP, or Internet Service Provider), which performs denoising, color correction, and white balance adjustment on the spectral modulation data. The ISP employs a non-local mean denoising algorithm to suppress noise, estimates scene white balance based on the gray-world algorithm, and adjusts the RGB channel gain. Data augmentation includes random rotation, scaling, and color transformation of the image data in the spectral modulation data to simulate different imaging conditions; histogram equalization expands the gray-level dynamic range, and adaptive filters are used to dynamically remove noise, improving low-light image quality. After preprocessing, the spectral modulation data undergoes standardization to ensure uniform data distribution and provide stable input for subsequent processing.

[0112] S30: Based on a pre-set multispectral calculation and color synthesis model, multispectral color image data is obtained according to the pre-processed spectral modulation data. The multispectral color image data includes material information for each pixel.

[0113] For example, such as Figure 8 As shown, the preprocessed spectral modulation data is output to other modules via the multispectral data MIPI-CSI interface through the image output interface unit. Combined with the module control commands set by the user, the preprocessed spectral modulation data is subjected to multispectral calculation and color synthesis. The image input interface unit outputs the chip control commands set by the user through the RAW data acquisition MIPI-CSI interface to control the acquisition of RAW data (raw spectral modulation data) in the multispectral detector 2 of the image input interface unit.

[0114] For example, the following section introduces the multispectral processing and color synthesis of preprocessed spectral modulation data:

[0115] Step S30, based on a pre-set multispectral calculation and color synthesis model, obtains multispectral color image data according to the preprocessed spectral modulation data. The multispectral color image data includes material information for each pixel, including:

[0116] S31: Obtain the spectral modulation data of each pixel in the preprocessed spectral modulation data, and reconstruct the spectral modulation data of each pixel. The reconstructed spectral modulation data of each pixel includes multi-channel spectral modulation data.

[0117] For example, such as Figure 8As shown, step S31 is performed in the multispectral image solving unit for spectral recovery and image reconstruction. It can be understood that each pixel corresponds to a spectral channel 204, and one spectral channel 204 can modulate the spectral modulation data of one spectral band. In order to obtain the true spectral information of this spectral channel 204, the spectral modulation data of the remaining seven spectral bands is estimated, and the estimated spectral modulation data of the remaining seven spectral bands is fused with the original spectral modulation data of the spectral band to obtain the spectral modulation data of each pixel after reconstruction. At this time, the spectral modulation data of each pixel includes the spectral modulation data of eight spectral bands corresponding to eight channels.

[0118] For example, the marginal gradient extension method is used to reconstruct the spectral modulation data of each pixel. First, based on the binary tree generation method, an equal spatial probability ratio spectral channel 204 distribution scheme is designed in the repeatedly arranged array. Then, according to the preprocessed spectral modulation data, the gradient information of each pixel is calculated. On the basis of maintaining the image structure features and texture information, the pixel values and gradient values of the marginal pixels are used to reconstruct the spectral modulation data of the remaining seven spectral bands of the pixel, so as to obtain the high spatial resolution spectral modulation data of eight spectral bands corresponding to eight channels of each pixel.

[0119] S32: Based on the pre-set multi-element multi-target optimization model, the mapping relationship between the spectral modulation data and RGB is established according to the spectral modulation data of each pixel after reconstruction.

[0120] For example, the multi-element multi-target optimization model is a pre-trained model, and the training process is as shown in Figure 9 As shown, first, a multi-element multi-target regression model is established, and then a partial least squares regression algorithm is used to solve the multi-element multi-target regression model for color prediction to obtain the mapping relationship between the spectral modulation data and RGB. During training, the historical spectral modulation data is used for training. The 9 independent variables include the spectral modulation data in the eight spectral channels 204 and one ordinary channel in each channel unit 203, and the 3 dependent variables include the R value, the G value and the B value. During training, the latent variables are extracted by iteration to obtain the mapping relationship between the spectral modulation data and RGB with the maximum correlation between the spectral modulation data and the RGB value. At the same time, other models such as random forest or support vector machine (SVM) or convolutional neural network (CNN) are included in the multi-element multi-target optimization model to improve the accuracy and robustness of the mapping relationship and the color prediction. During training, the hyperparameters of the multi-element multi-target optimization model are optimized by grid search or Bayesian optimization to obtain the best performance.

[0121] For example, as shown in Figure 8 The trained multi-element multi-target optimization model is set in the multispectral image solving unit.

[0122] S33: Color restoration according to the mapping relationship between the spectral modulation data and RGB to generate color image data.

[0123] As shown in the example, Figure 9 the method includes obtaining color restoration coefficients K, B according to the mapping relationship between the spectral modulation data and RGB, and performing color restoration calculation according to the color restoration coefficients K, B and the spectral modulation data to generate color image data.

[0124] As an example, the color restoration stage combines color constancy algorithms, such as white point estimation, to improve the accuracy of color restoration. CIEDE2000 color mapping algorithm is used to evaluate and optimize color difference, ensuring the naturalness and consistency of color conversion.

[0125] As shown in the example, Figure 8 Step S33 is performed in the multispectral image calculation unit.

[0126] As shown in the example, Figure 8 and Figure 9 After step S33, the method further includes defining image quality indicators for image quality evaluation, such as PSNR and SSIM, to quantitatively evaluate the quality of the image corresponding to the color image data, and to realize an automatic quality control process, and to reprocess images that do not meet the standard.

[0127] S34: According to the pre-set fusion model, the spectral modulation data of each pixel after reconstruction and the color image data are fused to obtain the fused image data.

[0128] As an example, in the fusion process of spectral modulation data and color image data, multi-scale fusion is performed using multi-resolution analysis or wavelet transform to preserve more detailed information. Deep learning fusion is performed using deep neural network CNN to improve the fusion effect. CNN can learn the complex features of the data, extract image features through convolution layers, reduce feature dimensions through pooling layers, and perform classification or regression through fully connected layers, thereby improving the fusion effect. At the same time, the method further includes obtaining user instructions for fine-tuning and parameter setting of color image based on user interaction interface. The method further includes a user feedback learning mechanism to continuously optimize the color restoration process according to user feedback, and to realize personalized image processing, such as Figure 9 the interaction and feedback model shown in the example. As shown in the example, Figure 9 The method further includes configuring algorithm resources through performance optimization and resource management model to improve processing speed, such as using GPU acceleration and parallel processing, and to realize resource management strategies, such as memory and CPU setting usage limit conditions, to ensure system stability.

[0129] S35: According to the pre-set solving model, the spectral modulation data of each pixel in the fused image data is solved to obtain the material information of each pixel and the multi-spectral color image data.

[0130] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification. Figure 9

[0131] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification.

[0132] S351: Based on the pre-set band partition rule, the spectral modulation data of each pixel after reconstruction is partitioned into multiple bands to obtain the spectral modulation data of each band.

[0133] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification.

[0134] S352: The spectral modulation data of each band is subjected to spectral inversion to obtain the spectral curve corresponding to the spectral modulation data of each band.

[0135] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification.

[0136] S353: Based on the pre-set ground object standard spectral curve, the spectral curve corresponding to the spectral modulation data of each band is compared with the ground object standard spectral curve, and the material information corresponding to the spectral modulation data of each band is obtained according to the comparison result.

[0137] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification.

[0138] As shown in the figure, in the multi-spectral imaging technology, the spectral bands are divided into multiple narrow-band bands by using the pre-set solving model, and the spectral distribution characteristics of the target are used for imaging and identification. Figure 8 ​As shown, step S353 is performed in the spectral calibration unit.

[0139] S354: Obtain the material information of each pixel according to the material information corresponding to the spectral modulation data of each waveband.

[0140] Exemplarily, since the spectral waveband corresponding to each pixel after reconstruction covers the full waveband 400nm~900nm of the micro-light imaging system, the spectral modulation data of each waveband can correspond to an image and the material information contained therein. The image data and material information obtained by each pixel at different spectral wavebands are combined to obtain the material information of each pixel and the multi-spectral color image data.

[0141] Exemplarily, after step S354, the method further comprises: performing gamma correction and white balance adjustment on the multi-spectral color image data after color restoration, assigning the recorded impact points as white, and removing noise by median filtering to obtain multi-spectral color image data output to the next model. Gamma correction adjusts the gamma value of the image to achieve nonlinear adjustment of image brightness and contrast, and improves the visual effect of the image. White balance adjustment ensures the accuracy of the color. The image is processed by applying a median filter to further remove noise and smooth the image. The median filter effectively removes noise while preserving edge details by replacing each pixel value with the median value in its neighborhood.

[0142] Exemplarily, as Figure 9 As shown, when obtaining the multi-spectral color image data output to the next model, the coordinates of the impact points affected by the overexposed points also need to be considered. The method for obtaining the coordinates of the impact points affected by the overexposed points is: obtaining the coordinates of the overexposed points from the spectral modulation data, and performing interpolation on the coordinates of the overexposed points to obtain the coordinates of the impact points affected by the overexposed points.

[0143] The image acquisition module 3 of the embodiment can realize high-precision color image synthesis and process a large amount of multi-spectral data through the above-mentioned process.

[0144] S40: Based on the pre-set feature extraction and target recognition model, obtain the target region and the position coordinates of the target region according to the multi-spectral color image data. Step S40 aims to improve the target recognition effect in micro-light conditions.

[0145] Exemplarily, the feature extraction and target recognition module is based on a deep learning architecture. The processing platform can optionally use a high-performance chip (such as RK3588, etc.). In terms of hardware, FPGA or GPU co-processing is supported. FPGA is suitable for real-time parallel processing of multispectral data, and GPU is suitable for efficient training and inference of deep learning models. In terms of software, deep learning frameworks such as TensorFlow or PyTorch are supported, which facilitates algorithm development and optimization. The system is equipped with an operating system and development tools for code writing, debugging and deployment, and realizes efficient operation of the algorithm. In terms of data, a large amount of multispectral image data sets covering multispectral images of targets in various scenes are included for algorithm training and testing.

[0146] Exemplarily, step S40 obtains a target region and position coordinates of the target region based on a pre-set feature extraction and target recognition model according to the multispectral color image data, including:

[0147] S41: Based on a pre-set deep learning convolutional neural network, shallow low-level features and deep high-level features in the spectral modulation data of each spectral channel 204 are extracted from the spectral modulation data of each pixel after reconstruction in the multispectral color image data.

[0148] Exemplarily, the shallow low-level features generally refer to features with a small difference from the adjacent feature values, and the corresponding material information cannot be determined according to the shallow low-level features.

[0149] Exemplarily, the deep high-level features generally refer to features with a large difference from the adjacent feature values, and the corresponding material information can be directly determined according to the deep high-level features.

[0150] Exemplarily, as shown in Figure 10 The feature extraction and target recognition model includes a feature extraction and fusion module and a target detection and recognition module. A deep learning-based convolutional neural network (CNN) is used for feature extraction and fusion. Through the CNN multi-branch network structure, each branch corresponds to a spectral channel 204. Through the convolutional layer and the pooling layer, the shallow low-level features and the deep high-level features of each spectral channel 204 are extracted.

[0151] S42: Based on a pre-set dynamic weighting model, the shallow low-level features and the deep high-level features of different spectral channels 204 are weighted and fused to obtain a fused feature map, wherein the weight of each spectral channel 204 is dynamically adjusted according to the environmental characteristics.

[0152] Exemplarily, the dynamic weighting model is trained according to the shallow low-level features and the deep high-level features of different scenes. The dynamic adjustment of the weights includes assigning an initial weight value to the features (the shallow low-level features and the deep high-level features) of different spectral channels 204, and after sensing the change of the scene, dynamically adjusting the weights of each spectral channel 204. The weights are dynamically adjusted according to the importance of the waveband of the spectral channel 204 in the scene. For example, at night, the weight corresponding to the near-infrared waveband spectral channel 204 is increased. The CNN cross-layer performs feature fusion to further integrate the shallow low-level features and the deep high-level features to form feature representation.

[0153] Exemplarily, the process of feature extraction and fusion is the process of recognizing the contour, and the boundary between different material information is obtained through the feature extraction and fusion module to distinguish different material information. At this time, only different material information can be distinguished, and the ground object corresponding to the material information cannot be determined.

[0154] For example, the boundary between the tree and the sky can be obtained through the feature extraction and fusion module. However, it cannot be determined whether it is a tree or the sky.

[0155] S43: According to the fusion feature map and the material information of each pixel, the target region and its corresponding position information are determined in the multi-spectral color image data, and a target box is generated.

[0156] Exemplarily, as shown in Figure 10 The target detection and recognition module uses the Faster R-CNN algorithm for target detection and recognition, and the target is a person. According to the fusion feature map and the material information of each pixel, the candidate region of the target is generated through the region proposal network (RPN), the candidate region is classified through the classification regression network, the candidate region is regressed through the frame, the target category and position are determined, and then the target region and its corresponding position information are determined and a target box is generated.

[0157] Exemplarily, as shown in Figure 10 When the Faster R-CNN algorithm in the target detection and recognition module is trained, the fusion feature map is input, the network parameters are adjusted through the back propagation algorithm, and the target detection and recognition effect is optimized.

[0158] S50: Output the multi-spectral color image data and display the target box in the multi-spectral color image data.

[0159] Exemplarily, as shown in Figure 8As shown, the post-processing output module compresses and visualizes the recognition result, including a data compression unit, a power management unit and an image output interface unit. The data compression unit adopts a compression algorithm suitable for multispectral images, hardware acceleration improves processing speed, and is compatible with multiple display interfaces. A video processing and display module is arranged in the data compression unit, which superimposes the detected target frame and category information on the multispectral color image to generate a visual multispectral color image data, encodes the image sequence into a video stream using an H.264 encoder, and outputs a visible video through a display. The power management unit dynamically adjusts the power dissipation and heat dissipation system to prevent the module from overheating and ensure stable operation. The image output interface unit is configured with an MIPI-DSI interface compatible with multiple display and storage devices, supporting different data formats and transmission rates. The synchronization signal processing logic ensures the accuracy and synchronization of data output.

[0160] The image acquisition module 3 of the embodiment can provide high-quality image acquisition, processing and transmission, is suitable for various multispectral image processing tasks, improves the adaptability and expansibility of the system, and ensures efficient, accurate and reliable detection performance.

[0161] In the present application, the terms "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance. The term "a plurality of" refers to two or more, unless otherwise explicitly limited.

[0162] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application following the general principles of the present application and including known or customary technical methods not disclosed in the present application. The specification and examples are only considered as exemplary.

[0163] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for low-light imaging based on multispectral recognition, characterized in that, The micro-light imaging system based on multi-spectral recognition comprises a shell, an objective lens, a multi-spectral detector and an image acquisition module. The objective lens and the shell are rotationally connected, the multi-spectral detector and the image acquisition module are electrically connected, and the multi-spectral detector, the image acquisition module and the shell are fixedly connected. The multi-spectral detector comprises a detector body and a micro-nano structure film layer, the micro-nano structure film layer is arranged on a side of the detector body facing the objective lens, and the micro-nano structure film layer is used for modulating a projection spectrum. The method comprises: acquiring spectral modulation data; preprocessing the spectral modulation data to obtain preprocessed spectral modulation data; based on a pre-set multi-spectral calculation and color synthesis model, obtaining multi-spectral color image data from the preprocessed spectral modulation data, wherein the multi-spectral color image data comprises material information of each pixel; based on a pre-set feature extraction and target recognition model, obtaining a target region and position coordinates of the target region from the multi-spectral color image data and generating a target frame; outputting the multi-spectral color image data and displaying the target frame in the multi-spectral color image data; based on a pre-set multi-spectral calculation and color synthesis model, obtaining multi-spectral color image data from the preprocessed spectral modulation data, wherein the multi-spectral color image data comprises material information of each pixel, comprising: acquiring spectral modulation data of each pixel in the preprocessed spectral modulation data, and reconstructing the spectral modulation data of each pixel, wherein the spectral modulation data of each pixel after reconstruction comprises multi-channel spectral modulation data; wherein the spectral modulation data of each pixel is reconstructed by using a marginal gradient extension method; based on a pre-set multi-element multi-target optimization model, solving the multi-element multi-target regression model by a partial least squares regression algorithm to perform color prediction according to the spectral modulation data of each pixel after reconstruction, and establishing a mapping relationship between the spectral modulation data and RGB; wherein 9 independent variables of the multi-element multi-target optimization model comprise spectral modulation data in eight spectral channels and one normal channel in each channel unit, and 3 dependent variables comprise R value, G value and B value; obtaining color restoration coefficients K and B according to the mapping relationship between the spectral modulation data and RGB, and generating color image data by color restoration of each pixel data according to the color restoration coefficients K and B and the spectral modulation data; according to a pre-set fusion model, performing multi-scale fusion on the spectral modulation data of each pixel after reconstruction and the color image data by using multi-resolution analysis or wavelet transform to obtain fused image data; according to a pre-set calculation model, calculating the spectral modulation data of each pixel after reconstruction in the fused image data to obtain material information of each pixel and the multi-spectral color image data. The micro-nano structure film layer comprises a substrate and a super material micro-nano array.

2. The method of claim 1, wherein the method is a multispectral recognition based low light imaging method. The super material micro-nano array is located on a side of the detector body facing the objective lens, and the substrate is located on a side of the super material micro-nano array facing the objective lens. ​ 3. The method according to claim 2, wherein, The super material micro-nano array comprises a plurality of channel units; Each of the channel units comprises a plurality of spectral channels; Each of the spectral channels comprises a micro-nano structure array; One of the spectral channels corresponds to one of the micro-nano structure arrays, and the micro-nano structure arrays in the plurality of spectral channels are different from each other.

4. The method of claim 3, wherein the method is a multispectral recognition based low light imaging method. The number of the spectral channels arranged in the super material micro-nano array is a first number; The number of the pixels arranged in the detector body is a second number; The first number and the second number are the same; The spectral channels and the pixels are arranged one by one.

5. The method of claim 3, wherein the method is a multispectral recognition based low light imaging method. The size of the spectral channels arranged in the super material micro-nano array is a first size; The size of the pixels arranged in the detector body is a second size; The first size and the second size are the same.

6. The method of claim 1, wherein the method is a multispectral recognition based low light imaging method. The objective lens comprises a first lens, a second lens, a third lens, a fourth lens, a fifth lens, and a sixth lens, which are arranged in the order of gradually approaching the multi-spectral detector along the optical axis of the objective lens. The first lens, the second lens, the fourth lens, the fifth lens, and the sixth lens are all meniscus lenses, the first lens, the second lens, the fourth lens, and the fifth lens are curved away from the multi-spectral detector, and the sixth lens is curved towards the multi-spectral detector. The third lens is a double-concave lens.

7. The micro-light imaging method based on multi-spectral recognition according to claim 1, wherein according to a pre-set solving model, the spectral modulation data of each pixel after reconstruction in the fused image data is solved to obtain the material information of each pixel and the multi-spectral color image data, including: based on a pre-set waveband partition rule, the spectral modulation data of each pixel after reconstruction is partitioned by waveband to obtain the spectral modulation data of multiple wavebands; the spectral modulation data of each waveband is subjected to spectral inversion to obtain the spectral curve corresponding to the spectral modulation data of each waveband; based on a pre-set standard spectral curve of ground objects, the spectral curve corresponding to the spectral modulation data of each waveband is compared with the standard spectral curve of ground objects, and the material information corresponding to the spectral modulation data of each waveband is obtained according to the comparison result; and the material information of each pixel is obtained according to the material information corresponding to the spectral modulation data of each waveband.

8. The micro-light imaging method based on multi-spectral recognition according to claim 1, wherein based on a pre-set feature extraction and target recognition model, the target region and the position coordinates of the target region are obtained according to the multi-spectral color image data, and a target frame is generated, including: based on a pre-set deep learning convolutional neural network, the shallow low-level features and the deep high-level features in the spectral modulation data of each spectral channel are extracted according to the spectral modulation data of each pixel after reconstruction in the multi-spectral color image data. ​ ​ ​ ​ ​ ​ ​ The shallow low-level features and the deep high-level features of different spectral channels are weighted and fused based on a preset dynamic weighting model to obtain a fused feature map, wherein the weight of each spectral channel is dynamically adjusted according to environmental characteristics; According to the fused feature map and the material information of each pixel, a target region and corresponding position information thereof are determined in the multi-spectral color image data, and a target frame is generated.

Images