Visible light, near infrared and thermal infrared multispectral imaging system

Abstract

The utility model relates to the technical field of agricultural remote sensing monitoring, and particularly discloses a visible light, near infrared and thermal infrared multispectral imaging system, which comprises an imaging module and a microcomputer, the imaging module is electrically connected with the microcomputer, the imaging module comprises a plurality of CMOS (complementary metal oxide semiconductor) cameras and a thermal infrared imaging camera, and the thermal infrared imaging camera is electrically connected with the microcomputer. The front end of the CMOS camera is provided with a narrow-band optical filter. According to the system, eight groups of IMX273 global shutter CMOS cameras and industrial cameras are adopted for parallel acquisition, and compared with a beam splitter prism scheme, the resolution is improved by 5 times, so that the obtained image can present more details, the unique waveband configuration of the system covers key wavebands such as vegetation red edges and water vapor absorption, the indexes are accurately calculated, and the image quality is improved. And information such as the growth state, the health degree and the biomass of the vegetation can be evaluated more accurately.

Description

Technical Field

[0001] This utility model relates to the technical field of agricultural remote sensing monitoring, specifically to a visible light, near-infrared and thermal infrared multispectral imaging system. Background Technology

[0002] Significant progress has been made in the application of multispectral imaging technology in agricultural remote sensing monitoring, but existing systems still face technical bottlenecks in terms of dynamic scene adaptability and multi-source data fusion. Traditional solutions mainly employ two architectures: rotating filter systems, which achieve multi-band acquisition by mechanically rotating and switching filters. However, experiments show that filter switching requires an interval of 50-200ms, resulting in inter-band displacement errors (>3 pixels) when the UAV's flight speed is >5m / s, severely affecting the calculation accuracy of vegetation indices (such as NDVI). In addition, frequent mechanical movement reduces system reliability, especially increasing the failure rate in high-temperature and high-humidity environments; and beam-splitting prism systems, which use optical beam splitting to achieve simultaneous multi-band acquisition. However, this type of system has inherent defects. A 5-band beam-splitting prism reduces the effective pixels of a single band to 20% of the total resolution, resulting in a spatial resolution difference of more than 4 times between the thermal infrared channel and visible light data, making it difficult to meet the requirements for the coordinated inversion of farmland canopy temperature and chlorophyll content. In recent years, multi-camera array solutions have improved resolution through parallel acquisition, but their design is not optimized for UAV platforms and urgently needs improvement. Therefore, we propose a visible light, near-infrared and thermal infrared multispectral imaging system. Utility Model Content

[0003] In view of the above-mentioned technical problems in related technologies, this utility model provides a visible light, near-infrared and thermal infrared multispectral imaging system that can solve the above problems.

[0004] To achieve the above-mentioned technical objectives, the technical solution of this utility model is implemented as follows:

[0005] A visible light, near-infrared and thermal infrared multispectral imaging system includes an imaging module and a microcomputer. The imaging module is electrically connected to the microcomputer. The imaging module includes several CMOS cameras and a thermal infrared imaging camera. A narrowband filter is provided at the front end of the CMOS camera.

[0006] Furthermore, there are eight CMOS cameras, which are arranged in a square around the thermal infrared imaging camera, with the thermal infrared imaging camera located at the center of the square area formed by the eight CMOS cameras.

[0007] Furthermore, the center wavelength of the narrowband filter is any one of 492, 560, 665, 704, 740, 783, 865, and 940 nm, and the bandwidth is set to 15~40 nm.

[0008] Furthermore, the microcomputer is equipped with a USB 3.0 expansion dock interface, and all eight CMOS cameras are wired to the microcomputer via the USB 3.0 expansion dock interface.

[0009] Furthermore, the thermal infrared imaging camera sensor is a vanadium oxide microbolometer, and the thermal infrared imaging camera sensor is wired to a microcomputer.

[0010] Furthermore, the microcomputer is an NVIDIA Jetson TX2 NX, which has an embedded solid-state drive.

[0011] The beneficial effects of this invention are as follows: This system uses 8 groups of IMX273 global shutter CMOS cameras, with parallel acquisition by industrial cameras. The single-band resolution is maintained at 1440×1080, which is 5 times higher than that of the beam splitter prism scheme. This allows the acquired images to present more details. In agricultural monitoring, it can clearly distinguish the leaf texture of crops, spots of pests and diseases, and other minute features. In ecological environment assessment, it can accurately identify vegetation types, distribution range, and minute ecological changes. In water quality detection, it helps to observe tiny suspended matter and algae aggregation areas in water bodies, providing more accurate data support for related research and decision-making.

[0012] High-efficiency data processing capabilities: Utilizing the NVIDIA Jetson TX2 NX development board, which features a built-in 256-core Pascal GPU, the powerful GPU acceleration reduces data processing latency to less than 200ms, compared to over 1.2s for traditional industrial PC solutions. This rapid data processing capability ensures real-time processing of large amounts of multispectral and thermal infrared data during UAV flight, providing timely feedback on monitoring results. In agriculture, it enables real-time monitoring of crop growth, allowing for rapid intervention upon detecting anomalies. In ecological environment monitoring, it promptly captures information on environmental changes, buying time for ecological protection and restoration efforts. In water quality testing, it allows for real-time monitoring of dynamic water quality changes, timely detection of pollution events, and determination of pollution extent.

[0013] Precise vegetation index calculation: The system’s unique band configuration covers key bands such as vegetation red edge (704nm) and water vapor absorption (940nm). The optimized selection of these bands greatly improves the calculation accuracy of vegetation indices such as NDVI (Normalized Difference Vegetation Index) and EVI (Enhanced Vegetation Index). By accurately calculating these indices, information such as vegetation growth status, health, and biomass can be assessed more precisely.

[0014] Lightweight design advantages: The integrated design concept runs through the entire system. From the aluminum alloy weight-reducing structure of the integrated mounting bracket for hardware to the compact and reasonable layout between modules, the system weight is effectively reduced. This is crucial for drones, as it reduces the load on the drone and improves its flight performance and endurance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this utility model or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this utility model. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] The present invention will now be described in further detail with reference to the accompanying drawings.

[0017] Figure 1 This is a schematic diagram of the structure of a visible light, near-infrared and thermal infrared multispectral imaging system.

[0018] Figure 2 This is a schematic diagram of the imaging module.

[0019] In the picture:

[0020] 1. CMOS camera; 101. Narrowband filter; 2. Thermal infrared imaging camera; 3. Imaging module; 4. Unmanned aerial vehicle (UAV). Detailed Implementation

[0021] The technical solutions of the present utility model will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present utility model, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present utility model are within the protection scope of the present utility model.

[0022] like Figure 1-2As shown, this utility model discloses a visible light, near-infrared, and thermal infrared multispectral imaging system, including an imaging module 3 and a microcomputer. The imaging module 3 is electrically connected to the microcomputer. The imaging module 3 includes several CMOS cameras 1 and one thermal infrared imaging camera 2. Each CMOS camera 1 has a narrow-band filter 101 at its front end. There are eight CMOS cameras 1 arranged in a square around the thermal infrared imaging camera 2. The CMOS cameras 1 are IMX273 global shutter industrial cameras (1 / 2.9'' target area, 1440×1080 pixels) arranged in a symmetrical square layout with a field-of-view overlap rate ≥15%. This layout effectively expands the imaging coverage, ensuring comprehensive observation of the target area from different angles. The required field-of-view overlap rate ensures the continuity and integrity of image acquisition, avoiding blind spots. The thermal infrared imaging camera 2 is located at the center of the square area formed by the eight CMOS cameras 1. The microcomputer is an NVIDIA Jetson TX2 NX with an embedded solid-state drive. The solid-state drive of the microcomputer is running Ubuntu. The 18.04 system deploys a radiation correction model and efficiently processes the data acquired by the imaging module 3. The NVIDIA Jetson TX2 NX comes pre-installed with multispectral image quantification software, which integrates the functions of data acquisition, processing, and output.

[0023] During the data acquisition phase, the software coordinates the operation of each camera in the imaging module to achieve synchronous acquisition and adaptively adjusts camera parameters (such as exposure time) according to actual environmental conditions. During the data processing phase, the software runs the radiometric correction model deployed on the Jetson TX2 NX to correct the acquired raw data and improve data quality. During the data output phase, the software outputs the processed data in a standard format, including 16-bit RAW multispectral cube (1440×1080×8), 14-bit thermal infrared temperature field (256×192), and spatiotemporal synchronization metadata (UTC timestamp, geographic coordinates, attitude angle, IOS).

[0024] Example 1: Multispectral Imaging Array Installation: Eight Sony IMX273 CMOS cameras 1 are mounted on an aluminum alloy bracket of imaging module 3. The aluminum alloy bracket is mounted under the nose of drone 4. During installation, high-precision measuring tools, such as electronic angle meters, are used to accurately measure the optical axis angle between adjacent cameras to ensure that the deviation is controlled within a very small range, so as to meet the requirement that the field of view overlap rate of adjacent cameras is ≥15%. At the same time, in order to ensure the stability and reliability of camera installation, a special camera mounting clamp and high-strength screws are used to firmly fix the camera on the aluminum alloy bracket to avoid loosening or displacement during subsequent use, which would affect the imaging effect. The lenses of the CMOS camera and thermal infrared imaging camera 2 in imaging module 3 are both facing downwards. The back of imaging module 3 is fixed under the nose of drone 4.

[0025] Installation of Thermal Infrared Imaging Camera 2: The thermal infrared imaging camera 2 is fixed to the center of the bracket using thermally conductive silicone pads. These pads not only provide fixation but also effectively conduct heat generated during the operation of the thermal infrared module, ensuring it operates in a stable temperature environment and improving image quality. During installation, precise optical alignment equipment, such as a laser alignment device, is required to adjust the position of the thermal infrared imaging camera, ensuring that the perpendicularity error between its optical axis and the multispectral array reference plane is ≤0.1°. This guarantees spatial consistency between thermal infrared imaging and multispectral imaging, laying the foundation for subsequent data fusion.

[0026] Electrical Connection: Both the CMOS camera 1 and the thermal infrared imaging camera are connected to the Jetson TX2NX docking station via a USB 3.0 interface. During the connection process, a high-quality USB 3.0 data cable should be used to ensure the stability and high speed of data transmission. At the same time, pay attention to the correct insertion and removal of the interface to avoid abnormal data transmission due to interface damage or poor contact. After the connection is completed, device identification and driver installation should be performed in the Jetson TX2 NX system to ensure that the system can correctly identify each camera device and allocate the corresponding resources to it, ensuring smooth data transmission.

[0027] Multispectral array acquisition: The multispectral array CMOS camera acquires data at full resolution at 60fps in global shutter mode, with adaptive exposure time adjustment (1 / 10000s~1 / 500s). The system monitors ambient light intensity in real time through a built-in ambient light sensor and automatically adjusts the camera's exposure time according to changes in light intensity. When the light intensity is strong, the exposure time is shortened to prevent the image from being too bright; when the light intensity is weak, the exposure time is extended to ensure that the image has sufficient brightness and detail. In addition, to ensure the accuracy and stability of the acquired data, the camera's acquisition parameters are calibrated and checked at regular intervals during the acquisition process to ensure that the camera is always in optimal working condition.

[0028] Thermal Infrared Module Acquisition: The thermal frame rate of the thermal infrared imaging camera is synchronized 1:1 with the visible light acquisition. When the thermal infrared imaging camera starts up, non-uniformity correction is automatically performed. By measuring the reference blackbody, the response differences of each pixel of the thermal infrared detector are obtained, and the acquired image data is corrected to eliminate image noise and non-uniformity caused by differences in the characteristics of the detector itself. At the same time, hardware synchronization signals are used to ensure that the acquisition frame rate of the thermal infrared imaging camera is consistent with the acquisition frame rate of the rectangular array CMOS camera, realizing the synchronous acquisition of multispectral and thermal infrared images, and providing time consistency guarantee for subsequent data fusion.

[0029] Data Processing: System Operating Environment: The microcomputer runs Ubuntu 18.04. During system installation and configuration, the system is optimized according to hardware resources and data processing requirements, including memory management, CPU scheduling, and file system optimization, to ensure efficient and stable operation. At the same time, necessary drivers and dependent libraries are installed to support the operation of data processing software and radiation correction models.

[0030] Data processing flow: Radiometric correction preprocessing: The original multispectral and thermal infrared images are preprocessed using dark current and vignetting correction factors generated by the software correction module; dark current correction removes inherent sensor noise, and vignetting correction eliminates image brightness unevenness, providing high-quality data for subsequent processing.

[0031] Core radiometric correction processing: Run the radiometric correction model on Jetson TX2 NX to perform flat field correction, dark current compensation and response inhomogeneity correction to improve image quality and ensure reliable analysis results;

[0032] Reflectance conversion calculation: After radiometric correction, the software converts the ground object reflectance radiance image into a reflectance image by band; by obtaining the solar downlink radiometric quantization parameters and combining them with the reflectance calculation of the radiometric calibration plate, a high-precision ground object reflectance image is generated.

[0033] Angle information calculation: The software's angle module calculates the angle between the sun and the observation based on the observation time, ground features, and sensor coordinates, constructs the observation geometry, and provides key angle parameters for BRDF model inversion and ground feature reflection analysis;

[0034] BRDF model inversion processing: The BRDF module first creates an inversion dataset, which involves the extraction and integration of sensor and ground feature information to generate the dataset; then, a suitable model is selected, and the dataset is used to fit coefficients to generate an inversion model, revealing the surface reflectance characteristics.

[0035] Data output management: The processed data is output in a standard format, including multispectral cubes, thermal infrared temperature fields, spatiotemporal synchronous metadata, and BRDF model coefficients.

[0036] In the preferred technical solution, the center wavelength of the narrowband filter 101 is any one of 492, 560, 665, 704, 740, 783, 865, and 940 nm, and the bandwidth is set to 15~40 nm, covering key bands such as vegetation red edge (704 nm) and water vapor absorption (940 nm). The selection of these specific bands can accurately capture important information related to agricultural monitoring, ecological environment assessment, etc.

[0037] In the preferred technical solution, the microcomputer is equipped with a USB 3.0 expansion dock interface, and all eight CMOS cameras 1 are wired to the microcomputer through the USB 3.0 expansion dock interface.

[0038] In the preferred technical solution, the thermal infrared imaging camera sensor of the thermal infrared imaging camera 2 is a vanadium oxide microbolometer. The thermal infrared imaging camera sensor is wired to a microcomputer. The vanadium oxide microbolometer is an uncooled infrared detector based on vanadium oxide material. Its core principle is to utilize the characteristic that the resistance of vanadium oxide material changes with temperature under infrared radiation to convert the infrared radiation signal into an electrical signal, thereby realizing thermal imaging or temperature measurement.

[0039] The above description is only a preferred embodiment of the present utility model and is not intended to limit the present utility model. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present utility model should be included within the protection scope of the present utility model.

Claims

1. A multispectral imaging system for visible, near-infrared and thermal infrared light, characterized in that Including an imaging module (3) and a microcomputer, the imaging module (3) is electrically connected with the microcomputer, the imaging module (3) includes a plurality of CMOS cameras (1) and a thermal infrared imaging camera (2), the front end of the CMOS camera (1) is provided with a narrowband filter (101).

2. The multispectral imaging system of claim 1, wherein, The number of the CMOS camera (1) is eight, and the eight CMOS cameras (1) are distributed in a square outside the thermal infrared imaging camera (2), and the thermal infrared imaging camera (2) is located at the center of the square formed by the eight CMOS cameras (1).

3. The multi-spectral imaging system of claim 1, wherein, The center wavelength of the narrowband filter (101) is any one of 492, 560, 665, 704, 740, 783, 865 and 940 nm, and the bandwidth is set to 15-40 nm.

4. The multi-spectral imaging system of claim 1, wherein, The microcomputer is provided with a docking station USB3.0 interface, and the eight CMOS cameras (1) are all wired connected with the microcomputer through the docking station USB3.0 interface.

5. The multispectral imaging system of claim 1, wherein, The thermal infrared imaging camera sensor of the thermal infrared imaging camera (2) is a vanadium oxide microbolometer, and the thermal infrared imaging camera sensor is wired connected with the microcomputer.

6. The multi-spectral imaging system of claim 1, wherein, The microcomputer is Nvidia Jetson TX2 NX, and the Jetson TX2 NX is embedded with a solid state disk.

Images