Intelligent optical sensing system based on multispectral fusion

Abstract

The application discloses an intelligent optical sensing system based on multispectral fusion, and relates to the technical field of optical sensing, comprising: a multispectral image acquisition module, which is used for synchronously acquiring original image data of a target scene under three spectral channels of visible light, near-infrared and short-wave infrared; a space-time registration and preprocessing module, which is used for high-precision space-time registration and radiation correction of images of different spectral channels; an adaptive feature extraction and fusion module, which is used for extracting multiscale spectral features from the registered multispectral images and dynamically selecting and weightedly fusing strategies according to scene contents; and a light-weight decision network module, which is used for outputting final target recognition and state discrimination results. Through the above technical scheme, the application can realize space-time consistency of multispectral data acquisition, improve cross-band registration accuracy, enhance robustness under low-contrast, occlusion and adverse weather conditions, greatly reduce calculation complexity and memory occupation, and is suitable for resource-limited edge computing devices.

Description

Technical Field

[0001] This invention relates to the field of optical sensing technology, and in particular to an intelligent optical sensing system based on multispectral fusion. Background Technology

[0002] Optical sensing technology, as a crucial means of modern information acquisition, has been widely applied in fields such as environmental monitoring, industrial inspection, medical diagnosis, and intelligent security. However, with the increasing complexity and sophistication of application scenarios, single-spectral-band optical sensing systems are gradually showing limitations in target recognition accuracy, environmental adaptability, and anti-interference capabilities. Multispectral imaging, by fusing spectral information from different bands, can effectively improve the resolution of targets and the depth of feature extraction, and has become a key development direction for next-generation intelligent sensing systems.

[0003] Among them, intelligent optical sensing systems based on multispectral fusion aim to comprehensively utilize data from multiple spectral channels, including visible light, near-infrared, and short-wave infrared, to achieve highly robust identification and state determination of targets in complex scenes through collaborative perception and information complementarity. These systems typically rely on simultaneous acquisition by multiple sensors, cross-band registration, and efficient fusion algorithms to construct more complete and reliable perception results.

[0004] Existing multispectral optical sensing systems generally suffer from problems such as insufficient spatiotemporal registration accuracy between spectral channels, static and fixed data fusion strategies, excessive computational resource consumption, and poor adaptability to dynamic lighting and severe weather. While some systems incorporate deep learning for feature fusion, the resulting model structures are bulky and difficult to deploy on edge devices. Other systems employ simple weighted or rule-based fusion methods, failing to fully exploit the semantic relationships between multispectral components, leading to significant performance degradation in low-contrast or occluded scenarios. These shortcomings severely restrict the promotion and implementation of multispectral sensing systems in real-time demanding and resource-constrained practical applications, necessitating a novel intelligent optical sensing architecture that balances accuracy, efficiency, and adaptability. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent optical sensing system based on multispectral fusion, which can effectively solve the problems in the background technology such as insufficient spatiotemporal registration accuracy between spectral channels, static data fusion strategy, excessive consumption of computing resources, and poor adaptability to dynamic lighting and severe weather.

[0006] To achieve the above objectives, the present invention provides the following technical solution: On one hand, an intelligent optical sensing system based on multispectral fusion, comprising the following components: a multispectral image acquisition module, used to simultaneously acquire raw image data of a target scene in three spectral channels: visible light, near-infrared, and short-wave infrared; a spatiotemporal registration and preprocessing module, receiving the raw image data output by the multispectral image acquisition module and performing high-precision spatiotemporal registration and radiometric correction on images of different spectral channels; an adaptive feature extraction and fusion module, connected to the spatiotemporal registration and preprocessing module, used to extract multi-scale spectral features from the registered multispectral image and dynamically select and weight fusion strategies based on scene content; a lightweight decision network module, receiving the fused feature map output by the adaptive feature extraction and fusion module and outputting the final target recognition and state discrimination results; and a system control and interface module, responsible for coordinating the timing control, data flow scheduling, and communication with external devices of the above modules.

[0007] Preferably, the multispectral image acquisition module specifically includes a visible light camera, a near-infrared camera, and a short-wave infrared camera. The three cameras are physically fixed by rigid structural components and share the same optical lens system. An optical prism separates the incident light onto sensor target surfaces with different spectral responses. The visible light camera operates in the wavelength range of 400nm to 700nm, the near-infrared camera operates in the wavelength range of 700nm to 1100nm, and the short-wave infrared camera operates in the wavelength range of 1100nm to 1700nm. The three cameras achieve microsecond-level synchronous exposure through hardware trigger signals to ensure that the acquired multispectral images are strictly aligned in time.

[0008] Furthermore, the spatiotemporal registration and preprocessing module performs the following operations: based on pre-calibrated camera intrinsic and extrinsic parameters, it performs geometric distortion correction on the images of the three spectral channels; it uses a hybrid registration algorithm based on phase correlation and SIFT feature point matching to calculate the sub-pixel-level spatial transformation matrix between the visible light channel image and the near-infrared and short-wave infrared channel images; it uses this transformation matrix to resample the near-infrared and short-wave infrared images to achieve pixel-level spatial alignment with the visible light image; and it performs non-uniformity correction and dark current compensation on the image of each channel to eliminate the fixed-mode noise of the sensor itself.

[0009] Furthermore, the adaptive feature extraction and fusion module includes a multi-branch feature extraction network and a dynamic weight generation network. The multi-branch feature extraction network has three parallel branches, which process the registered images of the visible light, near-infrared, and short-wave infrared channels, respectively. Each branch consists of four convolutional layers, used to extract multi-scale features from low-level texture to high-level semantics. The dynamic weight generation network takes the image of the visible light channel as the main input, and analyzes the global illumination conditions, target contrast, and occlusion of the current scene in real time through a compact convolutional neural network, and outputs a 3D fusion weight vector. ,in Final fused feature map From the feature map of each channel We obtain the weighted summation according to their corresponding weights: .

[0010] Furthermore, the lightweight decision network module is constructed using depthwise separable convolution and channel attention mechanisms. Its input is the fused feature map output by the adaptive feature extraction and fusion module, which is sequentially passed through two depthwise separable convolutional layers for feature compression and refinement, and then through a global average pooling layer to convert the feature map into a feature vector. Subsequently, the feature vector passes through a fully connected layer containing a channel attention mechanism to adaptively enhance the response of key feature channels, and finally through a Softmax classifier to output the probability of the target's class and its state confidence.

[0011] On the other hand, a smart optical sensing method based on multispectral fusion is proposed. The method comprises the following steps: Step S110, simultaneously capturing raw image data of the target scene in the visible, near-infrared, and short-wave infrared bands using a multispectral image acquisition module; Step S120, in the spatiotemporal registration and preprocessing module, performing geometric distortion correction, cross-channel sub-pixel-level spatial registration, and radiometric calibration on the raw image data to generate a spatiotemporally aligned and radiometrically consistent multispectral image group; Step S130, in the adaptive feature extraction and fusion module, extracting multi-scale spectral features from each channel image of the multispectral image group, and performing feature weighted fusion based on dynamically generated channel weights to generate a unified fused feature representation; Step S140, inputting the fused feature representation into a lightweight decision network module, and after feature compression, channel attention weighting, and classification decision, outputting the final target recognition result and state discrimination information.

[0012] The beneficial effects of this invention are:

[0013] 1. By using a shared optical lens and hardware synchronization triggering mechanism, the spatiotemporal consistency of multispectral data acquisition is ensured from the source, significantly improving cross-band registration accuracy;

[0014] 2. A scene-adaptive dynamic feature fusion strategy is adopted to replace static weighted fusion. It can dynamically adjust the contribution of each spectral channel according to the actual ambient lighting and target characteristics, thereby maintaining high robustness under low contrast, occlusion and harsh weather conditions.

[0015] 3. The entire processing flow, especially the feature extraction and decision network, adopts a deep separable convolution and compact network structure design, which significantly reduces computational complexity and memory consumption while ensuring recognition accuracy, enabling the system to be efficiently deployed on resource-constrained edge computing devices;

[0016] 4. The system architecture achieves full-process optimization from data acquisition, preprocessing, fusion to decision-making, taking into account perception accuracy, processing efficiency and adaptability to different application scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall technical solution architecture proposed in this invention;

[0018] Figure 2 This is a schematic diagram of the core principle framework of the adaptive feature extraction and fusion module in this invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0020] Example 1:

[0021] In border security monitoring applications, intelligent optical sensing systems based on multispectral fusion are deployed at border outpost observation points to provide continuous, 24 / 7 monitoring of the surrounding area. (See also...) Figure 1This system comprises a multispectral image acquisition module, a spatiotemporal registration and preprocessing module, an adaptive feature extraction and fusion module, a lightweight decision network module, and a system control and interface module. The multispectral image acquisition module uses rigid structural components to fix a visible light camera, a near-infrared camera, and a short-wave infrared camera onto the same optical platform. The three cameras share an optical lens system composed of multi-layered coated lenses. Incident light is precisely separated by a beam splitter onto sensor target surfaces with different spectral responses. The visible light camera uses a Sony IMX585 CMOS sensor with 12 million effective pixels and a strictly limited operating wavelength range of 400 nm to 700 nm, responsible for acquiring color texture information of the target scene. The near-infrared camera is equipped with a Hamamatsu G13122-512K InGaAs sensor with a pixel size of 512×512 and an operating wavelength range of 700 nm to 1100 nm, specifically designed to capture vegetation reflection characteristics and weak light sources at night. The shortwave infrared camera uses a FLIR A8580 SLS cooled sensor with a resolution of 640×512 and an operating wavelength extending from 1100 nm to 1700 nm, possessing the special ability to penetrate thin fog and identify camouflage materials. The three cameras achieve synchronized exposure control via hardware trigger signals generated by an FPGA, with an exposure time deviation of less than 1 microsecond, ensuring complete temporal alignment of the acquired multispectral images. After exposure, each camera transmits the raw image data in parallel to the spatiotemporal registration and preprocessing module via the Camera Link interface.

[0022] After receiving the raw image data, the spatiotemporal registration and preprocessing module first performs geometric distortion correction. Based on the pre-calibrated camera intrinsic parameter matrix and distortion coefficients, radial and tangential distortion corrections are performed on the image of each spectral channel. The intrinsic parameter matrix includes parameters such as focal length and principal point coordinates, and is obtained by segmental calibration within a temperature range of -10℃ to 50℃ using the Zhang Zhengyou calibration method. The corrected image enters the spatial registration stage, employing a hybrid registration algorithm combining phase correlation and SIFT feature point matching. The phase correlation algorithm calculates the translation parameters of the visible light and near-infrared channel images in the frequency domain, transforms the image to the frequency domain using Fast Fourier Transform, calculates the cross-power spectrum, locates the peak position, and obtains sub-pixel-level translation estimates. The SIFT feature point matching algorithm extracts stable feature points in the spatial domain, performs feature descriptor matching on the visible light and short-wave infrared images, uses the RANSAC algorithm to eliminate mismatched point pairs, and finally solves for a 3×3 homography matrix that includes rotation, scaling, and affine transformations. The calculated transformation matrix was applied to near-infrared and short-wave infrared images, and a bicubic interpolation algorithm was used for resampling to achieve pixel-level spatial alignment with the visible light images, with registration errors controlled within 0.3 pixels. In the radiometric preprocessing stage, non-uniformity correction was performed on each channel. A lookup table for the gain and bias coefficients of each pixel was established by acquiring uniform blackbody and whiteboard images to compensate for sensor response nonlinearity in real time. Simultaneously, dark current compensation was performed; a dark-field image was acquired before each exposure, and dark current noise was subtracted from the original image to ensure radiometric measurement accuracy reached a 12-bit quantization level.

[0023] See Figure 2The adaptive feature extraction and fusion module receives the registered multispectral image set and processes the three spectral channels in parallel through a multi-branch feature extraction network. The visible light branch inputs an RGB image with a size of 1024×1024×3, and extracts features step by step through four convolutional layers. The first convolutional layer uses 64 7×7 convolutional kernels with a stride of 2, and the output feature map size is 512×512×64; the second convolutional layer uses 128 5×5 convolutional kernels with a stride of 1, and the output feature map size is 512×512×128; the third convolutional layer uses 256 3×3 convolutional kernels with a stride of 2, and the output feature map size is 256×256×256; the fourth convolutional layer uses 512 3×3 convolutional kernels with a stride of 1, and the output feature map size is 256×256×512. The near-infrared and short-wave infrared branches use the same network structure, but the input is a single-channel image, and the number of convolutional kernels in each layer is halved to reduce computational complexity. The dynamic weight generation network takes the visible light channel image as input and analyzes scene characteristics through a compact convolutional neural network. This network contains three convolutional layers and two fully connected layers. The first convolutional layer uses 16 5×5 convolutional kernels to extract global illumination distribution features; the second convolutional layer uses 32 3×3 convolutional kernels to calculate image contrast indicators; and the third convolutional layer uses 64 3×3 convolutional kernels to detect potential occlusion region features. Subsequently, global average pooling converts the feature map into a 256-dimensional feature vector, which is then mapped to a 3-dimensional fusion weight vector through two fully connected layers. The fusion weight vector output by the dynamic weight generation network satisfies normalization constraints, with each weight component ranging from 0 to 1, and the sum of the three components strictly equal to 1. The final fusion feature map is obtained by weighted summation of the feature maps of each channel according to their corresponding weights. The specific calculation formula is as follows: The visible light feature map weight component reflects the importance of color texture information, the near-infrared feature map weight component reflects the value of thermal radiation characteristics, and the short-wave infrared feature map weight component characterizes the contribution of material recognition capability. In dense fog conditions, the system automatically increases the weight of the short-wave infrared channel to above 0.6 to fully utilize its penetration capability; in nighttime monitoring scenarios, the near-infrared channel weight is adjusted to the range of 0.5-0.7 to prioritize its night vision performance; during clear daytime conditions, the visible light channel weight is maintained at a level of 0.4-0.5 to ensure effective utilization of color information.

[0024] The lightweight decision network module receives a fused feature map of size 512×256×256. It first performs feature compression through two depthwise separable convolutional layers. The first depthwise separable convolutional layer uses 256 3×3 depthwise convolutional kernels and 512 1×1 point convolutional kernels to compress the feature map size to 256×128×512. The second depthwise separable convolutional layer uses 512 3×3 depthwise convolutional kernels and 1024 1×1 point convolutional kernels, outputting a feature map size of 128×64×1024. The computational cost of depthwise separable convolution is reduced by approximately 70% compared to standard convolution, while maintaining similar feature representation capabilities. After feature compression, a global average pooling layer transforms the feature map into a 1024-dimensional feature vector, which then enters a fully connected layer with channel attention mechanism. This layer first calculates the attention weights for each channel using two fully connected layers. The first fully connected layer compresses the 1024-dimensional input to 64 dimensions using the ReLU activation function; the second fully connected layer restores the 64-dimensional features to 1024 dimensions and uses the Sigmoid activation function to generate channel weights between 0 and 1. The original feature vector is then multiplied element-wise with the channel weights to enhance the response strength of key feature channels and suppress interference from irrelevant features. Finally, a Softmax classifier outputs the target recognition results. The classifier contains eight output nodes, corresponding to categories such as people, vehicles, drones, animals, vegetation, buildings, camouflaged targets, and unknown objects. It also outputs the probability value and state confidence score for each category. The confidence score threshold is set to 0.85; targets below this threshold are marked as requiring manual review.

[0025] The system control and interface module is implemented using the Xilinx Zynq UltraScale+ MPSoC platform, coordinating data flow transmission between modules via the AXI bus. This module generates precise timing control signals, including camera trigger pulses, exposure control signals, and data acquisition clocks, ensuring the entire system operates stably at a rate of 10 frames per second. It communicates with the command center server via a Gigabit Ethernet interface, using the UDP protocol to transmit recognition results and compressed image data with a transmission latency of less than 50 milliseconds. It also provides an RS-485 interface for connecting to an external pan-tilt control system, automatically adjusting the monitoring angle and focus based on the recognition results to achieve dynamic target tracking. The power management unit supports 12V DC input and internally generates multiple power supplies (3.3V, 1.8V, and 1.2V) through a DC-DC converter, keeping the overall power consumption below 15 watts, suitable for outdoor solar-powered environments.

[0026] Example 2:

[0027] In industrial product quality inspection scenarios, this system is applied to electronic product production lines for multispectral defect detection of circuit board assemblies. The multispectral image acquisition module is mounted at the end of a robotic arm, moving and capturing images in sync with the production line. The visible light camera uses a 20-megapixel global shutter sensor with a working distance of 30 cm and a resolution of 10 micrometers per pixel, clearly capturing solder joint morphology and component location. The near-infrared camera is equipped with an LED active illumination source, a wavelength of 850 nm, and an adjustable power range of 5-50 milliwatts, specifically designed to detect invisible internal connections such as cold solder joints and cracks. The short-wave infrared camera uses an InGaAs linear array sensor, with a scanning speed synchronized with the production line, specifically identifying bubbles and delamination defects within the package. The three cameras are equipped with a precision mechanical structure to ensure that the optical axis parallelism error is less than 0.01 degrees, and the robotic arm synchronously triggers image acquisition from all three cameras at each stop point.

[0028] The spatiotemporal registration and preprocessing module employs a combination of template matching and feature alignment, using standard qualified product images as reference templates to perform sub-pixel-level registration on the images to be inspected. During registration, special attention is paid to the corner, edge, and texture features of components to ensure that the positional deviation of the same component in different spectral images is less than 2 micrometers. Radiometric correction focuses on compensating for illumination non-uniformity, establishing an illumination compensation model by acquiring a standard white board to eliminate the influence of ambient light on measurement accuracy.

[0029] The adaptive feature extraction and fusion module optimizes the network structure for industrial inspection characteristics. The number of convolutional layers in each branch of the multi-branch feature extraction network is increased to 6, expanding the receptive field to capture a wider range of defect features. The dynamic weight generation network incorporates prior knowledge and establishes weight preference rules for common defect types: increasing the near-infrared weight to 0.7 for welding defect detection, increasing the short-wave infrared weight to 0.8 for packaging defect detection, and maintaining the visible light weight above 0.6 for appearance defect detection. In the feature fusion stage, in addition to weighted summation, channel stitching is used to preserve the unique features of each spectrum, forming a 1536-dimensional fused feature vector.

[0030] The lightweight decision network module output layer includes 12 defect categories, such as solder balls, cold solder joints, misalignment, warping, cracks, bubbles, contamination, oxidation, scratches, discoloration, missing parts, and excess parts. To meet the high reliability requirements of industrial inspection, a dual verification mechanism is implemented: initial identification confidence levels of 0.95 or higher result in direct judgment; levels between 0.85 and 0.95 trigger a multi-frame verification process; and levels below 0.85 are transferred to a manual re-inspection queue. The system control and interface module is deeply integrated with the production line PLC system, exchanging control signals in real time via the PROFINET protocol. Upon detecting a defect, an audible and visual alarm is immediately triggered, and the defect location and type are recorded. Simultaneously, a robotic arm is controlled to separate the defective product to the rework area. The system's detection cycle is 800 milliseconds, with a missed detection rate below 0.1% and a false detection rate below 0.5%, meeting the requirements of Industry 4.0 intelligent manufacturing.

[0031] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An intelligent optical sensing system based on multispectral fusion, characterized in that: It includes the following components: The multispectral image acquisition module is used to simultaneously acquire raw image data of the target scene in three spectral channels: visible light, near infrared, and short-wave infrared. The spatiotemporal registration and preprocessing module receives the raw image data output by the multispectral image acquisition module and performs high-precision spatiotemporal registration and radiometric correction on images of different spectral channels. An adaptive feature extraction and fusion module, connected to the spatiotemporal registration and preprocessing module, is used to extract multi-scale spectral features from the registered multispectral image and dynamically select and weighted fusion strategies based on scene content. The lightweight decision network module receives the fused feature map output by the adaptive feature extraction and fusion module, and outputs the final target recognition and state discrimination results; The system control and interface module is responsible for coordinating the timing control, data flow scheduling, and communication with external devices of the above modules. The multispectral image acquisition module specifically includes a visible light camera, a near-infrared camera, and a short-wave infrared camera. These three cameras are physically fixed by rigid structural components and share the same optical lens system. A beam-splitting prism separates the incident light onto sensor target surfaces with different spectral responses. The visible light camera operates in the 400nm to 700nm wavelength range, the near-infrared camera in the 700nm to 1100nm wavelength range, and the short-wave infrared camera in the 1100nm to 1700nm wavelength range. The three cameras achieve microsecond-level synchronous exposure via hardware trigger signals. The spatiotemporal registration and preprocessing module performs the following operations: based on pre-calibrated camera intrinsic and extrinsic parameters, it performs geometric distortion correction on the images of the three spectral channels; it uses a hybrid registration algorithm based on phase correlation and SIFT feature point matching to calculate the sub-pixel-level spatial transformation matrix between the visible light channel image and the near-infrared and short-wave infrared channel images; it uses this transformation matrix to resample the near-infrared and short-wave infrared images to achieve pixel-level spatial alignment with the visible light image; and it performs non-uniformity correction and dark current compensation on the image of each channel. The adaptive feature extraction and fusion module comprises a multi-branch feature extraction network and a dynamic weight generation network. The multi-branch feature extraction network has three parallel branches, which process the registered images of the visible light, near-infrared, and short-wave infrared channels, respectively. Each branch consists of four convolutional layers. The dynamic weight generation network takes the image of the visible light channel as the main input and analyzes the global illumination conditions, target contrast, and occlusion of the current scene in real time through a compact convolutional neural network, and outputs a 3D fusion weight vector. ,in Final fused feature map From the feature map of each channel We obtain the weighted summation according to their corresponding weights: ; The lightweight decision network module is constructed using depthwise separable convolution and channel attention mechanisms. Its input is the fused feature map output by the adaptive feature extraction and fusion module. It is then passed through two depthwise separable convolutional layers for feature compression and refinement, and then through a global average pooling layer to convert the feature map into a feature vector. Subsequently, the feature vector passes through a fully connected layer containing a channel attention mechanism to adaptively enhance the response of key feature channels. Finally, a Softmax classifier outputs the probability of the target's class and its state confidence. In the adaptive feature extraction and fusion module, the visible light branch of the multi-branch feature extraction network takes an RGB image with a size of 1024×1024×3 as input. Features are extracted step-by-step through four convolutional layers. The first convolutional layer uses 64 7×7 convolutional kernels with a stride of 2, outputting a feature map size of 512×512×64. The second convolutional layer uses 128 5×5 convolutional kernels with a stride of 1, outputting a feature map size of 512×512×128. The third convolutional layer uses 256 3×3 convolutional kernels with a stride of 2, outputting a feature map size of 256×256×256. The fourth convolutional layer uses 512 3×3 convolutional kernels with a stride of... The output feature map size is 256×256×512. The near-infrared branch and the short-wave infrared branch adopt the same network structure, with a single-channel image as input and the number of convolutional kernels in each layer halved. The dynamic weight generation network contains 3 convolutional layers and 2 fully connected layers. The first convolutional layer uses 16 5×5 convolutional kernels to extract global illumination distribution features. The second convolutional layer uses 32 3×3 convolutional kernels to calculate the image contrast index. The third convolutional layer uses 64 3×3 convolutional kernels to detect potential occlusion region features. Then, the feature map is converted into a 256-dimensional feature vector through global average pooling, and mapped to a 3-dimensional fusion weight vector through two fully connected layers.

2. The multispectral fusion based intelligent optical sensing system as claimed in claim 1, wherein: In the multispectral image acquisition module, the visible light camera uses a Sony IMX585 CMOS sensor with 12 million effective pixels; the near-infrared camera is equipped with a Hamamatsu G13122-512K InGaAs sensor with a pixel size of 512×512; and the short-wave infrared camera uses a FLIR A8580 SLS cooled sensor with a resolution of 640×512. The three cameras achieve synchronous exposure control through hardware trigger signals generated by the FPGA, with an exposure time deviation of less than 1 microsecond.

3. The intelligent optical sensing system based on multispectral fusion according to claim 1, characterized in that: In the spatiotemporal registration and preprocessing module, the geometric distortion correction includes radial distortion correction and tangential distortion correction; the camera intrinsic parameter matrix includes focal length and principal point coordinate parameters, which are obtained by segmental calibration within a temperature range of -10℃ to 50℃ using the Zhang Zhengyou calibration method; in the hybrid registration algorithm, the phase correlation algorithm calculates the translation parameters of the visible light channel and near-infrared channel images in the frequency domain, transforms the image to the frequency domain through fast Fourier transform, calculates the cross power spectrum and locates the peak position, and obtains sub-pixel level translation estimates; the SIFT feature point matching algorithm extracts stable feature points in the spatial domain, performs feature descriptor matching on the visible light and short-wave infrared images, uses the RANSAC algorithm to remove mismatched point pairs, and finally solves for a 3×3 homography matrix containing rotation, scaling and affine transformation.

4. The intelligent optical sensing system based on multispectral fusion according to claim 1, characterized in that: In the lightweight decision network module, the first depthwise separable convolutional layer uses 256 3×3 depth convolutional kernels and 512 1×1 point convolutional kernels to compress the feature map size to 256×128×512. The second depthwise separable convolutional layer uses 512 3×3 depthwise convolutional kernels and 1024 1×1 pointwise convolutional kernels, with an output feature map size of 128×64×1024. The channel attention mechanism fully connected layer first calculates the attention weights of each channel through two fully connected layers. The first fully connected layer compresses the 1024-dimensional input to 64 dimensions using the ReLU activation function. The second fully connected layer restores the 64-dimensional features to 1024 dimensions and uses the Sigmoid activation function to generate channel weights between 0 and 1.

5. The multispectral fusion based intelligent optical sensing system as claimed in claim 1, wherein: The system control and interface module is implemented using the Xilinx Zynq UltraScale+ MPSoC platform, coordinating data flow transmission between modules via the AXI bus. This module generates precise timing control signals, including camera trigger pulses, exposure control signals, and data acquisition clocks, ensuring the entire system operates stably at a rate of 10 frames per second. It communicates with the command center server via a gigabit Ethernet interface, using the UDP protocol to transmit recognition results and compressed image data with a transmission latency of less than 50 milliseconds. It also provides an RS-485 interface for connecting to an external pan-tilt control system.

6. A sensing method of the intelligent optical sensing system based on multispectral fusion according to claim 1, characterized in that, Includes the following steps: Step S110: Simultaneously capture raw image data of the target scene in the visible light, near-infrared and short-wave infrared bands through the multispectral image acquisition module; Step S120: In the spatiotemporal registration and preprocessing module, geometric distortion correction, cross-channel subpixel-level spatial registration, and radiometric calibration are performed on the original image data to generate a spatiotemporally aligned and radiometrically consistent multispectral image group. Step S130: In the adaptive feature extraction and fusion module, multi-scale spectral features are extracted from each channel image of the multispectral image group, and feature weighted fusion is performed based on dynamically generated channel weights to generate a unified fusion feature representation. Step S140: The fused feature representation is input into the lightweight decision network module. After feature compression, channel attention weighting and classification decision, the final target recognition result and state discrimination information are output.

7. The sensing method of the intelligent optical sensing system based on multispectral fusion according to claim 6, characterized in that: In step S130, the dynamically generated channel weights are obtained by analyzing the global illumination conditions, target contrast, and occlusion in the current scene in real time through a compact convolutional neural network, and outputting a 3D fusion weight vector. ,in Final fused feature map From the feature map of each channel We obtain the weighted summation according to their corresponding weights: .

8. The sensing method of the intelligent optical sensing system based on multispectral fusion according to claim 6, characterized in that: In step S140, the feature compression is achieved through two depthwise separable convolutional layers; the channel attention weighting is achieved through a fully connected layer containing a channel attention mechanism. This layer first calculates the attention weight of each channel through two fully connected layers, and then multiplies the original feature vector with the channel weights element by element; the classification decision outputs the probability of the target's class and its state confidence through a Softmax classifier.

Images