A multispectral intelligent recognition method based on optoelectronic technology

Abstract

The application discloses a multispectral intelligent recognition method based on photoelectric technology, and belongs to the field of machine vision and artificial intelligence. In view of the three defects that static weight cannot be self-adaptively adjusted, auxiliary modal learning is insufficient, and dynamic spectral drift cannot be online corrected in the existing multispectral fusion method, the light quality perception subnetwork is used to evaluate the scene light quality in real time, and the fusion gate weight of the visible light and thermal infrared two modes is dynamically output, so that the noise mode is automatically suppressed in the extreme light scene. Through the cross-modal comparison alignment framework, the features of the good light frame are taken as anchor points, the comparison alignment loss is calculated by combining the first-in-first-out negative sample queue, cross-domain spectral drift correction is realized without offline re-marking, and through the gradient decoupling modal balance training strategy, the gradient contribution degree of each mode is monitored in real time and is dynamically scaled in the back propagation stage, and the network is forced to learn the discrimination information of each spectral mode in a balanced manner.

Description

Technical Field

[0001] This invention belongs to the field of machine vision and artificial intelligence technology, specifically relating to a multispectral intelligent recognition method based on optoelectronic technology. It is particularly suitable for adaptive fusion, feature alignment and target recognition tasks of visible light and thermal infrared dual-modal images, and can be applied to all-weather industrial inspection, security monitoring and autonomous driving perception and other scenarios. Background Technology

[0002] Multispectral imaging technology, by fusing sensing information from multiple electromagnetic bands such as visible light and thermal infrared, can theoretically significantly improve the environmental robustness of machine vision systems. Thermal infrared sensors, based on the target's own thermal radiation imaging, do not rely on external light sources and have a natural advantage in dark or bright light environments. Visible light cameras, on the other hand, can capture rich texture, color, and shape details and have high recognition accuracy when there is sufficient light. The two complement each other, and their combined application has been widely practiced in fields such as industrial inspection, all-weather security monitoring, and autonomous driving perception.

[0003] Deep neural networks have become the mainstream algorithm paradigm for multispectral image fusion and recognition. Common architectures include: using a two-stream convolutional network to extract features from each modality and then fusing them, and adaptive weight fusion methods based on attention mechanisms. The closest existing technical solution to this invention is a two-stream multispectral fusion recognition method based on static attention weights. Its typical structure consists of four main modules: a visible light feature extraction backbone network, a thermal infrared feature extraction backbone network, a static channel attention fusion module, and a classification or detection output head. The processing flow of this solution is as follows: the system simultaneously acquires aligned visible light and thermal infrared images, inputs them into the corresponding backbone networks to extract feature maps, and then inputs the two feature maps into the static channel attention fusion module. A fixed channel attention weight vector is generated through global average pooling and fully connected layers. The fused features are obtained by weighting the stitched feature maps, and finally, the target recognition or classification task is completed through the output head.

[0004] The existing solutions described above have the following three fundamental flaws:

[0005] First, the static weight fusion mechanism leads to the failure of recognition in extreme scenes. The weights of the fusion module in the existing solution are fixed after training and cannot perceive the real-time illumination changes during the inference stage. In extremely dark or rainy / foggy scenes, the visible light modal feature map is included in the fusion calculation in a fixed proportion with high noise and low signal-to-noise ratio. The noise features pollute the fusion results, and the final recognition accuracy drops significantly, even lower than the baseline performance of the single infrared model, making the original design intention of the multispectral system completely fail.

[0006] Second, the lack of a modal equalization mechanism during the training phase leads to insufficient auxiliary modal learning. The high-frequency texture information density of visible light images is much higher than that of infrared images. During gradient descent, the network naturally tends to update parameters in the direction that minimizes the loss of the visible light branch, resulting in the parameters of the thermal infrared backbone network not being fully optimized. The feature extraction capability has been suppressed for a long time. Even in nighttime or occluded scenes where infrared should dominate, the system still cannot achieve the expected recognition robustness due to the low quality of thermal infrared branch features.

[0007] Third, static spectral alignment schemes cannot cope with spectral drift in dynamic scenes. Existing schemes rely on offline calibration to complete multispectral alignment. The calibration parameters are only effective for scenes under the training distribution. When dynamic shadows, occlusions, or sudden changes in illumination occur in the inference scene, the real-time spectral features of the same target deviate from the reference features during calibration. The recognition system experiences inter-frame jitter in continuous video streams, resulting in inconsistent recognition results and extremely poor cross-scene generalization ability. Summary of the Invention

[0008] The technical problem this invention aims to solve is to address the three shortcomings of existing multispectral fusion recognition methods: the inability to adaptively adjust static fusion weights, insufficient auxiliary modality learning, and the inability to correct dynamic scene spectral drift online. This invention provides a multispectral intelligent recognition method that can perceive illumination conditions in real time during the inference stage, adaptively adjust the fusion weights of each spectral modality, and actively balance the learning progress of each modality during the training stage. It also has online spectral consistency correction capabilities and can achieve cross-domain adaptation without the need for offline recalibration.

[0009] To address the aforementioned technical problems, this invention provides a multispectral intelligent recognition method based on optoelectronic technology, comprising the following steps:

[0010] A one-time geometric calibration is performed on the visible light camera and the thermal infrared camera to obtain the rotation matrix and translation vector between the two cameras. The thermal infrared image is reprojected onto the pixel coordinate system of the visible light image using homography transformation to achieve pixel-level spatial alignment. The aligned image pair is then used as the unified input for subsequent processing.

[0011] The aligned visible light image is input into the illumination quality perception subnetwork. This subnetwork extracts low-level visual statistical features such as brightness distribution, contrast, and local illumination uniformity of the image through multi-scale feature downsampling. After global illumination feature aggregation, the original gating scores of the two modalities are output through gating weight regression. Finally, the visible light modal dynamic gating weights are output after normalization. Dynamic gating weights with thermal infrared modes The constraints are satisfied:

[0012] ;

[0013] in, This represents the fusion contribution weight of visible light modes. This represents the fusion contribution weight of the thermal infrared modes. The sum of the two is always one, ensuring the normalization of the fusion ratio.

[0014] The aligned visible light image and thermal infrared image are input into a dual-stream backbone network, respectively, to extract visible light feature maps. With thermal infrared feature map ,in The number of feature map channels. and These represent the height and width of the feature map space after downsampling, respectively.

[0015] Using the dynamic gating weights output by the illumination quality sensing sub-network, the feature maps output by the two backbone networks are adaptively weighted and fused. The formula for calculating the fused features is as follows:

[0016] ;

[0017] in, This is the joint feature map after adaptive weighted fusion. and The broadcast method is applied to the entire feature map output by the corresponding backbone network. When the scene is in an extremely dark state, Approaching zero As the signal approaches a uniform intensity, the system automatically prioritizes thermal infrared characteristics to suppress interference from visible light noise modes.

[0018] Fusion features Input the recognition output head, complete the target detection or classification task, and output the final recognition result.

[0019] Furthermore, the present invention also includes an online correction step for cross-modal contrast alignment spectral consistency executed during the training phase. This step uses the multispectral joint features extracted under good lighting conditions as positive sample anchors, features under dynamic lighting or occlusion scenarios as queries, and feature sets belonging to different physical targets as negative samples to construct a cross-modal contrast loss function, thereby driving the network to learn stable spectral feature representations that are independent of lighting conditions online.

[0020] The cross-modal contrastive alignment loss function adopts the information-noise contrastive estimation form, and its calculation formula is as follows:

[0021] ;

[0022] in, For cross-modal contrast alignment loss, The number of samples in each training batch. For sample index, For the first Each sample was tested under dynamic lighting or occlusion scenarios. Normalized multispectral joint eigenvectors For the first Each sample was subjected to good lighting conditions. The normalized multispectral joint feature vector, i.e., the positive sample anchor point, For the first The feature vectors of the negative samples come from different physical targets. The number of negative samples. For negative sample index, The cosine similarity function is used. This is a temperature hyperparameter used to control the degree of concentration of the characteristic distribution.

[0023] The final total training loss is a weighted sum of the main loss for the recognition task and the contrast alignment loss:

[0024] ;

[0025] in, For the final total training loss, To identify the main loss of the task, The balance coefficient for the comparison loss is an adjustable hyperparameter used to control the strength of the spectral consistency constraint.

[0026] Furthermore, the present invention also includes a gradient decoupling modality balancing optimization strategy executed during the training phase, wherein in each training iteration step, the gradient norm of the main task loss with respect to the output feature map of the final layer of the visible light backbone network is calculated. Gradient norm of the final layer output feature map of the thermal infrared backbone network As an indicator of gradient contribution for each mode:

[0027] ;

[0028] in, The gradient received by the feature map of the final layer of the visible light backbone network The norm is used to measure the strength of the contribution of the visible light mode to the current training step. The gradient received by the feature map of the final layer of the thermal infrared backbone network The norm is used to measure the contribution intensity of thermal infrared modes. express Norm.

[0029] Based on the monitored modal contributions, the gradient scaling factor for each modal branch is dynamically calculated:

[0030] ;

[0031] in, The scaling factor applied to the backpropagation gradient of the visible light backbone network, with a value range of [value range missing]. , The scaling factor applied to the backpropagation gradient of the thermal infrared backbone network, with a value range of [value range missing]. , This is a numerical stability constant used to prevent division by zero, especially when the visible light mode gradient norm is much larger than the thermal infrared mode gradient norm. Approaching zero, forcibly suppressing visible light-dominated gradient updates. Approaching one ensures that the thermal infrared branch obtains sufficient gradients for adequate training.

[0032] To prevent the scaling factor from fluctuating wildly due to random variations in the gradient norm across batches, an exponential moving average is used to smooth the gradient norm.

[0033] ;

[0034] in, For the first The mode in the th ... The smoothed gradient norm after each training step. For the first Historical values ​​of the smoothed gradient norm for each training step. For the first The original gradient norm is calculated in real time for each training step. This is the smoothing coefficient for the exponential moving average; the larger the value, the smoother the effect. For modal indexing, The current training step is numbered, and the gradient scaling factor is actually calculated using the smoothed gradient norm instead of the original gradient norm to obtain stable training dynamics.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention utilizes a dynamic gating weight mechanism generated by an illumination quality perception sub-network, which can automatically suppress noisy modes and enhance effective modes in extreme illumination scenarios, thereby eliminating noise pollution effects. Compared with existing static fusion methods, it significantly reduces the false negative rate of target detection in extremely dark nighttime scenarios, effectively solving the problem that the performance of multispectral systems in extreme scenarios is even lower than that of single infrared model baselines.

[0037] 2. This invention uses a gradient decoupling modal balance optimization strategy to force the network to fully mine the discrimination information of auxiliary modes such as thermal infrared during the training phase, improve the feature extraction quality of the thermal infrared branch, and enable the multispectral system to give full play to the advantages of multi-sensor collaboration in various scenarios, thus overcoming the fundamental defect of long-term underlearning of auxiliary modes in existing solutions.

[0038] 3. This invention uses a cross-modal contrast alignment framework to correct spectral drift caused by dynamic lighting and occlusion in an online manner, achieving cross-domain adaptive recognition without offline recalibration, which greatly improves the temporal consistency of recognition results in continuous video streams. In industrial inspection scenarios containing dynamic shadows, the inter-frame consistency index of recognition is significantly improved.

[0039] 4. The illumination quality sensing subnetwork of this invention adopts a lightweight depthwise separable convolution design, with its parameter count not exceeding two percent of the parameter count of the backbone network. It meets the real-time inference latency requirements at the edge, with a single-frame processing latency of less than twenty milliseconds, and has good engineering deployment feasibility. Attached Figure Description

[0040] Figure 1 This is the overall reasoning flowchart of the present invention;

[0041] Figure 2 This is a flowchart of the training phase - cross-modal comparison and alignment process of the present invention;

[0042] Figure 3 This is a flowchart of the training phase-gradient decoupling mode balance optimization process of the present invention. Detailed Implementation

[0043] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] I. System Overall Architecture

[0045] The multispectral intelligent recognition method proposed in this invention is based on a deep neural network architecture. The entire system consists of five core functional modules working together: a multispectral input alignment module, an illumination quality perception sub-network, a dual-stream feature extraction backbone network, a cross-modal contrast alignment module, and a gradient decoupling modal balance training controller. Each module works together to support the recognition output head in completing target detection or classification tasks. The data flow direction is from left to right, with the leftmost side containing two inputs: a visible light image stream and a thermal infrared image stream, and the rightmost side containing the recognition result output.

[0046] II. Multispectral Input Alignment Module

[0047] Before system deployment, a one-time geometric calibration is performed on the visible light camera and the thermal infrared camera to obtain the rotation matrix between the two cameras. With translation vector Using standard homography transformation to transform thermal infrared images Reprojection to visible light image In the pixel coordinate system, the two are aligned pixel-level in space to obtain the aligned image pair. As a unified input for all subsequent modules, this module only needs to be calibrated offline once. During the inference phase, the calibration parameters can be directly reused for real-time alignment transformation without repeated calibration.

[0048] III. Illumination Quality Sensing Subnetwork and Dynamic Modal Gating

[0049] The illumination quality sensing subnetwork uses the aligned visible light image. As input, the thermal infrared image itself is radiatively stable and is not used as input for illumination quality assessment. This sub-network adopts a lightweight design, with the number of parameters not exceeding two percent of the number of parameters in the main backbone network. Its specific structure includes the following layers in sequence:

[0050] Three depthwise separable convolutional layers with a stride of 2, outputting 16, 32, and 64 channels respectively. Each layer is followed by a batch normalization layer and a linear rectified activation function to extract low-level visual statistical features such as image brightness distribution, contrast, and local illumination uniformity. The spatial size of the output feature map is halved layer by layer. A global average pooling layer compresses the spatial dimension, resulting in a global illumination description vector encoding the global illumination intensity and quality information of the current scene. Two fully connected layers: the first layer outputs a 32-dimensional parallel linear rectified activation function, and the second layer outputs a 2-dimensional raw gating score for both modalities. A normalized activation layer applies a normalization function to the raw gating score, outputting the dynamic gating weights for the visible light modality. Dynamic gating weights with thermal infrared modes ,satisfy Both are greater than zero.

[0051] After feature extraction is completed, and The feature maps output by the corresponding backbone networks are applied in a broadcast manner, and then fused according to the fusion formula. The system calculates adaptive fusion features to complete dynamic modal gating fusion. In normal daylight scenarios, the visible light gating weight is higher and the visible light features contribute more. In extremely dark night scenarios, the thermal infrared gating weight approaches one. The system relies almost entirely on thermal infrared features for identification and automatically avoids noise interference introduced by degraded visible light modes.

[0052] IV. Framework for Online Correction of Spectral Consistency Across Modal Contrast Alignment

[0053] This framework is executed online during the training phase. The specific implementation steps are as follows:

[0054] At the beginning of each training batch, the illumination quality-aware subnetwork is invoked to evaluate the illumination quality of all samples within the batch, and visible light gating weights are selected. Greater than the preset light quality threshold The samples are used as the anchor point candidate set, where The default value is 0.45. The larger the value, the better the lighting quality of the current scene.

[0055] For the selected anchor point samples, multispectral joint features are extracted through a dual-stream backbone network and a fusion layer. These features are then projected onto a low-dimensional spherical space for contrastive learning using a two-layer multilayer perceptron projector. Normalized anchor feature vector For non-anchor samples (i.e., samples under degraded lighting conditions), the same projection head is used to obtain the query feature vector. .

[0056] Maintain a first-in-first-out (FIFO) negative sample feature queue with a default queue length of 4,096. After each training step, enqueue the anchor features of the current batch and dequeue the oldest features that exceed the queue length. Negative samples are randomly selected from the queue to ensure the diversity and sufficiency of negative samples.

[0057] Based on the cross-modal contrastive loss function in the form of information-noise contrastive estimation, calculate the contrast alignment loss of all query samples in the current batch relative to their corresponding anchor points and the negative sample queue. and the main loss of the identification task Weighted merging constitutes the final training loss. The balance coefficient The default value is 0.1, which is used to control the strength of the spectral consistency constraint.

[0058] After training, the projection head and negative sample queue are only used during the training phase. During the inference phase, the trained dual-stream backbone network and fusion layer are used directly without any online correction calculations, and no inference latency overhead is introduced. Through this framework, the network is forced to learn during the training phase to map the multispectral features of the same physical target under different lighting conditions to similar positions in the feature space, thereby obtaining a stable spectral feature representation capability that is independent of lighting, and achieving cross-domain adaptive recognition during the inference phase.

[0059] V. Modal Equilibrium Optimization Strategy for Gradient Decoupling

[0060] This strategy is executed in each iteration step of the training phase, and the specific implementation steps are as follows:

[0061] During the forward propagation phase, the aligned multispectral images are paired... Inputting into a dual-stream backbone network, we obtain... and The light quality sensing subnetwork dynamically gated and fused to obtain The predicted output is obtained through forward computation by the recognition head, and the loss of the main task is calculated. Alignment loss with contrast , merged to obtain .

[0062] During the initial backpropagation and gradient acquisition phase, the automatic differentiation engine of the deep learning framework is invoked to complete one backpropagation, which is achieved through visible light feature maps. With thermal infrared feature map The registered gradient hooks capture and record the gradient tensors of each modality feature map, and calculate the gradient norm. and And apply an exponential moving average smoothing process to it, so that the smoothed gradient norm is obtained. and It replaces the original value in subsequent calculations.

[0063] During the gradient scaling correction phase, the gradient scaling coefficient is calculated in real time based on the acquired smooth gradient norm. and Clear the gradient cache already calculated in the current step, re-execute backpropagation with the equivalent loss corrected by the scaling factor, and use the optimizer to update the parameters of the entire network based on the corrected gradients, thus completing one training iteration step.

[0064] Through the above strategy, when the gradient contribution of the visible light mode is significantly higher than that of the thermal infrared mode, the system automatically reduces the gradient scaling factor of the visible light branch and increases the gradient scaling factor of the thermal infrared branch. This forces the network to fully explore the discrimination information of the thermal infrared mode during the training phase, avoiding the problem of long-term underlearning of the auxiliary mode. The exponential moving average smoothing mechanism ensures that the scaling factor does not fluctuate drastically due to the random fluctuation of the single batch gradient, thus keeping the training process stable.

[0065] VI. Complete Reasoning Process

[0066] The inference process of this invention does not include any training-related gradient calculation or loss calculation modules. The end-to-end inference process is as follows: a multispectral camera acquires visible light images and thermal infrared images, which undergo homography transformation through an input alignment module. The aligned visible light images are synchronously input into the illumination quality perception sub-network, which outputs dynamic gating weights for each modality in real time. The dual-stream backbone network extracts visible light feature maps and thermal infrared feature maps in parallel. The dynamic weighted fusion layer calculates fusion features based on the gating weights. The recognition output head completes target detection or classification based on the fusion features and outputs the final recognition result. The single-frame processing latency of the entire inference process is less than 20 milliseconds, which meets the requirements for real-time deployment at the edge.

[0067] VII. Typical Application Examples

[0068] In all-weather pedestrian detection scenarios for security monitoring, the method of this invention can be directly adapted to existing dual-light industrial cameras without hardware modifications. It only requires input of pixel-aligned dual video streams to operate. Under normal daylight conditions, the visible light gating weight output by the light quality perception sub-network is approximately 0.82, and the thermal infrared gating weight is approximately 0.18. The system primarily uses visible light features, fully utilizing their rich texture and color details to achieve precise recognition. In extremely dark nighttime scenes, the visible light gating weight is approximately 0.05, and the thermal infrared gating weight is approximately 0.95. The system automatically switches to using thermal infrared features. The dominant recognition mode effectively suppresses interference from low signal-to-noise ratio visible light features. When encountering dynamic shadows or brief occlusions, the illumination-independent feature representation learned during the training phase by the cross-modal contrast alignment framework enables the system to maintain stable recognition results without recalibration, avoiding inter-frame jitter. On the standard multispectral pedestrian detection benchmark dataset, compared with existing static fusion baseline methods, the present invention is expected to reduce the false negative rate by more than 15 percentage points in extreme nighttime scenes. In industrial inspection scenarios containing dynamic shadows, the recognition frame consistency index is expected to be improved by more than 20 percentage points.

[0069] In industrial inspection scenarios, the method of this invention can be used for equipment status monitoring and abnormal target identification on production lines. Dynamic shadows are frequently generated in factories due to robotic arm occlusion, equipment movement, and lighting changes. This invention automatically adjusts the modal fusion ratio at the moment of sudden change in illumination through a dynamic gating mechanism, and ensures sufficient learning of thermal infrared features through a gradient decoupling strategy, thereby ensuring the system's continuous and stable detection of equipment thermal anomalies and occluded targets.

[0070] In autonomous driving perception scenarios, this invention can be applied to pedestrian and obstacle detection using vehicle-mounted dual-light sensors. Drastic changes in illumination caused by cloud changes, tunnel entrances and exits, etc., can be perceived in real time and the modal weights can be adaptively adjusted through the dynamic gating mechanism of this invention. Combined with the spectral consistency achieved by cross-modal comparison alignment, the temporal stability of the recognition results during continuous driving is guaranteed, thereby improving the reliability of safe driving.

[0071] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A multispectral intelligent recognition method based on optoelectronic technology, characterized in that, Includes the following steps: The visible light image and the thermal infrared image, after pixel-level spatial alignment processing, are obtained to form an aligned dual-channel input image. The visible light image is input into the illumination quality perception subnetwork, which evaluates the illumination quality of the current scene in real time and outputs the visible light modal dynamic gating weights. Dynamic gating weights with thermal infrared modes The sum of the two weights is always one and both are greater than zero; The aligned dual-input images are input into a dual-stream backbone network to extract visible light feature maps. With thermal infrared feature map ; The visible light feature map and the thermal infrared feature map are adaptively weighted and fused using the dynamic gating weights to obtain a fused feature map. ,in ; The fused feature map is input into the recognition output head to complete the target detection or classification task and output the recognition result.

2. The method according to claim 1, characterized in that, The structure of the illumination quality sensing subnetwork includes, in sequence: three depthwise separable convolutional layers with a stride of 2, each layer having 16, 32, and 64 output channels respectively; each layer is followed by a batch normalization layer and a linear rectified activation function; a global average pooling layer to compress the spatial dimension into a global illumination description vector; two fully connected layers, the first layer having an output dimension of 32 and a parallel linear rectified activation function, and the second layer having an output dimension of 2; and a normalized activation layer to normalize the original gating scores of the two modalities into dynamic gating weights that satisfy the constraints. and The number of parameters in the illumination quality sensing subnetwork does not exceed two percent of the number of parameters in the dual-stream backbone network.

3. The method according to claim 1, characterized in that, It also includes a cross-modal contrast alignment step performed during the training phase: in each training batch, visible light gating weights greater than a preset illumination quality threshold are selected. The samples are used as anchor point samples. The anchor point samples are extracted, processed by a dual-stream backbone network and a fusion layer, and then projected by a projection head. The normalized anchor point feature vector is used as a positive sample, the corresponding feature vector of the same target under the illumination degradation scene is used as a query sample, and the feature vectors from different physical targets are used as negative samples. The cross-modal contrast alignment loss is calculated, and the cross-modal contrast alignment loss is weighted and merged with the main loss of the recognition task to form the final total training loss.

4. The method according to claim 3, characterized in that, The negative samples are stored in a first-in-first-out feature queue of length 4,096. After each training step, the anchor features of the current batch are enqueued, and the oldest features that exceed the queue length are dequeued. Negative samples are randomly selected from the queue to ensure the diversity of negative samples. After training, the projection head and the first-in-first-out feature queue are only used during the training phase, and no additional computation is introduced during the inference phase.

5. The method according to claim 1, characterized in that, It also includes a gradient decoupling mode balancing optimization step performed during the training phase: in each training iteration, by adjusting the visible light feature map... With thermal infrared feature map The registered gradient hooks capture the gradient tensors of each modality feature map, and calculate the partial derivative tensors of the main task loss with respect to each modality feature map. The norm serves as a gradient contribution index. Based on the ratio of the gradient contributions of each modality, the gradient scaling factor of each modality branch is dynamically calculated. A reduction processing is applied to the backpropagation gradient of the visible light backbone network, and a relative enhancement processing is applied to the backpropagation gradient of the thermal infrared backbone network. The backpropagation is then re-executed with the corrected gradient to update the parameters of the entire network.

6. The method according to claim 5, characterized in that, The gradient contribution index is smoothed using an exponential moving average with a default smoothing coefficient of 0.

9. The smoothed gradient norm is used instead of the original gradient norm in the calculation of the gradient scaling factor to prevent random fluctuations in the gradient norm from causing drastic jitter in the scaling factor and to ensure the stability of the training process.

7. The method according to claim 1, characterized in that, The pixel-level spatial alignment step includes: performing a one-time geometric calibration on the visible light camera and the thermal infrared camera, obtaining the rotation matrix and translation vector between the two cameras, and using homography transformation to reproject the thermal infrared image onto the pixel coordinate system of the visible light image to achieve pixel-level alignment. The calibration parameters are fixed after system deployment and can be directly reused during the inference phase.

8. The method according to claim 1, characterized in that, The method has a single-frame end-to-end processing latency of less than 20 milliseconds, which meets the real-time inference latency requirements at the edge. The illumination quality perception sub-network only uses visible light images as input and does not use thermal infrared images as input for illumination quality assessment.

9. A multispectral intelligent recognition system based on optoelectronic technology, characterized in that, include: The multispectral input alignment module is used to perform pixel-level spatial alignment between visible light images and thermal infrared images, and output the aligned dual-channel input images. The illumination quality perception subnetwork takes the aligned visible light image as input, evaluates the illumination quality of the current scene in real time, and outputs the visible light mode dynamic gating weight and the thermal infrared mode dynamic gating weight. The sum of the two weights is always one and both are greater than zero. A dual-stream feature extraction backbone network, comprising a parallel visible light backbone network and a thermal infrared backbone network, extracts aligned visible light image feature maps and thermal infrared image feature maps, respectively. The dynamic modal fusion module receives the dynamic gating weights output by the illumination quality sensing sub-network and the two feature maps output by the dual-stream backbone network, performs adaptive weighted fusion, and outputs a fused feature map. The recognition output head takes the fused feature map as input, completes the target detection or classification task, and outputs the final recognition result.

10. The system according to claim 9, characterized in that, The system also includes a cross-modal contrast alignment module and a gradient decoupling modal balance training controller. During the training phase, the cross-modal contrast alignment module maintains negative samples through a first-in-first-out negative sample queue, calculates the contrast alignment loss with the features of well-lit frames as positive sample anchors, and drives the network to learn stable spectral feature representations independent of illumination. The gradient decoupling modal balance training controller monitors the gradient contribution of each modal branch in real time through gradient hooks, dynamically adjusts the gradient scaling ratio of each modal branch according to the contribution difference, and forces the network to learn the discrimination information of each spectral mode in a balanced manner. Both the cross-modal contrast alignment module and the gradient decoupling modal balance training controller are involved only in the training phase and do not participate in the calculation during the inference phase.

Images