Intelligent target detection system based on multi-sensor data fusion

Abstract

The invention discloses an intelligent target detection system based on multi-sensor data fusion, particularly relates to the field of intelligent target detection, and comprises a heterogeneous sensor module, a data fusion module and a self-calibration module. According to the invention, at night or under low illumination, infrared thermal imaging dominates target detection, and the data weight of the visible light camera is reduced; under strong electromagnetic interference, the laser radar starts polarization filtering, and the millimeter wave radar performs frequency hopping interference resistance; data fusion adopts a three-level architecture: the first level is based on an improved D-S evidence theory to solve cross-modal data conflicts; in the second stage, aiming at sampling frequency and position difference, a time-space stamp compensation algorithm is used for alignment; the third level uses a bidirectional LSTM neural network of an attention mechanism to distribute weights; when the confidence coefficient of the sensor is low, a particle filter compensation mechanism is started, the deviation of the sensor is corrected, health assessment and predictive maintenance are carried out, historical data are recorded and calibrated, a health degree index is calculated, the remaining service life is predicted, and an alarm is triggered when the confidence coefficient is lower than a threshold value.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent target detection, and more specifically, to an intelligent target detection system based on multi-sensor data fusion. Background Art

[0002] Accurate target detection and identification are crucial in numerous fields, such as autonomous driving, intelligent security, and military reconnaissance. The limitations of traditional single-sensor systems in complex environments are becoming increasingly prominent. For example, in autonomous driving scenarios, relying solely on cameras can struggle to accurately perceive the surrounding environment at night or in inclement weather conditions; relying solely on radar can be inadequate for accurately identifying target details due to insufficient resolution. Faced with complex and changing environments, such as strong electromagnetic interference, low light levels, and high-speed moving targets, single-sensor systems often struggle to meet the requirements for high-precision and high-reliability detection, suffering from low detection accuracy, high false alarm rates, and poor adaptability.

[0003] The data obtained by different sensors are often complementary; multi-sensor data fusion technology came into being. By integrating information from multiple sensors, it can fully utilize the advantages of each sensor and make up for the defects of a single sensor, thereby improving the accuracy, reliability and robustness of target detection. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an intelligent target detection system based on multi-sensor data fusion to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Heterogeneous sensor module: including radar unit, infrared thermal imaging unit, lidar unit and visible light camera unit;

[0007] Data fusion module: adopts three-level fusion architecture:

[0008] Level 1: A sensor-level data conflict resolution module based on the improved DS evidence theory, which implements cross-modal data confidence assessment by introducing a dynamic trust weight factor.

[0009] The second level: multi-scale spatiotemporal alignment of target features, using a spatiotemporal stamp compensation algorithm to eliminate differences in sampling frequency and spatial position between sensors;

[0010] Level 3: A bidirectional LSTM neural network based on the attention mechanism dynamically allocates the contribution weights of different sensors in the feature extraction stage;

[0011] Self-calibration module: When it is detected that the confidence level of a single sensor is continuously lower than the threshold, the sensor deviation compensation mechanism based on particle filtering is activated.

[0012] Preferably, in the heterogeneous sensor module, at night or under low light conditions, the sampling frequency of the infrared thermal imaging module is set to 30Hz, and a non-uniformity correction algorithm is used to suppress thermal noise. The data weight of the visible light camera is dynamically reduced to below 0.2 through a weighted fusion algorithm to ensure that infrared data dominates target detection. If there is strong electromagnetic interference in the environment, the lidar unit immediately enables the polarization filtering mode and combines the Stokes parameter analysis to filter out stray light interference. At the same time, the millimeter wave radar triggers the frequency hopping anti-interference protocol, switches the carrier frequency in a pseudo-random sequence within the 76-81GHz frequency band, and uses space-time adaptive processing technology to suppress sidelobe noise.

[0013] Preferably, in the data fusion module, the first level is constructed based on the improved DS evidence theory, and the cross-modal data conflict problem is solved by a dynamic trust weight factor. In traditional DS theory, the basic probability distribution function has limitations in processing conflicting evidence. The present invention introduces a real-time sensor confidence assessment mechanism; the trust weight of the i-th sensor is defined as a time-varying function, comprehensively considering its signal-to-noise ratio, historical accuracy, and environmental adaptability.

[0014] The second level designs a joint calibration algorithm based on time-space stamp compensation to address differences in sampling frequency (radar 100Hz vs infrared 30Hz) and installation location between sensors:

[0015] For sensor data that arrives with a lag, such as the delay of LiDAR due to the scanning cycle, cubic spline interpolation is used to reconstruct the synchronous time series: LiDAR is represented as laser radar;

[0016]

[0017] Where L(t) represents the estimated value of the LiDAR data obtained by interpolation at time t; K represents the total number of reflection points received by the LiDAR before time t, and φ k Expressed as basis function, a k Expressed as the weight of the kth reflection point; t k It is expressed as the reception time of the kth reflection point, which means the time when the kth reflection point is received by LiDAR; Δt is expressed as the time scale parameter, which is used to normalize the time difference tt k , controls the shape and decay rate of the basis function;

[0018] The bidirectional LSTM neural network of the third-level attention mechanism includes:

[0019] Temporal feature extraction layer: A dilated convolutional layer is used to extract local temporal features of multi-sensor data;

[0020] Cross-modal attention layer: Calculates the feature correlation between different sensor channels through a learnable parameter matrix and generates dynamic attention weights;

[0021] Decision fusion layer: The integrated gated recurrent unit performs final target classification on the weighted feature sequence and outputs structured data including target type, spatial coordinates and motion vector.

[0022] Preferably, in the self-calibration module, after target recognition, the real-time confidence of each sensor is continuously evaluated;

[0023] The confidence calculation adopts the weighted exponential moving average method, and the specific calculation method is:

[0024] H i (t3) = μSNR i (t3)+(1-μ)H i (t3-1)

[0025] Among them, H i (t) represents the confidence at time step t3, μ represents the forgetting factor, SNR i (t3) is the signal-to-noise ratio of the i-th sensor at time step t3, H i (t3-1) confidence at time step t3-1;

[0026] When the H of a sensor i (t3) If the value is lower than the threshold for 5 consecutive cycles, it is judged as confidence abnormality and triggers the online calibration process;

[0027] When the sensor confidence is abnormal, the system starts a dynamic compensation mechanism driven by particle filtering. First, it assumes that there is a systematic deviation between the sensor measurement value and the true value, and then estimates the deviation through particle sampling and updating. The specific steps of the method are as follows:

[0028] Step A1: Initialize M particles Obey Gaussian distribution

[0029] Step A2: Calculate the weight of each particle;

[0030] Step A3: Resampling to avoid particle degradation and retain high-weight particles;

[0031] Step A4: Output bias estimate;

[0032] The output deviation estimate is fed back to the sensor data stream to calculate the corrected observation value. After sensor deviation compensation, the sensor health assessment and predictive maintenance are performed. The calibration history data of each sensor is recorded over a long period of time, the health index is calculated, and the remaining service life is predicted. The LSTM network is used to learn the sensor degradation trend. The input includes: historical deviation series and environmental stress data; the output is the probability distribution of RUL. When the predicted service life falls below the threshold, a maintenance alarm is triggered.

[0033] The technical effects and advantages of the present invention are as follows:

[0034] This invention uses infrared thermal imaging to prioritize target detection in intermittent low-light conditions, downweighting visible light camera data. Under strong electromagnetic interference, the lidar uses polarization filtering, while the millimeter-wave radar uses frequency hopping for interference mitigation. For high-speed moving targets, radar and lidar collaborate to track them. Data fusion employs a three-level architecture: the first level uses a modified DS evidence theory to resolve cross-modal data conflicts; the second level uses a time-space stamp compensation algorithm to align sampling frequency and position differences; and the third level uses a bidirectional LSTM neural network with an attention mechanism to assign weights. A self-calibration module activates a particle filter compensation mechanism to correct sensor bias when sensor confidence is low. It then performs health assessment and predictive maintenance, records calibration history, calculates a health index, and predicts remaining service life. Alarms are triggered when the remaining service life falls below a threshold. Through multimodal sensor fusion, an efficient data fusion architecture, sensor bias compensation, and a self-calibration mechanism, this invention significantly improves the accuracy and reliability of target detection and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a module schematic diagram of the present invention.

[0036] Figure 2 Schematic diagram of the bidirectional LSTM neural network of the present invention.

[0037] Figure 3 Schematic diagram of the self-calibration module flow of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] See also Figure 1 As shown, the present invention provides an intelligent target detection system based on multi-sensor data fusion, including a heterogeneous sensor module, a data fusion module, and a self-calibration module.

[0040] Heterogeneous sensor module: including radar unit, infrared thermal imaging unit, lidar unit and visible light camera unit;

[0041] In the heterogeneous sensor module, at night or in low light conditions, the infrared thermal imaging module's sampling frequency is set to 30Hz. A non-uniformity correction algorithm is used to suppress thermal noise, and a weighted fusion algorithm dynamically reduces the visible light camera's data weight to below 0.2, ensuring that infrared data dominates target detection. If strong electromagnetic interference is present, the lidar unit immediately activates polarization filtering mode, combining Stokes parameter analysis to filter out stray light interference. Simultaneously, the millimeter-wave radar triggers a frequency-hopping anti-interference protocol, switching carrier frequencies in a pseudo-random sequence within the 76-81GHz band, supplemented by space-time adaptive processing technology to suppress sidelobe noise. When a high-speed moving target is detected, the radar-lidar collaborative tracking mode is automatically activated. The millimeter-wave radar provides high-update radial velocity measurements, while the lidar extracts the target's three-dimensional features through super-resolution point cloud clustering. Both data are fed into a hybrid filtering algorithm: linear motion components are processed by a Kalman filter, while nonlinear maneuvers are predicted by a volumetric particle filter. A dynamic weight allocation mechanism optimizes sensor contributions in real time.

[0042] Data fusion module: adopts three-level fusion architecture:

[0043] Level 1: A sensor-level data conflict resolution module based on the improved DS evidence theory, which implements cross-modal data confidence assessment by introducing a dynamic trust weight factor.

[0044] The second level: multi-scale spatiotemporal alignment of target features, using a spatiotemporal stamp compensation algorithm to eliminate differences in sampling frequency and spatial position between sensors;

[0045] Level 3: A bidirectional LSTM neural network based on the attention mechanism dynamically allocates the contribution weights of different sensors in the feature extraction stage;

[0046] In the data fusion module, the first level is constructed based on the improved DS evidence theory, and the cross-modal data conflict problem is resolved through a dynamic trust weight factor. In traditional DS theory, the basic probability distribution function has limitations in processing conflicting evidence. The present invention introduces a real-time sensor confidence assessment mechanism. The trust weight of the i-th sensor is defined as a time-varying function, taking into account its signal-to-noise ratio, historical accuracy, and environmental adaptability. The specific calculation method of the trust weight is:

[0047]

[0048] Among them, w i(t) represents the trust weight of the i-th sensor at time t, α, β, γ represent the adjustment coefficients, SNR i (t) represents the signal-to-noise ratio of the i-th sensor at time t, N represents the number of historical data points, and f env (T, EMI) represents the environmental adaptation function of temperature and electromagnetic interference. It is expressed as an indicator function, indicating whether the true value of the k-th sample is consistent with the predicted value of the i-th sensor. If they are consistent, it is 1, otherwise it is 0;

[0049] EMI stands for electromagnetic interference;

[0050] The conflicting evidence of low-confidence sensors is suppressed by the weight factor, and the calculation method of the synthesized basic probability distribution is as follows:

[0051]

[0052] Among them, m(A) represents the synthesized BPA, which means the degree of confidence in proposition A; A represents a proposition or hypothesis, which is usually the target of basic probability distribution in DS evidence theory, m i (A j ) is represented by the i-th sensor pair Proposition A j The basic probability distribution, w i (t) is the weight factor of the i-th sensor at time t, ∩A j =A means that the intersection of all sensors is A, that is, all sensors support proposition A; The intersection of all sensors is an empty set, indicating that there are conflicts between sensors;

[0053] BPA stands for Basic Probability Allocation;

[0054] The second level designs a joint calibration algorithm based on time-space stamp compensation to address differences in sampling frequency (radar 100Hz vs infrared 30Hz) and installation location between sensors:

[0055] For sensor data that arrives with a lag, such as the delay of LiDAR due to the scanning cycle, cubic spline interpolation is used to reconstruct the synchronous time series: LiDAR is represented as laser radar;

[0056]

[0057] Where L(t) represents the estimated value of the LiDAR data obtained by interpolation at time t; K represents the total number of reflection points received by the LiDAR before time t, and φ k Expressed as basis function, a k Expressed as the weight of the kth reflection point; tk It is expressed as the reception time of the kth reflection point, which means the time when the kth reflection point is received by LiDAR; Δt is expressed as the time scale parameter, which is used to normalize the time difference tt k , controls the shape and decay rate of the basis function;

[0058]

[0059] Among them, S represents the transformation matrix from the sensor coordinate system to the world coordinate system, S0 represents the change matrix to be optimized, and P LiDAR Represented as the three-dimensional coordinates of the target point measured by the LiDAR sensor, P Radar Indicates the three-dimensional coordinates of the target point measured by the Radar sensor. It is expressed as solving the optimal transformation matrix by minimizing the objective function;

[0060] In the optimization process, S is an unknown variable, which is solved by minimizing the objective function;

[0061] Radar represents the radar sensor

[0062] The transformation matrix S contains rotation and translation parameters and is automatically updated every 5 seconds through feature point matching;

[0063] The bidirectional LSTM neural network of the third-level attention mechanism includes:

[0064] Temporal feature extraction layer: A dilated convolutional layer is used to extract local temporal features of multi-sensor data;

[0065] The temporal feature extraction layer captures multi-scale local features through dilated convolution. The raw sensor data is first input into the dilated causal convolution layer to extract local temporal patterns with different dilation coefficients.

[0066] The radar signal is convolved using three layers of expansion coefficients d = 1, 2, and 4 to capture short-term speed changes (such as sudden braking), medium-range maneuvers (such as turns), and long-range trajectory trends. The convolution kernel output dimension is calculated as follows:

[0067] F(t)=ReLU(W dil *X radar (td, t)+b dil )

[0068] Among them, F(t) represents the output eigenvalue, that is, the eigenvalue after dilated convolution and ReLU activation at time t; W dil Expressed as the weight matrix of the dilated convolution, X radarIt is represented as a radar data segment from time td to time t, * represents the hole convolution operation, which avoids the compression of the timing information of high-speed targets by ordinary convolution; b dil It is represented as the bias term of the dilated convolution; ReLU is represented as the activation function;

[0069] Infrared and visible light data share the same set of convolutional layers but have independent parameters. 2D dilated convolution is used to extract spatial-temporal joint features, such as target thermal radiation changes and texture motion.

[0070] The output of this stage is the local temporal feature tensor of each of the multiple sensors, with the dimension of [time step × number of channels × feature dimension];

[0071] Cross-modal attention layer: Calculates the feature correlation between different sensor channels through a learnable parameter matrix and generates dynamic attention weights;

[0072] The temporal features of each sensor are input into the cross-modal attention layer, and the inter-modal correlation is calculated through the learnable parameter matrix;

[0073] Q i =W1F i

[0074] Among them, Q i is represented as the query vector of the i-th sensor, W1 is represented as the weight matrix used to generate the query vector, and F i Represented as the feature sequence of the i-th sensor

[0075] K j =W2F j

[0076] Among them, K j It is represented as the key vector of the j-th sensor, W2 is used to generate the weight matrix of the key vector, F j Represented as the feature sequence of the jth sensor;

[0077] V j =W3F j

[0078] Among them, V j is represented as the value vector of the jth sensor, W3 is represented as the weight matrix for generating the value vector, and F j Represented as the feature sequence of the jth sensor;

[0079] Calculate the attention score of sensor i to j. The calculation method is as follows:

[0080]

[0081] Among them, δ ij Expressed as the attention score of sensor i to j, Qi Denotes the query vector of the i-th sensor, Denote the transpose of the key vector of sensor j, g k Expressed as key vector dimensions,

[0082] Weighted aggregation of multi-sensor value vectors to obtain the features after modal interaction. The calculation method of the features after modal interaction is as follows:

[0083]

[0084] The feature of sensor i after modal interaction, δ ij Expressed as the attention score of sensor i to j, V j It is represented as the value vector of sensor j;

[0085] For example, when the target is partially obscured by trees, the infrared feature’s attention weight to the visible light increases, using thermal radiation to complete the missing contour information;

[0086] The output of this stage is the context-aware features of each sensor, which integrates the complementary information of other modalities.

[0087] Decision fusion layer: The integrated gated recurrent unit performs final target classification on the weighted feature sequence and outputs structured data including target type, spatial coordinates, and motion vectors;

[0088] The decision fusion layer inputs the enhanced multimodal feature sequence into the gated recurrent unit for time series modeling and classification;

[0089] The calculation method of the update gate is as follows:

[0090]

[0091] Among them, z t It is represented as the update gate, W4 is represented as the weight matrix of the update gate, and h t-1 Represents the hidden state at the previous moment, It is represented as the enhanced feature, and σ is represented as the activation function;

[0092]

[0093] Among them, r t It is represented as the reset gate, W5 is represented as the weight matrix of the reset gate, and h t-1 Represents the hidden state at the previous moment, It is represented as the enhanced feature, and σ is represented as the activation function;

[0094]

[0095] Among them, ht is represented as the candidate hidden state, W6 is represented as the weight matrix of the candidate hidden state, r t Represented as reset gate, h t-1 Represents the hidden state at the previous moment, It is represented as the enhanced feature, and tanh is represented as the activation function, that is, the hyperbolic tangent function;

[0096] h t0 =(1-z t )⊙h t-1 +z t ⊙h t

[0097] Among them, h t0 Denote as the new hidden state, z t Represented as update gate, h t-1 Represents the hidden state at the previous moment, h t Denotes the candidate hidden state, and ⊙ denotes element-wise multiplication;

[0098] The bidirectional GRU further fuses the forward and backward context to capture the causality of target motion;

[0099] Classify the target type based on the new hidden state h t0 The fully connected layer + Softmax output category probability is calculated as follows:

[0100] P class =softmax(W7h t0 +b1)

[0101] Among them, P class Expressed as the output category probability, W7 fully connected layer weight matrix, h t0 Denotes the new hidden state, b1 denotes the bias vector of the fully connected layer, and softmax denotes the Softmax function, which is used to convert the output of the fully connected layer into a probability distribution;

[0102] The linear layer is used to predict the target center point (x, y, z) and the bounding box size. The calculation method is as follows:

[0103] B t =W8h t0 +b2

[0104] Among them, B t Represents the output target center point and bounding box size, W8 represents the linear layer weight matrix, which is used to map the hidden state to the target center point and bounding box size; b2 represents the linear layer bias vector, which is used to adjust the output offset;

[0105] The velocity and acceleration are calculated by time-difference hidden state. The velocity calculation method is as follows:

[0106]

[0107] in, Expressed as the velocity at time step t2, h t0 is represented as the new hidden state, It represents the hidden state at the previous moment, Δt0 represents the time step, and represents the time interval between adjacent time steps;

[0108] The calculation method of acceleration is as follows:

[0109]

[0110] in, Expressed as the acceleration at time step t, Expressed as the time step velocity, It is represented as the velocity at the previous moment, Δt0 is represented as the time step, and it represents the time interval between adjacent time steps;

[0111] The final output is in tuple form, which is as follows:

[0112]

[0113] Among them, P class Expressed as the output category probability, B t Represented as the output target center point and bounding box size, v t Expressed as the velocity at time step t, a t Expressed as the acceleration at time step t;

[0114] For example, for a car that suddenly changes lanes, the output is:

[0115] ("Vehicle", [x=12.3m, y=4.5m, z=0m], [v x =8.2m / s, v y

[0116] =3.1m / s], [a x =0.5m / s 2 , a y =2.3m / s 2 ])

[0117] Vehicle means vehicle;

[0118] Self-calibration module: When it is detected that the confidence level of a single sensor is continuously lower than the threshold, the sensor deviation compensation mechanism based on particle filtering is activated;

[0119] In the self-calibration module, after target recognition, the real-time confidence of each sensor is continuously evaluated;

[0120] The confidence calculation adopts the weighted exponential moving average method, and the specific calculation method is:

[0121] H i (t3) = μSNR i (t3)+(1-μ)H i (t3-1)

[0122] Among them, H i (t) represents the confidence at time step t3, μ represents the forgetting factor, SNR i (t3) is the signal-to-noise ratio of the i-th sensor at time step t3, H i (t3-1) confidence at time step t3-1;

[0123] When the H of a sensor i (t3) If the value is lower than the threshold for 5 consecutive cycles, it is judged as confidence abnormality and triggers the online calibration process;

[0124] If the confidence level continues to decrease (e.g., slope < -0.1 / cycle), it is predicted that the sensor has hardware aging or occlusion problems, and an early warning is issued;

[0125] When the sensor confidence is abnormal, the system starts the dynamic compensation mechanism driven by particle filtering. First, it assumes that there is a systematic deviation between the sensor measurement value and the true value. The calculation method of the sensor measurement value is as follows:

[0126] b t =c t +τ t +p t

[0127] Among them, b t Expressed as sensor measurement value, c t Expressed as the true value, τ t Denotes the time-varying bias to be estimated, p t is expressed as observation noise;

[0128] Then, the output deviation is estimated by particle sampling and updating. The specific steps of the method are as follows:

[0129] Step A1: Initialize M particles Obey Gaussian distribution

[0130] in, The state of the e-th particle, M represents the number of particles, is represented as the mean of the previous time step, φ 2Expressed as the variance of the Gaussian distribution;

[0131] Step A2: Calculate the weight of each particle. The calculation method is as follows:

[0132]

[0133] Among them, w t (e) represents the weight of the e-th particle, Expressed as the observation likelihood function, it means that in the particle state Observed b t The probability of ; R represents the variance of observation noise, Represented as the state of the e-th particle, b t Expressed as sensor measurement value, c t Represented as true value;

[0134] Step A3: Resampling to avoid particle degradation and retain high-weight particles;

[0135] Step A4: Output deviation estimate, the calculation method is as follows:

[0136]

[0137] Among them, τ t0 Expressed as the output deviation estimate, w t (e) represents the weight of the e-th particle, The state of the e-th particle, M represents the number of particles.

[0138] The output deviation estimate is fed back to the sensor data stream to calculate the corrected observation value. The calculation method is as follows:

[0139]

[0140] in, Represented as the corrected observation value, that is, the sensor data after bias correction; b t Expressed as sensor measurement value, τ t0 Expressed as an output deviation estimate;

[0141] For example, the millimeter-wave radar has a ranging deviation of +0.5m due to temperature drift, and the particle filter estimates τ t0 =0.48m, and the error is reduced to ±0.05m after correction; the infrared thermal imager has a radiation deviation due to lens contamination, which is restored to the calibration value after compensation;

[0142] After sensor deviation compensation, the sensor is evaluated for health and undergoes predictive maintenance. The calibration history of each sensor is recorded over a long period of time, and the health index is calculated. The calculation method is as follows:

[0143] HI=1-ω1E-ω2G-ω3U

[0144] Where HI is the health index, ω1, ω2, and ω3 are weight factors, E is the sensor operating time attenuation factor, G is the cumulative root mean square of historical deviations, and U is the frequency of recent calibration triggering.

[0145] The remaining useful life prediction is then performed using an LSTM network to learn the sensor degradation trend. The input includes historical deviation series and environmental stress data. The output is a probability distribution of RUL. When the predicted life falls below a threshold, a maintenance alarm is triggered.

[0146] For example, an autonomous vehicle driving in a dusty environment:

[0147] Problem detection: The LiDAR point cloud density decreased due to dust adhesion, and the confidence level dropped from 0.8 to 0.4 within 3 cycles;

[0148] Online compensation: Particle filtering is enabled to estimate the ranging deviation of the lidar using radar and visual data. After correction, the multi-sensor fusion positioning error is reduced from 1.2m to 0.3m.

[0149] Health management: The system records this abnormality, the HI index drops by 5%, and a "LiDAR lens cleaning recommendation" is prompted.

[0150] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent target detection system based on multi-sensor data fusion, characterized in that: include: Heterogeneous sensor module: including radar unit, infrared thermal imaging unit, lidar unit and visible light camera unit; Data fusion module: adopts three-level fusion architecture: Level 1: A sensor-level data conflict resolution module based on the improved DS evidence theory, which implements cross-modal data confidence assessment by introducing a dynamic trust weight factor. The second level: multi-scale spatiotemporal alignment of target features, using a spatiotemporal stamp compensation algorithm to eliminate differences in sampling frequency and spatial position between sensors; Level 3: A bidirectional LSTM neural network based on the attention mechanism dynamically allocates the contribution weights of different sensors in the feature extraction stage; Self-calibration module: When it is detected that the confidence level of a single sensor is continuously lower than the threshold, the sensor deviation compensation mechanism based on particle filtering is activated.

2. The intelligent target detection system based on multi-sensor data fusion according to claim 1, characterized in that: In the heterogeneous sensor module, the sampling frequency of the infrared thermal imaging module is set to 30Hz at night or in low light conditions. A non-uniformity correction algorithm is used to suppress thermal noise, and a weighted fusion algorithm is used to dynamically reduce the data weight of the visible light camera to below 0.

2. When a high-speed moving target is detected, the radar-lidar collaborative tracking mode is automatically activated. The millimeter-wave radar provides radial velocity measurements with a high update rate, and the lidar extracts the target's three-dimensional features through super-resolution point cloud clustering. The data of both are input into a hybrid filtering algorithm - the linear motion component is processed by the Kalman filter, and the nonlinear maneuver is predicted by the volumetric particle filter. The sensor contribution is optimized in real time through a dynamic weight allocation mechanism.

3. The intelligent target detection system based on multi-sensor data fusion according to claim 1, characterized in that: In the data fusion module, the first level is constructed based on the improved DS evidence theory, defining the trust weight of the i-th sensor as a time-varying function, taking into account its signal-to-noise ratio, historical accuracy, and environmental adaptability. The specific calculation method of the trust weight is: Among them, w i (t) represents the trust weight of the i-th sensor at time t, α, β, γ represent the adjustment coefficients, SNR i (t) represents the signal-to-noise ratio of the i-th sensor at time t, N represents the number of historical data points, and f env (T, EMI) represents the environmental adaptation function of temperature and electromagnetic interference. It is expressed as an indicator function, which indicates whether the true value of the k-th sample is consistent with the predicted value of the i-th sensor. If they are consistent, it is 1, otherwise it is 0.

4. The intelligent target detection system based on multi-sensor data fusion according to claim 1, characterized in that: The second stage designs a joint calibration algorithm based on time-space stamp compensation to address differences in sampling frequency and installation location between sensors. For delayed sensor data and the delay caused by the lidar scanning cycle, cubic spline interpolation is used to reconstruct the synchronized time series. Where L(t) represents the estimated value of the LiDAR data obtained by interpolation at time t; K represents the total number of reflection points received by the LiDAR before time t, and φ k Expressed as basis function, a k Expressed as the weight of the kth reflection point; t k It is expressed as the reception time of the kth reflection point, which means the time when the kth reflection point is received by LiDAR; Δt is expressed as the time scale parameter, which is used to normalize the time difference tt k , which controls the shape and decay rate of the basis function.

5. The intelligent target detection system based on multi-sensor data fusion according to claim 1, characterized in that: The bidirectional LSTM neural network of the third-level attention mechanism includes: Temporal feature extraction layer: A dilated convolutional layer is used to extract local temporal features of multi-sensor data; Cross-modal attention layer: Calculates the feature correlation between different sensor channels through a learnable parameter matrix and generates dynamic attention weights; Decision fusion layer: The integrated gated recurrent unit performs final target classification on the weighted feature sequence and outputs structured data including target type, spatial coordinates and motion vector.

6. The intelligent target detection system based on multi-sensor data fusion according to claim 5, characterized in that: The temporal feature extraction layer captures multi-scale local features through dilated convolution. The raw sensor data is first input into the dilated causal convolution layer to extract local temporal patterns with different dilation coefficients. The radar signal uses three layers of convolution with expansion coefficients d = 1, 2, and 4 to capture short-term speed mutations, medium-range maneuvers, and long-range trajectory trends respectively. The convolution kernel output dimension is calculated as follows: F(t)=ReLU(W dil *X radar (t-d,t)+b dil ) Among them, F(t) represents the output eigenvalue, that is, the eigenvalue after dilated convolution and ReLU activation at time t; W dil Expressed as the weight matrix of the dilated convolution, X radar It is represented as a radar data segment from time td to time t, * represents the hole convolution operation, which avoids the compression of the timing information of high-speed targets by ordinary convolution; b dil It is represented as the bias term of the dilated convolution; ReLU is represented as the activation function.

7. The intelligent target detection system based on multi-sensor data fusion according to claim 5, characterized in that: The temporal features of each sensor are input into the cross-modal attention layer, and the inter-modal correlation is calculated through the learnable parameter matrix; Weighted aggregation of multi-sensor value vectors to obtain the features after modal interaction. The calculation method of the features after modal interaction is as follows: The feature of sensor i after modal interaction, δ ij Expressed as the attention score of sensor i to j, V j is represented as a vector of values ​​for sensor j.

8. The intelligent target detection system based on multi-sensor data fusion according to claim 5, characterized in that: The decision fusion layer inputs the enhanced multimodal feature sequence into the gated recurrent unit for time series modeling and classification; Classify the target type based on the new hidden state h t0 The fully connected layer + Softmax output category probability is calculated as follows: P class =softmax(W7h t0 +b1) Among them, P class Expressed as the output category probability, W7 fully connected layer weight matrix, h t0 Denotes the new hidden state, b1 denotes the bias vector of the fully connected layer, and softmax denotes the Softmax function, which is used to convert the output of the fully connected layer into a probability distribution; Use the linear layer to predict the target center point (x, y, z) and the bounding box size. The calculation method is as follows: <h2 style=";text-align:left;direction:ltr">B<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> =W8h<h2 style=";text-align:left;direction:ltr"> t0 <h2 style=";text-align:left;direction:ltr"> +b2 Among them, B t Represents the output target center point and bounding box size, W8 represents the linear layer weight matrix, which is used to map the hidden state to the target center point and bounding box size; b2 represents the linear layer bias vector, which is used to adjust the output offset; The velocity and acceleration are calculated through the time difference hidden state, and the final output is in the form of a tuple. The specific form of the tuple is:

9. The intelligent target detection system based on multi-sensor data fusion according to claim 6, characterized in that: In the self-calibration module, after target recognition, the real-time confidence of each sensor is continuously evaluated; The confidence calculation adopts the weighted exponential moving average method, and the specific calculation method is: H i (t3)=μSNR i (t3)+(1-μ)H i (t3-1) Among them, H i (t) represents the confidence at time step t3, μ represents the forgetting factor, SNR i (t3) is the signal-to-noise ratio of the i-th sensor at time step t3, H i (t3-1) confidence at time step t3-1; When the sensor confidence is abnormal, the system activates a dynamic compensation mechanism driven by particle filtering. It assumes that there is a systematic deviation between the sensor measurement value and the true value, and then uses particle sampling and updating to estimate the output deviation. The output deviation estimate is fed back to the sensor data stream to calculate the corrected observation value. After sensor deviation compensation, the sensor is evaluated for health and undergoes predictive maintenance. The calibration history data of each sensor is recorded for a long time, the health index is calculated, and the remaining service life is predicted. The LSTM network is used to learn the sensor degradation trend. The input includes: historical deviation series and environmental stress data; the output is the probability distribution of RUL. When the predicted service life is lower than the threshold, a maintenance alarm is triggered.

Images