Target identification method for multi-sensor data fusion

Abstract

The invention discloses a target identification method for multi-sensor data fusion, particularly relates to the technical field of data fusion, and comprises a dynamic weight fusion module, a double-branch neural network and an abnormal sensing compensation system. Data are synchronously collected through a visible light camera, a thermal infrared imager and a millimeter wave radar, and a standardized feature map is generated through sensor specificity preprocessing; the dynamic weight fusion module generates an adaptive weight matrix based on the real-time confidence score and the environmental parameters, emphasizes infrared data when illumination suddenly changes, and improves intelligent distribution of radar weights in a rain and fog environment; the combined feature map after weight fusion is input into a double-branch neural network, a channel self-calibration branch suppresses interference noise, and a target category and coordinates are output after residual connection optimization; when the recognition confidence is insufficient, the abnormal compensation system starts visible light Wiener filtering restoration, generative adversarial network infrared compensation and radar multi-frame accumulation algorithms, and the system reliability is ensured when a single sensor fails.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion, more particularly, to a multi-sensor data fusion target recognition method. BACKGROUND

[0002] With the increasing demand for environmental perception accuracy in the fields of autonomous driving and intelligent security, single sensors (such as cameras and radars) have limitations in complex scenarios: visible light cameras fail in low light or rainy and foggy weather, infrared sensors have difficulty in distinguishing objects with similar heat sources, and millimeter wave radars lack target texture information. To overcome the above-mentioned defects, the industry gradually adopts multi-sensor fusion technology to improve the recognition robustness through data complementation of heterogeneous sensors such as visible light, infrared, and radar. However, existing fusion schemes mostly focus on simple integration at the data or decision level, and fail to effectively solve the core problems of multi-source data space-time alignment error and poor environmental adaptability, which restricts the practical landing of high-precision target recognition.

[0003] The current mainstream fusion method has three major defects:

[0004] Static fusion mechanism defects: traditional weighted average method, D-S evidence theory and other methods use fixed weights, which cannot adjust the contribution of sensors according to dynamic environments such as sudden changes in light and interference from rain and fog, resulting in a sharp drop in recognition accuracy in extreme environments.

[0005] Insufficient feature compatibility: when directly fusing raw data of heterogeneous sensors (such as images and point clouds), feature conflicts are easily caused due to differences in data dimensions and physical meanings, reducing the efficiency of neural network feature extraction.

[0006] Absence of abnormality processing: existing schemes do not establish a sensor fault response mechanism, and when some sensors fail due to obstruction or electromagnetic interference, the system still uses low-quality data for fusion, causing serious deviation in the recognition result. For example, the Tesla Autopilot system has repeatedly misidentified stationary vehicles in strong glare scenes, the root cause of which is the failure to dynamically reduce the weight of camera data.

[0007] The present application aims to break through the technical bottlenecks of the prior art and solve the following core problems:

[0008] Poor dynamic environmental adaptability: establish a real-time correlation model between sensor confidence and environmental parameters to achieve autonomous optimization and adjustment of fusion weights;

[0009] Inefficient heterogeneous feature fusion: design a cross-modal feature alignment mechanism to eliminate fusion conflicts of visible light texture, infrared temperature, and radar spatial features;

[0010] Abnormal sensor data pollution: develop a damaged data compensation algorithm based on generative adversarial networks (GAN) to maintain system reliability when some sensors fail.

[0011] Through the above technical innovation, the accuracy and robustness of target identification in complex environments are improved, and the needs of key scenes such as L4 level automatic driving are met.

[0012] Therefore, a target identification method based on multi-sensor data fusion is proposed to solve the above problems. SUMMARY

[0013] In order to overcome the above-mentioned defects of the prior art, embodiments of the present application provide a target identification method based on multi-sensor data fusion to solve the problems raised in the above background art.

[0014] To achieve the above object, the present application provides the following technical scheme: a target identification method based on multi-sensor data fusion, comprising the following operation processes: synchronously collecting target region data through a visible light camera, an infrared thermal imager and a millimeter wave radar; performing adaptive histogram equalization on the visible light image and extracting a texture feature map, performing non-uniformity correction on the infrared image and extracting a temperature distribution feature map, and performing dynamic clustering denoising on the millimeter wave radar point cloud and extracting a spatial position feature map; inputting the three types of feature maps into a dynamic weight fusion module, which generates confidence scores of each sensor based on real-time calculated signal-to-noise ratio, data integrity and environmental interference intensity, simultaneously obtains illumination intensity, rain and fog level and electromagnetic interference intensity feedback from an environmental parameter perception unit, and outputs an adaptive fusion weight matrix through a weight distribution function; generating a joint feature map by weighted fusion of the multi-source feature maps according to the weight matrix; inputting the joint feature map into a dual-branch deep convolutional neural network, wherein the spatial attention branch strengthens the target region features, and the channel self-calibration branch suppresses redundant feature channels, and the dual-branch outputs are output by a classification regression head to output target category and position coordinates after splicing.

[0015] Further, the weight distribution function adopts the following mathematical expression: Wi=α·Ci+β·Ei, wherein Wi represents the fusion weight value of the i-th sensor, Ci represents the normalized confidence score and satisfies 0≤Ci≤1, Ei represents an environmental adaptation coefficient and is calculated by the formula Ei=1-0.3L-0.5R-0.2E from the illumination intensity mapping value L, the rain and fog level mapping value R and the electromagnetic interference intensity mapping value E, and α and β are trainable parameters and satisfy the constraint conditions α+β=1 and 0.4≤α≤0.8.

[0016] Further, the operation of the dynamic weight fusion module includes a reinforcement learning optimization mechanism: a Q learning model is constructed with recognition accuracy as the reward function and weight distribution parameters as the action space; after each target recognition task is completed, the reward value is calculated according to the difference between the recognition result confidence and the true label; the Q value table is updated through the Bellman equation to dynamically adjust the parameters a and β in the weight distribution function; when the environmental parameters mutate beyond the threshold, the greedy strategy is started to explore the optimal weight combination.

[0017] Further, the dual-branch deep convolutional neural network includes the following specific structures: the spatial attention branch uses a dilated convolution layer with a dilated rate of [2, 4, 6] to extract multi-scale features, and generates a spatial weight mask through a Sigmoid activation function; the channel self-calibration branch compresses the spatial dimension through a global average pooling layer, generates a channel weight vector through two fully connected layers and a ReLU activation function; after the Hadamard product operation of the spatial weight mask and the channel weight vector, residual connection is performed with the original feature map; the classification regression head uses a parallel Softmax classifier and a bounding box regressor, where the bounding box regressor uses a Smooth L1 loss function to optimize coordinate prediction.

[0018] Further, it also includes a sensor anomaly processing procedure: when the output recognition confidence is lower than the preset threshold 0.7, the sensor health state diagnosis is started: the image blurriness index is detected for the visible light camera, if the blurriness exceeds 60, the image restoration based on Wiener filtering is started; the temperature gradient variance is detected for the infrared thermal imager, if the variance is lower than 0.05, the generative adversarial network compensation data is enabled, the generative adversarial network takes normal sensor data as input, generates infrared feature maps through a generator containing 3 residual blocks, and verifies data validity through a discriminator containing 5 convolution layers; the point cloud density is detected for the millimeter wave radar, if the density drops by 40%, the multi-frame data accumulation compensation algorithm is triggered.

[0019] Further, the generative adversarial network adopts a conditional training architecture: the generator input layer receives the spliced data of visible light texture feature maps and millimeter wave spatial position feature maps, reduces the dimension through a 3x3 convolution layer, and then inputs into a residual block group, each residual block contains a batch normalization layer, a LeakyReLU activation function and a skip connection; the discriminator input layer receives real infrared feature maps and generated feature maps, down-samples through 5 convolution layers with a step of 2, and finally outputs the authenticity probability through a Sigmoid function; the loss function uses an improved Wasserstein distance: where z is the generator input noise vector, x is the real infrared data, x^ is the linear interpolation of real and generated data, and λ is set to the gradient penalty coefficient 10.

[0020] Further, the application of the method in the automatic driving scene includes: the target category recognition covers vehicles (including cars, trucks, and buses), pedestrians (including adults and children), two-wheeled vehicles (including bicycles and motorcycles), traffic signs (including traffic lights and road signs), and obstacles (including cone barrels and rubble piles); the position information output is polar coordinate system data centered on the ego vehicle, including target azimuth angle θ (accuracy ±0.5°) and distance d (accuracy ±0.1 m); when the rain and fog level exceeds level 3, the weight of the infrared sensor is automatically increased to more than 0.8, and when the night illumination is lower than 10 lux, the weight of the millimeter wave radar is increased to more than 0.7.

[0021] Technical effects and advantages of the present application:

[0022] The core advantage of the present application lies in the synergistic effect of the dynamic weight fusion mechanism, the dual-branch neural network architecture, and the abnormal sensor compensation system, which improves the reliability of target recognition in complex environments:

[0023] The dynamic weight fusion mechanism calculates the sensor confidence score (based on signal-to-noise ratio and data integrity) and the environmental adaptation coefficient (related to illumination, rain and fog, and electromagnetic interference) in real time, dynamically adjusts the fusion weight using trainable parameters, automatically increases the weight of the infrared sensor in strong glare environment, and focuses on millimeter wave radar data in rainy and foggy weather, thereby fundamentally solving the environmental adaptability problem of fixed weight fusion.

[0024] The spatial attention branch of the dual-branch neural network uses dilated convolution to strengthen the target region feature expression (effectively dealing with partially occluded targets), and the channel self-calibration branch suppresses interference noise such as rain and snow through feature channel screening, and the two are connected through residual connection to optimize feature representation and improve heterogeneous feature compatibility.

[0025] The abnormal sensor compensation system is started when the recognition confidence is insufficient, uses Wiener filter to restore texture for visible light blur, generates adversarial network (generates compensation features based on normal sensor data) for failed infrared sensors, and implements multi-frame accumulation compensation for radar point cloud loss, ensuring that the system still maintains reliable output when a single sensor fails.

[0026] The above technical chain forms a closed-loop mechanism of "environmental perception-dynamic fusion-abnormal fault tolerance", which enables the multi-sensor system to have environment-adaptive robust recognition capability without increasing hardware costs. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The figure is the overall system framework diagram of the present application.

[0028] Figure 2 The figure is the dual-branch neural network structure diagram of the present application.

[0029] Figure 3This is a diagram of the exception handling mechanism of the present invention.

[0030] Figure 4 This is a flowchart of the dynamic weight fusion process of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] As attached Figures 1-4 A multi-sensor data fusion target recognition method is shown, comprising the following steps: Simultaneously acquiring target area data using a visible light camera, an infrared thermal imager, and a millimeter-wave radar; performing adaptive histogram equalization on the visible light image and extracting texture feature maps; performing non-uniformity correction on the infrared image and extracting temperature distribution feature maps; performing dynamic clustering denoising on the millimeter-wave radar point cloud and extracting spatial location feature maps; inputting the three feature maps into a dynamic weight fusion module, which generates confidence scores for each sensor based on real-time calculated signal-to-noise ratio, data integrity, and environmental interference intensity, while simultaneously acquiring light intensity, rain / fog level, and electromagnetic interference intensity fed back by the environmental parameter sensing unit, and outputting an adaptive fusion weight matrix through a weight allocation function; weighting and fusing the multi-source feature maps according to the weight matrix to generate a joint feature map; inputting the joint feature map into a dual-branch deep convolutional neural network, wherein the spatial attention branch enhances target area features, the channel self-calibration branch suppresses redundant feature channels, and the dual-branch outputs are spliced ​​and output by a classification and regression head to determine the target category and location coordinates;

[0034] The visible light camera (resolution 1920x1080@30fps), the infrared thermal imager (thermal sensitivity ≤50mK) and the millimeter wave radar (77GHz, detection distance 200m) are synchronously triggered by hardware synchronization to collect data of a target region; after the visible light image is processed by adaptive histogram equalization (CLAHE algorithm, grid size 8x8), a texture feature map is extracted using an LBP operator; the infrared image is corrected by a two-point correction method to eliminate non-uniform noise, and then a temperature distribution feature map is extracted through temperature calibration; the millimeter wave radar point cloud is processed by using a DBSCAN clustering algorithm (neighborhood radius 0.5m, minimum point number 5) to remove outlier noise, and a spatial position feature map is generated based on the point cloud centroid; the three types of feature maps are input into a dynamic weight fusion module, which is integrated on an FPGA chip, to calculate the signal-to-noise ratio (PSNR index for visible light, NETD index for infrared, and SCR index for radar) and the data integrity (effective pixel / point cloud ratio) of each sensor in real time, and to output a weight matrix in combination with an environmental parameter perception unit (light sensor range 0-100klux, rain and fog sensor to detect water droplet density, electromagnetic interference detector range 0-30V / m); the joint feature map generated after weighted fusion is input into a double-branch deep convolutional neural network (model architecture as shown in Figure 2 The spatial attention branch outputs a target region heat map, and the channel self-calibration branch outputs a feature channel weight vector, which are connected to the original feature map through Hadamard product operation, and finally the target class probability and the bounding box coordinates (x, y, w, h) are output by a classification regression head.

[0035] In the embodiment, the weight distribution function adopts the following mathematical expression: Wi=α·Ci+β·Ei, where Wi represents the fusion weight value of the i-th sensor, Ci represents a normalized confidence score and satisfies 0≤Ci≤1, Ei represents an environment adaptation coefficient and is calculated by the formula Ei=1-0.3L-0.5R-0.2E using the light intensity mapping value L, the rain and fog level mapping value R, and the electromagnetic interference intensity mapping value E, and α and β are trainable parameters and satisfy the constraint conditions α+β=1 and 0.4≤α≤0.8;

[0036] The weight distribution function is specifically implemented as: normalized confidence score Ci=0.4*(SNRi / SNRmax)+0.3*(Compi / Compmax)+0.3*(1-INTi / INTmax), wherein SNRmax is an upper limit of an ideal environment signal-to-noise ratio, Compmax is a theoretical data integrity, and INTmax is a maximum interference intensity; the environment adaptation coefficient Ei is realized through a quantitative mapping table: L=1.0 when the illumination intensity is greater than 50 klux, and L=0 when the illumination intensity is less than 1 klux; R=1.0 when the rain and fog level is greater than 6 levels, and R=0 when the rain and fog level is less than 2 levels; E=1.0 when the electromagnetic interference is greater than 20 V / m, and E=0 when the electromagnetic interference is less than 5 V / m; and finally Ei=1-0.3L-0.5R-0.2E; the initial value of the parameter a is 0.6, and β=0.4, and the parameters are automatically optimized through back propagation in the model training stage.

[0037] In the embodiment, the operation of the dynamic weight fusion module includes a reinforcement learning optimization mechanism: a Q learning model is constructed with recognition accuracy as a reward function and weight distribution parameters as an action space; after each target recognition task is completed, a reward value is calculated according to the difference between the recognition result confidence and the true label; the Q value table is updated through the Bellman equation to dynamically adjust the parameters a and β in the weight distribution function; when the environmental parameters mutate beyond a threshold, a greedy strategy is started to explore the optimal weight combination;

[0038] In the embodiment, the reinforcement learning optimization mechanism implementation process is: the state space is defined as the environmental parameter vector [L, R, E], and the action space is the α adjustment value (step size ±0.1); the reward function R=2*(Accuracy-0.85)-0.5*|Δa|, wherein Accuracy is the current frame recognition accuracy; the Q learning model update formula is Q(s, a)←Q(s, a)+η[R+γmaxQ(s', a')-Q(s, a)], the learning rate η=0.2, and the discount factor γ=0.9; when the environmental parameters mutate (for example, the L value changes by more than 0.5 in 0.2 seconds), an ε-greedy strategy (ε=0.3) is started to randomly explore the weight combination, and the exploration result is written into the Q value lookup table for real-time calling.

[0039] In the embodiment, the double-branch deep convolutional neural network includes the following specific structures: the spatial attention branch uses a dilated convolution layer with a dilated rate of [2, 4, 6] to extract multi-scale features, and generates a spatial weight mask through a Sigmoid activation function; the channel self-calibration branch compresses the spatial dimension through a global average pooling layer, generates a channel weight vector through two fully connected layers and a ReLU activation function; after the Hadamard product operation of the spatial weight mask and the channel weight vector, residual connection is performed with the original feature map; the classification regression head uses a parallel Softmax classifier and a bounding box regressor, wherein the bounding box regressor uses a Smooth L1 loss function to optimize coordinate prediction;

[0040] In the double-branch neural network, the input layer receives a joint feature map of 256x256x3; the spatial attention branch adopts three groups of dilated convolution (convolution kernel 3x3, dilated rate 2 / 4 / 6, channel number 64 / 128 / 256), and outputs a spatial weight mask after Sigmoid activation; the channel self-calibration branch is compressed to a feature vector of 1x1x512 through global average pooling, and generates a channel weight through a fully connected layer (512->128->512) and ReLU activation; the spatial mask and the channel weight are multiplied element by element, and then residual addition is performed with the original input feature map; the classification regression head includes a parallel Softmax layer (outputting 5-class target probabilities) and a bounding box regression layer (4 output nodes, using a Smooth L1 loss function L=0.5x 2 if|x|<1else|x|-0.5).

[0041] In the embodiment, a sensor anomaly processing procedure is further included: when the output recognition confidence is lower than a preset threshold 0.7, a sensor health state diagnosis is started: an image blurriness index is detected for the visible light camera, if the blurriness exceeds 60, an image restoration based on Wiener filtering is started; a temperature gradient variance is detected for the infrared thermal imager, if the variance is lower than 0.05, a generative adversarial network compensation data is enabled, the generative adversarial network takes normal sensor data as input, generates an infrared feature map through a generator containing three residual blocks, and verifies the data validity by a discriminator containing five convolution layers; a point cloud density is detected for the millimeter wave radar, if the density drops by 40%, a multi-frame data accumulation compensation algorithm is triggered;

[0042] In the sensor anomaly processing implementation procedure: when the recognition confidence is lower than 0.7, a sensor diagnosis is started: the visible light blurriness is calculated by Laplacian variance (threshold>60 is determined to be invalid), and Wiener filtering restoration (point spread function size 5x5, noise variance 0.01) is adopted; the infrared sensor temperature gradient variance is calculated by Sobel operator (threshold<0.05 is determined to be invalid), and the generative adversarial network compensation is triggered; when the millimeter wave radar point cloud density (unit volume point number<0.8 times the historical average) drops, a multi-frame accumulation algorithm (5 consecutive frames of point cloud motion compensation are superimposed) is started.

[0043] In this embodiment, the generative adversarial network adopts a conditional training architecture: the generator input layer receives the spliced data of the visible light texture feature map and the millimeter wave spatial position feature map, and after dimension reduction through a 3x3 convolutional layer, the input is a residual block group, each residual block containing a batch normalization layer, a LeakyReLU activation function and a skip connection; the discriminator input layer receives the real infrared feature map and the generated feature map, which is down-sampled through 5 convolutional layers with a step size of 2, and finally outputs the authenticity probability through a Sigmoid function; the loss function adopts an improved Wasserstein distance: where z is the generator input noise vector, x is the real infrared data, x^ is the linear interpolation of the real and generated data, and λ is set to the gradient penalty coefficient 10;

[0044] wherein the generative adversarial network implementation details: the generator input layer is the spliced data of the visible light texture feature map and the radar position feature map (size 256x256x6), which is input into the residual block group (3 residual blocks, each containing a BN layer + LeakyReLU + skip connection) after dimension reduction through a 3x3 convolution (step size 1, channel 64); the discriminator is composed of 5 convolutional layers (convolution kernel 5x5, channel 32 / 64 / 128 / 256 / 512, step size 2), and the output layer is activated by Sigmoid; the gradient penalty term in the loss function is realized by random interpolation sample x^ = εx + (1-ε)G(z) (ε ~ U[0,1]), and the discriminator weight is updated by the RMSProp optimizer (learning rate 0.0001).

[0045] In this embodiment, the application of the method in the automatic driving scene includes: the target category recognition covers vehicles (including cars, trucks, buses), pedestrians (including adults, children), two-wheeled vehicles (including bicycles, motorcycles), traffic signs (including traffic lights, road signs), and obstacles (including cones, rubble); the position information output is polar coordinate system data centered on the vehicle, including target azimuth angle θ (accuracy ±0.5°) and distance d (accuracy ±0.1m); when the rain and fog level exceeds level 3, the infrared sensor weight is automatically increased to above 0.8, and when the night light is below 10 lux, the millimeter wave radar weight is increased to above 0.7;

[0046] wherein the automatic driving scene application implementation specification: the target classification system adopts a five-level code (1xxx-vehicle class / 2xxx-pedestrian class / 3xxx-two-wheeled vehicle class / 4xxx-traffic sign class / 5xxx-obstacle class); the polar coordinate system output format is (θ, d), wherein θ is calibrated by the radar beam angle resolution 0.5°, and d is weighted and fused by the radar TOF ranging and binocular vision ranging; the environment strategy library preset rules: when the rain and fog level is ≥4, the infrared weight Wi is set to ≥0.8, and when the light is ≤10 lux, the millimeter wave radar weight Wr is ≥0.7.

[0047] Embodiment Two: Automatic Driving Scene Target Recognition Process (Multi-source Data Joint Modeling Scene)

[0048] Step 1: Multi-sensor Synchronous Acquisition and Preprocessing

[0049] The visible light camera (1920x1080@30fps), infrared thermal imager (NETD≤50mK), and 77GHz millimeter wave radar (detection distance 200m) are triggered by a hardware synchronization signal to simultaneously collect target area data; the visible light image is input into the CLAHE algorithm for adaptive histogram equalization (grid size 8x8), and after eliminating the effects of uneven illumination, the LBP operator is used to extract the texture feature map; the infrared image performs two-point non-uniformity correction (reference temperature points 20℃ / 50℃), and the temperature-gray scale mapping generates a temperature distribution feature map; the millimeter wave radar point cloud is input into the DBSCAN clustering algorithm (neighborhood radius 0.5m, minimum point number 5) after motion compensation, and after removing outliers, a spatial position feature map is generated. The three types of feature maps are uniformly scaled to 256x256 resolution to form a standardized input.

[0050] Step 2: Dynamic Weight Fusion Calculation

[0051] Real-time calculation of sensor confidence scores: visible light signal-to-noise ratio: PSNR=20·log10(MAX_I / √MSE), MAX_I=255 (8-bit image maximum value);

[0052] Infrared noise equivalent temperature difference: NETD=(4F 2 ·NETD_sensor) / D 2 (F is focal length, D is detection distance);

[0053] Radar signal clutter ratio: SCR=10·log10(P_signal / P_clutter);

[0054] Data completeness Comp_i=effective data area / total acquisition area;

[0055] Environmental parameters are obtained through a sensor network: illumination intensity L (unit klux), rain and fog level R (1-6 levels), electromagnetic interference E (unit V / m).

[0056] Weight calculation execution:

[0057] Ci=0.4·(PSNR / PSNR_max)+0.3·(Comp_i)+0.3·(1-E / E_max);

[0058] Ei = 1 - 0.3 · L_norm - 0.5 · R_norm - 0.2 · E_norm (L_norm is the illumination normalization value: L > 50 kLux, then = 1, L < 1 kLux, then = 0);

[0059] Most weight Wi = a · Ci + b · Ei (a initial value 0.6, b = 0.4, a + b = 1);

[0060] Step 3: Feature Weighted Fusion

[0061] Pixel-level weighted fusion is performed on the texture feature map F_v, the temperature feature map F_t, and the position feature map F_r:

[0062] F_fused = Wv · F_v + Wt · F_t + Wr · F_r;

[0063] Wherein Wv, Wt, Wr are weight values calculated by weight 2, and Wv + Wt + Wr = 1. During the fusion process, when the rain and fog level R is greater than or equal to 4, Wr is forcibly set to be greater than or equal to 0.7; when the illumination L is less than or equal to 10 lux, Wt is forcibly set to be greater than or equal to 0.6.

[0064] Step 4: Dual-branch neural network recognition

[0065] The fused feature map is input into the dual-branch CNN:

[0066] Spatial attention branch: Dilation Convolution Layer 1 (dilation rate 2, kernel 3x3, output channel 64) -> Dilation Convolution Layer 2 (dilation rate 4, channel 128) -> Dilation Convolution Layer 3 (dilation rate 6, channel 256) -> Sigmoid activation to generate spatial mask M_s;

[0067] Channel self-calibration branch: Global Average Pooling -> Fully Connected Layer 1 (512 -> 128 nodes) -> ReLU activation -> Fully Connected Layer 2 (128 -> 512 nodes) -> Sigmoid to generate channel weight vector W_c;

[0068] Feature optimization: F_optimized = F_fused + F_fused o M_s o W_c (o represents Hadamard product);

[0069] Classification regression head output: Classification branch: Softmax outputs class probability P = [p_vehicle, p_pedestrian, p_cyclist, p_sign, p_obstacle];

[0070] Regression branch: bounding box coordinates (x, y, w, h) = (delta_x · s + c_x, delta_y · s + c_y, s · exp(delta_w), s · exp(delta_h)), (s is the feature map step, c is the center coordinate, and delta is the network prediction offset);

[0071] Step 5: Abnormality diagnosis and compensation

[0072] Start diagnosis when target recognition confidence max(P) < 0.7:

[0073] Visible light diagnosis: Calculate image blur B = Var(Laplacian(F_v)), if B > 60 perform Wiener filtering:

[0074] F_v' = IFFT[FFT(F_v) · H*(u,v) / (|H(u,v)|2 + K)];

[0075] H(u,v) is the point spread function (size 5x5), K = 0.01 (noise variance);

[0076] Infrared diagnosis: Calculate temperature gradient variance V = Var(Sobel(F_t)), if V < 0.05 trigger GAN compensation:

[0077] Generator input G_in = Concat(F_v, F_r), generate F_t' through 3 residual blocks (each block contains BN + LeakyReLU + jump connection);

[0078] Discriminator output D_out = Sigmoid(Conv5(D_in)), loss function

[0079] Radar diagnosis: When point cloud density p < 0.8 · p_hist, perform motion compensation multi-frame accumulation:

[0080] P_comp = Σ_{k=1}^5 T_k · P_{t-k} (T_k is the k-th frame transformation matrix);

[0081] Step 6: Reinforcement learning weight optimization

[0082] Every 5 frames of recognition results trigger Q learning update:

[0083] State s = [L_norm, R_norm, E_norm] (normalized environment vector);

[0084] Action a = {α + 0.1, α - 0.1, keep} (α adjustment step ± 0.1);

[0085] Reward R = 2 · (Acc - 0.85) - 0.5 · |Δα| (Acc is the current frame accuracy);

[0086] Q value update: Q(s, a) ← Q(s, a) + 0.2 · [R + 0.9 · max_a' Q(s', a') - Q(s, a)];

[0087] When the environment mutates (such as |ΔL_norm|>0.5 / 0.2s), perform random action exploration with a probability of 30%.

[0088] Step 7: Polar coordinate output and policy execution

[0089] Final output target information: Category code: 1xxx (vehicle), 2xxx (pedestrian), 3xxx (two-wheeled vehicle), 4xxx (traffic sign), 5xxx (obstacle);

[0090] Spatial coordinates: (θ, d)=(arctan(y / x), √(x 2 +y 2 )) where: θ is calibrated by radar beam angle (accuracy ±0.5°);

[0091] d=w_radar·d_radar+(1-w_radar)·d_stereo (millimeter wave and binocular vision weighted fusion);

[0092] Automatically executed according to the environment strategy library: forced Wt≥0.6 at night (L<10lux), forced Wr≥0.8 in heavy rain (R≥5).

[0093] Finally, it should be pointed out that in the description of the present application, it should be pointed out that unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, which can be mechanical connection or electrical connection, or the communication between two elements, or direct connection, "up", "down", "left", "right" and the like are only used to indicate relative positional relationship, when the absolute position of the described object changes, the relative positional relationship may change;

[0094] Secondly: the present application discloses the structure involved in the embodiment of the present application, other structures can refer to the usual design, in the case of no conflict, the same embodiment and different embodiments of the present application can be combined with each other;

[0095] Finally: the above only for the preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A target recognition method based on multi-sensor data fusion, characterized in that, The process includes the following steps: Data on the target area is simultaneously acquired using a visible light camera, an infrared thermal imager, and a millimeter-wave radar; adaptive histogram equalization is performed on the visible light image to extract texture feature maps; non-uniformity correction is performed on the infrared image to extract temperature distribution feature maps; dynamic clustering denoising is performed on the millimeter-wave radar point cloud to extract spatial location feature maps; the three feature maps are input into a dynamic weighted fusion module, which generates confidence scores for each sensor based on real-time calculated signal-to-noise ratio, data integrity, and environmental interference intensity, while simultaneously acquiring the light intensity, rain / fog level, and electromagnetic interference intensity fed back by the environmental parameter sensing unit, and outputs an adaptive fusion weight matrix through a weight allocation function; the multi-source feature maps are weighted and fused according to the weight matrix to generate a joint feature map; the joint feature map is input into a dual-branch deep convolutional neural network, where the spatial attention branch enhances the target area features, the channel self-calibration branch suppresses redundant feature channels, and the dual-branch outputs are spliced ​​and output by a classification and regression head to determine the target category and location coordinates.

2. The target recognition method based on multi-sensor data fusion according to claim 1, characterized in that, The weight allocation function adopts the following mathematical expression: Wi=α·Ci+β·Ei, where Wi represents the fusion weight value of the i-th type of sensor, Ci represents the normalized confidence score and satisfies 0≤Ci≤1, Ei represents the environmental adaptation coefficient and is calculated by the light intensity mapping value L, the rain and fog level mapping value R, and the electromagnetic interference intensity mapping value E through the formula Ei=1-0.3L-0.5R-0.2E, and α and β are trainable parameters and satisfy the constraints α+β=1 and 0.4≤α≤0.

8.

3. The target recognition method based on multi-sensor data fusion according to claim 1, characterized in that, The operation of the dynamic weight fusion module includes a reinforcement learning optimization mechanism: constructing a Q-learning model with recognition accuracy as the reward function and weight allocation parameters as the action space; calculating the reward value based on the difference between the recognition result confidence and the true label after each target recognition task is completed; updating the Q-value table through the Bellman equation and dynamically adjusting the parameters α and β in the weight allocation function; and initiating a greedy strategy to explore the optimal weight combination when environmental parameters suddenly exceed the threshold.

4. The target recognition method based on multi-sensor data fusion according to claim 1, characterized in that, The dual-branch deep convolutional neural network includes the following unique structure: the spatial attention branch extracts multi-scale features using dilated convolutional layers with a dilation rate of [2, 4, 6], and generates a spatial weight mask using a Sigmoid activation function; the channel self-calibration branch compresses the spatial dimension through a global average pooling layer, and generates channel weight vectors through two fully connected layers and a ReLU activation function; the spatial weight mask and the channel weight vector are subjected to a Hadamard product operation, and then residual connections are performed with the original feature map; the classification and regression head uses a parallel Softmax classifier and a bounding box regressor, wherein the bounding box regressor uses a Smooth L1 loss function to optimize coordinate prediction.

5. The target recognition method based on multi-sensor data fusion according to claim 1, characterized in that, It also includes a sensor anomaly handling process: when the output recognition confidence level is lower than the preset threshold of 0.7, sensor health status diagnosis is initiated: for visible light cameras, if the image blur index exceeds 60, image restoration based on Wiener filtering is initiated; for infrared thermal imagers, if the temperature gradient variance is lower than 0.05, a generative adversarial network is enabled to compensate for the data. This generative adversarial network takes normal sensor data as input, generates an infrared feature map through a generator containing 3 residual blocks, and verifies the validity of the data by a discriminator containing 5 convolutional layers; for millimeter-wave radar, if the point cloud density decreases by 40%, a multi-frame data accumulation compensation algorithm is triggered.

6. The target recognition method based on multi-sensor data fusion according to claim 5, characterized in that, The generative adversarial network adopts a conditional training architecture: the generator input layer receives spliced ​​data of visible light texture feature map and millimeter-wave spatial location feature map, and after dimensionality reduction by 3×3 convolutional layers, it is input into residual block groups. Each residual block contains a batch normalization layer, a LeakyReLU activation function and skip connections; the discriminator input layer receives real infrared feature map and generated feature map, and downsamples through 5 convolutional layers with a stride of 2, and finally outputs the authenticity probability through the Sigmoid function. The loss function uses a modified Wasserstein distance: Where z is the generator input noise vector, and x is the real infrared data. For linear interpolation between real and generated data, λ is set to a gradient penalty coefficient of 10.

7. The target recognition method based on multi-sensor data fusion according to claim 1, characterized in that, The application of the method in autonomous driving scenarios includes: target category recognition covering vehicles (including cars, trucks, and buses), pedestrians (including adults and children), two-wheeled vehicles (including bicycles and motorcycles), traffic signs (including traffic lights and road signs), and obstacles (including cones and piles of gravel); the location information output is polar coordinate system data centered on the vehicle, including the target azimuth angle θ (accuracy ±0.5°) and distance d (accuracy ±0.1m); when the rain and fog level exceeds level 3, the weight of the infrared sensor is automatically increased to above 0.8, and when the nighttime light intensity is below 10 lux, the weight of the millimeter-wave radar is increased to above 0.7.

Images