Multi-sensor time sequence data fusion method, system and device and storage medium

Abstract

The invention discloses a multi-sensor time sequence data fusion method, system and device and a storage medium, and the method comprises the following steps: 1, obtaining a visible light image, an infrared image and a millimeter wave radar point cloud, and carrying out the data preprocessing; step 2, acquiring spatial features of the visible light image, the infrared image and the millimeter wave radar point cloud information by using a convolutional neural network, inputting continuous N frames of feature sequences into LSTM, and outputting enhanced feature representations of the visible light image, the infrared image and the millimeter wave radar point cloud information; and step 3, calculating weights of the visible light image, the infrared image and the millimeter wave radar point cloud information features in the step 2 through a learnable attention network, performing weighted summation to obtain a final fusion feature, and outputting the final fusion feature. According to the method, three common sensor combination time dimension feature change rules are adopted, a dynamic fusion strategy driven by an attention mechanism is introduced, multi-sensor data fusion is achieved, fusion features are output, and the time sequence dynamic association data fusion problem is solved.

Description

Technical Field

[0001] This invention belongs to the field of sensor data fusion, specifically relating to a method, system, device, and storage medium for multi-sensor time-series data fusion based on a long short-term memory network. Background Technology

[0002] Traditional multi-sensor fusion methods (such as Kalman filtering and DS evidence theory) focus primarily on static or short-term data fusion, lacking effective modeling of temporally dynamic correlation data. Furthermore, in complex environments (such as rain, snow, fog, haze, and darkness), the imaging quality of visible light sensors deteriorates drastically; while infrared sensors can partially penetrate smoke, their resolution is low and image texture information is weak; millimeter-wave radar provides accurate ranging but struggles to identify target details. Existing fusion strategies often fail to adaptively balance the spatiotemporal complementarity of sensors. Although some research has attempted to use Long Short-Term Memory (LSTM) networks for multi-sensor localization (e.g., combining LSTM with unscented Kalman filtering to improve the positioning accuracy of GPS / IMU / LiDAR), there is still a lack of solutions for end-to-end temporal fusion of these three types of sensors with significantly different characteristics: visible light, infrared, and millimeter-wave radar. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a multi-sensor time-series data fusion method, system, device, and storage medium based on Long Short-Term Memory (LSTM) networks. It employs LSTM networks to acquire the temporal dimension feature variation patterns of three common sensor combinations: visible light sensors, infrared sensors, and millimeter-wave radar sensors. An attention-driven dynamic fusion strategy is introduced to achieve multi-sensor data fusion, outputting fused features and solving the problem of time-series dynamic correlation data fusion.

[0004] The objective of this invention is achieved through the following technical solution: a multi-sensor time-series data fusion method, comprising the following steps:

[0005] Step 1: Acquire visible light images, infrared images, and millimeter-wave radar point clouds, and perform data preprocessing;

[0006] Step 2: Use a convolutional neural network to obtain the spatial features of visible light images, infrared images, and millimeter-wave radar point cloud information. Input the N consecutive frames of feature sequences into an LSTM and output the enhanced feature representations of visible light images, infrared images, and millimeter-wave radar point cloud information.

[0007] Step 3: Calculate the weights of the visible light image, infrared image, and millimeter-wave radar point cloud information features from Step 2 using a learnable attention network, sum them by weight to obtain the final fused features, and output them.

[0008] Preferably, in step 1, the data preprocessing process includes spatial alignment and temporal alignment.

[0009] Preferably, in step 1, spatial alignment is based on camera calibration results and coordinate transformation matrix, mapping infrared images and millimeter-wave radar point clouds to the visible light image coordinate system to achieve pixel-level registration;

[0010] Time alignment uses the same second pulse for all three sensors, and then divides the second pulse into corresponding frame frequency pulse signals according to the frame frequency of each sensor. Using the frame frequency pulse of the visible light sensor as a reference, the infrared sensor image information and the millimeter-wave radar sensor point cloud information are obtained through interpolation.

[0011] Preferably, in step 2, a ResNet50 CNN convolutional neural network model is used to obtain the spatial features of visible light images, infrared images, and millimeter-wave radar point cloud information.

[0012] Preferably, in step 2, the CNN convolutional neural network model includes: shallow convolution to extract basic features such as edges and textures; mid-level convolution to combine basic features to form component-level representations; deep convolution to integrate components to form complete object and scene features; and global average pooling applied after the last convolution to compress the spatial feature map into a feature vector of fixed length D, and processing each frame in the sequence sequentially.

[0013] Preferably, in step 2, after the LSTM network processes the entire sequence, a global pooling strategy is adopted to extract visible light enhanced temporal features, collect the hidden states of all time steps, and take the maximum value of all time steps.

[0014] Preferably, in step 3, the weights of the visible light image, infrared image, and millimeter-wave radar point cloud information features are adjusted according to different meteorological conditions.

[0015] In addition to providing a multi-sensor time-series data fusion method, this invention further provides a system for implementing the above method, the system comprising:

[0016] The data preprocessing module acquires visible light images, infrared images, and millimeter-wave radar point clouds, performs data preprocessing, and outputs the data to the book sequence feature extraction module.

[0017] The temporal feature extraction module uses a convolutional neural network to obtain spatial features of visible light images, infrared images, and millimeter-wave radar point cloud information. It inputs a continuous N-frame feature sequence into an LSTM and outputs enhanced feature representations of visible light images, infrared images, and millimeter-wave radar point cloud information.

[0018] The feature fusion module calculates the weights of the visible light image, infrared image, and millimeter-wave radar point cloud information features output by the temporal feature extraction module through a learnable attention network, and obtains the final fused features by weighted summation and output.

[0019] The present invention further provides an electronic device, including a memory and a processor.

[0020] The memory is used to store computer programs.

[0021] The processor is used to load the computer program in the memory so that when the above-mentioned electronic device is running, it can realize the operation of the above-mentioned multi-sensor time-series data fusion system.

[0022] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, enables the operation of the aforementioned multi-sensor time-series data fusion system.

[0023] The beneficial effects of this invention are as follows:

[0024] As can be seen from the above scheme, the present invention provides a multi-sensor time-series data fusion method, system, device and storage medium based on long short-term memory network. It uses LSTM to capture the evolution law of each sensor in the time dimension, designs a time-series feature extraction network for trimodal data, and introduces a dynamic fusion strategy driven by attention mechanism. It adaptively adjusts the feature contribution weight of different sensors according to weather, illumination, target characteristics, etc., to achieve end-to-end optimization from raw data to perception decision, and establishes a multimodal data fusion system of visible light-infrared-millimeter wave radar in complex environments, thereby improving the adaptability and reliability of complex scenarios. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multi-sensor time-series data fusion method based on a long short-term memory network in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] like Figure 1 As shown, the technical solution of the present invention provides a multi-sensor time-series data fusion method, including the following steps:

[0028] Step 1: Acquire visible light images, infrared images, and millimeter-wave radar point clouds, and perform data preprocessing;

[0029] Step 2: Use a convolutional neural network to obtain the spatial features of visible light images, infrared images, and millimeter-wave radar point cloud information. Input the N consecutive frames of feature sequences into an LSTM and output the enhanced feature representations of visible light images, infrared images, and millimeter-wave radar point cloud information.

[0030] Step 3: Calculate the weights of the visible light image, infrared image, and millimeter-wave radar point cloud information features from Step 2 using a learnable attention network, sum them by weight to obtain the final fused features, and output them.

[0031] In one embodiment of the present invention, step 1, the data preprocessing process includes spatial alignment and temporal alignment. Specifically, spatial alignment, based on camera calibration results and coordinate transformation matrix, maps infrared images and millimeter-wave radar point clouds to the visible light image coordinate system to achieve pixel-level registration. For temporal alignment, firstly, the three sensors use the same second pulse, and then the second pulse is divided into corresponding frame frequency pulse signals according to the frame frequency of each sensor. Typically, the visible light sensor has a higher frame frequency. Using the visible light sensor frame frequency pulse as a reference, the infrared sensor image information and the millimeter-wave radar sensor point cloud information are obtained through interpolation. In this embodiment, step 1 includes at least the following steps:

[0032] Step 1.1: Using the same second pulse, divide the second pulse into f... vis f ir f r Visible light video images, infrared video images, and millimeter-wave radar point cloud information are generated using an external triggering method.

[0033] Step 1.2: Acquire visible light video images (X) vis Y vis ), infrared video images (X) ir Y ir ), millimeter-wave radar point cloud information (X r Y r Z r );

[0034] Step 1.3: Based on the camera calibration results, the installation offset (ΔXΔYΔZ) of the radar sensor in the camera coordinate system can be obtained, and the millimeter-wave radar measurement value can be mapped to the visible light coordinate system through equation (1);

[0035]

[0036] In the formula (X) c Y c Z c () represents the radar measurement value in the visible light image coordinate system;

[0037] Step 1.4: Based on the coordinate transformation matrix, the infrared video image is mapped to visible light coordinates using equation (2);

[0038]

[0039] In the formula [x' ir ;y' ir [x] represents the projection of the infrared measurement point onto the visible light image coordinate system; ir ;y ir [Illustration point] is the infrared measurement point. [b1; b2] is the scaling and rotation matrix; [b1; b2] is the translation matrix; where [b1; b2] are obtained through sensor calibration;

[0040] Step 1.5: Using the visible light video frame rate as a reference, interpolate the infrared video image and millimeter-wave radar point cloud at the same time using equation (3);

[0041]

[0042] In one embodiment of the present invention, step 2 involves using a ResNet50 CNN convolutional neural network model to acquire spatial features from visible light images, infrared images, and millimeter-wave radar point cloud information. In this embodiment, step 2 includes at least the following steps:

[0043] Step 2.1: Select the ResNet50 CNN convolutional neural network model, input the video image information into the CNN network, and perform multiple convolution and pooling operations. The shallow convolution (layers 1-3) extracts basic features such as edges and textures; the middle convolution (layers 4-10) combines basic features to form component-level representations; the deep convolution (layer 11 to the end) integrates components to form complete object and scene features; after the last convolution, global average pooling is applied to compress the spatial feature map into a feature vector of fixed length D, and each frame in the sequence is processed sequentially.

[0044] Step 2.2: Assemble the N individual feature vectors into a sequence data structure in chronological order, where N is the temporal length and D is the feature dimension;

[0045] Step 2.3: Process the feature sequences sequentially using an LSTM network. The processing of the first frame includes:

[0046] (1) Input gate: determines which information in the features of the first frame is worth storing in long-term memory;

[0047] (2) Forgetting Gate: Since there is no historical memory, this gate has a relatively minor function;

[0048] (3) Cell state update: Update internal memory cells based on information filtered by the input gate;

[0049] (4) Output gate: Based on the current input and the new memory, generate the hidden state output of the first frame;

[0050] (5) Hidden state passing: passing the hidden state to the next frame;

[0051] Processing of the second and subsequent frames:

[0052] (1) Input gate: Evaluate the importance of the current frame input and determine the current frame input parameters;

[0053] (2) Forgetting Gate: Determines how much historical information to retain and how much old information to forget;

[0054] (3) Cell state renewal: selectively forgetting old memories and selectively adding new memories;

[0055] (4) Output gate: Generate a new hidden state based on the current input and the updated memory;

[0056] (5) Passing hidden state: Passing the hidden state to the next step.

[0057] Step 2.4: After the LSTM network processes the entire sequence, a global pooling strategy is used to extract the visible light enhanced temporal feature H_vis, collect the hidden states of all time steps, and take the maximum value of all time steps.

[0058] Step 2.5: For the infrared video image, use Steps 6 to 9 sequentially to obtain the infrared temporal enhancement feature H_ir;

[0059] Step 2.6: Convert the radar point cloud sequence into a feature map, and then use Step 6 to Step 9 to obtain the infrared temporal enhancement feature H_r.

[0060] Step 2.7: Concatenate the hidden states H_vis, H_ir, and H_radar output by the three LSTMs.

[0061] In one embodiment of the present invention, in step 3, the weights of the visible light image, infrared image, and millimeter-wave radar point cloud information features are adjusted according to different meteorological conditions, specifically including the following steps:

[0062] Step 3.1: Obtain current environmental status information, such as light intensity (day / night), weather conditions (sunny, rainy, foggy, snowy), visibility, electromagnetic complexity, etc., and encode them into an environmental vector C. Input H_vis, H_ir, H_r and C into a multi-layer fully connected neural network to generate importance scores S_vis, S_ir, and S_r. Make a preliminary judgment on the instantaneous reliability and value of each sensor. Normalize S_vis, S_ir, and S_r to obtain the modal feature weights α_vis, α_ir, and α_radar.

[0063] Step 3.2: Obtain the final fused features by weighted summation, as shown in the following formula:

[0064] H_fuse=α_vis*H_vis+α_ir*H_ir+α_radar*H_radar

[0065] In addition to providing a multi-sensor time-series data fusion method, the present invention further provides a system for implementing the above method, the system comprising:

[0066] The data preprocessing module acquires visible light images, infrared images, and millimeter-wave radar point clouds, performs data preprocessing, and outputs the data to the book sequence feature extraction module.

[0067] The temporal feature extraction module uses a convolutional neural network to obtain spatial features from visible light images, infrared images, and millimeter-wave radar point cloud information. It inputs a continuous N-frame feature sequence into an LSTM and outputs enhanced feature representations of visible light images, infrared images, and millimeter-wave radar point cloud information. Temporal feature extraction and LSTM modeling refer to using a convolutional neural network (CNN) to obtain spatial features from visible light images, infrared images, and millimeter-wave radar point cloud information, inputting a continuous N-frame feature sequence into an LSTM, and outputting enhanced feature representations of visible light images, infrared images, and millimeter-wave radar point cloud information, H_vis, H_ir, and H_radar.

[0068] The feature fusion module uses a learnable attention network to calculate the weights of the features from the visible light image, infrared image, and millimeter-wave radar point cloud information output by the temporal feature extraction module. It then performs a weighted sum to obtain the final fused feature and outputs it. The attention mechanism feature fusion module calculates the weights α_vis, α_ir, and α_radar for each modality feature using a learnable attention network. For example, it reduces α_vis under low light conditions and increases α_radar under severe weather conditions. The final fused feature is obtained by weighted summation: H_fuse = α_vis * H_vis + α_ir * H_ir + α_radar * H_radar.

[0069] The present invention further provides an electronic device, including a memory and a processor.

[0070] The memory is used to store computer programs.

[0071] The processor is used to load the computer program in the memory so that when the above-mentioned electronic device is running, it can realize the operation of the above-mentioned multi-sensor time-series data fusion system.

[0072] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, enables the operation of the aforementioned multi-sensor time-series data fusion system.

[0073] Compared with existing technologies, the multi-sensor time-series data fusion method, system, device, and storage medium based on long short-term memory networks provided in this embodiment of the invention have the following advantages:

[0074] 1. Temporal modeling capability: By introducing an LSTM network, the system adds historical temporal information, thereby improving the accuracy of predicting the behavior of dynamic targets;

[0075] 2. Adaptive Fusion: An attention mechanism is introduced to enable the system to dynamically adjust the fusion strategy according to environmental changes (such as severe weather or partial sensor failure), thereby enhancing the system's robustness.

[0076] 3. End-to-end learning: The entire model (feature extraction, LSTM, fusion module) can be trained jointly to achieve optimization from raw modal data to the perception task.

[0077] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-sensor time-series data fusion method, characterized in that: The method comprises the following steps: Step 1, acquiring visible light images, infrared images and millimeter wave radar point clouds, and performing data preprocessing; Step 2, using a convolutional neural network to obtain spatial features of the visible light images, the infrared images and the millimeter wave radar point cloud information, inputting a continuous N-frame feature sequence into an LSTM, and outputting enhanced feature representation of the visible light images, the infrared images and the millimeter wave radar point cloud information; Step 3, calculating weights of the visible light image, the infrared image and the millimeter wave radar point cloud information features in step 2 through a learnable attention network, weightedly summing to obtain final fusion features and outputting.

2. The multi-sensor time-series data fusion method of claim 1, wherein: In step 1, the data preprocessing process comprises spatial alignment and time alignment.

3. The multi-sensor time-series data fusion method of claim 2, wherein: In step 1, the spatial alignment is based on camera calibration results and a coordinate transformation matrix, and maps the infrared images and the millimeter wave radar point clouds to a visible light image coordinate system to realize pixel-level registration; The time alignment uses the same second pulse for the three sensors, and then divides the second pulse into corresponding frame frequency pulse signals according to the frame frequencies of the sensors, takes the visible light sensor frame frequency pulse as a reference, and obtains infrared sensor image information and millimeter wave radar sensor point cloud information through an interpolation method.

4. The multi-sensor time-series data fusion method of claim 1, wherein: In step 2, a ResNet50 CNN convolutional neural network model is selected to obtain spatial features of the visible light images, the infrared images and the millimeter wave radar point cloud information.

5. The multi-sensor time-series data fusion method of claim 4, wherein: In step 2, the CNN convolutional neural network model comprises: a shallow convolution for extracting basic features such as edges and textures; a middle convolution for combining the basic features to form part-level representation; and a deep convolution for integrating the parts to form complete object and scene features; and a global average pooling is applied after the last convolution to compress the spatial feature map into a fixed-length D feature vector, and each frame in the sequence is processed in turn.

6. The multi-sensor time-series data fusion method of claim 5, wherein: In step 2, after the LSTM network processes the entire sequence, a global pooling strategy is adopted to extract visible light enhanced time sequence features, collect hidden states of all time steps, and take the maximum value of all time steps.

7. The multi-sensor time-series data fusion method of claim 1, wherein: In step 3, the weights of the visible light image, the infrared image and the millimeter wave radar point cloud information features are adjusted according to different weather conditions.

8. A multi-sensor time-series data fusion system, characterized by: The system is used to implement the method of any one of claims 1-7, and the system comprises: a data preprocessing module for acquiring visible light images, infrared images and millimeter wave radar point clouds, and performing data preprocessing, and outputting to a sequence feature extraction module; a time sequence feature extraction module for using a convolutional neural network to obtain spatial features of the visible light images, the infrared images and the millimeter wave radar point cloud information, inputting a continuous N-frame feature sequence into an LSTM, and outputting enhanced feature representation of the visible light images, the infrared images and the millimeter wave radar point cloud information; a feature fusion module for calculating weights of the visible light image, the infrared image and the millimeter wave radar point cloud information features output by the time sequence feature extraction module through a learnable attention network, weightedly summing to obtain final fusion features and outputting.

9. The present application still further provides an electronic device comprising a memory and a processor, wherein, The memory is used to store a computer program, and the processor is used to load the computer program in the memory, and the electronic device can implement the operation of the multi-sensor time sequence data fusion system of claim 8 when running.

10. The application also provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, is capable of implementing the operation of the multi-sensor time-series data fusion system according to claim 8.

Images