Traffic safety early warning system and method based on thunder-vision fusion

Abstract

The invention discloses a traffic safety early warning system and method based on thunder-vision fusion. The system comprises an acquisition system, an extraction module, a construction module, a prediction module and an alarm module. The acquisition module is used for acquiring and processing image data in an urban traffic scene; the extraction module is used for performing feature extraction on the image data to obtain fusion features; the construction module constructs a prediction model based on the fusion features; the prediction module obtains a prediction result by using the prediction model; and the alarm module is used for giving an alarm when the prediction result shows that the behavior violating the traffic regulations occurs. According to the invention, by fusing visual and radar sensor data, the accuracy and robustness of early warning can be improved, false alarms caused by environmental changes and sensor limitations are reduced, meanwhile, the requirement of real-time monitoring is met, the traffic accident rate is effectively reduced, the traffic management efficiency is improved, and a basis is provided for future technical improvement and function extension.

Description

Technical Field

[0001] The present invention relates to the field of traffic safety, and in particular to a traffic safety early warning system and method based on radar and vision fusion. Background Art

[0002] In urban traffic scenes, vehicles, pedestrians, non-motor vehicles and other traffic participants are densely mixed, and their movement trajectories are complex and changeable, posing many traffic safety hazards. A single visual sensor is difficult to fully and accurately perceive and predict potential dangers in complex traffic scenes. Vision is easily disturbed by factors such as light and weather, and is easily blocked; while radar has a long detection distance and strong penetration, but it is difficult to obtain target appearance feature information. Therefore, there is an urgent need for a traffic safety warning system that integrates vision and radar, fully utilizing the complementary advantages of the two heterogeneous sensors to improve the accuracy and robustness of safety warnings in complex traffic scenes.

[0003] However, heterogeneous sensor data fusion faces many technical challenges: vision and radar have significant differences in spatial dimensions, timestamps, data formats, etc., making it difficult to directly match and associate; the feature representations of different traffic participants vary greatly, making it difficult to establish a unified fusion framework; the detection results of vision and radar may conflict and require effective coordination at the decision-making level. At the same time, the huge amount of data in traffic scenes and high real-time requirements place strict demands on the algorithm's computational efficiency and resource consumption. Summary of the invention

[0004] In order to solve the technical problems in the above background, the present invention aims to propose a solution for the characteristics of dense and mixed traffic participants and complex and changeable movement trajectories in urban traffic scenes.

[0005] To solve the above problems, the present invention provides a traffic safety warning system based on radar and visual fusion, comprising: a collection system, an extraction module, a construction module, a prediction module and an alarm module;

[0006] The acquisition module is used to acquire and process image data in urban traffic scenes;

[0007] The extraction module is used to extract features from the image data to obtain fusion features;

[0008] The construction module constructs a prediction model based on the fusion features;

[0009] The prediction module uses the prediction model to obtain a prediction result;

[0010] The alarm module is used to issue an alarm when the prediction result shows that there is a violation of traffic regulations.

[0011] Preferably, the acquisition module uses visual sensors and radar sensors to acquire images of urban traffic scenes; then, the image data acquired by the visual sensor and the point cloud data acquired by the radar sensor are preprocessed. First, a median filtering algorithm is used to denoise the image data, remove salt and pepper noise, and improve image quality; then, a voxelization method is used to downsample the point cloud data, dividing the point cloud data into 1-meter voxel grids, retaining only one point in each grid, and removing redundant information.

[0012] Preferably, the workflow of the extraction module includes:

[0013] Based on the image data and point cloud data, the data association results are obtained;

[0014] Based on the association result, obtaining the temporal dependency relationship between different sensor data;

[0015] Based on the temporal dependency, a fusion model is constructed to extract the fusion features.

[0016] Preferably, the process of constructing the fusion model is as follows: based on the temporal dependency, multimodal sequence features are obtained; the multimodal feature sequence includes data collected by different sensors; for the data collected by different sensors, the temporal features are extracted; the temporal features are modeled using a gated recurrent unit network, and the input of data at the current moment, the retention of historical information, and the output of the current state are controlled through the input gate, forget gate, and output gate in the gated recurrent unit network to achieve modeling of multimodal temporal data; in the modeling process of the gated recurrent unit network, the importance weights of different modal data are dynamically adjusted according to the hidden state and context information at the current moment by introducing an attention mechanism, so as to highlight the key temporal features and obtain the multimodal sequence features;

[0017] The output of the gated recurrent unit network is then combined with a convolutional neural network to construct the fusion model of multimodal data, wherein the convolutional neural network is used to extract local features in the multimodal data; the cross entropy loss function is used as the optimization target, and the network parameters of the fusion model are updated through the back propagation algorithm.

[0018] Preferably, the convolutional neural network model includes: 3 convolutional layers and 2 pooling layers; the convolution kernel size is 3x3, and the pooling kernel size is 2x2, which are used to extract local features in multimodal data; the output of the gated recurrent unit network is spliced ​​with the output of the convolutional neural network, and then the features are fused through the fully connected layer to obtain the final fused features.

[0019] Preferably, the construction module inputs the extracted fusion features into a pre-trained long-short term neural network model for learning to construct the final prediction model.

[0020] The present invention also provides a traffic safety early warning method based on radar and visual fusion, which is applied to the above system and comprises the following steps:

[0021] Collect and process image data in urban traffic scenes;

[0022] Extracting features from the image data to obtain fusion features;

[0023] Based on the fusion features, construct a prediction model;

[0024] Using the prediction model, obtaining a prediction result;

[0025] When the prediction results show that there is a violation of traffic regulations, an alarm is issued.

[0026] By fusing visual and radar sensor data, the present invention can improve the accuracy and robustness of early warnings, reduce false alarms caused by environmental changes and sensor limitations, and meet the needs of real-time monitoring, effectively reduce the incidence of traffic accidents, improve traffic management efficiency, and provide a foundation for future technological improvements and functional expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Embodiment 1

[0032] like Figure 1As shown, it is a schematic diagram of the system structure of this embodiment, including: an acquisition system, an extraction module, a construction module, a prediction module and an alarm module; the acquisition module is used to collect and process image data in urban traffic scenes; the extraction module is used to extract features from image data to obtain fusion features; the construction module constructs a prediction model based on the fusion features; the prediction module uses the prediction model to obtain prediction results; the alarm module is used to issue an alarm when the prediction results show that there is a violation of traffic regulations.

[0033] First, the acquisition module uses visual sensors and radar sensors to collect images of urban traffic scenes. Then, the image data collected by the visual sensor and the point cloud data collected by the radar sensor are preprocessed. First, the median filter algorithm is used to denoise the image data, remove salt and pepper noise, and improve the image quality; then, the voxelization method is used to downsample the point cloud data, dividing the point cloud data into 1-meter voxel grids, retaining only one point in each grid to remove redundant information.

[0034] After preprocessing, an extraction module is used to extract image features of the processed image data.

[0035] Firstly, based on the image data and point cloud data, the data association results are obtained.

[0036] Specifically, the VGG-16 convolutional neural network is used to extract image features and extract a 4096-dimensional feature vector; the LSTM recurrent neural network is used to extract the temporal features of the point cloud data and extract a 256-dimensional feature vector. The extracted image features and point cloud features are input into the attention mechanism module, and the attention weights between the features are calculated by Scaled Dot-ProductAttention to determine the correlation between them and obtain the top-5 associated feature pairs. According to the correlation of the feature pairs, a three-layer perceptron is used to perform nonlinear transformation on the image features and point cloud features, and the 4096-dimensional image features and 256-dimensional point cloud features are mapped to a 128-dimensional unified feature space. In the unified feature space, the semantic similarity between different sensor data is calculated by cosine similarity, and data with a similarity greater than 8 are associated.

[0037] Afterwards, based on the association results, the temporal dependencies between different sensor data are obtained.

[0038] Specifically, according to the data association results, the GRU gated recurrent unit network is used to perform sequence modeling on the associated multimodal features, and the temporal dependencies between different sensor data are mined to obtain 256-dimensional temporal features. The temporal features are used as prior knowledge, and the image features and point cloud features after association are globally optimized and fused through the conditional random field CRF. The optimal multimodal fusion features are obtained through the Viterbi algorithm to form a 128-dimensional robust representation of the state of traffic participants for subsequent behavior prediction and risk assessment tasks.

[0039] Finally, a fusion model is constructed based on the temporal dependency.

[0040] First, based on the temporal dependency, multimodal sequence features are obtained. The multimodal feature sequence includes data collected by different sensors; the temporal features are extracted from the data collected by different sensors; the temporal features are modeled using a gated recurrent unit network, and the input gate, forget gate, and output gate in the gated recurrent unit network are used to control the data input at the current moment, the retention of historical information, and the output of the current state to achieve modeling of multimodal temporal data; in the modeling process of the gated recurrent unit network, by introducing the attention mechanism, the importance weights of different modal data are dynamically adjusted according to the hidden state and context information at the current moment, highlighting the key temporal features and obtaining multimodal sequence features.

[0041] The output of the gated recurrent unit network is combined with the convolutional neural network to construct a fusion model of multimodal data, where the convolutional neural network is used to extract local features in multimodal data. The cross entropy loss function is used as the optimization target, and the network parameters of the fusion model are updated through the back propagation algorithm.

[0042] The convolutional neural network model used in the fusion model includes 3 convolutional layers and 2 pooling layers; the convolution kernel size is 3x3 and the pooling kernel size is 2x2, which is used to extract local features in multimodal data. The output of the gated recurrent unit network is concatenated with the output of the convolutional neural network, and then the features are fused through the fully connected layer to obtain the final fusion features. In the model training stage, the cross entropy loss function is used as the optimization target, the Adam optimizer is used, the learning rate is set to 0.01, the batch size is set to 32, and 100 epochs are trained.

[0043] The construction module inputs the extracted fusion features into the pre-trained model for learning to obtain a prediction model. Specifically, the extracted fusion features are input into the pre-trained long-term and short-term neural network model for learning to obtain the final prediction model.

[0044] The prediction module uses the trained fusion model to obtain the final prediction result. When the prediction result shows that there is a violation of traffic regulations, the alarm module will issue an alarm.

[0045] Embodiment 2

[0046] This embodiment also provides a traffic safety warning method based on radar and vision fusion, the steps comprising:

[0047] S1. Collect and process image data in urban traffic scenes.

[0048] Visual sensors and radar sensors are used to collect images of urban traffic scenes. The image data collected by the visual sensor and the point cloud data collected by the radar sensor are then preprocessed. First, the median filter algorithm is used to denoise the image data, remove salt and pepper noise, and improve image quality; then, the voxelization method is used to downsample the point cloud data, dividing the point cloud data into 1-meter voxel grids, retaining only one point in each grid to remove redundant information.

[0049] S2. Extract features from the image data to obtain fusion features.

[0050] S201. First, based on the image data and point cloud data, obtain data association results.

[0051] Specifically, the VGG-16 convolutional neural network is used to extract image features and extract a 4096-dimensional feature vector; the LSTM recurrent neural network is used to extract the temporal features of the point cloud data and extract a 256-dimensional feature vector. The extracted image features and point cloud features are input into the attention mechanism module, and the attention weights between the features are calculated by Scaled Dot-ProductAttention to determine the correlation between them and obtain the top-5 associated feature pairs. According to the correlation of the feature pairs, a three-layer perceptron is used to perform nonlinear transformation on the image features and point cloud features, and the 4096-dimensional image features and 256-dimensional point cloud features are mapped to a 128-dimensional unified feature space. In the unified feature space, the semantic similarity between different sensor data is calculated by cosine similarity, and data with a similarity greater than 8 are associated.

[0052] S202. Afterwards, based on the association result, the temporal dependency between different sensor data is obtained.

[0053] Specifically, according to the data association results, the GRU gated recurrent unit network is used to perform sequence modeling on the associated multimodal features, and the temporal dependencies between different sensor data are mined to obtain 256-dimensional temporal features. The temporal features are used as prior knowledge, and the image features and point cloud features after association are globally optimized and fused through the conditional random field CRF. The optimal multimodal fusion features are obtained through the Viterbi algorithm to form a 128-dimensional robust representation of the state of traffic participants for subsequent behavior prediction and risk assessment tasks.

[0054] S203. Finally, a fusion model is constructed based on the temporal dependency.

[0055] First, based on the temporal dependency, multimodal sequence features are obtained. The multimodal feature sequence includes data collected by different sensors; the temporal features are extracted from the data collected by different sensors; the temporal features are modeled using a gated recurrent unit network, and the input gate, forget gate, and output gate in the gated recurrent unit network are used to control the data input at the current moment, the retention of historical information, and the output of the current state to achieve modeling of multimodal temporal data; in the modeling process of the gated recurrent unit network, by introducing the attention mechanism, the importance weights of different modal data are dynamically adjusted according to the hidden state and context information at the current moment, highlighting the key temporal features and obtaining multimodal sequence features.

[0056] The output of the gated recurrent unit network is combined with the convolutional neural network to construct a fusion model of multimodal data, where the convolutional neural network is used to extract local features in multimodal data. The cross entropy loss function is used as the optimization target, and the network parameters of the fusion model are updated through the back propagation algorithm.

[0057] The convolutional neural network model used in the fusion model includes 3 convolutional layers and 2 pooling layers; the convolution kernel size is 3x3 and the pooling kernel size is 2x2, which is used to extract local features in multimodal data. The output of the gated recurrent unit network is concatenated with the output of the convolutional neural network, and then the features are fused through the fully connected layer to obtain the final fusion features. In the model training stage, the cross entropy loss function is used as the optimization target, the Adam optimizer is used, the learning rate is set to 0.01, the batch size is set to 32, and 100 epochs are trained.

[0058] S3. Based on the fusion features, a prediction model is constructed.

[0059] The extracted fusion features are input into a pre-trained model for learning to obtain a prediction model. Specifically, the extracted fusion features are input into a pre-trained long-term and short-term neural network model for learning to obtain a final prediction model.

[0060] S4. Using the prediction model, obtain prediction results.

[0061] S5. When the prediction results show that there is a violation of traffic regulations, an alarm is issued.

[0062] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A traffic safety warning system based on radar and visual fusion, characterized in that: include: Acquisition system, extraction module, construction module, prediction module and alarm module; The acquisition module is used to acquire and process image data in urban traffic scenes; The extraction module is used to extract features from the image data to obtain fusion features; The construction module constructs a prediction model based on the fusion features; The prediction module uses the prediction model to obtain a prediction result; The alarm module is used to issue an alarm when the prediction result shows that there is a violation of traffic regulations.

2. The traffic safety warning system based on radar and visual fusion according to claim 1 is characterized in that: The acquisition module uses a visual sensor and a radar sensor to acquire images of urban traffic scenes; then preprocesses the image data acquired by the visual sensor and the point cloud data acquired by the radar sensor. First, a median filter algorithm is used to denoise the image data, remove salt and pepper noise, and improve image quality; Then, the voxelization method is used to downsample the point cloud data, dividing the point cloud data into 1-meter voxel grids, retaining only one point in each grid to remove redundant information.

3. The traffic safety warning system based on radar and visual fusion according to claim 1 is characterized in that: The workflow of the extraction module includes: Based on the image data and point cloud data, the data association results are obtained; Based on the association result, obtaining the temporal dependency relationship between different sensor data; Based on the temporal dependency, a fusion model is constructed to extract the fusion features.

4. The traffic safety warning system based on radar and visual fusion according to claim 3 is characterized in that: The process of constructing the fusion model: based on the time series dependency, obtaining multimodal sequence features; the multimodal feature sequence includes data collected by different sensors; for the data collected by different sensors, extracting the time series features therein; The gated recurrent unit network is used to model the time series features. The input gate, forget gate and output gate in the gated recurrent unit network are used to control the data input at the current moment, the retention of historical information and the output of the current state, thus realizing the modeling of multimodal time series data. In the modeling process of the gated recurrent unit network, by introducing the attention mechanism, the importance weights of different modal data are dynamically adjusted according to the hidden state and context information at the current moment, the key time series features are highlighted, and the multimodal sequence features are obtained; The output of the gated recurrent unit network is then combined with a convolutional neural network to construct the fusion model of multimodal data, wherein the convolutional neural network is used to extract local features in the multimodal data; the cross entropy loss function is used as the optimization target, and the network parameters of the fusion model are updated through the back propagation algorithm.

5. The traffic safety warning system based on radar and visual fusion according to claim 4 is characterized in that: The convolutional neural network model includes: 3 convolutional layers and 2 pooling layers; the convolution kernel size is 3x3, and the pooling kernel size is 2x2, which are used to extract local features in multimodal data; the output of the gated recurrent unit network is spliced ​​with the output of the convolutional neural network, and then the features are fused through the fully connected layer to obtain the final fused features.

6. The traffic safety warning system based on radar and visual fusion according to claim 1 is characterized in that: The construction module inputs the extracted fusion features into the pre-trained long-term and short-term neural network model for learning to construct the final prediction model.

7. A traffic safety warning method based on radar and visual fusion, the method being applied to the system according to any one of claims 1 to 6, characterized in that the steps include: Collect and process image data in urban traffic scenes; Extracting features from the image data to obtain fusion features; Based on the fusion features, construct a prediction model; Using the prediction model, obtaining a prediction result; When the prediction results show that there is a violation of traffic regulations, an alarm is issued.

Images