Highway accident detection method, system, medium and equipment

Abstract

The invention belongs to the field of intelligent traffic, and provides a highway accident detection method, system, medium and equipment, and the method comprises the steps: carrying out the feature extraction of 4D millimeter wave radar point cloud data and image data, and obtaining a multi-scale image feature and a multi-scale local point cloud feature; performing Ro I feature extraction on the multi-scale local point cloud features and the multi-scale image features by adopting a hierarchical Ro I query strategy and a cross attention mechanism to obtain Ro I image features of each scale; performing feature fusion and global feature extraction on the multi-scale local point cloud features and all-scale Ro I image features to obtain a 3D target detection frame containing speed information; according to the method provided by the invention, the robustness of the system in all weather is enhanced by using the 4D radar, the accident detection accuracy of the system on the high-speed lane in extreme weather is improved, and the real-time efficiency of the system can also be ensured.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation and relates to a method, system, medium and equipment for detecting highway accidents based on images and 4D millimeter wave radar point clouds. Background Art

[0002] With the rapid development of science and technology, autonomous driving technology and intelligent transportation systems have become hot topics in the global transportation field. Intelligent transportation systems integrate advanced technologies such as the Internet of Things, big data, and cloud computing to collect, process, and analyze traffic information in real time to optimize traffic flow, reduce congestion, and improve traffic safety. In the intelligent transportation system, accident detection is a crucial link, which directly affects the efficiency of emergency response and the minimization of the impact of accidents.

[0003] At present, intelligent accident detection methods mainly rely on video surveillance, which performs real-time analysis and identification of accidents on images captured by cameras installed along highways. However, at night or in bad weather conditions, the effective field of view of the camera will be greatly reduced and affect the image quality, which in turn affects the accuracy of accident detection. In recent years, 4D millimeter-wave radar has gradually emerged as an emerging sensor device. Compared with traditional 3D radar, it can not only provide speed information, but also add high-resolution pitch angle measurement capabilities to obtain comprehensive 3D spatial information, which can achieve more accurate detection. At the same time, 4D radar also has the ability to work all-weather, is not affected by light and weather conditions, and can maintain stable performance even in harsh environments.

[0004] However, the current research on the fusion of image video and 4D millimeter wave radar point cloud is still in its infancy. The most commonly used fusion methods are data-level fusion and feature-level fusion. The data-level fusion method is to directly fuse the detection results of traditional radar clustering with the image detection results. The feature-level fusion method is to fuse the features of radar and camera through convolution method or attention mechanism in the neural network structure of deep learning technology. Although the above two methods use the additional information provided by radar to obtain better accident detection results, they have not yet studied the placement of radar modality in the main position of the model to better utilize the radar's all-weather detection capabilities. At the same time, the traditional method does not better integrate speed information into downstream tasks, which affects the accuracy of accident detection and prolongs the accident response time. Summary of the invention

[0005] In order to solve the technical problems of low accuracy of accident detection using images at night or in bad weather conditions, as well as low accuracy and long accident response time in the existing accident detection using images and 3D radar point clouds, the present invention discloses a highway accident detection method based on images and 4D millimeter wave radar point clouds, the method comprising the following steps:

[0006] S1. Extracting features from the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features;

[0007] S2. Using a hierarchical RoI query strategy and a cross attention mechanism to perform RoI feature extraction on the multi-scale local point cloud features and the multi-scale image features to obtain RoI image features of each scale;

[0008] S3, fusing the multi-scale local point cloud features with the RoI image features of all scales, and performing global feature extraction on the fused features to obtain a 3D target detection frame containing speed information;

[0009] S4, using a target tracking algorithm to perform multi-target tracking on the 3D target detection frame to obtain a multi-target vehicle trajectory tracking result;

[0010] S5. Perform event detection on the multi-target vehicle trajectory tracking results through speed information to obtain accident detection results.

[0011] The method of the present invention processes image features and point cloud features through a hierarchical RoI query strategy and a cross-attention mechanism to obtain multi-scale RoI image features, and realizes a multi-level RoI query strategy based on radar points, so as to better enable 4D radar to occupy a dominant position in detection, obtain all-weather detection capabilities, and improve the system operation speed. The multi-scale RoI image features are fused with point cloud features and global feature extraction is performed to obtain 3D target detection results with speed information, and finally the final result is obtained by the target tracking algorithm and the event detection algorithm in turn, which greatly reduces the computational load of the network, while improving the efficiency and accuracy of event detection, and fully reflects the better use of the radar's speed measurement capability.

[0012] An embodiment of the present invention also provides a highway accident detection system based on images and 4D millimeter-wave radar point clouds, including a feature extraction module, a RoIFPA module, a feature fusion module, a 3D target detection frame acquisition module, a target tracking module and an event detection module.

[0013] The feature extraction module is used to extract features from the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features;

[0014] The RoIFPA module is used to extract RoI features from the multi-scale local point cloud features and the multi-scale image features using a hierarchical RoI query strategy and a cross-attention mechanism to obtain RoI image features of each scale;

[0015] The feature fusion module is used to fuse the multi-scale local point cloud features with the RoI image features of all scales to obtain fused features;

[0016] The 3D target detection frame acquisition module is used to perform global feature extraction on the fused features to obtain a 3D target detection frame containing speed information;

[0017] The target tracking module is used to perform multi-target tracking on the 3D target detection frame using a target tracking algorithm to obtain a multi-target vehicle trajectory tracking result;

[0018] The event detection module is used to perform event detection on the multi-target vehicle trajectory tracking results through speed information to obtain accident detection results.

[0019] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned methods for highway accident detection based on images and 4D millimeter-wave radar point clouds is implemented to solve the technical problems of low accuracy of accident detection using images at night or in severe weather conditions, as well as low accuracy and long accident response time in the existing accident detection using images and 3D radar point clouds.

[0020] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned highway accident detection methods based on images and 4D millimeter-wave radar point clouds, so as to solve the technical problems of low accuracy of accident detection using images at night or in severe weather conditions, as well as low accuracy and long accident response time of existing accident detection using images and 3D radar point clouds.

[0021] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the highway accident detection method of the present invention, first, obtains multi-scale image features and multi-scale local point cloud features by extracting features from the collected 4D millimeter-wave radar point cloud data and image data; secondly, uses the trained RoIFPA module to perform RoI feature extraction on the multi-scale local point cloud features and multi-scale image features according to the hierarchical RoI query strategy and the cross-attention mechanism to obtain RoI image features of each scale; then, the multi-scale local point cloud features are feature fused with the RoI image features of all scales, and the fused features are subjected to global feature extraction to obtain a 3D target detection frame containing speed information; finally, the target tracking algorithm and the event detection algorithm are used to perform multi-target tracking and event detection on the 3D target detection frame to obtain an accident detection result.

[0022] The method and system of the present invention use 4D millimeter-wave radar to fill the gap in the application field of intelligent traffic accident detection, and develop an adaptive RoIFPA model for the integration of 4D radar and camera in this field. Through the RoIFPA module, the RoI image features are supplemented with the rich RGB semantic features (i.e., image features) in the camera on the basis of radar features. At the same time, a group of key radar points are used as the leading factor to ensure the accuracy of radar in all-weather detection, so that the accuracy will not be reduced due to camera degradation even in harsh conditions.

[0023] Furthermore, when acquiring RoI image features, only the RoI area is processed, which reduces the amount of data processing, improves the network's computing speed, and achieves real-time performance of the system. At the same time, the radar's speed detection capability enables the acquired trajectory to carry speed information, allowing the event detection algorithm to better perform its functions and improve the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of a highway accident detection method based on images and 4D millimeter wave radar point cloud is disclosed for an embodiment of the present invention;

[0026] Figure 2 The present invention discloses a block diagram of a highway accident detection method based on images and 4D millimeter wave radar point clouds;

[0027] Figure 3A schematic diagram of lane division when performing event detection on multi-target vehicle trajectory tracking results is disclosed in an embodiment of the present invention;

[0028] Figure 4 This is an architecture diagram of a highway accident detection system based on images and 4D millimeter wave radar point clouds disclosed in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0030] Among them, 401, feature extraction module; 402, RoI FPA module; 403, feature fusion module; 404, 3D target detection frame acquisition module; 405, target tracking module; 406, event detection module; 501, memory; 502, processor. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0033] The embodiment of the present invention discloses a highway accident detection method based on images and 4D millimeter wave radar point cloud, see Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0034] S1. Extracting features from the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features;

[0035] S2. Using a hierarchical Ro I query strategy and a cross attention mechanism to perform Ro I feature extraction on the multi-scale local point cloud features and the multi-scale image features to obtain Ro I image features of each scale;

[0036] S3, fusing the multi-scale local point cloud features with the RoI image features of all scales, and performing global feature extraction on the fused features to obtain a 3D target detection frame containing speed information;

[0037] S4, using a target tracking algorithm to perform multi-target tracking on the 3D target detection frame to obtain a multi-target vehicle trajectory tracking result;

[0038] S5. Perform event detection on the multi-target vehicle trajectory tracking results through speed information to obtain accident detection results.

[0039] The method of the present invention processes image features and point cloud features through hierarchical Ro I query strategy and cross attention mechanism to obtain multi-scale Ro I image features, and realizes a multi-level Ro I query strategy based on radar points, so as to better enable 4D radar to occupy a dominant position in detection, obtain all-weather detection capabilities, and improve the system operation speed. The multi-scale Ro I image features are fused with point cloud features and global feature extraction is performed to obtain 3D target detection results with speed information, and finally the final result is obtained by the target tracking algorithm and event detection algorithm in turn, which greatly reduces the computational load of the network, while improving the efficiency and accuracy of event detection, and fully reflects the better use of the radar's speed measurement capability.

[0040] In an optional embodiment of the above step S1, the 4D millimeter wave radar point cloud data and image data collected on the detection section of the highway at the same time can be extracted using different branches to obtain multi-scale image features and multi-scale local point cloud features, specifically including the following steps:

[0041] S11. A 4D millimeter-wave radar and a camera are set at the same position of a detection section of the highway, and the 4D millimeter-wave radar and the camera are used to collect radar point cloud data and image data at the same time.

[0042] Specifically, the 4D millimeter-wave radar and camera can be two separate devices or a radar-vision integrated device, which can be installed on the traffic poles of the detection section of the highway. After these devices are installed, they need to be synchronized with the software algorithm to ensure that the synchronized frame data collected in real time by different devices are within the time error range. In addition, the 4D millimeter-wave radar and camera need to be calibrated to ensure that the radar point cloud and image pixels can be converted to the same coordinate axis.

[0043] S12. Perform feature extraction on the image data through an image feature extraction network to obtain the multi-scale image features.

[0044] Specifically, the image feature extraction network can use the backbone network ResNet50 and the pyramid network FPN to process the image data. First, the backbone network ResNet50 is used to extract features from the image data to obtain an image feature map, and then the pyramid network FPN (Feature Pyramid Network) performs multi-level image feature extraction on the image feature map. Since FPN can upsample the small-size feature map of the deep network and add and fuse it with the feature map of the previous layer, the fused feature map contains both deeper small-scale semantic information and shallower larger-scale structural information, and thus multi-scale image features including three scales can be obtained.

[0045] S13, performing world coordinate system conversion and camera coordinate system conversion on the radar point cloud data to obtain the 4D millimeter wave radar point cloud data;

[0046] When extracting features from radar point cloud data, it is necessary to first perform coordinate conversion to ensure that the point cloud features and image feature positions are correctly matched during subsequent processing. This step converts the radar point cloud data from the radar coordinate system to the world coordinate system, and then projects it onto the image. The homogeneous coordinates P of the radar point at this time are r It can be expressed as P r =(x r ,y r ,z r ,1) T , by using the external parameter matrix T from the radar to the camera rc It can be converted to the camera coordinate system, the camera coordinate P c It can be expressed as:

[0047]

[0048] Among them, the external parameter matrix T rc Including the rotation matrix R and the translation vector t, the camera coordinates are projected onto the image plane using the camera's intrinsic parameter matrix K. The image homogeneous coordinates P on the image plane i It can be expressed as:

[0049]

[0050] Among them, the internal parameter matrix K includes the focal length f x ,f y and the image center point c x ,c y , x c ,y c 、z c are the coordinates of the point cloud in the radar point cloud data converted from the radar coordinate system to the camera coordinate system, which corresponds to the above coordinate system P c, u, v, s are the image pixel coordinates before normalization. The image pixel coordinates are (u′, v′) after normalization using the following formula:

[0051]

[0052] After coordinate transformation, the 4D millimeter-wave radar point cloud data includes the world coordinate system coordinates, image pixel coordinates and radial velocity of each point cloud, that is, the initial features of each point cloud are (x, y, z, v, rcs, u′, v′), where xyz is the coordinates of the point cloud projected onto the world coordinate system, and the radial velocity v and RCS value measured by the radar. RCS (Radar Cross Section) is a physical quantity that describes the ability of an object to reflect radar signals. It is widely used in radar systems and target detection. The unit of RCS value is square meter (m 2 ), which represents the effective reflection area of ​​an object under the irradiation of radar waves. The size of RCS directly affects the possibility of an object being detected by radar.

[0053] S14. Perform feature extraction on the 4D millimeter-wave radar point cloud data through a point cloud feature extraction network to obtain the multi-scale local point cloud features, where the multi-scale local point cloud features include each key radar point in each scale and its corresponding radar point features.

[0054] In the present invention, radar point cloud feature extraction is mainly based on the idea of ​​feature encoding. On the basis of the idea of ​​PointNet that uses the maximum pooling operation to extract unordered point cloud features, a multi-level group sampling process is introduced to gradually abstract the local features of adjacent point sets in spatial distance, and gradually concentrate discrete point features on multiple scales from small to large to achieve the capture of detailed features in the local space. At the same time, the point cloud feature extraction network of the present invention mainly adopts the network structure of PointNet++, and performs radar adaptive processing on it, so that it can extract radar speed information and RCS information.

[0055] In an optional embodiment of the above step S14, the point cloud feature extraction network is mainly composed of three parts: a sampling layer, a grouping layer, and a point network layer. Since the radar point cloud data input to the point cloud feature extraction network contains multiple point set abstraction layers, local context feature extraction from small to large sizes can be achieved. The point set abstraction layer abstracts features in three steps: sampling-grouping-point network. First, the central sampling points of the next level are sampled from the original point set through the farthest point sampling. Then, the local point sets are aggregated in a spherical area with a certain radius centered on the central sampling points. Finally, these points are input into the point network and encoded as features in the local area, thereby obtaining multi-level local features and sampling point information corresponding to the features. Specifically, the feature extraction of the 4D millimeter-wave radar point cloud data through the point cloud feature extraction network to obtain the multi-scale local point cloud features includes the following steps:

[0056] S141. According to the farthest point sampling strategy, a group of central sampling points of each layer are selected from the 4D millimeter-wave radar point cloud data through sampling layers as key radar points, each of the key radar points is used as the centroid, and a spherical radius area is given to each of the key radar points; the geometric information of the point cloud can be retained by selecting the key radar points.

[0057] S142: Based on the spherical radius area, extract the neighboring point cloud of each key radar point from the 4D millimeter wave radar point cloud data using a grouping layer to obtain a local area point cloud set of each key radar point. Due to the different radii of different layers, the area aggregated in each layer of the network becomes larger and larger from low to high.

[0058] S143. Using a point network layer, extracting context point cloud features from all the local area point clouds to obtain local area features, stacking and fusion of all the local area features layer by layer to obtain the multi-scale local point cloud features, wherein the multi-scale local point cloud features include each key radar point in each scale and its corresponding radar point features.

[0059] Through the operations of steps S141 to S143 above, the point cloud feature extraction network can learn the multi-scale local features in the 4D millimeter wave radar point cloud data, and can retain its key radar point information through sampling.

[0060] In the above step S1, if the system computing power allows, the two branches of the point cloud feature extraction network and the image feature extraction network can be calculated in parallel to improve the real-time calculation speed, and the feature maps obtained by the two branches are input into the subsequent RoIFPA module and processed by the method of the following step S2.

[0061] In an optional embodiment of the above step S2, the step of extracting RoI features from the multi-scale local point cloud features and the multi-scale image features using a hierarchical RoI query strategy and a cross attention mechanism to obtain RoI image features of each scale includes:

[0062] S21, calculating the distance between all key radar points and the camera source point in the multi-scale local point cloud feature, and dividing all the key radar points into multiple groups of sampling point sets according to the relationship between the distance and the distance threshold, each group of the sampling point sets corresponding to an image feature map of one scale in the multi-scale image feature;

[0063] S22, adopting a hierarchical RoI query strategy, using the image pixel coordinates in the sampling point set to query the image feature map of the corresponding scale, and drawing a square area in the image feature map with the queried pixel feature point as the center as the RoI area;

[0064] S23. Using a cross-attention mechanism, perform cross-attention calculation on each RoI region and its corresponding sampling point set to obtain the RoI image features of each scale.

[0065] Specifically, the key radar point in the multi-scale local point cloud feature obtained by the point cloud feature extraction network can be defined as p, and the distance d between the key radar point and the camera source point is calculated based on the coordinate information of the key radar point. Two thresholds d can be specified according to the distance. f and d n As the boundary between farther points and closer points, and corresponding to the image features of three scales, all key radar points can be divided into three groups of sampling point sets G f , G m , G n , respectively expressed as follows:

[0066] G f ={p|d>d f},G m ={p|d f ≥d≥d n},G n ={p|d>d n};

[0067] In each set of sampling points, the image pixel coordinates (u′, v′) projected from the point cloud to the image plane are used to query the image feature maps of different scales one by one. The key radar point of the feature map feature point corresponding to the queried image pixel coordinates is taken as the center / centroid, and a square area is divided into different feature map sizes as RoI, and bilinear interpolation is performed to extract features. Among them, the sampling point set with a longer distance queries the image feature map of the largest size, and the sampling point set with a larger range smax Divide RoI, and similarly use a smaller range s for the image feature map corresponding to two sampling point sets with smaller distances mid ,s min . The size of these RoIs projected into the image pixels is the same, such as a 512×512 pixel range, and features outside the image range are filled with 0. Dividing the RoI without using the entire image feature can reduce the subsequent calculation amount and improve real-time performance. At the same time, since larger feature maps have finer-grained feature details, using larger feature maps for farther targets can avoid loss due to deeper calculations. The feature extraction of the three Ro I regions is as follows:

[0068]

[0069] The features of the RoI area of ​​each group are cross-attentionally calculated with the radar point features of the corresponding group, and the RoI image features of each group can be obtained as the final output. p As the query Q in the cross attention, the coordinates of the upsampled points (i.e., key radar points) on the image plane are attached as the position embedding E p The key K and value V in the cross attention are both features of the grouped RoI area. The obtained features are then linearly transformed through the fully connected layer to obtain the final RoI image feature F fRoI ,F mROI ,F nRoI :

[0070] F fRoI ,F mROI ,F nRoI =Linear(CrossAttention(Q,K,V));

[0071] Q=[F p ; E p ]W Q ,K=F iRoI W K ,V=F iRoI W v , where W Q , W K , W v is the weight matrix.

[0072] Finally, the three grouped RoI image features are connected to obtain the final RoI image feature output by the RoIFPA module: RoI =[F fRoI ; F mROI ; F nRoI]. The present invention ensures that the features of subsequent feature fusion are dominated by radar points through the strategy of querying with key radar points, preserves the information of radar points to the greatest extent, and prevents the detection accuracy from decreasing due to camera degradation or failure under adverse conditions.

[0073] In an optional embodiment of the above step S3, the step of fusing the multi-scale local point cloud features with the RoI image features of all scales and performing global feature extraction on the fused features to obtain a 3D object detection frame containing speed information includes:

[0074] S31, fusing and connecting the local point cloud features of each scale in the multi-scale local point cloud features with the RoI image features of the corresponding scale to obtain a fused feature;

[0075] S33. Encode and decode the fused features to obtain global features, and detect the global features to obtain a 3D target detection frame containing speed information, wherein the 3D target detection frame includes the world coordinate system coordinates of each target, the detection frame size, the target speed, the target orientation, and the confidence.

[0076] In the specific implementation, the local point cloud features of each scale in the multi-scale local point cloud features and the RoI image features are fused and input into the encoder-decoder and the detection head to obtain the final detection result. The local point cloud features can be directly connected with the RoI image features through the fusion module and processed through a fully connected layer to obtain the fused features. The two are connected because the RoI image features are obtained by querying the features of the key radar points, so the features of the two parts are one-to-one corresponding. The fused features are input into the encoder-decoder. The encoder first encodes the fused features with local information into global features through multiple Transformer structures, and then inputs them into the decoder together with the target query to refine the target query features through multiple self-attention and cross-attention layers. These target query features are finally sent to the anchor-free detection head and directly regressed to obtain the 3D target detection box.

[0077] The final 3D target detection box can be expressed as (x, y, z, w, l, h, θ, v, cof), which contains the target world coordinate system coordinates (x, y, z), the detection box size (w, l, h), the target orientation θ, the target speed v and the confidence cof. The target speed can be better predicted due to the use of radar. Since the subsequent algorithm only needs to be applied to vehicle targets, this task can be regarded as a binary classification problem. Therefore, the 3D target detection box only contains information classified as the target vehicle, and no additional category information needs to be predicted. At the same time, when training the fusion network and encoder-decoder, except for the cross entropy loss for the target confidence, all other values ​​use smooth L1 loss.

[0078] In an optional embodiment of the above step S4, the using a target tracking algorithm to perform multi-target tracking on the 3D target detection frame to obtain a multi-target vehicle trajectory tracking result includes:

[0079] S41, extracting targets with confidence greater than a confidence threshold in the 3D target detection frame as target vehicles, modeling and trajectory prediction of each target vehicle through a Kalman filter algorithm, and obtaining a motion trajectory of each target vehicle;

[0080] S42. Using the Hungarian algorithm, the IoU value is calculated through the motion trajectory of each target vehicle and the predicted trajectory at the previous moment, and the target vehicles whose IoU values ​​are greater than the IoU threshold are regarded as the same target to obtain the multi-target vehicle trajectory tracking result.

[0081] In specific implementation, the IoU (Intersection over Union) value is an indicator used to evaluate the degree of overlap between the predicted box and the true box, and its value is the ratio of the intersection area of ​​the predicted box and the true box to their union area. When performing trajectory tracking, the 3D target detection box is input and the algorithm is initialized. A trajectory is established for each predicted target above the confidence threshold. Each trajectory is modeled using a Kalman filter and predicted and updated in real-time detection. When updating, the predicted trajectory is used to calculate the IoU with the current target box to establish a connection. The calculated IoU value can be compared with the IoU threshold. The one greater than the IoU threshold is regarded as the same target vehicle and is used as the observation value to update the Kalman filter of the trajectory. When a trajectory is updated without matching the target for a specified period of time, the trajectory is deleted from the target list. When a target cannot be matched with any current trajectory, it is added to the list as a new trajectory.

[0082] In an optional embodiment of the above step S5, performing event detection on the multi-target vehicle trajectory tracking result through speed information to obtain an accident detection result includes:

[0083] S51, dividing the non-emergency lane of the detection section into a plurality of lanes in a world coordinate system, and dividing each lane into a plurality of vehicle sections according to time intervals;

[0084] S52, projecting the multi-target vehicle trajectory tracking results to the corresponding segment, obtaining the minimum speed of all target vehicles in each segment, and if the minimum speed of a segment is less than a speed threshold, determining that an accident occurs in the segment, and outputting an accident detection result.

[0085] In specific implementation, when using multi-target vehicle trajectory tracking results for event detection, see Figure 3As shown, firstly, the non-emergency lanes on the highway are divided in the world coordinate system, and each lane is also divided into multiple vehicle segments according to a certain interval. The divided space only needs to cover the detection range of the equipment, and does not need to pay attention to the vertical coordinates. Then each vehicle segment S i It can be expressed as:

[0086]

[0087] Among them, (x i ,y i ) is the S of the train section i coordinate, and is the coordinate x i The upper and lower bounds of the range, and is the coordinate y i Then, the target tracking result is projected onto the vehicle section according to its plane coordinates, and we can get:

[0088]

[0089] Among them, P(S i ,T j (t)) is the time when the train segment S i Is there a function of the trajectory of j (t) is the coordinate of trajectory j at time t, including x j (t) and y j (t), from which we can calculate the segment S at a certain moment i The number of vehicles in i (t):

[0090] Among them, M and j are the total number of trajectories respectively.

[0091] When there is a target vehicle in the section at a certain moment, the lowest speed V among all the vehicle targets in the section can be calculated. min,i (t), and the speed is low-pass filtered and smoothed:

[0092]

[0093] Among them, v j (t) is the speed of trajectory j at time t. This step is mainly to detect accidents caused by single vehicle failure. When the minimum speed is calculated and the speed is lower than a threshold close to 0, it is judged that an accident has occurred in the section, thus completing the steps of the accident detection algorithm. When an accident occurs, the subsequent downstream algorithm will immediately send a report to the command center. At the same time, the device will record and save the sensor information from the beginning of this period, and upload it to the cloud for further accident processing.

[0094] The highway accident detection method of the present invention firstly extracts features from collected 4D millimeter wave radar point cloud data and image data to obtain multi-scale image features and multi-scale local point cloud features; secondly, a trained RoIFPA module is used to perform RoI feature extraction on the multi-scale local point cloud features and multi-scale image features according to a hierarchical RoI query strategy and a cross-attention mechanism to obtain RoI image features of each scale; then, the multi-scale local point cloud features are feature fused with RoI image features of all scales, and global feature extraction is performed on the fused features to obtain a 3D target detection frame containing speed information; finally, a target tracking algorithm and an event detection algorithm are used to perform multi-target tracking and event detection on the 3D target detection frame to obtain an accident detection result.

[0095] The method and system of the present invention use 4D millimeter-wave radar to fill the gap in the application field of intelligent traffic accident detection, and develop an adaptive RoI FPA model for the integration of 4D radar and camera in this field. Through the RoIFPA module, the RoI image features are supplemented with the rich RGB semantic features (i.e., image features) in the camera on the basis of radar features. At the same time, a group of key radar points are used as the leading factor to ensure the accuracy of radar in all-weather detection, so that the accuracy will not be reduced due to camera degradation even in harsh conditions.

[0096] Furthermore, when acquiring RoI image features, only the RoI area is processed, which reduces the amount of data processing, improves the network's computing speed, and achieves real-time performance of the system. At the same time, the radar's speed detection capability enables the acquired trajectory to carry speed information, allowing the event detection algorithm to better perform its functions and improve the accuracy of detection.

[0097] Based on the same inventive concept, a highway accident detection system based on images and 4D millimeter-wave radar point clouds is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the highway accident detection system is similar to the highway accident detection method disclosed in the above embodiments, the implementation of the highway accident detection system can refer to the implementation of the above-mentioned highway accident detection method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0098] Figure 4 is a structural block diagram of a highway accident detection system based on images and 4D millimeter wave radar point clouds disclosed in an embodiment of the present invention, such as Figure 4As shown, the highway accident detection system includes a feature extraction module 401, a RoIFPA module 402, a feature fusion module 403, a 3D target detection frame acquisition module 404, a target tracking module 405 and an event detection module 406, and the structure is described below.

[0099] The feature extraction module 401 is used to extract features from the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features;

[0100] The RoIFPA module 402 is used to extract RoI features from the multi-scale local point cloud features and the multi-scale image features using a hierarchical RoI query strategy and a cross-attention mechanism to obtain RoI image features of each scale;

[0101] The feature fusion module 403 is used to fuse the multi-scale local point cloud features with the RoI image features of all scales to obtain fused features;

[0102] The 3D object detection frame acquisition module 404 is used to perform global feature extraction on the fused features to obtain a 3D object detection frame containing speed information;

[0103] The target tracking module 405 is used to use a target tracking algorithm to perform multi-target tracking on the 3D target detection frame to obtain a multi-target vehicle trajectory tracking result;

[0104] The event detection module 406 is used to perform event detection on the multi-target vehicle trajectory tracking results through speed information to obtain accident detection results.

[0105] In an optional embodiment, the feature extraction module 401 includes an acquisition device, an image feature extraction network and a point cloud feature extraction network, wherein the acquisition device is a 4D millimeter wave radar and a camera set on a traffic pole of the detection section, or a radar-vision integrated device. The image feature extraction network is used to extract image pixel features from image data to obtain multi-scale image features, and the point cloud feature extraction network is used to extract point cloud features from the 4D millimeter wave radar point cloud data obtained after the coordinate system transformation to obtain multi-scale local point cloud features.

[0106] In an optional embodiment, the 3D object detection frame acquisition module 404 includes a feature fusion module, an encoder-decoder module and a detection head, wherein the feature fusion module is used to fuse and connect the local point cloud features of each scale in the multi-scale local point cloud features with the Ro I image features of the corresponding scale to obtain fused features. The encoder-decoder module is used to encode and decode the fused features to obtain global features. The detection head is used to detect the global features and output a 3D object detection frame containing speed information.

[0107] In this embodiment, a computer device is provided, such as Figure 5 As shown, it includes a memory 501, a processor 502, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned highway accident detection methods based on images and 4D millimeter-wave radar point clouds is implemented.

[0108] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0109] In this embodiment, a computer-readable storage medium is provided, which stores a computer program for executing any of the above-mentioned highway accident detection methods based on images and 4D millimeter-wave radar point clouds.

[0110] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0111] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A highway accident detection method based on images and 4D millimeter wave radar point cloud, characterized in that: include: Perform feature extraction on the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features; A hierarchical RoI query strategy and a cross-attention mechanism are used to extract RoI features from the multi-scale local point cloud features and the multi-scale image features to obtain RoI image features of each scale; The multi-scale local point cloud features are fused with the RoI image features of all scales, and global feature extraction is performed on the fused features to obtain a 3D target detection frame containing speed information; The 3D target detection frame is multi-target tracked using a target tracking algorithm to obtain a multi-target vehicle trajectory tracking result; and the multi-target vehicle trajectory tracking result is subjected to event detection using speed information to obtain an accident detection result.

2. The highway accident detection method based on image and 4D millimeter wave radar point cloud according to claim 1 is characterized in that: The feature extraction of the 4D millimeter wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features includes: A 4D millimeter wave radar and a camera are arranged at the same position of the detection section of the highway, and the radar point cloud data and image data at the same time are collected by the 4D millimeter wave radar and the camera; Extracting features from the image data through an image feature extraction network to obtain multi-scale image features; Performing world coordinate system conversion and camera coordinate system conversion on the radar point cloud data to obtain 4D millimeter wave radar point cloud data, wherein the 4D millimeter wave radar point cloud data includes world coordinate system coordinates, image pixel coordinates and radial velocity of each point cloud; The 4D millimeter-wave radar point cloud data is subjected to feature extraction through a point cloud feature extraction network to obtain the multi-scale local point cloud features, wherein the multi-scale local point cloud features include each key radar point in each scale and its corresponding radar point features.

3. The highway accident detection method based on image and 4D millimeter wave radar point cloud according to claim 2 is characterized in that: The step of extracting features from the 4D millimeter-wave radar point cloud data through a point cloud feature extraction network to obtain the multi-scale local point cloud features includes: According to the farthest point sampling strategy, the central sampling point of each layer is extracted from the 4D millimeter-wave radar point cloud data as a key radar point through the sampling layer of the point cloud feature extraction network, and each key radar point is taken as the centroid, and a spherical radius area is given to each key radar point; According to the spherical radius area, the grouping layer of the point cloud feature extraction network is used to extract the neighboring point cloud of each key radar point from the 4D millimeter wave radar point cloud data to obtain a local area point cloud set of each key radar point; The point network layer of the point cloud feature extraction network is used to extract context point cloud features from all the local area point clouds to obtain local area features, and all the local area features are stacked layer by layer and feature fused to obtain the multi-scale local point cloud features.

4. The highway accident detection method based on image and 4D millimeter wave radar point cloud according to claim 1 is characterized in that: The step of using a hierarchical RoI query strategy and a cross attention mechanism to extract RoI features from the multi-scale local point cloud features and the multi-scale image features to obtain RoI image features of each scale includes: Calculating the distance between all key radar points and the camera source point in the multi-scale local point cloud feature, and dividing all the key radar points into multiple groups of sampling point sets according to the relationship between the distance and the distance threshold, each group of the sampling point sets corresponding to an image feature map of one scale in the multi-scale image feature; A hierarchical RoI query strategy is adopted, and the image pixel coordinates of the sampling point set are used to query the image feature map of the corresponding scale, and a square area is drawn in the image feature map with the queried pixel feature point as the center as the RoI area; A cross-attention mechanism is adopted to perform cross-attention calculation on each RoI region and its corresponding sampling point set to obtain the RoI image features of each scale.

5. The method for detecting highway accidents based on images and 4D millimeter wave radar point clouds according to claim 1, characterized in that: The step of fusing the multi-scale local point cloud features with the RoI image features of all scales and extracting global features from the fused features to obtain a 3D target detection frame containing speed information includes: Performing feature fusion and connection on the local point cloud features of each scale in the multi-scale local point cloud features and the RoI image features of the corresponding scale to obtain fused features; The fused features are encoded and decoded to obtain global features, and the global features are detected to obtain a 3D object detection frame containing speed information, wherein the 3D object detection frame includes the world coordinate system coordinates of each object, the detection frame size, the object speed, the object orientation and the confidence.

6. The method for detecting highway accidents based on images and 4D millimeter wave radar point clouds according to claim 1, characterized in that: The using of the target tracking algorithm to perform multi-target tracking on the 3D target detection frame to obtain a multi-target vehicle trajectory tracking result includes: Extracting targets with confidence greater than a confidence threshold in the 3D target detection frame as target vehicles, modeling and trajectory prediction of each target vehicle through a Kalman filter algorithm, and obtaining a motion trajectory of each target vehicle; The Hungarian algorithm is used to calculate the IoU value through the motion trajectory of each target vehicle and the predicted trajectory at the previous moment, and the target vehicles with IoU values ​​greater than the IoU threshold are regarded as the same target to obtain the multi-target vehicle trajectory tracking result.

7. The method for detecting highway accidents based on images and 4D millimeter wave radar point clouds according to claim 1, characterized in that: The performing event detection on the multi-target vehicle trajectory tracking result through speed information to obtain an accident detection result includes: In a world coordinate system, the non-emergency lane of the detection section is divided into a plurality of lanes, and each lane is divided into a plurality of vehicle sections according to a time interval; The multi-target vehicle trajectory tracking results are projected to the corresponding segment, and the minimum speed of all target vehicles in each segment is obtained. If the minimum speed of a segment is less than a speed threshold, it is determined that an accident has occurred in the segment, and the accident detection result is output.

8. A highway accident detection system based on images and 4D millimeter wave radar point cloud, characterized in that: include: A feature extraction module, which is used to extract features from the 4D millimeter-wave radar point cloud data and image data collected at the same time on the detection section of the highway to obtain multi-scale image features and multi-scale local point cloud features; A RoIFPA module, wherein the RoIFPA module is used to extract RoI features from the multi-scale local point cloud features and the multi-scale image features using a hierarchical RoI query strategy and a cross-attention mechanism to obtain RoI image features of each scale; A feature fusion module, wherein the feature fusion module is used to fuse the multi-scale local point cloud features with the RoI image features of all scales to obtain fused features; A 3D target detection frame acquisition module, wherein the 3D target detection frame acquisition module is used to perform global feature extraction on the fused features to obtain a 3D target detection frame containing speed information; A target tracking module, wherein the target tracking module is used to perform multi-target tracking on the 3D target detection frame using a target tracking algorithm to obtain a multi-target vehicle trajectory tracking result; An event detection module is used to perform event detection on the multi-target vehicle trajectory tracking results through speed information to obtain an accident detection result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting highway accidents based on images and 4D millimeter-wave radar point clouds described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the highway accident detection method based on images and 4D millimeter-wave radar point clouds as described in any one of claims 1 to 7.

Images