Identification for traffic behavior

Abstract

Disclosed in the present application are a method and apparatus for identifying a traffic behavior, and a storage medium and an electronic device. The method comprises: acquiring a radar image on the basis of a collected signal of a radar sensor, and acquiring a video image on the basis of a collected signal of a video sensor; performing feature extraction on the radar image, performing feature extraction on the video image, and performing feature fusion on a feature extraction result of the radar image and a feature extraction result of the video image, so as to obtain a fused feature; performing traffic target detection on the basis of the fused feature, so as to obtain a detection result in a real physical coordinate space; performing traffic target tracking on the basis of the detection result in the real physical coordinate space, so as to generate a target queue trajectory of the traffic target; and identifying a target traffic behavior on the basis of the target queue trajectory.

Description

Traffic behavior recognition Technical Field

[0001] The present application relates to intelligent traffic control technology, and in particular to a method, device, storage medium and electronic device for identifying traffic behavior. Background Art

[0002] Traditional traffic behavior recognition methods use video image processing technology to perform feature recognition and behavior analysis on targets based on image pixels, and realize functions such as taking photos and recording target traffic behaviors.

[0003] Summary of the Invention

[0004] The present application provides a method for identifying traffic behavior, comprising: acquiring a radar image based on an acquisition signal of a radar sensor and acquiring a video image based on an acquisition signal of a video sensor; wherein the shooting angles of the radar sensor and the video sensor have an overlapping area, and the acquisition signal of the radar sensor and the acquisition signal of the video sensor correspond to the same time period; performing feature extraction on the radar image to obtain one or more first features, and performing feature extraction on the video image to obtain one or more second features, and performing feature fusion on the one or more first features and the one or more second features to obtain fused features; performing traffic target detection based on the fused features to obtain a detection result in a real physical coordinate space; performing traffic target tracking based on the detection result in the real physical coordinate space to generate a target queue trajectory of the traffic target; and identifying the target traffic behavior based on the target queue trajectory.

[0005] In some embodiments, acquiring the radar image based on the acquisition signal of the radar sensor includes: acquiring the radar image based on a radar track and the acquisition signal of the radar sensor.

[0006] In some embodiments, acquiring the radar image based on the radar track and the acquisition signal of the radar sensor includes: generating a first feature map including a first position of a pixel point, a first speed of the pixel point, and a distance of the pixel point based on a radar point cloud formed by the acquisition signal of the radar sensor; predicting and generating a second feature map including a second position of the pixel point within the time period, a second speed of the pixel point within the time period, and a confidence level of the pixel point based on the radar track; and generating the radar image based on the first feature map and the second feature map.

[0007] In some embodiments, extracting features from the radar image includes: extracting features from the radar image using a neural network including a random dropout layer.

[0008] In some embodiments, the feature fusion of the one or more first features and the one or more second features includes: processing the one or more first features using an attention network to obtain a weight coefficient of each first feature; and splicing and adding the one or more first features and the one or more second features based on the weight coefficient of each first feature.

[0009] In some embodiments, the detection result includes a target frame of the traffic target, a speed of the traffic target, and a position of the traffic target.

[0010] In some embodiments, traffic target tracking is performed based on the detection results in the real physical coordinate space, including: associating and matching the detection results for the first traffic target in the real physical coordinate space with each historical queue trajectory belonging to the first traffic target category, wherein the first traffic target belongs to the first traffic target category, and each of the historical queue trajectories represents a queue trajectory of a traffic target belonging to the first traffic target category saved before the current frame; if the match is successful, updating the historical queue trajectory by adding the detection result for the first traffic target to the historical queue trajectory with successful association and matching; if the match fails, creating a new queue trajectory corresponding to the first traffic target.

[0011] In some embodiments, after acquiring the radar image, the method further includes: converting the radar image to the video image coordinate system according to a predetermined conversion relationship between the radar coordinate space and the video image coordinate space; wherein the feature extraction, the feature fusion and the traffic target detection are all performed in the video image coordinate space; after performing traffic target detection based on the fused features, the detection result is converted to the real physical coordinate space based on a predetermined conversion relationship between the video image coordinate space and the real physical coordinate space.

[0012] In some embodiments, the method further includes: when the target traffic behavior is identified, collecting a picture containing the target traffic behavior through the video sensor.

[0013] A traffic behavior recognition device includes: an image acquisition unit, a feature fusion unit, a target detection unit, a target tracking unit, and a behavior analysis unit; the image acquisition unit is used to acquire a radar image based on a current frame acquisition signal of a radar sensor and a video image based on a current frame acquisition signal of a video sensor; wherein the position distance between the radar sensor and the video sensor is less than a set threshold; the feature fusion unit is used to perform feature extraction on the radar image to obtain one or more first features, and perform feature extraction on the video image to obtain one or more second features, and perform feature fusion on the one or more first features and the one or more second features to obtain fused features; the target detection unit is used to detect traffic targets based on the fused features and obtain detection results in a real physical coordinate space; the target tracking unit is used to track traffic targets based on the detection results in the real physical coordinate space and generate a target trajectory of the traffic target; and the behavior analysis unit is used to perform behavior analysis and recognition of the traffic target based on the target trajectory.

[0014] In some embodiments, in the image acquisition unit, acquiring the radar image based on the current frame acquisition signal of the radar sensor includes: acquiring the radar image based on a radar track and the current frame acquisition signal of the radar sensor.

[0015] In some embodiments, in the image acquisition unit, the radar image is acquired based on the radar track and the current frame acquisition signal of the radar sensor, including: generating a first feature map including a first position of a pixel point, a first speed of the pixel point, and a distance of the pixel point based on a radar point cloud formed by the acquisition signal of the radar sensor; based on the radar track, predicting and generating a second feature map including a second position of the pixel point within the time period, a second speed of the pixel point within the time period, and a confidence level of the pixel point; and generating the radar image based on the first feature map and the second feature map.

[0016] In some embodiments, in the feature fusion unit, extracting features from the radar image includes: extracting features from the radar image using a neural network including a random dropout layer.

[0017] In some embodiments, in the feature fusion unit, the feature fusion of the one or more first features and the one or more second features includes: processing the one or more first features using an attention network to obtain a weight coefficient of each first feature; and splicing and adding the one or more first features and the one or more second features based on the weight coefficient of each first feature.

[0018] In some embodiments, the detection result includes a target frame of the traffic target, a speed of the traffic target, and a position of the traffic target.

[0019] In some embodiments, in the target tracking unit, traffic target tracking is performed based on the detection results in the real physical coordinate space, including: associating and matching the detection results for the first traffic target in the real physical coordinate space with each historical queue trajectory belonging to the first traffic target category, wherein the first traffic target belongs to the first traffic target category, and each historical queue trajectory represents a queue trajectory of a traffic target belonging to the first traffic target category saved before the current frame; if the match is successful, updating the historical queue trajectory by adding the detection result for the first traffic target to the historical queue trajectory with successful association and matching; if the match fails, creating a new queue trajectory corresponding to the first traffic target.

[0020] In some embodiments, in the image acquisition unit, after acquiring the radar image, it is also used to: convert the radar image to the video image coordinate system according to a predetermined conversion relationship between the radar coordinate space and the video image coordinate space; wherein the feature extraction, the feature fusion and the traffic target detection are all performed in the video image coordinate space; after performing traffic target detection based on the fused features, the detection result is converted to the real physical coordinate space based on the predetermined conversion relationship between the video image coordinate space and the real physical coordinate space.

[0021] In some embodiments, the behavior analysis unit is further configured to instruct the video sensor to capture images containing the target traffic behavior when the target traffic behavior is identified.

[0022] A computer-readable storage medium stores computer instructions, wherein the instructions, when executed by a processor, can implement any of the above-mentioned methods for identifying traffic behaviors.

[0023] An electronic device, which includes at least a computer-readable storage medium and a processor; the processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the above-mentioned traffic behavior recognition methods.

[0024] As can be seen from the above technical solution, in this application, radar images and video images are generated based on the collected signals of the radar sensor and video sensor respectively; feature extraction is then performed on the radar image to obtain one or more first features and on the video image to obtain one or more second features, and the one or more first features and the one or more second features are feature fused to obtain fused features. This can achieve feature-level fusion of the radar image and the video image, providing more and richer image feature information as a reference for subsequent target detection and tracking, thereby improving the accuracy of target detection and tracking. In particular, in situations where environmental factors cause poor video imaging, the target detection and tracking performance will be greatly improved. Next, target detection is performed based on the fused features, and target tracking, behavior analysis, and traffic behavior recognition are performed in real physical space. In addition to improving the accuracy of target detection and tracking through the introduction of fused features as mentioned above, this part of the processing can also perform target tracking and behavior analysis in real physical space, improving the accuracy of traffic behavior recognition for occluded targets and distant targets in image space, effectively improving the accuracy of traffic behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of the basic flow of the traffic behavior recognition method in this application.

[0026] FIG2 is a schematic diagram of a specific flow chart of a traffic behavior recognition method in a specific embodiment of the present application.

[0027] FIG3a and FIG3b are schematic diagrams of the installation of the integrated radar and vision machine at a crossroads and a T-junction in an embodiment of the present application, respectively.

[0028] FIG4 is a schematic diagram showing an example of feature extraction and feature fusion of a radar image and a video image.

[0029] FIG5 is a schematic diagram of the basic structure of traffic behavior recognition in this application.

[0030] FIG6 is a schematic diagram of the basic structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical means and advantages of this application clearer, the application is further described in detail below with reference to the accompanying drawings.

[0032] Traditional traffic behavior recognition methods use video image processing technology to identify target features and analyze behavior based on image pixels, enabling features such as capturing and recording target traffic behavior. However, video images are significantly affected by lighting, weather, and environmental factors, and suffer from issues such as a narrow recognition range and an inability to perceive the target's true speed and position, significantly limiting their application. With the advancement of radar technology, millimeter-wave radar is unaffected by environmental factors like lighting and weather, and can accurately perceive the target's position and speed, with a detection range of up to 300-500 meters. Target tracking methods based on radar-visual fusion have emerged. These methods, in addition to using video images for target detection and recognition, supplement the detection and tracking of distant targets with radar sensors. However, current radar-visual fusion-based methods offer limited improvement in target detection and tracking accuracy.

[0033] The basic idea of ​​this application is to fuse radar images and video images at the feature level to provide richer reference information for target detection and target tracking, thereby improving the accuracy of traffic behavior recognition.

[0034] FIG1 is a schematic diagram of the basic flow of the traffic behavior recognition method in this application. As shown in FIG1 , the method includes steps 101 to 105.

[0035] Step 101 : Acquire a radar image based on a signal collected by a radar sensor, and acquire a video image based on a signal collected by a video sensor.

[0036] The radar sensor and video sensor's viewing angles overlap, and the radar sensor's signals and the video sensor's signals correspond to the same time period, for example, the current frame. This ensures simultaneous signal acquisition in the overlapping area. This allows radar features to be combined with video image features to form the basis for subsequent target detection and tracking.

[0037] When generating radar images, taking into account the sparsity of traffic radar point clouds and to further provide richer pixel information, optionally, in addition to using the signals directly collected by the radar sensor, radar tracks generated by historically collected signals from the radar sensor can also be combined to obtain track-related pixel information. This can, on the one hand, expand the information of existing pixel points, and on the other hand, predict and generate information of more pixel points to generate radar images.

[0038] Furthermore, since radar images and video images will be fused at the feature level later, they need to be unified into the same coordinate space. This can be done by converting the radar image into the video image coordinate space, or vice versa. Generally, radar images and video images unified into the same coordinate space are scaled to the same image size.

[0039] Step 102: Feature extraction is performed on the radar image and the video image, and the feature extraction results of the radar image and the video image are fused to obtain a fused feature. In other words, feature extraction is performed on the radar image to obtain one or more first features, and feature extraction is performed on the video image to obtain one or more second features, and the one or more first features and the one or more second features are fused to obtain a fused feature.

[0040] Feature extraction is performed on the radar image and video image to obtain radar image features and video image features. Specific feature extraction can be performed using various methods, such as using a deep convolutional neural network.

[0041] After feature extraction is complete, the feature extraction results of the radar image and the feature extraction results of the video image are fused to obtain fused features. The specific feature fusion method can be designed according to actual needs. For example, the simplest method can be directly through feature addition and splicing, or more complex fusion methods can be designed to more effectively combine radar image features with video image features.

[0042] Through the above processing, feature-level fusion of radar images and video images is achieved, which can provide more comprehensive and rich image global and detailed information, and provide richer reference for subsequent target detection and tracking.

[0043] Step 103: Perform traffic target detection based on the fusion features to obtain detection results in the real physical coordinate space.

[0044] In this application, since a rich variety of feature information for target detection can be provided, traffic target detection based on fused features can effectively improve target detection accuracy. Furthermore, since the fused feature information includes radar image features, it can reflect the characteristics of information acquired by the radar sensor. Therefore, even when video image quality is degraded due to environmental factors (such as wind, rain, snow, etc.), the feature information provided by the radar image can be used to effectively ensure the accuracy of traffic target detection.

[0045] Since this application requires identification of target traffic behaviors, the target detection performed in this step is to detect traffic targets that may cause traffic behaviors, such as motor vehicles, non-motor vehicles, etc., or, more specifically, to detect traffic targets that may cause target traffic behaviors. The target traffic behaviors here include traffic behaviors that need to be identified, such as abnormal parking, speeding, running a red light, etc.

[0046] In addition, due to the introduction of radar feature information, the detection results of traffic target detection are not limited to the traffic target frame and related parameters of the traffic target frame (such as the target center, etc.), but can also include the speed of the traffic target and the position and distance of the traffic target.

[0047] In the present application, the detection results of traffic target detection are expressed in real physical space, which can effectively mark all detected traffic targets. For example, traffic targets that cannot be marked due to occlusion in the image coordinate system can be marked. Specifically, the real physical space can adopt GPS (Global Positioning System) space, etc. It should be noted that the real physical coordinate space in this application does not include radar space. Specifically, the conversion relationship between radar coordinate space, video image coordinate space and real physical coordinate space can be obtained in advance.

[0048] Step 104 : Tracking the traffic target based on the detection results in the real physical coordinate space to generate a target queue trajectory of the traffic target.

[0049] This step is used to track traffic targets. Specifically, for each detected traffic target, the traffic target can be matched with each previously saved historical queue trajectory of each traffic target. When there is a successfully matched historical queue trajectory, that is, the traffic target matches a certain historical queue trajectory (for example, the position of the traffic target in the current frame is the next point of the historical queue trajectory), the historical queue trajectory is considered to be the historical queue trajectory of the corresponding traffic target. The trajectory information is updated by combining the historical queue trajectory and the detection results of the traffic target, and the updated historical queue trajectory is used as the target queue trajectory of the traffic target. Among them, the historical queue trajectory of the traffic target is previously obtained by looping multiple times based on the collected signals of the radar sensor and the video sensor according to the method of the present application. Therefore, the traffic target in the historical queue trajectory is also in the real physical coordinate space and is obtained by historical fusion features. The historical queue trajectory of the traffic target also includes rich and diverse information provided by the radar image and the video image.

[0050] Traffic target detection results are more accurate because they are determined by fusion features extracted from radar and video images. These fusion features also incorporate rich radar information, such as the target's speed, location, and distance. This fusion allows for more accurate traffic target matching and, consequently, for more accurate target queue trajectories.

[0051] Step 105 : Identify the target traffic behavior based on the target queue trajectory.

[0052] Target queue trajectories can be used to analyze traffic target behavior and identify target traffic behaviors. Target queue trajectories in real physical space can reflect more comprehensive target information, making target traffic behavior analysis based on target queue trajectories more accurate and effectively improving the accuracy of target traffic behavior identification.

[0053] This concludes the basic process of the traffic behavior recognition method shown in Figure 1 of this application. Through the above process, the introduction of radar images can overcome the interference of environmental factors, and feature-level fusion of radar images and video images can be used to provide richer feature information for traffic target detection, traffic target tracking, and traffic behavior recognition, thereby effectively improving the recognition accuracy of target traffic behavior.

[0054] The traffic behavior recognition method in this application is described below through specific examples.

[0055] FIG2 is a schematic diagram of a specific flow chart of the traffic behavior recognition method in an embodiment of the present application. In this embodiment, it is assumed that a radar-visual integrated machine is installed on a pole at an intersection or T-junction. The specific installation position can be as shown in FIG3a and FIG3b. The radar-visual integrated machine is equipped with a radar sensor, a video sensor, and a control unit (such as a central processing unit). The radar sensor is used to transmit and receive radar signals, and the video sensor is used to collect image signals. There is an overlapping area between the shooting angles of the radar sensor and the video sensor. The method of this embodiment is implemented in the control unit. The radar-visual integrated machine can simultaneously cover an area of ​​4 lanes in the near and 8 lanes in the far, and realize vehicle track tracking and behavior analysis and traffic behavior recognition functions within the covered area. In this embodiment, the processing of the radar acquisition signal and the acquisition signal of the video sensor of the current frame is used as an example for explanation. As shown in FIG2, the traffic behavior recognition method in this embodiment includes steps 201 to 210.

[0056] Step 201 : predetermine the conversion relationship between the radar coordinate space, the video image coordinate space, and the real physical coordinate space.

[0057] Because feature-level fusion of radar images and video images is required in this application, the radar images and video images need to be unified into the same coordinate space. Therefore, the transformation relationship between the radar coordinate space and the video image coordinate space needs to be pre-determined. Specifically, the radar trajectory of a traffic target in the radar space can be matched with the visual trajectory of the traffic target in the video image space to determine the radar trajectory and visual trajectory corresponding to the same traffic target. Based on the relationship between the two trajectories, the mapping transformation coefficients between the radar coordinate space and the video image coordinate space are calculated.

[0058] After obtaining the traffic target detection results, they need to be converted to a real physical coordinate space (GPS coordinate space is used in this embodiment). Therefore, it is necessary to determine the mutual conversion relationship between the radar space and the video image space and the real physical coordinate space. Specifically, multiple pixels corresponding to known landmark locations (such as landmark buildings) that are not collinear can be selected on the captured video image. The physical coordinates corresponding to the same locations can be found on a high-precision GPS (Global Position System) map. Based on the correspondence between these multiple sets of pixels and physical coordinates, the perspective transformation coefficients between the video image coordinate space and the GPS coordinate space can be determined.

[0059] After determining the conversion relationship between the radar coordinate space and the video image coordinate space, as well as the conversion relationship between the video image coordinate space and the GPS coordinate space, the rotation and translation relationship matrix of the radar coordinate space and the GPS coordinate space can be obtained by combining the above conversion relationships, and finally the mutual mapping coefficients of the three can be obtained.

[0060] Step 202: Acquire a video image generated based on the acquisition signal of the video sensor in the current frame.

[0061] The processing in this step can be performed in a conventional manner.

[0062] Step 203: Obtain a radar image based on the collected signal of the radar sensor in the current frame.

[0063] In this embodiment, a radar image may be generated based on the radar point cloud and the radar track. Specific processing may include:

[0064] 1) obtaining a radar point cloud formed by a signal collected by the radar sensor in a current frame, and generating a first feature map including a first position of a pixel point, a first velocity of the pixel point, and a distance of the pixel point;

[0065] Based on the collected signal of the radar sensor in the current frame, the radar point cloud can be directly determined, and the position, speed and distance of each pixel point on the point cloud can be determined. Among them, the position of the pixel point refers to the coordinate value of the point on the point cloud in the radar coordinate space. In order to distinguish it from the position information of the pixel point determined based on the radar track, this application refers to the position of the pixel point obtained based on the point cloud as the first position. The speed of the pixel point refers to the current speed of the point on the point cloud. In order to distinguish it from the speed information of the pixel point determined based on the radar track, this application refers to the speed of the pixel point obtained based on the point cloud as the first speed. The distance of the pixel point refers to the distance between the point on the point cloud and the radar sensor. Based on the above information of each pixel point on the point cloud, the first feature map in the radar coordinate space is generated in combination.

[0066] 2) Based on the radar track, a second feature map including a second position of the pixel point in the current frame, a second speed of the pixel point, and a confidence level is predicted and generated.

[0067] The traffic radar used in this application has a relatively long-range detection capability (300-500m), which also results in a relatively sparse point cloud. Therefore, the radar information determined based on the point cloud is relatively limited. Based on this, the radar track is further used in this embodiment to supplement the radar information.

[0068] The radar track is obtained based on the historical point cloud information of the radar sensor, and various methods can be used for specific determination. The radar track here refers to the latest radar track before the current frame. Based on the radar track, the position, speed and confidence of each pixel point related to the radar track in the current frame can be predicted. In this application, the position of the pixel point determined based on the radar track is referred to as the second position, and the speed of the pixel point determined based on the radar track is referred to as the second speed. Based on the predicted second position, second speed and confidence of the relevant pixel points, a second feature map is generated in the radar coordinate space.

[0069] The radar information can be supplemented by the above-mentioned second feature map. For pixel points that do not directly obtain information in the current frame, the corresponding pixel point information can be obtained through processing and prediction of the radar track. For pixel points that have already obtained corresponding pixel point information in the current frame, supplementary information (second position, second speed and confidence) can be obtained through processing and prediction of the radar track.

[0070] 3) Generate a radar image based on the first feature map and the second feature map.

[0071] The information from the first and second feature maps is combined to generate a radar image, which is actually a radar pseudo image. Specifically, for each pixel, six dimensions of information are obtained: first position, first velocity, distance, second position, second velocity, and confidence level. This generates a six-dimensional radar pseudo image, or radar image.

[0072] In this embodiment, radar images and video images are uniformly represented in the video image coordinate space. Based on the conversion relationship between the radar coordinate space and the video image coordinate space, the radar image can be converted to the video image coordinate space and scaled to match the size of the video image. This process is primarily based on the fact that video images can provide richer information.

[0073] Step 204 : extract features from the video image to obtain feature extraction results of the video image, such as one or more second features.

[0074] Video image feature extraction can be performed in various ways, such as using a deep convolutional network model based on ResNet-50.

[0075] Step 205 : performing feature extraction on the radar image to obtain a feature extraction result of the radar image, such as one or more first features.

[0076] Radar image feature extraction can also be performed in various ways, such as using a deep convolutional network model based on ResNet-50.

[0077] When extracting radar features, a random dropout layer can be introduced to mitigate overfitting issues that can arise from sparse point clouds and improve feature generalization. For example, a neural network with a random dropout layer can be used to extract features from radar images. To address the potential overfitting issue caused by sparse radar point clouds, introducing a random dropout layer to filter radar images before feature extraction can improve feature generalization.

[0078] Step 206 : Fusing the feature extraction results of the radar image and the feature extraction results of the video image to obtain fused features.

[0079] Optionally, in this embodiment, when fusing radar image features with video image features, an attention network is introduced to determine weight coefficients for each feature in the radar image feature extraction results. Based on these weight coefficients, the radar image feature extraction results and the video image feature extraction results are fused together by addition and concatenation to obtain fused features. In other words, the attention network is used to process the one or more first features (radar image features) to obtain weight coefficients for each first feature; based on the weight coefficients for each first feature, the one or more first features and the one or more second features (video image features) are concatenated and added.

[0080] Figure 4 shows an example of feature extraction and feature fusion of a radar image and a video image. Of course, this figure is only an example, and the image feature extraction and feature fusion processing of the present application is not limited to this.

[0081] Step 207 : Perform traffic target detection based on the fusion features to obtain detection results in the real physical coordinate space.

[0082] This step detects traffic objects based on the fused features, obtaining predicted results for the traffic object's frame location and category. Depending on the needs, only traffic objects can be detected and the detection results output. Before performing traffic object detection, the fused features can be further extracted for deeper semantic information before performing object detection. Traffic object detection can be performed using various methods, such as a typical object detection network.

[0083] Since the traffic target detection in this application is based on fusion features, the detection result information obtained is richer. The detection result may include the target frame of the traffic target, the speed of the traffic target and the position of the traffic target in the real physical space. It may also include other information of the traffic target. Assuming that the traffic target is a vehicle, it may also include the vehicle posture, vehicle model, etc.

[0084] Furthermore, to fully characterize the detection results obtained from traffic target detection based on fused features and for subsequent target tracking, the detection results are represented in a real physical coordinate space. Specifically, in this embodiment, the aforementioned feature extraction process is performed in the image coordinate space, and therefore the detection results obtained here are also in the image coordinate space. The detection results need to be converted to the GPS coordinate space. More specifically, the detection results can be converted to the GPS coordinate space based on the conversion relationship between the image coordinate space and the GPS coordinate space obtained in step 201.

[0085] Step 208 : Track the traffic target based on the detection result in the real physical coordinate space to generate a target queue trajectory of the traffic target.

[0086] This step is used to track the detected traffic targets in real time and record the relevant attributes of the traffic targets. Assuming that the traffic targets are vehicles, the relevant attributes of the vehicles can be recorded, such as vehicle ID (identifier), license plate number, vehicle location (for example, pixel location and GPS location), vehicle speed, vehicle queue trajectory and other information.

[0087] Specifically, as previously described, the traffic target detection result obtained by processing the current frame signal may include multiple traffic target frames, i.e., corresponding to multiple traffic targets (each traffic target frame corresponds to a traffic target). Traffic target tracking is performed separately for each of the multiple detected traffic targets, and the traffic target tracking process performed for each traffic target is identical. The following uses the tracking process for a single traffic target (hereinafter referred to as the first traffic target) as an example to illustrate the tracking process, which specifically includes the following two steps.

[0088] 1) Classify the first traffic target and determine the traffic target category to which it belongs. This traffic target category is referred to as the first traffic target category. The traffic target category may include a large category, such as motor vehicles, non-motor vehicles, etc., or a sub-category, such as trucks, cars, urban SUVs, etc.

[0089] 2) Based on the target detection result for the first traffic target, the first traffic target is associated and matched with the queue trajectories of each traffic target belonging to the first traffic target category. In other words, the detection result for the first traffic target in the real physical coordinate space is associated and matched with each historical queue trajectories belonging to the first traffic target category, where the first traffic target belongs to the first traffic target category, and each historical queue trajectory represents a queue trajectory of a traffic target belonging to the first traffic target category that was saved before the current frame.

[0090] The queue trajectory of a traffic target is the trajectory information of the traffic target obtained by synthesizing the target detection results for the same traffic target obtained before the current frame. Therefore, one queue trajectory corresponds to one traffic target. Here, the first traffic target needs to be individually correlated and matched with each of the historical queue trajectories of the same type to determine whether the first traffic target is identical to the traffic target corresponding to the historical queue trajectory. If so, the correlation match is considered successful; otherwise, the correlation match is considered unsuccessful. Specifically, to determine whether the first traffic target is identical to the traffic target corresponding to the historical queue trajectory (hereinafter referred to as the second traffic target), the similarity between the first traffic target and the traffic target corresponding to the historical queue trajectory can be compared. The higher the similarity, the more likely the two traffic targets are the same target.

[0091] More specifically, the similarity between the target detection result of the first traffic target and the target frame, position, speed, license plate, and other information of the second traffic target can be calculated one by one. If the similarity meets the threshold T, the two traffic targets are determined to be the same, meaning that the traffic target association matching is successful. Otherwise, the two traffic targets are determined to be different, meaning that the target association matching fails. Unlike association matching in image space, which can only use target frames for association matching, in GPS coordinate space, similarity matching can be performed by combining multiple perceptual information such as target frames, center points, speed, posture, and license plates output by feature-level fusion. Kalman filters are then used to perform temporal association and prediction of previous and next frames, effectively improving the accuracy of traffic target association matching.

[0092] When a first traffic target is successfully associated with a queue trajectory, the traffic target information for the current frame in the queue trajectory is updated. For example, the latest location, speed, and other information corresponding to the historical queue trajectory are updated. This updated historical queue trajectory becomes the queue trajectory of the first traffic target. Based on the historical information of the queue trajectory and the target detection results of the first traffic target, a comprehensive determination can be made as to whether the first traffic target's information needs to be updated. For example, information such as the license plate and vehicle model can be modified based on the results of multiple frames. In other words, when a first traffic target is successfully associated with a queue trajectory, the historical queue trajectory is updated by adding the detection results for the first traffic target to the successfully associated and matched historical queue trajectory.

[0093] When the first traffic target fails to be associated and matched with all similar queue trajectories, a new queue trajectory corresponding to the first traffic target is created, and queue trajectory related information of the first traffic target is added to the queue trajectory.

[0094] According to the above method, each traffic target corresponding to the target detection result is tracked in turn, and a queue trajectory corresponding to each traffic target is generated.

[0095] Step 209 : Identify the target traffic behavior based on the target queue trajectory.

[0096] Through the above processing, the trajectory of each target queue can be obtained. Based on the target queue trajectory, the behavior of the traffic target is analyzed in the GPS coordinate space to achieve the detection and identification of the target traffic behavior.

[0097] The specific behavior analysis method can be determined based on the target traffic behavior to be identified.

[0098] Assuming the target traffic behavior is "parking in a no-parking zone," the behavior analysis method can be as follows: The parking status of the traffic target is judged based on the speed, position, and other information of the target queue trajectory. For example, if the speed of the traffic target approaches 0 and the center point position of the traffic target remains relatively unchanged, the vehicle can be judged to be stationary. The no-parking area configured in the image coordinate space is converted to the GPS coordinate space and compared with the area where the traffic target is judged to be in a parking state to determine whether the target traffic behavior "parking in a no-parking zone" is established.

[0099] For the same target traffic behavior, when identifying it based on the video image coordinate space, the traffic target frame may jitter when determining whether the vehicle is parked based on its position in the video image coordinate system due to interference from lighting, surrounding vehicle lights, and streetlights. Therefore, it is necessary to determine whether the traffic target frame remains relatively stationary for a certain period of time to determine whether the vehicle is illegally parked. This means that an alarm will only be triggered some time after the vehicle has parked. Furthermore, due to video imaging distortion, even if a traffic target moves dozens of meters away from the camera, it will only move a few pixels in the image coordinate space, which can easily lead to misjudgment and affect the accuracy of reporting the target traffic behavior.

[0100] In contrast, the present application can judge the stationary state of traffic targets based on radar speed, avoiding misjudgment due to jitter of traffic target frames; for the related technology, in the video image coordinate space, the image trajectory displayed is close in distance range, and in the distant real space, it is impossible to judge whether the traffic target is in motion based on the video frame due to camera distortion. In the present application, the accuracy of identifying long-distance trajectories is improved through the radar-vision fusion feature, and is not affected by video imaging distortion. Therefore, the range of event detection is greatly improved, and the traffic target capture performance is improved. The capture rate and accuracy of events within 300m from the radar-vision integrated machine reach 98%.

[0101] In summary, in the identification of target traffic behaviors based on video images, video image acquisition will be "blind" without fill light or exposure to car lights, and it does not have the ability to accurately measure speed. However, in this application, target traffic behavior identification is performed based on radar feature results, which can achieve all-weather and all-time, wider-range, and higher-accuracy traffic behavior identification, and can determine the actual speed and physical position of the traffic target.

[0102] Step 210 : When the target traffic behavior is identified, a video sensor is used to capture a picture containing the target traffic behavior.

[0103] After identifying the target traffic behavior, the video sensor can also be used to capture images containing the target traffic behavior and retain them as evidence.

[0104] At this point, the process of the traffic behavior recognition method in the embodiment shown in Figure 2 ends. In this embodiment, the real physical coordinate space is the GPS coordinate space. In fact, it can also be other real physical coordinate spaces, such as the Beidou coordinate space.

[0105] As described above, this embodiment utilizes radar point cloud, radar track, and video image data, combined with feature-level fusion and target detection processing, to integrate radar features onto image features to achieve traffic target location and identification. Specifically, given the sparse nature of traffic radar point clouds, processing can only be performed based on single-frame radar images. Therefore, in addition to using point clouds, radar tracks are incorporated to assist in generating radar features. Selecting radar images as the primary sensor for feature fusion can further improve the detection performance of distant targets. In situations where video imaging is poor, integrating radar images can significantly enhance traffic target detection and tracking performance. Traffic target tracking and behavior analysis is achieved in real physical coordinate space. By integrating radar images, the accuracy of traffic target trajectory recognition can be improved, along with the capture rate and efficiency of event analysis, particularly at night, in long-distance scenarios, and for events requiring high speed. The capture rate and accuracy of target traffic behavior in this embodiment can reach 98%, further enhancing the application value of target traffic behavior recognition equipment in unmanned intersection management.

[0106] The above is a specific implementation of the traffic behavior recognition method in this application. This application also provides a traffic behavior recognition device that can be used to implement the above-mentioned traffic behavior recognition method. Figure 5 is a schematic diagram of the basic structure of the traffic behavior recognition device in this application. As shown in Figure 5, the device includes: an image acquisition unit 501, a feature fusion unit 502, a target detection unit 503, a target tracking unit 504, and a behavior analysis unit 505.

[0107] An image acquisition unit 501 is configured to acquire a radar image based on a current frame acquisition signal of a radar sensor and a video image based on a current frame acquisition signal of a video sensor; wherein a distance between the radar sensor and the video sensor is less than a set threshold;

[0108] a feature fusion unit 502 configured to perform feature extraction on the radar image to obtain one or more first features, perform feature extraction on the video image to obtain one or more second features, and perform feature fusion on the one or more first features and the one or more second features to obtain a fused feature;

[0109] The target detection unit 503 is used to detect traffic targets based on the fusion features and obtain detection results in a real physical coordinate space;

[0110] A target tracking unit 504 is configured to track a traffic target based on the detection result in the real physical coordinate space and generate a target trajectory of the traffic target;

[0111] The behavior analysis unit 505 is used to perform behavior analysis and recognition of traffic targets based on the target trajectory.

[0112] In some embodiments, the image acquisition unit 501 acquires the radar image based on the current frame acquisition signal of the radar sensor, including: acquiring the radar image based on the radar track and the current frame acquisition signal of the radar sensor.

[0113] In some embodiments, in the image acquisition unit 501, the radar image is acquired based on the radar track and the current frame acquisition signal of the radar sensor, including: generating a first feature map including a first position of a pixel point, a first speed of the pixel point, and a distance of the pixel point based on a radar point cloud formed by the acquisition signal of the radar sensor; predicting and generating a second feature map including a second position of the pixel point within the time period, a second speed of the pixel point within the time period, and a confidence level of the pixel point based on the radar track; and generating a radar image based on the first feature map and the second feature map.

[0114] In some embodiments, in the feature fusion unit 502 , extracting features from the radar image includes: extracting features from the radar image using a neural network including a random dropout layer.

[0115] In some embodiments, in the feature fusion unit 502, the feature fusion of the one or more first features and the one or more second features includes: processing the one or more first features using an attention network to obtain a weight coefficient of each first feature; and splicing and adding the one or more first features and the one or more second features based on the weight coefficient of each first feature.

[0116] In some embodiments, the detection result includes a target frame of the traffic target, a speed of the traffic target, and a position of the traffic target.

[0117] In some embodiments, in the target tracking unit 504, traffic target tracking is performed based on the detection results in the real physical coordinate space, including: associating and matching the detection results for the first traffic target in the real physical coordinate space with each historical queue trajectory belonging to the first traffic target category, wherein the first traffic target belongs to the first traffic target category, and each historical queue trajectory represents a queue trajectory of a traffic target belonging to the first traffic target category saved before the current frame; if the match is successful, updating the historical queue trajectory by adding the detection result for the first traffic target to the historical queue trajectory with successful association and matching; if the match fails, creating a new queue trajectory corresponding to the first traffic target.

[0118] In some embodiments, in the image acquisition unit 501, after acquiring the radar image, it is also used to: convert the radar image into the video image coordinate system according to a predetermined conversion relationship between the radar coordinate space and the video image coordinate space; wherein the feature extraction, the feature fusion and the traffic target detection are all performed in the video image coordinate space; after performing traffic target detection based on the fused features, the detection result is converted into the real physical coordinate space based on the predetermined conversion relationship between the video image coordinate space and the real physical coordinate space.

[0119] In some embodiments, the behavior analysis unit 505 is further configured to instruct the video sensor to capture images containing the target traffic behavior when the target traffic behavior is identified.

[0120] This application also provides a computer-readable storage medium storing instructions that, when executed by a processor, can perform the steps of the above-described method for identifying traffic behavior. In practical applications, the computer-readable medium can be included in the devices / apparatuses / systems described in the above-described embodiments, or it can exist independently and not be incorporated into the devices / apparatuses / systems. The computer-readable storage medium stores instructions that, when executed by a processor, can perform the steps of the above-described method for identifying traffic behavior.

[0121] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but is not intended to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] FIG6 is an electronic device provided in the present application. As shown in FIG6 , it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0123] The electronic device may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 602 is executed, the method for traffic behavior recognition may be implemented.

[0124] Specifically, in actual applications, the electronic device may further include components such as a power supply 603 and an input / output unit 604. Those skilled in the art will appreciate that the structure of the electronic device shown in FIG6 does not limit the electronic device, and the electronic device may include more or fewer components than shown, or may combine certain components, or arrange the components differently.

[0125] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, it executes various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0126] The memory 602 can be used to store software programs and modules, i.e., the computer-readable storage medium mentioned above. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created based on the use of the server, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0127] The electronic device also includes a power supply 603 for supplying power to various components. This power supply 603 can be logically connected to the processor 601 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 603 can also include any of one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other components.

[0128] The electronic device may further include an input / output unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, or optical signal input related to user settings and function control. The input / output unit 604 may also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which may be composed of graphics, text, icons, videos, or any combination thereof.

[0129] The above are some embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for identifying traffic behavior, comprising: Acquire a radar image based on a collection signal of a radar sensor and acquire a video image based on a collection signal of a video sensor; wherein the shooting angles of the radar sensor and the video sensor have an overlapping area, and the collection signal of the radar sensor and the collection signal of the video sensor correspond to the same time period; Performing feature extraction on the radar image to obtain one or more first features, and performing feature extraction on the video image to obtain one or more second features, and performing feature fusion on the one or more first features and the one or more second features to obtain fused features; Perform traffic target detection based on the fusion features to obtain detection results in a real physical coordinate space; Tracking a traffic target based on the detection result in the real physical coordinate space to generate a target queue trajectory of the traffic target; Based on the target queue trajectory, target traffic behavior is identified.

2. The method according to claim 1, wherein: Acquiring the radar image based on the collected signal of the radar sensor includes: The radar image is acquired based on the radar track and the acquisition signal of the radar sensor.

3. The method according to claim 2, wherein: Acquiring the radar image based on the radar track and the acquisition signal of the radar sensor includes: Generate a first feature map including a first position of a pixel point, a first speed of the pixel point, and a distance of the pixel point based on a radar point cloud formed by the collected signal of the radar sensor; Based on the radar track, predict and generate a second feature map including a second position of the pixel point within the time period, a second speed of the pixel point within the time period, and a confidence level of the pixel point; The radar image is generated based on the first feature map and the second feature map.

4. The method according to claim 1, wherein: Extracting features from the radar image includes: Feature extraction is performed on the radar image using a neural network including a random dropout layer.

5. The method according to any one of claims 1 to 4, wherein: The fusing the one or more first features with the one or more second features includes: Processing the one or more first features using an attention network to obtain a weight coefficient of each of the first features; Based on the weight coefficients of the first features, the one or more first features and the one or more second features are concatenated and added.

6. The method according to any one of claims 1 to 5, wherein: The detection result includes a target frame of the traffic target, a speed of the traffic target and a position of the traffic target.

7. The method according to any one of claims 1 to 6, wherein: Tracking a traffic target based on the detection result in the real physical coordinate space includes: Associating and matching the detection result for the first traffic target in the real physical coordinate space with each historical queue trajectory belonging to the first traffic target category, wherein the first traffic target belongs to the first traffic target category, and each of the historical queue trajectories represents a queue trajectory of a traffic target belonging to the first traffic target category saved before the current frame; If the match is successful, then by adding the detection result for the first traffic target to the associated matching success Historical queue trajectory to update the historical queue trajectory; If the match fails, a new queue trajectory corresponding to the first traffic target is created.

8. The method according to any one of claims 1 to 7, wherein: After acquiring the radar image, the method further includes: converting the radar image into the video image coordinate system according to a predetermined conversion relationship between the radar coordinate space and the video image coordinate space; Wherein, the feature extraction, the feature fusion and the traffic target detection are all performed in the video image coordinate space; After performing traffic target detection based on the fusion feature, the detection result is converted into the real physical coordinate space based on a predetermined conversion relationship between the video image coordinate space and the real physical coordinate space.

9. The method according to claim 8, further comprising: When the target traffic behavior is identified, the video sensor collects a picture containing the target traffic behavior.

10. A traffic behavior recognition device, comprising: Image acquisition unit, feature fusion unit, target detection unit, target tracking unit and behavior analysis unit; The image acquisition unit is used to acquire a radar image based on a current frame acquisition signal of the radar sensor and to acquire a video image based on a current frame acquisition signal of the video sensor; wherein the position distance between the radar sensor and the video sensor is less than a set threshold; The feature fusion unit is used to perform feature extraction on the radar image to obtain one or more first features, perform feature extraction on the video image to obtain one or more second features, and perform feature fusion on the one or more first features and the one or more second features to obtain fused features; The target detection unit is used to perform traffic target detection based on the fusion feature to obtain a detection result in a real physical coordinate space; The target tracking unit is used to track the traffic target based on the detection result in the real physical coordinate space to generate a target trajectory of the traffic target; The behavior analysis unit is used to perform behavior analysis and identification of traffic targets based on the target trajectory.

11. The device according to claim 10, wherein: In the image acquisition unit, acquiring the radar image based on the current frame acquisition signal of the radar sensor includes: acquiring the radar image based on the radar track and the current frame acquisition signal of the radar sensor.

12. The device according to claim 11, wherein In the image acquisition unit, acquiring the radar image based on the radar track and the current frame acquisition signal of the radar sensor includes: Generate a first feature map including a first position of a pixel point, a first speed of the pixel point, and a distance of the pixel point based on a radar point cloud formed by the collected signal of the radar sensor; Based on the radar track, predict and generate a second feature map including a second position of the pixel point within a time period, a second speed of the pixel point within the time period, and a confidence level of the pixel point; The radar image is generated based on the first feature map and the second feature map.

13. The device according to claim 10, wherein: In the feature fusion unit, feature extraction is performed on the radar image, including: extracting features from the radar image using a neural network including a random discard layer.

14. The device according to any one of claims 10 to 13, wherein: In the feature fusion unit, the step of fusing the one or more first features with the one or more second features includes: Processing the one or more first features using an attention network to obtain a weight coefficient of each of the first features; Based on the weight coefficients of the first features, the one or more first features and the one or more second features are concatenated and added.

15. The device according to any one of claims 10 to 14, wherein: The detection result includes a target frame of the traffic target, a speed of the traffic target and a position of the traffic target.

16. The device according to any one of claims 10 to 15, wherein: In the target tracking unit, tracking of the traffic target is performed based on the detection result in the real physical coordinate space, including: Associating and matching the detection result for the first traffic target in the real physical coordinate space with each historical queue trajectory belonging to the first traffic target category, wherein the first traffic target belongs to the first traffic target category, and each of the historical queue trajectories represents a queue trajectory of a traffic target belonging to the first traffic target category saved before the current frame; If the match is successful, updating the historical queue trajectory by adding the detection result for the first traffic target to the historical queue trajectory associated with the successful match; If the match fails, a new queue trajectory corresponding to the first traffic target is created.

17. The device according to any one of claims 10 to 16, wherein: In the image acquisition unit, after acquiring the radar image, it is also used to: convert the radar image into the video image coordinate system according to a predetermined conversion relationship between the radar coordinate space and the video image coordinate space; Wherein, the feature extraction, the feature fusion and the traffic target detection are all performed in the video image coordinate space; After performing traffic target detection based on the fusion feature, the detection result is converted into the real physical coordinate space based on a predetermined conversion relationship between the video image coordinate space and the real physical coordinate space.

18. The device according to claim 17, wherein: The behavior analysis unit is further configured to instruct the video sensor to capture a picture containing the target traffic behavior when the target traffic behavior is identified.

19. A computer-readable storage medium having computer instructions stored thereon, wherein: When the instructions are executed by a processor, the traffic behavior recognition method described in any one of claims 1 to 9 can be implemented.

20. An electronic device, wherein: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the traffic behavior recognition method described in any one of claims 1 to 9.

Images