A traffic behavior detection system based on visual recognition

Abstract

The application discloses a traffic behavior detection system based on visual recognition and belongs to the technical field of traffic, and solves the problem that when road traffic visual detection encounters rare special conditions, because a deep learning model relies on training data distribution and such samples are almost zero, target detection fails, behavior prediction deviates, complex working conditions are superimposed to amplify perception errors, and high-risk decisions such as emergency braking or blind lane changing are caused. A multi-modal data acquisition module is used to acquire image data, millimeter wave radar point cloud data and laser radar point cloud data. Through multi-modal data space-time synchronization and preprocessing, the application evaluates scene complexity and dynamically fuses features, detects targets, identifies unknown targets to generate rarity, constructs a causal diagram to perform counterfactual reasoning, predicts a track, comprehensively calculates a risk score to generate a decision and an evidence chain, and returns a difficult example to update a model, so that robust detection of a target and risk decision in a complex scene are realized.

Description

Technical Field

[0001] This invention relates to the field of traffic technology, and in particular to a traffic behavior detection system based on visual recognition. Background Technology

[0002] The vision-based traffic behavior detection system is an intelligent system that deeply integrates artificial intelligence, computer vision, and traffic engineering. It collects road monitoring video through cameras deployed at intersections, road sections, and on drones. Using deep learning algorithms, it processes the video stream continuously around the clock, performing target detection, cross-frame tracking, and behavioral feature analysis of traffic participants. This accurately identifies the movement trajectories and behavioral patterns of vehicles and pedestrians, determining whether they comply with traffic rules. The system can automatically identify explicit violations such as running red lights, speeding, and illegal parking, as well as implicit high-risk behaviors such as driver distraction and abnormal vehicle delays. It simultaneously performs data preprocessing and intelligent analysis, generating complete enforcement evidence and issuing timely warnings. Its applications cover multiple scenarios, including intelligent enforcement on urban roads, highway incident detection, public transportation monitoring, and drone-based mobile enforcement. It can replace manual monitoring, solving the problems of low efficiency and easy omissions in traditional monitoring, and realizing a shift in traffic management from passive response to proactive perception, effectively improving the level of intelligent traffic management and road safety.

[0003] However, visual recognition in road traffic detection encounters various rare and unusual situations. For example, live pigs transported on highways may fall from their containers and dart through traffic in panic; pedestrians dressed in inflatable costumes may suddenly appear from blind spots on holiday nights; or irregularly shaped large boulders and deep sinkholes may form on the road after a geological disaster. These scenarios are difficult to identify because deep learning models heavily rely on the distribution of training data. Such events are almost nonexistent in historical samples, making it difficult for object detection algorithms to extract effective features. This easily leads to misclassifying live animals as scattered goods or classifying unusual pedestrians as unknown backgrounds and ignoring them. Furthermore, behavioral prediction models often lack... The semantic gap arising from the understanding of random motion or unconventional intentions leads to completely erroneous predictions of future locations. In addition, complex operating conditions such as low light at night, high-speed driving, and dense traffic flow further amplify perception errors due to reflection distortion or shadow occlusion. These factors form a tight causal chain: data deficiency directly leads to the collapse of the model's generalization ability, which in turn causes interpretation bias. Interpretation bias forces the system to make high-risk decisions such as emergency braking or blind lane changes in a very short time. Once a mistake is made, it may induce major accidents such as chain rear-end collisions or rollovers in high-speed traffic. Afterwards, because the system cannot generate a complete and sufficiently confident chain of evidence, it may also lead to disputes over the determination of liability for the accident.

[0004] Therefore, a traffic behavior detection system based on visual recognition is proposed to solve or alleviate the above problems. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a traffic behavior detection system based on visual recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A traffic behavior detection system based on vision recognition, including The multimodal data acquisition module is used to acquire image data, millimeter-wave radar point cloud data, and lidar point cloud data. The data preprocessing and synchronization module, whose input end is connected to the output end of the multimodal data acquisition module, is used to perform spatiotemporal synchronization and preprocessing on the acquired multimodal data and output aligned multimodal data. The scene complexity assessment module, whose input is connected to the output of the data preprocessing and synchronization module, is used to assess scene complexity based on aligned multimodal data and generate scene risk coefficients. The multimodal feature extraction module, whose input is connected to the output of the data preprocessing and synchronization module, is used to extract visual features, radar features and lidar features from the aligned multimodal data respectively. The dynamic fusion module has its first input end connected to the output end of the multimodal feature extraction module and its second input end connected to the output end of the scene complexity assessment module. It is used to dynamically fuse visual features, radar features and lidar features according to the scene risk coefficient to generate a fused feature map. The target detection and unknown object recognition module, whose input is connected to the output of the dynamic fusion module, is used to perform target detection based on the fused feature map and to recognize unknown targets using a pre-trained normalized flow model, generating a list of detection results including known category targets and unknown targets. The causal graph construction module, whose input is connected to the output of the target detection and unknown identification module, is used to construct the causal graph of the current scene based on the detection result list and the pre-built causal rule base. The counterfactual prediction module has its first input connected to the output of the causal graph construction module and its second input connected to the output of the target detection and unknown identification module. It is used to perform counterfactual reasoning based on the causal graph and targets whose rarity exceeds a preset threshold in the detection results, and to generate the future trajectory probability distribution of each target. The risk decision-making and evidence generation module, whose input end is connected to the output end of the counterfactual prediction module and the output end of the target detection and unknown identification module, is used to calculate a comprehensive risk score based on the detection confidence, rarity score and trajectory probability distribution uncertainty of each target, and generate decision instructions and evidence chains based on the comprehensive risk score. The human-computer interaction and execution module has its input end connected to the output end of the risk decision and evidence generation module, and is used to output warning or control signals to the driver or vehicle control system according to the decision instructions. The difficult example feedback and cloud update module has its input end connected to the output end of the target detection and unknown identification module, and is used to feed back detected targets with a rarity exceeding a preset threshold as difficult example samples to the cloud. The cloud-based training and model management module is connected to the hard example feedback and cloud update module. It is used to receive hard example samples, fine-tune the normalized flow model and the object detection network, and send the updated model to the object detection and unknown identification module.

[0007] Preferably, acquiring image data, millimeter-wave radar point cloud data, and lidar point cloud data includes the following steps: The camera, millimeter-wave radar, and lidar are synchronously triggered according to the set sampling frequency to acquire raw image frames, millimeter-wave radar point cloud frames, and lidar point cloud frames, and a timestamp and sensor ID are added to each frame of data.

[0008] Preferably, the step of performing spatiotemporal synchronization and preprocessing on the acquired multimodal data to output aligned multimodal data includes the following steps: The data preprocessing and synchronization module uses the timestamp of the lidar point cloud data as a reference to perform linear interpolation on the image data and millimeter-wave radar point cloud data to achieve time alignment. It uses a calibration matrix to project the point cloud onto the image plane to generate a sparse depth map, performs histogram equalization adaptively according to the image brightness, and determines whether to perform dehazing based on the dark channel prior. It also performs ground filtering on the lidar point cloud based on the random sampling consistency algorithm to remove ground points.

[0009] Preferably, the step of evaluating scene complexity and generating scene risk coefficients based on aligned multimodal data includes the following steps: The scene complexity assessment module calculates the atmospheric light value and transmittance of the dark channel based on image data to obtain the visibility index, performs density-based clustering on the lidar point cloud to count the number of clusters to obtain the target density factor, inputs the image data into a lightweight convolutional neural network to obtain the weather category probability to obtain the weather severity, and weights and sums the complement of the visibility index, the target density factor and the weather severity to obtain the scene risk coefficient.

[0010] Preferably, the step of extracting visual features, radar features, and lidar features from the aligned multimodal data includes the following steps: The multimodal feature extraction module inputs the enhanced image data into the convolutional neural network backbone network to output a visual feature map, converts the millimeter-wave radar point cloud data into a bird's-eye view representation and maps it through a convolutional network to obtain a radar feature map, extracts columnar features from the lidar point cloud data through a PointPillars structure and maps them through a convolutional network to obtain a lidar feature map, and performs batch normalization processing on each modal feature map.

[0011] Preferably, the step of dynamically fusing visual features, radar features, and lidar features based on the scene risk coefficient to generate a fused feature map includes the following steps: The dynamic fusion module performs global average pooling on the visual feature map and concatenates it with the scene risk coefficient. Then, it passes the data through a multilayer perceptron and a sigmoid activation function to obtain the global visual weight. For each position of the visual feature map, it performs a 1×1 convolution and a sigmoid activation function to obtain the spatial visual weight map. This spatial weight map is then multiplied point-by-point by the global visual weight to obtain the point-by-point visual weight. Based on the feature energy of each position in the radar feature map and the lidar feature map, the radar secondary weight and the lidar secondary weight are calculated. Finally, the visual features and the radar-lidar fusion features are weighted and summed according to the point-by-point visual weight to obtain the fusion feature map.

[0012] Preferably, the step of performing target detection based on fused feature maps and identifying unknown targets using a pre-trained normalized flow model to generate a list of detection results including known category targets and unknown targets includes the following steps: The target detection and unknown identification module applies a region proposal network to generate candidate boxes on the fused feature map and extracts ROI feature vectors through ROIAlign. The ROI feature vectors are input into the classification branch and regression branch to obtain the known class probability and bounding box correction. The ROI feature vectors are input into a pre-trained normalized flow model to calculate the log-likelihood value and take the inverse to obtain the rarity score. The rarity score is compared with a preset threshold to determine unknown targets and set the corresponding confidence level. The module also maintains an online feature library to perform long-tail classification refinement on low-confidence known class samples. Finally, it outputs a list of detection results including bounding boxes, class, confidence level, and rarity score.

[0013] Preferably, the step of constructing a causal graph for the current scene based on the list of detection results and a pre-built causal rule base includes the following steps: The causal graph construction module generates entity nodes based on the detection result list, detects event nodes based on the detection results of multiple consecutive frames and point cloud data, traverses the pre-built causal rule library to add causal directed edges, and adds bidirectional edges based on the spatial proximity between entity nodes, thus organizing them into a causal graph for the current scene.

[0014] Preferably, the step of generating a future trajectory probability distribution for each target based on counterfactual reasoning of targets with a rarity exceeding a preset threshold in the causal graph and detection results includes the following steps: The counterfact prediction module generates a regular prediction distribution for targets whose rarity does not exceed a preset threshold using a social pooling model based on a long short-term memory network. For targets whose rarity exceeds the preset threshold, it generates multiple counterfact scenarios and performs intervention operations on the causal graph. After sampling trajectories based on behavioral templates and environmental constraints, it obtains a counterfact trajectory distribution. The regular prediction distribution and the counterfact trajectory distribution are then weighted and fused according to rarity and scenario similarity to obtain the final future trajectory probability distribution of each target.

[0015] Preferably, the step of calculating a comprehensive risk score based on the detection confidence, rarity score, and uncertainty of the trajectory probability distribution of each target, and generating a decision instruction and evidence chain based on the comprehensive risk score, includes the following steps: The risk decision-making and evidence generation module calculates the individual risk for each target based on detection confidence, rarity score, and entropy value of trajectory probability distribution. It calculates the interaction risk factor based on the collision time and minimum distance with the vehicle. The maximum value of the product of all target individual risks and interaction risk factors is taken as the comprehensive risk score. The comprehensive risk score is compared with a preset threshold to generate a decision instruction. The module also records multimodal data, detection results, rarity score, prediction distribution, causal graph, and decision instruction for a preset duration before and after the current frame to generate an evidence chain.

[0016] The present invention has the following beneficial effects: This invention acquires and spatiotemporally synchronizes multimodal sensor data in real time, performs image enhancement and point cloud filtering preprocessing, assesses scene complexity to generate scene risk coefficients, extracts multimodal features and dynamically fuses them based on the risk coefficients to generate a fused feature map, detects targets based on the fused feature map and uses a normalized flow model to identify unknown targets and generate detection results with sparseness, constructs a scene causal graph based on the detection results and a pre-built causal rule library, performs counterfactual reasoning on rare targets to generate future trajectory probability distributions, calculates a comprehensive risk score by comprehensively considering detection confidence, sparseness, and prediction uncertainty, and generates decision instructions and evidence chains, while simultaneously transmitting high-sparse and difficult examples back to the cloud for model updates, ultimately achieving robust detection and risk decision-making for known and unknown targets in complex traffic scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural block diagram of the present invention.

[0019] 1. Multimodal data acquisition module; 2. Data preprocessing and synchronization module; 3. Scene complexity assessment module; 4. Multimodal feature extraction module; 5. Dynamic fusion module; 6. Target detection and unknown identification module; 7. Causal graph construction module; 8. Counterfactual prediction module; 9. Risk decision-making and evidence generation module; 10. Human-computer interaction and execution module; 11. Difficult example feedback and cloud update module; 12. Cloud training and model management module. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In the description of this invention, it should be understood that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used to facilitate the description of this invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0024] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0025] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0026] A traffic behavior detection system based on visual recognition, such as Figure 1 As shown, including The multimodal data acquisition module 1 is used to acquire image data, millimeter-wave radar point cloud data and lidar point cloud data. Specifically, it includes the following steps: synchronously triggering the camera, millimeter-wave radar and lidar according to the set sampling frequency, acquiring the original image frame, millimeter-wave radar point cloud frame and lidar point cloud frame, and adding a timestamp and sensor ID to each frame of data. The data preprocessing and synchronization module 2, whose input end is connected to the output end of the multimodal data acquisition module 1, is used to perform spatiotemporal synchronization and preprocessing on the acquired multimodal data and output aligned multimodal data. Specifically, it includes the following steps: the data preprocessing and synchronization module 2 uses the timestamp of the lidar point cloud data as a reference to perform linear interpolation on the image data and the millimeter-wave radar point cloud data to achieve time alignment; it uses the calibration matrix to project the point cloud onto the image plane to generate a sparse depth map; it performs histogram equalization adaptively according to the image brightness and determines whether to perform dehazing based on the dark channel prior; and it performs ground filtering on the lidar point cloud based on the random sampling consistency algorithm to remove ground points. Scene complexity assessment module 3, whose input is connected to the output of data preprocessing and synchronization module 2, is used to assess scene complexity based on aligned multimodal data and generate scene risk coefficient. Specifically, it includes the following steps: the scene complexity assessment module 3 calculates the atmospheric light value and transmittance of the dark channel based on image data to obtain the visibility index; performs density-based clustering on the lidar point cloud to count the number of clusters to obtain the target density factor; inputs the image data into a lightweight convolutional neural network to obtain the weather category probability to obtain the weather severity; and weights and sums the complement of the visibility index, the target density factor, and the weather severity to obtain the scene risk coefficient. The multimodal feature extraction module 4, whose input is connected to the output of the data preprocessing and synchronization module 2, is used to extract visual features, radar features, and lidar features from the aligned multimodal data. Specifically, it includes the following steps: the multimodal feature extraction module 4 inputs the enhanced image data into the convolutional neural network backbone network to output a visual feature map; it converts the millimeter-wave radar point cloud data into a bird's-eye view representation and maps it through a convolutional network to obtain a radar feature map; it extracts columnar features from the lidar point cloud data through a PointPillars structure and maps them through a convolutional network to obtain a lidar feature map; and it performs batch normalization processing on each modal feature map. The dynamic fusion module 5, with its first input connected to the output of the multimodal feature extraction module 4 and its second input connected to the output of the scene complexity evaluation module 3, is used to dynamically fuse visual features, radar features, and lidar features according to the scene risk coefficient to generate a fused feature map. Specifically, it includes the following steps: The dynamic fusion module 5 performs global average pooling on the visual feature map and concatenates it with the scene risk coefficient. Then, it passes the map through a multilayer perceptron and a sigmoid activation function to obtain global visual weights. For each position of the visual feature map, it performs 1×1 convolution and a sigmoid activation function to obtain a spatial visual weight map. This spatial weight map is then multiplied point-by-point by the global visual weights to obtain point-by-point visual weights. Based on the feature energy of each position of the radar feature map and lidar feature map, it calculates the radar secondary weight and lidar secondary weight. Finally, it performs a weighted sum of the visual features and the radar-lidar fusion features according to the point-by-point visual weights to obtain the fused feature map. The target detection and unknown identification module 6, whose input is connected to the output of the dynamic fusion module 5, is used to perform target detection based on the fused feature map and to identify unknown targets using a pre-trained normalized flow model, generating a list of detection results including known category targets and unknown targets. Specifically, the target detection and unknown identification module 6 applies a region proposal network to the fused feature map to generate candidate boxes and extracts ROI feature vectors through ROIAlign. The ROI feature vectors are input into the classification branch and regression branch to obtain the known category probability and bounding box correction. The ROI feature vectors are input into the pre-trained normalized flow model to calculate the log-likelihood value and take the opposite to obtain the rarity score. The rarity score is compared with a preset threshold to determine the unknown target and set the corresponding confidence level. The module also maintains an online feature library to perform long-tail classification refinement on low-confidence known class samples. Finally, the output includes a list of detection results including bounding boxes, categories, confidence levels, and rarity scores. The causal graph construction module 7, whose input end is connected to the output end of the target detection and unknown identification module 6, is used to construct the causal graph of the current scene based on the detection result list and the pre-built causal rule library. Specifically, it includes the following steps: the causal graph construction module 7 generates entity nodes according to the detection result list, detects event nodes according to the detection results of multiple consecutive frames and point cloud data, traverses the pre-built causal rule library to add causal directed edges, and adds bidirectional edges according to the spatial proximity between entity nodes, and organizes them into the causal graph of the current scene. The counterfact prediction module 8 has its first input connected to the output of the causal graph construction module 7 and its second input connected to the output of the target detection and unknown identification module 6. It is used to perform counterfactual reasoning based on the causal graph and the targets whose rarity exceeds a preset threshold in the detection results, and to generate the future trajectory probability distribution of each target. Specifically, it includes the following steps: For targets whose rarity does not exceed the preset threshold, the counterfact prediction module 8 uses a social pooling model based on a long short-term memory network to generate a regular prediction distribution; for targets whose rarity exceeds the preset threshold, it generates multiple counterfactual scenarios and performs intervention operations on the causal graph. After sampling the trajectory according to the behavior template and environmental constraints, it obtains the counterfactual trajectory distribution; and it weights and fuses the regular prediction distribution and the counterfactual trajectory distribution according to rarity and scenario similarity to obtain the final future trajectory probability distribution of each target. The risk decision and evidence generation module 9, whose input is connected to the output of the counterfactual prediction module 8 and the output of the target detection and unknown identification module 6, is used to calculate a comprehensive risk score based on the detection confidence, rarity score, and uncertainty of the trajectory probability distribution of each target. Based on the comprehensive risk score, it generates decision instructions and evidence chains. Specifically, it includes the following steps: the risk decision and evidence generation module 9 calculates the individual risk of each target based on the detection confidence, rarity score, and entropy value of the trajectory probability distribution; calculates the interaction risk factor based on the collision time and minimum distance with the vehicle; takes the maximum value of the product of the individual risks of all targets and the interaction risk factor as the comprehensive risk score; compares the comprehensive risk score with a preset threshold to generate decision instructions; and records the multimodal data, detection results, rarity score, prediction distribution, causal graph, and decision instructions for a preset duration before and after the current frame to generate an evidence chain. The human-machine interaction and execution module 10, whose input end is connected to the output end of the risk decision and evidence generation module 9, is used to output warning or control signals to the driver or vehicle control system according to the decision command. Specifically, it includes the following steps: the human-machine interaction and execution module 10 receives the decision command; when the decision command is a warning, it issues an audio or visual prompt through the human-machine interaction interface; when the decision command is an active intervention, it sends a braking or steering request to the vehicle control system. The difficult example backhaul and cloud update module 11 has its input end connected to the output end of the target detection and unknown identification module 6. It is used to backhaul detected targets with a rarity exceeding a preset threshold as difficult example samples to the cloud. Specifically, it includes the following steps: The difficult example backhaul and cloud update module 11 monitors targets with a rarity exceeding a preset threshold in the detection results, extracts the original multimodal data and corresponding detection results of a preset number of frames before and after the appearance of the target, packages them into difficult example samples, and asynchronously uploads them to the cloud via a wireless network. The cloud training and model management module 12, which is connected to the hard example feedback and cloud update module 11, is used to receive hard example samples and fine-tune the normalized flow model and object detection network, and to send the updated model to the object detection and unknown recognition module 6. Specifically, the cloud training and model management module 12 receives and stores hard example samples from multiple vehicles, periodically uses hard example samples to fine-tune the normalized flow model and object detection network and adopts an elastic weight consolidation method to prevent catastrophic forgetting, generates a new model version and sends it to the object detection and unknown recognition module 6 of the vehicle terminal through over-the-air download technology.

[0027] This vision-based traffic behavior detection system operates by including the following steps: Step 1: Acquire multimodal sensor data, which includes image data, millimeter-wave radar point cloud data, and lidar point cloud data. Perform spatiotemporal synchronization and preprocessing on the multimodal sensor data to obtain aligned multimodal data. Using the timestamp of the lidar point cloud data as a reference, linear interpolation is performed on the image data and the millimeter-wave radar point cloud data to obtain time-aligned image and radar point cloud, where the interpolation weight is calculated based on the time difference. Using a pre-calibrated sensor extrinsic matrix, time-aligned lidar point clouds and millimeter-wave radar point clouds are projected onto the image plane to generate a sparse depth map corresponding to the image pixels. Image data quality enhancement: Determine whether the average brightness of the image is below a preset threshold. If so, apply adaptive histogram equalization to improve contrast. Estimate atmospheric light value and transmittance based on dark channel priors. If the average transmittance is below the visibility threshold, apply a dehazing algorithm to obtain enhanced image data. Ground filtering of LiDAR point clouds: The random sampling consensus algorithm is used to fit the ground plane model. Point clouds that are less than a preset distance threshold from the fitted plane are marked as ground points and removed. Obstacle point clouds are retained as LiDAR data for subsequent processing. Step 2: Based on the aligned multimodal data, assess the complexity of the current scene and generate a scene risk coefficient; The dark channel is calculated based on the enhanced image data. The average value of the brightest pixel in the dark channel is taken as the atmospheric light value. The transmittance of each pixel is calculated based on the atmospheric light value. The average transmittance of the whole image is taken as the visibility index. Density-based clustering is performed on the obstacle point cloud, the number of clusters is counted, the number of clusters is divided by the preset maximum target number and the minimum value of 1 is taken to obtain the target density factor; The enhanced image data is input into a pre-trained lightweight convolutional neural network, which outputs the probabilities of sunny, rainy, snowy, and foggy days. The maximum value of the probability of severe weather is taken as the severity of the weather. The scene risk coefficient is obtained by multiplying the complement of the visibility index, the target density factor, and the severity of the weather by preset weight coefficients and then summing them, where the sum of each weight coefficient is 1. Step 3: Extract visual features, radar features, and lidar features from the aligned multimodal data respectively; The enhanced image data is input into the backbone network of a convolutional neural network. After multiple convolutions and downsampling, a visual feature map is output. The spatial size of the visual feature map is one-eighth of the original image. The millimeter-wave radar point cloud data is converted into a bird's-eye view representation. The area around the vehicle is divided into grids of the same size as the visual feature map. The number of radar points, average radial velocity and average radar cross section in each grid are counted to form a three-channel original radar feature map. Then, the radar feature map is obtained by mapping through two layers of convolutional network. The LiDAR point cloud data is divided into cylinders of the same size as the visual feature map. The PointNet network is used to extract local features from the point cloud in each cylinder, and then the feature map is obtained by mapping through two convolutional networks. Each modal feature map is batch normalized to make the mean of the feature values ​​zero and the variance one. Step 4: Based on the scene risk coefficient, dynamically fuse visual features, radar features, and lidar features to generate a fused feature map; Global average pooling is performed on the visual feature map to obtain the global visual feature vector. The global visual feature vector is concatenated with the scene risk coefficient and then input into the multilayer perceptron. The sigmoid activation function outputs a scalar as the global visual weight. For the feature vector at each spatial location of the visual feature map, a spatial visual weight map of the same size as the visual feature map is generated by passing a 1×1 convolutional layer and a sigmoid activation function. The global visual weight is then multiplied element-wise with the spatial visual weight map to obtain the point-by-point visual weight. For each spatial location in the radar feature map and lidar feature map, the L2 norm of its feature vector is calculated as the energy value. The radar secondary weight is calculated based on the ratio of the radar energy value to the sum of the two energy values. The lidar secondary weight is 1 minus the radar secondary weight. For each spatial location, the visual features and the radar-liquid radar fusion features are weighted and summed according to the point-by-point visual weights. The radar-liquid radar fusion features are obtained by weighting and summing the radar features and the lidar features according to their respective sub-weights. Finally, a fusion feature map with the same size as the visual feature map is obtained. Step 5: Perform target detection based on the fused feature map, and at the same time use the pre-trained normalized flow model to calculate the feature log likelihood of each detected target, identify unknown targets based on the feature log likelihood, and generate a list of detection results containing known category targets and unknown targets. A region proposal network is applied to the fused feature map to generate candidate bounding boxes, and preliminary candidate bounding boxes are obtained by nonmaximum suppression. For each initial candidate box, the ROI Align operation is used to extract a fixed-size feature map from the fused feature map, and then mapped to an ROI feature vector through a fully connected layer; The ROI feature vector is input into the classification branch, and after passing through a fully connected layer and a softmax function, the known class probability distribution is obtained. The class corresponding to the highest probability is taken as the preliminary class, and the highest probability is used as the preliminary confidence level. At the same time, it is input into the regression branch to obtain the bounding box correction amount, and the candidate box position is fine-tuned. The ROI feature vector is input into a pre-trained normalized flow model, which is composed of multiple invertible affine coupling layers stacked together. The latent variables corresponding to the feature vector are calculated through forward transformation, and the log-likelihood value of the feature vector is obtained based on the Gaussian log probability of the latent variables and the sum of the absolute values ​​of the logarithms of the Jacobian determinants of each coupling layer. The negative of the log-likelihood value is taken as the rarity score. The rarity score is compared with a preset threshold. If it is greater than the threshold, the target is determined to be an unknown target, its category is marked as "unknown", and the confidence level is set to the sigmoid function value of the rarity score; otherwise, the known category and the initial confidence level are retained. Non-maximum suppression is applied again to all detection boxes, and the final list of detection results is output. Each detection result includes the bounding box coordinates, class, confidence score, and rarity score. Maintain an online feature library that stores the ROI feature vectors and their class labels for recently detected high-confidence, known-class samples in a queue. For known category samples in the current frame whose confidence level is lower than the preset confidence threshold, find the K samples with the highest cosine similarity to their ROI feature vector in the feature library, and these K samples must belong to the same category; The weighting coefficients are calculated based on the cosine similarity, and the feature vectors of the K samples are weighted and averaged to obtain the mixed feature vector. Input the mixed feature vector into the classification branch to obtain the probability distribution of the secondary classification, and take the maximum probability and the corresponding class; If the maximum probability of the secondary classification is greater than the original confidence score, then the category and confidence score of the target are replaced with the secondary classification result; otherwise, the original result is retained. Add samples and their categories in the current frame whose confidence level is higher than a preset high confidence threshold to the feature library, and remove the earliest added sample; Step 6: Based on the detection result list and the pre-built causal rule base, construct the causal graph for the current scene; For each target in the detection result list, an entity node is generated. The attributes of the entity node include target category, location coordinates, velocity vector, rarity score and current timestamp. Based on the detection results of multiple consecutive frames and point cloud data, specific events are detected, including: when a scattered object is detected near a truck and the object's trajectory starts in the truck's cargo compartment area, a cargo falling event node is generated; when the height of the road surface point cloud decreases beyond a preset depth threshold in consecutive frames, a road collapse event node is generated; when the vehicle deceleration exceeds a preset emergency braking threshold, an emergency braking event node is generated. Traverse the pre-built causal rule base. The rule base contains multiple causal rules. Each rule consists of a premise predicate, a conclusion predicate, and a rule confidence score. For an entity or event node in the current scenario that satisfies the rule premise, add a directed edge from the premise node to the conclusion node, and assign the edge weight to the confidence score of the rule. For any two entity nodes, calculate the Euclidean distance between them. If the distance is less than a preset proximity threshold, add a bidirectional edge. The edge weight is calculated based on the negative exponential function of the distance, with the edge weight being larger the closer the distance. Organize all nodes and directed edges into a causal graph for the current frame for subsequent inference; Step 7: Based on the causal graph and the targets with a rarity exceeding a preset threshold in the detection results, perform counterfactual reasoning to generate the probability distribution of the future trajectory of each target; For targets whose rarity does not exceed a preset threshold, a social pooling model based on a long short-term memory network is used for trajectory prediction: the encoder encodes the target's historical trajectory to obtain the hidden state, and the pooling layer aggregates the hidden states of surrounding targets to obtain the interaction vector. The decoder takes the hidden state and the interaction vector as input and outputs the Gaussian mixture model parameters of the trajectory at multiple future time points, including the weights, mean vectors and covariance matrices of each mixture component, thereby obtaining the conventional prediction distribution. For targets whose rarity exceeds a preset threshold, multiple counterfactual scenarios are first generated. Each counterfactual scenario corresponds to a preset abnormal behavior pattern, including sudden reversal, emergency stop, and lateral rush. Intervention operations are then performed on the target node in the causal graph to sever its causal edges with related nodes. In each counterfactual scenario, multiple sampling trajectories are generated based on the target's current position and speed, combined with the corresponding abnormal behavior template. Each sampling trajectory is assigned an initial weight, and the weights are then adjusted according to the current environmental context (such as road boundaries and surrounding obstacles) to obtain the counterfactual trajectory distribution in that scenario, which is represented as a Gaussian mixture model. Calculate the semantic similarity between each counterfactual scenario and the current scenario. The semantic similarity is calculated based on features such as obstacles and road structures in the scenario. The conventional prediction distribution and each counterfactual trajectory distribution are weighted and fused. The fusion weight is positively correlated with the rarity score of the target and the scene similarity, and the sum of all weights is 1, so as to obtain the final future trajectory probability distribution of the target. Step 8: Based on the detection confidence, rarity score and trajectory probability distribution uncertainty of each target, calculate the comprehensive risk score, and generate decision instructions and evidence chains based on the comprehensive risk score; For each target, the detection uncertainty is calculated as 1 minus its detection confidence, the rarity factor is its rarity score, and the prediction uncertainty is the entropy value of its trajectory probability distribution. The detection uncertainty, rarity factor, and prediction uncertainty are multiplied by preset weights and then summed to obtain the individual risk value of the target. For each target, the relative distance and relative speed are calculated based on its current position and speed and the position and speed of the vehicle, and then the collision time is calculated. If the relative speed is not positive, the collision time is taken as infinite. Based on the collision time and minimum distance, the interaction risk factor is calculated through a negative exponential function. The shorter the collision time or the smaller the distance, the larger the interaction risk factor. The individual risk value of each target is multiplied by the interaction risk factor, and the maximum value of the product of all targets is taken as the comprehensive risk score of the current frame. The comprehensive risk score is compared with the preset warning threshold and intervention threshold: if it is less than the warning threshold, the decision instruction is to drive normally; if it is between the warning threshold and the intervention threshold, the decision instruction is to issue a warning to alert the driver; if it is greater than or equal to the intervention threshold, the decision instruction is to intervene actively for safety, and to recommend automatic emergency braking or avoidance. Record multimodal sensor data segments of preset duration before and after the current frame, save the detection results, rarity scores, prediction distributions, causal graphs, and decision instructions for all targets in the current frame, and generate an evidence chain containing timestamps, geographical locations, vehicle speeds, and reasoning basis. The reasoning basis includes the ID of the highest-risk target and a description of its risk source. Step 9: Send the detected targets whose rarity exceeds the preset threshold back to the cloud as difficult examples for model updates; When the rarity score of a target exceeds a preset threshold, a difficult case record is triggered, saving the original multimodal data, corresponding detection results, and vehicle status for each preset number of frames before and after the target appears, and packaging them to generate a difficult case sample. Difficult sample data is asynchronously uploaded to the cloud server via wireless network, and the upload process does not block the real-time detection process. The cloud server periodically uses collected difficult examples to jointly fine-tune the normalized flow model and the object detection network. During the fine-tuning process, an elastic weight consolidation method is used, and a parameter penalty term weighted by the Fisher information matrix is ​​added to the loss function to preserve the performance of the original model in common scenarios. The fine-tuned model parameters are sent to all vehicle terminals via over-the-air download technology. After receiving the update, the vehicle terminals load the new model for subsequent detection.

[0028] When this system is working, the multimodal data acquisition module 1 synchronously acquires image, millimeter-wave radar, and lidar data. The data preprocessing and synchronization module 2 uses the lidar timestamp as a reference to perform linear interpolation on the image and millimeter-wave radar point cloud to ensure spatiotemporal alignment. At the same time, it adaptively performs histogram equalization based on image brightness and determines whether to perform dehazing based on dark channel priors to overcome interference from complex lighting and weather. It also performs ground filtering on the lidar point cloud to highlight non-ground obstacles, providing clean and aligned multi-source data for subsequent processing and avoiding perception starting point deviation caused by data asynchrony or noise interference.

[0029] Next, the scene complexity assessment module 3 calculates the atmospheric light value and transmittance of the dark channel based on the enhanced image to obtain the visibility index. It performs density-based clustering on the lidar point cloud to count the number of clusters and obtain the target density factor. The image is input into a lightweight convolutional neural network to obtain the weather category probability and takes the maximum probability of severe weather as the weather severity. Then, the complement of the visibility index, the target density factor and the weather severity are weighted and summed to obtain the scene risk coefficient. This coefficient reflects the current ambient light, target density and weather conditions in real time, providing a basis for adaptive adjustment for subsequent dynamic fusion.

[0030] Meanwhile, the multimodal feature extraction module 4 extracts visual feature maps from the aligned images through the convolutional neural network backbone, extracts radar feature maps containing velocity information from the millimeter-wave radar point cloud through bird's-eye view encoding, and extracts lidar feature maps containing geometric structures from the lidar point cloud through the PointPillars structure. It also performs batch normalization processing on each feature map, converting the original sensor data into high-dimensional semantic features, thus preparing rich representation information for subsequent fusion.

[0031] The dynamic fusion module 5 first performs global average pooling on the visual feature map to obtain global visual features. After concatenating these features with the scene risk coefficient, it inputs them into a multilayer perceptron and uses a sigmoid activation function to obtain global visual weights. Simultaneously, it generates spatial visual weight maps for each spatial location of the visual feature map through 1×1 convolution and a sigmoid activation function. The global visual weights are then multiplied point-by-point with the spatial visual weight maps to obtain point-by-point visual weights. Next, the feature energy at each location of the radar feature map and the lidar feature map is calculated to obtain the radar secondary weight and lidar secondary weight. Finally, the visual features and the radar-lidar fusion features are weighted and summed according to the point-by-point visual weights to obtain the fusion feature map. This allows the system to automatically reduce the visual modality weights and increase the radar and lidar contributions in scenarios such as pigs falling, where the visual appearance is unfamiliar but can be perceived by radar and lidar. This avoids missed detections caused by the visual model never having seen pigs before. At the same time, the visual weights are also reduced accordingly in complex working conditions such as rain, fog, and night to prevent the perception error from being amplified.

[0032] The target detection and unknown object recognition module 6 applies a region proposal network to generate candidate boxes on the fused feature map. ROI feature vectors for each candidate box are extracted using ROI Align. These feature vectors are then input into the classification and regression branches to obtain the known class probability and bounding box correction. Simultaneously, these feature vectors are input into a pre-trained normalized flow model to calculate its log-likelihood. Since the normalized flow model only learns the feature distribution of known classes such as vehicles and pedestrians during training, the feature vectors of people in unusual clothing or pigs will fall into low-probability regions of the known distribution, resulting in extremely small log-likelihoods. Taking the negative of this value yields a very high rarity score. When this score exceeds a preset threshold, the system classifies the target as an unknown object rather than misclassifying it as a known class. Based on the rarity score, a confidence level is calculated, thus successfully identifying previously unseen targets. For low-confidence known class samples, long-tail classification refinement is performed using an online feature library, improving the accuracy of tail category recognition. Finally, a list of detection results containing bounding boxes, classes, confidence levels, and rarity scores is output.

[0033] The causal graph construction module 7 generates entity nodes based on the detection result list. It detects event nodes such as cargo falling, road collapse, and sudden braking based on continuous multi-frame detection results and point cloud data. It traverses the pre-built causal rule library to match nodes in the current scene and adds causal directed edges. At the same time, it adds bidirectional edges based on the spatial proximity between entity nodes, organizing them into a causal graph of the current scene. This enables the system to understand that the pig falling was caused by the loose side of the truck, and that the strange pedestrian may have been startled, thus providing common sense guidance for subsequent predictions.

[0034] The counterfactual prediction module 8 uses a social pooling model based on long short-term memory networks to generate a regular prediction distribution for targets with a rarity not exceeding a preset threshold, based on historical trajectories and surrounding interactions. For targets with a rarity exceeding the preset threshold, it generates multiple counterfactual scenarios, each corresponding to a preset abnormal behavior pattern such as sudden reversal, emergency stop, or lateral rush. It then performs intervention operations on the target node on the causal graph, cutting off its causal edges with related nodes. Based on the behavior template and environmental constraints, it samples multiple trajectories to obtain the counterfactual trajectory distribution. The regular prediction distribution and each counterfactual trajectory distribution are weighted and fused according to rarity and scenario similarity to obtain the final future trajectory probability distribution. This ensures that when faced with a startled pig that may change direction at any time, the system no longer outputs a single deterministic erroneous trajectory, but instead generates a probability distribution covering multiple possibilities, thus solving the semantic gap problem of the regular prediction model.

[0035] The risk decision-making and evidence generation module 9 calculates the detection uncertainty for each target based on the detection confidence level, takes the rarity score as the rarity factor, calculates the prediction uncertainty based on the entropy value of the trajectory probability distribution, and weights and sums the three to obtain the individual risk value. It calculates the collision time and minimum distance based on the relative distance and relative speed with the vehicle and obtains the interaction risk factor. The maximum value of the product of the individual risk of all targets and the interaction risk factor is taken as the comprehensive risk score. The comprehensive risk score is compared with the preset warning threshold and intervention threshold to generate decision instructions for normal driving, warning or active intervention. At the same time, it records the multimodal data, detection results, rarity score, prediction distribution, causal graph and decision instructions for each preset time period before and after the current frame to generate an evidence chain containing timestamps, geographical location and reasoning basis. This ensures that the correct decision is made based on multidimensional risks in a very short time, avoids chain rear-end collisions caused by decision errors, and provides complete and traceable original evidence for subsequent liability determination.

[0036] The human-machine interaction and execution module 10 issues sound or visual prompts through the human-machine interaction interface based on the decision command, or sends a braking and steering request to the vehicle control system to achieve the final safety intervention.

[0037] The difficult example feedback and cloud update module 11 monitors targets with a rarity exceeding a preset threshold in the detection results, extracts the original multimodal data and corresponding detection results of a preset number of frames before and after the target's appearance, packages them into difficult example samples, and asynchronously uploads them to the cloud via a wireless network. The cloud training and model management module 12 receives and stores difficult example samples from multiple vehicles, periodically uses the difficult example samples to fine-tune the normalized flow model and the target detection network, and adopts an elastic weight consolidation method to prevent forgetting existing knowledge, generates a new model version, and distributes it to the vehicle terminal via over-the-air download technology, so that all vehicles can identify and predict similar scenarios such as pigs or strange pedestrians more quickly and accurately in the future, realizing the continuous evolution of the system.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A traffic behavior detection system based on visual recognition, characterized in that, include The multimodal data acquisition module (1) is used to acquire image data, millimeter-wave radar point cloud data and lidar point cloud data; The data preprocessing and synchronization module (2) has its input end connected to the output end of the multimodal data acquisition module (1) and is used to perform spatiotemporal synchronization and preprocessing on the acquired multimodal data and output the aligned multimodal data. The scene complexity assessment module (3) has its input end connected to the output end of the data preprocessing and synchronization module (2) and is used to assess the scene complexity based on the aligned multimodal data and generate the scene risk coefficient. The multimodal feature extraction module (4) has its input end connected to the output end of the data preprocessing and synchronization module (2) and is used to extract visual features, radar features and lidar features from the aligned multimodal data respectively. The dynamic fusion module (5) has its first input end connected to the output end of the multimodal feature extraction module (4) and its second input end connected to the output end of the scene complexity evaluation module (3). It is used to dynamically fuse visual features, radar features and lidar features according to the scene risk coefficient to generate a fused feature map. The target detection and unknown identification module (6) has its input end connected to the output end of the dynamic fusion module (5) for target detection based on the fused feature map and identification of unknown targets using a pre-trained normalized flow model, generating a list of detection results including known category targets and unknown targets; The causal graph construction module (7) has its input end connected to the output end of the target detection and unknown identification module (6), and is used to construct the causal graph of the current scene based on the detection result list and the pre-built causal rule library; The counterfact prediction module (8) has its first input end connected to the output end of the causal graph construction module (7) and its second input end connected to the output end of the target detection and unknown identification module (6). It is used to perform counterfactual reasoning based on the causal graph and the targets whose rarity exceeds a preset threshold in the detection results, and to generate the future trajectory probability distribution of each target. The risk decision and evidence generation module (9) is connected to the output of the counterfactual prediction module (8) and the output of the target detection and unknown identification module (6) for calculating a comprehensive risk score based on the detection confidence, rarity score and trajectory probability distribution uncertainty of each target, and generating decision instructions and evidence chains based on the comprehensive risk score. The human-computer interaction and execution module (10) has its input end connected to the output end of the risk decision and evidence generation module (9), and is used to output warning or control signals to the driver or vehicle control system according to the decision instructions; The difficult example feedback and cloud update module (11) has its input end connected to the output end of the target detection and unknown identification module (6), and is used to send back the detected targets with a rarity exceeding a preset threshold as difficult example samples to the cloud. The cloud training and model management module (12) is connected to the hard example return and cloud update module (11) to receive hard example samples and fine-tune the normalized flow model and the target detection network, and send the updated model to the target detection and unknown identification module (6).

2. The traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The acquisition of image data, millimeter-wave radar point cloud data, and lidar point cloud data includes the following steps: The camera, millimeter-wave radar, and lidar are synchronously triggered according to the set sampling frequency to acquire raw image frames, millimeter-wave radar point cloud frames, and lidar point cloud frames, and a timestamp and sensor ID are added to each frame of data.

3. The traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The process of performing spatiotemporal synchronization and preprocessing on the acquired multimodal data to output aligned multimodal data includes the following steps: The data preprocessing and synchronization module (2) uses the timestamp of the lidar point cloud data as a reference to perform linear interpolation on the image data and the millimeter-wave radar point cloud data to achieve time alignment. It uses the calibration matrix to project the point cloud onto the image plane to generate a sparse depth map. It performs histogram equalization based on the image brightness and determines whether to perform dehazing based on the dark channel prior. It also performs ground filtering on the lidar point cloud based on the random sampling consistency algorithm to remove ground points.

4. The traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The process of evaluating scene complexity and generating scene risk coefficients based on aligned multimodal data includes the following steps: The scene complexity assessment module (3) calculates the atmospheric light value and transmittance of the dark channel based on the image data to obtain the visibility index, performs density-based clustering of the lidar point cloud to count the number of clusters to obtain the target density factor, inputs the image data into a lightweight convolutional neural network to obtain the weather category probability to obtain the weather severity, and weights and sums the complement of the visibility index, the target density factor and the weather severity to obtain the scene risk coefficient.

5. A traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The step of extracting visual features, radar features, and lidar features from the aligned multimodal data includes the following steps: The multimodal feature extraction module (4) inputs the enhanced image data into the backbone network of the convolutional neural network to output the visual feature map, converts the millimeter-wave radar point cloud data into a bird's-eye view representation and maps it through the convolutional network to obtain the radar feature map, extracts the column features from the lidar point cloud data through the PointPillars structure and maps them through the convolutional network to obtain the lidar feature map, and performs batch normalization processing on each modal feature map.

6. The traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The step of dynamically fusing visual features, radar features, and lidar features based on the scene risk coefficient to generate a fused feature map includes the following steps: The dynamic fusion module (5) performs global average pooling on the visual feature map and concatenates it with the scene risk coefficient. Then, it obtains the global visual weight through a multilayer perceptron and a sigmoid activation function. It obtains the spatial visual weight map by performing 1×1 convolution and a sigmoid activation function on each position of the visual feature map and multiplies it point by point with the global visual weight to obtain the point-by-point visual weight. It calculates the radar secondary weight and the lidar secondary weight based on the feature energy of each position of the radar feature map and the lidar feature map, and then performs a weighted sum of the visual features and the radar-lidar fusion features according to the point-by-point visual weight to obtain the fusion feature map.

7. A traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The method of object detection based on fused feature maps and identification of unknown targets using a pre-trained normalized flow model, generating a list of detection results including known and unknown targets, includes the following steps: The target detection and unknown identification module (6) applies a region proposal network to generate candidate boxes on the fused feature map and extracts ROI feature vectors through ROIAlign. The ROI feature vectors are input into the classification branch and regression branch to obtain the known class probability and bounding box correction amount. The ROI feature vectors are input into the pre-trained normalized flow model to calculate the log likelihood value and take the opposite to obtain the rarity score. The rarity score is compared with a preset threshold to determine the unknown target and set the corresponding confidence level. The module also maintains an online feature library to perform long-tail classification and refinement of low-confidence known class samples. Finally, the module outputs a list of detection results including bounding boxes, class, confidence level and rarity score.

8. A traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The construction of the causal graph for the current scene based on the list of detection results and a pre-built causal rule base includes the following steps: The causal graph construction module (7) generates entity nodes based on the detection result list, detects event nodes based on the detection results of multiple consecutive frames and point cloud data, traverses the pre-built causal rule library to add causal directed edges, and adds bidirectional edges based on the spatial proximity between entity nodes, thus organizing the current scene into a causal graph.

9. A traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The step of generating the future trajectory probability distribution of each target by performing counterfactual reasoning based on the causal graph and the targets with a rarity exceeding a preset threshold in the detection results includes the following steps: The counterfact prediction module (8) generates a regular prediction distribution for targets whose rarity does not exceed a preset threshold using a social pooling model based on a long short-term memory network. For targets whose rarity exceeds a preset threshold, it generates multiple counterfact scenarios and performs intervention operations on the causal graph. After sampling the trajectory based on the behavior template and environmental constraints, it obtains the counterfact trajectory distribution. The regular prediction distribution and the counterfact trajectory distribution are then weighted and fused according to rarity and scenario similarity to obtain the final future trajectory probability distribution of each target.

10. A traffic behavior detection system based on visual recognition according to claim 1, characterized in that, The process of calculating a comprehensive risk score based on the detection confidence, rarity score, and uncertainty of trajectory probability distribution for each target, and generating decision instructions and a chain of evidence based on the comprehensive risk score, includes the following steps: The risk decision and evidence generation module (9) calculates the individual risk for each target based on the detection confidence, rarity score and the entropy value of the trajectory probability distribution, calculates the interaction risk factor based on the collision time and minimum distance with the vehicle, takes the maximum value of the product of the individual risk of all targets and the interaction risk factor as the comprehensive risk score, compares the comprehensive risk score with the preset threshold to generate a decision instruction, and records the multimodal data, detection results, rarity score, prediction distribution, causal graph and decision instruction for a preset duration before and after the current frame to generate an evidence chain.

Images