Automatic driving test scene extraction method based on roadside multimode perception

Abstract

The invention relates to an automatic driving test scene extraction method based on roadside multimode perception, and belongs to the field of automatic driving test. The method comprises the following steps: firstly, acquiring an image frame, millimeter wave radar data, laser radar data and local map data of a target roadside area; inputting the continuous image frames into a YOLOv8s model of a double-branch detection head, and synchronously identifying traffic participant information and traffic accident event information; matching and fusing the multi-sensor data in combination with local map data to generate traffic participant track point data; performing tracking management on the trajectory point data through a Kalman filtering algorithm to obtain initial trajectory information; through multi-dimensional abnormal point detection and layered trajectory reconstruction, final trajectory information is generated; based on the final trajectory, traffic violation detection and GGNN network driven high-interaction risk identification are completed; and finally extracting tracks before and after the event to form an automatic driving test scene. According to the method, the problems of poor multi-source data fusion, detection separation and the like in the prior art are solved, scene integrity and accuracy are improved, and high-order automatic driving test requirements can be met.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving testing, and in particular relates to a method for extracting autonomous driving test scenarios based on roadside multi-modal perception. Background Technology

[0002] The safety performance verification of autonomous driving systems requires a large number of real and complete test scenarios. Roadside multi-modal perception technology integrates image, millimeter-wave radar, lidar and map data to comprehensively collect information on the dynamics of traffic participants, traffic events and road environment. It is the core data source for extracting autonomous driving test scenarios. Its technological maturity directly affects the authenticity and effectiveness of the test scenarios and is crucial to ensuring the safe implementation of autonomous driving systems from research and development to commercial application.

[0003] Existing autonomous driving test scenario extraction technologies still have common limitations in key aspects: At the target and event detection level, most solutions can only achieve single identification of traffic participants (such as vehicles and pedestrians), making it difficult to simultaneously acquire traffic accident event information, resulting in the lack of key accident-related data in the scenario; In the multi-source data fusion stage, existing methods mostly remain at the level of simple coordinate matching or basic feature superposition of sensor data, failing to fully integrate road map information to establish deep correlations between data, making it difficult to generate core data that accurately reflects the movement status of traffic participants; In the trajectory processing stage, existing technologies have imperfect mechanisms for identifying and repairing abnormal data in the trajectory, which is prone to insufficient trajectory accuracy due to interference from abnormal points, making it impossible to truly restore the movement process of traffic participants; In the violation and risk identification stage, violation detection mostly relies on fixed parameter judgments, lacking adaptability to dynamic road conditions, while the identification of interaction risks between traffic participants is mostly based on single-dimensional indicators, making it difficult to comprehensively assess risk events under complex road conditions.

[0004] Existing technologies fail to establish a complete technological chain encompassing "collaborative detection of targets and events—deep fusion of multi-source data—precise trajectory optimization—intelligent identification of violations and risks," resulting in issues such as missing key information, insufficient data accuracy, and deviations in scene realism in the extracted autonomous driving test scenarios. This fails to meet the stringent requirements of high-level autonomous driving testing for comprehensive and accurate scenarios. Therefore, an optimized solution based on roadside multi-modal perception is urgently needed to address the common deficiencies of existing technologies throughout the entire scene extraction process, thereby improving the quality and reliability of test scenarios. Summary of the Invention

[0005] To address the problems existing in the background art, one aspect of the present invention provides a method for extracting autonomous driving test scenarios based on roadside multi-modal perception, comprising:

[0006] S1: Acquire image frames, millimeter-wave radar data, lidar data, and local map data of the target roadside area;

[0007] S2: Input the continuous image frame sequence into the YOLOv8s model of the dual-branch detection head for detection and identify traffic participant information and traffic accident event information;

[0008] S3: Based on local map data, the trajectory point data of each traffic participant is obtained by matching and fusing millimeter-wave radar data, lidar data, traffic participant information and traffic accident event information.

[0009] S4: Based on the trajectory point data of each traffic participant, the Kalman filter algorithm is used to track and manage the trajectory point data of each traffic participant, and generate the initial trajectory information of each traffic participant.

[0010] S5: Combine the initial trajectory information of each traffic participant with local map data to detect outliers, and reconstruct the detected outliers to generate the final trajectory information of each traffic participant.

[0011] S6: Based on the final trajectory information of each traffic participant and combined with local map data, detect traffic violations of each traffic participant and identify traffic violation information of each traffic participant;

[0012] S7: Identify the interaction risks between traffic participants based on their final trajectory information and local map data, and identify high-risk interaction events between traffic participants.

[0013] S8: Extract traffic accident events of traffic participants, traffic violation events of each traffic participant, and the final trajectory information of traffic participants within a preset time period before and after the occurrence of high-interaction risk events between traffic participants to form the scene information of the corresponding events.

[0014] Another aspect of the present invention provides an autonomous driving test scene extraction system based on roadside multi-modal perception, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the autonomous driving test scene extraction method based on roadside multi-modal perception as described above.

[0015] Another aspect of the present invention provides a computer storage medium storing a remote monitoring program, which, when executed by a processor, implements the aforementioned method for extracting autonomous driving test scenarios based on roadside multi-modal perception.

[0016] The present invention has at least the following beneficial effects

[0017] This invention first acquires image frames, millimeter-wave radar data, lidar data, and local map data of the target roadside area, laying a comprehensive data foundation for scene extraction and solving the problem of incomplete multi-source data acquisition in existing technologies. Then, it utilizes a YOLOv8s model with a dual-branch detection head to simultaneously identify traffic participants and traffic accident event information, overcoming the limitations of existing solutions where target and event detection are separated, leading to missed or delayed accident events. Subsequently, it combines local map data with multi-sensor data to generate trajectory point data, avoiding the insufficient fusion accuracy problem caused by traditional simple data overlay. Through Kalman filter tracking, multi-dimensional anomaly detection, and hierarchical trajectory reconstruction, it improves trajectory accuracy and smoothness, overcoming the defects of anomaly interference in existing trajectory processing. Based on the final trajectory and local map data, it achieves traffic violation detection and high-interaction risk identification, solving the problems of fixed violation judgments and singular risk identification in existing systems. Finally, it extracts the trajectories before and after key events to form a scene, ensuring that the extracted test scene is complete, realistic, and accurate, effectively meeting the stringent requirements of high-level autonomous driving system safety verification for high-quality test scenes. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0019] Figure 2 This is a schematic diagram of the YOLOv8s model structure of the dual-branch detection head of the present invention. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] Please see Figure 1 This invention provides a method for extracting autonomous driving test scenarios based on roadside multi-modal perception, including:

[0022] S1: Acquire image frames, millimeter-wave radar data, lidar data, and local map data of the target roadside area;

[0023] In this embodiment, a roadside high-definition RGB camera is installed on a roadside pole, and the camera parameters, including extrinsic and intrinsic parameters, are determined for subsequent coordinate transformation, aiming at the target area, and covering lanes, sidewalks, traffic lights, and stop lines. The camera continuously acquires images at a frame rate of 25fps, with each frame formatted as JPEG, outputting a continuous sequence of valid image frames. Millimeter-wave radar data is acquired via a roadside system, with the radar acquiring raw data at a frequency of 10Hz. The raw data includes radial distance, radial velocity, azimuth angle, and RCS (radar cross-section). Clutter targets with an RCS value < -30dB are removed, retaining valid targets such as vehicles and pedestrians. For each... Each valid target is assigned a temporary ID, its motion parameters are bound, and a timestamp synchronized with the image frame is added. LiDAR data is collected through roadside LiDAR, which collects raw 3D point cloud data at a frequency of 10Hz. Each frame contains millions of point clouds, and each point contains x / y / z world coordinates and reflection intensity. Local map data is downloaded from the target roadside area through an interface authorized by the traffic management department, in OpenDRIVE format, including: lane ID, lane type, world coordinates of the lane centerline, left and right boundary coordinates, and baseline speed limit; traffic facility information: traffic light ID and world coordinates, stop line world coordinates, and intersection topology.

[0024] S2: Input the continuous image frame sequence into the YOLOv8s model of the dual-branch detection head for detection and identify traffic participant information and traffic accident event information;

[0025] Please see Figure 2 Preferably, the YOLOv8 model with the dual-branch detection head includes: cascading the first and third C2f modules in the YOLOv8 backbone network through convolutional layers; cascading the second and fourth C2f modules in the YOLOv8 backbone network through convolutional layers; adding a DynamicConv layer between the third and fourth C2f modules; modifying each prediction head of the original YOLOv8 backbone network into a dual-branch detection head; the dual-branch detection head includes a target branch detection head and an accident branch detection head; the target branch detection head and the accident branch detection head share the same input feature; and adding a BiFormer spatial attention module to the accident branch detection head to preprocess the input feature so that the feature focuses on the accident collision region.

[0026] In this embodiment, the traffic participant information includes the target temporary ID, category (car / truck / pedestrian / non-motorized vehicle), 2D detection box coordinates (pixel coordinates, x1 / y1 / x2 / y2), and detection confidence (0~1); the traffic accident event information includes the accident ID, accident type (vehicle collision / pedestrian collision / object impact), accident area 2D detection box coordinates (pixel coordinates), and accident confidence (0~1); if the IoU between the accident area and the traffic participant detection box is ≥0.6, the accident ID and the target temporary ID are bound together to identify the involved participants. The main improvements to the dual-branch detection head YOLOv8 model in this embodiment include: concatenating the first C2f module (C2f1) and the third C2f module (C2f3) of the native YOLOv8 backbone network through 1×1 convolutional layers; similarly concatenating the second C2f module (C2f2) and the fourth C2f module (C2f4) through 1×1 convolutional layers; unifying the feature channel dimensions in the 1×1 convolutional layers (e.g., C2f1 output channels are 256, C2f3 input channels are 512, the 256 channels are converted to 512 channels through 1×1 convolution and then added element-wise), supplementing the detailed features of shallow networks (e.g., distant pedestrians, non-motorized vehicles), and avoiding feature attenuation in deeper layers; inserting a dynamic convolutional layer (DynamicConv) between the third C2f module (C2f3) and the fourth C2f module (C2f4); and improving the adaptability of the fixed convolutional kernels of the native YOLOv8 to occluded targets (e.g., vehicles occluding pedestrians, multiple vehicles occluding in parallel). To address shortcomings, dynamic convolution generates "convolutional kernels adapted to occluded scenes" through an attention mechanism, enhancing feature extraction in occluded areas. The dual-branch structure design replaces the native YOLOv8 single detection head with a dual-branch detection head, with both branches sharing three sets of multi-scale feature maps output by the neck network. Branch 1: Target branch detection head (responsible for traffic participant detection); Branch 2: Accident branch detection head (responsible for traffic accident event detection); The accident branch detection head is pre-loaded with a BiFormer spatial attention module. Accident areas (such as vehicle deformation after a collision, or fallen pedestrians) occupy a small proportion in the image and their features are not obvious. BiFormer focuses on the collision area features through "window partitioning + multi-head attention". In this embodiment, the target branch uses DFL loss to optimize the matching degree of the target category probability distribution and MPDIoU loss to optimize the detection box localization accuracy. The accident branch uses VFL loss to solve the imbalance of accident samples and NWD loss to optimize the localization of the accident area.

[0027] In this embodiment, step S2 involves inputting a continuous sequence of image frames of the target roadside area into an improved dual-branch detection head YOLOv8s model for detection. This model is specifically optimized from the original YOLOv8s: the first and third C2f modules, and the second and fourth C2f modules in the backbone network are cascaded through convolutional layers to supplement shallow detail features and avoid deep feature attenuation. A DynamicConv layer is added between the third and fourth C2f modules to improve adaptability to occluded scenes. Simultaneously, the original single detection head is changed to a dual-branch structure of a target branch detection head and an accident branch detection head (both share input features, and the accident branch is preceded by a BiFor layer). The spatial attention module focuses on the features of the accident collision area and ultimately simultaneously identifies and outputs traffic participant information (including category, 2D detection box coordinates, detection confidence, etc.) and traffic accident event information (including accident type, 2D detection box coordinates of the accident area, accident confidence, etc., and binds the involved participants through IoU matching). This step effectively solves the problem of missed or delayed detection of accident events caused by the separation of traffic participant and traffic accident event detection in the existing technology. At the same time, through the design of feature concatenation, dynamic convolution and spatial attention module, the detection robustness in occluded scenes and the detection accuracy of accident areas are improved, providing accurate and comprehensive visual basic data for subsequent multi-source sensor data fusion and test scene extraction.

[0028] S3: Based on local map data, the trajectory point data of each traffic participant is obtained by matching and fusing millimeter-wave radar data, lidar data, traffic participant information and traffic accident event information.

[0029] Preferably, step S3 includes:

[0030] S31: Perform DBSCAN clustering on the LiDAR data to generate 3D bounding boxes for each traffic participant, and project the 3D bounding boxes of each traffic participant onto the pixel plane to generate 2D projection boxes for each traffic participant.

[0031] In this embodiment, the LiDAR data (point cloud data (x / y / z world coordinates, reflection intensity), DBSCAN clustering parameters (pre-configured)) and sensor calibration parameters are used: LiDAR-camera extrinsic parameters and camera intrinsic parameters. Based on the 3D bounding box, the LiDAR-camera extrinsic parameters are used to convert the 8 vertices of the 3D bounding box into camera coordinate system coordinates. Then, the camera intrinsic parameters are used to convert them into pixel coordinates. The extreme values ​​of the pixel coordinates of the 8 vertices are taken to obtain the LiDAR 2D projection frame.

[0032] S32: Match the 2D projection box of the traffic participant with the visual detection box of the traffic participant, and perform weighted fusion on the successfully matched 2D projection box and visual detection box to obtain the fused detection box of the traffic participant.

[0033] In this embodiment, the visual detection box output by the camera is calculated. With LiDAR 2D projection frame Spatial overlap and category consistency are considered a successful match only if all three of the following conditions are met:

[0034] Condition 1: IoU (Intersection over Union) ≥ 0.5 (core matching criterion, excluding spatially unrelated boxes), IoU is calculated as follows: and The area of ​​the intersection divided by the area of ​​the union;

[0035] Condition 2: The point cloud density of the LiDAR 3D bounding box is ≥ Points / m² (Ensure the 3D bounding box is not generated by clutter clustering to avoid matching false targets; in this example...) Take 15);

[0036] Condition 3: If the camera has already output the category of the visual inspection box (e.g., car), then the size of the LiDAR 3D bounding box must match that category (e.g., the length, width, and height of the 3D bounding box corresponding to a car must be 3.5). 5m, 1.6 2m, 1.4 Within a 1.8m range, errors in matching the truck's 3D surround box with the car's detection frame are excluded.

[0037] If a visual bounding box matches multiple 3D bounding boxes (e.g., adjacent vehicles are clustered and split), the 3D bounding box with the highest point cloud density and the largest IoU is selected as the matching object; if a 3D bounding box matches multiple visual bounding boxes (e.g., the camera mistakenly detects the same vehicle as two targets), the visual bounding boxes are merged into the smallest bounding rectangle and the IoU is recalculated to ensure that there is a one-to-one correspondence in the end.

[0038] For successful matches and Weighted fusion of pixel coordinates is performed, with weights dynamically assigned based on sensor confidence and camera confidence. (When the occlusion area is greater than 30%, the confidence score of visual detection is multiplied by the occlusion compensation coefficient of 0.7), LiDAR confidence score. (When point cloud density > 20 points / m², Point cloud density is 15 At 20 points / ㎡, When the point cloud density is <15 points / m², The pixel coordinates after fusion are:

[0039]

[0040]

[0041] Dimensions after fusion:

[0042]

[0043]

[0044] Output data: Successfully matched fused detection bounding boxes (pixel coordinates) , , , Fusion confidence ); a successfully matched 3D bounding box for LiDAR.

[0045] S33: Based on the depth information of the 3D bounding box center, project the fused detection box of the traffic participant onto the world coordinate system to obtain the 3D fused detection box of the traffic participant; where the center of the 3D fused detection box is the coordinate of the trajectory point of the traffic participant.

[0046] In this embodiment, based on the world coordinate system depth of the 3D bounding box of the LiDAR, the coordinates of the center pixel of the fusion detection box are first reversed to the camera coordinate system coordinates through the camera intrinsic parameters; then, they are converted to world coordinate system coordinates through the camera-world coordinate system extrinsic parameters; the center of the 3D fusion detection box is the coordinate of the trajectory point of the traffic participant.

[0047] S34: Transform the polar coordinates of the center point of each traffic participant in the millimeter-wave radar data to the world coordinate system, and match the traffic participants according to the distance between the center point and the coordinates of the traffic participants' trajectory points; calculate the speed and acceleration information of the matched traffic participants based on their lidar data.

[0048] In this embodiment, the millimeter-wave radar polar coordinates are converted to world coordinates using extrinsic parameters, and the Euclidean distance between the polar coordinates and the center of the 3D fusion detection frame is calculated. ,like (In this embodiment) (Set to 2m), it is identified as a related target, and its velocity and acceleration information are calculated based on its lidar data.

[0049] S35: Generate the trajectory point data of traffic participants based on traffic participant information, traffic accident event information, trajectory point coordinates of traffic participants, 3D fusion detection boxes of traffic participants, and the speed and acceleration of traffic participants.

[0050] In this embodiment, for visual detection boxes that do not match the LiDAR 3D bounding box, the depth can be supplemented using historical depth interpolation and map constraints:

[0051] Historical depth tracing: Query the 3D bounding box depth of the LiDAR matched in the previous 3 frames for this visual target. Predicting the current frame depth using a constant velocity model ( For historical depth direction speed, (frame interval time).

[0052] World coordinate transformation: using predicted depth It replaces the lidar depth, generates world coordinates, and calculates its velocity and acceleration information based on its lidar data;

[0053] For 3D point cloud recognition and fusion of fully occluded targets, when the visual target is completely occluded (the camera has no visual detection box output or the LiDAR 3D bounding box does not match the visual detection box), new target category recognition and incident marking are achieved solely based on the LiDAR 3D point cloud, and cross-validated with millimeter-wave radar data, as follows:

[0054] 1. New target category recognition based on 3D point clouds

[0055] Point cloud feature extraction: For the 3D bounding boxes generated by LiDAR clustering (without matching visual boxes), extract the shape features (e.g., aspect ratio—car aspect ratio ≈ 2.5, pedestrian aspect ratio ≈ 0.3) and reflection intensity features (e.g., reflection intensity of metallic vehicles ≈ -10). -5dB, non-metallic pedestrian reflection intensity ≈-20 -15dB), motion characteristics (velocity calculated from the displacement of 3D boxes in adjacent frames—vehicle speed is usually >1m / s, pedestrian speed <3m / s);

[0056] Category determination: A random forest classifier is used. The above features are input and the target category (such as car, truck, pedestrian, non-motorized vehicle) and category confidence (such as 0.85) are output. This solves the problem of missing categories when there is no visual detection box. Since there is no visual detection box, its category confidence can be used as the fusion confidence.

[0057] 2. Accident marker recognition based on 3D point cloud

[0058] Accident Feature Detection: For the point cloud within the 3D bounding box, calculate the point cloud distribution anomaly degree—normal vehicle point clouds are distributed in regular cuboids, while accident vehicle point clouds (such as those deformed after a collision) will show local depressions (such as a sudden drop of 50% in point cloud density in a certain area) or protrusions (such as isolated point clouds of parts that have detached after a collision); at the same time, calculate the distance between adjacent target 3D bounding boxes—if the minimum distance between two target 3D bounding boxes is <0.5m and the relative speed is <0.2m / s (nearly stationary), it is judged as a "suspected collision accident";

[0059] Accident marker confirmation: If the point cloud distribution anomaly is greater than 0.7 (preset threshold) and meets the "suspected collision accident" condition, then add a "traffic accident marker" and an accident confidence level (set to 0.8) to the 3D annotation box.

[0060] 3. Cross-validation with millimeter-wave radar data

[0061] Velocity verification: If the velocity of the new target identified by the lidar is >1m / s, compare it with the radial velocity of the associated target output by the millimeter-wave radar. If the difference between the two is <0.5m / s, the target is confirmed to be real (excluding point cloud clutter).

[0062] Accident status verification: If the target is marked with an accident symbol, the millimeter-wave radar should detect a sudden change in the target's RCS (radar cross-section) (such as a 20% increase in RCS due to a change in vehicle shape after a collision). If this condition is met, the accident confidence level is increased to 0.90; otherwise, it is reduced to 0.7.

[0063] Output the fusion result of the fully occluded target: world coordinate system 3D coordinates, category, accident marker, motion speed, and fusion confidence, which serve as effective target data when there is no visual detection and are added to the overall fusion result.

[0064] The final fusion output includes structured traffic object trajectory point data containing accident markers and confidence scores, specifically including: target unified ID, category, world UTM coordinates (x / y / z), instantaneous velocity, acceleration (based on velocity difference), fusion confidence score, accident markers and confidence scores, and accident collision area. In this embodiment, special cases such as fully occluded targets, visual detection boxes that do not match the LiDAR 3D bounding box, one visual detection box matching multiple 3D bounding boxes, or one 3D bounding box matching multiple visual detection boxes are illustrated. These cases can be further extended based on the solution of this invention, but they do not affect the processing of normal cases in this solution. If the above situations do not occur in a visual sequence, this solution can be fully applied. For example, placing cameras, LiDAR, and millimeter-wave radar at a sufficiently high position in the roadside area can avoid target occlusion.

[0065] In this embodiment, step S3 uses local map data as a spatial reference to achieve deep matching and fusion of multi-sensor data and detection information in stages: First, DBSCAN clustering is performed on the LiDAR data to generate 3D bounding boxes for traffic participants, which are then projected onto the pixel plane to obtain 2D projection boxes. These are then matched with the visual detection boxes of traffic participants output in step S2 under multiple conditions (meeting IoU≥0.5, point cloud density meeting the standard, and category size matching). Through weighted fusion (dynamically allocating weights based on sensor confidence), a fused detection box is obtained. Subsequently, the fused detection box is projected onto the world coordinate system using the center depth information of the 3D bounding box to generate a 3D fused detection box (whose center is the coordinate of the trajectory point). At the same time, the polar coordinates of the center point of the millimeter-wave radar data are transformed to the world coordinate system. The target is associated by matching the distance with the trajectory point coordinates (≤ a set threshold), and the target velocity and acceleration are calculated by combining the LiDAR data. Finally, traffic participant information, traffic accident event information, trajectory point coordinates, 3D fused detection boxes, and motion parameters are integrated to generate structured traffic participant trajectory point data. This step effectively solves the problems of existing technologies where multi-sensor data fusion only involves simple overlay and lacks spatial constraints and multi-dimensional matching. It ensures coordinate uniformity through local map data, improves data association accuracy through multi-condition matching and weighted fusion, and integrates the advantages of vision, laser, and millimeter-wave radar (visual category recognition, laser 3D positioning, and millimeter-wave motion parameters). The generated trajectory point data has both spatial accuracy and motion integrity, providing high-quality basic data support for subsequent trajectory tracking and management.

[0066] S4: Based on the trajectory point data of each traffic participant, the Kalman filter algorithm is used to track and manage the trajectory point data of each traffic participant, and generate the initial trajectory information of each traffic participant.

[0067] Preferably, the step of using the Kalman filter algorithm to track and manage the trajectory point data of each traffic participant and generate the initial trajectory information of each traffic participant includes:

[0068] S41: Convert the trajectory point data of traffic participants to trajectory point data in a rectangular coordinate system, and define the state vector of each traffic participant based on the coordinates, velocity and acceleration of the traffic participants in the rectangular coordinate system.

[0069] In this embodiment, the state vectors of each traffic participant are represented as follows:

[0070]

[0071] in, Represents the coordinates of traffic participants. Indicates the speed of traffic participants; Indicates the acceleration of traffic participants;

[0072] S42: Define the state transition matrix based on the uniform acceleration model, and define the process noise covariance matrix based on the speed and acceleration of traffic participants in the previous frame;

[0073] In this embodiment, the state transition matrix Represented as:

[0074]

[0075] in, It is a 3x3 identity matrix. It is a 3x3 zero matrix. (Determined by the tracking frame rate F of the static initialization parameters;)

[0076] In this embodiment, the process noise covariance matrix Represented as:

[0077]

[0078] in, , , The modulus of the target velocity in the previous frame (unit: m / s); The magnitude of the target acceleration in the previous frame (unit: m / s²). It is a 9th-order identity matrix (with the same dimension as the state vector).

[0079] S43: Predict the prior state of traffic participants in the previous frame based on their optimal state and optimal covariance. and prior covariance ;

[0080] In this embodiment, the optimal state of the target in the previous frame is read from the historical tracking list. With covariance ;

[0081] Prior state calculation: ;

[0082] Prior covariance calculation: ;

[0083] S44: Generate the 3D prior bounding box of the traffic participants in the previous frame based on their prior states and the standard 3D bounding box size of the traffic participants.

[0084] S45: Match the 3D fusion detection boxes of each traffic participant in the current frame with the 3D prior bounding boxes of the traffic participants in the current frame. If the match is successful, add the trajectory point data of the traffic participants in the current frame to the trajectory of the traffic participants to generate the initial trajectory information of the traffic participants. Then, based on the state observation values ​​of the traffic participants in the current frame, correct the prior state and prior covariance of the traffic participants in the previous frame to obtain the highest priority prior state and highest priority covariance of the traffic participants in the current frame.

[0085] In this embodiment, for each observation target With historical goals Calculate the following two types of features:

[0086] (1) The formula for spatial overlap (IoU_MPD) is:

[0087]

[0088] in: For the target observed in the current frame, Set its 3D detection bounding box; For historical tracking targets, For its prior 3D frame ; , They are respectively , Center UTM coordinates The distance between the centers of the two frames is the UTM distance. The maximum diagonal length of the sensing area.

[0089] (2) The formula for SimMotion is:

[0090]

[0091] in, For observation target With historical goals The angle between the velocity vectors, For adjustment coefficients, For observation target speed, For historical goals The speed.

[0092] 2. Matching cost matrix and two-level matching

[0093] Construct the cost matrix Perform two-level matching:

[0094] Basic matching: They are directly identified as the same target;

[0095] Secondary match: Pairs that were not initially matched. Furthermore, if the IoU between the prior bounding box and the observation bounding box is ≥0.3, they are determined to be the same target.

[0096] 3. Generate three types of results and transmit them to the dynamic management stage of tracking status.

[0097] List of matching pairs: and Treat them as the same target and add them to the corresponding trajectory.

[0098] New target list: Unmatched observation targets (create new target IDs and update them);

[0099] List of missing targets: Unmatched historical targets (if there are 3 consecutive frames without a match, the trajectory information of the target will be output and tracking will be terminated).

[0100] In this embodiment, step S4 uses the Kalman filter algorithm to generate initial trajectory information based on the trajectory point data of each traffic participant: First, the trajectory point data is converted to a rectangular coordinate system, and a state vector containing three-dimensional position, velocity, and acceleration is defined by combining the coordinates, velocity, and acceleration of the traffic participants; then, a state transition matrix is ​​constructed based on the uniform acceleration model, and the process noise covariance matrix is ​​determined based on the velocity and acceleration of the target in the previous frame. In this way, the prior state and prior covariance of the current frame are predicted by the optimal state and optimal covariance of the target in the previous frame; then, a 3D prior bounding box is generated by combining the target standard 3D bounding box size and the prior state, and the 3D fusion detection boxes of each traffic participant in the current frame are matched with the 3D prior bounding box. If the matching is successful, the trajectory point data of the current frame is added to the corresponding target trajectory to form initial trajectory information. At the same time, the prior state and covariance of the previous frame are corrected according to the current frame state observation value to obtain the optimal state and covariance of the current frame. This step effectively solves the problem of tracking interruption or trajectory deviation caused by sensor noise and temporary target occlusion in existing trajectory tracking by using the "prediction-correction" closed-loop mechanism of Kalman filtering combined with spatial matching of 3D bounding boxes. It improves the continuity and accuracy of trajectory tracking of traffic participants, and the generated initial trajectory information provides a complete and logically coherent basic data for subsequent anomaly detection and trajectory reconstruction.

[0101] S5: Combine the initial trajectory information of each traffic participant with local map data to detect outliers, and reconstruct the detected outliers to generate the final trajectory information of each traffic participant.

[0102] Preferably, step S5 includes:

[0103] S51: Based on the coordinates of each trajectory point in the initial trajectory information of traffic participants and combined with local map data, if a trajectory point in the initial trajectory information of a traffic participant that exceeds a set threshold is within the left and right boundaries of a certain lane, then that lane is taken as the lane to which the initial trajectory of the traffic participant belongs; and trajectory points that are not within that lane are removed as abnormal points.

[0104] In this embodiment, the left and right boundary UTM coordinate fitting equations and the lane centerline UTM coordinate fitting equations are obtained based on local map data. By calculating the average distance from each trajectory point in the initial trajectory information of traffic participants to the centerline of each lane, the lane with the smallest distance is selected as the lane to which the initial trajectory of the traffic participant belongs.

[0105] S52: Based on the category of traffic participants, determine the speed and acceleration conditions for each traffic participant, and remove trajectory points that do not meet the speed and acceleration conditions as outliers.

[0106] S53: Set up a sliding window to process the initial trajectory information of traffic participants. Within each sliding window, calculate the mean and standard deviation of the trajectory points. Based on the mean and standard deviation of the trajectory points within the sliding window, identify outlier trajectory points within the sliding window as abnormal points and remove them.

[0107] S54: In the initial trajectory of the traffic participant after processing in steps S51 to S53, for missing trajectories with a trajectory interruption duration less than or equal to a set threshold, the missing trajectory is fitted and repaired by using third-order B-spline curve fitting based on the valid trajectory points before and after the interruption; for missing trajectories with an interruption duration greater than the set threshold, based on the valid trajectory points before the interruption, and constrained by the centerline equation of the lane to which the initial trajectory belongs, the distance between the predicted trajectory and the centerline of the lane is constrained to be less than the set threshold, and H∞ filtering prediction is performed frame by frame to repair the missing trajectory and obtain the final trajectory information of the traffic participant.

[0108] Preferably, the objective function used when fitting and repairing the missing trajectory using B-spline curve fitting includes:

[0109]

[0110] in, This indicates that the third-order B-spline curve has the following parameters. The coordinates at that location These are curve parameters associated with timestamps, representing the trajectory points predicted by the fitted curve; This represents the actual trajectory points before and after the interruption; The second derivative of the curve is represented. This represents the smoothing coefficient.

[0111] In this embodiment, step S5, based on the initial trajectory information of each traffic participant, combines local map data to perform multi-dimensional anomaly detection and hierarchical trajectory reconstruction to generate the final trajectory information: First, by determining whether trajectory points exceeding a set threshold in the initial trajectory are within the left and right boundaries of a lane, the lane to which the trajectory belongs is determined, and trajectory points not within the lane are removed as anomalies; then, based on the category of traffic participants (such as vehicles, pedestrians, and non-motorized vehicles), the corresponding reasonable range of speed and acceleration is determined, and trajectory points that do not meet the range are judged as anomalies and removed; subsequently, a sliding window is set to process the initial trajectory segment by segment, and the trajectory is calculated within each window. The mean and standard deviation of the track points are used to identify and remove outlier track points that deviate from the mean and exceed the standard deviation range. For track interruptions that may occur after outlier removal, if the interruption duration is less than or equal to a set threshold, the missing track is repaired by fitting a third-order B-spline curve based on the valid track points before and after the interruption (balancing fitting accuracy and curve smoothness through an objective function with a smoothing coefficient). If the interruption duration is greater than the set threshold, the centerline equation of the lane to which the track belongs is used as a constraint (ensuring that the distance between the predicted track and the centerline is less than the set threshold), and H∞ filtering prediction is performed frame by frame based on the valid track points before the interruption to complete the missing track repair and finally obtain the final track information of the traffic participants.

[0112] S6: Based on the final trajectory information of each traffic participant and combined with local map data, detect traffic violations of each traffic participant and identify traffic violation information of each traffic participant;

[0113] Preferably, the information identifying traffic violations by each traffic participant includes:

[0114] S61: Define the lane to which the final trajectory of a traffic participant belongs as the target lane, calculate the angle between the velocity direction of the traffic participant's trajectory point and the permissible driving direction of the target lane, and determine that the traffic participant has committed a wrong-way driving violation if the angle is greater than a set threshold.

[0115] S62: Count the number of traffic participants in the target lane in the most recent preset time period. Based on the count and the maximum allowable number of traffic in the target lane, the maximum speed limit of the target lane is weighted to obtain the dynamic speeding threshold of the target lane. If the speed of the traffic participant's trajectory point is greater than the dynamic speeding threshold of the target lane, it is determined that the traffic participant has committed a speeding violation.

[0116] S63: Extract the stop line data of the target lane based on the local map data, calculate the distance from each trajectory point of the traffic participant to the stop line, and if the distance between the trajectory point and the stop line is less than a set threshold and the speed of the trajectory point of the traffic participant is greater than the set threshold during the continuous red light event corresponding to the stop line, then it is determined that the traffic participant has committed a red light violation.

[0117] In this embodiment, step S6, based on the final trajectory information of each traffic participant generated in step S5, combines local map data to conduct multi-dimensional traffic violation event detection to identify the traffic violation event information of each traffic participant: First, the lane to which the final trajectory of the traffic participant belongs is defined as the target lane. By calculating the angle between the speed direction of the trajectory point and the permitted driving direction of the target lane, if the angle exceeds a set threshold, it is determined that the traffic participant has a wrong-way driving violation event; Second, the number of traffic participants in the target lane in the most recent preset time period is counted. Based on the counted number and the maximum permitted number of traffic participants in the target lane, the maximum speed limit of the lane is weighted and calculated to obtain a dynamic speeding threshold adapted to real-time traffic conditions. If the speed of the trajectory point is greater than the dynamic speeding threshold, it is determined that there is a speeding violation event; Finally, the stop line data corresponding to the target lane is extracted from the local map data, and the distance from each trajectory point to the stop line is calculated. If, during the continuous red light period corresponding to the stop line, the distance between the trajectory point and the stop line is less than the set threshold and the speed of the trajectory point is greater than the set threshold (excluding stationary or slow stopping situations), it is determined that there is a red light violation event.

[0118] S7: Identify the interaction risks between traffic participants based on their final trajectory information and local map data, and identify high-risk interaction events between traffic participants.

[0119] Preferably, the information for identifying high-risk interaction events between traffic participants includes:

[0120] S71: Based on the final trajectory information of each traffic participant, calculate the Euclidean distance, relative azimuth, speed difference and collision time between each traffic participant at each moment, construct a multi-dimensional spatiotemporal relationship matrix and perform standardization processing;

[0121] S72: Weighted aggregation is performed on each row of the standardized multidimensional spatiotemporal relationship matrix to generate the interaction features of each traffic participant;

[0122] S73: Generate connection weights between traffic participants using a social force model based on the Euclidean distance between them and their position vectors.

[0123] S74: Using traffic participants as nodes and their interaction features as initial features, a traffic participant interaction graph is constructed using the connection weights between traffic participants as edges. The traffic participant interaction graph is then input into the GGNN network model to predict the probability of interaction risks occurring between traffic participants. If the probability value is greater than a set threshold, a high-risk interaction event is determined to have occurred between traffic participants.

[0124] In this embodiment, step S7 constructs a multidimensional analysis model based on the final trajectory information of each traffic participant and local map data to identify high-risk interaction events between traffic participants: First, at each moment, the Euclidean distance, relative azimuth angle, speed difference, and collision time between any two traffic participants are calculated. These spatiotemporal features are integrated into a multidimensional spatiotemporal relationship matrix, and the differences in the dimensions of different features are eliminated through standardization. Then, each row of the standardized matrix is ​​weighted and aggregated to generate interaction features that reflect the interaction relationship between a single traffic participant and all other participants. Next, based on the Euclidean distance between participants (the closer the distance, the higher the weight) and the position vector, the interaction strength between participants is calculated through a social force model and used as the connection weights of the interaction graph. Finally, with traffic participants as nodes, interaction features as initial node features, and connection weights as edges, a traffic participant interaction graph is constructed and input into the GGNN network model for risk probability prediction. If the predicted probability value is greater than a set threshold, it is determined that a high-risk interaction event has occurred between the corresponding participants. This step effectively solves the problem that existing high-interaction risk identification relies solely on a single distance indicator and cannot fully reflect complex spatiotemporal interaction relationships. By constructing multi-dimensional spatiotemporal features and modeling with graph neural networks, it not only captures the positional relationships between participants but also integrates dynamic interaction information such as speed and orientation, significantly improving the accuracy and comprehensiveness of risk identification. At the same time, by combining the connection weights generated by the social force model, the interaction graph more closely reflects the mutual influence patterns of participants in real traffic scenarios. Ultimately, the identified high-interaction risk events can accurately supplement key dangerous scenario elements in autonomous driving test scenarios, providing more targeted scenario support for verifying the risk response capabilities of autonomous driving systems.

[0125] S8: Extract traffic accident events of traffic participants, traffic violation events of each traffic participant, and the final trajectory information of traffic participants within a preset time period before and after the occurrence of high-interaction risk events between traffic participants to form the scene information of the corresponding events.

[0126] Another aspect of the present invention provides an autonomous driving test scene extraction system based on roadside multi-modal perception, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the aforementioned autonomous driving test scene extraction method based on roadside multi-modal perception.

[0127] Another aspect of the present invention provides a computer storage medium storing a remote monitoring program, which, when executed by a processor, implements the aforementioned method for extracting autonomous driving test scenarios based on roadside multi-modal perception.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] In summary, this invention first acquires image frames, millimeter-wave radar data, lidar data, and local map data of the target roadside area, laying a comprehensive data foundation for scene extraction and solving the problem of incomplete multi-source data acquisition in existing technologies. Then, it utilizes a YOLOv8s model with a dual-branch detection head to simultaneously identify traffic participants and traffic accident event information, overcoming the limitations of existing solutions where target and event detection are separated, leading to missed or delayed accident events. Subsequently, it combines local map data with multi-sensor data for matching and fusion to generate trajectory point data, avoiding the insufficient fusion accuracy problem caused by traditional simple data overlay. Through Kalman filter tracking, multi-dimensional anomaly detection, and hierarchical trajectory reconstruction, it improves trajectory accuracy and smoothness, overcoming the defects of anomaly interference in existing trajectory processing. Based on the final trajectory and local map data, it achieves traffic violation detection and high-interaction risk identification, solving the problems of fixed violation judgments and singular risk identification in existing systems. Finally, it extracts the trajectories before and after key events to form a scene, ensuring that the extracted test scene is complete, realistic, and accurate, effectively meeting the stringent requirements of high-level autonomous driving system safety verification for high-quality test scenes.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for extracting autonomous driving test scenarios based on roadside multi-modal perception, characterized in that, include: S1: Acquire image frames, millimeter-wave radar data, lidar data, and local map data of the target roadside area; S2: Input the continuous image frame sequence into the YOLOv8s model of the dual-branch detection head for detection and identify traffic participant information and traffic accident event information; S3: Based on local map data, the trajectory point data of each traffic participant is obtained by matching and fusing millimeter-wave radar data, lidar data, traffic participant information and traffic accident event information. S4: Based on the trajectory point data of each traffic participant, the Kalman filter algorithm is used to track and manage the trajectory point data of each traffic participant, and generate the initial trajectory information of each traffic participant. S5: Combine the initial trajectory information of each traffic participant with local map data to detect outliers, and reconstruct the detected outliers to generate the final trajectory information of each traffic participant. S6: Based on the final trajectory information of each traffic participant and combined with local map data, detect traffic violations of each traffic participant and identify traffic violation information of each traffic participant; S7: Identify the interaction risks between traffic participants based on their final trajectory information and local map data, and identify high-risk interaction events between traffic participants. S8: Extract traffic accident events of traffic participants, traffic violation events of each traffic participant, and the final trajectory information of traffic participants within a preset time period before and after the occurrence of high-interaction risk events between traffic participants to form the scene information of the corresponding events.

2. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, The YOLOv8 model with the dual-branch detection head includes: cascading the first and third C2f modules in the YOLOv8 backbone network through convolutional layers; cascading the second and fourth C2f modules in the YOLOv8 backbone network through convolutional layers; adding a Dynamic Conv layer between the third and fourth C2f modules; modifying each prediction head of the original YOLOv8 backbone network into a dual-branch detection head; the dual-branch detection head includes a target branch detection head and an accident branch detection head; the target branch detection head and the accident branch detection head share the same input feature; and adding a BiFormer spatial attention module to the accident branch detection head to preprocess the input feature so that the feature focuses on the accident collision region.

3. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, Step S3 includes: S31: Perform DBSCAN clustering on the LiDAR data to generate 3D bounding boxes for each traffic participant, and project the 3D bounding boxes of each traffic participant onto the pixel plane to generate 2D projection boxes for each traffic participant. S32: Match the 2D projection box of the traffic participant with the visual detection box of the traffic participant, and perform weighted fusion on the successfully matched 2D projection box and visual detection box to obtain the fused detection box of the traffic participant. S33: Based on the depth information of the 3D bounding box center, project the fused detection box of the traffic participant onto the world coordinate system to obtain the 3D fused detection box of the traffic participant; where the center of the 3D fused detection box is the coordinate of the trajectory point of the traffic participant. S34: Transform the polar coordinates of the center point of each traffic participant in the millimeter-wave radar data to the world coordinate system, and match the traffic participants according to the distance between the center point and the coordinates of the traffic participants' trajectory points; calculate the speed and acceleration information of the matched traffic participants based on their lidar data. S35: Generate the trajectory point data of traffic participants based on traffic participant information, traffic accident event information, trajectory point coordinates of traffic participants, 3D fusion detection boxes of traffic participants, and the speed and acceleration of traffic participants.

4. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, The process of using the Kalman filter algorithm to track and manage the trajectory point data of each traffic participant, and generating initial trajectory information for each traffic participant, includes: S41: Convert the trajectory point data of traffic participants to trajectory point data in a rectangular coordinate system, and define the state vector of each traffic participant based on the coordinates, velocity and acceleration of the traffic participants in the rectangular coordinate system. S42: Define the state transition matrix based on the uniform acceleration model, and define the process noise covariance matrix based on the speed and acceleration of traffic participants in the previous frame; S43: Predict the prior state and prior covariance of traffic participants in the previous frame based on their optimal state and optimal covariance in the previous frame. S44: Generate the 3D prior bounding box of the traffic participants in the previous frame based on their prior states and the standard 3D bounding box size of the traffic participants. S45: Match the 3D fusion detection boxes of each traffic participant in the current frame with the 3D prior bounding boxes of the traffic participants in the current frame. If the match is successful, add the trajectory point data of the traffic participants in the current frame to the trajectory of the traffic participants to generate the initial trajectory information of the traffic participants. Then, based on the state observation values ​​of the traffic participants in the current frame, correct the prior state and prior covariance of the traffic participants in the previous frame to obtain the highest priority prior state and highest priority covariance of the traffic participants in the current frame.

5. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, Step S5 includes: S51: Based on the coordinates of each trajectory point in the initial trajectory information of traffic participants and combined with local map data, if a trajectory point in the initial trajectory information of a traffic participant that exceeds a set threshold is within the left and right boundaries of a certain lane, then that lane is taken as the lane to which the initial trajectory of the traffic participant belongs; and trajectory points that are not within that lane are removed as abnormal points. S52: Based on the category of traffic participants, determine the speed and acceleration conditions for each traffic participant, and remove trajectory points that do not meet the speed and acceleration conditions as outliers. S53: Set up a sliding window to process the initial trajectory information of traffic participants. Within each sliding window, calculate the mean and standard deviation of the trajectory points. Based on the mean and standard deviation of the trajectory points within the sliding window, identify outlier trajectory points within the sliding window as abnormal points and remove them. S54: In the initial trajectory of the traffic participant after processing in steps S51 to S53, for missing trajectories with a trajectory interruption duration less than or equal to a set threshold, the missing trajectory is fitted and repaired by using third-order B-spline curve fitting based on the valid trajectory points before and after the interruption; for missing trajectories with an interruption duration greater than the set threshold, based on the valid trajectory points before the interruption, and constrained by the centerline equation of the lane to which the initial trajectory belongs, the distance between the predicted trajectory and the centerline of the lane is constrained to be less than the set threshold, and H∞ filtering prediction is performed frame by frame to repair the missing trajectory and obtain the final trajectory information of the traffic participant.

6. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, The objective function used when fitting and repairing the missing trajectory using B-spline curve fitting includes: in, This indicates that the third-order B-spline curve has the following parameters. The coordinates at that location These are curve parameters associated with timestamps, representing the trajectory points predicted by the fitted curve; This represents the actual trajectory points before and after the interruption; The second derivative of the curve is represented. represents the smoothing coefficient; n represents the number of fitted trajectory points.

7. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, The information identifying traffic violations by each traffic participant includes: S61: Define the lane to which the final trajectory of a traffic participant belongs as the target lane, calculate the angle between the velocity direction of the traffic participant's trajectory point and the permissible driving direction of the target lane, and determine that the traffic participant has committed a wrong-way driving violation if the angle is greater than a set threshold. S62: Count the number of traffic participants in the target lane in the most recent preset time period. Based on the count and the maximum allowable number of traffic in the target lane, the maximum speed limit of the target lane is weighted to obtain the dynamic speeding threshold of the target lane. If the speed of the traffic participant's trajectory point is greater than the dynamic speeding threshold of the target lane, it is determined that the traffic participant has committed a speeding violation. S63: Extract the stop line data of the target lane based on the local map data, calculate the distance from each trajectory point of the traffic participant to the stop line, and if the distance between the trajectory point and the stop line is less than a set threshold and the speed of the trajectory point of the traffic participant is greater than the set threshold during the continuous red light event corresponding to the stop line, then it is determined that the traffic participant has committed a red light violation.

8. The method for extracting autonomous driving test scenarios based on roadside multi-modal perception according to claim 1, characterized in that, The information for identifying high-risk interaction events between traffic participants includes: S71: Based on the final trajectory information of each traffic participant, calculate the Euclidean distance, relative azimuth, speed difference and collision time between each traffic participant at each moment, construct a multi-dimensional spatiotemporal relationship matrix and perform standardization processing; S72: Weighted aggregation is performed on each row of the standardized multidimensional spatiotemporal relationship matrix to generate the interaction features of each traffic participant; S73: Generate connection weights between traffic participants using a social force model based on the Euclidean distance between them and their position vectors. S74: Using traffic participants as nodes and their interaction features as initial features, a traffic participant interaction graph is constructed using the connection weights between traffic participants as edges. The traffic participant interaction graph is then input into the GGNN network model to predict the probability of interaction risks occurring between traffic participants. If the probability value is greater than a set threshold, a high-risk interaction event is determined to have occurred between traffic participants.

9. A system for extracting autonomous driving test scenarios based on roadside multi-modal perception, characterized in that, The system includes a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the autonomous driving test scenario extraction method based on roadside multimodal perception as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores a remote monitoring program, which, when executed by the processor, implements an autonomous driving test scenario extraction method based on roadside multi-modal perception as described in any one of claims 1 to 8.

Images