Processing system for sensing performance test and sensor deployment scheme evaluation

Abstract

The embodiment of the invention relates to a processing system for sensing performance testing and sensor deployment scheme evaluation. The system comprises a high-precision map library, a basic element library, a scene modeling module, a scene template library, a scene simulation module, a scene simulator, a data labeling module, an intelligent model library, a test database, a scene evaluation module, a perception test module and an evaluation / test report storage module. Through the method, a large amount of test data can be constructed, the scene complexity of the test data can be improved, the evaluation efficiency of a sensor deployment scheme can be improved, and the test sufficiency of a tested person can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing system for perception performance testing and sensor deployment scheme evaluation. Background Art

[0002] The roadside perception algorithm processing interface (module, device, service or system) used in the intelligent connected vehicle-road cooperative system is used to perform target detection or event recognition processing based on the sensor data (such as images or point clouds) collected from the roadside. The sensitivity and accuracy of the roadside perception algorithm processing interface (module, device, service or system) depends on the test adequacy of its own perception performance test. In order to have a high enough test adequacy, not only a large amount of test data is required, but also the test data must have sufficient scene complexity (such as various meteorological conditions, various lighting conditions, various road conditions, etc.). However, in the actual roadside data collection process, it is difficult to collect such a large amount of complex scene data.

[0003] In addition, a corresponding sensor deployment plan needs to be drawn up before deploying roadside equipment on roads / intersections. In order to ensure that the sensors deployed by the plan can meet the pre-established design requirements (such as coverage, overlapping effects, etc.), the implementation effect of the plan is usually verified through manual on-site mapping and measurement. If the verification conclusion is that it does not meet the standards, multiple plan adjustments and multiple manual verifications will be required. Obviously, this traditional approach is affected by human factors, which will lead to a long overall plan formulation time and low efficiency.

[0004] Therefore, how to improve the test adequacy of the perception performance test of the subject (roadside perception algorithm processing interface, module, equipment, service or system) when roadside data is limited and how to evaluate the sensor deployment plan without human intervention are the technical problems that the present invention needs to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a processing system for perception performance testing and sensor deployment scheme evaluation in view of the defects of the prior art, the system comprising: a high-precision map library, a basic element library, a scenario modeling module, a scenario template library, a scenario simulation module, a scenario simulator, a data annotation module, an intelligent model library, a test database, a scenario evaluation module, a perception test module and an evaluation / test report storage module; wherein the high-precision map library is used to store high-precision maps of road traffic; the basic element library is used to store static / dynamic / signal light / environment / sensor / traffic flow element templates; the scenario modeling module is used to provide a manual editing interface and perform traffic scene modeling through manual configuration by the user, and is also used to automatically model traffic scenes according to the real scene data set and environment description text input by the user as well as the intelligent model library and the basic element library, and is also used to construct a traffic scene model of any type of specified road conditions by configuring and adjusting the traffic scene template. The board is also used to expand the template of the traffic scene template of the scene template library by data enhancement; the scene template library is used to store multiple traffic scene templates; the scene simulation module is used to distribute the traffic scene template to the scene simulator and / or the scene evaluation module according to the simulation operation instruction input by the user; the scene simulator is used to load the traffic scene template and simulate it dynamically, and simulate the sensor data based on the types and properties of various sensors loaded in the scene during the simulation process, and provide multiple data output interfaces; the data annotation module is used to produce test data according to the scene simulator and the intelligent model library; the intelligent model library at least includes a visual / point cloud target detection / event recognition model; the scene evaluation module is used to evaluate the sensor deployment plan of the traffic scene template; the perception test module is used to test the perception performance of the testee (roadside perception algorithm processing interface, module, device, service or system) according to the test database. Through the scene modeling module + scene simulator + data annotation module of the present invention, not only a large amount of test data can be constructed, but also the scene complexity of the test data can be improved. Through the scene evaluation module of the present invention, the sensor deployment plan can be evaluated without manual intervention. Through the perception test module of the present invention, the test adequacy of the testee can be improved based on the massive data with sufficient scene complexity.

[0006] To achieve the above-mentioned purpose, an embodiment of the present invention provides a processing system for perception performance testing and sensor deployment scheme evaluation, the system comprising: a high-precision map library, a basic element library, a scene modeling module, a scene template library, a scene simulation module, a scene simulator, a data annotation module, an intelligent model library, a test database, a scene evaluation module, a perception test module and an evaluation / test report storage module;

[0007] The scenario modeling module is respectively connected to the high-precision map library, the basic element library, the scenario template library and the intelligent model library; the scenario simulation module is respectively connected to the scenario template library, the scenario simulator and the scenario evaluation module; the scenario simulator is connected to the data annotation module; the data annotation module is respectively connected to the intelligent model library and the test database; the evaluation / test report storage module is respectively connected to the scenario evaluation module and the perception test module; the perception test module is connected to the test database; the perception test module is also connected to a subject outside the system; the subject is a type of roadside perception algorithm processing interface, module, device, service or system;

[0008] The high-precision map library is used to store multiple high-precision road traffic maps;

[0009] The basic element library is used to store multiple types of static element templates, multiple types of dynamic element templates, multiple types of signal light element templates, multiple types of environmental element templates, multiple types of sensor element templates and traffic flow element templates;

[0010] The scene modeling module is used to perform traffic scene modeling according to the high-precision map library, the basic element library and the manual configuration results of the user to obtain the corresponding traffic scene template and store it in the scene template library; the scene modeling module is also used to automatically model the traffic scene according to the real scene data set and environment description text input by the user, the intelligent model library and the basic element library to obtain the corresponding traffic scene template and store it in the scene template library; the scene modeling module is also used to construct the traffic scene template of any type of specified road condition by configuring and adjusting the traffic scene template and store it in the scene template library; the scene modeling module is also used to perform template expansion on the traffic scene template in the scene template library by data enhancement;

[0011] The scene template library is used to store a plurality of traffic scene templates; each of the traffic scene templates corresponds to a unique template identifier;

[0012] The scenario simulation module is used to perform traffic scenario template distribution processing on the scenario simulator and / or the scenario evaluation module according to the simulation operation instruction input by the user;

[0013] The scene simulator is used to load the current traffic scene template sent by the scene simulation module when the corresponding simulator state is in the idle state, and perform dynamic simulation based on the currently loaded scene; and clear the historical data in the built-in data cache at the start of the simulation; and set the simulator state to the working state during the simulation process, and set the simulator state to the idle state at the end of the current simulation process; and perform sensor data simulation based on the types and properties of various sensors loaded in the scene during the simulation process and store the simulation data into the built-in data cache; and provide a simulator state query interface for outputting the current simulator state; and provide a sensor query interface for outputting the sensor list in the current simulation scene; and provide a simulation data query interface for outputting any sensor simulation data stored in the built-in data cache; the simulator state includes idle state and working state;

[0014] The data annotation module is used to generate test data according to the scenario simulator and the intelligent model library and add the obtained test data to the test database;

[0015] The intelligent model library includes at least a visual target detection model, a point cloud target detection model, a visual event recognition model, and a point cloud event recognition model;

[0016] The test database includes at least a visual target detection dataset, a point cloud target detection dataset, a visual event recognition dataset, and a point cloud event recognition dataset;

[0017] The scenario evaluation module is used to evaluate the sensor deployment scheme of the current traffic scenario template to generate a corresponding first evaluation report and store it in the evaluation / test report storage module;

[0018] The perception test module is used to test the perception performance of the subject according to the test database to obtain a corresponding first test report and store it in the evaluation / test report storage module.

[0019] Preferably, the template parameters of the static element template include static element types and static element attributes; the static element types include at least road sections, intersections, ramps, roundabouts, guide signs, instruction signs, construction signs, warning signs, prohibition signs, instruction markings, warning markings, prohibition markings, road poles, obstacles, roadside equipment, buildings, and plants; the static element attributes are the attribute sets of the current static element, including at least ground coordinates, shape and size, orientation, and color;

[0020] The template parameters of the dynamic element template include dynamic element type, dynamic element attribute, element motion mode and element motion trajectory; the dynamic element type includes at least cars, motorcycles, bicycles and pedestrians; the dynamic element attribute is the attribute set of the current dynamic element, including at least shape, size and color; the element motion mode includes multiple types of motion models; the element motion trajectory is composed of multiple motion trajectory points; each of the motion trajectory points includes at least trajectory point coordinates, trajectory point speed, trajectory point acceleration and trajectory point heading angle;

[0021] The template parameters of the signal light element template include signal light type, signal light attribute and signal light transformation rule; the signal light type includes at least motor vehicle signal light, non-motor vehicle signal light, pedestrian crossing signal light, direction indicator light, lane signal light, flashing warning signal light, road and railway level crossing signal light; the signal light attribute is the attribute set of the current signal light, including at least spatial coordinates and shape type; the signal light transformation rule includes the color of the signal light at the initial moment, the rotation order of the signal light color and the duration of each signal light color;

[0022] The template parameters of the environmental element template include the environmental element type and the intensity change sequence; the environmental element type includes at least sunny, rainy, snowy, foggy, natural light source, and street light source; when the environmental element type is sunny, rainy, snowy, and foggy, the intensity change sequence includes the visibility change sequence and the road surface slipperiness change sequence; when the environmental element type is natural light source and street light source, the intensity change sequence includes the illumination intensity change sequence, the illumination angle change sequence, and the illumination range change sequence;

[0023] The template parameters of the sensor element template include sensor type, sensor attributes and sensor jitter settings; the sensor type includes at least camera and laser radar; when the sensor type is a camera, the sensor attributes include at least camera internal parameters, camera external parameters, camera field of view angle, camera sampling rate, camera installation position, camera installation angle, and camera maximum shooting distance; when the sensor type is a laser radar, the sensor attributes include at least radar internal parameters, radar external parameters, radar field of view angle, radar sampling rate, radar installation position, radar installation angle, and radar maximum scanning distance; the sensor jitter settings include jitter direction, jitter frequency, and jitter amplitude;

[0024] The template parameters of the traffic flow element template at least include vehicle traffic flow, pedestrian traffic flow, driving speed and congestion index;

[0025] The designated road conditions at least include normal driving of vehicles on the road / intersection, rear-end collision of vehicles on the road / intersection, turning of vehicles on the road / intersection, scratching of vehicles on the road / intersection, collision of vehicles on the road / intersection, road construction, driving on narrow roads, driving on curves, driving on slopes, congestion during rush hour, driving on roads / intersections in bad weather, driving on roads / intersections under strong light conditions, and driving on roads / intersections under weak light conditions;

[0026] The traffic scene template includes a first template high-precision map and a first template element set; the first template element set is composed of a plurality of first template elements, and each of the first template elements corresponds to a unique element identifier;

[0027] The query parameter of the simulator status query interface is empty; the interface return data of the simulator status query interface is the corresponding simulator status;

[0028] The query parameter of the sensor query interface is empty; the interface return data of the sensor query interface is the corresponding sensor list; the sensor list includes multiple sensor records; each sensor record includes a sensor identifier and a sensor type; the sensor type includes at least a camera and a laser radar;

[0029] The query parameters of the simulation data query interface include sensor identification parameters, sensor type parameters and data time parameters; the sensor type parameters include at least camera and lidar; the data time parameter is a time period; the return data of the simulation data query interface is the image sequence and point cloud sequence corresponding to the camera or lidar specified by the current query parameters.

[0030] Preferably, the scene modeling module is specifically used for, when the traffic scene modeling is performed according to the high-precision map library, the basic element library and the manual configuration result of the user to obtain the corresponding traffic scene template and store it in the scene template library:

[0031] Providing a visual map editing interface for the user; and providing the user with a map retrieval option and a map editing window on the visual map editing interface; and after the user retrieves and selects a road traffic high-precision map from the high-precision map library through the map retrieval option, making a map copy for the current road traffic high-precision map and loading it into the map editing window; and receiving the user's element modification and / or map rotation and / or map cropping operation results on the current map copy through the map editing window, and saving the current map copy as a corresponding first template high-precision map when the user confirms saving through the map editing window;

[0032] When the first template high-precision map is successfully saved, a visual element editing interface is provided for the user; a map loading window and an element toolbar are provided for the user on the visual element editing interface; and the first template high-precision map is loaded with a map through the map loading window; and each element template of the basic element library is assigned a corresponding tool icon, and all the tool icons are loaded through the element toolbar; and when the user selects any one of the tool icons, an icon dragging function is provided for the user to drag the current tool icon to the map loading window, and an element template copy is generated for the element template corresponding to the current tool icon, and a unique element identifier is generated for the current element template copy; and when the user selects any one of the tool icons on the map loading window, a visual element configuration window is provided for the element template copy corresponding to the current tool icon for receiving the user's The modification result of any other template parameter of the current element template copy except the element type is saved; and when any tool icon is newly added or displaced on the map loading window, the position attribute of the element template copy corresponding to the current tool icon is modified based on the map coordinates of the current tool icon on the map loading window; and when the user confirms saving through the visual element editing interface, the element template copy corresponding to each tool icon loaded on the map loading window is saved as a corresponding first template element, and the corresponding first template element set is composed of all the first template elements obtained; and a corresponding traffic scene template is composed of the obtained first template high-precision map and the first template element set; and a unique template identifier is set for the traffic scene template, and the traffic scene template with template identifier information is stored in the scene template library;

[0033] Among them, there is and can only be loaded one tool icon corresponding to the traffic flow element template on the map loading window.

[0034] Preferably, the real scene data set includes a first high-precision map, a first overhead view sequence, and one or more first horizontal view sequences; the real scene data set corresponds to a real traffic scene; the first high-precision map is a high-precision map of road traffic corresponding to the current real traffic scene; the first overhead view sequence and each of the first horizontal view sequences each correspond to a set of camera internal and external parameters; the first overhead view sequence is formed by a plurality of first overhead views sorted in chronological order, and each of the first overhead views corresponds to a camera position coordinate and an image timestamp; each of the first horizontal view sequences is formed by a plurality of first horizontal views sorted in chronological order, and each of the first horizontal view sequences corresponds to a camera position coordinate and an image timestamp; when the number of the first horizontal view sequences is greater than 1, the shooting angles of every two of the first horizontal view sequences are different.

[0035] Preferably, the scene modeling module is specifically used for automatically modeling the traffic scene according to the real scene data set and the environment description text input by the user as well as the intelligent model library and the basic element library to obtain the corresponding traffic scene template and store it in the scene template library:

[0036] Receiving the real scene data set and the environment description text input by the user as the corresponding first scene data set and first environment description text;

[0037] and extracting the corresponding first high-precision map, the first overhead view sequence and all the first horizontal view sequences from the first scene data set; and using the first high-precision map as a corresponding first template high-precision map; and performing weather type and light source type keyword recognition on the first environment description text to obtain corresponding weather type keywords and light source type keywords; when the weather type keyword is not empty, it at least includes sunny, rainy, snowy and foggy; when the light source type keyword is not empty, it at least includes natural light source and street light source;

[0038] and performing target detection processing on each of the first overhead view and each of the first plan view based on the visual target detection model of the intelligent model library to obtain a corresponding plurality of first target objects to form a corresponding first target object group; and all of the first target object groups corresponding to the first overhead view are sorted in chronological order to form a corresponding first target object group sequence; and all of the first target object groups corresponding to each of the plan views are sorted in chronological order to form a corresponding first target object group sequence; and each of the first target objects corresponds to a group of target object attributes, including target type, target coordinates, target size, and target orientation;

[0039] Based on the camera internal and external parameters and camera coordinates of each of the first target objects, the target coordinates, target size and target orientation of the current target object are converted from the image coordinate system to the world coordinate system; and the image timestamps corresponding to each of the first target object groups are added as corresponding target timestamps to the target object attributes of each of the target objects in the current first target object group; the updated target object attributes of the first target object include target type, target coordinates, target size, target orientation and target timestamp;

[0040] And each of the first target object group sequences is used as the corresponding current target object group sequence; based on a preset target association algorithm, target association is performed on every two adjacent first target object groups in the current target object group sequence, and based on the target association result, the target object attributes of each of the first target objects corresponding to the same associated target are used as the trajectory point attributes of a corresponding first trajectory point, and all the first trajectory points corresponding to the same associated target are sorted in chronological order to form a corresponding first target trajectory; the target association algorithm at least includes the Hungarian algorithm; the first target trajectory is formed by sequentially sorting a plurality of the first trajectory points, and the trajectory point attributes of each of the first trajectory points include target type, target coordinates, target size, target orientation and target timestamp; the target types corresponding to all the first trajectory points on the first target trajectory are the same and will not change;

[0041] Taking target coordinates and target orientation as observation quantities, and taking target speed, target acceleration and target heading angle as state quantities, the target speed, target acceleration and target heading angle corresponding to each first trajectory point on each first target trajectory are estimated according to a preset filter algorithm, and the estimation result is added to the trajectory point attributes corresponding to each first trajectory point; the updated trajectory point attributes of the first trajectory point include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the filter algorithm includes at least a Kalman filter algorithm;

[0042] The sequence time lengths of the first overhead view sequence and each of the first horizontal view sequences are counted to obtain the corresponding first sequence time length; and the longest first sequence time length is used as the corresponding maximum time length; and the maximum time length is divided into a plurality of time scales based on a preset minimum unit time length; and the target timestamps of the first track points of each of the first target trajectories on each of the time scales are aligned with the current time scale; and the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of the first track points of each of the first target trajectories on each of the time scales are estimated according to the filter algorithm or the interpolation algorithm; and the corresponding second target trajectory is formed by the first track points of each of the first target trajectories on all the time scales;

[0043] The trajectory similarity of every two second target trajectories is calculated to obtain the corresponding first trajectory similarity; all the second target trajectories are clustered based on a preset trajectory similarity threshold to obtain one or more corresponding second target trajectory sets; all the first trajectory points of all the second target trajectories of each second target trajectory set on each time scale are fused to obtain a corresponding second trajectory point, and all the obtained second trajectory points are sequentially sorted to form a corresponding first target fused trajectory; the second target trajectory set is composed of one or more second target trajectories; when the number of the second target trajectories in the second target trajectory set is greater than 1, any two of the second target trajectories in the current set are The first trajectory similarities between the target trajectories all exceed the trajectory similarity threshold; the first target fusion trajectory is formed by sequentially sorting a plurality of second trajectory points, and the trajectory point attributes of each of the second trajectory points include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the target type and target timestamp of each of the second trajectory points are consistent with the target type and target timestamp of all corresponding first trajectory points; the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of each of the second trajectory points are fused from the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of all corresponding first trajectory points;

[0044] Based on the target type corresponding to each of the first target fusion trajectories, a corresponding static, dynamic, signal light or sensor element template is selected from the basic element library as a corresponding first template element; and a corresponding unique element identifier is assigned to the current first template element; and template parameters of the current first template element are configured based on all trajectory point attributes of the current first target fusion trajectory;

[0045] When the weather type keyword is not empty, the environment element template whose environment element type in the basic element library matches the weather type keyword is used as a corresponding first template element; and a corresponding unique element identifier is assigned to the current first template element; and the intensity change timing of the current first template element is configured based on a set of preset visibility change timings and preset road slipperiness change timings corresponding to the weather type keyword;

[0046] When the light source type keyword is not empty, the environment element template whose environment element type in the basic element library matches the light source type keyword is used as a corresponding first template element; and a corresponding unique element identifier is assigned to the current first template element; and the intensity transformation timing of the current first template element is configured based on a set of preset illumination intensity transformation timings, preset illumination angle transformation timings, and preset illumination range transformation timings corresponding to the light source type keyword;

[0047] And all the first template elements obtained this time form a corresponding first template element set; and the first template high-precision map and the first template element set form a corresponding traffic scene template; and a unique template identifier is set for the traffic scene template, and the traffic scene template with template identifier information is stored in the scene template library.

[0048] Preferably, the scene modeling module is specifically used to use each of the existing traffic scene templates in the scene template library as the corresponding current base template when the template expansion of the traffic scene template in the scene template library is carried out by data enhancement; and the first template elements corresponding to each type of the dynamic elements in the current base template are recorded as corresponding first dynamic elements; and multiple derived scene templates are generated by dynamically adjusting the element motion mode and the element motion trajectory of one or more of the first dynamic elements in the current base template; and a unique template identifier is set for each of the derived scene templates; and a corresponding derived scene template set is composed of all the derived scene templates corresponding to the current base template; and all the obtained derived scene template sets are added to the scene template library.

[0049] Preferably, the scenario simulation module is specifically used for, when the scenario simulator and / or the scenario evaluation module performs traffic scenario template distribution processing according to the simulation operation instruction input by the user:

[0050] Receive the simulation operation instruction input by the user; and extract the corresponding instruction type and instruction parameter from the simulation operation instruction; the simulation operation instruction includes the instruction type and the instruction parameter; the instruction type includes the first type, the second type and the third type; when the instruction type is the first or second type, the instruction parameter is a template identifier; when the instruction type is the third type, the instruction parameter is a template identifier sequence;

[0051] and identifying the instruction type;

[0052] If the instruction type is the first or second type, the template identifier in the instruction parameter is used as the corresponding current template identifier; and the traffic scene template corresponding to the current template identifier in the scene template library is used as the corresponding current traffic scene template; and when the instruction type is the first type, the current traffic scene template is sent to the scene simulator; and when the instruction type is the second type, the current traffic scene template is sent to the scene simulator and the scene evaluation module respectively;

[0053] If the instruction type is the third type, the template identification sequence in the instruction parameter is used as the corresponding current template identification sequence; and the template identifications of the current template identification sequence are sequentially traversed; and in the traversal process, the currently traversed template identification is used as the corresponding current template identification; and the traffic scene template corresponding to the current template identification in the scene template library is used as the corresponding current traffic scene template; and the simulation duration of the current traffic scene template is estimated to obtain the corresponding current simulation duration; and based on the preset first delay factor and the current simulation duration, the corresponding first waiting duration = current simulation duration × (1 + first delay factor), 0<first delay factor<1 is calculated; and the current traffic scene template is sent to the scene simulator; and after waiting for the first waiting time from the completion of this scene sending, it goes to the next template identification for traversal until the traversal of the last template identification is completed.

[0054] Preferably, all types of models in the intelligent model library have completed model training based on a large data set in advance; the visual target detection model is used to perform target detection processing based on the model input image and output a corresponding detection target set; the point cloud target detection model is used to perform target detection processing based on the model input point cloud and output a corresponding detection target set; the visual event recognition model is used to perform traffic event recognition processing based on the model input image and output one or more corresponding event types; the point cloud event recognition model is used to perform traffic event recognition processing based on the model input point cloud and output one or more corresponding event types;

[0055] The visual target detection dataset includes multiple first data records; the first data record includes a first training image and a first labeled target set; the point cloud target detection dataset includes multiple second data records; the second data record includes a first training point cloud and a second labeled target set; the visual event recognition dataset includes multiple third data records; the third data record includes a second training image and a first labeled event set; the point cloud event recognition dataset includes multiple fourth data records; the fourth data record includes a second training point cloud and a second labeled event set.

[0056] Preferably, the data annotation module is specifically used for, when the test data is produced according to the scenario simulator and the intelligent model library and the obtained test data is added to the test database:

[0057] Initializing a first waiting flag in advance as the end of waiting; the first waiting flag includes starting waiting and ending waiting;

[0058] and obtaining the corresponding simulator state through the simulator state query interface of the scenario simulator regularly at a preset first query frequency;

[0059] And identify the simulator state; if the simulator state is in the working state, set the first waiting flag to start waiting; if the simulator state is in the idle state, then when the first waiting flag is start waiting, the sensor data of each camera and each laser radar in the current simulation scene of the scene simulator are acquired through the sensor query interface and the simulation data query interface of the scene simulator to obtain the corresponding multiple first image sequences and multiple first point cloud sequences, and set the first waiting flag to end waiting after all sensor data are obtained; each of the first image sequences corresponds to a camera, and each of the first point cloud sequences corresponds to a laser radar; the first image sequence is composed of multiple first images sorted in chronological order; the first point cloud sequence is composed of multiple first point clouds sorted in chronological order;

[0060] and merging all the first image sequences obtained this time to obtain a corresponding first image set; and merging all the first point cloud sequences obtained this time to obtain a corresponding first point cloud set;

[0061] And perform a round of traversal on all the first images of the first image set; and in this round of traversal, take the first image currently traversed as the corresponding current image; and input the current image into the visual target detection model for target detection processing and use the detection target set output by the model as a corresponding first label target set; and input the current image into the visual event recognition model for traffic event recognition processing and form a corresponding first label event set with all event types output by the model; and use the current image as a corresponding first training image and a corresponding second training image; and form a corresponding first data record with the first training image corresponding to the current image and the first label target set; and form a corresponding third data record with the second training image corresponding to the current image and the first label event set; and at the end of this round of traversal, all the first data records obtained in this round of traversal form a corresponding first data record set, and all the third data records obtained in this round of traversal form a corresponding third data record set;

[0062] And perform a round of traversal on all the first point clouds of the first point cloud set; and in this round of traversal, take the currently traversed first point cloud as the corresponding current point cloud; and input the current point cloud into the point cloud target detection model for target detection processing and use the detection target set output by the model as a corresponding second label target set; and input the current point cloud into the point cloud event recognition model for traffic event recognition processing and form a corresponding second label event set with all event types output by the model; and use the current point cloud as a corresponding first training point cloud and a corresponding second training point cloud; and form a corresponding second data record with the first training point cloud corresponding to the current point cloud and the second label target set; and form a corresponding fourth data record with the second training point cloud corresponding to the current point cloud and the second label event set; and at the end of this round of traversal, form a corresponding second data record set with all the second data records obtained in this round of traversal, and form a corresponding fourth data record set with all the fourth data records obtained in this round of traversal;

[0063] And add the first data record set obtained this time to the visual target detection data set of the test database; and add the second data record set obtained this time to the point cloud target detection data set of the test database; and add the third data record set obtained this time to the visual event recognition data set of the test database; and add the fourth data record set obtained this time to the point cloud event recognition data set of the test database.

[0064] Preferably, the scenario evaluation module is specifically used for, when evaluating the sensor deployment scheme of the current traffic scenario template to generate a corresponding first evaluation report and storing it in the evaluation / test report storage module:

[0065] Extracting the first template high-precision map and the first template element set corresponding to the current traffic scene template as the corresponding current high-precision map and current element set; and taking each sensor element in the current element set as a corresponding first sensor; and counting the total number of the first sensors to obtain a corresponding first total number N;

[0066] And perform a round of traversal for all the first sensors; and in this round of traversal, use the first sensor currently traversed as the corresponding current sensor; and identify the sensor type corresponding to the current sensor; if the sensor type corresponding to the current sensor is a camera, then based on the current high-precision map and the camera field of view, camera installation position, camera installation angle and camera maximum shooting distance corresponding to the current sensor, identify the road coverage of the current sensor to obtain the corresponding first coverage range; if the sensor type corresponding to the current sensor is a lidar, then based on the current high-precision map and the radar field of view, radar installation position, radar installation angle and radar maximum scanning distance corresponding to the current sensor, identify the road coverage of the current sensor to obtain the corresponding first coverage range;

[0067] After the current round of traversal of all the first sensors is completed, all the first coverage ranges obtained are range fused to obtain the corresponding first fusion range; all the road / intersection ranges on the current high-precision map are fused to obtain the corresponding second fusion range; the intersection range of the first and second fusion ranges is identified to obtain the corresponding first intersection range; based on the first intersection range and the second fusion range, the corresponding first coverage rate is calculated = (first intersection range / second fusion range) × 100%; whether the first coverage rate is lower than the preset first coverage rate threshold is identified, if so, the corresponding coverage rate evaluation state is set to be unsatisfactory, otherwise, the corresponding coverage rate evaluation state is set to be up to standard; the coverage rate evaluation state includes up to standard and unsatisfactory;

[0068] and identifying the road / intersection type corresponding to the current high-precision map to obtain a corresponding first road / intersection type; and selecting the road / intersection sensor overlapping deployment requirement corresponding to the first road / intersection type from the preset multiple road / intersection sensor overlapping deployment requirements as the corresponding first sensor overlapping deployment requirement; the first road / intersection type at least includes a straight / curved / ramp section of a highway, a straight / curved / ramp section of a national highway, a straight / curved / ramp section of an urban road, a straight / curved / ramp section of a rural road, a T-intersection, a cross intersection, an X-intersection, a compound intersection, and a roundabout;

[0069] And based on the first total number N and all the first coverage ranges, an N×N two-dimensional matrix is ​​constructed and recorded as the corresponding first intersection-and-union ratio matrix; the first row to the Nth row of the first intersection-and-union ratio matrix correspond one-to-one to the first to the Nth first sensors, and the first column to the Nth column correspond one-to-one to the first to the Nth first sensors; the cell value of each matrix cell of the first intersection-and-union ratio matrix is ​​the intersection-and-union ratio of the two first coverage ranges corresponding to the two first sensors corresponding to the current matrix cell;

[0070] and identifying whether the first intersection-to-union ratio matrix meets the first sensor overlapping deployment requirement, and if so, setting the corresponding overlapping deployment evaluation state to meet the standard; if not, setting the corresponding overlapping deployment evaluation state to fail to meet the standard; the overlapping deployment evaluation state includes meeting the standard and failing to meet the standard;

[0071] And the template identifier corresponding to the current traffic scene template is used as the corresponding first evaluation template identifier; and each of the first sensors and the corresponding first coverage range constitutes a corresponding first sensor deployment feature, and all the obtained first sensor deployment features constitute a corresponding first sensor deployment feature set; and the first template high-precision map, the first template element set, the first total number N, the first sensor deployment feature set, the first road / intersection type, the first sensor overlapping deployment requirement, the first coverage rate, the first intersection-to-union ratio matrix, the coverage rate evaluation status and the overlapping deployment evaluation status constitute a corresponding first evaluation report which is stored in the evaluation / test report storage module.

[0072] Preferably, the perception test module is specifically used for, when the perception performance of the subject is tested according to the test database to obtain a corresponding first test report which is stored in the evaluation / test report storage module:

[0073] Step 111, receiving a first test request sent by the subject; and extracting a corresponding first test request identifier, a first test type and a first test total amount from the first test request;

[0074] The first test request includes the first test request identifier, the first test type and the first test total amount; the first test type includes visual target detection, point cloud target detection, visual event recognition, and point cloud event recognition;

[0075] Step 112, identifying the first test type; if the first test type is visual target detection, then based on a preset test data selection rule, the first data record of the total amount of the first test is selected from the visual target detection data set of the test database to form a corresponding first data set; if the first test type is point cloud target detection, then based on the test data selection rule, the second data record of the total amount of the first test is selected from the point cloud target detection data set of the test database to form a corresponding first data set; if the first test type is visual event recognition, then based on the test data selection rule, the third data record of the total amount of the first test is selected from the visual event recognition data set of the test database to form a corresponding first data set; if the first test type is point cloud event recognition, then based on the test data selection rule, the fourth data record of the total amount of the first test is selected from the visual event recognition data set of the test database to form a corresponding first data set;

[0076] Wherein, the first data set includes multiple first test records; the first test record includes first training data and first label data; when the first test type is visual target detection, the first training data and the first label data are the corresponding first training image and the first label target set; when the first test type is point cloud target detection, the first training data and the first label data are the corresponding first training point cloud and the second label target set; when the first test type is visual event recognition, the first training data and the first label data are the corresponding second training image and the first label event set; when the first test type is point cloud event recognition, the first training data and the first label data are the corresponding second training point cloud and the second label event set;

[0077] Step 113, establishing a test channel with the subject and recording it as the corresponding first channel;

[0078] Step 114, taking the first of the first test records in the first data set as the corresponding current test record;

[0079] Step 115, sending the first training data of the current test record to the subject through the first channel; receiving the first prediction data sent back by the subject through the first channel; and forming a corresponding first prediction-label pair by the first prediction data and the first label data of the current test record;

[0080] Step 116, identifying whether the current test record is the last of the first test record of the first data set; if so, closing the first channel and going to step 117; if not, extracting the next first test record of the first data set as the new current test record, and returning to step 115;

[0081] Step 117, identifying the first test type; if the first test type is visual target detection or point cloud target detection, then based on all the first prediction-label pairs obtained, accuracy, recall and prediction deviation analysis is performed to obtain a corresponding first accuracy, first recall and first prediction deviation set to form a corresponding first analysis data set; if the first test type is visual event recognition or point cloud event recognition, then based on all the first prediction-label pairs obtained, accuracy and recall analysis is performed to obtain a corresponding second accuracy and second recall to form a corresponding first analysis data set;

[0082] Wherein, the first prediction deviation set includes at least a first coordinate prediction deviation and a first orientation prediction deviation;

[0083] Step 118, forming a corresponding first prediction-label pair set from all the first prediction-label pairs obtained; and forming a corresponding first test report from the first test request, the first data set, the first prediction-label pair set and the first analysis data set, which is stored in the evaluation / test report storage module.

[0084] An embodiment of the present invention provides a processing system for perception performance testing and sensor deployment scheme evaluation. As can be seen from the above content, the system includes: a high-precision map library, a basic element library, a scene modeling module, a scene template library, a scene simulation module, a scene simulator, a data annotation module, an intelligent model library, a test database, a scene evaluation module, a perception test module and an evaluation / test report storage module; wherein, the high-precision map library is used to store high-precision maps of road traffic; the basic element library is used to store static / dynamic / signal light / environment / sensor / traffic flow element templates; the scene modeling module is used to provide a manual editing interface and perform traffic scene modeling through manual configuration by the user, and is also used to automatically model traffic scenes based on the real scene data set and environment description text input by the user as well as the intelligent model library and the basic element library, and is also used to construct traffic scene templates of any type of specified road conditions by configuring and adjusting the traffic scene templates, and is also used to enhance the scene template library by data enhancement. The template is expanded according to the traffic scene template; the scene template library is used to store multiple traffic scene templates; the scene simulation module is used to distribute the traffic scene template to the scene simulator and / or the scene evaluation module according to the simulation operation instructions input by the user; the scene simulator is used to load the traffic scene template and dynamically simulate it, and during the simulation process, the sensor data is simulated based on the types and properties of various sensors loaded in the scene, and multiple data output interfaces are provided; the data annotation module is used to produce test data according to the scene simulator and the intelligent model library; the intelligent model library at least includes visual / point cloud target detection / event recognition models; the scene evaluation module is used to evaluate the sensor deployment plan of the traffic scene template; the perception test module is used to test the perception performance of the testee (roadside perception algorithm processing interface, module, equipment, service or system) according to the test database. Through the scenario modeling module + scenario simulator + data annotation module of the embodiment of the present invention, not only a large amount of test data can be constructed, but also the scenario complexity of the test data can be improved, that is, the data production efficiency is accelerated and the data richness is enhanced; through the scenario evaluation module of the embodiment of the present invention, the sensor deployment plan can be evaluated without human intervention, thereby improving the efficiency of the plan evaluation; through the perception test module of the embodiment of the present invention, the test adequacy of the subject can be improved based on massive data with sufficient scenario complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A module structure diagram of a processing system for perception performance testing and sensor deployment solution evaluation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0087] The embodiment of the present invention provides a processing system 1 for sensing performance testing and sensor deployment scheme evaluation. Figure 1 As shown in the module structure diagram of a processing system for perception performance testing and sensor deployment scheme evaluation provided by an embodiment of the present invention, the system mainly includes: a high-precision map library 101, a basic element library 102, a scene modeling module 103, a scene template library 104, a scene simulation module 105, a scene simulator 106, a data annotation module 107, an intelligent model library 108, a test database 109, a scene evaluation module 110, a perception test module 111 and an evaluation / test report storage module 112.

[0088] The connection relationship of the components of the system of the embodiment of the present invention is as follows: the scene modeling module 103 is connected to the high-precision map library 101, the basic element library 102, the scene template library 104 and the intelligent model library 108 respectively. The scene simulation module 105 is connected to the scene template library 104, the scene simulator 106 and the scene evaluation module 110 respectively. The scene simulator 106 is connected to the data annotation module 107. The data annotation module 107 is connected to the intelligent model library 108 and the test database 109 respectively. The evaluation / test report storage module 112 is connected to the scene evaluation module 110 and the perception test module 111 respectively. The perception test module 111 is connected to the test database 109. The perception test module 111 is also connected to the subject 2 outside the system. Here, the subject 2 of the embodiment of the present invention is a type of roadside perception algorithm processing interface, module, device, service or system.

[0089] (I) High-precision map library 101:

[0090] The high-precision map library 101 of the embodiment of the present invention is used to store multiple high-precision road traffic maps.

[0091] (II) Basic element library 102:

[0092] The basic element library 102 of the embodiment of the present invention is used to store multiple types of static element templates, multiple types of dynamic element templates, multiple types of signal light element templates, multiple types of environmental element templates, multiple types of sensor element templates and traffic flow element templates.

[0093] Here, the template parameters of the static element template of the embodiment of the present invention include static element types and static element attributes; wherein the static element types include at least road sections, intersections, ramps, roundabouts, road signs, direction signs, construction signs, warning signs, prohibition signs, direction markings, warning markings, prohibition markings, road poles, obstacles, roadside equipment, buildings, and plants; the static element attributes are the attribute sets of the current static elements, including at least ground coordinates, shape and size, orientation, and color.

[0094] The template parameters of the dynamic element template include dynamic element type, dynamic element attributes, element motion mode and element motion trajectory; among them, the dynamic element type includes at least cars, motorcycles, bicycles and pedestrians; the dynamic element attributes are the attribute set of the current dynamic element, including at least shape, size and color; the element motion mode includes multiple types of motion models; the element motion trajectory is composed of multiple motion trajectory points; each motion trajectory point includes at least trajectory point coordinates, trajectory point velocity, trajectory point acceleration and trajectory point heading angle.

[0095] The template parameters of the traffic light element template include traffic light type, traffic light attributes and traffic light change rules; among them, traffic light types include at least motor vehicle traffic lights, non-motor vehicle traffic lights, pedestrian crossing traffic lights, direction indicator lights, lane traffic lights, flashing warning traffic lights, and road and railway level crossing traffic lights; traffic light attributes are the attribute set of the current traffic light, including at least spatial coordinates and shape type; traffic light change rules include the traffic light color at the initial moment, the order of traffic light color rotation and the duration of each traffic light color.

[0096] The template parameters of the environmental element template include the environmental element type and the intensity change timing; wherein, the environmental element type includes at least sunny, rainy, snowy, foggy, natural light source, and street light source; when the environmental element type is sunny, rainy, snowy, and foggy, the intensity change timing includes the visibility change timing and the road surface slipperiness change timing; when the environmental element type is a natural light source or a street light source, the intensity change timing includes the light intensity change timing, the light angle change timing, and the light range change timing.

[0097] The template parameters of the sensor element template include sensor type, sensor attributes and sensor jitter settings; among which, the sensor type includes at least camera and lidar; when the sensor type is camera, the sensor attributes include at least camera intrinsic parameters, camera extrinsic parameters, camera field of view angle, camera sampling rate, camera installation position, camera installation angle, and camera maximum shooting distance; when the sensor type is lidar, the sensor attributes include at least radar intrinsic parameters, radar extrinsic parameters, radar field of view angle, radar sampling rate, radar installation position, radar installation angle, and radar maximum scanning distance; the sensor jitter settings include jitter direction, jitter frequency, and jitter amplitude.

[0098] The template parameters of the traffic flow element template at least include vehicle traffic flow, pedestrian traffic flow, driving speed and congestion index.

[0099] (III) Scene modeling module 103:

[0100] The scene modeling module 103 of the embodiment of the present invention is used to perform traffic scene modeling based on the high-precision map library 101, the basic element library 102 and the manual configuration results of the user to obtain the corresponding traffic scene template and store it in the scene template library 104.

[0101] The scene modeling module 103 of the embodiment of the present invention is also used to automatically model the traffic scene according to the real scene data set and environment description text input by the user, as well as the intelligent model library 108 and the basic element library 102 to obtain the corresponding traffic scene template and store it in the scene template library 104.

[0102] The scene modeling module 103 of the embodiment of the present invention is also used to construct a traffic scene template of any type of specified road condition by configuring and adjusting the traffic scene template and store it in the scene template library 104. Here, the specified road conditions of the embodiment of the present invention at least include normal driving of vehicles on roads / intersections, rear-end collision of vehicles on roads / intersections, turning of vehicles on roads / intersections, scratching of vehicles on roads / intersections, collision of vehicles on roads / intersections, road construction, driving on narrow roads, driving on curved roads, driving on slopes, congestion during peak hours, driving on roads / intersections in bad weather, driving on roads / intersections under strong light conditions, and driving on roads / intersections under weak light conditions.

[0103] The scene modeling module 103 of the embodiment of the present invention is also used to expand the traffic scene templates of the scene template library 104 by data enhancement.

[0104] It should be noted that the scene modeling module 103 of the present invention can also generate a large number of high-simulation scene data sets by calling some large models dedicated to traffic visual scene generation, and configure multiple environment description texts for each high-simulation scene data set to obtain a set of high-simulation scene data sets + environment description texts similar to the real scene data set + environment description text; then the automatic modeling of traffic scenes based on the real scene data set + environment description text + intelligent model library 108 + basic element library 102 is realized: automatic modeling of traffic scenes based on high-simulation scene data sets + environment description text + intelligent model library 108 + basic element library 102. The large models dedicated to traffic visual scene generation mentioned here are commonly traffic visual scene generation models based on diffusion large models.

[0105] In a specific implementation of the embodiment of the present invention, the scene modeling module 103 is specifically used to perform traffic scene modeling according to the high-precision map library 101, the basic element library 102 and the manual configuration results of the user to obtain the corresponding traffic scene template and store it in the scene template library 104:

[0106] Step A1, providing a visual map editing interface for the user; and providing the user with a map retrieval option and a map editing window on the visual map editing interface; and after the user retrieves and selects a road traffic high-precision map from the high-precision map library 101 through the map retrieval option, making a map copy for the current road traffic high-precision map and loading it into the map editing window; and receiving the user's element modification and / or map rotation and / or map cropping operation results on the current map copy through the map editing window, and saving the current map copy as a corresponding first template high-precision map when the user confirms to save through the map editing window;

[0107] Step A2, when the first template high-precision map is successfully saved, a visual element editing interface is provided for the user; a map loading window and an element toolbar are provided for the user on the visual element editing interface; and the first template high-precision map is loaded with a map through the map loading window; a corresponding tool icon is assigned to each element template of the basic element library 102, and all tool icons are loaded through the element toolbar; and when the user selects a tool icon, an icon dragging function is provided for the user to drag the current tool icon to the map loading window, and an element template copy is generated for the element template corresponding to the current tool icon, and a unique element identifier is generated for the current element template copy; and when the user selects a tool icon on the map loading window, a visual element configuration window is provided for the element template copy corresponding to the current tool icon The interface is used to receive and save the modification results of any template parameter other than the element type made by the user to the copy of the current element template; and when any tool icon is newly added or displaced on the map loading window, the position attribute of the copy of the element template corresponding to the current tool icon is modified based on the map coordinates of the current tool icon on the map loading window; and when the user confirms saving through the visual element editing interface, the copy of the element template corresponding to each tool icon loaded on the map loading window is saved as a corresponding first template element, and all the obtained first template elements constitute a corresponding first template element set; and the obtained first template high-precision map and the first template element set constitute a corresponding traffic scene template; and a unique template identifier is set for the traffic scene template, and the traffic scene template with the template identifier information is stored in the scene template library 104.

[0108] It should be noted here that there is and can only be loaded one tool icon corresponding to the traffic flow element template on the map loading window.

[0109] In another specific implementation of the embodiment of the present invention, the scene modeling module 103 is specifically used to automatically model the traffic scene according to the real scene data set and the environment description text input by the user, as well as the intelligent model library 108 and the basic element library 102 to obtain the corresponding traffic scene template and store it in the scene template library 104:

[0110] Step B1, receiving a real scene data set and an environment description text input by a user as a corresponding first scene data set and a first environment description text;

[0111] Here, the real scene data set of the embodiment of the present invention includes a first high-precision map, a first overhead view sequence, and one or more first horizontal view sequences; the real scene data set corresponds to a real traffic scene; the first high-precision map is a high-precision map of road traffic corresponding to the current real traffic scene; the first overhead view sequence and each first horizontal view sequence each correspond to a set of camera internal and external parameters; the first overhead view sequence is formed by sorting a plurality of first overhead views in chronological order, and each first overhead view corresponds to a camera position coordinate and an image timestamp; each first horizontal view sequence is formed by sorting a plurality of first horizontal views in chronological order, and each first horizontal view corresponds to a camera position coordinate and an image timestamp; when the number of first horizontal view sequences is greater than 1, the shooting angles of every two first horizontal view sequences are different;

[0112] It should be noted that the top view and the horizontal view in the embodiment of the present invention are both from the roadside sensor by default; the shooting angle of the top view is not strictly defined as 90° perpendicular to the ground, but it can be regarded as a top view after the angle between the axis of the camera and the ground exceeds a set angle, such as the image of the road below taken by the camera installed on the lamp pole, the image of the road below taken by the camera installed on the buildings on both sides of the road, etc.; the shooting angle of the horizontal view is not strictly defined as parallel to the ground, but it can be regarded as a horizontal view after the angle with the ground is less than a set angle, such as the surrounding environment image taken by the low-position camera installed at the low-position observation position on the roadside, the surrounding environment image taken by the camera installed on the vehicle end, the environment image taken by the pedestrian's mobile phone camera, etc.;

[0113] Step B2, extracting the corresponding first high-precision map, the first overhead view sequence and all the first horizontal view sequences from the first scene data set; and using the first high-precision map as a corresponding first template high-precision map; and performing weather type and light source type keyword recognition on the first environment description text to obtain corresponding weather type keywords and light source type keywords;

[0114] Among them, when the weather type keyword is not empty, it at least includes sunny, rainy, snowy, and foggy; when the light source type keyword is not empty, it at least includes natural light source and street light source;

[0115] Step B3, performing target detection processing on each first top view and each first plan view based on the visual target detection model of the intelligent model library 108 to obtain a corresponding plurality of first target objects to form a corresponding first target object group; and sorting all first target object groups corresponding to the first top view in chronological order to form a corresponding first target object group sequence; and sorting all first target object groups corresponding to each plan view in chronological order to form a corresponding first target object group sequence;

[0116] Here, each first target object corresponds to a set of target object attributes, including target type, target coordinates, target size, and target orientation;

[0117] Step B4, based on the camera internal and external parameters and camera coordinates of each first target object, the target coordinates, target size and target orientation of the current target object are converted from the image coordinate system to the world coordinate system; and the image timestamps corresponding to each first target object group are added as corresponding target timestamps to the target object attributes of each target object in the current first target object group;

[0118] Here, the updated target object attributes of the first target object include target type, target coordinates, target size, target orientation, and target timestamp;

[0119] Step B5, and use each first target object group sequence as the corresponding current target object group sequence; perform target association on each two adjacent first target object groups in the current target object group sequence based on a preset target association algorithm, and use the target object attributes of each first target object corresponding to the same associated target as the trajectory point attributes of a corresponding first trajectory point based on the target association result, and sort all the first trajectory points corresponding to the same associated target in chronological order to form a corresponding first target trajectory;

[0120] Here, the target association algorithm at least includes the Hungarian algorithm; the first target trajectory is formed by sequentially sorting a plurality of first trajectory points, and the trajectory point attributes of each first trajectory point include a target type, a target coordinate, a target size, a target orientation, and a target timestamp; the target types corresponding to all the first trajectory points on the first target trajectory are the same and will not change;

[0121] Step B6, using the target coordinates and the target orientation as observation quantities, and the target speed, the target acceleration and the target heading angle as state quantities, estimating the target speed, the target acceleration and the target heading angle corresponding to each first trajectory point on each first target trajectory according to a preset filter algorithm, and adding the estimation results to the trajectory point attributes corresponding to each first trajectory point;

[0122] Here, the updated trajectory point attributes of the first trajectory point include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the filter algorithm includes at least a Kalman filter algorithm;

[0123] Step B7, statistically analyzing the sequence time lengths of the first overhead view sequence and each first horizontal view sequence to obtain the corresponding first sequence time length; and taking the longest first sequence time length as the corresponding maximum time length; and dividing the maximum time length into multiple time scales based on a preset minimum unit time length; and aligning the target timestamps of the first trajectory points of each first target trajectory on each time scale with the current time scale; and estimating the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of the first trajectory points of each first target trajectory on each time scale according to a filter algorithm or an interpolation algorithm; and forming the corresponding second target trajectory from the first trajectory points of each first target trajectory on all time scales;

[0124] Here, the minimum unit duration is a small unit time length;

[0125] Step B8, and calculate the trajectory similarity of every two second target trajectories to obtain the corresponding first trajectory similarity; and cluster all the second target trajectories based on the preset trajectory similarity threshold to obtain one or more corresponding second target trajectory sets; and perform trajectory point attribute fusion on all the first trajectory points of all the second target trajectories of each second target trajectory set at each time scale to obtain a corresponding second trajectory point, and sequentially sort all the obtained second trajectory points to form a corresponding first target fused trajectory;

[0126] Here, the trajectory similarity threshold is a preset threshold parameter; the second target trajectory set is composed of one or more second target trajectories; when the number of second target trajectories in the second target trajectory set is greater than 1, the first trajectory similarity between any two second target trajectories in the current set exceeds the trajectory similarity threshold; the first target fused trajectory is formed by sequentially sorting a plurality of second trajectory points, and the trajectory point attributes of each second trajectory point include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the target type and target timestamp of each second trajectory point are consistent with the target type and target timestamp of all corresponding first trajectory points; the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of each second trajectory point are fused from the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of all corresponding first trajectory points; the fusion algorithm here can be customized based on application requirements, and under normal circumstances, fusion can be achieved by calculating the mean;

[0127] Step B9, based on the target type corresponding to each first target fusion trajectory, select the corresponding static, dynamic, signal light or sensor element template from the basic element library 102 as a corresponding first template element; and assign a corresponding unique element identifier to the current first template element; and configure the template parameters of the current first template element based on all trajectory point attributes of the current first target fusion trajectory;

[0128] Step B10, when the weather type keyword is not empty, the environment element template whose environment element type matches the weather type keyword in the basic element library 102 is used as a corresponding first template element; and a corresponding unique element identifier is assigned to the current first template element; and the intensity transformation timing of the current first template element is configured based on a set of preset visibility transformation timings and preset road slipperiness transformation timings corresponding to the weather type keyword;

[0129] Here, the preset visibility change sequence is a preset visibility sequence, which can be adaptively customized to various weather conditions based on application requirements; the preset road slipperiness change sequence is a preset road slipperiness sequence, which can be adaptively customized to various weather conditions based on application requirements;

[0130] Step B11, when the light source type keyword is not empty, the environment element template whose environment element type matches the light source type keyword in the basic element library 102 is used as a corresponding first template element; and a corresponding unique element identifier is assigned to the current first template element; and the intensity change timing of the current first template element is configured based on a set of preset illumination intensity change timings, preset illumination angle change timings, and preset illumination range change timings corresponding to the light source type keyword;

[0131] Here, the preset illumination intensity change sequence, the preset illumination angle change sequence, and the preset illumination range change sequence are respectively a preset illumination intensity sequence, an incident light angle sequence, and a ground range sequence, which can be adaptively customized to various regions / altitudes, various seasons, various time periods, various light source types, and / or various weather conditions based on application requirements;

[0132] Step B12, and all the first template elements obtained this time form a corresponding first template element set; and the first template high-precision map and the first template element set form a corresponding traffic scene template; and set a unique template identifier for the traffic scene template, and store the traffic scene template with the template identifier information into the scene template library 104.

[0133] In another specific implementation of an embodiment of the present invention, the scene modeling module 103 is specifically used to, when expanding the traffic scene template of the scene template library 104 by data enhancement, use the existing traffic scene templates in the scene template library 104 as the corresponding current base template; and record the first template element corresponding to each type of dynamic element in the current base template as the corresponding first dynamic element; and generate multiple derived scene templates by dynamically adjusting the element motion mode and element motion trajectory of one or more first dynamic elements in the current base template; and set a unique template identifier for each derived scene template; and form a corresponding derived scene template set by all derived scene templates corresponding to the current base template; and add all the obtained derived scene template sets to the scene template library 104.

[0134] It should be noted that, as can be seen from the above, the various freely customizable basic elements provided by the basic element library 102 of the embodiment of the present invention and the three scene template construction methods provided by the scene modeling module 103 (manual configuration method based on human-computer interaction, automatic generation method based on intelligent model, and derivative customization method based on configuration adjustment of existing templates) are bound to be easily created. A large number of traffic scene templates can be created, and good coverage of various unconventional specific traffic scene templates can be achieved, especially for various extreme conditions or critical situations (corner case) Specific traffic scene templates can also be easily customized. As can be seen from the following, the automatic production process of test data in the embodiment of the present invention is realized by the scene simulation module 105, the scene simulator 106 and the data annotation module 107. Then, after a large number of rich traffic scene templates are constructed through the basic element library 102+scene modeling module 103, based on the common scene simulation module 105+scene simulator 106+data annotation module 107, it is naturally possible to output test data sets under various traffic scenes, especially a large number of test data sets for various extreme conditions or critical situations can be generated. In this way, by testing the subject 2 based on the data set output by the embodiment of the present invention, the subject 2's ability to solve long-tail problems can naturally be fully evaluated. The so-called long-tail problems here are actually the potential risks brought by special scenarios with low probability of occurrence but a wide variety of types, and there are many specific traffic scene templates corresponding to long-tail problems, such as dynamic elements of abnormal traffic behavior in the scene (such as vehicles traveling in the wrong direction), environmental elements of extreme intensity (such as blizzards, rainstorms, sandstorms), complex road conditions composed of complex static road elements (such as construction areas, congested roads, narrow roads), etc., and these scenes can be easily constructed through the embodiments of the present invention.

[0135] (IV) Scene template library 104:

[0136] The scene template library 104 of the embodiment of the present invention is used to store multiple traffic scene templates.

[0137] Here, each traffic scene template of the embodiment of the present invention corresponds to a unique template identifier; the traffic scene template includes a first template high-precision map and a first template element set; the first template element set is composed of multiple first template elements, and each first template element corresponds to a unique element identifier.

[0138] (V) Scene simulation module 105:

[0139] The scenario simulation module 105 of the embodiment of the present invention is used to perform traffic scenario template distribution processing on the scenario simulator 106 and / or the scenario evaluation module 110 according to the simulation operation instruction input by the user.

[0140] In another specific implementation of the embodiment of the present invention, the scene simulation module 105 is specifically used to distribute the traffic scene template to the scene simulator 106 and / or the scene evaluation module 110 according to the simulation operation instruction input by the user:

[0141] Step C1, receiving a simulation operation instruction input by a user; and extracting a corresponding instruction type and instruction parameters from the simulation operation instruction;

[0142] Wherein, the simulation operation instruction includes an instruction type and an instruction parameter; the instruction type includes a first type, a second type and a third type; when the instruction type is the first or second type, the instruction parameter is a template identifier; when the instruction type is the third type, the instruction parameter is a template identifier sequence;

[0143] When the instruction type is the first type, it indicates that a single template is loaded and simulated, and test data is produced simultaneously; when it is the second type, it indicates that a single template is loaded and simulated, and test data is produced simultaneously, and the sensor deployment plan on the currently loaded template is evaluated simultaneously; when it is the third type, it indicates that batch templates are loaded and simulated, and test data is produced simultaneously;

[0144] Step C2, identifying the instruction type;

[0145] Step C3, if the instruction type is the first or second type, the template identifier in the instruction parameter is used as the corresponding current template identifier; and the traffic scene template corresponding to the current template identifier in the scene template library 104 is used as the corresponding current traffic scene template; and when the instruction type is the first type, the current traffic scene template is sent to the scene simulator 106; and when the instruction type is the second type, the current traffic scene template is sent to the scene simulator 106 and the scene evaluation module 110 respectively;

[0146] Step C4, if the instruction type is the third type, the template identification sequence in the instruction parameter is used as the corresponding current template identification sequence; and the template identifications of the current template identification sequence are sequentially traversed; and in the traversal process, the currently traversed template identification is used as the corresponding current template identification; and the traffic scene template corresponding to the current template identification in the scene template library 104 is used as the corresponding current traffic scene template; and the simulation duration of the current traffic scene template is estimated to obtain the corresponding current simulation duration; and based on the preset first delay factor and the current simulation duration, the corresponding first waiting duration is calculated = current simulation duration × (1 + first delay factor), 0<first delay factor<1; and the current traffic scene template is sent to the scene simulator 106; and after waiting for the first waiting duration from the completion of this scene sending, it goes to the next template identification for traversal until the traversal of the last template identification is completed.

[0147] (VI) Scenario simulator 106:

[0148] The scene simulator 106 of the embodiment of the present invention is used to load the current traffic scene template sent by the scene simulation module 105 when the corresponding simulator state is an idle state, and perform dynamic simulation based on the currently loaded scene; and clear the historical data in the built-in data cache at the start of the simulation; and set the simulator state to the working state during the simulation process, and set the simulator state to the idle state at the end of the simulation process; and perform sensor data simulation based on the types and properties of various sensors loaded in the scene during the simulation process and store the simulation data in the built-in data cache; and provide a simulator state query interface for outputting the current simulator state; and provide a sensor query interface for outputting the sensor list in the current simulation scene; and provide a simulation data query interface for outputting any sensor simulation data stored in the built-in data cache.

[0149] Here, the scenario simulator 106 of the embodiment of the present invention is a type of tool software used for simulating road traffic scenarios and capable of simulating and outputting sensor data in the scenarios. Common tool software of this type include: SUMO, ErgoSIM, RoadRunner, VISSIM, and the like.

[0150] The simulator status of the embodiment of the present invention includes an idle state and a working state. The query parameters of the simulator status query interface are empty; the interface return data of the simulator status query interface is the corresponding simulator status. The query parameters of the sensor query interface are empty; the interface return data of the sensor query interface is the corresponding sensor list; the sensor list includes multiple sensor records; each sensor record includes a sensor identifier and a sensor type; the sensor type includes at least a camera and a lidar. The query parameters of the simulation data query interface include a sensor identification parameter, a sensor type parameter, and a data time parameter; the sensor type parameter includes at least a camera and a lidar; the data time parameter is a time period; the return data of the simulation data query interface is the image sequence and point cloud sequence corresponding to the camera or lidar specified by the current query parameters.

[0151] (VII) Data annotation module 107:

[0152] The data annotation module 107 of the embodiment of the present invention is used to generate test data according to the scenario simulator 106 and the intelligent model library 108 and add the obtained test data to the test database 109 .

[0153] In another specific implementation of the embodiment of the present invention, the data annotation module 107 is specifically used to generate test data according to the scenario simulator 106 and the intelligent model library 108 and add the obtained test data to the test database 109:

[0154] Step D1, pre-initializing a first waiting flag to end the waiting;

[0155] Here, the first waiting flag includes starting waiting and ending waiting;

[0156] Step D2, and regularly obtaining the corresponding simulator status through the simulator status query interface of the scene simulator 106 according to the preset first query frequency;

[0157] Here, the first query frequency is a preset time frequency parameter;

[0158] Step D3, and identify the simulator state; if the simulator state is in the working state, set the first waiting flag to start waiting; if the simulator state is in the idle state, then when the first waiting flag is the start waiting, the sensor data of each camera and each laser radar in the current simulation scene of the scene simulator 106 are acquired through the sensor query interface and the simulation data query interface of the scene simulator 106 to obtain the corresponding multiple first image sequences and multiple first point cloud sequences, and after all the sensor data are obtained, set the first waiting flag to end waiting;

[0159] Here, each first image sequence corresponds to a camera, and each first point cloud sequence corresponds to a laser radar; the first image sequence is formed by sorting a plurality of first images in chronological order; the first point cloud sequence is formed by sorting a plurality of first point clouds in chronological order;

[0160] Step D4, merging all first image sequences obtained this time to obtain a corresponding first image set; and merging all first point cloud sequences obtained this time to obtain a corresponding first point cloud set;

[0161] Step D5, and perform a round of traversal on all the first images of the first image set; and in this round of traversal, take the currently traversed first image as the corresponding current image; and input the current image into the visual target detection model for target detection processing and use the detection target set output by the model as a corresponding first label target set; and input the current image into the visual event recognition model for traffic event recognition processing and form a corresponding first label event set with all event types output by the model; and use the current image as a corresponding first training image and a corresponding second training image; and form a corresponding first data record with the first training image corresponding to the current image and the first label target set; and form a corresponding third data record with the second training image corresponding to the current image and the first label event set; and at the end of this round of traversal, form a corresponding first data record set with all the first data records obtained in this round of traversal, and form a corresponding third data record set with all the third data records obtained in this round of traversal;

[0162] Step D6, and perform a round of traversal on all the first point clouds of the first point cloud set; and in this round of traversal, use the currently traversed first point cloud as the corresponding current point cloud; and input the current point cloud into the point cloud target detection model for target detection processing and use the detection target set output by the model as a corresponding second label target set; and input the current point cloud into the point cloud event recognition model for traffic event recognition processing and form all event types output by the model into a corresponding second label event set; and use the current point cloud as a corresponding first training point cloud and a corresponding second training point cloud; and form a corresponding second data record by the first training point cloud corresponding to the current point cloud and the second label target set; and form a corresponding fourth data record by the second training point cloud corresponding to the current point cloud and the second label event set; and at the end of this round of traversal, form a corresponding second data record set by all the second data records obtained in this round of traversal, and form a corresponding fourth data record set by all the fourth data records obtained in this round of traversal;

[0163] Step D7, and add the first data record set obtained this time to the visual target detection data set of the test database 109; and add the second data record set obtained this time to the point cloud target detection data set of the test database 109; and add the third data record set obtained this time to the visual event recognition data set of the test database 109; and add the fourth data record set obtained this time to the point cloud event recognition data set of the test database 109.

[0164] (VIII) Intelligent Model Library 108:

[0165] The intelligent model library 108 of the embodiment of the present invention at least includes a visual target detection model, a point cloud target detection model, a visual event recognition model, and a point cloud event recognition model.

[0166] Here, all types of models in the intelligent model library 108 of the embodiment of the present invention have completed model training based on a large data set in advance. The visual target detection model is used to perform target detection processing based on the model input image and output the corresponding detection target set. The point cloud target detection model is used to perform target detection processing based on the model input point cloud and output the corresponding detection target set. The visual event recognition model is used to perform traffic event recognition processing based on the model input image and output one or more corresponding event types. The point cloud event recognition model is used to perform traffic event recognition processing based on the model input point cloud and output one or more corresponding event types.

[0167] It should be noted that the intelligent models in the intelligent model library 108 of the embodiment of the present invention should correspond to the data sets in the test database 109; the purpose of various intelligent models is actually to perform corresponding label data annotation on the training data required for various data sets; and the processing efficiency of data annotation can be improved by using intelligent models to process label annotation.

[0168] It should also be noted that, in addition to the above four basic models, the intelligent models in the intelligent model library 108 of the embodiment of the present invention can also add more other models, and correspondingly other types of data sets can be constructed in the test database 109.

[0169] (IX) Test database 109:

[0170] The test database 109 of the embodiment of the present invention at least includes a visual target detection dataset, a point cloud target detection dataset, a visual event recognition dataset, and a point cloud event recognition dataset.

[0171] Here, the visual object detection dataset of the embodiment of the present invention includes a plurality of first data records; the first data record includes a first training image and a first label target set. The point cloud object detection dataset includes a plurality of second data records; the second data record includes a first training point cloud and a second label target set. The visual event recognition dataset includes a plurality of third data records; the third data record includes a second training image and a first label event set. The point cloud event recognition dataset includes a plurality of fourth data records; the fourth data record includes a second training point cloud and a second label event set.

[0172] (X) Scenario Assessment Module 110:

[0173] The scenario assessment module 110 of the embodiment of the present invention is used to assess the sensor deployment scheme of the current traffic scenario template and generate a corresponding first assessment report to store in the assessment / test report storage module 112 .

[0174] In another specific implementation of the embodiment of the present invention, the scenario evaluation module 110 is specifically used to evaluate the sensor deployment scheme of the current traffic scenario template, generate a corresponding first evaluation report and store it in the evaluation / test report storage module 112:

[0175] Step E1, extracting a corresponding first template high-precision map and a first template element set from the current traffic scene template as the corresponding current high-precision map and current element set; and taking each sensor element in the current element set as a corresponding first sensor; and counting the total number of the first sensors to obtain a corresponding first total number N;

[0176] Step E2, and perform a round of traversal for all first sensors; and in this round of traversal, use the first sensor currently traversed as the corresponding current sensor; and identify the sensor type corresponding to the current sensor; if the sensor type corresponding to the current sensor is a camera, then based on the current high-precision map and the camera field of view, camera installation position, camera installation angle and camera maximum shooting distance corresponding to the current sensor, the road coverage range of the current sensor is identified to obtain the corresponding first coverage range; if the sensor type corresponding to the current sensor is a lidar, then based on the current high-precision map and the radar field of view, radar installation position, radar installation angle and radar maximum scanning distance corresponding to the current sensor, the road coverage range of the current sensor is identified to obtain the corresponding first coverage range;

[0177] Step E3, after the current round of traversal of all first sensors is completed, all first coverage ranges obtained are range fused to obtain a corresponding first fusion range; all road / intersection ranges on the current high-precision map are fused to obtain a corresponding second fusion range; the intersection range of the first and second fusion ranges is identified to obtain a corresponding first intersection range; the corresponding first coverage rate is calculated based on the first intersection range and the second fusion range = (first intersection range / second fusion range) × 100%; and whether the first coverage rate is lower than a preset first coverage rate threshold is identified, if so, the corresponding coverage rate evaluation state is set to be unsatisfactory, otherwise, the corresponding coverage rate evaluation state is set to be up to standard;

[0178] Here, the first coverage rate threshold is a preset threshold parameter; the coverage rate evaluation status includes reaching the standard and failing to reach the standard;

[0179] Step E4, identifying the road / intersection type corresponding to the current high-precision map to obtain a corresponding first road / intersection type; and selecting a road / intersection sensor overlapping deployment requirement corresponding to the first road / intersection type from the preset multiple road / intersection sensor overlapping deployment requirements as the corresponding first sensor overlapping deployment requirement;

[0180] Here, the first road / intersection type includes at least a straight / curved / ramp section of a highway, a straight / curved / ramp section of a national highway, a straight / curved / ramp section of an urban road, a straight / curved / ramp section of a rural road, a T-intersection, a cross intersection, an X-intersection, a compound intersection, and a roundabout;

[0181] It should be noted that the multiple road / intersection sensor overlapping deployment requirements are a set of multiple pre-set sensor overlapping deployment requirements that are suitable for different road / intersection types (highway straight / curved / ramp sections, national highway straight / curved / ramp sections, urban road straight / curved / ramp sections, rural road straight / curved / ramp sections, T-intersections, cross intersections, X-intersections, compound intersections, and roundabouts); each type of sensor overlapping deployment requirement can be customized based on application requirements;

[0182] Step E5, constructing an N×N two-dimensional matrix based on the first total number N and all first coverages and recording it as the corresponding first intersection-and-union ratio matrix;

[0183] Here, the 1st to Nth rows of the first intersection-and-union ratio matrix correspond one-to-one to the 1st to Nth first sensors, and the 1st to Nth columns correspond one-to-one to the 1st to Nth first sensors; the cell value of each matrix cell of the first intersection-and-union ratio matrix is ​​the intersection-and-union ratio of the two first coverage ranges corresponding to the two first sensors corresponding to the current matrix cell;

[0184] Here, the diagonal from the upper left vertex to the lower right vertex of the first intersection-and-union matrix is ​​all 1 and can be ignored; the first intersection-and-union matrix can be used to count the number of isolated sensors that do not overlap with any other sensor in coverage, that is, sensors whose corresponding matrix rows / columns are all 0 except for the matrix cells intersecting the diagonal; the first intersection-and-union matrix can be used to count the number of key sensors that overlap with other sensors the most times in coverage, that is, sensors whose corresponding matrix rows / columns have the largest number of matrix cells greater than 0 except for the matrix cells intersecting the diagonal; the first intersection-and-union matrix can be used to count highly overlapping sensor pairs, that is, two sensors corresponding to matrix cells whose matrix cell values ​​exceed a preset large intersection-and-union threshold value except for the matrix cells on the diagonal; the first intersection-and-union matrix can be used to count the overlapping distribution status of all sensors;

[0185] Step E6, identifying whether the first intersection-over-union ratio matrix meets the first sensor overlapping deployment requirement, if so, setting the corresponding overlapping deployment evaluation state to meet the requirement, if not, setting the corresponding overlapping deployment evaluation state to fail to meet the requirement;

[0186] Here, the overlapping deployment assessment status includes meeting the standard and not meeting the standard;

[0187] Step E7, and use the template identifier corresponding to the current traffic scene template as the corresponding first evaluation template identifier; and each first sensor and the corresponding first coverage range constitute the corresponding first sensor deployment feature, and all the obtained first sensor deployment features constitute the corresponding first sensor deployment feature set; and the first template high-precision map, the first template element set, the first total number N, the first sensor deployment feature set, the first road / intersection type, the first sensor overlapping deployment requirement, the first coverage rate, the first intersection-to-union ratio matrix, the coverage rate evaluation status and the overlapping deployment evaluation status constitute the corresponding first evaluation report which is stored in the evaluation / test report storage module 112.

[0188] (XI) Perception test module 111:

[0189] The perception test module 111 of the embodiment of the present invention is used to test the perception performance of the subject 2 according to the test database 109 to obtain a corresponding first test report and store it in the evaluation / test report storage module 112 .

[0190] In another specific implementation of the embodiment of the present invention, the perception test module 111 is specifically used for testing the perception performance of the subject 2 according to the test database 109 to obtain a corresponding first test report which is stored in the evaluation / test report storage module 112:

[0191] Step F1, receiving a first test request sent by the testee 2; and extracting a corresponding first test request identifier, a first test type and a first test total amount from the first test request;

[0192] The first test request includes a first test request identifier, a first test type, and a first test total amount; the first test type includes visual target detection, point cloud target detection, visual event recognition, and point cloud event recognition;

[0193] Step F2, identifying the first test type; if the first test type is visual target detection, then based on the preset test data selection rule, the first data record of the first test total amount is selected from the visual target detection data set of the test database 109 to form a corresponding first data set; if the first test type is point cloud target detection, then based on the test data selection rule, the second data record of the first test total amount is selected from the point cloud target detection data set of the test database 109 to form a corresponding first data set; if the first test type is visual event recognition, then based on the test data selection rule, the third data record of the first test total amount is selected from the visual event recognition data set of the test database 109 to form a corresponding first data set; if the first test type is point cloud event recognition, then based on the test data selection rule, the fourth data record of the first test total amount is selected from the visual event recognition data set of the test database 109 to form a corresponding first data set;

[0194] Here, the test data selection rule is a pre-set data selection rule, which can be customized based on application requirements; for example, random sampling is performed in segments according to the data record upload time;

[0195] The first data set output by the current step includes multiple first test records; wherein the first test record includes first training data and first label data; when the first test type is visual target detection, the first training data and the first label data are the corresponding first training image and the first label target set; when the first test type is point cloud target detection, the first training data and the first label data are the corresponding first training point cloud and the second label target set; when the first test type is visual event recognition, the first training data and the first label data are the corresponding second training image and the first label event set; when the first test type is point cloud event recognition, the first training data and the first label data are the corresponding second training point cloud and the second label event set;

[0196] Step F3, establishing a test channel with the subject 2 and recording it as the corresponding first channel;

[0197] Here, the specific construction method of the first channel can be customized based on application requirements; for example, it can be constructed by using a network dedicated line, a wireless channel, an Internet protocol channel, etc.

[0198] Step F4, taking the first first test record of the first data set as the corresponding current test record;

[0199] Step F5, sending the first training data of the current test record to the subject 2 through the first channel; receiving the first prediction data sent back by the subject 2 through the first channel; and forming a corresponding first prediction-label pair by the first prediction data and the first label data of the current test record;

[0200] Step F6, identifying whether the current test record is the last first test record of the first data set; if so, closing the first channel and going to step F7; if not, extracting the next first test record of the first data set as the new current test record and returning to step F5;

[0201] Step F7, identifying the first test type; if the first test type is visual target detection or point cloud target detection, then based on all the obtained first prediction-label pairs, the accuracy, recall rate and prediction deviation analysis are performed to obtain the corresponding first accuracy, first recall rate and first prediction deviation set to form the corresponding first analysis data set; if the first test type is visual event recognition or point cloud event recognition, then based on all the obtained first prediction-label pairs, the accuracy and recall analysis are performed to obtain the corresponding second accuracy and second recall rate to form the corresponding first analysis data set;

[0202] The first prediction deviation set includes at least a first coordinate prediction deviation and a first orientation prediction deviation; the smaller the prediction deviation of the first coordinate prediction deviation or the first orientation prediction deviation, the higher the corresponding prediction accuracy; conversely, if the prediction deviation is larger, the corresponding prediction accuracy is lower;

[0203] In step F8, all the first prediction-label pairs obtained are used to form a corresponding first prediction-label pair set; and the first test request, the first data set, the first prediction-label pair set and the first analysis data set are used to form a corresponding first test report which is stored in the evaluation / test report storage module 112.

[0204] In another specific implementation of the embodiment of the present invention, the perception test module 111 is specifically used to perform accuracy, recall and prediction deviation analysis based on all the first prediction-label pairs obtained to obtain the corresponding first accuracy, first recall and first prediction deviation set to form a corresponding first analysis data set:

[0205] Step G1, perform a round of traversal on all first prediction-label pairs; and in this round of traversal, take the currently traversed first prediction-label pair as the corresponding current prediction-label pair; and take the first prediction data and the first label data of the current prediction-label pair as the corresponding current prediction data and the current label data; and extract the target type, target coordinates and target orientation of each detection target in the current label data to form a corresponding first label target; and extract the target type, target coordinates and target orientation of each detection target in the current prediction data to form a corresponding first prediction target; and count the total number of first label targets to obtain the corresponding total number of first label targets, and count the total number of first prediction targets to obtain the corresponding total number of first prediction targets; and identify whether the total number of first label targets and the total number of first prediction targets are equal; if the total number of first label targets and the total number of first prediction targets are not equal, set the corresponding first test result to failure; if the total number of first label targets and the total number of first prediction targets are equal, take the first prediction target closest to each first label target as the corresponding first matching target, and identify whether the target types of all first label targets and their corresponding first prediction targets are the same; if at least one first If the target type of the label target and the corresponding first predicted target are different, the corresponding first test result is set to failure; if the target type of all first label targets and their corresponding first predicted targets are the same, the straight-line distance between the two target coordinates of each first label target and its corresponding first matching target is calculated to obtain the corresponding first deviation distance, and whether all first deviation distances are less than the preset first deviation distance threshold is identified; if at least one first deviation distance is greater than or equal to the first deviation distance threshold, the corresponding first test result is set to failure; if all first deviation distances are less than the first deviation distance threshold, the angle difference between the two target orientations of each first label target and its corresponding first matching target is calculated to obtain the corresponding first deviation angle, and whether all first deviation angles are less than the preset first deviation angle threshold is identified; if at least one first deviation angle is greater than or equal to the first deviation angle threshold, the corresponding first test result is set to failure; if all first deviation angles are less than the first deviation angle threshold, the corresponding first test result is set to success; and after the end of this round of traversal, the accuracy and recall rates are calculated based on all the first test results obtained to obtain the corresponding first accuracy and first recall rates;

[0206] Here, the first deviation distance threshold is a preset distance threshold parameter, and the first deviation angle threshold is a preset angle threshold parameter;

[0207] Step G2: the first deviation distances corresponding to all the first test results that are specifically successful form a corresponding first deviation distance set, and the first deviation angles corresponding to all the first test results that are specifically successful form a corresponding first deviation angle set; and based on the first deviation distance set, a standard deviation is calculated to obtain a corresponding first coordinate prediction deviation; and based on the first deviation angle set, a standard deviation is calculated to obtain a corresponding first orientation prediction deviation; and the obtained first coordinate prediction deviation and first orientation prediction deviation form a corresponding first prediction deviation set;

[0208] Step G3, the obtained first accuracy, first recall and first prediction deviation set form a corresponding first analysis data set.

[0209] In another specific implementation of the embodiment of the present invention, the perception test module 111 is specifically used to perform accuracy and recall analysis based on all the obtained first prediction-label pairs to obtain the corresponding second accuracy and second recall to form a corresponding first analysis data set:

[0210] Step H1, perform a round of traversal on all first prediction-label pairs; and in this round of traversal, use the currently traversed first prediction-label pair as the corresponding current prediction-label pair; and use the first prediction data and the first label data of the current prediction-label pair as the corresponding current prediction data and the current label data; and form a corresponding current label event set from all event types of the current label data; and form a corresponding current prediction event set from all event types of the current prediction data; and identify whether the current label event set and the current prediction event set are completely matched, if so, set the corresponding second test result to success, otherwise set the corresponding second test result to failure; and after the end of this round of traversal, calculate the accuracy and recall rate based on all the obtained second test results to obtain the corresponding second accuracy and second recall rate;

[0211] Step H2, the obtained second precision and second recall rate form a corresponding first analysis data set.

[0212] It should be noted that the division of the various system components / modules of the above system can be regarded as a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these system components / modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the scene modeling module can be a separately established processing element, or it can be integrated in a chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called by a processing element of the above system to execute the corresponding module function. The implementation of other system components / modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, the processing steps of each module of the aforementioned system can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0213] For example, the system components / modules of the above system may be one or more integrated circuits configured to implement the functions of the corresponding components / modules, such as: one or more specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a system component / module of the above system is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0214] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above system are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state drive (SSD)), etc.

[0215] In summary, an embodiment of the present invention provides a processing system for perception performance testing and sensor deployment scheme evaluation; from the above content, it can be seen that the system includes: a high-precision map library, a basic element library, a scene modeling module, a scene template library, a scene simulation module, a scene simulator, a data annotation module, an intelligent model library, a test database, a scene evaluation module, a perception test module and an evaluation / test report storage module; wherein the high-precision map library is used to store high-precision maps of road traffic; the basic element library is used to store static / dynamic / signal light / environment / sensor / traffic flow element templates; the scene modeling module is used to provide a manual editing interface and perform traffic scene modeling through manual configuration by the user, and is also used to automatically model traffic scenes according to the real scene data set and environment description text input by the user as well as the intelligent model library and the basic element library, and is also used to construct traffic scenes of any type of specified road conditions by configuring and adjusting the traffic scene templates. The template is also used to expand the traffic scene template of the scene template library by data enhancement; the scene template library is used to store multiple traffic scene templates; the scene simulation module is used to distribute the traffic scene template to the scene simulator and / or the scene evaluation module according to the simulation operation instructions input by the user; the scene simulator is used to load the traffic scene template and dynamically simulate it, and during the simulation process, the sensor data is simulated based on the types and properties of various sensors loaded in the scene, and multiple data output interfaces are provided; the data annotation module is used to produce test data according to the scene simulator and the intelligent model library; the intelligent model library at least includes visual / point cloud target detection / event recognition models; the scene evaluation module is used to evaluate the sensor deployment plan of the traffic scene template; the perception test module is used to test the perception performance of the subject (roadside perception algorithm processing interface, module, equipment, service or system) according to the test database. Through the scenario modeling module + scenario simulator + data annotation module of the embodiment of the present invention, not only a large amount of test data can be constructed, but also the scenario complexity of the test data can be improved, that is, the data production efficiency is accelerated and the data richness is enhanced; through the scenario evaluation module of the embodiment of the present invention, the sensor deployment plan can be evaluated without human intervention, thereby improving the efficiency of the plan evaluation; through the perception test module of the embodiment of the present invention, the test adequacy of the subject can be improved based on massive data with sufficient scenario complexity.

[0216] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0217] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0218] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing system for sensing performance testing and sensor deployment scheme evaluation, characterized in that: The system includes: a high-precision map library, a basic element library, a scene modeling module, a scene template library, a scene simulation module, a scene simulator, a data annotation module, an intelligent model library, a test database, a scene evaluation module, a perception test module and an evaluation / test report storage module; The scenario modeling module is respectively connected to the high-precision map library, the basic element library, the scenario template library and the intelligent model library; the scenario simulation module is respectively connected to the scenario template library, the scenario simulator and the scenario evaluation module; the scenario simulator is connected to the data annotation module; the data annotation module is respectively connected to the intelligent model library and the test database; the evaluation / test report storage module is respectively connected to the scenario evaluation module and the perception test module; the perception test module is connected to the test database; the perception test module is also connected to a subject outside the system; the subject is a type of roadside perception algorithm processing interface, module, device, service or system; The high-precision map library is used to store multiple high-precision road traffic maps; The basic element library is used to store multiple types of static element templates, multiple types of dynamic element templates, multiple types of signal light element templates, multiple types of environmental element templates, multiple types of sensor element templates and traffic flow element templates; The scene modeling module is used to perform traffic scene modeling according to the high-precision map library, the basic element library and the manual configuration results of the user to obtain the corresponding traffic scene template and store it in the scene template library; the scene modeling module is also used to automatically model the traffic scene according to the real scene data set and environment description text input by the user, the intelligent model library and the basic element library to obtain the corresponding traffic scene template and store it in the scene template library; the scene modeling module is also used to construct the traffic scene template of any type of specified road condition by configuring and adjusting the traffic scene template and store it in the scene template library; the scene modeling module is also used to perform template expansion on the traffic scene template in the scene template library by data enhancement; The scene template library is used to store a plurality of traffic scene templates; each of the traffic scene templates corresponds to a unique template identifier; The scenario simulation module is used to perform traffic scenario template distribution processing on the scenario simulator and / or the scenario evaluation module according to the simulation operation instruction input by the user; The scene simulator is used to load the current traffic scene template sent by the scene simulation module when the corresponding simulator state is in the idle state, and perform dynamic simulation based on the currently loaded scene; and clear the historical data in the built-in data cache at the start of the simulation; and set the simulator state to the working state during the simulation process, and set the simulator state to the idle state at the end of the current simulation process; and perform sensor data simulation based on the types and properties of various sensors loaded in the scene during the simulation process and store the simulation data into the built-in data cache; and provide a simulator state query interface for outputting the current simulator state; and provide a sensor query interface for outputting the sensor list in the current simulation scene; and provide a simulation data query interface for outputting any sensor simulation data stored in the built-in data cache; the simulator state includes idle state and working state; The data annotation module is used to generate test data according to the scenario simulator and the intelligent model library and add the obtained test data to the test database; The intelligent model library includes at least a visual target detection model, a point cloud target detection model, a visual event recognition model, and a point cloud event recognition model; The test database includes at least a visual target detection dataset, a point cloud target detection dataset, a visual event recognition dataset, and a point cloud event recognition dataset; The scenario evaluation module is used to evaluate the sensor deployment scheme of the current traffic scenario template to generate a corresponding first evaluation report and store it in the evaluation / test report storage module; The perception test module is used to test the perception performance of the subject according to the test database to obtain a corresponding first test report and store it in the evaluation / test report storage module.

2. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 1, characterized in that: The template parameters of the static element template include static element types and static element attributes; the static element types include at least road sections, intersections, ramps, roundabouts, guide signs, instruction signs, construction signs, warning signs, prohibition signs, instruction markings, warning markings, prohibition markings, road poles, obstacles, roadside equipment, buildings, and plants; the static element attributes are the attribute sets of the current static element, including at least ground coordinates, shape and size, orientation, and color; The template parameters of the dynamic element template include dynamic element type, dynamic element attribute, element motion mode and element motion trajectory; the dynamic element type includes at least cars, motorcycles, bicycles and pedestrians; the dynamic element attribute is the attribute set of the current dynamic element, including at least shape, size and color; the element motion mode includes multiple types of motion models; the element motion trajectory is composed of multiple motion trajectory points; each of the motion trajectory points includes at least trajectory point coordinates, trajectory point speed, trajectory point acceleration and trajectory point heading angle; The template parameters of the signal light element template include signal light type, signal light attribute and signal light transformation rule; the signal light type includes at least motor vehicle signal light, non-motor vehicle signal light, pedestrian crossing signal light, direction indicator light, lane signal light, flashing warning signal light, road and railway level crossing signal light; the signal light attribute is the attribute set of the current signal light, including at least spatial coordinates and shape type; the signal light transformation rule includes the color of the signal light at the initial moment, the rotation order of the signal light color and the duration of each signal light color; The template parameters of the environmental element template include the environmental element type and the intensity change sequence; the environmental element type includes at least sunny, rainy, snowy, foggy, natural light source, and street light source; when the environmental element type is sunny, rainy, snowy, and foggy, the intensity change sequence includes the visibility change sequence and the road surface slipperiness change sequence; when the environmental element type is natural light source and street light source, the intensity change sequence includes the illumination intensity change sequence, the illumination angle change sequence, and the illumination range change sequence; The template parameters of the sensor element template include sensor type, sensor attributes and sensor jitter settings; the sensor type includes at least camera and laser radar; when the sensor type is a camera, the sensor attributes include at least camera internal parameters, camera external parameters, camera field of view angle, camera sampling rate, camera installation position, camera installation angle, and camera maximum shooting distance; when the sensor type is a laser radar, the sensor attributes include at least radar internal parameters, radar external parameters, radar field of view angle, radar sampling rate, radar installation position, radar installation angle, and radar maximum scanning distance; the sensor jitter settings include jitter direction, jitter frequency, and jitter amplitude; The template parameters of the traffic flow element template at least include vehicle traffic flow, pedestrian traffic flow, driving speed and congestion index; The designated road conditions at least include normal driving of vehicles on the road / intersection, rear-end collision of vehicles on the road / intersection, turning of vehicles on the road / intersection, scratching of vehicles on the road / intersection, collision of vehicles on the road / intersection, road construction, driving on narrow roads, driving on curves, driving on slopes, congestion during rush hour, driving on roads / intersections in bad weather, driving on roads / intersections under strong light conditions, and driving on roads / intersections under weak light conditions; The traffic scene template includes a first template high-precision map and a first template element set; The first template element set consists of a plurality of first template elements, each of which corresponds to a unique element identifier; The query parameter of the simulator status query interface is empty; The interface return data of the simulator status query interface is the corresponding simulator status; The query parameter of the sensor query interface is empty; the interface return data of the sensor query interface is the corresponding sensor list; the sensor list includes multiple sensor records; each sensor record includes a sensor identifier and a sensor type; the sensor type includes at least a camera and a laser radar; The query parameters of the simulation data query interface include sensor identification parameters, sensor type parameters and data time parameters; the sensor type parameters include at least camera and lidar; the data time parameter is a time period; the return data of the simulation data query interface is the image sequence and point cloud sequence corresponding to the camera or lidar specified by the current query parameters.

3. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: The scene modeling module is specifically used for: when the traffic scene modeling is performed according to the high-precision map library, the basic element library and the manual configuration result of the user to obtain the corresponding traffic scene template and store it in the scene template library: Providing a visual map editing interface for the user; and providing the user with a map retrieval option and a map editing window on the visual map editing interface; and after the user retrieves and selects a road traffic high-precision map from the high-precision map library through the map retrieval option, making a map copy for the current road traffic high-precision map and loading it into the map editing window; and receiving the user's element modification and / or map rotation and / or map cropping operation results on the current map copy through the map editing window, and saving the current map copy as a corresponding first template high-precision map when the user confirms saving through the map editing window; When the first template high-precision map is successfully saved, a visualization element editing interface is provided for the user; a map loading window and an element toolbar are provided for the user on the visualization element editing interface; and the first template high-precision map is loaded through the map loading window; A corresponding tool icon is assigned to each element template of the basic element library, and all the tool icons are loaded through the element toolbar; when the user selects any tool icon, an icon dragging function is provided for the user to drag the current tool icon to the map loading window, and an element template copy is generated for the element template corresponding to the current tool icon, and a unique element identifier is generated for the current element template copy; When a user selects any of the tool icons on the map loading window, a visual element configuration window is provided for the element template copy corresponding to the current tool icon for receiving and saving the user's modification results of any template parameters other than the element type of the current element template copy; when any of the tool icons on the map loading window is newly added or displaced, the position attribute of the element template copy corresponding to the current tool icon is modified based on the map coordinates of the current tool icon on the map loading window; when the user confirms saving through the visual element editing interface, the element template copy corresponding to each of the tool icons loaded on the map loading window is saved as a corresponding first template element, and all the first template elements obtained constitute a corresponding first template element set; and the obtained first template high-precision map and the first template element set constitute a corresponding traffic scene template; and setting a unique template identifier for the traffic scene template, and storing the traffic scene template with the template identifier information into the scene template library; Among them, there is and can only be loaded one tool icon corresponding to the traffic flow element template on the map loading window.

4. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: The real scene data set includes a first high-precision map, a first overhead view sequence, and one or more first horizontal view sequences; the real scene data set corresponds to a real traffic scene; the first high-precision map is a high-precision map of road traffic corresponding to the current real traffic scene; the first overhead view sequence and each of the first horizontal view sequences each correspond to a set of camera internal and external parameters; the first overhead view sequence is formed by a plurality of first overhead views sorted in chronological order, and each of the first overhead views corresponds to a camera position coordinate and an image timestamp; each of the first horizontal view sequences is formed by a plurality of first horizontal views sorted in chronological order, and each of the first horizontal view sequences corresponds to a camera position coordinate and an image timestamp; when the number of the first horizontal view sequences is greater than 1, the shooting angles of every two of the first horizontal view sequences are different.

5. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 4, characterized in that: The scene modeling module is specifically used for automatically modeling the traffic scene according to the real scene data set and the environment description text input by the user as well as the intelligent model library and the basic element library to obtain the corresponding traffic scene template and store it in the scene template library: Receiving the real scene data set and the environment description text input by the user as the corresponding first scene data set and first environment description text; and extracting the corresponding first high-precision map, the first overhead view sequence, and all the first horizontal view sequences from the first scene data set; and using the first high-precision map as a corresponding first template high-precision map; and performing weather type and light source type keyword recognition on the first environment description text to obtain corresponding weather type keywords and light source type keywords; when the weather type keyword is not empty, it at least includes sunny, rainy, snowy, and foggy; when the light source type keyword is not empty, it at least includes natural light source and street light source; and performing target detection processing on each of the first top view and each of the first plan view based on the visual target detection model of the intelligent model library to obtain a corresponding plurality of first target objects to form a corresponding first target object group; All the first target object groups corresponding to the first overhead view are sorted in chronological order to form a corresponding first target object group sequence; all the first target object groups corresponding to each of the plan views are sorted in chronological order to form a corresponding first target object group sequence; each of the first target objects corresponds to a group of target object attributes, including target type, target coordinates, target size, and target orientation; and performing a coordinate system conversion from an image coordinate system to a world coordinate system on a target coordinate, a target size, and a target orientation of a current target object based on the camera intrinsic and extrinsic parameters and the camera coordinates of each of the first target objects; and adding the image timestamps corresponding to each of the first target object groups as corresponding target timestamps to the target object attributes of each of the target objects in the current first target object group; The updated target object attributes of the first target object include target type, target coordinates, target size, target orientation and target timestamp; and taking each of the first target object group sequences as the corresponding current target object group sequence; Based on a preset target association algorithm, target association is performed on every two adjacent first target object groups in the current target object group sequence, and based on the target association result, the target object attributes of each first target object corresponding to the same associated target are used as the trajectory point attributes of a corresponding first trajectory point, and all the first trajectory points corresponding to the same associated target are sorted in chronological order to form a corresponding first target trajectory; the target association algorithm at least includes the Hungarian algorithm; the first target trajectory is formed by sequentially sorting a plurality of the first trajectory points, and the trajectory point attributes of each of the first trajectory points include target type, target coordinates, target size, target orientation and target timestamp; the target types corresponding to all the first trajectory points on the first target trajectory are the same and will not change; Taking the target coordinates and the target orientation as observation quantities, and the target speed, the target acceleration and the target heading angle as state quantities, estimating the target speed, the target acceleration and the target heading angle corresponding to each of the first trajectory points on each of the first target trajectories according to a preset filter algorithm, and adding the estimation results to the trajectory point attributes corresponding to each of the first trajectory points; The updated trajectory point attributes of the first trajectory point include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the filter algorithm includes at least a Kalman filter algorithm; Counting the sequence time lengths of the first top view sequence and each of the first plan view sequences to obtain corresponding first sequence time lengths; and taking the longest duration of the first sequence as the corresponding maximum duration; and dividing the maximum duration into a plurality of time scales based on a preset minimum unit duration; and aligning the target timestamps of the first track points of each of the first target tracks on each of the time scales with the current time scale; and estimating the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of each of the first target trajectories at each of the time scales according to the filter algorithm or the interpolation algorithm; The first track points of each of the first target tracks on all the time scales form a corresponding second target track; The trajectory similarity of every two second target trajectories is calculated to obtain the corresponding first trajectory similarity; all the second target trajectories are clustered based on a preset trajectory similarity threshold to obtain one or more corresponding second target trajectory sets; all the first trajectory points of all the second target trajectories of each second target trajectory set on each time scale are fused to obtain a corresponding second trajectory point, and all the obtained second trajectory points are sequentially sorted to form a corresponding first target fused trajectory; the second target trajectory set is composed of one or more second target trajectories; when the number of the second target trajectories in the second target trajectory set is greater than 1, any two of the second target trajectories in the current set are The first trajectory similarities between the target trajectories all exceed the trajectory similarity threshold; the first target fusion trajectory is formed by sequentially sorting a plurality of second trajectory points, and the trajectory point attributes of each of the second trajectory points include target type, target coordinates, target size, target orientation, target timestamp, target speed, target acceleration and target heading angle; the target type and target timestamp of each of the second trajectory points are consistent with the target type and target timestamp of all corresponding first trajectory points; the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of each of the second trajectory points are fused from the target coordinates, target size, target orientation, target speed, target acceleration and target heading angle of all corresponding first trajectory points; and based on the target type corresponding to each of the first target fusion trajectories, selecting a corresponding static, dynamic, signal light or sensor element template from the basic element library as a corresponding first template element; And assign a corresponding unique element identifier to the current first template element; and configuring the template parameters of the current first template element based on all trajectory point attributes of the current first target fusion trajectory; and when the weather type keyword is not empty, taking the environmental element template whose environmental element type in the basic element library matches the weather type keyword as a corresponding first template element; And assign a corresponding unique element identifier to the current first template element; and configuring the intensity change sequence of the current first template element based on a set of preset visibility change sequences and preset road slipperiness change sequences corresponding to the weather type keyword; and when the light source type keyword is not empty, taking the environment element template whose environment element type in the basic element library matches the light source type keyword as a corresponding first template element; And assign a corresponding unique element identifier to the current first template element; and configuring the intensity change timing of the current first template element based on a set of preset illumination intensity change timings, preset illumination angle change timings, and preset illumination range change timings corresponding to the light source type keyword; All the first template elements obtained this time form a corresponding first template element set; and the first template high-precision map and the first template element set form a corresponding traffic scene template; A unique template identifier is set for the traffic scene template, and the traffic scene template with the template identifier information is stored in the scene template library.

6. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: The scene modeling module is specifically used for, when the traffic scene template of the scene template library is expanded by the data enhancement method, taking each existing traffic scene template in the scene template library as the corresponding current base template; and recording the first template element corresponding to each type of the dynamic element in the current base template as the corresponding first dynamic element; and generating a plurality of derived scene templates by dynamically adjusting the element motion mode and the element motion trajectory of one or more of the first dynamic elements in the current base template; A unique template identifier is set for each of the derived scene templates; a corresponding derived scene template set is formed by all the derived scene templates corresponding to the current base template; and all the derived scene template sets obtained are added to the scene template library.

7. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 1, characterized in that: The scenario simulation module is specifically used for, when the scenario simulator and / or the scenario evaluation module performs traffic scenario template distribution processing according to the simulation operation instruction input by the user: Receive the simulation operation instruction input by the user; and extract the corresponding instruction type and instruction parameter from the simulation operation instruction; the simulation operation instruction includes the instruction type and the instruction parameter; the instruction type includes the first type, the second type and the third type; when the instruction type is the first or second type, the instruction parameter is a template identifier; when the instruction type is the third type, the instruction parameter is a template identifier sequence; and identifying the instruction type; If the instruction type is the first or second type, the template identifier in the instruction parameter is used as the corresponding current template identifier; and taking the traffic scene template corresponding to the current template identifier in the scene template library as the corresponding current traffic scene template; and sending the current traffic scene template to the scene simulator when the instruction type is the first type; and sending the current traffic scene template to the scene simulator and the scene evaluation module respectively when the instruction type is the second type; If the instruction type is the third type, the template identification sequence in the instruction parameter is used as the corresponding current template identification sequence; and the template identifications of the current template identification sequence are sequentially traversed; And during the traversal process, the template identifier currently traversed is used as the corresponding current template identifier; and taking the traffic scene template corresponding to the current template identifier in the scene template library as the corresponding current traffic scene template; and estimating the simulation duration of the current traffic scene template to obtain the corresponding current simulation duration; and calculating the corresponding first waiting time=current simulation time×(1+first delay factor) based on the preset first delay factor and the current simulation time, 0<first delay factor<1; and sending the current traffic scene template to the scene simulator; And after waiting for the first waiting time from the time when the current scene is sent, it turns to the next template identifier for traversal until the traversal of the last template identifier is completed.

8. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: All types of models in the intelligent model library have completed model training based on large data sets in advance; the visual target detection model is used to perform target detection processing based on the model input image and output the corresponding detection target set; the point cloud target detection model is used to perform target detection processing based on the model input point cloud and output the corresponding detection target set; the visual event recognition model is used to perform traffic event recognition processing based on the model input image and output one or more corresponding event types; the point cloud event recognition model is used to perform traffic event recognition processing based on the model input point cloud and output one or more corresponding event types; The visual target detection dataset includes multiple first data records; the first data record includes a first training image and a first labeled target set; the point cloud target detection dataset includes multiple second data records; the second data record includes a first training point cloud and a second labeled target set; the visual event recognition dataset includes multiple third data records; the third data record includes a second training image and a first labeled event set; the point cloud event recognition dataset includes multiple fourth data records; the fourth data record includes a second training point cloud and a second labeled event set.

9. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 8, characterized in that: The data annotation module is specifically used for: when the test data is generated according to the scenario simulator and the intelligent model library and the obtained test data is added to the test database: Initializing a first waiting flag in advance as the end of waiting; the first waiting flag includes starting waiting and ending waiting; and obtaining the corresponding simulator state through the simulator state query interface of the scenario simulator regularly at a preset first query frequency; And identify the simulator state; if the simulator state is in the working state, set the first waiting flag to start waiting; if the simulator state is in the idle state, then when the first waiting flag is start waiting, the sensor data of each camera and each laser radar in the current simulation scene of the scene simulator are acquired through the sensor query interface and the simulation data query interface of the scene simulator to obtain the corresponding multiple first image sequences and multiple first point cloud sequences, and set the first waiting flag to end waiting after all sensor data are obtained; each of the first image sequences corresponds to a camera, and each of the first point cloud sequences corresponds to a laser radar; the first image sequence is composed of multiple first images sorted in chronological order; the first point cloud sequence is composed of multiple first point clouds sorted in chronological order; and merging all the first image sequences obtained this time to obtain a corresponding first image set; and merging all the first point cloud sequences obtained this time to obtain a corresponding first point cloud set; and performing a round of traversal on all the first images in the first image set; And in this round of traversal, the first image currently traversed is used as the corresponding current image; the current image is input into the visual target detection model for target detection processing and the detection target set output by the model is used as a corresponding first label target set; the current image is input into the visual event recognition model for traffic event recognition processing and all event types output by the model are combined into a corresponding first label event set; and the current image is used as a corresponding first training image and a corresponding second training image; The first training image corresponding to the current image and the first label target set form a corresponding first data record; and the second training image corresponding to the current image and the first label event set form a corresponding third data record; At the end of this round of traversal, all the first data records obtained in this round of traversal form a corresponding first data record set, and all the third data records obtained in this round of traversal form a corresponding third data record set; And perform a round of traversal on all the first point clouds of the first point cloud set; and in this round of traversal, use the currently traversed first point cloud as the corresponding current point cloud; and input the current point cloud into the point cloud target detection model for target detection processing and use the detection target set output by the model as a corresponding second label target set; and input the current point cloud into the point cloud event recognition model for traffic event recognition processing and use all event types output by the model as a corresponding second label event set; and use the current point cloud as a corresponding first training point cloud and a corresponding second training point cloud; and forming a corresponding second data record from the first training point cloud and the second label target set corresponding to the current point cloud; The second training point cloud and the second label event set corresponding to the current point cloud form a corresponding fourth data record; At the end of this round of traversal, all the second data records obtained in this round of traversal form a corresponding second data record set, and all the fourth data records obtained in this round of traversal form a corresponding fourth data record set; and adding the first data record set obtained this time to the visual target detection data set of the test database; and adding the second data record set obtained this time to the point cloud target detection data set of the test database; and adding the third data record set obtained this time to the visual event recognition data set of the test database; And the fourth data record set obtained this time is added to the point cloud event recognition data set of the test database.

10. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: The scenario evaluation module is specifically used for evaluating the sensor deployment scheme of the current traffic scenario template to generate a corresponding first evaluation report and storing it in the evaluation / test report storage module: Extracting the first template high-precision map and the first template element set corresponding to the current traffic scene template as the corresponding current high-precision map and current element set; and taking each sensor element in the current element set as a corresponding first sensor; and counting the total number of the first sensors to obtain a corresponding first total number N; and performing a round of traversal on all the first sensors; and in this round of traversal, taking the first sensor currently traversed as the corresponding current sensor; and identifying the sensor type corresponding to the current sensor; If the sensor type corresponding to the current sensor is a camera, the road coverage range of the current sensor is identified based on the current high-precision map and the camera field of view, camera installation position, camera installation angle, and camera maximum shooting distance corresponding to the current sensor to obtain a corresponding first coverage range; If the sensor type corresponding to the current sensor is a laser radar, the road coverage range of the current sensor is identified based on the current high-precision map and the radar field of view angle, radar installation position, radar installation angle and radar maximum scanning distance corresponding to the current sensor to obtain the corresponding first coverage range; After the current round of traversal of all the first sensors is completed, all the first coverage ranges obtained are range fused to obtain the corresponding first fusion range; all the road / intersection ranges on the current high-precision map are fused to obtain the corresponding second fusion range; the intersection range of the first and second fusion ranges is identified to obtain the corresponding first intersection range; based on the first intersection range and the second fusion range, the corresponding first coverage rate is calculated = (first intersection range / second fusion range) × 100%; whether the first coverage rate is lower than the preset first coverage rate threshold is identified, if so, the corresponding coverage rate evaluation state is set to be unsatisfactory, otherwise, the corresponding coverage rate evaluation state is set to be up to standard; the coverage rate evaluation state includes up to standard and unsatisfactory; and identifying the road / intersection type corresponding to the current high-precision map to obtain a corresponding first road / intersection type; and selecting the road / intersection sensor overlapping deployment requirement corresponding to the first road / intersection type from the preset multiple road / intersection sensor overlapping deployment requirements as the corresponding first sensor overlapping deployment requirement; the first road / intersection type at least includes a straight / curved / ramp section of a highway, a straight / curved / ramp section of a national highway, a straight / curved / ramp section of an urban road, a straight / curved / ramp section of a rural road, a T-intersection, a cross intersection, an X-intersection, a compound intersection, and a roundabout; And based on the first total number N and all the first coverage ranges, an N×N two-dimensional matrix is ​​constructed and recorded as the corresponding first intersection-and-union ratio matrix; the first row to the Nth row of the first intersection-and-union ratio matrix correspond one-to-one to the first to the Nth first sensors, and the first column to the Nth column correspond one-to-one to the first to the Nth first sensors; the cell value of each matrix cell of the first intersection-and-union ratio matrix is ​​the intersection-and-union ratio of the two first coverage ranges corresponding to the two first sensors corresponding to the current matrix cell; and identifying whether the first intersection-to-union ratio matrix meets the first sensor overlapping deployment requirement, and if so, setting the corresponding overlapping deployment evaluation state to meet the standard; if not, setting the corresponding overlapping deployment evaluation state to fail to meet the standard; the overlapping deployment evaluation state includes meeting the standard and failing to meet the standard; And the template identifier corresponding to the current traffic scene template is used as the corresponding first evaluation template identifier; and each of the first sensors and the corresponding first coverage range constitutes a corresponding first sensor deployment feature, and all the obtained first sensor deployment features constitute a corresponding first sensor deployment feature set; and the first template high-precision map, the first template element set, the first total number N, the first sensor deployment feature set, the first road / intersection type, the first sensor overlapping deployment requirement, the first coverage rate, the first intersection-to-union ratio matrix, the coverage rate evaluation status and the overlapping deployment evaluation status constitute a corresponding first evaluation report which is stored in the evaluation / test report storage module.

11. The processing system for sensing performance testing and sensor deployment scheme evaluation according to claim 2, characterized in that: The perception test module is specifically used for, when the perception performance of the subject is tested according to the test database to obtain a corresponding first test report which is stored in the evaluation / test report storage module: Step 111, receiving a first test request sent by the subject; and extracting a corresponding first test request identifier, a first test type and a first test total amount from the first test request; The first test request includes the first test request identifier, the first test type and the first test total amount; the first test type includes visual target detection, point cloud target detection, visual event recognition, and point cloud event recognition; Step 112, identifying the first test type; if the first test type is visual target detection, then based on a preset test data selection rule, the first data record of the total amount of the first test is selected from the visual target detection data set of the test database to form a corresponding first data set; if the first test type is point cloud target detection, then based on the test data selection rule, the second data record of the total amount of the first test is selected from the point cloud target detection data set of the test database to form a corresponding first data set; if the first test type is visual event recognition, then based on the test data selection rule, the third data record of the total amount of the first test is selected from the visual event recognition data set of the test database to form a corresponding first data set; if the first test type is point cloud event recognition, then based on the test data selection rule, the fourth data record of the total amount of the first test is selected from the visual event recognition data set of the test database to form a corresponding first data set; Wherein, the first data set includes multiple first test records; the first test record includes first training data and first label data; when the first test type is visual target detection, the first training data and the first label data are the corresponding first training image and the first label target set; when the first test type is point cloud target detection, the first training data and the first label data are the corresponding first training point cloud and the second label target set; when the first test type is visual event recognition, the first training data and the first label data are the corresponding second training image and the first label event set; when the first test type is point cloud event recognition, the first training data and the first label data are the corresponding second training point cloud and the second label event set; Step 113, establishing a test channel with the subject and recording it as the corresponding first channel; Step 114, taking the first of the first test records in the first data set as the corresponding current test record; Step 115, sending the first training data of the current test record to the subject through the first channel; receiving the first prediction data sent back by the subject through the first channel; and forming a corresponding first prediction-label pair by the first prediction data and the first label data of the current test record; Step 116, identifying whether the current test record is the last of the first test record of the first data set; if so, closing the first channel and going to step 117; if not, extracting the next first test record of the first data set as the new current test record, and returning to step 115; Step 117, identifying the first test type; if the first test type is visual target detection or point cloud target detection, then based on all the first prediction-label pairs obtained, accuracy, recall and prediction deviation analysis is performed to obtain a corresponding first accuracy, first recall and first prediction deviation set to form a corresponding first analysis data set; if the first test type is visual event recognition or point cloud event recognition, then based on all the first prediction-label pairs obtained, accuracy and recall analysis is performed to obtain a corresponding second accuracy and second recall to form a corresponding first analysis data set; Wherein, the first prediction deviation set includes at least a first coordinate prediction deviation and a first orientation prediction deviation; Step 118, forming a corresponding first prediction-label pair set from all the first prediction-label pairs obtained; and forming a corresponding first test report from the first test request, the first data set, the first prediction-label pair set and the first analysis data set, which is stored in the evaluation / test report storage module.

Images