Automatic test system and method for mining area unmanned driving perception algorithm

Abstract

The invention relates to an automatic test system and method for a mining area unmanned driving perception algorithm, belongs to the technical field of automatic driving, solves the problem of driving risks caused by inaccurate perception results in the prior art, and comprises the following steps: S1, executing data acquisition, data classification and data preparation by a data management tool module; obtaining a test scene database and a to-be-tested list, wherein the test scene database stores the matched total data and truth value files; s2, the quantitative test software module executes automatic test and test index analysis processing to obtain a test result, a test report and a perception algorithm evaluation result file; and S3, a visualization and result analysis module analyzes and visualizes the perception algorithm evaluation result file.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an automated testing system and method for an unmanned driving perception algorithm in a mining area. Background Art

[0002] The perception system is the core part of the unmanned driving technology system. It is responsible for collecting surrounding environmental information, processing and analyzing this information to form an understanding and cognition of the environment. On the one hand, the operating link of the perception system is complex, and it is affected by the stability of sensors, voltage, communication links, etc., which may lead to incomplete perception output results. On the other hand, due to external factors such as the complexity and variability of scene input, the rugged roads in the mining area, the constraints of the actual mining driving environment, and environmental uncertainties (high pressure, cold, strong winds and dust, etc.), missed detections and false detections often occur in the perception system recognition, which brings driving risks.

[0003] At present, the commonly used perception algorithm testing methods are mostly based on manual methods, which replay the original data collected by the vehicle sensors, cut the replayed data into frames, and manually compare the perception algorithm detection results and the true value frame by frame to calculate the test results.

[0004] The above method requires a lot of manpower, time and economic costs, external errors have a great impact on the test results, and the credibility of the test results is low. Summary of the invention

[0005] In view of this, the present invention provides an automatic quantitative testing system for autonomous driving perception algorithms for mining areas, which aims to quantify the accuracy, recall rate and processing time of obstacle recognition in various scenarios during driving in mining areas, solving the high cost problem caused by reliance on manual evaluation or labeling in the prior art, while improving the credibility of the test results, and providing a more powerful basis for targeted adjustment of the robustness and adaptability of the algorithm in various scenarios.

[0006] According to one embodiment of the present invention, an automated testing device testing system for unmanned driving perception algorithms in mining areas is provided, comprising: a data management tool module, a quantitative testing software module, and a visualization and result analysis module, wherein:

[0007] The data management tool module includes a field module and a research and development module. The field module records the collected raw data. The research and development module includes a data management submodule and a data consumption submodule. The data management submodule manages the true value files and the raw data, and classifies, stores and queries the returned full data. The data consumption submodule performs file system mounting operations on the storage node and the consumption node.

[0008] The quantitative test software module includes a data reading and communication module, a quantitative index calculation module, and a test result output and push module;

[0009] The data reading and communication module interacts with the perception algorithm program being tested, reads the truth value file, and interacts with the visualization module in the visualization and result analysis module, and sends the real-time perception target output information and the truth value information to the visualization module for display;

[0010] The quantitative index calculation submodule compares the perception result output by the perception algorithm program with the true value information corresponding to the target obstacle according to the custom algorithm, calculates the accuracy and recall rate of the same type of obstacle for a single frame of data, calculates the precision for all frames of the same type of obstacle, calculates the pass rate for each category of target, calculates the overall pass rate for all categories of obstacles, and calculates the average precision mean of all categories of targets;

[0011] The test result output and push submodule counts and saves the quantitative test results calculated by the quantitative index calculation submodule, and pushes the quantitative test results to relevant users;

[0012] The visualization and result analysis module includes a visualization module and a historical result display module. The visualization module visualizes the original data, real-time perception target output information and true value information; and compares the test results of the historical iteration version and the current optimized version, and generates and displays charts in different dimensional quantitative indicator panels.

[0013] According to another embodiment of the present invention, there is provided an automated testing method for an unmanned driving perception algorithm in a mine area, comprising the following steps:

[0014] Step S1, the data management tool module performs data collection, data classification and data preparation, obtains a test scenario database storing full data files, searches and extracts full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested;

[0015] Step S2, the quantitative test software module starts the perception algorithm program, performs automated testing and test index analysis and processing, and obtains test results, test reports and perception algorithm evaluation result files;

[0016] Step S3, the visualization and result analysis module analyzes and visualizes the perception algorithm evaluation result file, and the contents in the perception algorithm evaluation result file and the visualization chart are used to optimize the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm result is obtained and provided to the downstream prediction planning module for collision risk prediction and path planning of vehicles in the mining area;

[0017] Wherein, the step S1 specifically includes the following steps:

[0018] Step S1.1, the field module of the data management tool module performs data collection, obtains raw data marked with a timestamp and stores it as a raw data file, the raw data including the vehicle's own GNSS data, and uses the RTK device to mark each two-dimensional corner point and center point of the target obstacle to record the GPS true value information;

[0019] Step S1.2, establishing a test scenario database based on the pre-acquired high-precision map file and calibration parameter file, and obtaining an original data file, matching the original data file with the pre-acquired high-precision map file and calibration parameter file to obtain matched data, and storing the matched data in the test scenario database;

[0020] Step S1.3, the research and development module of the data management tool module classifies and stores the data, classifies the corresponding data, and stores it in the test scenario database according to functions and requirements;

[0021] Step S1.4, the research and development module of the data management tool module obtains GPS true value information from the RTK device, converts it into a UTM true value, and calculates an attribute true value based on the UTM true value and the collected GNSS data of the vehicle itself, in which a frame number representing a timestamp is preset;

[0022] Step S1.5, the research and development module of the data management tool module generates a truth value file based on the obtained attribute truth value, and stores the generated truth value file in the test scenario database. The truth value file in the test scenario database and the obtained corresponding data together constitute a full data file;

[0023] Step S1.6, the R&D module of the data management tool module prepares data, searches and extracts the full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested, wherein the extracted full data files include the corresponding data and true value files related to the target type to be tested, and the list to be tested includes the test items of the target scenario to be tested indicating the target type to be tested.

[0024] Optionally, step S2 specifically includes the following steps:

[0025] Step S2.1, the quantitative test software module deploys the software version to be tested on the Jenkins continuous integration tool and performs preliminary commissioning, and starts the quantitative test software module, matches the original data file in the extracted full data file with the high-precision map file and the calibration parameter file to obtain a matched data file, and the matched data file retains the marked timestamp;

[0026] Step S2.2, the quantitative test software module reads the list to be tested and determines whether the list to be tested is empty. If the list to be tested is not empty, execute step S2.3; otherwise, execute step S2.4;

[0027] Step S2.3, the quantitative index calculation module of the quantitative test software module starts a specific version of the perception program as the perception algorithm program, reads the true value file related to the test item in the test list from the extracted full data file, tests the performance of the perception algorithm program, obtains the test results, and generates a perception algorithm evaluation result file according to the test results;

[0028] Step S2.4, the quantitative index calculation module of the quantitative test software module analyzes the test index in the test result to determine whether the index requirements are met. If yes, execute step S2.5; otherwise, analyze the reason for the test failure, return to step S1.6, and regenerate the list to be tested;

[0029] Step S2.5, after the test is completed, the quantitative test software module generates a detailed test report based on the test results, records all key information during the test, and provides the current version of the test report including the key information to the development and algorithm personnel through the output and push module;

[0030] Step S2.6, the test passes, and the version of the perception algorithm evaluation result file is updated according to the test results to complete the test process.

[0031] Optionally, the step S2.3 specifically includes the following steps:

[0032] Step S2.3.1, the quantitative test software module starts a specific version of the perception program, including pulling the latest version of the perception program as the perception algorithm program;

[0033] Step S2.3.2, the quantitative test software module pulls the latest version of the quantitative test program and configures the quantitative test parameters according to the test requirements;

[0034] Step S2.3.3, the quantitative test software module compiles and starts the pulled quantitative test program, and starts the perception algorithm program;

[0035] Step S2.3.4, the quantitative test software module reads the true value file related to the to-be-tested items in the to-be-tested list from the extracted full data file according to the to-be-tested list;

[0036] Step S2.3.5, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, sends it to the quantitative test program for testing, obtains the test result, and generates the perception algorithm evaluation result file based on the test result.

[0037] Optionally, the step S2.3.5 specifically includes the following steps:

[0038] Step S2.3.5.1, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, which is marked with a frame number according to the timestamp and includes the detection and tracking information of the target obstacle, and the detection and tracking information of the target obstacle includes the coordinate information of the center point and corner points of the obstacle detection frame in the vehicle body coordinate system, the size information of the obstacle detection frame and the yaw angle information of the obstacle detection frame;

[0039] Step S2.3.5.2, the quantitative test software module obtains the corresponding perception result from the perception algorithm program according to the frame number, and retrieves the true value information from the read truth value file according to the above frame number to obtain the attribute true value corresponding to the frame number;

[0040] Step S2.3.5.3, determine the center of the circle according to the true value of the attribute corresponding to the current frame number, determine the radius according to the threshold in the configured quantization test parameter, and obtain a circular area range;

[0041] Step S2.3.5.4, determine whether the detection and tracking information of the target obstacle in the perception result corresponding to the current frame number is within the circular area. If so, it is determined to be a positive detection and the next step is executed; if not, it is determined to be a missed detection or a false detection and returns to step S2.3.5.2;

[0042] Step S2.3.5.5, calculating the accuracy and recall rate based on the detection and tracking information of the target obstacle in the current perception result and the read truth file;

[0043] Step S2.3.5.6, the quantitative test software module calculates the average precision mean of the current perception algorithm program, and calculates the average precision for each individual test category as the test result, and calculates the accuracy and recall rate based on the test results and generates a perception algorithm evaluation result file, and returns to step S2.2.

[0044] Optionally, the step S3 specifically includes the following steps:

[0045] Step S3.1, obtaining the current version of the perception algorithm evaluation result file;

[0046] Step S3.2, obtaining the perception algorithm evaluation result files of the last five tests;

[0047] Step S3.3, the visualization and result analysis module analyzes the accuracy of each test category and the pass rate of each version of the perception algorithm evaluation result file as the analysis result;

[0048] Step S3.4, the visualization and result analysis module uses the analysis results to draw trend graphs of the quantitative indicators of each version and displays them for optimizing the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm results are obtained and provided to the downstream prediction and planning module for collision risk prediction and path planning of vehicles in the mining area.

[0049] Compared with the prior art, an automated testing system and method for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention has at least the following advantages:

[0050] (1) Automatic quantitative testing of perception algorithms is implemented, and the test system is automatically triggered and test reports are output for different versions of perception algorithms.

[0051] (2) Save costs and minimize manual intervention in the testing process. It also saves time and economic costs of annotating data and improves the efficiency of perception algorithm iteration.

[0052] (3) Improve the credibility of the results. The test results obtained have a low error rate and a high fault tolerance rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A schematic diagram of the module composition of an automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention.

[0055] Figure 2 A workflow diagram of a data management tool module and a quantitative test software module of an automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention.

[0056] Figure 3This is a workflow diagram of a quantitative testing software module of an automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention.

[0057] Figure 4 A visualization workflow diagram of a visualization and result analysis module of an automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention.

[0058] Figure 5 A result analysis workflow diagram of a visualization and result analysis module of an automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0060] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0061] An automated testing system for an unmanned driving perception algorithm in a mining area provided according to an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0062] like Figure 1 As shown, an automatic testing system for an unmanned driving perception algorithm in a mining area provided by an embodiment of the present invention includes multiple modules and lower-level sub-modules.

[0063] like Figure 1 As shown, an automated testing system for unmanned driving perception algorithms in mining areas provided according to an embodiment of the present invention includes: a data management tool module, a quantitative testing software module, and a visualization and result analysis module.

[0064] Specifically, the data management tool module includes a field module and a research and development module, and the research and development module includes a data management submodule and a data consumption submodule.

[0065] The field module records the true value information of the target. The true value information includes the GPS true value information and the real size obtained by marking each corner point and center point of the target through RTK (real-time kinematic differential), and the field module is used as an intermediary to preview, copy and transmit the true value file composed of the true value information and the original data collected by the vehicle sensor.

[0066] The R&D module includes a data management submodule and a data consumption submodule. The data management submodule manages the true value files and the original data, and performs operations such as classification, warehousing, and query on the full amount of data returned. The data consumption submodule performs file system mounting operations on the storage nodes and consumption nodes.

[0067] The quantitative test software module includes a data reading and communication module, a quantitative index calculation module, and a test result output and push module.

[0068] The data reading and communication module interacts with the perception algorithm program as input, reads the corresponding true value file according to the original point cloud file serial number read by the perception algorithm program, and obtains real-time perception target output information, timeout information, etc., and interacts with the visualization module in the visualization and result analysis module, and sends the real-time perception target output information and true value information to the visualization module for display.

[0069] The quantitative index calculation submodule compares the perception target output information output by the perception algorithm program and the true value information corresponding to the target obstacle according to a custom algorithm, calculates the accuracy and recall rate of the same type of obstacles for single frame data, and outputs whether the current frame passes the test; calculates the accuracy (Average Precision, referred to as AP) of all frames of the same type of obstacles, calculates the pass rate for each category of targets, and calculates the overall pass rate for all categories of obstacles; calculates the mean average precision (mean Average Precision, referred to as mAP) of all categories of targets.

[0070] The test result output and push submodule collects statistics on the quantitative test results calculated by the quantitative index calculation submodule, saves them in the form of text to the local hard disk, and pushes the quantitative test results to subscribed users.

[0071] The visualization and result analysis module includes a visualization module and a historical result display module.

[0072] The visualization module visualizes the original data, real-time perception target output information and true value information; and visualizes the high-precision map information matched with the original data.

[0073] The historical result display module compares the test results of the historical iteration version and the current optimized version, generates and displays charts in different dimensional quantitative indicator panels.

[0074] like Figure 1-Figure 5 As shown, according to another embodiment of the present invention, an automated testing method for an unmanned driving perception algorithm in a mining area includes the following steps.

[0075] Step S1: The data management tool module performs data collection, data classification and data preparation to obtain a test scenario database storing full data files, searches and extracts full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested. Figure 2 As shown, step S1 specifically includes the following steps.

[0076] Step S1.1, the field module of the data management tool module performs data collection, obtains raw data marked with a timestamp and stores it as a raw data file, which includes the vehicle's own GNSS data. Among them, the raw data can be collected by the sensors of the vehicles deployed in the mining area and stored in the built-in hard disk of the vehicle's onboard industrial computer as a raw data file. In this step, the RTK equipment can also be used to mark each two-dimensional corner point and center point of the collected target obstacle to record the GPS true value information.

[0077] Step S1.2, establish a test scenario database based on the pre-acquired high-precision map file and calibration parameter file, obtain the original data file, correspond the original data file with the pre-acquired high-precision map file and calibration parameter file, obtain the corresponding data, and store the corresponding data in the test scenario database. In this step, a distributed storage node can also be used to remotely transmit the corresponding data back to the field module of the data management tool module. The corresponding data includes the original data file, the high-precision map file and the calibration parameter file. Through this corresponding processing, different types of data in the test scenario database, for example, the original data file and the high-precision map file and the calibration parameter file, can be matched.

[0078] Step S1.3, the R&D module of the data management tool module establishes a test scenario database, and performs data classification and storage, classifies the corresponding data, and stores it in the test scenario database according to functions and requirements. During classification, the full amount of data can also be labeled for subsequent rapid retrieval and use.

[0079] Specifically, the specific scenarios of the established test scenario database can be pre-designed, and each test scenario is guaranteed to target only one type of obstacle. The test scenario database stores the original data files, high-precision map files, calibration parameter files, and the true value files mentioned below.

[0080] Step S1.4, the research and development module of the data management tool module obtains GPS true value information from the RTK device, converts it into utm (UNIVERSAL TRANSVERSE MERCARTOR GRID SYSTEM) true value, and calculates the attribute true value based on the utm true value and the collected vehicle's own GNSS (Global Navigation Satellite System) data. The attribute true value may include target timestamp information, type information, identity information, x, y, z coordinate information of the target center point in the utm coordinate system, x, y, z coordinate information of the target center point in the vehicle body coordinate system, length, width, height size information, absolute speed and horizontal and longitudinal component information, absolute acceleration and horizontal and longitudinal component information, target heading information, degree of occlusion, etc. It is also possible to preset a frame number representing the timestamp in the attribute true value.

[0081] Step S1.5, the research and development module of the data management tool module generates a truth value file based on the obtained attribute truth value, and stores the generated truth value file in the test scenario database. The truth value file in the test scenario database and the obtained corresponding data together constitute a full data file.

[0082] Step S1.6, the R&D module of the data management tool module prepares data, searches and extracts the full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested, wherein the extracted full data files include the corresponding data and true value files related to the target type to be tested, and the list to be tested includes the test items of the target scenario to be tested indicating the target type to be tested.

[0083] Step S2: The quantitative test software module performs automated testing and test index analysis and processing to obtain test results, test reports and perception algorithm evaluation result files. Figure 2 and Figure 3 As shown, step S2 specifically includes the following steps.

[0084] Step S2.1, the quantitative test software module deploys the software version to be tested on the Jenkins continuous integration tool (Jenkins continuous integration tool), and performs preliminary commissioning to ensure that its basic functions are running normally; the data of the test scenario database is mounted to the quantitative test software module, the quantitative test software module is started, and the original data in the extracted full data file is matched with the high-precision map file and the calibration parameter file to obtain a matched data file, which retains the marked timestamp. This matching process can ensure that the different types of data read by the quantitative test software module come from the same time, the same scene, etc., to ensure the accuracy of the data during the test.

[0085] Step S2.2, the quantitative test software module reads the list to be tested and determines whether the list to be tested is empty. If the list to be tested is not empty, step S2.3 is executed, otherwise, step S2.4 is executed. The quantitative test software module of the system can automatically trigger the test task through the Jenkins continuous integration tool to ensure the automation and continuity of the test process.

[0086] Step S2.3, the quantitative index calculation module of the quantitative test software module starts a specific version of the perception program as the perception algorithm program, reads the true value file related to the to-be-tested items in the to-be-tested list from the extracted full data file, tests the performance of the perception algorithm program, obtains the test results, and generates a perception algorithm evaluation result file according to the test results. This step S2.3 specifically includes the following steps.

[0087] Step S2.3.1, the quantitative test software module starts a specific perception version program, including pulling the latest version of the perception program as the perception algorithm program to ensure the update and accuracy of the perception algorithm program version to be tested.

[0088] Step S2.3.2, the quantitative test software module pulls the latest version of the quantitative test program and configures the quantitative test parameters according to the test requirements. The test parameters include the test threshold, performance indicators, etc., which are crucial to the accuracy of the test results.

[0089] Step S2.3.3, the quantitative test software module compiles and starts the pulled quantitative test program, and starts the perception algorithm program.

[0090] Step S2.3.4, the quantitative test software module reads the truth value information in the truth value file related to the to-be-tested items in the to-be-tested list from the extracted full data file according to the to-be-tested list.

[0091] Step S2.3.5: During the test, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, sends it to the quantitative test program for testing, obtains the test result, and generates the perception algorithm evaluation result file based on the test result. The specific test process is as follows.

[0092] Step S2.3.5.1, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, which is marked with a frame number according to the timestamp and includes the detection and tracking information of the target obstacle. The loaded perception scene database provides the necessary data support for simulating the real perception environment. The detection and tracking information of the target obstacle can be the coordinate information of the center point and corner point of the obstacle detection frame in the vehicle body coordinate system, the obstacle detection frame size information, and the yaw angle information of the obstacle detection frame.

[0093] Step S2.3.5.2, the quantitative test software module obtains the corresponding perception result from the perception algorithm program according to the frame number, and retrieves the true value information from the read truth value file according to the above frame number to obtain the attribute true value corresponding to the frame number.

[0094] Step S2.3.5.3, determine the center of the circle according to the true value of the attribute corresponding to the current frame number, determine the radius according to the threshold in the configured quantization test parameter, and obtain a circular area range.

[0095] Step S2.3.5.4, determine whether the detection and tracking information of the target obstacle in the perception result corresponding to the current frame number is within the circular area. If it is within the area, it is determined to be a positive detection and the next step is executed; if it is not within the area, it is determined to be a missed detection or a false detection and returns to step S2.3.5.2. When it is determined to be a missed detection or a false detection, it can be further determined whether there is a detection target output by its perception algorithm program within the entire area. If the detection and tracking information of the target obstacle does not exist within the above-mentioned area but exists in other areas, it is determined to be a false detection; if it does not exist in the above-mentioned area nor in other areas, it is determined to be a missed detection.

[0096] Step S2.3.5.5, calculate the accuracy and recall rate based on the current detection and tracking information of the target obstacle and the read true value file.

[0097] Step S2.3.5.6, the quantitative test software module calculates the mean average precision (mAP) of the perception algorithm program, and calculates the average precision (AP) for each individual test category as the test result, and calculates the accuracy and recall rate based on the test results and generates a perception algorithm evaluation result file, and returns to step S2.2. The mean average precision (mAP) of the model is an important indicator for evaluating the performance of the perception algorithm program. After completing these calculations, compile the program of the quantitative test software module and prepare for actual test execution. The system calculates the average precision (AP) for each individual test category, and starts the quantitative indicator calculation module in the quantitative test software module to perform specific test tasks. During the test, the system calculates the accuracy and recall rate based on the test results. These two indicators are used to evaluate the predictive ability of the model.

[0098] In the above test process, if a test failure occurs, you can use the analysis tool to find out the cause of the failure and match the problem.

[0099] Step S2.4, the quantitative index calculation module of the quantitative test software module analyzes the test index in the test result to determine whether the index requirements are met. If yes, execute step S2.5, otherwise analyze the reason for the test failure, return to step S1.6, regenerate the list to be tested, and modify the perception algorithm and retrain the perception algorithm to improve performance. The test results can also be remotely transmitted back to the development and algorithm team through the test result output and push module of the quantitative test software module, and manual acceptance and fine-tuning are performed to ensure the quality of the final product.

[0100] Step S2.5, complete the test, the quantitative test software module generates a detailed test report based on the test results, records all key information in the test process, and provides the current version of the test report perception algorithm evaluation result file including the key information to the development and algorithm personnel through the output and push module, and pushes the test report perception algorithm evaluation result file to relevant personnel for subsequent iteration and optimization.

[0101] Step S2.6, the test passes, and the version of the perception algorithm evaluation result file is updated according to the test results to complete the test process.

[0102] In the process of executing automated tests in the above quantitative test software module, the visualization and result analysis module can visualize the test information. Figure 4 As shown, the visualization in this implementation manner, the specific process is as follows.

[0103] When the quantitative test software module is started, the visualization and result analysis module can be started and enter the initialization phase to prepare for the test process. The user enters the version information of the perception algorithm program software to be tested into the visualization and result analysis module. During the test, the visualization and result analysis module displays the detection and tracking information of the target obstacle received from the perception algorithm program, obtains the true value file from the test scenario database, and receives the perception algorithm evaluation result file from the quantitative test software module. The visualization module can also display the test results when the test is completed.

[0104] Step S3, the visualization and result analysis module analyzes and visualizes the perception algorithm evaluation result file, and the contents in the perception algorithm evaluation result file and the visualization chart are used to optimize the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm result is obtained and provided to the downstream prediction and planning module for collision risk prediction and path planning of vehicles in the mining area. Specifically, the following steps are included.

[0105] Step S3.1, obtain the current version of the perception algorithm evaluation result file.

[0106] Step S3.2, obtain the perception algorithm evaluation result files of the last five tests. In order to conduct in-depth analysis, the system needs to input the perception algorithm evaluation result files of the last five tests. The data in these historical perception algorithm evaluation result files will be used to compare with the current perception algorithm evaluation result files to evaluate the improvement of software performance.

[0107] Step S3.3, the visualization and result analysis module analyzes the accuracy (AP) of each test category and the pass rate of each version of the perception algorithm evaluation result file. As the analysis result, the pass rate reflects the stability and reliability of the software under specific test conditions. The mean average precision (mAP) of each version of the perception algorithm evaluation result file can also be further calculated and analyzed. mAP provides a comprehensive performance evaluation indicator.

[0108] Step S3.4, the visualization and result analysis module uses the analysis results to draw trend graphs of the quantitative indicators of each version and displays them for optimizing the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm results are obtained and provided to the downstream prediction and planning module for collision risk prediction and path planning of vehicles in the mining area. This trend graph provides users with an intuitive tool for observing the evolution of software performance over time.

[0109] The optimized perception algorithm program obtained in the above manner can achieve more accurate and stable target detection and tracking in mining scenarios, and provide the perception algorithm results obtained by the optimized perception algorithm program to the downstream prediction planning model, so that the vehicle can complete real-time and reliable collision risk prediction and efficient and safe path planning.

[0110] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0111] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0112] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An automated testing system for unmanned driving perception algorithms in mining areas, characterized in that: include: Data management tool module, quantitative test software module, and visualization and result analysis module, among which, The data management tool module includes a field module and a research and development module. The field module records the collected raw data. The research and development module includes a data management submodule and a data consumption submodule. The data management submodule manages the true value files and the raw data, and classifies, stores and queries the returned full data. The data consumption submodule performs file system mounting operations on the storage node and the consumption node. The quantitative test software module includes a data reading and communication module, a quantitative index calculation module, and a test result output and push module; The data reading and communication module interacts with the perception algorithm program being tested, reads the truth value file, and interacts with the visualization module in the visualization and result analysis module, and sends the real-time perception target output information and the truth value information to the visualization module for display; The quantitative index calculation submodule compares the perception result output by the perception algorithm program with the true value information corresponding to the target obstacle according to the custom algorithm, calculates the accuracy and recall rate of the same type of obstacle for a single frame of data, calculates the precision for all frames of the same type of obstacle, calculates the pass rate for each category of target, calculates the overall pass rate for all categories of obstacles, and calculates the average precision mean of all categories of targets; The test result output and push submodule counts and saves the quantitative test results calculated by the quantitative index calculation submodule, and pushes the quantitative test results to relevant users; The visualization and result analysis module includes a visualization module and a historical result display module. The visualization module visualizes the original data, real-time perception target output information and true value information; and compares the test results of the historical iteration version and the current optimized version, and generates and displays charts in different dimensional quantitative indicator panels.

2. A method for an automated testing system for an unmanned driving perception algorithm in a mining area based on claim 1, characterized in that: The following steps are involved: Step S1, the data management tool module performs data collection, data classification and data preparation, obtains a test scenario database storing full data files, searches and extracts full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested; Step S2, the quantitative test software module starts the perception algorithm program, performs automated testing and test index analysis and processing, and obtains test results, test reports and perception algorithm evaluation result files; Step S3, the visualization and result analysis module analyzes and visualizes the perception algorithm evaluation result file, and the contents in the perception algorithm evaluation result file and the visualization chart are used to optimize the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm result is obtained and provided to the downstream prediction planning module for collision risk prediction and path planning of vehicles in the mining area; Wherein, the step S1 specifically includes the following steps: Step S1.1, the field module of the data management tool module performs data collection, obtains raw data marked with a timestamp and stores it as a raw data file, the raw data including the vehicle's own GNSS data, and uses the RTK device to mark each two-dimensional corner point and center point of the target obstacle to record the GPS true value information; Step S1.2, establishing a test scenario database based on the pre-acquired high-precision map file and calibration parameter file, and obtaining an original data file, matching the original data file with the pre-acquired high-precision map file and calibration parameter file to obtain matched data, and storing the matched data in the test scenario database; Step S1.3, the research and development module of the data management tool module classifies and stores the data, classifies the corresponding data, and stores it in the test scenario database according to functions and requirements; Step S1.4, the research and development module of the data management tool module obtains GPS true value information from the RTK device, converts it into a UTM true value, and calculates an attribute true value based on the UTM true value and the collected GNSS data of the vehicle itself, in which a frame number representing a timestamp is preset; Step S1.5, the research and development module of the data management tool module generates a truth value file based on the obtained attribute truth value, and stores the generated truth value file in the test scenario database. The truth value file in the test scenario database and the obtained corresponding data together constitute a full data file; Step S1.6, the R&D module of the data management tool module prepares data, searches and extracts the full data files related to the target type to be tested from the test scenario database according to the test requirements, and generates a list to be tested according to the target type to be tested, wherein the extracted full data files include the corresponding data and true value files related to the target type to be tested, and the list to be tested includes the test items of the target scenario to be tested indicating the target type to be tested.

3. The method according to claim 2, characterized in that The step S2 specifically includes the following steps: Step S2.1, the quantitative test software module deploys the software version to be tested on the Jenkins continuous integration tool and performs preliminary commissioning, and starts the quantitative test software module, matches the original data file in the extracted full data file with the high-precision map file and the calibration parameter file to obtain a matched data file, and the matched data file retains the marked timestamp; Step S2.2, the quantitative test software module reads the list to be tested and determines whether the list to be tested is empty. If the list to be tested is not empty, execute step S2.3; otherwise, execute step S2.4; Step S2.3, the quantitative index calculation module of the quantitative test software module starts a specific version of the perception program as the perception algorithm program, reads the true value file related to the test item in the test list from the extracted full data file, tests the performance of the perception algorithm program, obtains the test results, and generates a perception algorithm evaluation result file according to the test results; Step S2.4, the quantitative index calculation module of the quantitative test software module analyzes the test index in the test result to determine whether the index requirements are met. If yes, execute step S2.5; otherwise, analyze the reason for the test failure, return to step S1.6, and regenerate the list to be tested; Step S2.5, after the test is completed, the quantitative test software module generates a detailed test report based on the test results, records all key information during the test, and provides the current version of the test report including the key information to the development and algorithm personnel through the output and push module; Step S2.6, the test passes, and the version of the perception algorithm evaluation result file is updated according to the test results to complete the test process.

4. The method according to claim 3, characterized in that The step S2.3 specifically includes the following steps: Step S2.3.1, the quantitative test software module starts a specific version of the perception program, including pulling the latest version of the perception program as the perception algorithm program; Step S2.3.2, the quantitative test software module pulls the latest version of the quantitative test program and configures the quantitative test parameters according to the test requirements; Step S2.3.3, the quantitative test software module compiles and starts the pulled quantitative test program, and starts the perception algorithm program; Step S2.3.4, the quantitative test software module reads the true value file related to the to-be-tested items in the to-be-tested list from the extracted full data file according to the to-be-tested list; Step S2.3.5, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, sends it to the quantitative test program for testing, obtains the test result, and generates the perception algorithm evaluation result file based on the test result.

5. The method according to claim 4, characterized in that The step S2.3.5 specifically includes the following steps: Step S2.3.5.1, the perception algorithm program obtains the matched data file, calculates and outputs the perception result, which is marked with a frame number according to the timestamp and includes the detection and tracking information of the target obstacle, and the detection and tracking information of the target obstacle includes the coordinate information of the center point and corner points of the obstacle detection frame in the vehicle body coordinate system, the size information of the obstacle detection frame and the yaw angle information of the obstacle detection frame; Step S2.3.5.2, the quantitative test software module obtains the corresponding perception result from the perception algorithm program according to the frame number, and retrieves the true value information from the read truth value file according to the above frame number to obtain the attribute true value corresponding to the frame number; Step S2.3.5.3, determine the center of the circle according to the true value of the attribute corresponding to the current frame number, determine the radius according to the threshold in the configured quantization test parameter, and obtain a circular area range; Step S2.3.5.4, determine whether the detection and tracking information of the target obstacle in the perception result corresponding to the current frame number is within the circular area. If so, it is determined to be a positive detection and the next step is executed; if not, it is determined to be a missed detection or a false detection and returns to step S2.3.5.2; Step S2.3.5.5, calculating the accuracy and recall rate based on the detection and tracking information of the target obstacle in the current perception result and the read truth file; Step S2.3.5.6, the quantitative test software module calculates the average precision mean of the current perception algorithm program, and calculates the average precision for each individual test category as the test result, and calculates the accuracy and recall rate based on the test results and generates a perception algorithm evaluation result file, and returns to step S2.

2.

6. The method according to claim 5, characterized in that The step S3 specifically comprises the following steps: Step S3.1, obtaining the current version of the perception algorithm evaluation result file; Step S3.2, obtaining the perception algorithm evaluation result files of the last five tests; Step S3.3, the visualization and result analysis module analyzes the accuracy of each test category and the pass rate of each version of the perception algorithm evaluation result file as the analysis result; Step S3.4, the visualization and result analysis module uses the analysis results to draw trend graphs of the quantitative indicators of each version and displays them for optimizing the perception algorithm program. Based on the optimized perception algorithm program, the optimized perception algorithm results are obtained and provided to the downstream prediction and planning module for collision risk prediction and path planning of vehicles in the mining area.

Images