Automatic driving performance information acquisition method and device and computer storage medium

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving performance information acquisition method and device and a computer storage medium. The method comprises the following steps: preprocessing vehicle end sensing data to generate evaluation data with a sequence timestamp; obtaining a check point file in a model training process, and loading the check point file by using a deep learning framework to initialize an algorithm model to be evaluated; reasoning the evaluation data by using the initialized algorithm model to obtain a model output result, the model output result having a time sequence characteristic; and evaluating a model output result by using the truth value information of the vehicle end sensing data so as to obtain the automatic driving performance information of the algorithm model.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and more specifically to a method and device for acquiring autonomous driving performance information, and a computer storage medium. Background Art

[0002] In the field of autonomous driving, deep learning models can enable vehicles to perceive the surrounding environment based on sensor information in order to perform further control. In order to measure the performance of deep learning models, model evaluation of deep learning models is required.

[0003] Some solutions use the evaluator in the deep learning framework to evaluate the performance of the model on training and test data, and tune the model accordingly. However, the evaluation of the evaluator in the existing solutions is all performed on unordered data, and its evaluation results do not involve timing indicators. Therefore, it is not suitable for fields that need to pay attention to the timing performance capabilities of the model, such as autonomous driving. Some other solutions use the Model In the Loop (MIL) test method to evaluate the model. However, because the packaging process of the model in the loop test is too complicated, for algorithm models that require rapid iteration, it not only consumes a lot of time, but also reduces the efficiency of model evaluation.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to a person of ordinary skill in the art. Summary of the invention

[0005] In order to solve or at least alleviate one or more of the above problems, the following technical solutions are provided. The embodiments of the present application provide a method, device and computer storage medium for obtaining automatic driving performance information, which can support the high-frequency iteration requirements of the model and provide fast and accurate model evaluation results.

[0006] According to the first aspect of the present application, a method for obtaining automatic driving performance information is provided. The method comprises the following steps: preprocessing the vehicle-side perception data to generate evaluation data with sequential timestamps; obtaining the checkpoint file in the model training process, and loading the checkpoint file using the deep learning framework to initialize the algorithm model to be evaluated; using the initialized algorithm model to infer the evaluation data to obtain the model output result; using the true value information of the vehicle-side perception data to evaluate the model output result to obtain the automatic driving performance information of the algorithm model.

[0007] As an alternative or supplement to the above scheme, in a method for acquiring autonomous driving performance information according to an embodiment of the present application, preprocessing the vehicle-side perception data includes: sampling the vehicle-side perception data at a preset sampling frame rate; and temporally and spatially aligning data from different sensors according to a preset alignment logic, so that the temporal and spatial characteristics of the evaluation data are consistent with the actual operating environment of the vehicle.

[0008] As an alternative or supplement to the above scheme, in a method for acquiring autonomous driving performance information according to an embodiment of the present application, the preset sampling frame rate and the preset alignment logic are the same as the sampling frame rate and alignment logic used in the whole vehicle data playback simulation, and the evaluation logic used to evaluate the model output results is the same as the evaluation logic of the model-in-the-loop test of the algorithm model performed in the whole vehicle data playback simulation environment.

[0009] As an alternative or supplement to the above scheme, in a method for obtaining autonomous driving performance information according to an embodiment of the present application, the method also includes: comparing the autonomous driving performance information with the evaluation results of the model-in-the-loop test to verify the effectiveness and accuracy of the method for obtaining autonomous driving performance information.

[0010] As an alternative or supplement to the above solution, in a method for acquiring autonomous driving performance information according to an embodiment of the present application, the method also includes: converting the obtained model output result into an output in Proto format to adapt the evaluation logic.

[0011] As an alternative or supplement to the above solution, in a method for acquiring autonomous driving performance information according to an embodiment of the present application, reasoning is performed by a GPU and evaluation is performed by a CPU.

[0012] As an alternative or supplement to the above scheme, in a method for obtaining autonomous driving performance information according to an embodiment of the present application, the method also includes: decoding data units of evaluation data using an asynchronous decoder; caching the decoded data into a shared memory; and extracting data from the decoded data in chronological order through a sampler to form evaluation data for reasoning.

[0013] As an alternative or supplement to the above scheme, in a method for obtaining autonomous driving performance information according to an embodiment of the present application, the evaluation data is video data, the data unit is a group of pictures, the decoded data is frame data, and the evaluation data used for reasoning consists of continuous frame data.

[0014] As an alternative or supplement to the above scheme, in an autonomous driving performance information acquisition method according to an embodiment of the present application, using the true value information of the vehicle-side perception data to evaluate the model output result includes: measuring the performance of the model through time series indicators, wherein the time series indicators include one or more of the following: continuity, consistency, autocorrelation and cross-correlation, wherein continuity represents the smoothness of the model output result in time, consistency represents the stability and reliability of the model output result in the time dimension, autocorrelation represents the self-correlation of the model output result in the time series, and cross-correlation represents the correlation between the model output result and other time series data.

[0015] According to the second aspect of the present application, there is provided an apparatus for acquiring autonomous driving performance information, the apparatus comprising: a memory; a processor; and a computer program stored on the memory and executable on the processor, wherein the execution of the computer program enables any one of the methods for acquiring autonomous driving performance information described in the first aspect of the present application to be executed.

[0016] According to a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium includes instructions, and the instructions, when run, execute any one of the methods for obtaining autonomous driving performance information described in the first aspect of the present application.

[0017] According to one or more embodiments of the present application, the method for obtaining autonomous driving performance information performs model inference on evaluation data with sequential timestamps, so that the model output results have time series characteristics, and the evaluation of the model output results can quantify the performance of the model through time series indicators. The method also initializes the algorithm model by loading the checkpoint file using a deep learning framework to meet the high-frequency iteration requirements during the model development process. Compared with existing model evaluation methods, the method for obtaining autonomous driving performance information in the present application provides a fast and accurate model evaluation method suitable for the field of autonomous driving to help developers more effectively evaluate the autonomous driving performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or other aspects and advantages of the present disclosure will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar units are represented by the same reference numerals. In the accompanying drawings:

[0019] Figure 1 This is the overall process of model evaluation according to the embodiments of this application;

[0020] Figure 2 is a flowchart of a method 20 for obtaining automatic driving performance information according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of a method 30 for preprocessing vehicle-side sensing data according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of a method 40 for obtaining automatic driving performance information according to another embodiment of the present application;

[0023] Figure 5A It is a flow chart of the data loading and decoding process of a traditional evaluator when evaluating video data;

[0024] Figure 5B is a flowchart of a data loading and decoding process of a method for acquiring autonomous driving performance information when evaluating video data according to another embodiment of the present application;

[0025] Figure 6 It is a block diagram of an automatic driving performance information acquisition device 60 according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The description of the following specific embodiments is merely exemplary in nature and is not intended to limit the disclosed technology or the application and use of the disclosed technology. In addition, it is not intended to be bound by any express or implied theory presented in the aforementioned technical field, background technology or the following specific embodiments.

[0027] In the following detailed description of the embodiments, many specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it is apparent to one of ordinary skill in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0028] Terms such as "having" and "including" indicate that in addition to the units (modules) and steps that are directly and clearly stated in the specification and claims, the technical solution of the present disclosure does not exclude the situation where it has other units (modules) and steps that are not directly or clearly stated. Terms such as "first" and "second" do not indicate the order of units in terms of time, space, size, etc., but are only used to distinguish each unit. Moreover, the steps in this application are not limited to being implemented in the order of writing, but the steps written later can also be implemented at the same time as the steps written earlier, or before the steps written earlier.

[0029] As mentioned above, the evaluator in the deep learning framework can be used to evaluate the performance of the model on the training and test data, and the model can be tuned accordingly. The evaluation results of the evaluator involve a variety of evaluation indicators, such as the accuracy, precision, recall rate, F1 score, etc. of the model. It should be noted that the evaluation of the evaluator in the existing scheme is performed on unordered data, so its evaluation results do not involve time series indicators. However, in the field of autonomous driving, the focus is usually on the performance of the model for a period of time, so it is expected to evaluate continuous data and measure the performance of the model through time series indicators.

[0030] In addition, the model can also be evaluated by model-in-the-loop testing. Model-in-the-loop testing is a process of testing a single or integrated module in a model-based development environment. In model-in-the-loop testing, the algorithm model needs to be connected to the controlled object model to form a closed loop at the model level to verify whether the algorithm model meets the requirements. In model-in-the-loop testing, real data can be replayed through data playback simulation (Logsim) to simulate various possible scenarios or use cases, thereby more effectively verifying the safety, comfort and intelligence of the autonomous driving system. However, before performing model-in-the-loop testing, the model needs to be packaged and deployed. During the deployment process, it is necessary to ensure that all necessary dependencies are packaged correctly so that they can be executed correctly in the target environment. This may include library files, environment variables, and configuration files. These dependencies may not be related to the update of the model (for example, the adjustment of model parameters), but each time the model is iterated or updated, the packaging process needs to be repeated to ensure the smooth start of the test. Therefore, for algorithm models that require rapid iteration, this will take a lot of time and reduce the efficiency of model evaluation.

[0031] In this regard, this application proposes an improved model evaluation scheme to quickly and accurately evaluate the algorithm model in the field of autonomous driving. Figures 1 to 6 This improved model evaluation scheme is explained.

[0032] Figure 1 The figure shows an overall process 100 of model evaluation according to an embodiment of the present application.

[0033] like Figure 1As shown in the figure, after creating the evaluation task 102, the evaluation data for the evaluation model is called from the database 104. In addition, it is necessary to issue instructions to the resource library 106 to call the evaluation resources (such as GPU resources) that need to be used. Then, according to the workflow template 108, the overall process, stage and node of the evaluation (main workflow 110), as well as the more detailed tasks (sub-workflow 112) in the specific stage or task are determined. Next, the evaluation resources 114 assigned to the evaluation task in the resource library perform model reasoning 116 in parallel based on the evaluation data provided in the database 104. After completing the model reasoning 116, the model evaluation 118 is performed according to the model reasoning result, and a corresponding evaluation report 120 is generated.

[0034] Figure 2 It is a flowchart of the method 20 for obtaining the autonomous driving performance information according to the embodiment of the present application.

[0035] In step S202, the vehicle-side perception data is preprocessed to generate evaluation data with sequential timestamps.

[0036] The vehicle-side perception data is real data in the real environment collected by various sensors on the vehicle, or fused data generated based on real data in the real environment collected by various sensors. They can provide the vehicle with accurate environmental perception (for example, the location, shape, category and speed information of obstacles) and understanding of special scenes. The vehicle-side perception data includes video data collected by cameras, environmental sound data collected by microphones, and point cloud data collected by radar systems, etc. These data are temporally continuous data and have sequential timestamps. In addition, the vehicle-side perception data can also come from public databases, including high-quality traffic data collected at different locations, such as vehicle trajectory data, vehicle flow and occupancy data, etc. The vehicle-side perception data can be divided into multiple data sets according to the scene of the vehicle-side perception data. When you want to optimize the model for a specific scene, you can call the corresponding data set to simulate the data input in the scene.

[0037] As mentioned above, the method for obtaining autonomous driving performance information in this application focuses on the performance of the model for a period of time, and needs to evaluate continuous data through timing indicators. However, in the actual storage and processing process, the time sequence of the vehicle-side perception data may be disrupted due to various reasons (for example, network transmission, sensor failure, etc.). Therefore, it is necessary to ensure that the vehicle-side perception data used for evaluation has sequential timestamps through preprocessing in step S202. Exemplarily, the preprocessing in step S202 may include one or more of the following: data formatting - used to convert data in different formats into a unified format for subsequent processing and analysis, which may include converting video data into a frame sequence of images, converting audio data into a waveform representation, and converting point cloud data into a unified coordinate system, etc.; data cleaning - used to remove outliers and noise to improve the quality of evaluation data, which may include filtering, denoising, outlier detection and removal, etc.; data interception - in some cases, the original continuous data stream may be too long and not suitable for direct model evaluation, or in order to simulate specific scenarios and conditions, it is necessary to intercept fragments of a specific length from the original data; data sampling - used to ensure the consistency and processability of the data while reducing unnecessary computational burden; data alignment - since the data from different sensors may be out of sync in time, the timestamps of the data need to be aligned. In addition, the data can be transformed into a unified coordinate system to achieve spatial alignment; data labeling - used to assign a label to each sample in the data set to indicate the expected output.

[0038] Reference below Figure 3 , Figure 3 FIG. 3 is an exemplary flow chart of a method 30 for preprocessing vehicle-side sensing data according to an embodiment of the present application. Figure 3 As shown, the method 30 includes the above-mentioned data alignment step and data sampling step.

[0039] Specifically, in step S3021, the vehicle-side perception data is aligned in time and space according to the preset alignment logic. Step S3021 involves synchronizing and calibrating data from different sensors (such as cameras, radars, lidars, etc.) in time and space to ensure that the data can accurately reflect objects and environments in the real world.

[0040] Since different sensors usually have their own internal clocks, these clocks may have deviations, resulting in different timestamps at the same time when they collect data. This may cause problems in data fusion and analysis. For example, if the camera and radar respectively collect images and point cloud data of the same object, but the timestamps are inconsistent, it is impossible to accurately associate the image pixels with the point cloud. Therefore, it is necessary to time-align the vehicle-side perception data. The specific time alignment logic may include interpolation, extrapolation, or linear correction.

[0041] Spatial alignment refers to converting the perception data from different sensors into the same unified coordinate system so that they can be correctly fused and analyzed. The core of spatial alignment is to establish the conversion relationship between different sensor coordinate systems and apply this relationship to transform the data into a unified coordinate system. For example, the video data from the left, right, front and rear cameras of a vehicle are converted into a unified coordinate system to construct a panoramic road condition around the vehicle at a certain moment.

[0042] In addition, in step 3022, the vehicle-side perception data is sampled according to a preset sampling frame rate. In one possible implementation, a suitable sampling frame rate is first determined based on the model requirements and processing capabilities. For example, video data can be sampled at a rate of 30 frames per second (FPS), while point cloud data can be sampled at a rate of 10 times per second. In addition, in order to ensure that the sampling clocks of all sensors are synchronized, a master clock can be configured to trigger the sampling of all sensors to ensure the time consistency of the data.

[0043] return Figure 2 ,After preprocessing, in step S204, a checkpoint file in the model training process is obtained, and the checkpoint file is loaded using the deep learning framework to initialize the algorithm model to be evaluated.

[0044] In the deep learning framework, a checkpoint file is a file used to save the parameters (for example, weights and biases) and training status of the model at a specific point in time. Checkpoint files are lightweight file formats that can easily store a large number of model parameters, and when model training is interrupted or the model needs to be reloaded, the checkpoint file can be loaded to initialize or restore the model to continue model training or model reasoning. Compared with the two methods of updating a large number of model parameters and deploying the model in its entirety, the application of checkpoint files to the model evaluation process can achieve a faster and more flexible model initialization process. Through lightweight checkpoint files, model updates and changes can be reflected in an efficient manner, which is especially suitable for fast-iteration vehicle-side algorithm models.

[0045] Then, in step S206, the initialized algorithm model is used to infer the evaluation data to obtain a model output result.

[0046] Since these evaluation data have sequential timestamps, the model output results obtained by reasoning on these evaluation data have time series characteristics and can be measured by time series indicators.

[0047] In the traditional deep learning framework, the reasoning and evaluation of the evaluator are both run on the GPU, and the evaluation of the evaluator is for disordered random data. Therefore, the evaluation of the evaluator is usually to evaluate the reasoning result of the data after completing the model reasoning of a single data, and then to perform the reasoning of the next data, and then to evaluate the reasoning result of the next data, which will take up a lot of GPU time. It should be noted that the evaluation of the model reasoning result usually involves more indicator calculations, which are more suitable for CPU to perform, while model reasoning is more suitable for GPU to perform. Therefore, when the evaluator is evaluated on the GPU, the efficiency of the overall model evaluation is reduced. In view of this, according to some embodiments of the present application, before step S206, a workflow can be used to split the reasoning and evaluation tasks, wherein the reasoning task is completed by the GPU and the evaluation task is completed by the central processing unit (CPU), thereby avoiding the evaluation task from occupying the use of the GPU.

[0048] Then, in step S208, the model output results are evaluated using the true value information of the vehicle-side perception data to obtain the autonomous driving performance information of the algorithm model. The true value information is the real label or result of the data, which is used as a benchmark for evaluating, verifying and calibrating the data. When the model reasoning result of the data is consistent with the true value information, it indicates that the model reasoning is accurate. Autonomous driving performance information refers to the result of quantitative and qualitative evaluation of the autonomous driving related performance of the algorithm model. It is obtained by comparing the model output results with the true value information of the vehicle-side perception data, and the specific information it contains depends on the specific goals and tasks of the algorithm model. Exemplarily, the autonomous driving performance information can be measured by one or more of the following evaluation indicators: target detection indicators, including precision, recall, F1 value, average precision mean, etc., which are used to evaluate the accuracy of the algorithm in detecting targets such as pedestrians, vehicles and traffic signals; positioning accuracy, which is used to measure the accuracy of the algorithm in estimating the vehicle's position and direction (posture); classification accuracy, if the algorithm involves classifying objects or their attributes (for example, the color of traffic lights), the classification accuracy is used to measure the accuracy of the classification results; trajectory accuracy, which is used to measure the degree of match between the predicted trajectory output by the algorithm and the actual driving trajectory of the vehicle.

[0049] In step S208, the model output result obtained by inference is compared with the true value information of the vehicle-side perception data, and the autonomous driving performance of the model is quantified by calculating the corresponding evaluation indicators (for example, target detection indicators, classification accuracy, positioning accuracy, etc.). With these evaluation indicators, developers can verify whether the current version of the model meets the design requirements for autonomous driving performance.

[0050] As described above, the model output results inferred from these evaluation data have time series characteristics and can be measured by time series indicators. Exemplarily, time series indicators may include one or more of the following: continuity, consistency, autocorrelation, and mutual correlation. Among them, continuity is used to evaluate whether the model output results are smooth and continuous in time. For example, in an autonomous driving scenario, if the model predicts that there are mutations or unreasonable jumps between adjacent time points in the vehicle motion trajectory, the continuity index will be poor, which can be obtained by the slope, rate of change, or smoothing function of the model output results in time. Consistency is used to evaluate the stability and reliability of the model output results in the time dimension. For example, if the algorithm model has large fluctuations in the prediction results and lacks consistency when facing the same input data, the indicator will be poor, which can be evaluated by calculating the variance or standard deviation of the predicted value, or by comparing the statistical characteristics of the prediction results in different time periods to judge the consistency. Autocorrelation is used to evaluate the self-correlation of the model output results in the time series. For example, if the model output results are highly correlated between adjacent time points, the autocorrelation is high, which can be used to determine whether the model captures the potential periodicity or trend. Cross-correlation is used to evaluate the correlation between the model output results and other time series data (such as sensor data) in time series. For example, if the model's prediction results are highly correlated with the radar's obstacle detection results in time, the cross-correlation is high, which can be used to verify whether the model prediction results are consistent with the real environment.

[0051] Figure 4 It is a flowchart of a method 40 for acquiring autonomous driving performance information according to another embodiment of the present application.

[0052] exist Figure 4 In the embodiment, step S402 includes preprocessing the vehicle-side sensing data to generate evaluation data with sequential timestamps. This step can be performed in the same manner as Figure 2 The process is implemented in the same manner as step S202 in the above process and will not be described in detail here.

[0053] exist Figure 4 Step S403 includes decoding the data units of the evaluation data using an asynchronous decoder, and caching the decoded data into a shared memory, so that the sampler can extract data from the decoded data in time sequence and compose the evaluation data for inference.

[0054] In the deep learning framework, the evaluator needs to load the evaluation data before performing model inference on the evaluation data. For example, the evaluation data can be an image frame in a video, and its data unit is a group of pictures (GOP). GOP refers to a group of continuous frames in a video sequence. GOP usually starts with a key frame, followed by a certain number of forward prediction frames and bidirectional prediction frames. In the process of model evaluation for video data, it is necessary to decode GOP to obtain each frame in GOP.

[0055] Figure 5A FIG. 5 is a flow chart of a data loading and decoding process 510 of a conventional evaluator when evaluating video data. Figure 5A As shown in the figure, during the loading process of the model inference of the unordered data, the data loader 5102 calls the sampler 5104 to obtain a certain number of random frame indexes. Then, the sampler 5104 can divide the number into several batches and process each batch in parallel to speed up the sampling speed ( Figure 5A Only two batches are shown in the figure), and then the dataset workers 5106-1, 5106-2 execute the loop of loading videos 5108-1, 5108-2, decoding GOP and selecting frames 5110-1, 5110-2 according to the corresponding batch size (the number of frames required in the batch), until all frames in the batch have been selected, and finally batches 5112-1, 5112-2 are generated and recombined to output sample data 5114. Because the frames within the same batch are random frames, these frames may come from the same or different GOPs. At the same time, different frames within the same GOP may be selected by multiple batches, and these selections may be separated by a long time. Therefore, in this case, there may be a situation where the same GOP is repeatedly decoded to obtain different frames, which increases the time required for the loading process.

[0056] The evaluation method of the present application is to infer the temporally continuous data, so if a frame in a GOP is selected, other temporally continuous frames in the same GOP will also be selected soon (in the same batch or in the next batch). Figure 5B FIG. 5 is a flow chart of a data loading and decoding process 520 of the method for obtaining automatic driving performance information when evaluating video data according to an embodiment of the present application. Figure 5BAs shown in the figure, the data loader 5202 loads the video 5204 and calls the decoder to decode the GOP 5206-1 and 5206-2 in the video, and after decoding the GOP, caches the frames in the decoded GOP into the shared memory 5208. At the same time, the data loader 5202 calls the sampler 5210 to directly select frames 5212-1 and 5212-2 from the frames cached in the shared memory to extract data in the order of timestamps, thereby avoiding the overhead of repeated decoding to further speed up the speed. Figure 5B In the illustrated data loading and decoding process 520 , the measurement data is video data, the data unit is GOP, the decoded data is frame data, and the measurement data used for inference consists of continuous frame data.

[0057] return Figure 4 Step S404 includes obtaining a checkpoint file during the model training process and loading the checkpoint file using a deep learning framework to initialize the algorithm model to be evaluated. Figure 2 The process is implemented in the same manner as step S204 in the above process and will not be described in detail here.

[0058] Step S406 includes using the initialized algorithm model to infer the evaluation data to obtain the model output result. Figure 2 The process is implemented in the same manner as step S206 in the above process and will not be described in detail here.

[0059] Step S407 includes converting the model output result obtained in step S406 into Proto format. Proto format is a lightweight and efficient structured data storage format. Due to its high efficiency, cross-platform and cross-language characteristics, it has been widely used in many fields. The purpose of this step will be described below.

[0060] Step S408 includes evaluating the model output results using the true value information of the vehicle-side perception data. Figure 2 Additionally or alternatively, after step S408, the evaluation results can be visualized using a visualization code to generate an evaluation report to provide a more intuitive display of the model performance.

[0061] Step S409 includes comparing the acquired autonomous driving performance information with the evaluation results of the model-in-the-loop test to verify the effectiveness and accuracy of the method for acquiring the autonomous driving performance information. As mentioned above, the present application aims to provide a fast and accurate model evaluation method, which is expected to have the same evaluation results as the evaluation results of the model-in-the-loop test in the whole vehicle data playback simulation, but faster than the model-in-the-loop test. Therefore, in step S409, by verifying the results of the two evaluation methods, the effectiveness and accuracy of the evaluation using this method can be ensured. At the same time, in order to obtain evaluation results that are consistent with the model-in-the-loop test as much as possible, Figure 3 The preset sampling frame rate in step S3021 and the Figure 3 The preset alignment logic in step S3022 should be the same as the sampling frame rate and alignment logic of the whole vehicle data playback simulation, and the evaluation logic used to evaluate the model output results should be the same as the evaluation logic of the model-in-the-loop test for the algorithm model executed in the whole vehicle data playback simulation, so as to minimize interference factors. Since the evaluation logic is the same as the evaluation logic of the model-in-the-loop test, in order to adapt the evaluation logic, in step S407, the model reasoning result is converted to Proto for use as an intermediate storage data format to further narrow the difference with the model-in-the-loop test method. In some embodiments, step S409 can also be executed before the automatic driving performance information acquisition method of the present application is put into use. For example, before the automatic driving performance information acquisition method of the present application is formally used for model evaluation, the reliability of the method can be verified by step S409. After the verification is completed, the method is used to replace the model-in-the-loop test in the process of automatic driving model development, and after it is put into use, step S409 may no longer be executed.

[0062] According to one or more embodiments of the present application, the method 20, 40 performs model reasoning on the evaluation data with sequential timestamps, so that the evaluation of the model output results can quantify the performance of the model through timing indicators. In addition, the method 20, 40 realizes the rapid and accurate deployment of the evaluation model by applying the checkpoint file to meet the high-frequency iteration requirements in the model development process. In addition, the method 20, 40 ensures that the data used for reasoning is continuous in time and has consistent specifications and quality by preprocessing the vehicle-side perception data. In addition, the method 20, 40 according to one or more embodiments of the present application avoids the time consumption of repeated decoding by asynchronously decoding and caching the data, and avoids excessive GPU occupation by splitting the reasoning and evaluation tasks, thereby further improving the efficiency of the model evaluation. In addition, the method 20, 40 according to one or more embodiments of the present application further ensures the accuracy of the method for obtaining the autonomous driving performance information by using the same sampling frame rate, alignment logic and evaluation logic as the whole vehicle data playback simulation, and by applying the Proto format.

[0063] Figure 6 6 is a schematic block diagram of an automatic driving performance information acquisition device 60 according to an embodiment of the present application. The device includes a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620. The execution of the computer program 630 enables the automatic driving performance information acquisition method 20 or 40 to be executed.

[0064] Software (such as program code and / or data) according to the present disclosure may be stored on one or more computer storage media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein may be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0065] The embodiments and examples presented herein are provided to best illustrate embodiments according to the present disclosure and its specific applications, and thereby enable those skilled in the art to implement and use the present disclosure. However, those skilled in the art will appreciate that the above description and examples are provided only for ease of illustration and example. The presented description is not intended to cover all aspects of the present disclosure or to limit the present disclosure to the precise form disclosed.

Claims

1. A method for obtaining automatic driving performance information, characterized in that: The method comprises the following steps: Preprocess the vehicle-side perception data to generate evaluation data with sequential timestamps; Obtain the checkpoint file during the model training process, and use the deep learning framework to load the checkpoint file to initialize the algorithm model to be evaluated; Using the initialized algorithm model to infer the evaluation data to obtain a model output result, wherein the model output result has a time series characteristic; and The true value information of the vehicle-side perception data is used to evaluate the model output results to obtain the autonomous driving performance information of the algorithm model.

2. The method of claim 1, wherein: Preprocessing of vehicle-side perception data includes: Sampling the vehicle-side perception data according to a preset sampling frame rate; and The data from different sensors are aligned in time and space according to the preset alignment logic, so that the time and space characteristics of the evaluation data are consistent with the actual operating environment of the vehicle.

3. The method of claim 2, wherein: The preset sampling frame rate and the preset alignment logic are the same as the sampling frame rate and alignment logic used in the whole vehicle data playback simulation, and the evaluation logic used to evaluate the model output results is the same as the evaluation logic of the model-in-the-loop test of the algorithm model performed in the whole vehicle data playback simulation environment.

4. The method of claim 3, further comprising: The autonomous driving performance information is compared with the evaluation results of the model-in-the-loop test to verify the effectiveness and accuracy of the method for obtaining the autonomous driving performance information.

5. The method of claim 3, further comprising: The obtained model output result is converted into an output in Proto format to adapt to the evaluation logic.

6. The method of claim 1, wherein: The reasoning is performed by the GPU, and the evaluation is performed by the CPU.

7. The method of claim 1, further comprising: Decoding the data unit of the evaluation data using an asynchronous decoder; Cache the decoded data into a shared memory; as well as A sampler is used to extract data from the decoded data in time sequence and compose the evaluation data for inference.

8. The method of claim 7, wherein: The measurement data is video data, the data unit is a picture group, the decoded data is frame data, and the measurement data for inference consists of continuous frame data.

9. The method of claim 1, wherein: Using the true value information of the vehicle-side perception data to evaluate the model output result includes: The performance of the model is measured by time series indicators, wherein the time series indicators include one or more of the following: continuity, consistency, autocorrelation and cross-correlation, wherein the continuity characterizes the smoothness of the model output results in time, the consistency characterizes the stability and reliability of the model output results in the time dimension, the autocorrelation characterizes the self-correlation of the model output results in the time series, and the cross-correlation characterizes the correlation between the model output results and other time series data.

10. An automatic driving performance information acquisition device, characterized in that: The device includes: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein the execution of the computer program enables the method for acquiring autonomous driving performance information according to any one of claims 1-9 to be executed.

11. A computer storage medium, characterized in that: The computer storage medium includes instructions, which, when run, execute the method for obtaining automatic driving performance information according to any one of claims 1-9.

Images