Simulation environment test self-evaluation system and method based on automatic driving system

Abstract

The invention provides a simulation environment test self-evaluation system based on an automatic driving system, and the system comprises a data collection module which is used for collecting the test data of the automatic driving system in a simulation environment; the data preprocessing module is used for preprocessing the test data; the model construction training module is used for constructing a recurrent neural network model according to the processed data, and training the recurrent neural network model by taking a known simulation test result of the automatic driving system as label data; and the automatic evaluation module is used for inputting to-be-evaluated simulation test data of the automatic driving system into the trained recurrent neural network model, arranging the input data according to time steps to form a time sequence input, updating a hidden state, extracting key features from the updated hidden state, and obtaining a corresponding performance evaluation result. The technical problems that the prior art is easily influenced by subjective factors and the evaluation result is inaccurate are solved.

Description

Technical Field

[0001] The present application relates to the field of simulation testing technology, and in particular to a simulation environment testing self-assessment system and method based on an autonomous driving system. Background Art

[0002] With the rapid development of autonomous driving technology, the safety and reliability of autonomous driving systems have become the focus of industry attention. In order to verify the performance of autonomous driving systems, simulation testing has become an important means. Simulation testing tests autonomous driving systems by simulating various road and traffic scenarios in a virtual environment to evaluate their performance in a real environment. However, traditional simulation test result evaluation methods mostly rely on manual analysis, and the evaluation process is cumbersome, requiring manual detailed analysis of test data, which consumes a lot of time and energy; the evaluation results are highly subjective, and different evaluators may have different evaluation conclusions on the same test results; the evaluation results lack comparability, and it is difficult to objectively compare and rank the performance of different autonomous driving systems. At present, there are also some methods that attempt to automatically evaluate the simulation test results of autonomous driving systems through machine learning algorithms. These methods usually use traditional machine learning algorithms such as support vector machines and decision trees to analyze test data to extract key features and output evaluation results. However, these methods have limitations in processing time series data, and it is difficult to accurately capture the dynamic characteristics of the performance of autonomous driving systems over time. In addition, the simulation test data of autonomous driving systems usually contains rich time series information, such as vehicle driving trajectory, speed, acceleration, etc. When processing these data, traditional machine learning algorithms often fail to fully capture the temporal dependencies of the data, resulting in limited accuracy of the evaluation results. Summary of the invention

[0003] In view of this, the present invention provides a simulation environment test self-evaluation system and method based on an autonomous driving system to solve the technical problems that the prior art relies too much on manual analysis, is time-consuming and labor-intensive, is easily affected by subjective factors, and has inaccurate evaluation results.

[0004] The present invention provides a simulation environment test self-evaluation system based on an automatic driving system, characterized in that the system comprises: a data acquisition module, used to acquire test data of the automatic driving system in a simulation environment; a data preprocessing module, connected to the data acquisition module, used to preprocess the test data; a model building and training module, connected to the data preprocessing module, used to build a recurrent neural network model according to the processed data, and use known automatic driving system simulation test results as label data to train the recurrent neural network model; an automatic evaluation module, connected to the model building and training module, used to input the automatic driving system simulation test data to be evaluated into the trained recurrent neural network model, arrange the input data according to time steps to form a time series input, and perform a time series evaluation according to the function h. t =tanh(W ihxt +W hh h t-1 +b h )Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results.

[0005] Furthermore, the system also includes: a result output module, connected to the automatic evaluation module, for outputting the evaluation results in the form of charts and reports.

[0006] Furthermore, the system also includes: a model update optimization module, which is connected to the model construction training module and is used to adjust model parameters and add new known autonomous driving system simulation test results to train the recurrent neural network model.

[0007] Furthermore, the test data includes vehicle driving trajectory, speed, acceleration, and sensor data.

[0008] Furthermore, the collection method is to export through simulation software, or to obtain through real-time communication with a simulation environment.

[0009] Furthermore, the preprocessing includes data cleaning and data normalization.

[0010] Furthermore, the method for training the recurrent neural network model is specifically: optimizing the parameters of the recurrent neural network model by minimizing the loss function so that it learns the mapping relationship between the performance of the autonomous driving system and the simulation test data.

[0011] Furthermore, the output evaluation results include numerical values, trend graphs, and rankings of various performance indicators.

[0012] The present invention also provides a simulation environment test self-evaluation method based on an autonomous driving system, the method comprising: step 1, a data acquisition module acquires test data of the autonomous driving system in a simulation environment; step 2, a data preprocessing module performs pre-data cleaning and data normalization on the test data; step 3, a model construction training module constructs a recurrent neural network model according to the processed data, and uses known autonomous driving system simulation test results as label data to train the recurrent neural network model; step 4, an automated evaluation module inputs the autonomous driving system simulation test data to be evaluated into the trained recurrent neural network model, arranges the input data according to time steps to form a time series input, and performs a time series input according to the function h t =tanh(W ihxt +W hh h t-1 +b h )Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results.

[0013] Furthermore, the method also includes: step 5, the result output module outputs the evaluation results in the form of charts and reports; step 6, the model update optimization module adjusts the model parameters and adds new known autonomous driving system simulation test results to train the recurrent neural network model.

[0014] The present invention provides a self-evaluation system and method for simulation environment testing based on an autonomous driving system. The technical solution collects test data, performs preprocessing, constructs a recurrent neural network (RNN) model, and then uses known autonomous driving system simulation test results for training. The autonomous driving system simulation test to be evaluated is input into the trained model to obtain the evaluation result and complete the self-evaluation. The technical solution can effectively solve the problem of automated evaluation of autonomous driving system simulation test results, improve evaluation efficiency and accuracy, and provide strong support for the research and development and optimization of autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of a framework of a simulation environment test self-evaluation system based on an autonomous driving system provided by the present invention; Figure 2 It is a flow chart of a self-evaluation method for simulation environment testing based on an autonomous driving system provided by the present invention; Figure 3 It is a flowchart of another self-evaluation method for simulation environment testing based on an autonomous driving system provided by the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Embodiment 1: The present invention provides a simulation environment test self-evaluation system and method based on an automatic driving system. Figure 1 As shown, the system includes a data acquisition module, a data preprocessing module, a model building and training module, and an automated evaluation module. Figure 2 As shown, the method steps are as follows.

[0018] Step 1: The data acquisition module collects test data of the autonomous driving system in a simulation environment; Step 2: The data preprocessing module performs pre-data cleaning and data normalization on the test data; Step 3: The model building training module builds a recurrent neural network model based on the processed data, and uses the known autonomous driving system simulation test results as label data to train the recurrent neural network model; Step 4: The automated evaluation module inputs the simulated test data of the autonomous driving system to be evaluated into the trained recurrent neural network model, arranges the input data according to time steps, and forms a time series input. t =tanh(W ihxt +W hh h t-1 +b h )Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results.

[0019] The present invention provides a self-evaluation system and method for simulation environment testing based on an autonomous driving system. The technical solution collects test data, performs preprocessing, constructs a recurrent neural network model, and then trains it with known autonomous driving system simulation test results. The autonomous driving system simulation test to be evaluated is input into the trained model to obtain the evaluation result and complete the self-evaluation. The technical solution can effectively solve the problem of automated evaluation of the simulation test results of the autonomous driving system and improve the evaluation efficiency and accuracy.

[0020] Embodiment 2: The present invention provides a simulation environment test self-evaluation system and method based on an automatic driving system. Figure 1As shown, the system includes a data acquisition module, a data preprocessing module, a model building and training module, an automated evaluation module, a result output module, and a model update and optimization module. Figure 3 As shown, the method steps are as follows.

[0021] Step 1: The data acquisition module collects test data of the autonomous driving system in a simulation environment; When using simulation software to simulate the autonomous driving system, the simulation software will generate test data such as vehicle trajectory, speed, acceleration, sensor data, etc. Then, these data are exported to local storage through the data export function. In the data acquisition module, although the traditional data acquisition method can already meet basic needs, it is possible to consider using more efficient data acquisition technology, such as using higher-precision sensors or optimizing data acquisition algorithms to improve data accuracy and real-time performance. This technical improvement can not only improve the performance evaluation accuracy of the autonomous driving system, but also provide more reliable data support for subsequent manufacturing processes.

[0022] Step 2: The data preprocessing module preprocesses the test data; The preprocessing is data cleaning and data normalization. In the data cleaning process, invalid data, abnormal data and duplicate data are removed to ensure the accuracy and integrity of the data. In the data normalization process, the data is converted to the same scale range to facilitate subsequent data analysis and processing. Data preprocessing is one of the key steps in the present invention. The present invention adopts methods such as data cleaning and data normalization to ensure the accuracy and consistency of the data, but other data preprocessing methods can also be used to process the simulation test data of the autonomous driving system. For example, data enhancement technology can be used to increase the diversity and richness of data, or feature selection methods can be used to extract key features that have an important impact on performance evaluation. These methods can be selected and optimized according to specific application scenarios and requirements. In the data preprocessing module, although the existing data cleaning and data normalization technologies are relatively mature, more efficient preprocessing algorithms can still be explored to reduce the time and cost of data processing. For example, parallel computing or distributed computing technology is used to accelerate the data processing process, or more advanced machine learning algorithms are used to optimize the effects of data cleaning and normalization.

[0023] Step 3: The model building training module builds a recurrent neural network model based on the processed data, and uses the known autonomous driving system simulation test results as label data to train the recurrent neural network model; Based on the preprocessed data, a recurrent neural network model is constructed, that is, an RNN model. The RNN model is a neural network model that can capture the temporal dependency of data and is suitable for processing time series data. In the RNN model construction and training module, other types of neural network models, such as convolutional neural network (CNN) or deep belief network (DBN), can be used to construct the performance evaluation model of the autonomous driving system. It is also possible to consider using an integrated learning method to combine multiple different types of neural network models to improve the overall evaluation performance. In the analysis of the simulation test data of the autonomous driving system, the RNN model can use the temporal characteristics of data such as vehicle driving trajectory, speed, acceleration, etc. to extract key features that have an important impact on the performance evaluation of the autonomous driving system. The construction of the RNN model includes an input layer, a hidden layer, and an output layer. The input layer receives the preprocessed data, the hidden layer captures the temporal dependency in the data through a cyclic connection, and the output layer outputs the evaluation results of the performance of the autonomous driving system. When building the model, it is necessary to select a suitable RNN model type, such as a simple RNN, a long short-term memory network (LSTM), or a gated recurrent unit (GRU), etc., to adapt to different data and task requirements.

[0024] In general, the LSTM model is constructed as an implementation of the RNN model. The LSTM model has a strong ability to process time series data and can capture the time series dependencies in the data. During the model construction process, select the appropriate input layer, hidden layer, and output layer structure, and set the corresponding parameters. Use the known simulation test results of the autonomous driving system as label data to train the LSTM model. The label data includes the performance evaluation results obtained through manual analysis. During the training process, the parameters of the LSTM model are optimized by minimizing the loss function so that it can learn the mapping relationship between the performance of the autonomous driving system and the simulation test data.

[0025] Step 4: The automated evaluation module inputs the simulated test data of the autonomous driving system to be evaluated into the trained recurrent neural network model, arranges the input data according to time steps, and forms a time series input. t =tanh(W ihxt +W hh h t-1 +b h )Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results.

[0026] The simulated test data of the autonomous driving system to be evaluated is input into the trained LSTM model, and the model will output the corresponding performance evaluation results. The evaluation results include multiple performance indicators such as driving stability, obstacle avoidance ability, and compliance with traffic rules. These indicators are selected and defined according to actual needs to fully reflect the performance of the autonomous driving system.

[0027] Step 5, the result output module outputs the evaluation results in the form of charts and reports; The evaluation results are output in the form of charts, reports, etc., so that users can intuitively understand the performance of the autonomous driving system. The output results can include the values, trend charts, rankings, etc. of various performance indicators. Users can intuitively understand the performance of the autonomous driving system through these results, and conduct further analysis and mining to help users better understand the evaluation results and the performance characteristics of the autonomous driving system. At the same time, visualization tools are provided to allow users to further analyze and mine the evaluation results. Visualization tools can include data visualization software, interactive dashboards, etc., through which users can filter, sort, compare and other operations on the evaluation results to obtain more detailed information and insights.

[0028] Step 6: The model updating and optimization module adjusts the model parameters and adds new known autonomous driving system simulation test results to train the recurrent neural network model.

[0029] With the continuous development of autonomous driving technology and the continuous increase of simulation test data, the RNN model needs to be continuously optimized and updated to adapt to new data and task requirements. Model optimization can be performed by adjusting model parameters, increasing training data, introducing new features, etc. Model update means that when a new autonomous driving system or simulation test scenario appears, the RNN model needs to be retrained or adjusted to adapt to new data and task requirements. In the process of model optimization and updating, it is necessary to continuously collect and analyze new simulation test data, and adjust and improve the RNN model according to actual needs. At the same time, the optimized model needs to be verified and tested to ensure its accuracy and reliability.

[0030] The present invention provides a self-evaluation system and method for simulation environment testing based on an autonomous driving system. The technical solution collects test data, performs preprocessing, constructs a recurrent neural network model, and then trains it with known autonomous driving system simulation test results. The autonomous driving system simulation test to be evaluated is input into the trained model to obtain the evaluation result and complete the self-evaluation. The technical solution can effectively solve the problem of automated evaluation of the simulation test results of the autonomous driving system and improve the evaluation efficiency and accuracy.

[0031] Embodiment three: The present invention provides a simulation environment test self-evaluation system and method based on an automatic driving system. Figure 1 As shown, the system includes a data acquisition module, a data preprocessing module, a model building and training module, and an automated evaluation module. Figure 2 As shown, the method steps are as follows.

[0032] Step 1: The data acquisition module collects test data of the autonomous driving system in a simulation environment; First, it is necessary to collect test data of the autonomous driving system in the simulation environment. These data include vehicle trajectory, speed, acceleration, sensor data, etc., which can fully reflect the performance of the autonomous driving system in the simulation environment. The data can be directly exported through the simulation software or acquired through real-time communication with the simulation environment.

[0033] Step 2: The data preprocessing module performs pre-data cleaning and data normalization on the test data; Preprocess the collected data. The preprocessing process includes data cleaning and data normalization. Data cleaning mainly removes invalid data, abnormal data and duplicate data to ensure the accuracy and completeness of the data. Data normalization converts the data to the same scale to facilitate subsequent data analysis and processing.

[0034] Step 3: The model building training module builds a recurrent neural network model based on the processed data, and uses the known autonomous driving system simulation test results as label data to train the recurrent neural network model; The RNN model is trained using known autonomous driving system simulation test results as label data. Label data can be performance evaluation results obtained through manual analysis or other reliable reference data. The training process optimizes the parameters of the RNN model by minimizing the loss function so that it can learn the mapping relationship between the autonomous driving system performance and the simulation test data.

[0035] Step 4: The automated evaluation module inputs the simulated test data of the autonomous driving system to be evaluated into the trained recurrent neural network model, arranges the input data according to time steps, and forms a time series input. t =tanh(W ihxt +W hh h t-1 +b h )Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results.

[0036] The simulation test data of the autonomous driving system to be evaluated is input into the trained RNN model, and the model will output the corresponding performance evaluation results. The evaluation results may include multiple performance indicators, such as driving stability, obstacle avoidance ability, degree of compliance with traffic rules, etc. These indicators can be selected and defined according to actual needs to fully reflect the performance of the autonomous driving system. During the automatic evaluation process, the RNN model will extract key features based on the timing characteristics of the input data, and output the evaluation results based on the learned mapping relationship. Since the RNN model has a powerful timing data processing capability, it can more accurately reflect the performance of the autonomous driving system. In addition, the system and method provided by the present invention can be embedded in the research and development and production process of the autonomous driving system to achieve more efficient autonomous driving system performance evaluation and optimization.

[0037] The specific process of this step includes the following steps.

[0038] Step 41, input and initialize data; In the autonomous driving simulation evaluation system, the RNN model first receives preprocessed time series data. These data include vehicle location information (such as changes in longitude and latitude over time), speed information (speed values ​​at different times), sensor data (such as camera image sequences, radar distance data fluctuations over time), and traffic environment information (time series records of the location and motion status of surrounding vehicles and pedestrians). These data are arranged in time steps to form a time series input. A hidden state inside the RNN model is used to store information from previous time steps. When processing the data of the first time step, this hidden state will be initialized to a default value, usually a full zero vector or a value obtained by a random initialization method.

[0039] Step 42, processing data time by time step; For each time step, the RNN model takes the data of the current time step and the hidden state of the previous time step as input. Taking the vehicle speed data as an example, assuming the time series is v1, v2, v3, ..., when processing the second time step v2, the model will take v2 and the hidden state h1 obtained after processing v1 as input. The function of this process is: h t =tanh(W ihxt +W hh h t-1 +b h ), where h t is the hidden state at the current time step t, x t is the input data v2 of the current time step, W ih and W hh is the weight matrix of the model, b his the bias term, and tanh is the activation function, which is used to introduce nonlinear transformations so that the model can learn complex relationships. Through the above calculations, the RNN model updates the hidden state, and this new hidden state contains the input information of the current time step and the accumulated information of the previous time step. Continuing with the vehicle speed as an example, h2 contains the information of v1 and v2, and this information will continue to accumulate and update as the time step progresses. When processing the subsequent time step v3, this updated hidden state will be used together as input, and the above calculation process will be repeated to realize the transmission of information in the time series.

[0040] Step 43: extract key features.

[0041] In the process of processing data step by step, the RNN model extracts key features from time series data by continuously updating the above hidden states. For example, for a segment of vehicle driving trajectory data, the model will extract features such as the vehicle's acceleration and deceleration patterns, steering frequency and amplitude. These features are not based on data at a single time point, but rather integrate information over a period of time, reflecting the vehicle's driving habits and state changes.

[0042] The present invention provides a self-evaluation system and method for simulation environment testing based on an autonomous driving system. The technical solution collects test data, performs preprocessing, constructs a recurrent neural network model, and then trains it with known autonomous driving system simulation test results. The autonomous driving system simulation test to be evaluated is input into the trained model to obtain the evaluation result and complete the self-evaluation. The technical solution can effectively solve the problem of automated evaluation of the simulation test results of the autonomous driving system and improve the evaluation efficiency and accuracy.

[0043] In summary, the embodiment of the present invention provides a self-evaluation system and method for simulation environment testing based on an autonomous driving system. The technical solution greatly reduces the time and cost of manual analysis through an automated evaluation process. Users can quickly obtain the performance evaluation results of the autonomous driving system without tedious data analysis and processing, thereby improving the evaluation efficiency. The technical solution also uses the RNN algorithm to perform intelligent analysis on the simulation test data, which can more accurately reflect the performance of the autonomous driving system. The RNN model can capture the temporal dependencies in the data and extract the key features that have an important impact on the performance evaluation, thereby improving the accuracy of the evaluation. In addition, the technical solution realizes the objective comparison and ranking of the performance of different autonomous driving systems through unified evaluation standards and indicators. Users can intuitively compare and evaluate the performance of different autonomous driving systems based on the evaluation results, which provides strong support for the research and development and optimization of autonomous driving systems and enhances comparability. The method and system provided by the present invention can be expanded and customized according to actual needs. Users can select appropriate RNN model types, evaluation indicators, visualization tools, etc. according to different autonomous driving systems and simulation test scenarios to adapt to different needs and application scenarios.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A simulation environment test self-evaluation system based on an autonomous driving system, characterized in that: The system comprises: A data acquisition module is used to collect test data of the autonomous driving system in a simulation environment; A data preprocessing module, connected to the data acquisition module, is used to preprocess the test data; The model building training module is connected to the data preprocessing module and is used to build a recurrent neural network model based on the processed data and train the recurrent neural network model using the known simulation test results of the autonomous driving system as label data; The automated evaluation module is connected to the model building and training module and is used to input the simulated test data of the autonomous driving system to be evaluated into the trained recurrent neural network model, arrange the input data according to time steps, form a time series input, and calculate the time series input according to the function h. t =tanh(W ihxt +W hh h t-1 +b h ) Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results, where h t is the hidden state at the current time step t, x t is the input data v2 of the current time step, W ih and W hh is the weight matrix of the model, b h is the bias term and tanh is the activation function.

2. According to claim 1, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The system further comprises: a result output module, which is connected to the automatic evaluation module and is used for outputting the evaluation results in the form of charts and reports.

3. According to claim 1, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The system also includes: a model update optimization module, which is connected to the model construction training module and is used to adjust model parameters and add new known autonomous driving system simulation test results to train the recurrent neural network model.

4. According to claim 1, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The test data includes vehicle driving trajectory, speed, acceleration, and sensor data.

5. According to claim 1, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The acquisition method is to export through simulation software, or to obtain through real-time communication with the simulation environment.

6. According to claim 1, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The preprocessing includes data cleaning and data normalization.

7. The self-evaluation system for simulation environment testing based on the automatic driving system according to claim 1, characterized in that: The method for training the recurrent neural network model is specifically: optimizing the parameters of the recurrent neural network model by minimizing the loss function so that it learns the mapping relationship between the performance of the autonomous driving system and the simulation test data.

8. According to claim 2, a simulation environment test self-evaluation system based on an automatic driving system is characterized in that: The output evaluation results include the numerical values, trend graphs, and rankings of various performance indicators.

9. A method for using the simulation environment test self-evaluation system based on the automatic driving system according to claims 1-8, characterized in that: The method comprises: Step 1: The data acquisition module collects test data of the autonomous driving system in a simulation environment; Step 2: The data preprocessing module performs pre-data cleaning and data normalization on the test data; Step 3: The model building training module builds a recurrent neural network model based on the processed data, and uses the known autonomous driving system simulation test results as label data to train the recurrent neural network model; Step 4: The automated evaluation module inputs the simulated test data of the autonomous driving system to be evaluated into the trained recurrent neural network model, arranges the input data according to time steps, and forms a time series input. t =tanh(W ihxt +W hh h t-1 +b h ) Update the updated hidden state in the recurrent neural model, extract key features from the updated hidden state, and obtain the corresponding performance evaluation results, where h t is the hidden state at the current time step t, x t is the input data v2 of the current time step, W ih and W hh is the weight matrix of the model, b h is the bias term and tanh is the activation function.

10. The method for self-evaluation of a simulation environment test based on an autonomous driving system according to claim 9, characterized in that: The method further comprises: Step 5, the result output module outputs the evaluation results in the form of charts and reports; Step 6: The model updating and optimization module adjusts the model parameters and adds new known autonomous driving system simulation test results to train the recurrent neural network model.

Images