Automatic driving-oriented key scene generation method and device, and computer device

Abstract

The application discloses an automatic driving-oriented key scene generation method and device and computer equipment, and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring scene description information corresponding to a test task of automatic driving test; acquiring a test area conforming to the scene description information from a scene map, and setting a driving route conforming to the test task for a measured object in the test area; extracting atomic actions of scene participants from the scene description information; initializing an experience replay pool corresponding to a reinforcement learning model, and constructing sample data based on the scene description information, the test area and the driving route to store the sample data to the experience replay pool; extracting sample data from the experience replay pool to update the reinforcement learning model, and acquiring a determined parameter value of the atomic action through the updated reinforcement learning model; and generating a key scene for realizing the test task based on the determined parameter values of all atomic actions. The application can improve the generation efficiency of the key scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and particularly relates to a key scene generation method and device for automatic driving and a computer device. BACKGROUND

[0002] The scene-based test method is a key link in the engineering of an automatic driving system. Due to the complexity of a driving environment and a driving task, the number of driving scenes that the automatic driving system may encounter is infinite. Therefore, it is necessary to identify scenes and generate key scenes with high risks (vehicle collision), which is particularly important for the design, confirmation and verification of the automatic driving system and its safety basis, and is related to the efficiency, safety and quality of the research and development of automatic driving.

[0003] Key scene generation aims to find specific key scenes from given logical scenes, wherein the logical scene is a state space level scene with parameter definition and value range, and the specific scene is a specificization of the logical scene with specific parameter values. Therefore, how to find specific critical parameters from an infinite parameter space becomes a core challenge faced by the key scene generation technology. The key scene generation method adopted in the related technology is based on random testing or combinatorial testing. The former directly samples from the parameter range of the logical scene to obtain the parameters required for the action of the participants in the specific scene through random testing; the latter generates multiple parameter combinations that cover the parameter range as much as possible through combinatorial testing, and then simulates the execution of the specific scene based on the parameter combinations in the simulation software to observe whether a high-risk accident occurs. If a high-risk accident occurs, the specific scene formed by the parameter combination is labeled as a key scene, otherwise, a new parameter combination is obtained by continuing sampling.

[0004] However, due to the small number of key scenes, the random testing or combinatorial testing method in the related technology usually needs to sample nearly one hundred times to obtain a key scene, which is low in generation efficiency. SUMMARY

[0005] To solve the above problems, the present application provides a key scene generation method, device and computer equipment for automatic driving, which can improve the generation efficiency of key scenes and realize the rapid generation of key scenes.

[0006] To achieve the above purpose, the technical content of the present application includes:

[0007] A key scene generation method for automatic driving, the method comprising:

[0008] Obtaining scene description information corresponding to a test task of automatic driving test, the scene description information being used to indicate information related to a scene occurrence area and a scene participant;

[0009] acquire a test area conforming to the scene description information from a scene map, and set a driving route conforming to the test task for the measured object in the test area;

[0010] extract an atomic action of the scene participant from the scene description information;

[0011] initialize an experience replay pool corresponding to the reinforcement learning model, and construct sample data based on the scene description information, the test area, and the driving route to store to the experience replay pool;

[0012] update the reinforcement learning model by extracting sample data from the experience replay pool, and obtain a determined parameter value of the atomic action through the updated reinforcement learning model;

[0013] generate a key scene for implementing the test task based on the determined parameter values of all atomic actions, the key scene being a test scene in which the measured object and the scene participant have the maximum probability of collision.

[0014] Optionally, the constructing sample data based on the scene description information, the test area, and the driving route to store to the experience replay pool comprises: for atomic action a i , obtaining a training parameter value of the atomic action a i through a random generation function, the i being a positive integer; accessing an automatic driving simulation test platform having the measured object, and processing the training parameter value of the environment state s i and the training parameter value of the atomic action a i according to the test area and the driving route to obtain a training parameter value of the environment state s i+1 , a training reward corresponding to the atomic action a i , and a collision state; wherein the training parameter value of the initial environment state s1 is obtained based on the scene description information; based on the i, the training parameter value of the environment state s i , the training parameter value of the atomic action a i , the training parameter value of the environment state s i+1 , the training reward corresponding to the atomic action a i , and the collision state, the sample data corresponding to the atomic action a i is constructed; let i=i+1, and repeat the above steps for atomic action a i . iThe step of setting the training parameter values ​​is executed again until the sample data has been constructed for all atomic actions; the process of constructing the sample data for all atomic actions is repeated for u rounds, and all sample data constructed in the u rounds is stored in the experience replay pool.

[0015] Optionally, the total number of atomic actions is n, where n is a positive integer greater than or equal to i; the atomic action a is obtained through a random generation function. i The training parameter values ​​include: values ​​generated from the atomic action a by the random generation function. i Randomly sample within the parameter range, and add a padding mask and a timing mask to the sampling results to obtain the atomic action a. i The training parameter values; wherein, the padding mask is used to fill the sampling result with the parameter values ​​of n-1 atomic actions, so that the atomic action a i The training parameter values ​​are represented by an action space of length n; the temporal mask is used to define the execution timing of the atomic action a within the action space. i The parameter values ​​of the subsequent ni atomic actions are set to zero, and the execution timing in the action space is set to the atomic action a. i The parameter values ​​of the previous i-1 atomic actions are set to the training parameter values ​​of the corresponding atomic actions.

[0016] Optionally, the atomic action a i The corresponding training reward is the reward obtained by the reinforcement learning model during training. The calculation process includes: for each scene participant, calculating the longitudinal collision time between the tested object and the corresponding scene participant based on the difference between the ordinate of the tested object and the ordinate of the corresponding scene participant, and the difference between the longitudinal velocity of the tested object and the longitudinal velocity of the corresponding scene participant; for each scene participant, calculating the lateral collision time between the tested object and the corresponding scene participant based on the difference between the abscissa of the tested object and the abscissa of the corresponding scene participant, and the difference between the lateral velocity of the tested object and the lateral velocity of the corresponding scene participant; calculating the risk factor between the tested object and each scene participant based on the longitudinal collision time and the lateral collision time; and taking the maximum value of the risk factor between the tested object and all scene participants and the sum of the additional reward as the reward obtained by the reinforcement learning model.

[0017] Optionally, the calculating the risk factor between the measured object and each scene participant based on the longitudinal collision time and the lateral collision time comprises: for each scene participant, if the longitudinal collision time is greater than zero or the lateral collision time is greater than zero, taking the reciprocal of the maximum of the longitudinal collision time and the lateral collision time as the risk factor between the measured object and the corresponding scene participant; for each scene participant, if the longitudinal collision time is less than or equal to zero and the lateral collision time is less than or equal to zero, taking the absolute value of the difference between the longitudinal collision time and the lateral collision time and the reciprocal of the difference between the minimum of the longitudinal collision time and the lateral collision time as the risk factor between the measured object and the corresponding scene participant.

[0018] Optionally, the experience replay pool comprises a positive sample buffer and a negative sample buffer; and the storing all sample data constructed in the u rounds into the experience replay pool comprises: classifying all sample data constructed in the u rounds according to the collision state to obtain at least one positive sample and at least one negative sample; storing the at least one positive sample in the positive sample buffer and storing the at least one negative sample in the negative sample buffer; wherein the positive sample refers to sample data in which the measured object collides with the scene participant, and the negative sample refers to sample data in which the measured object does not collide with the scene participant.

[0019] Optionally, the reinforcement learning model comprises an actor network, a target network corresponding to the actor network, a first critic network, a target network corresponding to the first critic network, a second critic network, and a target network corresponding to the second critic network; wherein the actor network is configured to process a parameter value of an input current environment state to output a parameter value of a current atomic action; the first critic network and the second critic network are configured to process input sample data to respectively output a first evaluation value and a second evaluation value corresponding to a respective atomic action; and the target network is configured to stabilize an expected return of the reinforcement learning model.

[0020] Optionally, the updating the reinforcement learning model by extracting sample data from the experience replay pool and obtaining the determined parameter value of the atomic action through the updated reinforcement learning model comprises: for each atomic action a iextract a target number of sample data from the experience replay pool for updating the reinforcement learning model; process, by the first critic network and the second critic network, the training parameter value of the corresponding environment state and the training parameter value of the corresponding atomic action in each sample data to respectively obtain the first evaluation value and the second evaluation value corresponding to the corresponding atomic action; process, by the target network of the actor network, the training parameter value of the next environment state of the corresponding environment state in each sample data to obtain the predicted parameter value of the next atomic action of the corresponding atomic action; process, by the target network of the first critic network and the target network of the second critic network, the training parameter value of the next environment state of the corresponding environment state and the predicted parameter value of the next atomic action of the corresponding atomic action in each sample data to respectively obtain the first evaluation value and the second evaluation value corresponding to the next atomic action of the corresponding atomic action; construct a loss function of the first critic network based on the rewards corresponding to the corresponding atomic actions, the first evaluation values corresponding to the corresponding atomic actions, and the first evaluation values corresponding to the next atomic actions of the corresponding atomic actions in all sample data; construct a loss function of the second critic network based on the rewards corresponding to the corresponding atomic actions, the second evaluation values corresponding to the corresponding atomic actions, and the second evaluation values corresponding to the next atomic actions of the corresponding atomic actions in all sample data; respectively perform reverse gradient propagation according to the loss function of the first critic network and the loss function of the second critic network to update the parameters of the first critic network and the second critic network; construct a loss function of the actor network based on the first evaluation values and the second evaluation values corresponding to the corresponding atomic actions in all sample data; perform reverse gradient propagation according to the loss function of the actor network to update the parameters of the actor network; process the determined parameter value of the environment state s i by the actor network to obtain the determined parameter value of the atomic action a i ; wherein the process of obtaining the determined parameter value of the environment state s i includes processing the determined parameter value of the environment state s i-1 and the determined parameter value of the atomic action a i-1 by the autonomous driving simulation test platform according to the test area and the driving route to obtain the determined parameter value of the environment state s i ; the determined parameter value of the initial environment state s1 is obtained based on the scene description information, and the determined parameter value of the initial atomic action a1 is obtained by processing the determined parameter value of the input environment state s1 by the actor network.

[0021] Optionally, the experience replay pool comprises a positive sample buffer and a negative sample buffer; the extracting the target number of sample data from the experience replay pool for updating the reinforcement learning model comprises: extracting a first number of positive samples from the positive sample buffer and extracting a second number of negative samples from the negative sample buffer, the target number of sample data comprising the first number of positive samples and the second number of negative samples, the sum of the first number and the second number being equal to the target number; wherein the positive sample refers to sample data in which the measured object collides with the scene participant, and the negative sample refers to sample data in which the measured object does not collide with the scene participant.

[0022] Optionally, the parameters of the target network of the first critic network and the parameters of the target network of the second critic network are updated based on the parameters of the critic network with the smaller output evaluation value in the first critic network and the second critic network; and the parameter update frequency of the target network of the first critic network and the target network of the second critic network is lower than the parameter update frequency of the first critic network and the second critic network; the parameters of the target network of the actor network are updated based on the parameters of the actor network; and the parameter update frequency of the target network of the actor network is lower than the parameter update frequency of the actor network, and the parameter update frequency of the actor network is lower than the parameter update frequency of the first critic network and the second critic network.

[0023] Optionally, the processing of the determined parameter value of the environment state s i by the actor network to obtain the determined parameter value of the atomic action a i further comprises: processing the determined parameter value of the environment state s i and the determined parameter value of the atomic action a i by the automatic driving simulation test platform according to the test area and the driving route to obtain the determined parameter value of the environment state s i+1 , the training reward corresponding to the atomic action a i , and the collision state; based on the i, the determined parameter value of the environment state s i , the determined parameter value of the atomic action a i , the determined parameter value of the environment state s i+1 , the training reward corresponding to the atomic action a i , and the collision state, constructing sample data corresponding to the atomic action a i ; storing the sample data corresponding to the atomic action a i to the experience replay pool to update the experience replay pool.

[0024] Optionally, the parameter value of the atomic action comprises a speed of the scene participant, a target position of the scene participant; the parameter value of the environment state comprises a lateral distance between the measured object and the scene participant, a longitudinal distance between the measured object and the scene participant, a running distance of the measured object, a running distance of the scene participant, and a collision state between the measured object and the scene participant.

[0025] A key scene generation device for automatic driving, the device comprising:

[0026] An information acquisition module is configured to acquire scene description information corresponding to a test task of automatic driving testing, the scene description information being used to indicate information related to a scene occurrence area and a scene participant;

[0027] A running setting module is configured to acquire a test area conforming to the scene description information from a scene map, and set a running route conforming to the test task for a measured object in the test area;

[0028] An action extraction module is configured to extract an atomic action of the scene participant from the scene description information;

[0029] A sample construction module is configured to initialize an experience replay pool corresponding to a reinforcement learning model, and construct sample data based on the scene description information, the test area, and the running route, and store the sample data to the experience replay pool;

[0030] A parameter acquisition module is configured to extract sample data from the experience replay pool to update the reinforcement learning model, and acquire a determined parameter value of the atomic action through the updated reinforcement learning model;

[0031] A scene generation module is configured to generate a key scene for implementing the test task based on the determined parameter values of all atomic actions, the key scene being a test scene with the largest probability of collision between the measured object and the scene participant.

[0032] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the key scene generation method for automatic driving.

[0033] A computer device comprising a memory and a processor, the memory having a computer program stored therein, and the processor being configured to run the computer program to execute the key scene generation method for automatic driving.

[0034] A computer program product, when the computer program product runs on a computer device, causes the computer device to execute the automatic driving-oriented key scene generation method.

[0035] Compared with the related art, the present application has at least the following advantages:

[0036] 1. The test scene is disassembled into atomic action sequences, the timing relationship between the atomic actions is modeled, and the parameter values of each atomic action in the key scene are sampled through the reinforcement learning model.

[0037] The present application first extracts the atomic actions of the scene participants from the scene description information, and then disassembles the test scene into atomic action sequences of the scene participants according to the timing logic relationship between the atomic actions. Then a reinforcement learning model is constructed, the atomic action sequence is modeled as the output of the reinforcement learning model, and the action parameters of the scene participants are set as the actions of the reinforcement learning model, and the state parameters of the traffic environment are set as the states of the reinforcement learning model. Then the number of atomic actions in the atomic action sequence is set as the number of steps executed by the reinforcement learning model, and in each step, the reinforcement learning model samples a parameter value of a corresponding atomic action in combination with the timing mask. Based on this, on the one hand, the present application samples the parameter values of the atomic actions through the reinforcement learning model, effectively avoiding the huge test verification amount caused by random parameter sampling, and improving the generation efficiency of the key scene; on the other hand, the present application fits the timing logic relationship between the atomic actions through the timing mask, so that the parameter sampling of the current atomic action is only affected by the previously executed atomic action, ensuring the correctness of the timing logic relationship.

[0038] 2. A reward function is constructed based on the collision information between the measured object and the scene participants, guiding the reinforcement learning model to explore the parameter space of the key scene.

[0039] In the present application, the collision information includes two parts: collision risk information and collision behavior information. The collision risk information is used to indicate the risk degree of collision between the measured object and the scene participants, and the collision behavior information is used to indicate the collision or non-collision between the measured object and the scene participants. Among them, the risk degree (risk factor) is measured by the lateral collision time and the longitudinal collision time, and the collision or non-collision is measured by the additional reward. When designing the reward function, the greater the risk of collision between the measured object and the scene participants, or the greater the reward obtained by the reinforcement learning model when the measured object and the scene participants collide. The present application guides the reinforcement learning model to tend to high-risk rewards and explore the parameter space of the key scene where the collision is most likely to occur, which is more in line with the generation principle of the key scene, and effectively improves the generation efficiency of the key scene.

[0040] 3. Optimizing the design of the experience replay pool to ensure that the distribution of the extracted sample data is balanced, so that the collision information is effectively learned by the reinforcement learning model.

[0041] In the related art, the experience replay pool only has one buffer, and all sample data is stored in the one buffer. When extracting sample data, random extraction is performed from the buffer, which can easily lead to unbalanced distribution of collision samples and non-collision samples. The present application divides the experience replay pool into a positive sample buffer and a negative sample buffer. When storing sample data, if the simulation process of the sample data occurs collision, the sample data is regarded as a positive sample and is placed in the positive sample buffer, otherwise the sample data is regarded as a negative sample and is placed in the negative sample buffer. When a certain amount of sample data is extracted from the experience replay pool for subsequent training of the reinforcement learning model, the sample data can be extracted from the positive and negative sample buffers according to a predetermined proportion, so as to ensure the balance of the distribution of positive and negative samples, avoid the single pattern of the sample data for training of the reinforcement learning model, and enable the reinforcement learning model to effectively learn the collision information. The experience replay pool designed by the present application is more in line with the generation of key scenes for autonomous driving, and the extracted sample data is balanced to effectively guide the direction of reinforcement learning, so that more stable and efficient reinforcement learning is performed, which helps to improve the generation efficiency of key scenes. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Schematic diagram of the calculation principle of the risk factor.

[0043] Figure 2 Schematic diagram of the network structure of the reinforcement learning model.

[0044] Figure 3 Schematic diagram of the network structure of the actor network and its target network and the critic network and its target network.

[0045] Figure 4 Flowchart of the key scene generation method for autonomous driving.

[0046] Figure 5 Schematic diagram of the initialization process of the experience replay pool.

[0047] Figure 6 Schematic diagram of the training process of the reinforcement learning model.

[0048] Figure 7 Block diagram of the key scene generation device for autonomous driving. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are only specific embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0050] Firstly, the state, action and reward of the reinforcement learning model used in the present application are introduced and described.

[0051] In the automatic driving test, the measured object is decided to run by the automatic driving algorithm carried on the measured object, and the scene participants (such as environmental vehicles, pedestrians, etc.) other than the measured object are controlled to run by the simulation platform. Therefore, the test scene for the automatic driving test needs to specify the actions and parameters of the scene participants, so that the simulation platform can control the scene participants to run according to these information, and realize the automatic driving test based on the test scene.

[0052] The present application aims to generate the key scene in which the measured object and the scene participant most probably collide, and obtain the action parameters of the scene participant by the reinforcement learning model for scene generation. Based on this, in the present application, the reinforcement learning model takes the action parameters of the scene participant as the output, and optionally, the action parameters of the scene participant include but are not limited to: the speed of the scene participant, the target position of the scene participant. In addition, in the real traffic driving scene, each participant (such as vehicles, pedestrians, etc.) determines its own traffic behavior based on the traffic conditions it is in, therefore, in the present application, in order to simulate the real traffic driving scene, the reinforcement learning model takes the state parameters of the traffic environment as the input, and optionally, the state parameters of the traffic environment include but are not limited to: the lateral distance between the measured object and the scene participant, the longitudinal distance between the measured object and the scene participant, the driving distance of the measured object, the driving distance of the scene participant, the collision state between the measured object and the scene participant. The collision state is used to indicate whether the measured object and the scene participant collide or not, and the collision state can be binary processed to convert the collision state into a numerical value, for example, the collision state of collision is binary as 1, and the collision state of no collision is binary as 0.

[0053] To guide the reinforcement learning model to output action parameters with a higher probability of causing collisions, this invention calculates the reward acquired by the reinforcement learning model during training based on collision information between the tested object and scene participants. Optionally, the collision information includes two parts: collision risk information and collision behavior information. Collision risk information indicates the degree of risk of collision between the tested object and scene participants, while collision behavior information indicates whether a collision has occurred or not. It should be understood that, in this invention, a collision between the tested object and scene participants includes a distance of 0 between them, or a distance within a preset range (e.g., between 0 and 0.1 meters).

[0054] In one example, collision risk information is measured by the collision time between the tested object and the scene participants. Based on this, the calculation process of the reward obtained by the reinforcement learning model during training includes the following steps.

[0055] (1) For each scene participant, calculate the longitudinal collision time between the tested object and the scene participant based on the difference between the longitudinal coordinate of the tested object and the longitudinal coordinate of the scene participant, as well as the difference between the longitudinal velocity of the tested object and the longitudinal velocity of the scene participant.

[0056] Longitudinal collision time refers to the time required for the tested object to collide with a scene participant along the longitudinal axis. For example, the formula for calculating longitudinal collision time is shown below.

[0057]

[0058] in, Y refers to the longitudinal collision time between the tested object and the kth scene participant, where k is a positive integer; ego This refers to the ordinate of the measured object. This refers to the longitudinal velocity of the object being measured; This refers to the ordinate of the participant in the k-th scene. This refers to the longitudinal velocity of the participant in the k-th scene.

[0059] (2) For each scene participant, calculate the lateral collision time between the test object and the scene participant based on the difference between the horizontal coordinate of the test object and the horizontal coordinate of the scene participant, as well as the difference between the lateral velocity of the test object and the lateral velocity of the scene participant.

[0060] Lateral collision time refers to the time required for the tested object to collide with a scene participant along the horizontal axis. For example, the formula for calculating lateral collision time is shown below.

[0061]

[0062] wherein, TCTk refers to the lateral collision time between the measured object and the kth scene participant, k being a positive integer; X ego Y refers to the longitudinal coordinate of the measured object, Vx refers to the lateral velocity of the measured object; Xk refers to the longitudinal coordinate of the kth scene participant, Vxk refers to the lateral velocity of the kth scene participant.

[0063] It should be noted that in the present application, the coordinates, velocities and collision times are all vectors.

[0064] (3) Based on the longitudinal collision time and the lateral collision time, the risk factor between the measured object and the scene participant is calculated.

[0065] If one of the longitudinal collision time and the lateral collision time is greater than zero, it means that the measured object and the scene participant are not likely to collide, so the risk factor of the collision between the measured object and the scene participant is small. If both the longitudinal collision time and the lateral collision time are less than or equal to zero, it means that after the longer collision time of the two collision times, the measured object and the scene participant are likely to collide, and the smaller the difference between the two collision times, the more likely the measured object and the scene participant are to collide, and the shorter the longer collision time of the two collision times, the faster the measured object and the scene participant collide, so the smaller the difference between the two collision times, the shorter the longer collision time, and the greater the risk factor of the collision between the measured object and the scene participant.

[0066] Based on this, optionally, if the longitudinal collision time is greater than zero or the lateral collision time is greater than zero, the reciprocal of the maximum value of the longitudinal collision time and the lateral collision time is taken as the risk factor between the measured object and the scene participant; if the longitudinal collision time is less than or equal to zero and the lateral collision time is less than or equal to zero, the absolute value of the difference between the longitudinal collision time and the lateral collision time and the reciprocal of the difference between the minimum value of the longitudinal collision time and the lateral collision time are taken as the risk factor between the measured object and the scene participant. Exemplarily, the calculation formula of the risk factor between the measured object and the scene participant is as follows.

[0067]

[0068] wherein, TCTk refers to the risk factor between the measured object and the kth scene participant.

[0069] Exemplarily, as shown in Figure 1 , the driving trajectory of the measured object (A) is straight, the driving trajectory of the kth scene participant (B) is first right turn and then straight, at the current time, the longitudinal coordinate Yego The ordinate of the participant in the kth scene is less than (Y ego and All are positive numbers), the x-coordinate of the measured object is X. ego It is also smaller than the x-coordinate of the participant in the k-th scene. (X ego and All are positive numbers. Due to the lateral velocity of the measured object... The lateral velocity of the participant in the kth scene is 0. If less than 0, the lateral collision time between the tested object and the kth scene participant is... Less than 0, after a period of time The participant in the kth scene will catch up with the test subject on the horizontal axis. If the test subject's vertical velocity... The vertical velocity of the participant in the kth scene is less than ( and If all numbers are positive, then the longitudinal collision time between the tested object and the kth scene participant is... If the value is greater than 0, the tested object cannot catch up with the k-th scene participant on the vertical axis, and therefore cannot collide with the k-th scene participant. Greater than ( and If all numbers are positive, then the relationship between the tested object and the participant in the k-th scene is... Less than 0, after a period of time The tested subject will catch up with the k-th scene participant on the vertical axis; in this case, if and The absolute value of the difference Greater than the simulation time t of the test s This indicates that during the simulation time, the object being tested cannot catch up with the k-th scene participant on the vertical axis, or the k-th scene participant cannot catch up with the object being tested on the horizontal axis, thus a collision is impossible; if Less than the simulation time t of the test s This indicates that within the simulation time, the tested object can catch up with the kth scene participant on the vertical axis, and the kth scene participant can catch up with the tested object on the horizontal axis, which may result in a collision. The smaller the value, the more likely the tested object and the kth scene participant are to collide.

[0070] (4) The sum of the maximum value of the risk factors between the tested object and all participants in the scenario and the additional reward is used as the reward obtained by the reinforcement learning model.

[0071] Wherein, the extra reward is the first value in the case that the scene participant collides with the measured object, and the extra reward is the second value in the case that the scene participant does not collide with the measured object, and the first value is greater than the second value. In actual application, the specific values of the first value and the second value can be adjusted based on the scene scale and the reinforcement learning experience of the user, and the present application does not limit this. For example, the first value is a constant greater than 0 (such as 5), and the second value is 0.

[0072] Through the calculation of the above risk factor, the greater the risk factor between the measured object and the scene participant, the more likely the measured object and the scene participant collide. In order to guide the reinforcement learning model to output action parameters that are more likely to cause collision, the present application adds the maximum value of all risk factors to the extra reward as the reward obtained by the reinforcement learning model, so that the reinforcement learning model tends to high-risk reward in the training process. Exemplarily, the calculation formula of the reward obtained by the reinforcement learning model in the training process is as follows.

[0073]

[0074] Wherein, R refers to the reward obtained by the reinforcement learning model; z refers to the total number of scene participants, z is a positive integer greater than or equal to k; r c refers to the extra reward.

[0075] Secondly, the network structure of the reinforcement learning model used in the present application is introduced and described.

[0076] The present application uses a double-delay deep deterministic policy gradient network to construct a reinforcement learning model, which can calculate a continuous action space. As Figure 2 shown, the reinforcement learning model used in the present application includes an actor network, a target network corresponding to the actor network, a first critic network, a target network corresponding to the first critic network, a second critic network, and a target network corresponding to the second critic network. Optionally, the actor network, the critic network and the target network can adopt the structure of a three-layer neural network, which includes an input layer, a hidden layer and an output layer, wherein the hidden layer can include one or more convolutional layers, and the present application does not limit this. Exemplarily, the network structure of the actor network and its target network and the critic network and its target network is as shown in Figure 3 .

[0077] The actor network is used for processing the parameter value of the input current environment state (the state parameter described above) and outputting the parameter value of the current atomic action (the action parameter described above); the first critic network and the second critic network are used for processing the input sample data and outputting the first evaluation value and the second evaluation value corresponding to the sample data respectively; and the target network is used for stabilizing the expected return of the reinforcement learning model. In the present application, the sample data in the experience replay pool is constructed by simulation data obtained through an automatic driving simulation test platform. The parameter value of the atomic action is obtained by sampling through a random generation function or by inputting the parameter value of the environment state into the actor network, and then the parameter value of the atomic action is input into the automatic driving simulation test platform to obtain simulation data for constructing sample data. Subsequently, the sample data is extracted from the experience replay pool to train the actor network and the two critic networks, so as to realize the parameter update of the reinforcement learning model.

[0078] For other introduction and description of the construction process of the sample data, the experience replay pool and the initialization process thereof, the training process and the use process of the reinforcement learning model, etc., please refer to the following method embodiment, which will not be described here.

[0079] Next, the steps of the key scene generation method provided by the present application will be introduced and described as follows. Figure 4 As shown in the method embodiment, the key scene generation method in the present application includes the following steps (steps 1 to 6).

[0080] Step 1: Obtain the scene description information corresponding to the test task of the automatic driving test, and the scene description information is used to indicate the information related to the scene occurrence area and the scene participants.

[0081] The key scene generation of the automatic driving test first needs to model the test scene. The present application uses the scene description information to model the test scene for realizing the automatic driving test, and the scene description information is used to indicate or describe the information related to the scene occurrence area and the scene participants corresponding to the test task of the automatic driving test. Optionally, the scene description information includes scene occurrence area information, scene participant information and scene participant action interaction information.

[0082] The scene occurrence area information is used for describing the area where the scene is initialized, such as a crossroad, a one-way two-lane road, a curved road and the like; the scene participant information is used for describing all the scene participants involved in the scene initialization, such as the type (such as an environmental vehicle, a pedestrian and the like) and the number of the scene participants, the position (including the relative position relationship and the absolute position relationship between different scene participants) and the orientation of the scene participants, the color of the scene participants and the like; and the scene participant action interaction information is used for indicating the action interaction relationship between all the scene participants except the measured object, the action including but not limited to straight driving, parking, reversing, overtaking, cutting in, turning and lane changing, and the action interaction relationship can be described in one or more ways such as series (the end of the previous action is the start of the next action), parallel (multiple actions start at the same time or the execution time of the multiple actions has an intersection) and repetition (the action is repeatedly executed).

[0083] Step 2: Obtain a test area meeting the scene description information from the scene map, and set a driving route meeting the test task for the measured object in the test area.

[0084] The computer device models the scene occurrence area and the position of the scene participant according to the scene description information, searches a suitable test area from the scene map to meet the modeling constraint, and initializes the test scene. As known from the above step 1, the modeling constraint based on the scene description information includes the position constraint of the scene occurrence area (such as a crossroad, a straight road, a curved road and the like) and the position constraint of the scene participant (such as the distance between two environmental vehicles, the relative position relationship and the like).

[0085] When the test scene is initialized, the scene participant and the measured object need to be generated in the test area. The initial model parameters of the scene participant mainly include the initial position, the driving route and the driving speed; the measured object is decided and operated by the automatic driving algorithm carried on the measured object, and the computer device needs to set a driving route meeting the test task for the measured object in the test area, so as to serve as the basis for the automatic driving algorithm to control the driving of the measured object. It should be understood that the driving route in the present application refers to the running route formed after the start point, the end point and the task point in the test process are set (similar to the navigation in the vehicle-mounted map), and does not include the specific parameter information such as the speed and the distance of the measured object and the scene participant.

[0086] Step 3: Extract the atomic action of the scene participant from the scene description information.

[0087] The atomic action is a minimum action unit. Optionally, the atomic action of the environment vehicle includes, but is not limited to, parking, reversing, lane changing, U-turning, straight driving, left turning, right turning, anchoring, overtaking, rear-ending, etc. The atomic action of the environment vehicle flow includes, but is not limited to, vehicle speed, vehicle driving speed, etc. The atomic action of the pedestrian and the pedestrian flow includes, but is not limited to, crossing the road, walking along the road, etc.

[0088] In the present application, the action interaction information of the scene participants in the scene description information is used to indicate the action interaction relationship between all the scene participants except the measured object. Based on the action interaction information of the scene participants, the atomic actions of all the scene participants can be extracted. Optionally, one action described in the action interaction information of the scene participants can be one atomic action, or a combination of multiple atomic actions.

[0089] For example, the action interaction information of the scene participants describes that the environment vehicle 1 decelerates and drives straight, and the environment vehicle 1 drives straight at a constant speed, which is a serial relationship, the environment vehicle 1 drives straight at a constant speed and the environment vehicle 2 changes lane to the left, which is a parallel relationship, and the environment vehicle 2 changes lane to the left and the environment vehicle 2 drives straight at a constant speed, which is a serial relationship. Therefore, from the action interaction information of the scene participants, the atomic actions of the scene participants can be extracted, including: the environment vehicle 1 decelerates and drives straight, the environment vehicle 1 drives straight at a constant speed, the environment vehicle 2 changes lane to the left, and the environment vehicle 2 drives straight at a constant speed. For another example, the action interaction information of the scene participants describes that the environment vehicle 1 changes lane and overtakes, and the environment vehicle 2 drives straight, which is a serial relationship. Therefore, from the action interaction information of the scene participants, the atomic actions of the scene participants can be extracted, including: the environment vehicle 1 drives straight, the environment vehicle 1 changes lane to the left, the environment vehicle 1 drives straight, the environment vehicle 1 changes lane to the right, the environment vehicle 1 drives straight, and the environment vehicle 2 drives straight. It should be understood that in this example, the three atomic actions are all "the environment vehicle 1 drives straight", but the speeds of the environment vehicle 1 driving straight in these three atomic actions can be different.

[0090] In actual application, the computer device is pre-stored with an atomic action set, which includes the atomic actions that may exist for various types of scene participants. Therefore, when extracting the atomic actions of the scene participants, the computer device can convert the actions in the action interaction information of the scene participants into the atomic actions in the atomic action set. Of course, in some embodiments, the action interaction relationship between the scene participants can not be included in the scene description information, but the driving routes of the scene participants are included. The computer device can also decompose the driving routes of the scene participants into multiple atomic actions based on the atomic actions included in the atomic action set.

[0091] Step 4: initializing an experience replay pool corresponding to the reinforcement learning model, and constructing sample data based on the scene description information, the test area, and the driving route to store in the experience replay pool.

[0092] The application initializes an experience replay pool corresponding to the reinforcement learning model before training the reinforcement learning model, and constructs sample data required for training and stores the sample data into the experience replay pool; wherein, initializing the experience replay pool includes determining the sample data type (class) in the experience replay pool, that is, instantiating an object. Since the sample data stored in the experience replay pool is constructed based on various information of the current test task, such as scene description information, test area and driving route of the measured object, the sample data in the experience replay pool can better meet the test requirements of the current test task, and the reinforcement learning model trained based on the experience replay pool can also more accurately obtain the key scene for implementing the current test task.

[0093] In one example, the above constructing sample data based on scene description information, test area and driving route and storing the sample data into the experience replay pool includes the following sub-steps (steps 4.1 to 4.5).

[0094] Step 4.1: For an atomic action a i , obtaining a training parameter value of the atomic action a i by a random generation function.

[0095] The atomic action extracted from the scene description information in the above step 3 is only a description of the action type, and does not include specific action parameters (such as speed, target position, etc.), and the application obtains the parameter value of the atomic action by the actor network in the reinforcement learning model for generating the key scene. Wherein, in order to facilitate description and distinction, the application refers to the parameter value input and output by the reinforcement learning model during training as a training parameter value, and refers to the parameter value input and output by the reinforcement learning model during use as a determined parameter value. In the application, the computer device can generate the parameter value in two ways, one way is to obtain it by the actor network in the reinforcement learning model as shown in Figure 3 , and the other way is to obtain it by a random generation function (such as rand function, etc.). Before training the reinforcement learning model (that is, in step 4), a sufficient amount of sample data needs to be constructed and stored into the experience replay pool, at this time, the construction of the sample data needs the computer device to run the random generation function to obtain the parameter value of the atomic action (the training parameter value of the atomic action); then in the training process of the reinforcement learning model and the updating of the experience replay pool, the computer device obtains the parameter value of the atomic action (the determined parameter value of the atomic action) by the actor network as shown in Figure 3 .

[0096] In step 4.1, as shown in Figure 5 , a specific parameter value is randomly sampled from the parameter range of the atomic action a i by a random generation function, to obtain the training parameter value of the atomic action a i , and i is a positive integer. Wherein, the atomic action ai The parameter range of the atomic action a i may be obtained through the scene description information. If the parameter range of the atomic action a i is not included in the scene description information, the random generation function may combine the parameter value sampled from the last atomic action a i-1 to obtain the training parameter value of the atomic action a i . For example, assuming that the atomic action a i is straight driving, if the parameter range (such as speed) set for the atomic action a i in the scene description information is 10 to 20 meters per second, the random generation function may randomly sample a parameter value (such as 12 meters per second) from the interval of 10 to 20 meters per second as the training parameter value of the atomic action a i .

[0097] Step 4.2: The automatic driving simulation test platform processes the training parameter value of the environment state s i and the training parameter value of the atomic action a i according to the test area and the driving route, to obtain the training parameter value of the environment state s i+1 , the training reward corresponding to the atomic action a i , and the collision state.

[0098] The automatic driving simulation test platform renders the current traffic environment in the test area according to the training parameter value of the environment state s i , controls the scene participant corresponding to the atomic action a i to perform the atomic action a i according to the training parameter value of the atomic action a i in the test area, and in addition, the measured object is connected to the automatic driving simulation test platform and runs according to the pre-set driving route in the test area. Through the processing of the automatic driving simulation test platform, the training reward corresponding to the atomic action a i and the collision state are obtained. The collision state can indicate whether there is a collision between the scene participant and the measured object when the atomic action a i is performed; the training reward is the reward obtained by the reinforcement learning model in the training process, and the calculation process is described above and will not be described here. In addition, the processing of the automatic driving simulation test platform can also update the parameter value of the environment state, that is, obtain the training parameter value of the environment state s i+1 for subsequent test processing; based on this, the training parameter value of the environment state s i is output by the automatic driving simulation test platform based on the training parameter value of the environment state s i-1 and the training parameter value of the atomic action a i-1 .

[0099] The training parameter values ​​of environment state s1 are obtained based on scene description information. For example, the parameter values ​​of environment state include: the lateral distance between the tested object and scene participants, the longitudinal distance between the tested object and scene participants, the travel distance of the tested object, the travel distance of scene participants, and the collision state between the tested object and scene participants. Since environment state s1 is the initial environment state, the travel distance between the tested object and each scene participant is 0, and the collision state between the tested object and each scene participant is no collision. The lateral and longitudinal distances between the tested object and each scene participant can be obtained from the scene description information regarding scene initialization.

[0100] Step 4.3: Based on i and environment state s i Training parameter values, atomic action a i Training parameter values, environment state s i+1 Training parameter values, atomic action a i The corresponding training rewards and collision states are used to construct the atomic action a. i The corresponding sample data.

[0101] Based on this, atomic action a i The corresponding sample data includes: atomic action a i The sequence number (i.e., i) and the environmental state s i Training parameter values, atomic action a i Training parameter values, environment state s i+1 Training parameter values, atomic action a i The corresponding training rewards and collision states.

[0102] Step 4.4: Let i = i + 1, and start executing from step 4.1 above again until the sample data has been constructed for all atomic actions.

[0103] like Figure 5 As shown, the environmental state s was obtained through the processing in step 4.2 above. i+1 Given the training parameter values, let i = i + 1, and start executing step 4.1 above again, sampling and obtaining the atomic action a through a random generation function. i+1 The training parameter values ​​are used to test the environmental state s through an autonomous driving simulation test platform. i+1 Training parameter values ​​and atomic actions a i+1 By processing the training parameter values, atomic action a can be constructed. i+1corresponding to the atomic action. The process is repeated in this way until sample data corresponding to all atomic actions is constructed. It should be understood that in the present application, the order of construction of sample data corresponding to atomic actions (the order of the serial numbers of the atomic actions) is consistent with the timing logic relationship (the execution timing) between the atomic actions, wherein the action interaction relationship between all scene participants is included in the scene description information, and the timing logic relationship between the atomic actions can be obtained based on the action interaction relationship.

[0104] In practical applications, there is a certain correlation between the parameter values of the atomic actions, which means that the selection of the parameter values of the atomic action executed first will affect the selection of the parameter values of the atomic action executed later, i.e., the selection of the parameter values of the atomic action a i will affect the selection of the parameter values of the atomic action a i+1 . In addition, theoretically, when using the parameter values of the atomic action executed first for prediction, no information of the atomic action executed later should be shared, i.e., the selection of the parameter values of the atomic action a i+1 will not affect the selection of the parameter values of the atomic action a i .

[0105] Based on this, optionally, as shown in Figure 5 , the present application adds a padding mask and a sequence mask to the parameter values of the atomic actions to solve the problem of the non-fixed length of the action space while ensuring the correctness of the timing. Taking the processing of the training parameter values of the atomic action a i as an example, the above step 4.1 includes: randomly sampling from the parameter range of the atomic action a i by using a random generation function, and adding a padding mask and a sequence mask to the sampling result to obtain the training parameter values of the atomic action a i ; wherein the padding mask is used to fill the parameter values of n-1 atomic actions in the sampling result, so that the training parameter values of the atomic action a i are represented in the action space with a length of n, and n is a positive integer greater than or equal to i; the sequence mask is used to set the parameter values of the n-i atomic actions whose execution timing is located after the atomic action a i to zero in the action space, and set the parameter values of the i-1 atomic actions whose execution timing is located before the atomic action a i to the training parameter values of the corresponding atomic actions, and the determination of the execution timing in the processing of the sequence mask is based on the serial number of the atomic action. After that, the parameter values of the processed atomic actions are input into the automatic driving simulation test platform for testing.

[0106] Step 4.5: Repeat the construction process of the sample data corresponding to all atomic actions for u rounds, and store all the sample data constructed in the u rounds to the experience replay pool.

[0107] Before training the reinforcement learning model, the training parameter values of the atomic actions in the constructed sample data are obtained by a random generation function, so that the training parameter values of the atomic action a1 obtained by the random generation function are different each time for the same initial environment state s1, and the training parameter values of the next environment state are also different. This cycle is executed, and in different sample data construction rounds, the same atomic action corresponds to different training parameter values, different training parameter values of the environment state, different collision states, and different collision rewards. Therefore, after the execution of multiple rounds, the computer device constructs multiple sample data for each atomic action. For example, assuming that there are n atomic actions, the sample data construction process is repeatedly executed for u rounds (u is a positive integer greater than 1), n sample data are constructed in each round, and n×u sample data are constructed in total after the execution of u rounds. Alternatively, the specific value of u can be determined in combination with the number of sample data required for training the reinforcement learning model, for example, the number of sample data used to train the reinforcement learning model is ten times the total number of atomic actions, and u is an integer greater than or equal to 10 (such as 10, 11, or 12).

[0108] Finally, the computer device stores all the sample data constructed in the u rounds to the experience replay pool. Of course, the sample data can also be stored to the experience replay pool in real time after the construction of one sample data is completed. Alternatively, in order to avoid the single pattern of the sample data for model training, so that the reinforcement learning model effectively learns the collision information, as shown in Figure 5 The present application divides the experience replay pool corresponding to the reinforcement learning model into a positive sample buffer and a negative sample buffer, which are used to store positive samples and negative samples, respectively, and a certain number of positive samples and negative samples are extracted from the two buffers in proportion for model training. Among them, the positive sample refers to the sample data in which the measured object collides with the scene participant, and the negative sample refers to the sample data in which the measured object does not collide with the scene participant.

[0109] For example, the computer device classifies all the sample data according to the collision state, obtains at least one positive sample and at least one negative sample, stores the at least one positive sample in the positive sample buffer of the experience replay pool, and stores the at least one negative sample in the negative sample buffer of the experience replay pool. Alternatively, after the construction of one sample data is completed, the sample data is stored based on the collision state of the sample data, and the atomic action a i For example, the computer device classifies all the sample data according to the collision state, obtains at least one positive sample and at least one negative sample, stores the at least one positive sample in the positive sample buffer of the experience replay pool, and stores the at least one negative sample in the negative sample buffer of the experience replay pool. Alternatively, after the construction of one sample data is completed, the sample data is stored based on the collision state of the sample data, and the atomic action a i corresponding sample data is a positive sample or a negative sample; if it is a positive sample, the atomic action a iThe corresponding sample data is stored in the positive sample buffer; if it is a negative sample, the atomic action a i The corresponding sample data is stored in the negative sample buffer. Of course, in the case where the experience replay pool is divided into a positive sample buffer and a negative sample buffer, the sample data in the experience replay pool can carry the collision state or can not carry the collision state, and the present application does not limit this.

[0110] Step 5: sample data is extracted from the experience replay pool to update the reinforcement learning model, and the determined parameter value of the atomic action is obtained through the updated reinforcement learning model.

[0111] In the present application, the sampling of the determined parameter value of each atomic action requires training of the reinforcement learning model once, i.e., updating the parameters of the reinforcement learning model, so that the reinforcement learning model can better learn the characteristics of the current traffic environment, tend to high-risk rewards, and output action parameters that are more likely to cause collisions. The computer device extracts a certain number of sample data from the experience replay pool to update the reinforcement learning model. Optionally, in order to enable the reinforcement learning model to effectively learn the collision information, the present application divides the experience replay pool corresponding to the reinforcement learning model into a positive sample buffer and a negative sample buffer, and the computer device can extract sample data from the two buffers according to a pre-set proportion. For example, the pre-set ratio of the number of sample data extracted from the positive sample buffer to the number of sample data extracted from the negative sample buffer is η (0≤η≤1), and if η is 0.5, it means that the same number of sample data is extracted from the positive sample buffer and the negative sample buffer for training of the reinforcement learning model. Figure 6

[0112] In one example, the process of obtaining the determined parameter value of the atomic action a i is taken as an example, and the above step 5 includes the following sub-steps (steps 5.1 to 5.9).

[0113] Step 5.1: a target number of sample data is extracted from the experience replay pool for updating the reinforcement learning model. i

[0114] The target number can be a pre-set number, such as a pre-set target number of ten times the total number of atomic actions. In the case of the atomic action a i ​​When obtaining a specific parameter value, a target number of sample data is first extracted from the experience replay pool to update the reinforcement learning model. Optionally, when obtaining specific parameter values ​​for different atomic actions, the number of sample data used for updating the reinforcement learning model can be the same (e.g., the target number of sample data is extracted for each) or different; this invention does not limit this. However, the content of the sample data used for updating the reinforcement learning model must be at least partially different. For example, each time sample data is extracted for updating, the computer device can randomly extract a target number of sample data from the experience replay pool, and at least some of the sample data extracted in any two updates must be different. For further descriptions of the sample data extraction process, please refer to the above embodiments, which will not be repeated here.

[0115] Step 5.2: Process the training parameter values ​​of the corresponding environmental state and the training parameter values ​​of the corresponding atomic actions in each sample data through the first commentator network and the second commentator network, and obtain the first evaluation value and the second evaluation value corresponding to the corresponding atomic action respectively.

[0116] In reinforcement learning models, the commentator network evaluates the input environment state and the parameter values ​​of atomic actions, outputting an action-state evaluation value (Q). For example... Figure 6 As shown, the parameters of the first commentator network are represented by θ1, and the parameters of the second commentator network are represented by θ2. When processing each sample data, both the first and second commentator networks process the training parameter values ​​of the corresponding environmental state s and the training parameter values ​​of the corresponding atomic action a in the sample data, and output the first evaluation value corresponding to the corresponding atomic action a. Second evaluation value

[0117] Step 5.3: Through the target network of the actor network, process the training parameter values ​​of the next environmental state of the corresponding environmental state in each sample data to obtain the predicted parameter values ​​of the next atomic action of the corresponding atomic action.

[0118] The training objective of a reinforcement learning model is to find the optimal policy that maximizes the expected reward. The expected reward is obtained by fitting the reward for each atomic action to the evaluation value of the next atomic action. The reward for each atomic action is directly included in the sample data; however, the evaluation value of the next atomic action requires the parameter values ​​of the next atomic action (predicted parameter values) and the parameter values ​​of the next environment state (trained parameter values).

[0119] In this invention, the predicted parameter values ​​of the next atomic action of the corresponding atomic action are obtained through the target network of the actor network, such as...Figure 6 As shown, the parameters of the actor network are represented by φ, and the parameters of the target network of the actor network are represented by φ'; when processing each sample data, the training parameter value of the next environment state s' of the corresponding environment state s in the sample data is input into the target network of the actor network to obtain the predicted parameter value of the next atom a' of the corresponding atom action a (a' = π φ′ (s').

[0120] Step 5.4: The training parameter value of the next environment state of the corresponding environment state in each sample data and the predicted parameter value of the next atom action of the corresponding atom action are processed by the target network of the first critic network and the target network of the second critic network to obtain the first evaluation value and the second evaluation value corresponding to the next atom action of the corresponding atom action, respectively.

[0121] In the present application, the evaluation value corresponding to the next atom action of the corresponding atom action is obtained by the target network of the critic network. As shown, Figure 6 As shown, the parameters of the target network of the first critic network are represented by θ1', and the parameters of the target network of the second critic network are represented by θ2'; when processing each sample data, the target network of the first critic network and the target network of the second critic network both process the training parameter value of the next environment state s' of the environment state s in the sample data and the predicted parameter value of the next atom a' of the corresponding atom action a output by the target network of the actor network to output the first evaluation value and the second evaluation value corresponding to the next atom a' of the corresponding atom action a, respectively. In the present application, the parameters of the target networks of the two critic networks are the same, that is, θ1' is equal to θ2', so that is also equal to

[0122] Step 5.5: A loss function of the first critic network is constructed based on the rewards corresponding to the corresponding atom actions, the first evaluation values corresponding to the corresponding atom actions, and the first evaluation values corresponding to the next atom actions of the corresponding atom actions in all sample data; a loss function of the second critic network is constructed based on the rewards corresponding to the corresponding atom actions, the second evaluation values corresponding to the corresponding atom actions, and the second evaluation values corresponding to the next atom actions of the corresponding atom actions in all sample data.

[0123] The expected return (Q_target) can be fitted through the rewards corresponding to the corresponding atom actions and the evaluation values corresponding to the next atom actions of the corresponding atom actions, and the loss function of the critic network is constructed through the mean square error between the expected return and the evaluation values of the corresponding atom actions. Based on this, Figure 6As shown, the calculation formulas of the loss function Loss1 of the first critic network and the loss function Loss2 of the second critic network are as follows.

[0124]

[0125]

[0126] wherein r is the training reward corresponding to the corresponding atomic action a; γ is a weight coefficient (discount factor) of the evaluation value of the next atom a' of the corresponding atomic action a in the expected return; is to add noise ∈ in the predicted parameter value of the next atom a' of the corresponding atomic action a (i.e. ), so as to explore more possible parameter values and improve the diversity of sample data; is the evaluation value corresponding to the next atom a' of the corresponding atomic action a, which is output by the target network of the first critic network and the target network of the second critic network (the parameters of the target networks of the two critic networks are the same), and the parameters of the target networks of the two critic networks are equal to the parameters of the critic network with smaller evaluation value output in the first critic network and the second critic network.

[0127] Step 5.6: Perform backward gradient propagation according to the loss function of the first critic network and the loss function of the second critic network respectively, and update the parameters of the first critic network and the second critic network.

[0128] Based on the loss functions of the two critic networks, the parameters θ1 and θ2 of the two critic networks are updated by backward gradient propagation, and the parameter update gradients of the two critic networks are as follows.

[0129]

[0130]

[0131] Wherein, M is a target number, that is, the total number of sample data extracted from the experience replay pool for training of the reinforcement learning model in step 5.1. In addition, the parameters of the target network of the first critic network and the parameters of the target network of the second critic network are updated based on the parameters of the critic network with a smaller output evaluation value in the first critic network and the second critic network; and the parameter update frequency of the target network of the first critic network and the target network of the second critic network is lower than the parameter update frequency of the first critic network and the second critic network, so as to stabilize the expected return. For example, the computer device updates the parameters of the two critic networks for each sampling of the determined parameter value of the atom action; but updates the parameters of the target network of the two critic networks once every other sampling of the determined parameter value of the atom action; that is, the parameters of the two critic networks are updated once for each reinforcement learning model training, and the parameters of the target network of the two critic networks are updated once for every two reinforcement learning model trainings. Of course, in actual application, the parameter update frequency can also be set in other ways, and the present application does not limit this.

[0132] Step 5.7: Constructing a loss function of the actor network based on the first evaluation value and the second evaluation value corresponding to the corresponding atom action in all sample data.

[0133] The actor network needs to cater to the evaluation of the critic network, so the optimization of the actor network needs to maximize the evaluation value of the critic network. In the present application, in order to maximize the evaluation value of the critic network, the inverse of the evaluation value of the critic network is taken as the loss function of the actor network, so as to realize the maximization of the evaluation value based on the minimization of the loss value of the loss function. Alternatively, the computer device randomly selects one of the first critic network and the second critic network, or pre-sets one of the critic networks, or selects the critic network with a smaller output evaluation value, and constructs the loss function of the actor network through the inverse of the evaluation value output by the critic network. For example, as shown in Figure 6 The calculation formula of the loss function Loss of the actor network is as follows.

[0134] Loss=-Q θ (s,a)

[0135] Wherein, a=π φ (s);θ is the parameter of the critic network, which is any one of the parameter θ1 of the first critic network and the parameter θ2 of the second critic network, which is determined by selection in combination with actual application.

[0136] Step 5.8: Updating the parameters of the actor network according to the loss function of the actor network.

[0137] The loss function of the actor network is used for reverse gradient propagation to update the parameters of the actor network, and the gradient of the parameter update of the actor network is as follows.

[0138]

[0139] The parameters of the target network of the actor network are updated based on the parameters of the actor network, and the parameter update frequency of the target network of the actor network is lower than the parameter update frequency of the actor network. For example, the parameters of the actor network are updated once every two samples of the determined parameter values of the atomic actions, and the parameters of the target network of the actor network are updated once every three samples of the determined parameter values of the atomic actions, that is, the parameters of the actor network are updated once every two times of reinforcement learning model training, and the parameters of the target network of the actor network are updated once every four times of reinforcement learning model training. In addition, it can be known from the above example that the parameter update frequency of the actor network is lower than the parameter update frequency of the first critic network and the second critic network, which helps to stabilize the training process. Generally, as shown in the following table, the parameters of the two critic networks are updated once every time of reinforcement learning model training, and the parameters of the actor network are updated once every two times of reinforcement learning model training. Of course, in actual applications, the parameter update frequency can also be set in other ways, and the present application does not limit this. Figure 6

[0140] It should be understood that the above steps 5.7 and 5.8 are not performed every time the determined parameter values of the atomic actions are sampled. For example, steps 5.7 and 5.8 are performed for the sampling of the determined parameter values of the atomic action a i , but not for the sampling of the determined parameter values of the atomic action a i+1 .

[0141] Step 5.9: The determined parameter values of the environment state s i are processed by the actor network to obtain the determined parameter values of the atomic action a i .

[0142] For the sampling of the determined parameter values of the atomic action a i , after completing the training of the reinforcement learning model this time, the determined parameter values of the environment state s i are processed by the actor network in the reinforcement learning model to output the determined parameter values of the atomic action a i . In the present application, the process of obtaining the determined parameter values of the environment state s i includes: processing the determined parameter values of the environment state s i-1 and the determined parameter values of the atomic action a i-1 according to the test area and the driving route by the automatic driving simulation test platform to obtain the determined parameter values of the environment state s i ​The determined parameter values ​​are as follows: the determined parameter values ​​of the initial environment state s1 are obtained based on the scene description information (please refer to the process of obtaining the training parameter values ​​of environment state s1 in step 4.2 above), and the determined parameter values ​​of the initial atomic action a1 are obtained by the actor network processing the determined parameter values ​​of the input environment state s1.

[0143] like Figure 6 As shown, after obtaining atomic action a i After determining the parameter values, the atomic action a will be performed. i Determine the parameter values ​​and input them into the autonomous driving simulation test platform (environmental state s). i The parameter values ​​are determined by the autonomous driving simulation test platform for atomic action a. i-1 (Storing the corresponding sample data during the acquisition process) allows us to obtain the next environmental state s. i+1 Determine the parameter values, and then target the atomic action a. i+1 By repeating steps 5.1 to 5.9 above, atomic action a can be obtained. i+1 The determined parameter values ​​are obtained. By repeating steps 5.1 to 5.9 above, the determined parameter values ​​of all atomic actions can be obtained.

[0144] Optionally, such as Figure 6 As shown, in this invention, after obtaining the determined parameter values ​​for each atomic action, the computer device inputs these values ​​into an autonomous driving simulation test platform to obtain simulation data to construct sample data corresponding to the atomic actions. This sample data is then stored in the experience replay pool to update the pool. This invention updates the experience replay pool in a timely manner based on the current training status of the reinforcement learning model during training, enriching the sample data and making it more accurate. Subsequent training draws sample data from the updated experience replay pool, effectively improving the training performance of the reinforcement learning model. Further details regarding the sample data construction process can be found in step 4 above and will not be elaborated upon here.

[0145] It should be understood that in actual application, the computer device can train the reinforcement learning model for multiple rounds so that the reinforcement learning model is effectively trained and the parameter sampling accuracy of the reinforcement learning model is improved. For example, assuming that the total number of atomic actions is n, the reinforcement learning model is trained for v rounds (v is a positive integer greater than 1), and in each round of training, the parameters of the reinforcement learning model are updated n times (one update corresponds to one atomic action), and after v rounds of training, the parameters of the reinforcement learning model are updated n x v times in total. After each parameter update, the determined parameter value of the corresponding atomic action is obtained through the updated reinforcement learning model, and the determined parameter value of the corresponding atomic action is processed through the automatic driving simulation test platform to obtain the sample data corresponding to the corresponding atomic action and update the experience replay pool. That is, a new sample data is obtained after each parameter update of the reinforcement learning model, n new sample data are added to the experience replay pool after one round of training, and n x v new sample data are added to the experience replay pool after v rounds of training.

[0146] It should be noted that for the same atomic action, the reinforcement learning model may sample different determined parameter values in different rounds of training. Since new sample data is continuously constructed in the training process of the reinforcement learning model to update the experience replay pool, the later the training process is, the more rich and accurate sample data can be extracted for parameter update of the reinforcement learning model, and the reinforcement learning model can achieve better sampling effect. Based on this, the determined parameter value of each atomic action sampled by the reinforcement learning model in the last round of training is used to generate the key scene.

[0147] Step 6: Based on the determined parameter values of all atomic actions, generate a key scene for implementing a test task.

[0148] The key scene refers to a test scene in which the probability of collision between the measured object and the scene participant is the largest. In the present application, the sampling of the determined parameter value of each atomic action trains the reinforcement learning model to tend to a high-risk collision reward, so that the determined parameter value of the atomic action that most likely causes a collision to occur under the corresponding environment state is obtained through the reinforcement learning model after parameter update. After obtaining the determined parameter values of all atomic actions, the determined parameter values can be supplemented to the scene description information to generate a key scene matching the test task. It should be understood that from the above description of step 5, it can be known that in the case where the reinforcement learning model is trained for multiple rounds, the determined parameter values of all atomic actions in step 6 are obtained by the reinforcement learning model in the last round of training.

[0149] In summary, the present application disassembles the test scene into atomic action sequences, models the timing relationship between atomic actions, and samples the parameter values of each atomic action in the key scene through the reinforcement learning model. The present application first extracts the atomic actions of the scene participants from the scene description information, and then disassembles the test scene into the atomic action sequences of the scene participants according to the timing logic relationship between the atomic actions. Then, the reinforcement learning model is constructed, the atomic action sequences are modeled as the output of the reinforcement learning model, and the action parameters of the scene participants are set as the actions of the reinforcement learning model, and the state parameters of the traffic environment are set as the states of the reinforcement learning model. Then, the number of atomic actions in the atomic action sequence is set as the number of steps executed by the reinforcement learning model, and in each step, the reinforcement learning model samples a parameter value of a corresponding atomic action in combination with the timing mask. Based on this, on the one hand, the present application samples the parameter values of the atomic actions through the reinforcement learning model, effectively avoiding the huge test verification amount caused by random parameter sampling, and improving the generation efficiency of the key scene; on the other hand, the present application fits the timing logic relationship between the atomic actions through the timing mask, so that the parameter sampling of the current atomic action is only affected by the previously executed atomic action, and the correctness of the timing logic relationship is ensured.

[0150] In addition, the present application constructs a reward function based on the collision information between the measured object and the scene participants, and guides the reinforcement learning model to explore the parameter space of the key scene. The collision information includes two parts: collision risk information and collision behavior information. The collision risk information is used to indicate the risk degree of collision between the measured object and the scene participants, and the collision behavior information is used to indicate the collision or non-collision between the measured object and the scene participants. Among them, the risk degree (risk factor) is measured by the lateral collision time and the longitudinal collision time, and the collision or non-collision is measured by the additional reward. When designing the reward function, the greater the risk of collision between the measured object and the scene participants, or the greater the reward obtained by the reinforcement learning model when the measured object collides with the scene participants. Through the design of greater risk, collision, and greater reward, the present application guides the reinforcement learning model to tend to high-risk rewards and explore the parameter space of the key scene where the collision is most likely to occur, which is more in line with the generation principle of the key scene, and effectively improves the generation efficiency of the key scene.

[0151] In addition, the application optimizes the design of the experience replay pool, ensures the balanced distribution of the extracted sample data, and enables the reinforcement learning model to effectively learn the collision information. In the related art, the experience replay pool only has one buffer, and all sample data are stored in the buffer. When the sample data is extracted, it is randomly extracted from the buffer, which can easily lead to an unbalanced distribution of collision samples and non-collision samples. The application divides the experience replay pool into a positive sample buffer and a negative sample buffer. When storing sample data, if the simulation process of the sample data occurs collision, the sample data is regarded as a positive sample and is stored in the positive sample buffer, otherwise, the sample data is regarded as a negative sample and is stored in the negative sample buffer. When a certain amount of sample data is extracted from the experience replay pool for training the reinforcement learning model, the sample data can be extracted from the positive and negative sample buffers according to a preset proportion, so as to ensure the balance of the positive and negative sample distribution, avoid the single pattern of the sample data for training the reinforcement learning model, and enable the reinforcement learning model to effectively learn the collision information. The application designs an experience replay pool that is more suitable for generating key scenes of autonomous driving, effectively guides the reinforcement learning direction through the balanced distribution of the extracted sample data, performs more stable and efficient reinforcement learning, and helps to improve the generation efficiency of the key scenes.

[0152] Please refer to Figure 7 which shows a block diagram of a key scene generation device for autonomous driving provided by an embodiment of the application. The key scene generation device can be a computer device or can be arranged in a computer device. As shown in Figure 7 , the key scene generation device 700 includes an information acquisition module 710, a driving setting module 720, an action extraction module 730, a sample construction module 740, a parameter acquisition module 750, and a scene generation module 760.

[0153] The information acquisition module 710 is configured to acquire scene description information corresponding to a test task of autonomous driving test, wherein the scene description information is used to indicate information related to a scene occurrence area and a scene participant.

[0154] The driving setting module 720 is configured to acquire a test area conforming to the scene description information from a scene map, and set a driving route conforming to the test task for a measured object in the test area.

[0155] The action extraction module 730 is configured to extract an atomic action of the scene participant from the scene description information.

[0156] The sample construction module 740 is configured to initialize an experience replay pool corresponding to a reinforcement learning model, and construct sample data based on the scene description information, the test area, and the driving route, and store the sample data to the experience replay pool.

[0157] The parameter acquisition module 750 is used to: extract sample data from the experience replay pool to update the reinforcement learning model, and obtain the determination parameter values ​​of the atomic action through the updated reinforcement learning model;

[0158] The scene generation module 760 is used to generate a key scene for implementing the test task based on the determined parameter values ​​of all atomic actions. The key scene refers to the test scene in which the probability of collision between the tested object and the scene participants is the highest.

[0159] Optionally, the sample construction module 740 is used for: targeting atomic action a i The atomic action a is obtained through a random generation function. i The training parameter values, where i is a positive integer; by accessing the autonomous driving simulation test platform with the test object, according to the test area and the driving route, the environmental state s is... i The training parameter values ​​and the atomic action a i The training parameter values ​​are processed to obtain the environmental state s. i+1 The training parameter values, the atomic action a i The corresponding training rewards and collision states; wherein, the training parameter values ​​of the initial environment state s1 are obtained based on the scene description information; based on i, the environment state s i The training parameter values, the atomic action a i The training parameter values, the environmental state s i+1 The training parameter values, the atomic action a i The corresponding training rewards and collision states are used to construct the atomic action a. i The corresponding sample data; let i = i + 1, starting from the atomic action a i The atomic action a is obtained through a random generation function. i The step of setting the training parameter values ​​is executed again until the sample data has been constructed for all atomic actions; the process of constructing the sample data for all atomic actions is repeated for u rounds, and all sample data constructed in the u rounds is stored in the experience replay pool.

[0160] Optionally, the total number of atomic actions is n, where n is a positive integer greater than or equal to i; the sample construction module 740 is further configured to: generate samples from the atomic actions a using the random generation function. i Randomly sample within the parameter range, and add a padding mask and a timing mask to the sampling results to obtain the atomic action a. i The training parameter values; wherein, the padding mask is used to fill the sampling result with the parameter values ​​of n-1 atomic actions, so that the atomic action a itraining parameter value of the atom action a i training parameter value of the atom action a i training parameter value of the atom action a

[0161] training parameter value of the atom action a i The corresponding training reward is a reward obtained by the reinforcement learning model in the training process. The device 700 is further configured to: for each scene participant, calculate a longitudinal collision time between the measured object and the corresponding scene participant based on a difference between a longitudinal coordinate of the measured object and a longitudinal coordinate of the corresponding scene participant, and a difference between a longitudinal speed of the measured object and a longitudinal speed of the corresponding scene participant; for each scene participant, calculate a lateral collision time between the measured object and the corresponding scene participant based on a difference between a lateral coordinate of the measured object and a lateral coordinate of the corresponding scene participant, and a difference between a lateral speed of the measured object and a lateral speed of the corresponding scene participant; calculate a risk factor between the measured object and each scene participant based on the longitudinal collision time and the lateral collision time; and take a sum of the maximum risk factor between the measured object and all scene participants and an additional reward as the reward obtained by the reinforcement learning model.

[0162] The device 700 is further configured to: for each scene participant, if the longitudinal collision time is greater than zero or the lateral collision time is greater than zero, take a reciprocal of a maximum value of the longitudinal collision time and the lateral collision time as the risk factor between the measured object and the corresponding scene participant; and for each scene participant, if the longitudinal collision time is less than or equal to zero and the lateral collision time is less than or equal to zero, take an absolute value of a difference between the longitudinal collision time and the lateral collision time and a reciprocal of a difference between a minimum value of the longitudinal collision time and the lateral collision time as the risk factor between the measured object and the corresponding scene participant.

[0163] The experience replay pool includes a positive sample buffer and a negative sample buffer. The sample construction module 740 is further configured to: classify all sample data constructed in the u rounds according to the collision state, to obtain at least one positive sample and at least one negative sample; store the at least one positive sample in the positive sample buffer and store the at least one negative sample in the negative sample buffer; wherein the positive sample refers to sample data in which the measured object collides with the scene participant, and the negative sample refers to sample data in which the measured object does not collide with the scene participant.

[0164] Optionally, the reinforcement learning model comprises: an actor network, a target network corresponding to the actor network, a first critic network, a target network corresponding to the first critic network, a second critic network, and a target network corresponding to the second critic network; wherein the actor network is configured to process the parameter value of the input current environment state, and output the parameter value of the current atomic action; the first critic network and the second critic network are configured to process the input sample data, and output the first evaluation value and the second evaluation value corresponding to the respective atomic action, respectively; and the target network is configured to stabilize the expected return of the reinforcement learning model.

[0165] Optionally, the parameter obtaining module 750 is configured to: for the atomic action a i extract a target number of sample data from the experience replay pool for updating the reinforcement learning model; process the training parameter value of the respective environment state and the training parameter value of the respective atomic action in each sample data through the first critic network and the second critic network, and obtain the first evaluation value and the second evaluation value corresponding to the respective atomic action, respectively; process the training parameter value of the next environment state of the respective environment state in each sample data through the target network of the actor network, and obtain the predicted parameter value of the next atomic action of the respective atomic action; process the training parameter value of the next environment state of the respective environment state and the predicted parameter value of the next atomic action of the respective atomic action in each sample data through the target network of the first critic network and the target network of the second critic network, and obtain the first evaluation value and the second evaluation value corresponding to the next atomic action of the respective atomic action, respectively; construct a loss function of the first critic network based on the reward corresponding to the respective atomic action, the first evaluation value corresponding to the respective atomic action, and the first evaluation value corresponding to the next atomic action of the respective atomic action in all sample data; construct a loss function of the second critic network based on the reward corresponding to the respective atomic action, the second evaluation value corresponding to the respective atomic action, and the second evaluation value corresponding to the next atomic action of the respective atomic action in all sample data; perform reverse gradient propagation according to the loss function of the first critic network and the loss function of the second critic network, respectively, and update the parameters of the first critic network and the second critic network; construct a loss function of the actor network based on the first evaluation value and the second evaluation value corresponding to the respective atomic action in all sample data; perform reverse gradient propagation according to the loss function of the actor network, and update the parameters of the actor network; and process the determined parameter value of the environment state s i through the actor network, and obtain the determined parameter value of the atomic action a i ; wherein the environment state s iThe obtaining process of the determined parameter value of the environment state s i-1 The determined parameter value of the atomic action a i-1 The determined parameter value of the environment state s i The determined parameter value of the initial environment state s1 is obtained based on the scene description information, and the determined parameter value of the initial atomic action a1 is obtained by processing the input determined parameter value of the environment state s1 by the actor network.

[0166] Optionally, the experience replay pool includes a positive sample buffer and a negative sample buffer; the parameter obtaining module 750 is further configured to: extract a first number of positive samples from the positive sample buffer, and extract a second number of negative samples from the negative sample buffer, the target number of sample data including the first number of positive samples and the second number of negative samples, the sum of the first number and the second number being equal to the target number; wherein the positive sample refers to sample data in which the measured object collides with the scene participant, and the negative sample refers to sample data in which the measured object does not collide with the scene participant.

[0167] Optionally, the parameters of the target network of the first critic network and the parameters of the target network of the second critic network are updated based on the parameters of the critic network with a smaller output evaluation value in the first critic network and the second critic network; and the parameter update frequency of the target network of the first critic network and the target network of the second critic network is lower than the parameter update frequency of the first critic network and the second critic network; the parameters of the target network of the actor network are updated based on the parameters of the actor network; and the parameter update frequency of the target network of the actor network is lower than the parameter update frequency of the actor network, and the parameter update frequency of the actor network is lower than the parameter update frequency of the first critic network and the second critic network.

[0168] Optionally, the apparatus 700 is further configured to: process the determined parameter value of the environment state s i and the determined parameter value of the atomic action a i to obtain the determined parameter value of the environment state s i+1 , the corresponding training reward and the collision state of the atomic action a i based on the i, the determined parameter value of the environment state s i , the determined parameter value of the atomic action a i , the determined parameter value of the environment state s i+1 , the determined parameter value of the atomic action ai corresponding training reward and collision state, constructing the atomic action a i corresponding sample data; storing the atomic action a i corresponding sample data into the experience replay pool to update the experience replay pool.

[0169] Optionally, the parameter value of the atomic action includes: a speed of the scene participant, a target position of the scene participant; the parameter value of the environment state includes: a lateral distance between the measured object and the scene participant, a longitudinal distance between the measured object and the scene participant, a running distance of the measured object, a running distance of the scene participant, and a collision state between the measured object and the scene participant.

[0170] For specific implementation process, beneficial effects and the like of the device module, please refer to the above introduction of the method embodiments, which will not be repeated here.

[0171] In an example embodiment, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above-mentioned key scene generation method for automatic driving.

[0172] In an example embodiment, a computer readable storage medium is also provided, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned key scene generation method for automatic driving.

[0173] In an example embodiment, a computer program product is also provided, which, when running on a computer device, causes the computer device to execute the above-mentioned key scene generation method for automatic driving.

[0174] The above embodiments are only used to illustrate the technical solutions of the present application rather than limit it, and the ordinary skilled in the art can modify or equivalently replace the technical solutions of the present application, and the protection scope of the present application should be subject to the claims.

Claims

1. A method for generating key scenarios for autonomous driving, characterized in that, The method includes: Obtain scene description information corresponding to the test task of autonomous driving test, wherein the scene description information is used to indicate information related to the scene occurrence area and scene participants; Obtain a test area that matches the scene description information from the scene map, and set a driving route that matches the test task for the test object in the test area; Extract the atomic actions of the scene participants from the scene description information; An experience replay pool corresponding to the reinforcement learning model is initialized, and sample data is constructed and stored in the experience replay pool based on the scene description information, the test area, and the driving route. The reinforcement learning model includes: an actor network, a target network corresponding to the actor network, a first commentator network, a target network corresponding to the first commentator network, a second commentator network, and a target network corresponding to the second commentator network. The actor network processes the parameter values ​​of the current environment state and outputs the parameter values ​​of the current atomic action. The first commentator network and the second commentator network process the input sample data and output the first evaluation value and the second evaluation value corresponding to the respective atomic action. The target network is used to stabilize the expected return of the reinforcement learning model. The reinforcement learning model is updated by extracting sample data from the experience replay pool, and the determined parameter values ​​of the atomic actions are obtained through the updated reinforcement learning model; wherein, the step of extracting sample data from the experience replay pool to update the reinforcement learning model and obtaining the determined parameter values ​​of the atomic actions through the updated reinforcement learning model includes: For atomic action a i A target number of sample data are extracted from the experience replay pool for updating the reinforcement learning model; The training parameter values ​​of the corresponding environmental state and the training parameter values ​​of the corresponding atomic action in each sample data are processed through the first commentator network and the second commentator network to obtain the first evaluation value and the second evaluation value corresponding to the corresponding atomic action respectively. The target network of the actor network processes the training parameter values ​​of the next environmental state of the corresponding environmental state in each sample data to obtain the predicted parameter values ​​of the next atomic action of the corresponding atomic action. The training parameter values ​​of the next environmental state of the corresponding environmental state and the prediction parameter values ​​of the next atomic action of the corresponding atomic action in each sample data are processed through the target networks of the first commentator network and the second commentator network to obtain the first evaluation value and the second evaluation value corresponding to the next atomic action of the corresponding atomic action. Based on the reward corresponding to the corresponding atomic action, the first evaluation value corresponding to the corresponding atomic action, and the first evaluation value corresponding to the next atomic action in all sample data, the loss function of the first commentator network is constructed; based on the reward corresponding to the corresponding atomic action, the second evaluation value corresponding to the corresponding atomic action, and the second evaluation value corresponding to the next atomic action in all sample data, the loss function of the second commentator network is constructed. Backward gradient propagation is performed according to the loss functions of the first commentator network and the second commentator network respectively to update the parameters of the first commentator network and the second commentator network; Based on the first and second evaluation values ​​corresponding to the atomic actions in all sample data, the loss function of the actor network is constructed. Backward gradient propagation is performed according to the loss function of the actor network to update the parameters of the actor network; The environmental state s is assessed through the actor network. i The determined parameter values ​​are processed to obtain the atomic action a. i The determined parameter values; wherein, the environmental state s i The process of obtaining the parameter values ​​includes: using an autonomous driving simulation test platform to determine the environmental state s according to the test area and the driving route. i-1 Determine the parameter values ​​and atomic actions a i-1 The determined parameter values ​​are processed to obtain the environmental state s. i The initial environmental state s1 is determined based on the scene description information, and the initial atomic action a1 is determined by the actor network through processing the input environmental state s1. Based on the determined parameter values ​​of all atomic actions, a key scenario is generated to achieve the test task. The key scenario refers to the test scenario in which the probability of collision between the tested object and the scenario participants is the highest.

2. The method according to claim 1, characterized in that, The step of constructing sample data and storing it in the experience playback pool based on the scene description information, the test area, and the driving route includes: For atomic action a i The atomic action a is obtained through a random generation function. i The training parameter values, where i is a positive integer; By connecting to an autonomous driving simulation test platform containing the tested object, and according to the test area and the driving route, the environmental state s is tested. i The training parameter values ​​and the atomic action a i The training parameter values ​​are processed to obtain the environmental state s. i+1 The training parameter values, the atomic action a i The corresponding training rewards and collision states; wherein, the training parameter values ​​of the initial environment state s1 are obtained based on the scene description information; Based on i and the environmental state s i The training parameter values, the atomic action a i The training parameter values, the environmental state s i+1 The training parameter values, the atomic action a i The corresponding training rewards and collision states are used to construct the atomic action a. i Corresponding sample data; Let i = i + 1, starting from the atomic action a i The atomic action a is obtained through a random generation function. i The step of training parameter values ​​is executed again until the sample data has been constructed for all atomic actions; The process of constructing sample data corresponding to all atomic actions is repeated for u rounds, and all sample data constructed in the u rounds are stored in the experience replay pool.

3. The method according to claim 2, characterized in that, The total number of atomic actions is n, where n is a positive integer greater than or equal to i; the atomic action a is obtained through a random generation function. i The training parameter values ​​include: From the atomic action a through the random generation function i Randomly sample within the parameter range, and add a padding mask and a timing mask to the sampling results to obtain the atomic action a. i The training parameter values; The filling mask is used to fill the sampling result with parameter values ​​of n-1 atomic actions, so that the atomic action a i The training parameter values ​​are represented by an action space of length n; the temporal mask is used to define the execution timing of the atomic action a within the action space. i The parameter values ​​of the subsequent ni atomic actions are set to zero, and the execution timing in the action space is set to the atomic action a. i The parameter values ​​of the previous i-1 atomic actions are set to the training parameter values ​​of the corresponding atomic actions.

4. The method according to claim 2, characterized in that, The atomic action a i The corresponding training reward is the reward obtained by the reinforcement learning model during training, and the calculation process includes: For each scene participant, the longitudinal collision time between the tested object and the corresponding scene participant is calculated based on the difference between the longitudinal coordinate of the tested object and the longitudinal coordinate of the corresponding scene participant, as well as the difference between the longitudinal velocity of the tested object and the longitudinal velocity of the corresponding scene participant. For each scene participant, the lateral collision time between the tested object and the corresponding scene participant is calculated based on the difference between the horizontal coordinate of the tested object and the horizontal coordinate of the corresponding scene participant, as well as the difference between the lateral velocity of the tested object and the lateral velocity of the corresponding scene participant. Based on the longitudinal collision time and the lateral collision time, calculate the risk factor between the tested object and each scene participant; The sum of the maximum value of the risk factors among the tested object and all participants in the scenario, and the additional reward, is used as the reward obtained by the reinforcement learning model.

5. The method according to claim 4, characterized in that, The calculation of risk factors between the tested object and each participant in the scene, based on the longitudinal collision time and the lateral collision time, includes: For each scenario participant, if the longitudinal collision time is greater than zero, or the lateral collision time is greater than zero, then the opposite of the maximum value of the longitudinal collision time and the lateral collision time is used as the risk factor between the tested object and the corresponding scenario participant. For each scenario participant, if the longitudinal collision time is less than or equal to zero and the lateral collision time is less than or equal to zero, then the reciprocal of the difference between the absolute value of the difference between the longitudinal collision time and the lateral collision time and the minimum value of the longitudinal collision time and the lateral collision time is used as the risk factor between the tested object and the corresponding scenario participant.

6. The method according to claim 2, characterized in that, The experience replay pool includes a positive sample buffer and a negative sample buffer; storing all sample data constructed in the u rounds into the experience replay pool includes: Classify all sample data constructed in the u rounds according to the collision state, and obtain at least one positive sample and at least one negative sample; The at least one positive sample is stored in the positive sample buffer, and the at least one negative sample is stored in the negative sample buffer; Wherein, the positive sample refers to the sample data in which the tested object collides with the scene participants, and the negative sample refers to the sample data in which the tested object does not collide with the scene participants.

7. The method according to claim 1, characterized in that, The experience replay pool includes a positive sample buffer and a negative sample buffer; the step of extracting a target number of sample data from the experience replay pool for updating the reinforcement learning model includes: A first number of positive samples are drawn from the positive sample buffer, and a second number of negative samples are drawn from the negative sample buffer. The target number of sample data includes the first number of positive samples and the second number of negative samples, and the sum of the first number and the second number is equal to the target number. Wherein, the positive sample refers to the sample data in which the tested object collides with the scene participants, and the negative sample refers to the sample data in which the tested object does not collide with the scene participants.

8. The method according to claim 1, characterized in that, The parameters of the target network of the first commentator network and the target network of the second commentator network are updated based on the parameters of the commentator network with the smaller output evaluation value in the first commentator network and the second commentator network; and the parameter update frequency of the target network of the first commentator network and the target network of the second commentator network is lower than the parameter update frequency of the first commentator network and the second commentator network. The parameters of the target network of the actor network are updated based on the parameters of the actor network; and the parameter update frequency of the target network of the actor network is lower than the parameter update frequency of the actor network, and the parameter update frequency of the actor network is lower than the parameter update frequency of the first commentator network and the second commentator network.

9. The method according to claim 1, characterized in that, The environmental state s is obtained through the actor network. i The determined parameter values ​​are processed to obtain the atomic action a. i After determining the parameter values, the following is also included: The autonomous driving simulation test platform is used to test the environmental state s according to the test area and the driving route. i The determined parameter values ​​and the atomic action a i The determined parameter values ​​are processed to obtain the environmental state s. i+1 The determined parameter value, the atomic action a i The corresponding training rewards and collision states; Based on i and the environmental state s i The determined parameter value, the atomic action a i The determined parameter values, the environmental state s i+1 The determined parameter value, the atomic action a i The corresponding training rewards and collision states are used to construct the atomic action a. i Corresponding sample data; The atomic action a i The corresponding sample data is stored in the experience replay pool to update the experience replay pool.

10. The method according to any one of claims 2 to 9, characterized in that, The parameter values ​​of the atomic action include: the speed of the scene participant and the target position of the scene participant; The parameters of the environmental state include: the lateral distance between the tested object and the scene participants, the longitudinal distance between the tested object and the scene participants, the travel distance of the tested object, the travel distance of the scene participants, and the collision state between the tested object and the scene participants.

11. A key scenario generation device for autonomous driving, characterized in that, The device includes: The information acquisition module is used to acquire scene description information corresponding to the test task of autonomous driving test. The scene description information is used to indicate information related to the scene occurrence area and scene participants. The driving setting module is used to obtain a test area that matches the scene description information from the scene map, and set a driving route that matches the test task for the test object in the test area; An action extraction module is used to extract atomic actions of the scene participants from the scene description information; A sample construction module is used to initialize the experience replay pool corresponding to the reinforcement learning model, and construct sample data based on the scene description information, the test area, and the driving route to store it in the experience replay pool; wherein, the reinforcement learning model includes: an actor network, a target network corresponding to the actor network, a first commentator network, a target network corresponding to the first commentator network, a second commentator network, and a target network corresponding to the second commentator network; the actor network is used to process the parameter values ​​of the input current environment state and output the parameter values ​​of the current atomic action; the first commentator network and the second commentator network are used to process the input sample data and output the first evaluation value and the second evaluation value corresponding to the corresponding atomic action, respectively; the target network is used to stabilize the expected return of the reinforcement learning model; The parameter acquisition module is used to extract sample data from the experience replay pool to update the reinforcement learning model, and to obtain the definite parameter values ​​of the atomic actions through the updated reinforcement learning model; wherein, the step of extracting sample data from the experience replay pool to update the reinforcement learning model, and obtaining the definite parameter values ​​of the atomic actions through the updated reinforcement learning model, includes: For atomic action a i A target number of sample data are extracted from the experience replay pool for updating the reinforcement learning model; The training parameter values ​​of the corresponding environmental state and the training parameter values ​​of the corresponding atomic action in each sample data are processed through the first commentator network and the second commentator network to obtain the first evaluation value and the second evaluation value corresponding to the corresponding atomic action respectively. The target network of the actor network processes the training parameter values ​​of the next environmental state of the corresponding environmental state in each sample data to obtain the predicted parameter values ​​of the next atomic action of the corresponding atomic action. The training parameter values ​​of the next environmental state of the corresponding environmental state and the prediction parameter values ​​of the next atomic action of the corresponding atomic action in each sample data are processed through the target networks of the first commentator network and the second commentator network to obtain the first evaluation value and the second evaluation value corresponding to the next atomic action of the corresponding atomic action. Based on the reward corresponding to the corresponding atomic action, the first evaluation value corresponding to the corresponding atomic action, and the first evaluation value corresponding to the next atomic action in all sample data, the loss function of the first commentator network is constructed; based on the reward corresponding to the corresponding atomic action, the second evaluation value corresponding to the corresponding atomic action, and the second evaluation value corresponding to the next atomic action in all sample data, the loss function of the second commentator network is constructed. Backward gradient propagation is performed according to the loss functions of the first commentator network and the second commentator network respectively to update the parameters of the first commentator network and the second commentator network; Based on the first and second evaluation values ​​corresponding to the atomic actions in all sample data, the loss function of the actor network is constructed. Backward gradient propagation is performed according to the loss function of the actor network to update the parameters of the actor network; The environmental state s is assessed through the actor network. i The determined parameter values ​​are processed to obtain the atomic action a. i The determined parameter values; wherein, the environmental state s i The process of obtaining the parameter values ​​includes: using an autonomous driving simulation test platform to determine the environmental state s according to the test area and the driving route. i-1 Determine the parameter values ​​and atomic actions a i-1 The determined parameter values ​​are processed to obtain the environmental state s. i The initial environmental state s1 is determined based on the scene description information, and the initial atomic action a1 is determined by the actor network through processing the input environmental state s1. The scene generation module is used to generate key scenes for implementing the test task based on the determined parameter values ​​of all atomic actions. The key scene refers to the test scene with the highest probability of collision between the tested object and the scene participants.

12. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images