Training method of reinforcement learning model for automatic driving and vehicle meeting method

Abstract

The invention provides a training method of a reinforcement learning model for automatic driving and a vehicle meeting method, and the training method comprises the steps: obtaining an initial state space parameter of a current simulation environment, inputting the initial state space parameter into an initial reinforcement learning model, and enabling the initial reinforcement learning model to output a prediction action for controlling the execution of a target vehicle. An action award is determined based on the predicted action and an execution action of the other vehicle. And after the target vehicle is controlled to execute the prediction action, acquiring a next state space parameter of the current simulation environment. And repeatedly executing the steps of inputting the next state space parameter into the initial reinforcement learning model and the subsequent process until the vehicle meeting process of the vehicle meeting scene is finished. And determining a reward function based on a plurality of action rewards in the meeting process. And training the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a training method and a meeting method of a reinforcement learning model for autonomous driving. Background Art

[0002] Autonomous driving refers to the process of leveraging technologies such as sensors, computer vision, artificial intelligence, and control systems to enable vehicles to autonomously perceive their environment, make decisions, and complete driving tasks without a human driver. Meeting vehicles are common in autonomous driving scenarios, especially in narrow roads, illegally parked vehicles, and overtaking on the wrong side of the road. These decision-making tasks are extremely challenging.

[0003] The mainstream autonomous driving solution for current meeting scenarios is an algorithm designed based on expert rules. When designing expert rules, traffic regulations can be referenced to develop solutions (i.e., autonomous driving algorithms (models)) for different meeting scenarios. This allows the autonomous driving algorithm to control the vehicle and complete the meeting task. However, designing an autonomous driving algorithm based on expert rules requires considering a variety of meeting scenarios in advance. Manually designing an autonomous driving algorithm for each scenario consumes considerable manpower and time. Furthermore, during the design process, the algorithm's parameters or logic must be manually adjusted based on the algorithm's results, affecting the algorithm's accuracy. Summary of the Invention

[0004] In view of this, the present application is dedicated to providing a training method and a meeting method of a reinforcement learning model for autonomous driving, so as to improve the efficiency and accuracy of obtaining a model for autonomous driving.

[0005] In a first aspect, the present application provides a method for training a reinforcement learning model for autonomous driving, the method comprising:

[0006] Inputting initial state space parameters of a current simulation environment into an initial reinforcement learning model, and outputting a predicted action of a target vehicle, wherein the current simulation environment represents a meeting scenario between the target vehicle and other vehicles in the simulation environment, and the initial reinforcement learning model is deployed on the target vehicle;

[0007] determining an action reward based on the predicted action and the executed action of the other vehicle;

[0008] After controlling the target vehicle to execute the predicted action, obtaining next state space parameters of the current simulation environment;

[0009] Repeating the inputting of the next state space parameter into the initial reinforcement learning model and subsequent processes until the meeting process is completed;

[0010] Determining a reward function based on the plurality of action rewards during the meeting process;

[0011] The initial reinforcement learning model is trained based on the reward function to obtain a trained target reinforcement learning model.

[0012] In one possible implementation, training the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model includes:

[0013] Reversely updating the parameters of the initial reinforcement learning model based on the reward function, and repeating the above training process until a preset number of times is reached, and determining a preferred reinforcement learning model based on multiple reward functions corresponding to the preset number of training processes;

[0014] Modifying the current simulation environment to obtain another simulation environment, wherein the other simulation environment represents a vehicle-meeting scene different from the vehicle-meeting scene represented by the current simulation environment;

[0015] The preferred reinforcement learning model is deployed on a target vehicle in the other simulation environment, and the preferred reinforcement learning model is trained based on the other simulation environment to obtain the target reinforcement learning model.

[0016] In a possible implementation, the method further includes:

[0017] After deploying the target reinforcement learning model on a target real vehicle traveling in a real environment, when the target real vehicle is in a real meeting scenario with other real vehicles, obtaining multiple sets of state space parameters in the real meeting scenario, where the other real vehicle represents the vehicle in front of the target real vehicle or the vehicle to the left of the target real vehicle;

[0018] classifying the multiple sets of state space parameters in the simulation environment and the multiple sets of state space parameters in the real vehicle meeting scenario;

[0019] When the multiple groups of state space parameters in the real vehicle meeting scenario and the multiple groups of state space parameters in the simulation environment are of different categories, the target reinforcement learning model is trained based on the state space parameters of different categories in the real vehicle meeting scenario to obtain an optimized reinforcement learning model.

[0020] In one possible implementation, determining an action reward based on the predicted action and the executed action of the other vehicle includes:

[0021] Determining a first action reward corresponding to the predicted action based on a first preset action reward strategy for the target vehicle, wherein the first preset action reward strategy includes rewards corresponding to different actions of the target vehicle;

[0022] Determining a second action reward corresponding to the executed action based on a second preset action reward strategy of the other vehicle, wherein the second preset action reward strategy includes rewards corresponding to different actions of the other vehicle;

[0023] The action reward is determined based on the first action reward and the second action reward.

[0024] In a possible implementation, determining a reward function based on the plurality of action rewards during the meeting process includes:

[0025] Determining a reward score corresponding to an evaluation indicator of the meeting process, the evaluation indicator including at least one of: whether the meeting is successful, the duration of the meeting, the time the target vehicle stops and waits, and the acceleration smoothness of the target vehicle;

[0026] The reward function is determined based on the reward score and a plurality of the action rewards.

[0027] In a possible implementation, when the evaluation indicator includes a successful meeting, the reward score corresponding to the successful meeting is a first preset score;

[0028] When the evaluation index includes a failure to meet another vehicle, the reward score corresponding to the failure to meet another vehicle is a second preset score, and the second preset score is a negative number;

[0029] When the evaluation index includes the duration of the meeting, the bonus score corresponding to the duration of the meeting is determined based on the duration of the meeting;

[0030] When the evaluation index includes the parking waiting time of the target vehicle and the parking waiting time of the target vehicle is greater than the preset time, the reward score corresponding to the parking waiting time of the target vehicle is a third preset score, and the third preset score is a negative number;

[0031] When the evaluation index includes acceleration smoothness of the target vehicle, the bonus score corresponding to the acceleration smoothness of the target vehicle is determined based on the acceleration of the target vehicle.

[0032] In a possible implementation, the initial state space parameters include: state parameters of the target vehicle, state parameters of the other vehicles, and lane parameters;

[0033] The state parameter of the target vehicle includes at least one of position, heading angle, speed, or size;

[0034] The other vehicles include: the left vehicle of the target vehicle and / or the vehicle in front of the target vehicle;

[0035] The lane parameters include: the area of ​​the lane line or the lane direction.

[0036] In a second aspect, the present application provides a method for meeting an autonomous vehicle, the method comprising:

[0037] When the target real vehicle is in a real meeting scene with other real vehicles, obtaining initial state space parameters in the real meeting scene, where the other real vehicles represent the vehicle in front of the target real vehicle and / or the vehicle to the left of the target real vehicle;

[0038] Inputting the initial state space parameters into a target reinforcement learning model, and outputting a predicted action of the target real vehicle;

[0039] After the target real vehicle executes the predicted action, obtaining the next state space parameters of the real meeting scene;

[0040] Repeating the inputting of the next state space parameter into the target reinforcement learning model and subsequent processes until the meeting process is completed;

[0041] The target reinforcement learning model is trained based on the method described in any one of the implementation methods of the first aspect above.

[0042] In a third aspect, the present application provides a training device for a reinforcement learning model for autonomous driving, the device comprising:

[0043] a first prediction unit, configured to input initial state space parameters of a current simulation environment into an initial reinforcement learning model, and output a predicted action of a target vehicle, wherein the current simulation environment represents a meeting scenario between the target vehicle and other vehicles in the simulation environment, and the initial reinforcement learning model is deployed on the target vehicle;

[0044] a first determining unit, configured to determine an action reward based on the predicted action and the executed action of the other vehicle;

[0045] A first acquisition unit is configured to acquire next state space parameters of the current simulation environment after controlling the target vehicle to execute the predicted action;

[0046] a first iterative unit, configured to repeatedly input the next state space parameter into the initial reinforcement learning model and subsequent processes until the meeting process is completed;

[0047] A second determining unit is configured to determine a reward function based on the plurality of action rewards during the meeting process;

[0048] A training unit is used to train the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model.

[0049] In a fourth aspect, the present application provides an autonomous driving meeting device, the device comprising:

[0050] a second acquisition unit, configured to acquire initial state space parameters in a real meeting scene when the target real vehicle is in a real meeting scene with other real vehicles, wherein the other real vehicles represent a preceding vehicle of the target real vehicle and / or a vehicle to the left of the target real vehicle;

[0051] a second prediction unit, configured to input the initial state space parameters into a target reinforcement learning model and output a predicted action of the target real vehicle;

[0052] The second acquisition unit is further configured to acquire next state space parameters of the real vehicle meeting scene after the target real vehicle performs the predicted action;

[0053] The second iterative unit is used to repeatedly execute the input of the next state space parameter into the target reinforcement learning model and subsequent processes until the meeting process is completed; wherein, the target reinforcement learning model is trained based on the method described in any implementation method of the first aspect above.

[0054] In a fifth aspect, the present application provides an electronic device, the device comprising: a memory and a processor;

[0055] The memory is used to store relevant program codes;

[0056] The processor is used to call the program code and execute the method described in any one of the implementation modes of the first aspect or the second aspect.

[0057] In a sixth aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method described in any one of the implementation methods of the first aspect or the second aspect above.

[0058] In a seventh aspect, the present application provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements the method described in any one of the implementation methods of the first aspect or the second aspect.

[0059] In the above-described implementation of the present application, to train a reinforcement learning model for a meeting scenario for autonomous driving, a simulation environment can be pre-created, and a meeting scenario involving a target vehicle and other vehicles can be established within the simulation environment to train the reinforcement learning model. Specifically, the initial state-space parameters of the current simulation environment are first obtained and input into the initial reinforcement learning model. The initial reinforcement learning model then outputs a predicted action for controlling the target vehicle. The initial reinforcement learning model is deployed on the target vehicle to control the target vehicle. The actions executed by the other vehicles can then be obtained. Based on the predicted action and the actions executed by the other vehicles, an action reward is determined. After controlling the target vehicle to execute the predicted action, the next state-space parameters of the current simulation environment are obtained. Inputting the next state-space parameters into the initial reinforcement learning model and subsequent steps are repeated until the meeting scenario is complete. During the meeting, due to changes in the state-space parameters, the initial reinforcement learning model will output multiple predicted actions for controlling the target vehicle. Accordingly, the other vehicles will also generate multiple actions, thus determining multiple action rewards. A reward function is determined based on the multiple action rewards during the meeting. Thus, the initial reinforcement learning model can be trained based on the reward function until the training cutoff condition is met, and a trained target reinforcement learning model is obtained. Through the method provided in the present application, the meeting scene in the simulation environment can be used to automatically train the reinforcement learning model that controls the target vehicle to complete the meeting, and the parameters can be modified more conveniently in the simulation environment to train the reinforcement learning model based on a variety of meeting scenes. There is no need to manually design different autonomous driving algorithms, which can improve the efficiency and accuracy of obtaining the reinforcement learning model. Since it is a process of automatically training the reinforcement learning model, no manual adjustment is required, and the efficiency and accuracy of obtaining the reinforcement learning model can also be improved, thereby improving the accuracy of controlling the vehicle to complete the meeting based on the reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments provided in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0061] Figure 1 A flowchart of a method for training a reinforcement learning model for autonomous driving provided in an embodiment of the present application.

[0062] Figure 2a A schematic diagram of a meeting scene provided in an embodiment of the present application.

[0063] Figure 2b A schematic diagram of another meeting scenario provided in an embodiment of the present application.

[0064] Figure 3 A flowchart of an autonomous driving meeting method provided in an embodiment of the present application.

[0065] Figure 4 A schematic diagram of a training device for a reinforcement learning model for autonomous driving provided in an embodiment of the present application.

[0066] Figure 5 A schematic diagram of an autonomous driving meeting device provided in an embodiment of the present application.

[0067] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. The described embodiments are only exemplary implementation methods of the present application and are not all implementation methods. Those skilled in the art can combine the embodiments of the present application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of the present application.

[0069] The mainstream autonomous driving solution for current meeting scenarios is an algorithm designed based on expert rules. When designing expert rules, traffic regulations can be referenced to develop solutions (i.e., autonomous driving algorithms (models)) for different meeting scenarios. This allows the autonomous driving algorithm to control the vehicle and complete the meeting task. However, designing an autonomous driving algorithm based on expert rules requires considering a variety of meeting scenarios in advance. Manually designing an autonomous driving algorithm for each scenario consumes considerable manpower and time. Furthermore, during the design process, the algorithm's parameters or logic must be manually adjusted based on the algorithm's results, affecting the algorithm's accuracy.

[0070] Based on this, embodiments of the present application provide a method for training a reinforcement learning model for autonomous driving, thereby improving the efficiency and accuracy of acquiring models for autonomous driving. To train a reinforcement learning model for an autonomous driving meeting scenario, a simulation environment can be pre-created and a meeting scenario between a target vehicle and other vehicles can be established within the simulation environment to train the reinforcement learning model. Specifically, initial state space parameters of the current simulation environment are obtained and input into an initial reinforcement learning model, which then outputs a predicted action for controlling the target vehicle. The initial reinforcement learning model is deployed on the target vehicle to control the target vehicle. The actions executed by the other vehicles can then be obtained. Based on the predicted action and the actions executed by the other vehicles, an action reward is determined. After the target vehicle executes the predicted action, the next state space parameters of the current simulation environment are obtained. Inputting the next state space parameters into the initial reinforcement learning model and subsequent steps are repeated until the meeting scenario is complete. During the meeting, due to changes in the state space parameters, the initial reinforcement learning model will output multiple predicted actions for controlling the target vehicle. Accordingly, the other vehicles will also generate multiple actions, thus determining multiple action rewards. A reward function is determined based on the multiple action rewards during the meeting. Thus, the initial reinforcement learning model can be trained based on the reward function until the training cutoff condition is met, and a trained target reinforcement learning model is obtained. Through the method provided in the embodiment of the present application, the meeting scene in the simulation environment can be used to automatically train the reinforcement learning model that controls the target vehicle to complete the meeting, and the parameters can be modified more conveniently in the simulation environment to train the reinforcement learning model based on a variety of meeting scenes. There is no need to manually design different autonomous driving algorithms, which can improve the efficiency and accuracy of obtaining the reinforcement learning model. Since it is a process of automatically training the reinforcement learning model, no manual adjustment is required, and the efficiency and accuracy of obtaining the reinforcement learning model can also be improved, thereby improving the accuracy of controlling the vehicle to complete the meeting based on the reinforcement learning model.

[0071] The elements of reinforcement learning mainly include the agent, environment, state, action, and reward. In the embodiment of this application, the agent represents the reinforcement learning model, the environment represents the simulation environment, the state space parameters of the simulation environment are the state, which serves as the input of the reinforcement learning model, and the action is the output of the reinforcement learning model. After the reinforcement learning model outputs an action, it can obtain a reward, which is used to constrain the reinforcement learning strategy and update the parameters of the reinforcement learning model. Through continuous feedback updates, the reward is maximized.

[0072] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, a detailed introduction will be given below in conjunction with the drawings in the embodiments.

[0073] See also Figure 1As shown, it is a flowchart of a training method of a reinforcement learning model for autonomous driving provided in an embodiment of the present application.

[0074] Optionally, the method can be executed by an industrial computer, in which a simulation environment can be created and a reinforcement learning model can be deployed.

[0075] The method may include the following steps:

[0076] S101: Input the initial state space parameters of the current simulation environment into the initial reinforcement learning model and output the predicted action of the target vehicle.

[0077] In embodiments of the present application, a meeting scenario involving a target vehicle and other vehicles can be pre-established in a simulation environment, and an initial reinforcement learning model can be deployed on the target vehicle in the simulation environment to achieve control of the target vehicle. A meeting scenario can be understood as a scenario where the target vehicle meets a vehicle in an oncoming lane, or the target vehicle overtakes a preceding vehicle in the same lane by changing lanes. The simulation environment can be a three-dimensional simulation environment, used to simulate vehicles, roads, and environments in a real-world environment.

[0078] In an embodiment of the present application, the other vehicles may represent at least one of the preceding vehicle or the vehicle to the left of the target vehicle in a meeting scenario. That is, when there is a preceding vehicle of the target vehicle in the meeting scenario, the other vehicles may represent both the preceding vehicle and the vehicle to the left of the target vehicle. When there is no preceding vehicle of the target vehicle in the meeting scenario, the other vehicles may represent the vehicle to the left of the target vehicle. For example, when the target vehicle is meeting a vehicle from an oncoming lane, and there is also a vehicle in front of the target vehicle, in this meeting scenario, the other vehicles may represent both the vehicle in front of the target vehicle and the vehicle from the oncoming lane.

[0079] See also Figure 2a , which is a schematic diagram of a meeting scene provided by an embodiment of the present application. Vehicle 1 represents the target vehicle, vehicle 2 is the preceding vehicle of vehicle 1, vehicle 3 is the left vehicle of vehicle 1, vehicle 1 and vehicle 3 are meeting, and other vehicles include vehicle 2 and vehicle 3. Figure 2b FIG. 1 is a schematic diagram of another vehicle-meeting scenario provided by an embodiment of the present application, wherein vehicle 4 represents the target vehicle, vehicle 5 is the preceding vehicle of vehicle 4, vehicle 6 is the vehicle to the left of vehicle 4, and vehicles 4 and 6 are meeting each other. Other vehicles include vehicles 5 and 6.

[0080] To train the initial reinforcement learning model, the initial state space parameters of the current simulation environment can be obtained. These initial state space parameters may include the state parameters of the target vehicle, the state parameters of other vehicles, and lane parameters. The state parameters of the target vehicle include at least one of its position, heading angle, velocity, or size. The state parameters of other vehicles include at least one of their position, heading angle, velocity, or size. Lane parameters include the area of ​​the lane line and the lane direction.

[0081] The initial state space parameters are input into the initial reinforcement learning model, which then outputs a predicted action for the target vehicle, thereby controlling the target vehicle to execute the predicted action. The predicted action may include acceleration, steering angle, etc., meaning the vehicle is controlled to drive according to the acceleration or to turn according to the steering angle.

[0082] S102: Determine an action reward based on the predicted action and the executed actions of other vehicles.

[0083] In a meeting scenario, while controlling the target vehicle to execute a predicted action based on the initial reinforcement learning model, the actions of other vehicles can also be obtained. These actions can be measured by the target vehicle's sensors in the current simulation environment. After obtaining the actions of other vehicles, the action reward can be determined based on the target vehicle's predicted action and the actions of the other vehicles.

[0084] In one possible implementation, the action reward can be determined in the following manner: based on the first preset action reward strategy of the target vehicle, a first action reward corresponding to the predicted action is determined, wherein the first preset action reward strategy includes rewards corresponding to different actions of the target vehicle. That is, rewards corresponding to different actions are set in advance for all actions that may be performed by the target vehicle. Similarly, based on the second preset action reward strategy of other vehicles, a second action reward corresponding to the executed action is determined, and the second preset action reward strategy includes rewards corresponding to different actions of other vehicles. That is, rewards corresponding to different actions are set in advance for all actions that may be performed by other vehicles. Thus, the action reward can be determined based on the first action reward and the second action reward.

[0085] For example, for different actions of the target vehicle, a score corresponding to each action of the target vehicle is set as a reward. For different actions of other vehicles, a score corresponding to each action of other vehicles is set as a reward.

[0086] In an embodiment of the present application, the actions of the target vehicle / other vehicles can be pre-set to include: left turn acceleration, left turn deceleration, straight-ahead acceleration, straight-ahead deceleration, cruising, reversing, etc., where cruising means that the vehicle is driving in its original state. For each action of the target vehicle and other vehicles, a score is set as a reward. In one possible implementation, the scores corresponding to the same actions of the target vehicle and other vehicles can be set to be the same. For example, when the action of the target vehicle / other vehicle is left turn acceleration, the corresponding reward is 4 points; when the action of the target vehicle / other vehicle is left turn deceleration, the corresponding reward is 1 point; when the action of the target vehicle / other vehicle is straight-ahead acceleration, the corresponding reward is 2 points; when the action of the target vehicle / other vehicle is straight-ahead deceleration, the corresponding reward is 3 points; when the action of the target vehicle / other vehicle is cruising, the corresponding reward is 0 points; when the action of the target vehicle / other vehicle is reversing, the corresponding reward is 0 points.

[0087] Based on this, when determining the action reward based on the first action reward and the second action reward, the first action reward and the second action reward can be summed to obtain the action reward. For example, the reward factor corresponding to the first action reward of the target vehicle can be set to k1, the reward factor corresponding to the reward for the target vehicle's left vehicle performing the action can be set to k2, and the reward factor corresponding to the reward for the target vehicle's front vehicle performing the action can be set to k3. Then, the product of the reward factor corresponding to the first action reward and the first action reward is calculated, as well as the product of the reward factor corresponding to the second action reward and the second action reward, and the sum of the two products is calculated to obtain the action reward. Among them, k1, k2, and k3 are all positive numbers less than 1.

[0088] Among them, when the other vehicles include the target vehicle's left vehicle, the reward factor corresponding to the second action reward is k2; when the other vehicles include the target vehicle's front vehicle, the reward factor corresponding to the second action reward is k3; when the other vehicles include the target vehicle's left vehicle and the target vehicle's front vehicle, the second action reward includes the reward for the target vehicle's left vehicle performing the action and the reward for the target vehicle's front vehicle performing the action. Then, the product of the reward factor corresponding to the second action reward and the second action reward includes: the product of the target vehicle's left vehicle performing the action and k2, and the product of the target vehicle's front vehicle performing the action and k3. In other words, action reward = reward for the target vehicle's predicted action * k1 + reward for the target vehicle's left vehicle performing the action * k2 + reward for the target vehicle's front vehicle performing the action * k3.

[0089] It should be noted that the rewards corresponding to different actions of the target vehicle and other vehicles provided in the embodiment of the present application are merely exemplary and are not limited to the above form. The embodiment of the present application does not limit the specific values ​​of the reward points corresponding to different actions, and the rewards corresponding to the same action of the target vehicle and other vehicles can also be set to be different.

[0090] S103: After controlling the target vehicle to execute the predicted action, obtaining the next state space parameters of the current simulation environment.

[0091] When the target vehicle performs the predicted action, the state of the current simulation environment changes, and the next state space parameters of the current simulation environment can be obtained as the new input of the initial reinforcement learning model.

[0092] It should be noted that in the embodiment of the present application, the current simulation environment refers to the current meeting process. Before the current meeting process ends, even if the position, speed, etc. of the target vehicle / other vehicles change, the simulation environment in which the target vehicle and other vehicles are located still belongs to the current simulation environment.

[0093] S104: Repeat the process of inputting the next state space parameter into the initial reinforcement learning model and the subsequent process until the meeting process is completed.

[0094] In other words, the next state space parameters are re-input into the initial reinforcement learning model. Similarly, the initial reinforcement learning model can output the next predicted action of the target vehicle and obtain the next actions of other vehicles. Based on the next predicted and next actions, the next action reward is determined. Afterward, the new state space parameters of the current simulation environment can be obtained.

[0095] The above process is repeated until the end of the meeting process can be determined based on the state space parameters of the current simulation environment. Since the state space parameters include the position of the target vehicle and the positions of other vehicles, it is possible to determine whether the current meeting process is over by the position of the target vehicle and the positions of other vehicles. For example, when the target vehicle meets the left vehicle from the opposite lane, when the distance between the target vehicle and the left vehicle is less than the preset distance, the meeting process can be considered to be over. When the target vehicle attempts to overtake the preceding vehicle in the same lane (meeting the preceding vehicle), when the target vehicle is in front of the preceding vehicle, the meeting process can be considered to be over. Among them, the end of the meeting process can include two situations: successful meeting or failed meeting. Failed meeting can be understood as a collision between the target vehicle and other vehicles.

[0096] S105: Determine a reward function based on multiple action rewards during the meeting process.

[0097] During a meeting process, multiple predicted actions of the target vehicle can be determined based on multiple state space parameters of the current simulation environment, and multiple executed actions of other vehicles can also be obtained. Therefore, multiple action rewards can be determined based on multiple predicted actions and the executed actions corresponding to each predicted action. Therefore, based on multiple action rewards, the reward function corresponding to the meeting process can be determined. The goal of training the reinforcement learning model is to maximize the reward function.

[0098] In one possible implementation, evaluation indicators for the meeting process can be predetermined, and the reward scores corresponding to the evaluation indicators can be determined. The evaluation indicators are used to evaluate the quality of the meeting process, and the higher the reward scores corresponding to the evaluation indicators, the more successful the meeting process. The evaluation indicators for the meeting process can include at least one of: whether the meeting was successful (successful or unsuccessful), the duration of the meeting, the time the target vehicle stopped and waited, and the acceleration smoothness of the target vehicle. When the evaluation indicators for the meeting process include multiple evaluation indicators, the reward scores corresponding to the multiple evaluation indicators can be summed to serve as the reward score corresponding to the meeting process.

[0099] After determining the reward score for the evaluation metric, a reward function can be determined based on the reward score and the rewards for the multiple actions during the meeting process. For example, the reward function can be calculated by summing the rewards for the multiple actions with the reward score.

[0100] Optionally, when the evaluation metric includes a successful meeting, the reward score corresponding to a successful meeting is a first preset score. For example, the first preset score can be set to 10 points. When the evaluation metric includes a failed meeting, the reward score corresponding to a failed meeting is a second preset score, and the second preset score is a negative number. Since the goal of training a reinforcement learning model is to maximize the reward, the reward score corresponding to a failed meeting can be set to a smaller value, so that the reinforcement learning model training process can focus more on successful meeting. For example, the second preset score can be set to -100 points.

[0101] When the evaluation metric includes meeting duration, the bonus score corresponding to the meeting duration is determined based on the meeting duration. For example, the bonus score corresponding to the meeting duration can be set equal to the product of the meeting duration and a first preset ratio. Alternatively, because a longer meeting duration indicates a less smooth meeting process, the bonus score should be smaller. Therefore, the first preset ratio can be set to a negative number less than 0.

[0102] When the evaluation metric includes the target vehicle's parked waiting time, and the target vehicle's parked waiting time is greater than a preset time, the reward score corresponding to the target vehicle's parked waiting time is a third preset score, and the third preset score is a negative number. For example, the third preset score can be set to -0.2 points. When the target vehicle's parked waiting time is less than or equal to the preset time, the reward score corresponding to the target vehicle's parked waiting time can be set to 0, i.e., it has no effect on the determination of the reward function.

[0103] When the evaluation metric includes the target vehicle's acceleration smoothness, the bonus score corresponding to the target vehicle's acceleration smoothness is determined based on the target vehicle's acceleration. For example, the bonus score corresponding to the target vehicle's acceleration smoothness is equal to the product of the rate of change of the target vehicle's acceleration and a second preset ratio. The embodiments of this application do not limit the specific value of the second preset ratio. For example, the second preset ratio can be set to a positive number less than 1.

[0104] Optionally, when the evaluation indicators of the meeting process include one or more of meeting success, meeting failure, meeting duration, target vehicle stop waiting time, and target vehicle acceleration smoothness, the sum of the bonus scores corresponding to the evaluation indicators included in the meeting process can be calculated as the bonus score corresponding to the evaluation indicators of the meeting process.

[0105] S106: Train the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model.

[0106] After obtaining the reward function for the current meeting process, the parameters of the initial reinforcement learning model can be reversely updated based on the reward function using methods such as backpropagation. The initial state space parameters of the current simulation environment are then input into the modified reinforcement learning model, followed by subsequent training. That is, steps S101-S105 are repeated based on the modified reinforcement learning model until a training cutoff condition is met, resulting in a trained target reinforcement learning model. The training cutoff condition can be a predetermined number of training cycles, i.e., the number of repeated modifications of the reinforcement learning model parameters reaches a predetermined number, at which point the trained target reinforcement learning model is obtained.

[0107] In one possible implementation, in order to further improve the accuracy of the target reinforcement learning model so that the target reinforcement learning model can adapt to a variety of meeting scenarios and can more accurately control the target vehicle to complete the meeting in a variety of meeting scenarios, the current simulation environment can be modified, that is, a new meeting scenario can be constructed, and training can be continued in the new meeting scenario to obtain the target reinforcement learning model.

[0108] Specifically, step S106, "Training the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model," can be implemented by reversely updating the parameters of the initial reinforcement learning model based on the reward function of the current meeting process, and repeating the training process of steps S101-S105 until a preset number of times is reached. Each meeting process corresponds to a training process, and each training process can correspond to a corresponding reward function. The preferred reinforcement learning model can be determined based on the multiple reward functions corresponding to the preset number of training processes. For example, the reinforcement learning model corresponding to the maximum value among the multiple reward functions can be determined as the preferred reinforcement learning model.

[0109] The current simulation environment is then modified to obtain another simulation environment, wherein the other simulation environment represents a different meeting scene than the meeting scene represented by the current simulation environment. That is, the current simulation environment is modified to obtain a new meeting scene. Optionally, the modification of the current simulation environment can be based on an initial state of the current simulation environment. For example, the position / speed / size of the target vehicle, the position / speed / size of other vehicles, or the area of ​​lane lines corresponding to the initial state of the current simulation environment can be modified.

[0110] After obtaining the other simulation environments, the preferred reinforcement learning model can be deployed on the target vehicle in the other simulation environments. The preferred reinforcement learning model is then trained based on the other simulation environments to obtain the target reinforcement learning model. In other words, the training process of steps S101-S106 is performed based on the other simulation environments and the preferred reinforcement learning model.

[0111] Optionally, after completing the above training process, another preferred reinforcement learning model can be obtained (for example, at this time the model can handle the meeting under two meeting scenarios). Based on this, other simulation environments can be further modified to obtain different meeting scenarios represented by another simulation environment, and the training process of steps S101-S106 can be continued using another simulation environment and another preferred reinforcement learning model. After the number of times the above-mentioned modified simulation environment is repeated to execute the training process reaches a threshold, the simulation environment can no longer be modified, and the trained target reinforcement learning model is obtained. Through the above process, the generalization ability of the reinforcement learning model can be improved, so that the reinforcement learning model can adapt to a variety of meeting scenarios, improve the accuracy of the reinforcement learning model, and improve the accuracy of controlling the target vehicle to complete the meeting based on the reinforcement learning model.

[0112] In one possible implementation, in order to further improve the accuracy of the reinforcement learning model, after the target reinforcement learning model is deployed in a real-world vehicle-passing scenario, the target reinforcement learning model can be optimized based on feedback from the real environment.

[0113] Specifically, the target reinforcement learning model can be deployed on a target real vehicle traveling in a real environment. When the target real vehicle and other real vehicles are in a real meeting scene, multiple sets of state space parameters in the real meeting scene are obtained. Among them, the other real vehicles represent the front vehicle of the target real vehicle and / or the left vehicle of the target real vehicle. The real meeting scene can represent a meeting scene in a real environment, and the target real vehicle and the other real vehicles have a safe meeting space. For example, the target real vehicle has a certain safety distance from the other real vehicles, and the relative speed of the other real vehicles and the target real vehicle is less than the preset speed. Among them, the specific form of the real meeting scene can be determined in combination with actual needs, and the embodiments of the present application do not limit this.

[0114] Among them, the state space parameters in the real meeting scenario are of the same type as the state space parameters in the simulation environment, and may include the state parameters of the target real vehicle, the state parameters of other real vehicles, and the real lane parameters, etc. The state parameters of the target real vehicle / other real vehicles include: at least one of: position, heading angle, speed, or size.

[0115] According to the above embodiment, when training a reinforcement learning model based on a simulation environment, multiple sets of state-space parameters for the simulation environment can be obtained. Therefore, the multiple sets of state-space parameters for the simulation environment can be classified with the multiple sets of state-space parameters in a real-world meeting scenario. When the multiple sets of state-space parameters in the real-world meeting scenario and the multiple sets of state-space parameters in the simulation environment are of different categories, the target reinforcement learning model is trained based on the state-space parameters of the different categories in the real-world meeting scenario to obtain an optimized reinforcement learning model.

[0116] The multiple sets of state-space parameters for the simulation environment can be classified to obtain a first category set. The multiple sets of state-space parameters for the real environment can be classified to obtain a second category set. If a category in the second category set does not exist in the first category set, this indicates that the multiple sets of state-space parameters in the real-world meeting scenario and the multiple sets of state-space parameters in the simulation environment are of different categories. For example, for each set of state-space parameters in the simulation environment, the relative coordinates of the target vehicle and the other vehicles, the relative speed of the target vehicle and the other vehicles, and the relative heading difference between the target vehicle and the other vehicles can be calculated. The relative coordinates, relative speed, relative heading difference, size of the target vehicle, and size of the other vehicles can then be combined into a one-dimensional vector. In this way, multiple one-dimensional vectors can be obtained for the multiple sets of state-space parameters in the simulation environment. These multiple one-dimensional vectors can then be clustered to obtain a first category set consisting of multiple categories, each of which includes one or more one-dimensional vectors. Similarly, a second category set corresponding to the real environment can also be obtained, which can include multiple categories.

[0117] Alternatively, multiple groups of state-space parameters in the real meeting scenario and multiple groups of state-space parameters in the simulation environment can be classified at the same time. When there is only the state-space parameters in the real meeting scenario but no state-space parameters in the simulation environment in a certain category, it means that the multiple groups of state-space parameters in the real meeting scenario and the multiple groups of state-space parameters in the simulation environment are in different categories.

[0118] Optionally, when training the target reinforcement learning model based on state-space parameters of different categories in a real-world meeting scenario, a meeting scenario in a target simulation environment can be created based on the state-space parameters of different categories in the real-world meeting scenario. The target reinforcement learning model can be trained in the target simulation environment to obtain an optimized reinforcement learning model. The process of training the target reinforcement learning model can be found in the above embodiment and will not be further described here.

[0119] In one possible implementation, the ratio of the number of different state-space parameters in the real-world meeting scenario and the simulation environment to the number of multiple sets of state-space parameters in the simulation environment can be calculated. When this ratio is less than a preset value, it indicates that the real-world meeting scenario is relatively close to the simulation environment, and the degree of modification to the target reinforcement learning model is relatively small. The target reinforcement learning model can be trained based on all the different state-space parameters in the real-world meeting scenario. When this ratio is greater than or equal to the preset value, it indicates that the real-world meeting scenario is significantly different from the simulation environment, and the degree of modification to the target reinforcement learning model is relatively large. If the target reinforcement learning model is trained based on all the different state-space parameters, it is likely to overfit the target reinforcement learning model, affecting its performance in the original training data. Therefore, the target reinforcement learning model can be trained based on a portion of the different state-space parameters in the real-world meeting scenario, and the parameters of the target reinforcement learning model can be updated based on the learning rate k. The learning rate k represents a hyperparameter used to control the update step size of the target reinforcement learning model parameters in each iteration and the amplitude of parameter adjustment of the target reinforcement learning model during the optimization process.

[0120] Through the method provided in the embodiment of the present application, the reinforcement learning model for controlling the target vehicle to complete the meeting can be automatically trained using the meeting scene in the simulation environment, and the parameters can be modified more conveniently in the simulation environment so as to train the reinforcement learning model based on a variety of meeting scenes. There is no need to manually design different autonomous driving algorithms to adapt to a variety of meeting scenes, which can improve the efficiency and accuracy of obtaining the reinforcement learning model. Since it is a process of automatically training the reinforcement learning model, no manual adjustment is required, and the efficiency and accuracy of obtaining the reinforcement learning model can also be improved, thereby improving the accuracy of controlling the vehicle to complete the meeting based on the reinforcement learning model. After obtaining the trained target reinforcement learning model, the target reinforcement learning model can also be optimized based on the state space parameters of the real environment to further improve the accuracy of the reinforcement learning model.

[0121] Based on the above method embodiment, the present application embodiment also provides an automatic driving meeting method. Figure 3 As shown, it is a flow chart of an automatic driving meeting method provided in an embodiment of the present application.

[0122] The method may include the following steps:

[0123] S301: When the target real vehicle and other real vehicles are in a real meeting scene, initial state space parameters in the real meeting scene are obtained.

[0124] The other real vehicles refer to the preceding vehicle of the target real vehicle and / or the left vehicle of the target real vehicle.

[0125] S302: Input the initial state space parameters into the target reinforcement learning model and output the predicted action of the target real vehicle.

[0126] S303: After the target real vehicle performs the predicted action, the next state space parameters of the real meeting scene are obtained.

[0127] S304: Repeat the process of inputting the next state space parameter into the target reinforcement learning model and the subsequent process until the meeting process is completed.

[0128] The target reinforcement learning model is trained based on the method described in the above method embodiment.

[0129] Based on the above method embodiment, the present application embodiment also provides a training device for a reinforcement learning model for autonomous driving. Figure 4 , which is a schematic diagram of a training device for a reinforcement learning model for autonomous driving provided in an embodiment of the present application.

[0130] The apparatus 400 comprises:

[0131] A first prediction unit 401 is configured to input initial state space parameters of a current simulation environment into an initial reinforcement learning model, and output a predicted action of a target vehicle, wherein the current simulation environment represents a meeting scenario between the target vehicle and other vehicles in the simulation environment, and the initial reinforcement learning model is deployed on the target vehicle;

[0132] A first determining unit 402 is configured to determine an action reward based on the predicted action and the executed action of the other vehicle;

[0133] A first acquisition unit 403 is configured to acquire next state space parameters of the current simulation environment after controlling the target vehicle to perform the predicted action;

[0134] A first iterative unit 404 is configured to repeatedly input the next state space parameter into the initial reinforcement learning model and subsequent processes until the meeting process is completed;

[0135] A second determining unit 405 is configured to determine a reward function based on the plurality of action rewards during the meeting process;

[0136] The training unit 406 is configured to train the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model.

[0137] In one possible implementation, the training unit 406 is specifically used to reversely update the parameters of the initial reinforcement learning model based on the reward function, and repeatedly perform the above training process until a preset number of times is reached, and determine the preferred reinforcement learning model based on multiple reward functions corresponding to the preset number of training processes; modify the current simulation environment to obtain other simulation environments, and the meeting scenes represented by the other simulation environments are different from the meeting scenes represented by the current simulation environment; deploy the preferred reinforcement learning model on the target vehicle in the other simulation environment, and train the preferred reinforcement learning model based on the other simulation environment to obtain the target reinforcement learning model.

[0138] In one possible implementation, the device further includes: an optimization training unit, configured to deploy the target reinforcement learning model on a target real vehicle traveling in a real environment, and when the target real vehicle and other real vehicles are in a real meeting scene, obtain multiple sets of state space parameters in the real meeting scene, wherein the other real vehicles represent the preceding vehicle of the target real vehicle and / or the left vehicle of the target real vehicle; classify the multiple sets of state space parameters in the simulation environment and the multiple sets of state space parameters in the real meeting scene; when the multiple sets of state space parameters in the real meeting scene and the multiple sets of state space parameters in the simulation environment are of different categories, train the target reinforcement learning model based on the state space parameters of different categories in the real meeting scene to obtain an optimized reinforcement learning model.

[0139] In one possible implementation, the first determination unit 402 is specifically used to determine a first action reward corresponding to the predicted action based on a first preset action reward strategy for the target vehicle, the first preset action reward strategy including rewards corresponding to different actions of the target vehicle; determine a second action reward corresponding to the executed action based on a second preset action reward strategy for the other vehicles, the second preset action reward strategy including rewards corresponding to different actions of the other vehicles; and determine the action reward based on the first action reward and the second action reward.

[0140] In one possible implementation, the second determination unit 405 is specifically configured to determine a reward score corresponding to an evaluation indicator of the meeting process, where the evaluation indicator includes at least one of: whether the meeting is successful, the duration of the meeting, the waiting time of the target vehicle, and the acceleration smoothness of the target vehicle; and determine the reward function based on the reward score and the multiple action rewards.

[0141] In one possible implementation, when the evaluation indicator includes a successful meeting, the reward score corresponding to the successful meeting is a first preset score; when the evaluation indicator includes a failed meeting, the reward score corresponding to the failed meeting is a second preset score, and the second preset score is a negative number; when the evaluation indicator includes a meeting duration, the reward score corresponding to the meeting duration is determined based on the meeting duration; when the evaluation indicator includes the target vehicle's parking and waiting time and the target vehicle's parking and waiting time is greater than the preset time, the reward score corresponding to the target vehicle's parking and waiting time is a third preset score, and the third preset score is a negative number; when the evaluation indicator includes the acceleration smoothness of the target vehicle, the reward score corresponding to the acceleration smoothness of the target vehicle is determined based on the acceleration of the target vehicle.

[0142] In one possible implementation, the initial state space parameters include: state parameters of the target vehicle, state parameters of the other vehicles, and lane parameters; the state parameters of the target vehicle include: at least one of position, heading angle, speed, or size; the other vehicles include: the left vehicle of the target vehicle and / or the vehicle in front of the target vehicle; the lane parameters include: the area of ​​the lane line or the lane direction.

[0143] In addition, the embodiment of the present application also provides an automatic driving vehicle meeting device. Figure 5 , which is a schematic diagram of an automatic driving meeting device provided in an embodiment of the present application.

[0144] The apparatus 500 comprises:

[0145] A second acquisition unit 501 is configured to acquire initial state space parameters in a real meeting scene when the target real vehicle is in a real meeting scene with other real vehicles, where the other real vehicles represent a preceding vehicle of the target real vehicle and / or a vehicle to the left of the target real vehicle;

[0146] A second prediction unit 502 is configured to input the initial state space parameters into a target reinforcement learning model and output a predicted action of the target real vehicle;

[0147] The second acquisition unit 501 is further configured to acquire next state space parameters of the real vehicle-meeting scenario after the target real vehicle performs the predicted action;

[0148] The second iterative unit 503 is used to repeatedly input the next state space parameters into the target reinforcement learning model and subsequent processes until the meeting process is completed; wherein, the target reinforcement learning model is trained based on the method described in the above method embodiment.

[0149] Based on the above method embodiment and device embodiment, the present application also provides an electronic device, which will be described below with reference to the accompanying drawings.

[0150] See also Figure 6 , Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application.

[0151] The device 600 includes: a memory 601 and a processor 602;

[0152] The memory 601 is used to store relevant program codes;

[0153] The processor 602 is used to call the program code to execute the training method of the reinforcement learning model for autonomous driving described in the above method embodiment.

[0154] In addition, an embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the training method of the reinforcement learning model for autonomous driving described in the above method embodiment.

[0155] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the training method of the reinforcement learning model for autonomous driving described in the above method embodiment is implemented.

[0156] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0157] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0158] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiments described above are merely illustrative, wherein the units or modules described as separate components may or may not be physically separated, and the components shown as units or modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units or modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without expending creative work.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations that may be implemented according to the methods, devices and equipment of various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0161] It should also be noted that, in this application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0162] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0163] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to encompass the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a reinforcement learning model for autonomous driving, characterized in that: The method comprises: Inputting initial state space parameters of a current simulation environment into an initial reinforcement learning model, and outputting a predicted action of a target vehicle, wherein the current simulation environment represents a meeting scenario between the target vehicle and other vehicles in the simulation environment, and the initial reinforcement learning model is deployed on the target vehicle; determining an action reward based on the predicted action and the executed action of the other vehicle; After controlling the target vehicle to execute the predicted action, obtaining next state space parameters of the current simulation environment; Repeating the inputting of the next state space parameter into the initial reinforcement learning model and subsequent processes until the meeting process is completed; Determining a reward function based on the plurality of action rewards during the meeting process; The initial reinforcement learning model is trained based on the reward function to obtain a trained target reinforcement learning model.

2. The method according to claim 1, characterized in that The training of the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model includes: Reversely updating the parameters of the initial reinforcement learning model based on the reward function, and repeating the above training process until a preset number of times is reached, and determining a preferred reinforcement learning model based on multiple reward functions corresponding to the preset number of training processes; Modifying the current simulation environment to obtain another simulation environment, wherein the other simulation environment represents a vehicle-meeting scene different from the vehicle-meeting scene represented by the current simulation environment; The preferred reinforcement learning model is deployed on a target vehicle in the other simulation environment, and the preferred reinforcement learning model is trained based on the other simulation environment to obtain the target reinforcement learning model.

3. The method according to claim 1 or 2, characterized in that The method further comprises: After deploying the target reinforcement learning model on a target real vehicle traveling in a real environment, when the target real vehicle is in a real meeting scenario with other real vehicles, obtaining multiple sets of state space parameters in the real meeting scenario, where the other real vehicles represent the vehicle in front of the target real vehicle and / or the vehicle to the left of the target real vehicle; classifying the multiple sets of state space parameters in the simulation environment and the multiple sets of state space parameters in the real vehicle meeting scenario; When the multiple groups of state space parameters in the real vehicle meeting scenario and the multiple groups of state space parameters in the simulation environment are of different categories, the target reinforcement learning model is trained based on the state space parameters of different categories in the real vehicle meeting scenario to obtain an optimized reinforcement learning model.

4. The method according to claim 1, wherein The determining of an action reward based on the predicted action and the executed action of the other vehicle includes: Determining a first action reward corresponding to the predicted action based on a first preset action reward strategy for the target vehicle, wherein the first preset action reward strategy includes rewards corresponding to different actions of the target vehicle; Determining a second action reward corresponding to the executed action based on a second preset action reward strategy of the other vehicle, wherein the second preset action reward strategy includes rewards corresponding to different actions of the other vehicle; The action reward is determined based on the first action reward and the second action reward.

5. The method according to claim 1, wherein The determining of a reward function based on the plurality of action rewards during the meeting process includes: Determining a reward score corresponding to an evaluation indicator of the meeting process, the evaluation indicator including at least one of: whether the meeting is successful, the duration of the meeting, the time the target vehicle stops and waits, and the acceleration smoothness of the target vehicle; The reward function is determined based on the reward score and a plurality of the action rewards.

6. The method according to claim 5, characterized in that When the evaluation index includes a successful meeting, the reward score corresponding to the successful meeting is a first preset score; When the evaluation index includes a failure to meet another vehicle, the reward score corresponding to the failure to meet another vehicle is a second preset score, and the second preset score is a negative number; When the evaluation index includes the duration of the meeting, the bonus score corresponding to the duration of the meeting is determined based on the duration of the meeting; When the evaluation index includes the parking waiting time of the target vehicle and the parking waiting time of the target vehicle is greater than the preset time, the reward score corresponding to the parking waiting time of the target vehicle is a third preset score, and the third preset score is a negative number; When the evaluation index includes acceleration smoothness of the target vehicle, the bonus score corresponding to the acceleration smoothness of the target vehicle is determined based on the acceleration of the target vehicle.

7. The method according to claim 1, characterized in that The initial state space parameters include: state parameters of the target vehicle, state parameters of the other vehicles, and lane parameters; The state parameter of the target vehicle includes at least one of position, heading angle, speed, or size; The other vehicles include: the left vehicle of the target vehicle and / or the vehicle in front of the target vehicle; The lane parameters include: the area of ​​the lane line or the lane direction.

8. An automatic driving method for meeting other vehicles, characterized in that: The method comprises: When the target real vehicle is in a real meeting scene with other real vehicles, obtaining initial state space parameters in the real meeting scene, where the other real vehicles represent the vehicle in front of the target real vehicle and / or the vehicle to the left of the target real vehicle; Inputting the initial state space parameters into a target reinforcement learning model, and outputting a predicted action of the target real vehicle; After the target real vehicle executes the predicted action, obtaining the next state space parameters of the real meeting scene; Repeating the inputting of the next state space parameter into the target reinforcement learning model and subsequent processes until the meeting process is completed; Wherein, the target reinforcement learning model is trained based on the method of any one of claims 1-7.

9. A training device for a reinforcement learning model for autonomous driving, characterized in that: The device comprises: a first prediction unit, configured to input initial state space parameters of a current simulation environment into an initial reinforcement learning model, and output a predicted action of a target vehicle, wherein the current simulation environment represents a meeting scenario between the target vehicle and other vehicles in the simulation environment, and the initial reinforcement learning model is deployed on the target vehicle; a first determining unit, configured to determine an action reward based on the predicted action and the executed action of the other vehicle; A first acquisition unit is configured to acquire next state space parameters of the current simulation environment after controlling the target vehicle to execute the predicted action; a first iterative unit, configured to repeatedly input the next state space parameter into the initial reinforcement learning model and subsequent processes until the meeting process is completed; A second determining unit is configured to determine a reward function based on the plurality of action rewards during the meeting process; A training unit is used to train the initial reinforcement learning model based on the reward function to obtain a trained target reinforcement learning model.

10. An automatic driving meeting device, characterized in that: The device comprises: a second acquisition unit, configured to acquire initial state space parameters in a real meeting scene when the target real vehicle is in a real meeting scene with other real vehicles, wherein the other real vehicles represent a preceding vehicle of the target real vehicle and / or a vehicle to the left of the target real vehicle; a second prediction unit, configured to input the initial state space parameters into a target reinforcement learning model and output a predicted action of the target real vehicle; The second acquisition unit is further configured to acquire next state space parameters of the real vehicle meeting scene after the target real vehicle performs the predicted action; a second iterative unit, configured to repeatedly input the next state space parameter into the target reinforcement learning model and subsequent processes until the meeting process is completed; Wherein, the target reinforcement learning model is trained based on the method of any one of claims 1-7.

Images