Generator training method, vehicle control method, device, equipment and medium

Abstract

The invention discloses a generator training method, a vehicle control method, a device, equipment and a medium. Comprising the steps that a plurality of first training samples are acquired, the first training samples comprise first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, the first behavior sequence is output according to a pre-acquired target decision maker, and the target decision maker is used for simulating decision behaviors of human experts; a plurality of first training samples are adopted to train an initial generator to obtain a first target generator, a target decision maker is adopted to assist the training of the generator, and the output of the target decision maker is taken as a training sample, so that the generator can learn a decision behavior which is basically the same as that of the target decision maker, and the generator can be well trained; meanwhile, the learning process can be accelerated and the decision accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and in particular relates to a generator training method, a vehicle control method, a device, a equipment and a medium. Background Art

[0002] Generative Adversarial Network (GAIL) is an imitation learning method that can achieve good results. GAIL uses data from human-environment interactions for learning. The Generative Adversarial Network consists of two networks: the generator and the discriminator, which continuously compete with each other for optimization. The generator uses existing human demonstration samples to generate imitation samples, and uses the imitation samples to deceive the discriminator, making it impossible for the discriminator to distinguish the imitation samples.

[0003] With the rapid development of autonomous driving technology, generative adversarial networks have also been applied to autonomous driving. However, since the generator does not fully utilize human knowledge in its imitation process, the training effect of the generator is poor and it cannot be well applied to autonomous driving. Summary of the invention

[0004] The embodiments of the present application provide a generator training method, a vehicle control method, an apparatus, a device and a medium, which can solve the problem of poor training effect of the generator.

[0005] In a first aspect, an embodiment of the present application provides a generator training method, the method comprising:

[0006] Acquire a plurality of first training samples, wherein the first training samples include first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, wherein the first behavior sequence is obtained according to a pre-acquired target decision maker output, wherein the target decision maker is used to simulate the decision-making behavior of a human expert;

[0007] The initial generator is trained using a plurality of first training samples to obtain a first target generator.

[0008] In one example, obtaining a plurality of first training samples includes:

[0009] The process of obtaining each of the first training samples includes:

[0010] Acquiring first status information of the vehicle;

[0011] Inputting the first state information of the vehicle into the target decision maker to obtain a first behavior sequence corresponding to the first state information, wherein the target decision maker is trained based on a plurality of second training samples, each of which includes the second state information of the vehicle and a second behavior sequence corresponding to the vehicle, wherein the second behavior sequence is obtained by simulated driving according to the second state information of the vehicle in a simulation environment using preset rules;

[0012] The first training sample is constructed according to the first state information and a first behavior sequence corresponding to the first state information.

[0013] In one example, constructing the first training sample according to the first state information and a first behavior sequence corresponding to the first state information includes:

[0014] Inputting the first state information into an initial generator to obtain a third behavior sequence output by the initial generator;

[0015] If the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is less than a preset threshold, then increasing the penalty value of the initial generator, adjusting the parameters of the initial generator according to the penalty value, and constructing the first training sample according to the first state information and the first behavior sequence corresponding to the first state information;

[0016] If the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is greater than or equal to the preset threshold, increasing the reward value of the initial generator, and adjusting the parameters of the initial generator according to the reward value;

[0017] The method of using a plurality of first training samples to train the initial generator to obtain a first target generator includes:

[0018] The first target generator is obtained by training the initial generator with adjusted parameters using a plurality of the first training samples.

[0019] In one example, after the initial generator is trained using a plurality of first training samples to obtain a first target generator, the method further includes:

[0020] Acquiring third status information of a plurality of the vehicles;

[0021] Inputting the third state information of each vehicle into the first target generator to obtain a behavior sequence corresponding to each third state information;

[0022] Based on the feedback indication input by the user, sample construction is performed on the plurality of third state information and the behavior sequence corresponding to each of the third state information to obtain a plurality of third training samples;

[0023] The first target generator is trained using a plurality of the third training samples to obtain a second target generator.

[0024] In one example, the feedback indication based on the user input, constructing samples of the behavior sequences corresponding to the plurality of the third state information and each of the third state information to obtain a plurality of third training samples, includes:

[0025] Displaying a plurality of information pairs, each of the information pairs comprising one of the third state information and a behavior sequence corresponding to the third state information;

[0026] Receive the feedback indication input by the user, determine a plurality of the third training samples, each of the third training samples includes a third state information in the plurality of information pairs, and a fourth behavior sequence corresponding to the third state information, and the fourth behavior sequence is determined according to the feedback indication.

[0027] In an example, the fourth behavior sequence is the behavior sequence corresponding to the third state information, or the fourth behavior sequence is obtained by adjusting the behavior sequence corresponding to the third state information through the feedback indication.

[0028] In a second aspect, an embodiment of the present application provides a vehicle control method, including:

[0029] Get the current status information of the vehicle;

[0030] Inputting the current state information into a generator to obtain a target behavior sequence;

[0031] The vehicle is controlled according to the target behavior sequence, and the generator is obtained based on the generator training method described in the first aspect.

[0032] In a third aspect, an embodiment of the present application provides a generator training device, the device comprising:

[0033] a first acquisition module, configured to acquire a plurality of first training samples, wherein the first training samples include first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, wherein the first behavior sequence is obtained according to a pre-acquired target decision maker output, wherein the target decision maker is configured to simulate a decision-making behavior of a human expert;

[0034] A training module is used to train the initial generator using multiple first training samples to obtain a first target generator.

[0035] In a fourth aspect, an embodiment of the present application provides a vehicle control device, the device comprising:

[0036] The second acquisition module is used to acquire the current state information of the vehicle;

[0037] A processing module, used for inputting the current state information into a generator to obtain a target behavior sequence;

[0038] A control module is used to control the vehicle according to the target behavior sequence, and the generator is obtained based on the generator training method described in the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0040] When the processor executes the computer program instructions, it implements the generator training method as described in the first aspect or the vehicle control method as described in the second aspect.

[0041] In a sixth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the generator training method as described in the first aspect or the vehicle control method as described in the second aspect is implemented.

[0042] In the seventh aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the generator training method as described in the first aspect or the vehicle control method as described in the second aspect.

[0043] In an eighth aspect, an embodiment of the present application provides a vehicle, comprising: a processor and a memory storing computer program instructions;

[0044] When the processor executes the computer program instructions, the vehicle control method as described in the second aspect is implemented.

[0045] The generator training method, vehicle control method, device, equipment and medium of the embodiments of the present application adopt a target decision maker to assist the training of the generator, and use the output of the target decision maker as a training sample, so that the generator can learn decision-making behavior that is basically the same as the target decision maker, and can better train the generator, while accelerating the learning process and improving the accuracy of the decision. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 It is a flowchart of an embodiment of the generator training method provided by the present application;

[0048] Figure 2 is another flowchart of an embodiment of the generator training method provided by the present application;

[0049] Figure 3 It is a schematic diagram of the structure of the generator, decision maker and discriminator provided by this application;

[0050] Figure 4 It is a flow chart of an embodiment of the vehicle control method provided by the present application;

[0051] Figure 5 is a structural schematic diagram of an embodiment of a generator training device provided by the present application;

[0052] Figure 6 is a structural schematic diagram of an embodiment of a vehicle control device provided by the present application;

[0053] Figure 7 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0054] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0055] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0056] The vehicle may be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle may also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle may be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it may be a hybrid vehicle or a pure electric vehicle.

[0057] In related technologies, Generative Adversarial Network (GAIL) is an imitation learning method that can achieve good results. GAIL uses data from human-environment interactions for learning. The Generative Adversarial Network consists of two networks: the generator and the discriminator. The two networks continuously compete with each other for optimization.

[0058] With the rapid development of autonomous driving technology, generative adversarial networks have also been applied to autonomous driving. However, since the generator does not fully utilize human knowledge in its imitation process, the training effect of the generator is poor and it cannot be well applied to autonomous driving.

[0059] Based on this, the embodiments of the present application provide a generator training method, a vehicle control method, an apparatus, a device and a medium. By adopting a target decision maker to assist in the training of the generator and using the output of the target decision maker as a training sample, the generator can learn decision-making behaviors that are basically the same as the target decision maker, which can better train the generator, accelerate the learning process and improve the accuracy of the decision.

[0060] In order to solve the technical problems in the related art, the embodiments of the present application provide a generator training method, device, equipment, computer storage medium and computer program product. The generator training method provided in the embodiments of the present application is first introduced below.

[0061] Figure 1 A flow chart of an embodiment of the generator training method provided by the present application is shown. The generator training method of the present application can be applied to electronic devices.

[0062] like Figure 1 As shown, the generator training method provided in the embodiment of the present application includes the following steps S101 to S102, wherein:

[0063] S101. Acquire a plurality of first training samples, where the first training sample includes first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, where the first behavior sequence is obtained according to a pre-acquired target decision maker output, and the target decision maker is used to simulate the decision-making behavior of a human expert.

[0064] In this embodiment, multiple first training samples are obtained, and the first training sample includes first state information of the vehicle, and the first state information includes: the speed, position, and direction of the vehicle itself, the speed, position, and direction of surrounding vehicles, and the position and direction of obstacles. The first training sample also includes a first behavior sequence of the vehicle corresponding to the first state information, and the first behavior sequence includes acceleration, deceleration, left lane change, right lane change, and overtaking. The first behavior sequence is obtained according to the pre-acquired output of the target decision maker, and the output of the target decision maker is used as a sample for training.

[0065] Among them, the first state information in the first training sample can be obtained from a next generation simulation (NGSIM) dataset, and the NGSIM dataset includes natural vehicle dataset trajectories, which are captured from high-rise buildings and capture vehicle trajectories on highways and urban traffic roads from a bird's-eye view.

[0066] Among them, the pre-acquired target decider is a human experience decider, which is a tool that uses human knowledge and experience to assist the generator in learning to make decisions. The target decider is used to simulate the decision-making behavior of human experts. By inputting the first state information to the target decider, the target decider analyzes, judges and makes decisions based on these data, and then outputs the corresponding behavior sequence.

[0067] S102. Use multiple first training samples to train the initial generator to obtain a first target generator.

[0068] In this embodiment, a plurality of first training samples are used to train the initial generator. During the training process, when the first state information of the input of the generator and the target decision maker is consistent, if the output of the generator and the target decision maker is the same and the same first behavior sequence is output, it indicates that the decision is consistent, and a positive reward is given to the generator; if the output of the generator and the target decision maker is different, it indicates that the decision is inconsistent, and a negative reward is given to the generator until the initial generator converges to obtain the first target generator.

[0069] In an embodiment of the present application, a plurality of first training samples are obtained, the training samples including first state information of the vehicle and a first behavior sequence corresponding to the first state information, the first behavior sequence being obtained based on the output of a target decision maker, the target decision maker being used to simulate the decision-making behavior of a human expert, and the target decision maker being used to assist in the training of the generator, and the output of the target decision maker being used as a training sample, so that the generator can learn decision-making behaviors that are substantially the same as those of the target decision maker, and can better train the generator, while accelerating the learning process and improving the accuracy of decisions.

[0070] Figure 2 Another flow chart of an embodiment of the generator training method provided by the present application is shown. The generator training method of the present application can be applied to electronic devices.

[0071] like Figure 2 As shown, the generator training method provided in the embodiment of the present application includes the following steps S201 to S205, wherein:

[0072] S201. Acquire a plurality of first training samples, where the first training sample includes first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, where the first behavior sequence is obtained according to a pre-acquired target decision maker output, and the target decision maker is used to simulate the decision-making behavior of a human expert.

[0073] In this embodiment, multiple first training samples are obtained, and the first training sample includes first state information of the vehicle, and the first state information includes: the speed, position, and direction of the vehicle itself, the speed, position, and direction of surrounding vehicles, and the position and direction of obstacles. The first training sample also includes a first behavior sequence of the vehicle corresponding to the first state information, and the first behavior sequence includes acceleration, deceleration, left lane change, right lane change, and overtaking. The first behavior sequence is obtained according to the pre-acquired output of the target decision maker, and the output of the target decision maker is used as a sample for training.

[0074] Among them, the first state information in the first training sample can be obtained from a next generation simulation (NGSIM) dataset, and the NGSIM dataset includes natural vehicle dataset trajectories, which are captured from high-rise buildings and capture vehicle trajectories on highways and urban traffic roads from a bird's-eye view.

[0075] Among them, the pre-acquired target decider is a human experience decider, which is a tool that uses human knowledge and experience to assist the generator in learning to make decisions. The target decider is used to simulate the decision-making behavior of human experts. By inputting the first state information to the target decider, the target decider analyzes, judges and makes decisions based on these data, and then outputs the corresponding behavior sequence.

[0076] In one example, step S201 includes:

[0077] The process of obtaining each first training sample includes:

[0078] Acquire first state information of the vehicle; input the first state information of the vehicle into a target decision maker to obtain a first behavior sequence corresponding to the first state information, the target decision maker is trained based on a plurality of second training samples, each second training sample includes the second state information of the vehicle, and a second behavior sequence corresponding to the vehicle, the second behavior sequence is obtained by simulated driving according to the second state information of the vehicle in a simulation environment using preset rules; construct a first training sample based on the first state information and the first behavior sequence corresponding to the first state information.

[0079] In this embodiment, the acquisition process of each first training sample is specifically to acquire first state information of the vehicle, input the first state information to the target decision maker, and the target decision maker outputs a first behavior sequence corresponding to the first state information.

[0080] Specifically, the target decision maker is obtained based on multiple second training samples, each of which includes the second state information of the vehicle and the second behavior sequence corresponding to the vehicle. The second state information is obtained from a pre-acquired data set, which is an NGSIM data set. The data set includes multiple second state information, including: the speed, position, and direction of the vehicle itself, the speed, position, and direction of surrounding vehicles, and the position and direction of obstacles. In a simulation environment, the target decision maker is obtained by simulated driving according to the second state information of the vehicle using preset rules.

[0081] Among them, the preset rules are expert rules, which are driving rules pre-selected by experts and cover basic driving principles, driving skills and habits, safe driving awareness, and emergency response. Alternatively, in a simulation environment, a preset driving model, such as a vehicle following model, is used to input the second state information of the vehicle into the preset driving model to obtain a second behavior sequence. The driving rules set by existing human experts or the corresponding driving model are used to simulate what actions should be taken in each scenario in the data set, thereby obtaining the corresponding second behavior sequence, and the second state information of the vehicle and the second behavior sequence constitute the second training sample. The decision maker is trained using the second training sample obtained in the above manner, so that the decision maker can output a human experience strategy, i.e., a behavior sequence, based on the state information of the vehicle.

[0082] In one example, constructing a first training sample according to first state information and a first behavior sequence corresponding to the first state information includes:

[0083] Input the first state information into the initial generator to obtain a third behavior sequence output by the initial generator; if the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is less than a preset threshold, increase the penalty value of the initial generator, adjust the parameters of the initial generator according to the penalty value, and construct a first training sample according to the first state information and the first behavior sequence corresponding to the first state information; if the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is greater than or equal to the preset threshold, increase the reward value of the initial generator, and adjust the parameters of the initial generator according to the reward value;

[0084] The initial generator is trained using a plurality of first training samples to obtain a first target generator, including:

[0085] The initial generator with adjusted parameters is trained using a plurality of first training samples to obtain a first target generator.

[0086] In the above embodiment, whether to give the generator a positive reward or a negative reward is determined based on the similarity, and the similarity is determined by the discriminator. For example, see Figure 3 , Figure 3 The left side of the figure represents the NGSIM dataset. The state information in the dataset has its own corresponding timestamp st. The first state information of the vehicle is obtained from the dataset. Each first state information has its own corresponding timestamp st. Whether two first state information are the same is determined based on the timestamps corresponding to the first state information. The first state information se is output to the target decision maker to obtain the first behavior sequence ae output by the target decision maker. The first state information se is input to the initial generator to obtain the third behavior sequence ag output by the initial generator. The target decision maker sends (se, ae) to the discriminator, and the initial generator sends (se, ag) to the discriminator. The discriminator calculates the similarity based on (se, ae) and (se, ag).

[0087] Specifically, if the similarity between the third behavior sequence and the first behavior sequence is less than a preset threshold, ag and ae correspond to different decision-making behaviors, indicating that the current initial generator is not learning very well, then a negative reward is given to the generator, that is, the penalty value r1(πθ) of the initial generator is increased, and the parameters of the initial generator are adjusted according to the penalty value. Specifically, according to the penalty value r1(πθ) and the loss value loss, a first value r(πθ) is obtained, the parameters of the initial generator are adjusted according to the first value, and a first training sample is constructed according to the first state information and the first behavior sequence corresponding to the first state information.

[0088] If the similarity between the third behavior sequence and the first behavior sequence is greater than or equal to the preset threshold, ag and ae correspond to the same decision behavior. In order to motivate the initial generator to learn, a positive reward is given to the generator, that is, the reward value r1(πθ) of the initial generator is increased, and the parameters of the initial generator are adjusted according to the reward value. Specifically, according to the reward value r1(πθ) and the loss value loss, a first value r(πθ) is obtained, and the parameters of the initial generator are adjusted according to the first value. Further, a plurality of first training samples are used to train the parameter-adjusted initial generator until the generator converges, which means that it is difficult for the discriminator to distinguish the behavior sequence of the initial generator from that of the target decision maker, thereby obtaining the first target generator.

[0089] Further, see Figure 3 , the target decision maker and the initial generator can interact with each other. If the similarity between the third behavior sequence and the first behavior sequence is less than a preset threshold, the target decision maker takes the first behavior sequence as the target sequence, sends the first behavior sequence πH to the initial generator, and replaces the third behavior sequence with the first behavior sequence, so that the initial generator can learn better decision-making behaviors. Alternatively, after the initial generator has learned a better decision-making behavior, the point behavior πθ of the initial generator can also be sent to the target decision maker, so that the target decision maker can learn a better decision-making behavior and can generate a behavior sequence close to the target sequence of the target decision maker more quickly. Specifically, a user with driving experience evaluates the third behavior sequence of the initial generator. If the third behavior sequence is considered to be better, it is marked and provided to the target decision maker for learning, so that the target decision maker and the initial generator interact and share, and learn better decision-making behaviors from each other.

[0090] S202. Use multiple first training samples to train the initial generator to obtain a first target generator.

[0091] In this embodiment, a plurality of first training samples are used to train the initial generator. During the training process, when the first state information of the input of the generator and the target decision maker is consistent, if the output of the generator and the target decision maker is the same and the same first behavior sequence is output, it indicates that the decision is consistent, and a positive reward is given to the generator; if the output of the generator and the target decision maker is different, it indicates that the decision is inconsistent, and a negative reward is given to the generator until the initial generator converges to obtain the first target generator.

[0092] S203: Obtain third status information of multiple vehicles.

[0093] In this embodiment, the first target generator is obtained through the previous first stage of learning. The first target generator can output better decision-making behaviors. In order to make the first target generator more suitable for actual traffic scenarios, learn better driving strategies, output better decision-making behaviors, and conform to human values, the second stage of learning can be carried out.

[0094] In actual traffic scenarios, humans will have different behaviors when driving different vehicles. For example, under the same traffic scenario and traffic rules, in the same highway or urban road environment, when humans drive small cars, the speed is relatively faster and the distance between cars is relatively shorter. But when humans drive large vehicles, the speed is relatively slower and the distance between cars is longer. Decision-making behaviors in the same environment will show different behaviors due to their different values, and need to be aligned according to human values. Therefore, the second stage of learning can be carried out.

[0095] In this embodiment, the third state information of multiple vehicles is obtained for the first target generator to learn. The third state information includes the vehicle's own speed, position, and direction, the speed, position, and direction of surrounding vehicles, and the position and direction of obstacles. The third state information can also be obtained from the above-mentioned data set.

[0096] S204: Input the third state information of each vehicle into the first target generator to obtain a behavior sequence corresponding to each piece of third state information.

[0097] In this embodiment, the third state information of each vehicle is input into the first target generator, and the first target generator outputs the behavior sequence corresponding to each third state information, displays the third state information of each vehicle and the corresponding behavior sequence, and the user provides feedback based on the displayed third state information of the vehicle and the corresponding behavior sequence.

[0098] S205 . Based on the feedback indication input by the user, construct samples of the plurality of third state information and the behavior sequences corresponding to each of the third state information to obtain a plurality of third training samples.

[0099] In this embodiment, based on the feedback indication input by the user, sample construction is performed on a plurality of third state information and a behavior sequence corresponding to each third state information to obtain a plurality of third training samples.

[0100] In one example, step S205 includes:

[0101] Display multiple information pairs, each information pair includes a third state information, and a behavior sequence corresponding to the third state information; receive feedback instructions input by the user, determine multiple third training samples, each third training sample includes a third state information in the multiple information pairs, and a fourth behavior sequence corresponding to the third state information, and the fourth behavior sequence is determined according to the feedback instructions.

[0102] In this embodiment, a pair of information pairs are displayed, each of which includes a third state information and a behavior sequence corresponding to the third state information. The user can determine whether to accept or reject the behavior sequence by viewing each third state information and the corresponding behavior sequence. If the user thinks that the behavior corresponding to the execution of the behavior sequence is approved in the traffic scene corresponding to the third state information, the user can click to accept the behavior sequence; if the user thinks that the behavior corresponding to the execution of the behavior sequence is not approved in the traffic scene corresponding to the third state information, the user can click to reject the behavior sequence and give the corresponding behavior, i.e., the behavior sequence. Feedback instructions input by the user are received, such as acceptance or rejection, and multiple third training samples are determined. Each third training sample includes the third state information in multiple information pairs and the fourth behavior sequence corresponding to the third state information. The fourth behavior sequence is determined based on the user's feedback instructions.

[0103] Optionally, the fourth behavior sequence is a behavior sequence corresponding to the third state information, or the fourth behavior sequence is obtained by adjusting the behavior sequence corresponding to the third state information through feedback indication.

[0104] In this embodiment, if the user agrees with the behavior sequence output by the first target generator, the behavior sequence is accepted, and the fourth behavior sequence is consistent with the behavior sequence corresponding to the third state information. If the user disagrees with the behavior sequence output by the first target generator, the behavior training is rejected and a new behavior is given. The fourth behavior sequence is a new behavior sequence obtained by adjusting the behavior sequence corresponding to the third state information through feedback instructions. Through user feedback, multiple third state information and fourth behavior sequences are collected to form multiple third training samples for the first target generator to perform the second stage of learning.

[0105] S206. Use multiple third training samples to train the first target generator to obtain a second target generator.

[0106] In this embodiment, the third training sample data is used to train the first target generator to learn human behavior strategies, thereby obtaining the second target generator, completing the learning of driving strategies based on human values, which can be applied to actual traffic scenarios.

[0107] In the embodiment of the present application, aligned reinforcement learning is used to enable the generator to provide human-like strategies and achieve the same behavior as humans, which can be better used in actual traffic scenarios.

[0108] Figure 4 FIG. 1 is a flow chart of an embodiment of the vehicle control method provided by the present application. The vehicle control method of the present application can be applied to a vehicle. Figure 4 As shown, the vehicle control method provided in the embodiment of the present application includes the following steps S401 to S403, wherein:

[0109] S401. Obtain current status information of the vehicle.

[0110] In this embodiment, the current state information of the vehicle is obtained, and the current state information includes the current speed, current position, current direction of the vehicle, the current speed, current position, current direction of surrounding vehicles, and the current position and current direction of obstacles. The state information can be obtained through sensors set on the vehicle body.

[0111] S402: Input the current state information into the generator to obtain the target behavior sequence.

[0112] In this embodiment, the current state information of the vehicle is input into the generator, which is obtained through the above-mentioned generator training method and can be used for vehicle driving to achieve automatic control of the vehicle.

[0113] S403, controlling the vehicle according to the target behavior sequence, the generator is obtained based on the above generator training method.

[0114] In this embodiment, the vehicle is controlled according to the target behavior sequence, and the generator obtained through the above learning is applied to autonomous driving, which can be applied to actual traffic scenarios to better control vehicle driving.

[0115] The following is an example of the generator training method provided in the embodiment of the present application.

[0116] The discriminator and the generator constitute GAIL (Generative Adversarial Imitation Learning), which is a reinforcement learning method that combines generative adversarial networks (GANs) and imitation learning. The core idea of ​​GAIL is to learn a strategy that can imitate expert behavior through an adversarial training process between a generator and a discriminator. In GAIL, the goal of the generator is to learn a strategy that makes the generated policy behavior difficult to distinguish from the expert behavior; while the goal of the discriminator is to distinguish the policy behavior generated by the generator from the real behavior of the expert. Through this adversarial training, the generator gradually learns a strategy that can produce behavior similar to that of an expert.

[0117] The embodiment of the present application proposes a new generative adversarial imitation learning framework. On the basis of GAIL, a human experience decision maker (i.e., the target decision maker mentioned above) is added to the generator, and a two-stage behavior strategy learning method is performed on the generator. The first stage is an imitation learning stage based on human experience, and the second stage is a reinforcement learning stage. In the first stage, human strategy learning is completed using methods such as human experience, imitation learning, and control sharing. In the second stage, a value alignment method for intelligent agents is performed using feedback reinforcement learning.

[0118] Specifically, the first phase of learning content is as follows:

[0119] 1) Initialize the human experience decision maker (i.e. the decision maker mentioned above) and the discriminator (also called the discriminator). The input of the human experience decision maker is the vehicle's state information, including: the vehicle's own speed, position, direction, the speed, position, direction of surrounding vehicles, and the position and direction of obstacles. The output of the human experience decision maker is a behavior sequence, including acceleration, deceleration, left lane change, right lane change, and overtaking.

[0120] Initialize the generator. The generator in this embodiment is an intelligent agent that needs to self-learn to control the intelligent vehicle. The input is the vehicle status information and the output is the behavior sequence. The generator adopts a deep neural network-like structure.

[0121] The discriminator uses a deep neural network structure to determine the similarity between the human experience decision maker and the generator. The discriminant method is to determine the similarity by comparing the similarity between the behavior sequence output by the human experience decision maker and the behavior sequence output by the generator. Or, by comparing the action probability output by the human experience decision maker and the action probability output by the generator, the similarity is determined.

[0122] See also Figure 3, the human expert strategy corresponding to the human experience decision maker is πH, and the strategy learned by the generator is πθ. The human experience decision maker guides the generator to learn by judging the quality of the samples generated by the generator. For example, when the similarity is small, the human expert strategy πH is used to replace the strategy of the generator for the generator to learn. At the same time, the strategy learned by the generator is πθ, which has a better effect. By collecting the strategies of the generator as learning samples for the human experience decision maker to learn, the interaction between πH and πθ is completed. To determine whether the strategy learned by the generator is πθ, the strategy of the generator can be evaluated by users with driving experience.

[0123] 2) Using the target behavior samples to train the human experience decision maker to obtain the target human experience decision maker (i.e., the target decision maker mentioned above). Specifically, the target behavior samples are obtained in the following manner: in a simulation environment, based on the NGSIM data set, the NGSIM data set includes the state information of the vehicle (i.e., the second state information mentioned above), and expert rules are used to perform simulated driving in the simulation environment to obtain the behavior sequence of the vehicle (i.e., the second behavior sequence mentioned above), collect the state information of the vehicle (i.e., the second state information mentioned above) and the corresponding behavior sequence (i.e., the second behavior sequence mentioned above), and obtain the target behavior samples (i.e., the second training samples mentioned above).

[0124] Alternatively, the kinematic formula (i.e., the driving model mentioned above) is used to re-simulate what actions should be taken in each scenario in the data set. Specifically, in a simulation environment, based on the NGSIM data set, a driving model is used to obtain the vehicle's behavior sequence (i.e., the second behavior sequence mentioned above), collect the vehicle's state information (i.e., the second state information mentioned above) and the corresponding behavior sequence (i.e., the second behavior sequence mentioned above), and obtain the target behavior sample (i.e., the second training sample mentioned above).

[0125] The target behavior samples are used and clone learning is utilized to train the human experience decision maker. Finally, the learned human experience decision maker can output the human experience strategy (i.e., the first behavior sequence in the above text) according to the environmental state information (i.e., the first state information in the above text, and input the first state information into the target decision maker), thereby obtaining the target human experience decision maker.

[0126] 3) The generator learns strategies by interacting with the environment and combining the target human experience decision maker. The generator interacts with the environment in the autonomous driving simulator, uses the environmental state and its own state perceived by the intelligent vehicle environment during the interaction, and gives behavioral decision actions as optimization samples, which are saved in the experience pool to obtain the strategy training experience pool.

[0127] 4) During the training process, the generator outputs a behavior sequence based on the environmental state information and inputs the behavior sequence into the discriminator. The discriminator uses the strategy of the target human experience decision maker as a guiding strategy based on the strategy similarity between the generator and the target human experience decision maker in the same situation and the principle of control sharing, and replaces the strategy generated by the generator (i.e., the first behavior sequence is used to replace the third behavior sequence in the above text, so that the initial generator can learn better decision-making behavior). At the same time, the generator and the target human experience decision maker make consistent decisions in the same situation, increasing the learning reward of the generator (i.e., if the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is greater than or equal to the preset threshold, the reward value of the initial generator is increased). The parameters of the generator are adjusted using the replaced strategy experience pool and the similarity of the discriminator (i.e., adjusting the parameters of the initial generator according to the reward value / adjusting the parameters of the initial generator according to the penalty value in the above text), so that human experience can guide the generator to generate a behavior sequence closer to the target behavior faster during the intelligent vehicle learning process.

[0128] 5) Repeat steps 3 and 4 until the generator and discriminator converge, which means that the behavior generated by the generator is getting closer and closer to the target behavior, and the discriminator is increasingly unable to distinguish between the behavior generated by the generator and the target behavior. Through the above steps, the generative adversarial imitation learning training process is applied in the autonomous driving simulation environment, so that the generator learns driving skills by observing and imitating human behavior.

[0129] In the above process, through the previous first stage of learning, the first target generator is obtained. The first target generator can output better decision-making behaviors. In order to make the first target generator more suitable for actual traffic scenarios, learn better driving strategies, output better decision-making behaviors, and conform to human values, the second stage of learning can be carried out.

[0130] Specifically, the second phase of learning content is as follows:

[0131] It is hoped that a better driving strategy than humans can be learned while being in line with human values, so the second stage of learning can be carried out: behavioral learning based on human values. In traffic scenarios, humans driving different types of vehicles will produce different behaviors. For example, under the same traffic scenario and traffic rules, in the same highway or urban road environment, when humans drive small cars, the speed is relatively faster and the distance between vehicles is relatively shorter. But when humans drive large vehicles, the speed is relatively slower and the distance between vehicles is longer. Similarly, when driving special vehicles such as police cars and ambulances, human values ​​such as driving speed and distance maintenance are more prominent. Generally speaking, the behavior of intelligent agents in the same environment (i.e., the first target generator above) will show different behaviors due to their different values, and they need to be aligned according to human values. In order to solve this common problem, the embodiment proposes to learn human values ​​using reinforcement learning methods.

[0132] Agent D R The model (i.e., the first target generator above) is expressed as Indicates that the agent is at value A R The strategy π generated under R . Human Model D H Defined as D H =<π H ,A H >, indicating that humans value A H The strategy π generated under H . Humans may have different views on the agent model, and this difference reflects the underlying value differences between humans and agents. When the agent's strategy is inconsistent with the human strategy, a value alignment problem arises. Currently, the solution to the value alignment problem is mainly for humans to sort the data generated by the agent and generate a reward model to guide learning. Although humans may not be able to accurately define human goals, they can identify states that meet human expectations. In this case, this information is included in the human's strategy π H That is, when humans follow their own values ​​to do something, they will generate π H Therefore, the intelligent agent should learn how to determine its own value function, give human-like strategies, and achieve the same behavior as humans. Different strategies will be generated under different value drivers. Conversely, the difference in strategies reflects the mismatch of values.

[0133] The agent uses its own model D R Interactive learning with the environment, in value A R The strategy trajectory τ is generated under the action R ={(s 1 ,a 1 ), (s 2 ,a 2 ),…,(s n ,an )}, the strategic trajectory of human value generation is According to the two trajectories, R Make adjustments. The strategy is based on the value function A R The state behavior generated under the influence of the overall value is the combination of the components of each value dimension. For example, the overall value is a complete value system composed of time value, safety value, etc., that is: A R =θ 1 m 1 +θ 2 m 2 +…+θ n m m To find a reasonable θ i Make A R The resulting strategy is similar to human values. The difference lies in whether humans think the current strategy is consistent with their own values, that is, τ R With τ H The difference is as follows: R Whether it is accepted by humans. Human samples are equivalent to supervisory signals, which supervise and adjust the value function of the intelligent agent. The degree of human acceptance of machine value is reflected in the degree of acceptance of actions under the same state. The probability p of the machine value parameter θ being accepted by human value, the intelligent agent adjusts θ to make A R As big as possible H accept.

[0134] The second stage value alignment process is:

[0135] 1) Use the driving strategy learned in the first stage to drive in the simulation environment. The strategy will make a series of driving decisions based on the environment. The environment (i.e., the third state information mentioned above) is represented by s, and each environment will have an action a (i.e., input the third state information into the first target generator to obtain the behavior sequence corresponding to each third state information).

[0136] 2) The intelligent agent (i.e., the first goal generator mentioned above) shows the strategy to be adopted at the future moment, that is, the action a to be taken at time s, to the human, allowing the human to choose to accept or reject the action, or directly execute action a to let the human take over and interrupt the action, so as to obtain human feedback (i.e., the feedback indication of the user input mentioned above).

[0137] 3) In the loop 1) and 2), we can collect many (s, a) state pairs, which are called trajectory sequences in reinforcement learning (i.e., based on the feedback instructions of the user input mentioned above, samples of multiple third state information and the behavior sequences corresponding to each third state information are constructed to obtain multiple third training samples).

[0138] 4) Use the collected (s, a) state pairs to update again and complete the driving strategy learning of human values ​​(i.e., use multiple third training samples to train the first target generator to obtain the second target generator in the above text).

[0139] The embodiment of the present application provides GAIL with rich human prior data knowledge and embeds human behavior data knowledge into the learning process, aiming to solve the bottleneck problem of random imitation behavior of the GAIL generator. It is found that the behavior of agents in different environments will show different value-driven behaviors due to their different attributes. Through aligned reinforcement learning, the agent system learns how to determine its own value function, gives human-like strategies, and achieves the same behavior as humans.

[0140] In the generator training method provided in the embodiment of the present application, the execution subject may be a generator training device. In the embodiment of the present application, the generator training device performing the generator training method is taken as an example to illustrate the generator training device provided in the embodiment of the present application.

[0141] Figure 5 FIG. 1 shows a schematic diagram of the structure of the generator training device provided in an embodiment of the present application. Figure 5 As shown, the generator training device 50 of the present application includes: a first acquisition module 501 and a training module 502.

[0142] Among them, the first acquisition module 501 is used to obtain multiple first training samples, the first training sample includes the first state information of the vehicle, and the first behavior sequence of the vehicle corresponding to the first state information, the first behavior sequence is obtained according to the pre-acquired target decision maker output, and the target decision maker is used to simulate the decision-making behavior of human experts.

[0143] The training module 502 is used to train the initial generator using multiple first training samples to obtain a first target generator.

[0144] In one example, the first acquisition module 501 is also used to acquire the first state information of the vehicle. The training module 502 is also used to input the first state information of the vehicle into the target decision maker to obtain the first behavior sequence corresponding to the first state information, and the target decision maker is trained according to a plurality of second training samples, each second training sample includes the second state information of the vehicle, and the second behavior sequence corresponding to the vehicle, and the second behavior sequence is obtained by simulated driving according to the second state information of the vehicle in a simulation environment using preset rules; the first training sample is constructed according to the first state information and the first behavior sequence corresponding to the first state information.

[0145] In one example, the training module 502 is also used to input the first state information into the initial generator to obtain a third behavior sequence output by the initial generator; if the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is less than a preset threshold, the penalty value of the initial generator is increased, and the parameters of the initial generator are adjusted according to the penalty value, and a first training sample is constructed according to the first state information and the first behavior sequence corresponding to the first state information; if the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is greater than or equal to a preset threshold, the reward value of the initial generator is increased, and the parameters of the initial generator are adjusted according to the reward value; and the initial generator with adjusted parameters is trained using multiple first training samples to obtain a first target generator.

[0146] In one example, the first acquisition module 501 is further used to acquire the third state information of multiple vehicles. The training module 502 is further used to input the third state information of each vehicle into the first target generator to obtain the behavior sequence corresponding to each third state information; based on the feedback indication input by the user, sample construction is performed on the multiple third state information and the behavior sequence corresponding to each third state information to obtain multiple third training samples; the multiple third training samples are used to train the first target generator to obtain the second target generator.

[0147] In one example, the generator training apparatus further includes an output module and an input module.

[0148] The output module is used to display multiple information pairs, each of which includes a third state information and a behavior sequence corresponding to the third state information. The input module receives feedback instructions input by the user. The training module 502 is also used to determine multiple third training samples, each of which includes a third state information in the multiple information pairs and a fourth behavior sequence corresponding to the third state information, and the fourth behavior sequence is determined according to the feedback instructions.

[0149] The generator training device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0150] The vehicle control method provided in the embodiment of the present application can be executed by a vehicle control device. In the embodiment of the present application, the vehicle control device provided in the embodiment of the present application is described by taking the vehicle control method executed by the vehicle control device as an example.

[0151] Figure 6 FIG. 1 shows a schematic diagram of the structure of a vehicle control device provided in an embodiment of the present application. Figure 6 As shown, the vehicle control device 60 of the present application includes: a second acquisition module 601 , a processing module 602 and a control module 603 .

[0152] The second acquisition module 601 is used to acquire the current state information of the vehicle.

[0153] The processing module 602 is used to input the current state information into the generator to obtain the target behavior sequence.

[0154] The control module 603 is used to control the vehicle according to the target behavior sequence, and the generator is obtained based on the above generator training method.

[0155] The vehicle control device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0156] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0157] The electronic device may include a processor 701 and a memory 702 storing computer program instructions.

[0158] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0159] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.

[0160] In some embodiments, the memory 702 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0161] The processor 701 implements any one of the generator training methods in the above embodiments by reading and executing computer program instructions stored in the memory 702.

[0162] In one example, the electronic device may further include a communication interface 707 and a bus 710. Figure 7 As shown, the processor 701, the memory 702, and the communication interface 707 are connected via a bus 710 and communicate with each other.

[0163] The communication interface 707 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0164] Bus 710 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 710 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0165] The electronic device can execute the generator training method in the embodiment of the present application, thereby realizing the combination Figure 1 and Figure 5 or, executing the vehicle control method in the embodiment of the present application, thereby realizing the combination of Figure 4 and Figure 6 A vehicle control method and device are described.

[0166] In addition, in combination with the generator training method or the vehicle control method in the above embodiment, the embodiment of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the generator training method or the vehicle control method in the above embodiment is implemented.

[0167] In combination with the generator training method or vehicle control method in the above-mentioned embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements the generator training method or vehicle control method in the above-mentioned embodiments.

[0168] An embodiment of the present application provides a vehicle, comprising: a processor and a memory storing computer program instructions;

[0169] When the processor executes the computer program instructions, the vehicle control method as described above is implemented.

[0170] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0171] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0172] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0173] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0174] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A generator training method, characterized in that: include: Acquire a plurality of first training samples, wherein the first training samples include first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, wherein the first behavior sequence is obtained according to a pre-acquired target decision maker output, wherein the target decision maker is used to simulate the decision-making behavior of a human expert; The initial generator is trained using a plurality of first training samples to obtain a first target generator.

2. The method according to claim 1, characterized in that The obtaining of a plurality of first training samples comprises: The process of obtaining each of the first training samples includes: Acquiring first status information of the vehicle; Inputting the first state information of the vehicle into the target decision maker to obtain a first behavior sequence corresponding to the first state information, wherein the target decision maker is trained based on a plurality of second training samples, each of which includes the second state information of the vehicle and a second behavior sequence corresponding to the vehicle, wherein the second behavior sequence is obtained by simulated driving according to the second state information of the vehicle in a simulation environment using preset rules; The first training sample is constructed according to the first state information and a first behavior sequence corresponding to the first state information.

3. The method according to claim 2, characterized in that The constructing the first training sample according to the first state information and the first behavior sequence corresponding to the first state information includes: Inputting the first state information into an initial generator to obtain a third behavior sequence output by the initial generator; If the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is less than a preset threshold, then increasing the penalty value of the initial generator, adjusting the parameters of the initial generator according to the penalty value, and constructing the first training sample according to the first state information and the first behavior sequence corresponding to the first state information; If the similarity between the third behavior sequence and the first behavior sequence corresponding to the first state information is greater than or equal to the preset threshold, increasing the reward value of the initial generator, and adjusting the parameters of the initial generator according to the reward value; The method of using a plurality of first training samples to train the initial generator to obtain a first target generator includes: The first target generator is obtained by training the initial generator with adjusted parameters using a plurality of the first training samples.

4. The method according to any one of claims 1 to 3, characterized in that After the initial generator is trained using a plurality of first training samples to obtain a first target generator, the method further includes: Acquiring third status information of a plurality of the vehicles; Inputting the third state information of each vehicle into the first target generator to obtain a behavior sequence corresponding to each third state information; Based on the feedback indication input by the user, sample construction is performed on the plurality of third state information and the behavior sequence corresponding to each of the third state information to obtain a plurality of third training samples; The first target generator is trained using a plurality of the third training samples to obtain a second target generator.

5. The method according to claim 4, characterized in that The feedback indication based on the user input, constructing samples of the behavior sequences corresponding to the plurality of the third state information and each of the third state information to obtain a plurality of third training samples, including: Displaying a plurality of information pairs, each of the information pairs comprising one of the third state information and a behavior sequence corresponding to the third state information; Receive the feedback indication input by the user, determine a plurality of the third training samples, each of the third training samples includes a third state information in the plurality of information pairs, and a fourth behavior sequence corresponding to the third state information, and the fourth behavior sequence is determined according to the feedback indication.

6. The method according to claim 5, characterized in that The fourth behavior sequence is the behavior sequence corresponding to the third state information, or the fourth behavior sequence is obtained by adjusting the behavior sequence corresponding to the third state information according to the feedback indication.

7. A vehicle control method, characterized in that: The method comprises: Get the current status information of the vehicle; Inputting the current state information into a generator to obtain a target behavior sequence, wherein the generator is obtained based on the generator training method described in any one of claims 1 to 6; The vehicle is controlled according to the target behavior sequence.

8. A generator training device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of first training samples, wherein the first training samples include first state information of a vehicle and a first behavior sequence of the vehicle corresponding to the first state information, wherein the first behavior sequence is obtained according to a pre-acquired target decision maker output, and the target decision maker is used to simulate the decision-making behavior of a human expert; A training module is used to train the initial generator using multiple first training samples to obtain a first target generator.

9. A vehicle control device, characterized in that: The device comprises: The second acquisition module is used to acquire the current state information of the vehicle; A processing module, used for inputting the current state information into a generator to obtain a target behavior sequence, wherein the generator is obtained based on the generator training method according to any one of claims 1 to 6; A control module is used to control the vehicle according to the target behavior sequence.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the generator training method as described in any one of claims 1-6, or the vehicle control method as described in claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the generator training method as described in any one of claims 1 to 6, or the vehicle control method as described in claim 7.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the generator training method as described in any one of claims 1 to 6, or the vehicle control method as described in claim 7.

13. A vehicle, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the vehicle control method as claimed in claim 7 is implemented.

Images