A network training and unmanned driving equipment control method and device

Abstract

The present specification discloses a network training and control method and device for unmanned driving equipment, which relates to the field of unmanned driving, obtains training samples, and then inputs the state information corresponding to the target object and the state information of the acquisition device as comprehensive state information into a preset attention network to obtain the attention of the acquisition device to the target object. For each driving strategy preset for the target object, based on the comprehensive state information, the evaluation value for the driving strategy is determined, and then the degree of difference in the evaluation values ​​between the driving strategies is determined, and the deviation between the degree of difference in the evaluation values ​​and the attention of the acquisition device to the target object is minimized as the optimization goal, and the attention network is trained, so that the unmanned driving device can select the driving strategy in combination with the attention to each target object when driving automatically, so that when there are an uncertain number of targets around the unmanned driving device, a reasonable driving strategy can also be determined.

Description

Technical Field

[0001] This specification relates to the field of unmanned driving, and in particular to a method and device for controlling network training and unmanned driving equipment. Background Art

[0002] In unmanned driving technology, unmanned driving equipment needs to control itself according to the status of surrounding pedestrians, vehicles and other targets during driving to ensure its own driving safety.

[0003] In the prior art, unmanned driving equipment can determine the next driving strategy through set rules, but the rules need to be set manually, and it is difficult to cope with all situations. Of course, unmanned driving equipment can also determine the next driving strategy by inputting the status of surrounding targets into the neural network, but usually the input dimension of the neural network is fixed, and the number of surrounding targets is usually not fixed. In this way, it is also difficult to judge the driving strategy.

[0004] Therefore, how to determine the driving strategy when the number of targets is not fixed is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a network training and unmanned driving equipment control method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a network training method, including:

[0008] Acquire training samples, wherein the training samples include state information of a collection device and state information of targets around the collection device;

[0009] Inputting the state information corresponding to the target object and the state information of the acquisition device into a preset attention network as comprehensive state information to obtain the attention degree of the acquisition device to the target object;

[0010] For each driving strategy preset for the target object, determining an evaluation value for the driving strategy based on the comprehensive state information;

[0011] The degree of difference in evaluation values ​​between the driving strategies is determined, and the attention network is trained with the optimization goal of minimizing the deviation between the degree of difference in evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to determine the attention degree to the target object, and to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object according to the determined attention degree, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

[0012] Optionally, for each driving strategy preset for the target object, determining an evaluation value for the driving strategy based on the comprehensive state information specifically includes:

[0013] For each driving strategy, the comprehensive state information and the driving strategy are input into a pre-trained evaluation network to determine an evaluation value for the driving strategy.

[0014] Optionally, training the policy network specifically includes:

[0015] Inputting the comprehensive state information into the strategy network to obtain the probability corresponding to each driving strategy determined by the strategy network based on the state of the target object around the acquisition device;

[0016] Determining an expected evaluation value according to the probability corresponding to each driving strategy and the evaluation value for each driving strategy;

[0017] In order to maximize the expected value of the evaluation, the policy network is subjected to reinforcement learning training.

[0018] Optionally, the training sample includes: state information of the acquisition device at a first historical moment and state information of the target object around the acquisition device, state information of the acquisition device at a second historical moment and state information of the target object around the acquisition device, a target driving strategy selected by the acquisition device at the first historical moment, an actual reward value obtained by the acquisition device executing the target driving strategy at the first historical moment, and the second historical moment is a moment after the first historical moment;

[0019] Training the evaluation network specifically includes:

[0020] The state information of the acquisition device at the first historical moment and the state information of the target object around the acquisition device are used as the first comprehensive state information, and the state information of the acquisition device at the second historical moment and the state information of the target object around the acquisition device are used as the second comprehensive state information;

[0021] Inputting the first comprehensive state information and the target driving strategy into an evaluation network to obtain an evaluation value corresponding to the target driving strategy;

[0022] determining, according to the second comprehensive status information, an evaluation value corresponding to the second historical moment;

[0023] determining a target evaluation value according to the evaluation value corresponding to the second historical moment and the actual reward value;

[0024] The evaluation value corresponding to the target driving strategy is used to approximate the target evaluation value as an optimization target, and reinforcement learning training is performed on the evaluation network.

[0025] Optionally, determining a target evaluation value according to the evaluation value corresponding to the second historical moment and the actual reward value specifically includes:

[0026] The target evaluation value is determined based on the evaluation value corresponding to the second historical moment, the actual reward value and the discount factor corresponding to the second historical moment, wherein for each historical moment after the first historical moment, the farther the first historical moment is from the historical moment, the smaller the impact of the evaluation value of the driving strategy corresponding to the historical moment on the target evaluation value under the influence of the discount factor corresponding to the historical moment.

[0027] Optionally, determining the evaluation value corresponding to the second historical moment according to the second comprehensive state information specifically includes:

[0028] For each driving strategy, input the second comprehensive state information and the driving strategy into the auxiliary evaluation network to obtain an evaluation value corresponding to each driving strategy at the second historical moment;

[0029] According to the evaluation value corresponding to each driving strategy at the second historical moment, the evaluation value corresponding to the second historical moment is determined, the auxiliary evaluation network is used to assist the evaluation network in reinforcement learning training, the auxiliary evaluation network has the same network structure as the evaluation network, and the parameter adjustment step size of the auxiliary evaluation network during the network reinforcement learning training process is smaller than the parameter adjustment step size of the evaluation network during the network reinforcement learning training process.

[0030] Optionally, determining the evaluation value corresponding to the second historical moment according to the evaluation value corresponding to each driving strategy at the second historical moment specifically includes:

[0031] Inputting the second comprehensive state information into an auxiliary strategy network to obtain the probability corresponding to each driving strategy at the second historical moment, and inputting the second comprehensive state information into an auxiliary attention network to obtain the attention degree for the target object around the collection device at the second historical moment;

[0032] According to the probability corresponding to each driving strategy at the second historical moment, the attention to the target objects around the collection device at the second historical moment, and the evaluation value corresponding to each driving strategy at the second historical moment, the evaluation value corresponding to the second historical moment is determined, the auxiliary strategy network is used to assist the evaluation network in reinforcement learning training, the auxiliary strategy network has the same network structure as the strategy network, and the parameter adjustment step of the auxiliary strategy network during the network reinforcement learning training process is smaller than the parameter adjustment step of the strategy network during the network reinforcement learning training process; the auxiliary attention network is used to assist the evaluation network in reinforcement learning training, the auxiliary attention network has the same network structure as the attention network, and the parameter adjustment step of the auxiliary attention network during the network reinforcement learning training process is smaller than the parameter adjustment step of the attention network during the network reinforcement learning training process.

[0033] Optionally, determining the actual reward value specifically includes:

[0034] The actual reward value is determined according to the driving state at the next moment after the acquisition device selects the target driving strategy at the first historical moment.

[0035] Optionally, determining the actual reward value according to the driving state after the acquisition device selects the target driving strategy at the first historical moment specifically includes:

[0036] Determine, according to the driving state, the speed and acceleration of the acquisition device after selecting the target driving strategy at the first historical moment;

[0037] According to the driving state, determining whether a collision occurs after the acquisition device selects the target driving strategy at the first historical moment, and obtaining a first determination result;

[0038] According to the driving state, determining whether the acquisition device passes through the location where the acquisition device is located after selecting the target driving strategy at the first historical moment, and obtaining a second determination result, wherein the location includes a crosswalk;

[0039] The actual reward value is determined according to the speed, the acceleration, the first judgment result, and the second judgment result.

[0040] This specification provides a control method for an unmanned driving device, including:

[0041] Obtaining status information of the current unmanned driving device and status information of each target object around the unmanned driving device;

[0042] For each target object, the state information of the target object and the state information of the unmanned driving device are used as the comprehensive state information corresponding to the target object;

[0043] Inputting the comprehensive state information corresponding to the target object into the attention network and the strategy network, obtaining the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object, the probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object, and the attention network and the strategy network are trained by a network training method;

[0044] According to the attention degree of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, a target driving strategy is selected from the driving strategies, and the unmanned driving device is controlled according to the selected target driving strategy.

[0045] This specification provides a network training device, including:

[0046] An acquisition module, used to acquire training samples, wherein the training samples include state information of a collection device and state information of targets around the collection device;

[0047] A comprehensive input module, used to input the state information corresponding to the target object and the state information of the acquisition device as comprehensive state information into a preset attention network to obtain the attention degree of the acquisition device to the target object;

[0048] An evaluation value determination module, for determining, for each driving strategy preset for the target object, an evaluation value for the driving strategy based on the comprehensive state information;

[0049] A training module is used to determine the degree of difference in evaluation values ​​between the driving strategies, and to train the attention network with the optimization goal of minimizing the deviation between the degree of difference in evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object through the determined attention degree to the target object, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

[0050] This specification provides a control device for unmanned driving equipment, including:

[0051] An acquisition module, used to acquire the status information of the current unmanned driving device and the status information of each target object around the unmanned driving device;

[0052] A determination module, for each target object, using the state information of the target object and the state information of the unmanned driving device as the comprehensive state information corresponding to the target object;

[0053] An input module, used to input the comprehensive state information corresponding to the target object into the attention network and the strategy network, to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object, the probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object, and the attention network and the strategy network are trained by a network training method;

[0054] The control module is used to select a target driving strategy from various driving strategies according to the attention degree of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, and control the unmanned driving device according to the selected target driving strategy.

[0055] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned network training and unmanned driving equipment control method is implemented.

[0056] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a network training method when executing the program.

[0057] This specification provides an unmanned driving device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a control method for the unmanned driving device when executing the program.

[0058] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0059] It can be seen from the above method that a training sample is obtained, which includes the state information of the acquisition device and the state information of the target object around the acquisition device. Then, the state information corresponding to the target object and the state information of the acquisition device are input into the preset attention network as comprehensive state information to obtain the attention of the acquisition device to the target object. For each driving strategy preset for the target object, based on the comprehensive state information, the evaluation value for the driving strategy is determined, and then the degree of difference in the evaluation values ​​between the driving strategies is determined, and the deviation between the degree of difference in the evaluation values ​​and the attention of the acquisition device to the target object is minimized as the optimization goal, and the attention network is trained. The trained attention network is used to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object through the determined attention to the target object, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

[0060] It can be seen from the above method that this method can train the attention network, so that when the unmanned driving device is driving automatically, through the attention network, the unmanned driving device can determine the attention to each target object, and then select the driving strategy based on the attention to each target object. In this way, when there are multiple targets around the unmanned driving device, a reasonable driving strategy can also be determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0062] Figure 1 A flowchart of a network training method in this specification;

[0063] Figure 2 A schematic diagram of training each network provided in this specification;

[0064] Figure 3 A flowchart of a control method for an unmanned driving device in this specification;

[0065] Figure 4 A schematic diagram of a network training device provided in this specification;

[0066] Figure 5 A schematic diagram of a control device for an unmanned driving device provided in this manual;

[0067] Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment;

[0068] Figure 7 The corresponding Figure 3 Schematic diagram of unmanned driving equipment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0070] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0071] Figure 1 The flowchart of a network training method in this specification specifically includes the following steps:

[0072] S101: Acquire training samples, where the training samples include status information of a collection device and status information of targets around the collection device.

[0073] In the field of unmanned driving, when an unmanned driving device is driving automatically, it needs to determine the current driving strategy at each moment based on its own driving status and the driving status of surrounding targets (such as pedestrians, vehicles, etc.) to control itself. In addition, there are often multiple targets around the unmanned driving device, so the unmanned driving device needs to determine the driving strategy in the presence of multiple targets. For example, when the unmanned driving device needs to pass through a crosswalk where there is no traffic light, and there are often multiple pedestrians at the crosswalk, in this case, in addition to the status of the unmanned driving device itself, the unmanned driving device also needs to determine the driving strategy based on the driving status of each pedestrian to ensure that it can safely pass through the crosswalk.

[0074] Therefore, the attention network and the strategy network in the network training method provided in this specification can be used together to determine a driving strategy for the unmanned driving device when there are multiple targets around the unmanned driving device.

[0075] Specifically, training samples can be obtained, which may include status information of a collection device and status information of targets around the collection device. The collection device may refer to an unmanned driving device or an ordinary driving device. The collection device may be used to collect status information of surrounding targets in real time during the training phase to perform certain training on the attention network, the policy network, and the evaluation network. Of course, the collection device may also refer to a virtual collection device in a simulation system, which simulates a real driving environment through a simulation system to obtain the status information of the above-mentioned collection device and the status information of targets around the collection device, thereby training the attention network, the policy network, and the evaluation network.

[0076] The status information of the acquisition device may refer to information related to the driving status of the acquisition device, and the status information of the target object around the acquisition device may refer to information related to the driving status of the target object around the acquisition device. Therefore, the status information of the acquisition device may include the real-time position information, speed magnitude, speed direction, acceleration magnitude and acceleration direction of the acquisition device, and the status information of the target object may include the relative position information, speed magnitude, speed direction, acceleration magnitude and acceleration direction between the target object and the acquisition device.

[0077] S102: Inputting the state information corresponding to the target object and the state information of the acquisition device into a preset attention network as comprehensive state information to obtain the attention of the acquisition device to the target object.

[0078] S103: For each driving strategy preset for the target object, determine an evaluation value for the driving strategy based on the comprehensive state information.

[0079] S104: Determine the degree of difference in evaluation values ​​between the driving strategies, and train the attention network with the optimization goal of minimizing the deviation between the degree of difference in evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to determine the attention degree to the target object, and according to the determined attention degree, compensate the probability of each driving strategy determined by the strategy network based on the state of the target object, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

[0080] After obtaining the above-mentioned training samples, the state information corresponding to the target object in the training sample and the state information of the acquisition device can be used as comprehensive state information, and the comprehensive state information can be input into a preset attention network to obtain the attention of the acquisition device to the target object. Then, for each driving strategy, based on the comprehensive state information, an evaluation value corresponding to each driving strategy can be obtained, wherein each driving strategy can include a preset acceleration and steering wheel angle.

[0081] The attention network can be used to determine the attention of the collection device (or unmanned driving device) to the surrounding targets. For a target object, the higher the attention of the collection device to the target object, the greater the influence of the target object on the selection of driving strategy of the collection device, which means that the collection device (or unmanned driving device) needs to pay more attention to the target object.

[0082] Therefore, when training the attention network, the difference in evaluation values ​​between the driving strategies can be determined, and the attention network can be trained with the optimization goal of minimizing the deviation between the difference in evaluation values ​​and the attention of the acquisition device to the target object. The evaluation value of a driving strategy can indicate the quality of the driving strategy. If the driving strategy can make the subsequent driving condition of the acquisition device better, the evaluation value of the driving strategy is high. Otherwise, the evaluation value of the driving strategy may be low.

[0083] The reason why minimizing the deviation between the difference in the evaluation value and the attention of the acquisition device to the target object is taken as the optimization goal is that for a target object, if the target object has little impact on the acquisition device (such as an unmanned driving device), then based on the state information of the target object, the evaluation value of each driving strategy determined may be similar, and if the target object has a greater impact on the acquisition device, then there may be a driving strategy whose evaluation value is much higher than the evaluation values ​​of other driving strategies. Therefore, the attention network can be optimized by the difference in the evaluation values ​​of each driving strategy, that is, the difference in the evaluation values, so that for a target object, the higher the difference in the evaluation values ​​of each driving strategy determined based on the state information of the target object (indicating that the target object has a greater impact on the driving strategy), the higher the attention of the target object, and the more the unmanned driving device will consider the target object when driving.

[0084] The degree of difference in the evaluation values ​​mentioned above can be determined in a variety of ways. For example, the variance of the evaluation values ​​corresponding to each driving strategy can be determined as the degree of difference in the evaluation values. For another example, the differences between the evaluation values ​​corresponding to different driving strategies in each driving strategy can be determined, and the differences can be summed up as the degree of difference in the evaluation values. The trained attention network can be used to determine the degree of attention to the target object, and according to the determined degree of attention, the probability of each driving strategy determined by the strategy network based on the state of the target object is compensated to obtain the driving strategy of the unmanned driving device under the influence of the target object (the specific process of this part will be described in the control method of the unmanned driving device below).

[0085] The evaluation value mentioned above can also be determined in a variety of ways, for example, it can be determined by a pre-trained evaluation network, that is, for each driving strategy, the driving strategy and the comprehensive state information can be input into the pre-trained evaluation network to determine the evaluation value for the driving strategy. Of course, other methods can also be used, such as pre-building a relationship table storing the evaluation values ​​corresponding to each comprehensive state information, and querying and determining through the relationship table.

[0086] Among them, the evaluation network also needs to be trained in advance. The evaluation network can only be used to train the above-mentioned attention network and strategy network, and does not need to be applied in the actual selection of driving strategies by unmanned driving equipment.

[0087] The specific process of training the evaluation network is as follows:

[0088] When training the evaluation network, the training samples need to include the state information of the collection device at the first historical moment and the state information of the target objects around the collection device, the state information of the collection device at the second historical moment and the state information of the target objects around the collection device, the target driving strategy selected by the collection device at the first historical moment, and the actual reward value obtained by the collection device when executing the target driving strategy at the first historical moment. The second historical moment is the moment after the first historical moment (which can be the next moment of the first historical moment).

[0089] The reason why both the status information of the acquisition device and the surrounding target objects at the first historical moment and the status information of the acquisition device and the surrounding target objects at the second historical moment are needed is because it is hoped that the evaluation value for a driving strategy at a certain moment determined by the evaluation network takes into account the driving status of the acquisition device at several moments after that moment, rather than just considering the reward (i.e., the actual reward value) that can be obtained immediately at the next moment after the driving strategy is selected and executed at that moment. That is to say, for that moment, if the acquisition device selects a driving strategy that brings better driving conditions in the future, it is hoped that the evaluation value of the driving strategy determined by the status information of the acquisition device and the status information of the target object at that moment will be higher.

[0090] Based on this, the server can use the status information corresponding to the target object at the first historical moment and the status information of the collection device at the first historical moment as the first comprehensive status information, and use the status information corresponding to the target object at the second historical moment and the status information of the collection device at the second historical moment as the second comprehensive status information, and input the first comprehensive status information and the target driving strategy into the evaluation network to obtain the evaluation value corresponding to the target driving strategy.

[0091] Then, the evaluation value corresponding to the second historical moment can be determined based on the second comprehensive state information, and the target evaluation value can be determined based on the evaluation value corresponding to the second historical moment and the actual reward value. The evaluation value corresponding to the target driving strategy is used to approximate the target evaluation value as the optimization target, and reinforcement learning training is performed on the evaluation network.

[0092] That is to say, for the evaluation value output by the evaluation network for the target driving strategy selected at the first historical moment, in addition to requiring the evaluation network to take into account the real reward (i.e., the actual reward value) obtained at the next moment after the target driving strategy is selected at the first historical moment, the evaluation network is also required to consider the evaluation values ​​at subsequent moments. In this way, the evaluation value determined by the evaluation network can give as comprehensive consideration to the driving process of the collection device as possible.

[0093] It should also be noted that in order to reduce the impact of the evaluation values ​​of subsequent moments on the evaluation value of the target driving strategy at the first historical moment, when training the evaluation network, the target evaluation value can be determined based on the evaluation value corresponding to the second historical moment, the actual reward value and the discount factor corresponding to the second historical moment. Among them, for each historical moment after the first historical moment, the farther the first historical moment is from this historical moment, the smaller the impact of the evaluation value of the driving strategy corresponding to this historical moment on the target evaluation value under the influence of the discount factor corresponding to this historical moment.

[0094] The evaluation value corresponding to the second historical moment mentioned above can be determined based on the evaluation value corresponding to each driving strategy at the second historical moment. In addition, the evaluation value corresponding to the second historical moment can be obtained not only through the evaluation network itself, but also through another auxiliary evaluation network with the same network structure as the evaluation network but with slower network parameter update.

[0095] Specifically, for each driving strategy, the second comprehensive state information and the driving strategy can be input into the auxiliary evaluation network to obtain the evaluation value corresponding to each driving strategy at the second historical moment, and the evaluation value corresponding to the second historical moment can be determined according to the evaluation value corresponding to each driving strategy at the second historical moment, wherein the auxiliary evaluation network can be used to assist the above-mentioned evaluation network in performing reinforcement learning training, the auxiliary evaluation network has the same network structure as the evaluation network, and the parameter adjustment step size of the auxiliary evaluation network during the network reinforcement learning training process is smaller than the parameter adjustment step size of the evaluation network during the network reinforcement learning training process.

[0096] That is to say, the training process of the auxiliary evaluation network is slower than that of the evaluation network. The auxiliary evaluation network is required to assist in calculating the objective function of the evaluation network and adjusting the network parameters of the evaluation network each time. In this way, the evaluation network will not be trained quickly, and the evaluation network can gradually learn the evaluation value of each driving strategy under each different comprehensive state information during the training process.

[0097] Of course, in order to slow down the training process of the evaluation network, in addition to adding an auxiliary evaluation network, an auxiliary strategy network and an auxiliary attention network can also be added during the training process of the evaluation network, such as Figure 2 shown.

[0098] Figure 2 A schematic diagram of training each network provided in this specification.

[0099] from Figure 2 It can be seen that the auxiliary policy network, auxiliary attention network and auxiliary evaluation network can all be used to strengthen the training of the evaluation network. The auxiliary policy network, auxiliary attention network and auxiliary evaluation network are consistent with the information and network results output by their corresponding policy network, attention network and evaluation network, but the parameter adjustment is slow during the training process. For example, for the auxiliary policy network and the policy network, after the network parameters of the policy network are updated once, the auxiliary policy network can be updated according to the network parameters of the policy network, and the network parameters of the policy network are given a lower weight, so that the network parameters of the auxiliary policy network are updated slower than the policy network parameters. The same is true for the auxiliary attention network and the auxiliary evaluation network. Then, when training the evaluation network, the evaluation network is reinforced with the results of the auxiliary policy network, the auxiliary attention network and the auxiliary evaluation network.

[0100] Specifically, the second comprehensive state information can be input into the auxiliary strategy network to obtain the probability corresponding to each driving strategy at the second historical moment, and the second comprehensive state information can be input into the auxiliary attention network to obtain the attention to the target objects around the collection device at the second historical moment, and the evaluation value corresponding to each driving strategy at the second historical moment is determined. The auxiliary strategy network is used to assist the evaluation network in reinforcement learning training. The auxiliary strategy network has the same network structure as the policy network. The parameter adjustment step of the auxiliary strategy network during the network training process is smaller than the parameter adjustment step of the policy network during the network training process. The auxiliary attention network is used to assist the evaluation network in reinforcement learning training. The auxiliary attention network has the same network structure as the attention network. The parameter adjustment step of the auxiliary attention network during the network reinforcement learning training process is smaller than the parameter adjustment step of the attention network during the network reinforcement learning training process.

[0101] The actual reward value mentioned above can be obtained by the real reward obtained by the collection device at the next moment after the collection device selects the target driving strategy at the first historical moment. Therefore, the above actual reward value can be determined based on the driving status at the next moment after the collection device selects the target driving strategy at the first historical moment.

[0102] Specifically, the speed and acceleration of the acquisition device after it selects the target driving strategy at the first historical moment can be determined based on the driving status. Whether a collision occurs after the acquisition device selects the target driving strategy at the first historical moment can be determined based on the driving status, and a first judgment result can be obtained. Based on the driving status, whether the acquisition device passes through the location where the acquisition device is located after selecting the target driving strategy at the first historical moment can be determined, and a second judgment result can be obtained. The locations mentioned here include crosswalks. Finally, the actual reward value can be determined based on the speed, acceleration, the first judgment result, and the second judgment result.

[0103] Among them, when calculating the actual reward value, some of the above factors are beneficial to the unmanned driving device, while some factors are undesirable for the unmanned driving device. For example, in terms of whether a collision occurs and acceleration, it is undesirable for the unmanned driving device to collide, and it is undesirable for the unmanned driving device to brake suddenly or start suddenly. Since a collision is a more serious situation, a higher weight can be given to the first judgment result. Therefore, after the unmanned driving device selects a certain driving strategy, if the unmanned driving device collides or the acceleration of the unmanned driving device is high, the actual reward value may be lower.

[0104] For another example, with respect to the speed and the second judgment result, if the speed is high, it can be judged that the driving condition of the collection device is relatively smooth, and if the second judgment result is that the collection device has passed the crosswalk, it also indicates that the driving condition of the collection device is good. Therefore, if the collection device has passed the crosswalk or the speed is high, the actual reward value may be higher.

[0105] The actual reward value can be calculated using the following formula:

[0106] r(s t , a t )=α1r collision +α2r acceleration +α3r done +α4r speed

[0107] The above r(s t ,a t ) is the number of a selected at time t t The actual reward value of this driving strategy, r collision Indicates whether a collision occurs. If a collision occurs, it can be 1, and if no collision occurs, it can be 0. α1 is the weight corresponding to the first judgment result. The weight can be negative and the value is large. Of course, r collision Assign a larger value, r acceleration can be the value of acceleration, α2 is the weight corresponding to the acceleration, which can be a negative value, r done Indicates whether the location where the collection device is located (such as a crosswalk), α3 is the corresponding weight, which can be a large positive value, r speed is the value of the speed, α3 is the weight corresponding to the speed, and the weight can be a positive value.

[0108] It should also be noted that the attention network and strategy network provided in this specification can be used to select the driving strategy of the unmanned driving device when the unmanned driving device is at a crosswalk where there is no traffic light. Therefore, when training the network, a real unmanned driving device can also be used as a collection device to drive at the crosswalk to obtain the state information of the unmanned driving device, the state information of the surrounding targets, the driving strategy of the unmanned driving device, the real reward value obtained by the unmanned driving device after adopting the driving strategy at that moment, and the state information of the unmanned driving device and the state information of the surrounding targets at the next moment, and store them as training samples.

[0109] A round of training can be performed on the attention network, policy network, and evaluation network every T seconds until the unmanned driving device crashes or crosses a pedestrian crossing, or the first two situations do not occur for a certain period of time, then this round of training is terminated, and after N cycles of training, the training of the attention network, policy network, and evaluation network is completed.

[0110] The above describes in detail various ways to train the evaluation network. The following formula is an example of an objective function for training the evaluation network:

[0111]

[0112] in,

[0113] In the above objective function, D is the sample set, which can include N training samples. To evaluate the output of the network for t i The comprehensive state information at this moment is t The evaluation value of this driving strategy, r(s t ,a t ) is the actual reward value obtained, is the evaluation value corresponding to each driving strategy at time t+1 output by the auxiliary evaluation network, is the regularization term, To assist the strategy network, a comprehensive state information at time t+1 is used to obtain a strategy vector, which contains the probability corresponding to each driving strategy. γ is the discount factor. means Satisfy the probability distribution corresponding to the auxiliary attention network.

[0114] Here is an example of an objective function for a policy network:

[0115]

[0116] in, is the regularization term, is the strategy vector. It can be seen from the above formula that the evaluation value is also needed when training the strategy network. That is to say, when training the strategy network, the comprehensive state information can be input into the strategy network to obtain the probability corresponding to each driving strategy determined by the strategy network based on the state of the target objects around the acquisition device, and the evaluation expectation value is determined according to the probability corresponding to each driving strategy and the evaluation value for each driving strategy, and then the strategy network is reinforced for learning and training by maximizing the evaluation expectation value. The evaluation expectation value mentioned here can refer to the weighted sum of the probability corresponding to each driving strategy and the evaluation value for each driving strategy, that is, by multiplying the evaluation value of each driving strategy by the probability corresponding to the corresponding driving strategy, and obtaining the sum value, the strategy network is reinforced for learning and training by maximizing the sum value, so that the driving strategy with a larger evaluation value has a higher corresponding probability.

[0117] The following objective function is an example of an objective function for an attention network:

[0118]

[0119] Among them, J C (ω) can be understood as the attention output by the attention network. t i The variance of the evaluation value corresponding to each driving strategy at this moment.

[0120] The above network training method is mainly used to explain the training process of the attention network, the policy network and the evaluation network. The attention network and the policy network need to be applied to the selection of driving strategies when the unmanned driving device is actually driving automatically. Therefore, the following is a description of a control method for an unmanned driving device provided in this specification. Figure 3 shown.

[0121] Figure 3 The following is a flow chart of a control method for an unmanned driving device in this specification, which specifically includes the following steps:

[0122] S301: Acquire status information of the current unmanned driving device and status information of each target object around the unmanned driving device.

[0123] S302: For each target object, the state information of the target object and the state information of the unmanned driving device are used as the comprehensive state information corresponding to the target object.

[0124] S303: The comprehensive state information corresponding to the target object is input into the attention network and the strategy network to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object. The probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object. The attention network and the strategy network are trained by a network training method.

[0125] S304: According to the attention degree of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, a target driving strategy is selected from the driving strategies, and the unmanned driving device is controlled according to the selected target driving strategy.

[0126] When determining the driving strategy of an unmanned driving device, the driving strategy can be selected only through the attention network and the strategy network. Specifically, the unmanned driving device can obtain its current state information and the state information of each target object around it, and for each target object, the state information of the target object and the state information of the unmanned driving device are used as the comprehensive state information corresponding to the target object.

[0127] Then, the comprehensive state information corresponding to the target object is input into the attention network and the strategy network to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object. The probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object.

[0128] According to the attention degree of the unmanned driving device to each target object, and the probability of each driving strategy corresponding to each target object (the probability of each driving strategy corresponding to each target object can be compensated by the attention degree of the unmanned driving device to each target object, thereby determining the target driving strategy), the target driving strategy is selected from the various driving strategies, and the unmanned driving device is controlled according to the selected target driving strategy, wherein the attention network and the strategy network are trained by the above-mentioned network training method.

[0129] Among them, the output of the strategy network can be a strategy vector, which contains the probability corresponding to each driving strategy. The output of the attention network is the attention. For each target object, the strategy vector corresponding to the target object can be determined, thereby obtaining the probability corresponding to each driving strategy determined for the unmanned driving device based on the state of the target object.

[0130] The strategy vectors corresponding to each target object and the attention degree corresponding to each target object can be obtained, and the strategy vectors corresponding to each target object are summarized to obtain the strategy matrix P = (p1,…,pN ), and summarize the attention corresponding to each target object to obtain the attention vector c=(c1,…,c N ), perform dot multiplication and normalization of the strategy matrix and the attention vector, and we can get the probability of each driving strategy corresponding to the state of each target object:

[0131] p out =softmax(P·c T )

[0132] Among them, p out The probability corresponding to each driving strategy is included, and the driving strategy with the highest probability can be selected as the target driving strategy, so that the unmanned driving device can control itself according to the target driving strategy. It should be noted that multiple driving strategies can be preset, and a strategy vector contains the probability determined for each driving strategy. Through the strategy vector corresponding to each target object and the corresponding attention degree of the target object, the final probability of each driving strategy determined by combining various target objects can be determined to select the driving strategy.

[0133] It should be noted that the network training method and the control method of the unmanned driving device in this specification are implemented based on reinforcement learning, that is, the evaluation network, strategy network and attention network in this specification are implemented based on the framework of reinforcement learning, and the training samples in this specification are derived from a constructed experience pool, which is obtained by driving a driving device (such as an unmanned driving device) or a simulated driving device on a road (or a simulated road environment) at each moment, The state information of the driving device, the state information of the pedestrian, the calculated reward value and the state information at the next moment, so that this information is stored in the experience pool as a piece of experience information, and each training sample can be obtained through a piece of experience information in the experience pool.

[0134] The unmanned driving equipment mentioned above may refer to unmanned vehicles, drones, automatic delivery equipment and other equipment that can realize automatic driving. Based on this, the network training and unmanned driving equipment methods provided in this specification can be used to select driving strategies for unmanned driving equipment when there are multiple targets around. The unmanned driving equipment can be specifically applied to the field of delivery through unmanned driving equipment, such as the business scenarios of using unmanned driving equipment for express delivery, logistics, take-out and other delivery.

[0135] It can be seen from the above method that the attention network can be trained. During the training, by minimizing the deviation between the difference in evaluation values ​​and the attention of the acquisition device to the target object, the attention network can output a higher attention to the target object that has a greater impact on the driving strategy. Therefore, when the unmanned driving device is driving automatically, the unmanned driving device can determine the attention to each target object through the attention network, and then select the driving strategy based on the attention to each target object. In this way, when there are multiple target objects around the unmanned driving device, a reasonable driving strategy can also be determined.

[0136] The above is a network training and unmanned driving equipment control method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding network training and unmanned driving equipment control device, such as Figure 4 , Figure 5 shown.

[0137] Figure 4 A schematic diagram of a network training device provided in this specification specifically includes:

[0138] An acquisition module 401 is used to acquire training samples, wherein the training samples include state information of a collection device and state information of objects around the collection device;

[0139] A comprehensive input module 402 is used to input the state information corresponding to the target object and the state information of the acquisition device as comprehensive state information into a preset attention network to obtain the attention degree of the acquisition device to the target object;

[0140] An evaluation value determination module 403, for determining, for each driving strategy preset for the target object, an evaluation value for the driving strategy based on the comprehensive state information;

[0141] The training module 404 is used to determine the degree of difference in evaluation values ​​between the driving strategies, and to train the attention network with the optimization goal of minimizing the deviation between the degree of difference in evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object through the determined attention degree to the target object, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

[0142] Optionally, the evaluation value determination module 403 is specifically used to, for each driving strategy, input the comprehensive state information and the driving strategy into a pre-trained evaluation network to determine the evaluation value for the driving strategy.

[0143] Optionally, the training module 404 is also used to input the comprehensive state information into the strategy network to obtain the probability corresponding to each driving strategy determined by the strategy network based on the state of the target objects around the acquisition device; determine the evaluation expectation value according to the probability corresponding to each driving strategy and the evaluation value of each driving strategy; and perform reinforcement learning training on the strategy network to maximize the evaluation expectation value.

[0144] Optionally, the training sample includes: state information of the acquisition device at a first historical moment and state information of the target object around the acquisition device, state information of the acquisition device at a second historical moment and state information of the target object around the acquisition device, a target driving strategy selected by the acquisition device at the first historical moment, an actual reward value obtained by the acquisition device executing the target driving strategy at the first historical moment, and the second historical moment is a moment after the first historical moment;

[0145] The training module 404 is specifically used to use the state information of the collection device at the first historical moment and the state information of the target object around the collection device as the first comprehensive state information, and the state information of the collection device at the second historical moment and the state information of the target object around the collection device as the second comprehensive state information; input the first comprehensive state information and the target driving strategy into the evaluation network to obtain the evaluation value corresponding to the target driving strategy; determine the evaluation value corresponding to the second historical moment according to the second comprehensive state information; determine the target evaluation value according to the evaluation value corresponding to the second historical moment and the actual reward value; use the evaluation value corresponding to the target driving strategy to approximate the target evaluation value as the optimization target, and perform reinforcement learning training on the evaluation network.

[0146] Optionally, the training module 404 is specifically used to determine the target evaluation value based on the evaluation value corresponding to the second historical moment, the actual reward value and the discount factor corresponding to the second historical moment, wherein for each historical moment after the first historical moment, the farther the first historical moment is from the historical moment, the evaluation value of the driving strategy corresponding to the historical moment has a smaller impact on the target evaluation value under the influence of the discount factor corresponding to the historical moment.

[0147] Optionally, the training module 404 is specifically used to, for each driving strategy, input the second comprehensive state information and the driving strategy into the auxiliary evaluation network to obtain the evaluation value corresponding to each driving strategy at the second historical moment; determine the evaluation value corresponding to the second historical moment according to the evaluation value corresponding to each driving strategy at the second historical moment, the auxiliary evaluation network is used to assist the evaluation network in performing reinforcement learning training, the auxiliary evaluation network has the same network structure as the evaluation network, and the parameter adjustment step size of the auxiliary evaluation network during the network reinforcement learning training process is smaller than the parameter adjustment step size of the evaluation network during the network reinforcement learning training process.

[0148] Optionally, the training module 404 is specifically used to input the second comprehensive state information into the auxiliary strategy network to obtain the probability corresponding to each driving strategy at the second historical moment, and input the second comprehensive state information into the auxiliary attention network to obtain the attention to the target objects around the collection device at the second historical moment; determine the evaluation value corresponding to the second historical moment according to the probability corresponding to each driving strategy at the second historical moment, the attention to the target objects around the collection device at the second historical moment, and the evaluation value corresponding to each driving strategy at the second historical moment, the auxiliary strategy network is used to assist the evaluation network in reinforcement learning training, the auxiliary strategy network has the same network structure as the strategy network, and the parameter adjustment step of the auxiliary strategy network during the network reinforcement learning training process is smaller than the parameter adjustment step of the strategy network during the network reinforcement learning training process, the auxiliary attention network is used to assist the evaluation network in reinforcement learning training, the auxiliary attention network has the same network structure as the attention network, and the parameter adjustment step of the auxiliary attention network during the network reinforcement learning training process is smaller than the parameter adjustment step of the attention network during the network reinforcement learning training process.

[0149] Optionally, the training module 404 is specifically used to determine the actual reward value according to the driving state at the next moment after the acquisition device selects the target driving strategy at the first historical moment.

[0150] Optionally, the training module 404 is specifically used to determine, based on the driving state, the speed and acceleration of the acquisition device after it selects the target driving strategy at the first historical moment; based on the driving state, determine whether a collision occurs after the acquisition device selects the target driving strategy at the first historical moment, and obtain a first judgment result; based on the driving state, determine whether the acquisition device passes through a location where the acquisition device is located after selecting the target driving strategy at the first historical moment, and obtain a second judgment result, wherein the location includes a crosswalk; and determine the actual reward value based on the speed, acceleration, the first judgment result, and the second judgment result.

[0151] Figure 5 A schematic diagram of a control device for an unmanned driving device provided in this specification specifically includes:

[0152] An acquisition module 501 is used to acquire the status information of the current unmanned driving device and the status information of each target object around the unmanned driving device;

[0153] A determination module 502 is used for, for each target object, using the state information of the target object and the state information of the unmanned driving device as the comprehensive state information corresponding to the target object;

[0154] An input module 503 is used to input the comprehensive state information corresponding to the target object into the attention network and the strategy network to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object. The probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object. The attention network and the strategy network are trained by a network training method;

[0155] The control module 504 is used to select a target driving strategy from various driving strategies according to the attention of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, and control the unmanned driving device according to the selected target driving strategy.

[0156] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 or Figure 3 The network training and control method of unmanned driving equipment.

[0157] This manual also provides Figure 6 , 7 The schematic structure diagram of the electronic equipment and unmanned driving equipment shown in the figure. Figure 6 , 7As mentioned above, at the hardware level, the electronic device and the unmanned driving device include a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 or Figure 3 The network training and control method of the unmanned driving device. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0158] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0159] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0160] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0161] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0162] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0168] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0169] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0170] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0172] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A network training method, characterized in that: include: Acquire training samples, wherein the training samples include state information of a collection device and state information of targets around the collection device; Inputting the state information corresponding to the target object and the state information of the acquisition device into a preset attention network as comprehensive state information to obtain the attention degree of the acquisition device to the target object; For each driving strategy preset for the target object, determining an evaluation value for the driving strategy based on the comprehensive state information; The degree of difference in evaluation values ​​between the driving strategies is determined, and the attention network is trained with the optimization goal of minimizing the deviation between the degree of difference in the evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to determine the attention degree to the target object, and to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object according to the determined attention degree, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

2. The method according to claim 1, characterized in that For each driving strategy preset for the target object, determining an evaluation value for the driving strategy based on the comprehensive state information specifically includes: For each driving strategy, the comprehensive state information and the driving strategy are input into a pre-trained evaluation network to determine an evaluation value for the driving strategy.

3. The method according to claim 1, characterized in that Training the policy network includes: Inputting the comprehensive state information into the strategy network to obtain the probability corresponding to each driving strategy determined by the strategy network based on the state of the target object around the acquisition device; Determining an expected evaluation value according to the probability corresponding to each driving strategy and the evaluation value for each driving strategy; In order to maximize the expected value of the evaluation, the policy network is subjected to reinforcement learning training.

4. The method according to claim 2, characterized in that The training samples include: state information of the acquisition device at a first historical moment and state information of the target objects around the acquisition device, state information of the acquisition device at a second historical moment and state information of the target objects around the acquisition device, the target driving strategy selected by the acquisition device at the first historical moment, and the actual reward value obtained by the acquisition device executing the target driving strategy at the first historical moment, wherein the second historical moment is a moment after the first historical moment; Training the evaluation network specifically includes: The state information of the acquisition device at the first historical moment and the state information of the target object around the acquisition device are used as the first comprehensive state information, and the state information of the acquisition device at the second historical moment and the state information of the target object around the acquisition device are used as the second comprehensive state information; Inputting the first comprehensive state information and the target driving strategy into an evaluation network to obtain an evaluation value corresponding to the target driving strategy; determining, according to the second comprehensive status information, an evaluation value corresponding to the second historical moment; determining a target evaluation value according to the evaluation value corresponding to the second historical moment and the actual reward value; The evaluation value corresponding to the target driving strategy is used to approximate the target evaluation value as an optimization target, and reinforcement learning training is performed on the evaluation network.

5. The method according to claim 4, characterized in that Determining a target evaluation value according to the evaluation value corresponding to the second historical moment and the actual reward value specifically includes: The target evaluation value is determined based on the evaluation value corresponding to the second historical moment, the actual reward value and the discount factor corresponding to the second historical moment, wherein for each historical moment after the first historical moment, the farther the first historical moment is from the historical moment, the smaller the impact of the evaluation value of the driving strategy corresponding to the historical moment on the target evaluation value under the influence of the discount factor corresponding to the historical moment.

6. The method according to claim 4, characterized in that Determining, according to the second comprehensive state information, an evaluation value corresponding to the second historical moment specifically includes: For each driving strategy, input the second comprehensive state information and the driving strategy into the auxiliary evaluation network to obtain an evaluation value corresponding to each driving strategy at the second historical moment; According to the evaluation value corresponding to each driving strategy at the second historical moment, the evaluation value corresponding to the second historical moment is determined, the auxiliary evaluation network is used to assist the evaluation network in reinforcement learning training, the auxiliary evaluation network has the same network structure as the evaluation network, and the parameter adjustment step size of the auxiliary evaluation network during the network reinforcement learning training process is smaller than the parameter adjustment step size of the evaluation network during the network reinforcement learning training process.

7. The method according to claim 6, characterized in that Determining the evaluation value corresponding to the second historical moment according to the evaluation value corresponding to each driving strategy at the second historical moment specifically includes: Inputting the second comprehensive state information into an auxiliary strategy network to obtain the probability corresponding to each driving strategy at the second historical moment, and inputting the second comprehensive state information into an auxiliary attention network to obtain the attention degree for the target object around the collection device at the second historical moment; According to the probability corresponding to each driving strategy at the second historical moment, the attention to the target objects around the collection device at the second historical moment, and the evaluation value corresponding to each driving strategy at the second historical moment, the evaluation value corresponding to the second historical moment is determined, the auxiliary strategy network is used to assist the evaluation network in reinforcement learning training, the auxiliary strategy network has the same network structure as the strategy network, and the parameter adjustment step of the auxiliary strategy network during the network reinforcement learning training process is smaller than the parameter adjustment step of the strategy network during the network reinforcement learning training process; the auxiliary attention network is used to assist the evaluation network in reinforcement learning training, the auxiliary attention network has the same network structure as the attention network, and the parameter adjustment step of the auxiliary attention network during the network reinforcement learning training process is smaller than the parameter adjustment step of the attention network during the network reinforcement learning training process.

8. The method according to claim 4, characterized in that Determining the actual reward value specifically includes: The actual reward value is determined according to the driving state at the next moment after the acquisition device selects the target driving strategy at the first historical moment.

9. The method according to claim 8, characterized in that Determining the actual reward value according to the driving state after the acquisition device selects the target driving strategy at the first historical moment specifically includes: Determine, according to the driving state, the speed and acceleration of the acquisition device after selecting the target driving strategy at the first historical moment; According to the driving state, determining whether a collision occurs after the acquisition device selects the target driving strategy at the first historical moment, and obtaining a first determination result; According to the driving state, determining whether the acquisition device passes through the location where the acquisition device is located after selecting the target driving strategy at the first historical moment, and obtaining a second determination result, wherein the location includes a crosswalk; The actual reward value is determined according to the speed, the acceleration, the first judgment result, and the second judgment result.

10. A control method for an unmanned driving device, characterized in that: include: Obtaining status information of the current unmanned driving device and status information of each target object around the unmanned driving device; For each target object, the state information of the target object and the state information of the unmanned driving device are used as the comprehensive state information corresponding to the target object; The comprehensive state information corresponding to the target object is input into the attention network and the strategy network to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object, the probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object, and the attention network and the strategy network are trained by the method according to any one of claims 1 to 9; According to the attention degree of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, a target driving strategy is selected from the driving strategies, and the unmanned driving device is controlled according to the selected target driving strategy.

11. A network training device, characterized in that: include: An acquisition module, used to acquire training samples, wherein the training samples include status information of a collection device and status information of targets around the collection device; A comprehensive input module, used to input the state information corresponding to the target object and the state information of the acquisition device as comprehensive state information into a preset attention network to obtain the attention degree of the acquisition device to the target object; An evaluation value determination module, for determining, for each driving strategy preset for the target object, an evaluation value for the driving strategy based on the comprehensive state information; A training module is used to determine the degree of difference in evaluation values ​​between the driving strategies, and to train the attention network with the optimization goal of minimizing the deviation between the degree of difference in the evaluation values ​​and the attention degree of the acquisition device to the target object. The trained attention network is used to compensate the probability of each driving strategy determined by the strategy network based on the state of the target object through the determined attention degree to the target object and according to the determined attention degree, so as to obtain the driving strategy of the unmanned driving device under the influence of the target object.

12. A control device for unmanned driving equipment, characterized in that: include: An acquisition module, used to acquire the status information of the current unmanned driving device and the status information of each target object around the unmanned driving device; A determination module, for each target object, using the state information of the target object and the state information of the unmanned driving device as the comprehensive state information corresponding to the target object; An input module, used to input the comprehensive state information corresponding to the target object into an attention network and a strategy network, to obtain the attention of the unmanned driving device to the target object and the probability of each driving strategy corresponding to the target object, the probability of each driving strategy corresponding to the target object is the probability corresponding to each driving strategy determined by the unmanned driving device based on the state of the target object, the attention network and the strategy network are trained by the method according to any one of claims 1 to 9; The control module is used to select a target driving strategy from various driving strategies according to the attention degree of the unmanned driving device to each target object and the probability of each driving strategy corresponding to each target object, and control the unmanned driving device according to the selected target driving strategy.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 9 or 10 is implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 9 is implemented.

15. An unmanned driving device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to claim 10 is implemented.

Images