Decision-making method, decision-making model training method and device based on multi-task learning

Abstract

The present application discloses a decision method, a decision model training method and a device based on multi-task learning in the field of artificial intelligence. The decision model training method comprises: randomly acquiring a plurality of sample data from a first sample database, wherein the first sample database comprises sample data of a plurality of candidate tasks, the sample data of a target task comprises a task vector corresponding to the target task, the task vector corresponding to the target task is obtained based on common subtasks and characteristic subtasks in the plurality of candidate tasks, and the target task is any one of the plurality of candidate tasks; adjusting the decision model M according to the plurality of sample data; t The parameters in the decision model M t+1 ; Judgment decision model M t+1 Whether it converges; when the decision model M t+1 When convergence occurs, the decision model M is determined t+1 The target decision model is adopted. The decision effect and convergence ability of the decision model are improved by using the embodiment of the present application, and the mutual influence between multiple tasks is avoided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a decision-making method based on multi-task learning, a decision-making model training method and a device. Background Art

[0002] Reinforcement learning is an important branch of artificial intelligence, and its ability to complete certain specific tasks has surpassed that of ordinary humans. However, for reinforcement learning algorithms, the model obtained after one training can usually only be used for a specific task. If it is to be applied to another new task, it needs to be retrained to obtain a new model. This means that although the training algorithm is general, the learned model can only be applied to specific task scenarios.

[0003] With the increasing application of reinforcement learning algorithms in the industry, many application scenarios require not only reinforcement learning models to handle a single task, but also models to achieve better results in multi-task scenarios. Multiple tasks mean that the reinforcement learning algorithm needs to learn multiple Markov models, but the non-unique state transition probability may cause the reinforcement learning algorithm to converge poorly or even fail to converge. In addition, due to the different reward mechanisms for different tasks, a simple task may quickly dominate the model effect, and other sparse reward tasks may not be explored enough, resulting in uneven learning effects and poor overall model effects. In response to the above problems, there is an urgent need for a reinforcement learning algorithm that can learn multiple tasks simultaneously.

[0004] An existing solution is to balance limited resources in a single learning algorithm to meet the learning of multiple tasks. Many learning algorithms will have poor results due to the trade-offs between multiple tasks. For example, during the learning process, some tasks have large reward values, which causes the algorithm to focus on tasks with outstanding reward values ​​at the expense of versatility, resulting in other tasks not being able to achieve good results. There are also algorithms that unify the reward values ​​of each task by reducing rewards. This approach may change the optimization goal. If the reward values ​​are all large non-negative values, then after the reduction, it becomes optimizing the frequency of obtaining rewards rather than accumulating expected rewards. In addition, the balance of the algorithm between tasks depends not only on the size of the reward value, but also on the reward density. Reward reduction will still lead to imbalance in the algorithm between different tasks.

[0005] Another solution is called distillation-based learning: it mainly builds a student network through supervised learning of multiple expert networks for specific tasks. This learning algorithm provides a compromise result of multi-task strategy, and each expert network needs to be trained on a large scale in advance. Although this learning algorithm avoids the problem of unbalanced reward values, it still balances among multiple tasks, the learning effect is not ideal, and its performance is limited by the expert network and cannot be further improved. Summary of the invention

[0006] The embodiments of the present application provide a decision-making method, a decision model training method and a device based on multi-task learning. In the scheme of the embodiments of the present application, by jointly characterizing the tasks, a task vector obtained by characteristic sub-tasks and common sub-tasks is obtained. When training the model, the mutual influence between multiple tasks can be avoided, and the strategy learning of multiple tasks can be promoted through common sub-tasks. Targeted learning of tasks is performed through characteristic sub-tasks, thereby improving the multi-task strategy effect and the convergence speed of the model. When making decisions, the same model can be used to make decisions on multiple tasks, thereby avoiding the mutual influence between multiple tasks.

[0007] In a first aspect, an embodiment of the present application provides a training method based on a multi-task learning decision model, comprising:

[0008] S1: Obtain the target task from multiple candidate tasks; and obtain the status information s of the target task according to the target task t and obtain the task vector corresponding to the target task according to the target task. The task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks of multiple candidate tasks; S2: according to the state information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample annual data of the target task, add the sample data of the target task to the preliminary sample database, and obtain a first sample database; S3: randomly obtain multiple sample data from the first sample database; the multiple sample data are sample data of some or all of the candidate tasks in the multiple candidate tasks; S4: use the reinforcement learning method to adjust the decision model M according to the multiple sample data t The parameters in the decision model M t+1 ; S5: Judgment decision model M t+1 Whether it converges; when the decision model M t+1 When convergence, determine the decision model M t+1 The target decision model.

[0009] Among them, the common subtask is a subtask shared by the subtasks of multiple candidate tasks, and the characteristic subtask is a subtask unique to a candidate task among the subtasks of multiple candidate tasks. For example, in the intersection scenario, the above multiple candidate tasks may include going straight at the intersection, turning left at the intersection, and turning right at the intersection; going straight at the intersection includes two subtasks: collision or arrival at the intersection and improving traffic efficiency; turning left at the intersection includes two subtasks: collision or arrival at the intersection and improving traffic efficiency; turning right at the intersection includes two subtasks: collision or arrival at the intersection and improving traffic efficiency. Among them, collision or arrival at the intersection, collision or arrival at the intersection, collision or arrival at the intersection, and collision or arrival at the intersection are characteristic subtasks, and improving traffic efficiency is a common subtask.

[0010] Optionally, the above reinforcement learning method may be a reinforcement learning method based on a value function.

[0011] Based on the characteristic subtasks and common subtasks in multiple candidate tasks, the task vector of any candidate task in multiple candidate tasks is obtained, and the strategy of learning multiple candidate tasks through one model is realized. Among them, the common subtasks of multiple candidate tasks can promote the learning of multiple candidate task strategies and improve the model convergence ability; the characteristic subtasks are used to carry out targeted learning of multiple candidate tasks, which avoids the mutual influence between multiple tasks and the compromise of the model between multiple tasks, and realizes that the same model can achieve excellent results when making decisions on multiple tasks.

[0012] In a feasible embodiment, obtaining a task vector corresponding to the target task according to the target task includes:

[0013] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0014] In a feasible embodiment, according to the state information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample data for the target task, including:

[0015] The status information of the target task s t The task vector corresponding to the target task is input into the decision model M t In order to obtain the candidate actions of the target task, the target action of the target task is selected from the candidate actions of the target task and the actions randomly obtained from the action space according to the preset probability. The probability of the candidate action of the target task being selected is the preset probability. After the target action is executed, the state information s of the target task is obtained. t+1 , and according to the status information s of the target task t+1 Get the reward value vector of the target task; the reward value in the reward value vector corresponds to the subtask in the task vector corresponding to the target task; the sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task t , target action of target task, state information of target task s t+1 and the reward value vector of the target task.

[0016] By constructing a reward value vector, the execution status of each subtask in the target task is fed back to the decision model for learning, thereby improving the accuracy of the decision model.

[0017] In a feasible embodiment, according to the state information s of the target task t The task vector corresponding to the target task is input into the decision model M t In order to obtain candidate actions for the target task, the following are processed:

[0018] Decision Model M t According to the status information s of the target task t The task vector corresponding to the target task obtains the action value function vector of the target task, and the action value function in the action value function vector of the target task corresponds one to one with the subtask corresponding to the element in the task vector corresponding to the target task; the decision model M t The value function of the target task is obtained according to the action value function vector of the target task and the task vector corresponding to the target task; the decision model M t According to the value function of the target task, the candidate action of the target task is obtained from the action space. The candidate action of the target task is the action in the action space that maximizes the value function of the target task.

[0019] By obtaining the value function of the target task according to the task vector and action value function vector corresponding to the target task, and then determining the target action based on the value function, it is avoided that the action value function of the subtask irrelevant to the target task in the action value function vector affects the selection of the target action when making decisions on the target task.

[0020] In a feasible embodiment, selecting a target action of the target task from candidate actions of the target task and actions randomly obtained from the action space according to a preset probability includes:

[0021] When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action of the target task; the first parameter is a random number in the value range [0,1]; when the first parameter is not greater than the preset probability, the action randomly obtained from the action space is determined as the target action of the target task.

[0022] By randomly obtaining actions from the action space and determining them as the target actions of the target task, it is possible to explore new actions when training the decision model, thereby obtaining better actions when using the decision model.

[0023] It should be pointed out here that the initial value of the preset probability is 1 or a larger value close to 1; the preset probability gradually decreases as the number of training times increases.

[0024] In a feasible embodiment, the decision model M is adjusted according to a plurality of sample data.t The parameters in the decision model M t+1 include:

[0025] According to the loss function and the state information s of each sample data in multiple sample data t , task vector, target action, state information s t+1 The loss value is calculated by the reward value vector; the decision model M is adjusted according to the loss value t The parameters in the decision model M t+1 .

[0026] The first state information is obtained before the target action is executed, and the second state information is obtained after the target action is executed.

[0027] Among them, the loss value can be expressed as:

[0028]

[0029] Among them, r in the above formula is the reward value vector in the above sample data, g is the task vector in the above sample data, and a t is the target action in the above sample data, and the discount coefficient γ is a constant.

[0030] In a feasible embodiment, the decision model M is judged t+1 Whether it is convergent, including:

[0031] According to the status information s of the target task t+1 Determine whether the target task is completed; when it is determined that the target task is not completed, set t=t+1, and repeat steps S2-S5 until the target task is completed;

[0032] When the target task is determined to be completed, the decision model M t+1 Convergence; in determining the decision model M t+1 If it does not converge, set t = t + 1 and repeat steps S1-S5 until the decision model M t+1 convergence.

[0033] In a second aspect, an embodiment of the present application provides a decision-making method based on multi-task learning, including:

[0034] Get multiple candidate tasks, get the target task from the multiple candidate tasks; and get the status information of the target task according to the target task t ; Perform task joint characterization on the target task according to multiple candidate tasks to obtain the task vector corresponding to the target task. The task vector corresponding to each candidate task is obtained based on the characteristic subtasks and common subtasks of multiple candidate tasks; According to the task vector corresponding to the target task and the state information s of the target task tDetermine the target action from the action space.

[0035] The multiple candidate tasks may be tasks in the same scenario or tasks in different scenarios.

[0036] By jointly representing the target task based on the subtasks of multiple candidate tasks, it is possible to make decisions on multiple tasks using the same model and avoid mutual influence between multiple tasks.

[0037] In a feasible embodiment, a target task is jointly characterized according to a plurality of candidate tasks to obtain a task vector corresponding to the target task, including:

[0038] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0039] In a feasible embodiment, according to the task vector corresponding to the target task and the state information s of the target task t Determine the target action from the action space, including:

[0040] The task vector corresponding to the target task and the state information s of the target task are t The input is processed into the target decision model to obtain the target action. The target decision model is implemented based on a neural network.

[0041] Optionally, the neural network may be a fully connected neural network, a convolutional neural network, a recurrent neural network or other neural networks.

[0042] In a feasible embodiment, the task vector corresponding to the target task and the state information s of the target task are t Input into the target decision model for processing to obtain the target action, including:

[0043] According to the task vector corresponding to the target task and the state information s of the target task t Obtain an action value function vector for the target task, wherein the action value function in the action value function vector corresponds one-to-one to the subtask corresponding to the element in the task vector corresponding to the target task; obtain the value function of the target task according to the task vector and the action value function vector corresponding to the target task; obtain the target action from the action space according to the value function of the target task, and the target action is the action in the action space that maximizes the value function of the target task.

[0044] By obtaining the value function of the target task according to the task vector and action value function vector corresponding to the target task, and then determining the target action based on the value function, it is avoided that the action value function of the subtask irrelevant to the target task in the action value function vector affects the selection of the target action when making decisions on the target task, thereby improving the decision-making effect of the target task.

[0045] Optionally, the above value function may be a Q value function, and the Q value function of the target task may be expressed as: R(s t ,a k ,g), the target action can be expressed as:

[0046] Among them, g is the task vector corresponding to the target task, a k is the action in the action space.

[0047] In a third aspect, an embodiment of the present application provides a decision model training device based on multi-task learning, comprising:

[0048] an acquisition unit, configured to randomly acquire a plurality of sample data from a first sample database; the first sample database includes sample data of a plurality of candidate tasks, the sample data of a target task includes a task vector corresponding to the target task, the task vector corresponding to the target task is obtained based on common subtasks and characteristic subtasks in the plurality of candidate tasks, and the target task is any one of the plurality of candidate tasks;

[0049] An adjustment unit is used to adjust the decision model M according to multiple sample data using a reinforcement learning method. t The parameters in the decision model M t+1 ;

[0050] Determination unit, used when the decision model M t+1 When convergence, determine the decision model M t+1 The target decision model.

[0051] In a feasible embodiment, the acquisition unit is used to acquire a target task from multiple candidate tasks; and acquire the state information s of the target task according to the target task. t And obtain the task vector corresponding to the target task according to the target task,

[0052] The training device further comprises:

[0053] Update unit, used to update the state information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample data of the target task, and add the sample data of the target task to the preliminary sample database to obtain a first sample database;

[0054] In a feasible embodiment, in terms of acquiring the task vector corresponding to the target task according to the target task, the acquiring unit is specifically used to:

[0055] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0056] In a feasible embodiment, according to the state information s of the target task t , the task vector of the target task and the decision model M t In terms of generating sample data for the target task, the update unit is specifically used to:

[0057] According to the status information of the target task s t The task vector corresponding to the target task is input into the decision model M t , to obtain the candidate actions of the target task; select the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to the preset probability, and the probability of the candidate action of the target task being selected is the preset probability; after the target action is executed, obtain the state information s of the target task t+1 , and according to the status information s of the target task t+1 Get the reward value vector of the target task; the reward value in the reward value vector corresponds to the subtask corresponding to the element in the task vector corresponding to the target task; the sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task t , target action of target task, state information of target task s t+1 and the target task reward value vector.

[0058] In a feasible embodiment, when the state information s of the target task is t The task vector corresponding to the target task is input into the decision model M t In order to obtain the target action of the target task, the update unit is specifically used to:

[0059] According to the status information of the target task s tThe action value function vector of the target task is obtained from the task vector corresponding to the target task, and the action value function in the action value function vector of the target task corresponds one to one to the subtasks corresponding to the elements in the task vector corresponding to the target task; the value function of the target task is obtained according to the action value function vector of the target task and the task vector corresponding to the target task; the target action of the target task is obtained from the action space according to the value function of the target task, and the target action is the action in the action space that maximizes the value function of the target task.

[0060] In a feasible embodiment, in selecting the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to a preset probability, the updating unit is specifically used to:

[0061] When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action of the target task; the first parameter is a random number in the value range [0,1]; when the first parameter is not greater than the preset probability, the action randomly obtained from the action space is determined as the target action of the target task.

[0062] In a feasible embodiment, the adjustment unit is specifically used for:

[0063] According to the loss function and the state information s of each sample data in multiple sample data t , task vector, target action, state information s t+1 The loss value is calculated by the reward value vector; the decision model M is adjusted according to the loss value t The parameters in the decision model M t+1 .

[0064] In a fourth aspect, an embodiment of the present application provides a decision-making device based on multi-task learning, including:

[0065] The acquisition unit is used to acquire multiple candidate tasks, acquire the target task from the multiple candidate tasks, and acquire the status information s of the target task according to the target task t ;

[0066] A joint characterization unit is used to perform task joint characterization on a target task based on multiple candidate tasks to obtain a task vector corresponding to the target task, wherein the task vector corresponding to each candidate task is obtained based on characteristic subtasks and common subtasks of the multiple candidate tasks;

[0067] A determination unit is used to determine the task vector corresponding to the target task and the state information s of the target task. t , determine the target action from the action space.

[0068] In a feasible embodiment, the joint representation unit is specifically used for:

[0069] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0070] In a feasible embodiment, the determining unit is specifically configured to:

[0071] The task vector corresponding to the target task and the state information s of the target task are t The input is processed into a target decision model to obtain the target action of the target task. The target decision model is implemented based on a neural network.

[0072] In a feasible embodiment, the task vector corresponding to the target task and the state information s of the target task are t The input is processed into the target decision model to obtain the target action aspects of the target task. The determination unit is specifically used to:

[0073] According to the task vector corresponding to the target task and the state information s of the target task t Obtain the action value function vector of the target task, where the action value functions in the action value function vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; obtain the value function of the target task according to the task vector and the action value function vector corresponding to the target task; obtain the target action from the action space according to the value function of the target task, where the target action is the action in the action space that maximizes the value function of the target task.

[0074] In a fifth aspect, an embodiment of the present application provides another decision model training device based on multi-task learning, comprising:

[0075] a memory for storing instructions; and

[0076] at least one processor coupled to the memory;

[0077] Wherein, when the at least one processor executes the instruction, the instruction causes the processor to execute part or all of the method described in the first aspect.

[0078] In a sixth aspect, an embodiment of the present application provides another decision-making device based on multi-task learning, including:

[0079] a memory for storing instructions; and

[0080] at least one processor coupled to the memory;

[0081] Wherein, when the at least one processor executes the instruction, the instruction causes the processor to execute part or all of the method described in the second aspect.

[0082] In the seventh aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes part or all of the method described in the first aspect or the second aspect.

[0083] In an eighth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes part or all of the method described in the first aspect or the second aspect.

[0084] In a ninth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes part or all of the method described in the first aspect or the second aspect.

[0085] Among them, the computer program product can be run on an intelligent carrier installed with a computer system (such as a movable vehicle, a robotic arm, a recommendation search engine, etc.). The executable code for obtaining task / status information, system status processing, decision selection, and control is on the storage component of the computer system, and the CPU / GPU and the storage system need to be coordinated during operation. At the same time, the network communication component of the computer system is also required, and the decision model is stored on the storage component of the computer system.

[0086] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0088] Figure 1a A functional block diagram of a vehicle provided in an embodiment of the present application;

[0089] Figure 1b A schematic diagram of a computer system architecture provided in an embodiment of the present application;

[0090] Figure 1c A schematic diagram of a framework of an autonomous driving system provided in an embodiment of the present application;

[0091] Figure 1d A schematic diagram of another framework of an autonomous driving system provided in an embodiment of the present application;

[0092] Figure 2 A flowchart of a decision-making method based on multi-task learning provided in an embodiment of the present application;

[0093] Figure 3 This is a schematic diagram of the intersection scene;

[0094] Figure 4 A flowchart of a training method for a decision model based on multi-task learning provided in an embodiment of the present application;

[0095] Figure 5 A flowchart of another method for training a decision model based on multi-task learning provided in an embodiment of the present application;

[0096] Figure 6 A schematic diagram of the structure of a decision-making device provided in an embodiment of the present application;

[0097] Figure 7 A schematic diagram of the structure of a vehicle-mounted device provided in an embodiment of the present application;

[0098] Figure 8 A schematic diagram of the structure of a decision-making device based on multi-task learning provided in an embodiment of the present application;

[0099] Fig. 9 A schematic diagram of the structure of a decision model training device provided in an embodiment of the present application;

[0100] Fig.10 A schematic diagram of the structure of another decision-making device provided in an embodiment of the present application;

[0101] Fig.11 A schematic diagram of the structure of another decision model training device provided in an embodiment of the present application;

[0102] Fig.12 A partial schematic diagram of a computer program product provided in an embodiment of the present application. DETAILED DESCRIPTION

[0103] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0104] Figure 1ais a functional block diagram of a vehicle 100 provided by an embodiment of the present invention. In one embodiment, the vehicle 100 is configured to be in a fully or partially autonomous driving mode. For example, the vehicle 100 can control itself while being in the autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the possibility of the other vehicle performing the possible behavior, and control the vehicle 100 based on the determined information. When the vehicle 100 is in the autonomous driving mode, the vehicle 100 can be set to operate without human interaction.

[0105] The vehicle 100 may include various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, and a power source 110, a computer system 112, and a user interface 116. Optionally, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each subsystem and element of the vehicle 100 may be interconnected by wire or wirelessly.

[0106] The travel system 102 may include components that provide powered movement for the vehicle 100. In one embodiment, the travel system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy.

[0107] Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 may also provide energy to other systems of vehicle 100.

[0108] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

[0109] The sensor system 104 may include several sensors that sense information about the environment surrounding the vehicle 100. For example, the sensor system 104 may include a positioning system 122 (the positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors of the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the autonomous vehicle 100.

[0110] Positioning system 122 may be used to estimate the geographic location of vehicle 100. IMU 124 is used to sense position and orientation changes of vehicle 100 based on inertial acceleration. In one embodiment, IMU 124 may be a combination of an accelerometer and a gyroscope.

[0111] Radar 126 may utilize radio signals to sense objects within the surrounding environment of vehicle 100. In some embodiments, in addition to sensing objects, radar 126 may also be used to sense the speed and / or heading of an object.

[0112] The laser rangefinder 128 may utilize laser light to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0113] The camera 130 may be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 may be a still camera or a video camera.

[0114] Control system 106 controls the operation of vehicle 100 and its components. Control system 106 may include various elements, including a steering system 132 , a throttle 134 , a brake unit 136 , a sensor fusion system 138 , a computer vision system 140 , a path control system 142 , and an obstacle avoidance system 144 .

[0115] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system.

[0116] The throttle 134 is used to control the operating speed of the engine 118 and, in turn, the speed of the vehicle 100 .

[0117] The brake unit 136 is used to control the deceleration of the vehicle 100. The brake unit 136 can use friction to slow down the wheel 121. In other embodiments, the brake unit 136 can convert the kinetic energy of the wheel 121 into electric current. The brake unit 136 can also take other forms to slow down the rotation speed of the wheel 121 to control the speed of the vehicle 100.

[0118] The computer vision system 140 can be operated to process and analyze the images captured by the camera 130 in order to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signs, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, and the like.

[0119] The route control system 142 is used to determine the driving route of the vehicle 100. In some embodiments, the route control system 142 may combine data from the sensors 138, the GPS 122, and one or more predetermined maps to determine the driving route for the vehicle 100.

[0120] The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise negotiate potential obstacles in the environment of the vehicle 100 .

[0121] Of course, in one example, the control system 106 may include additional or alternative components other than those shown and described, or may also reduce some of the components shown above.

[0122] The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through peripherals 108. The peripherals 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152.

[0123] In some embodiments, the peripheral device 108 provides a means for the user of the vehicle 100 to interact with the user interface 116. For example, the onboard computer 148 can provide information to the user of the vehicle 100. The user interface 116 can also operate the onboard computer 148 to receive input from the user. The onboard computer 148 can be operated via a touch screen. In other cases, the peripheral device 108 can provide a means for the vehicle 100 to communicate with other devices located in the vehicle. For example, the microphone 150 can receive audio (e.g., voice commands or other audio input) from the user of the vehicle 100. Similarly, the speaker 152 can output audio to the user of the vehicle 100.

[0124] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system 146 can communicate with a wireless local area network (WLAN) using WiFi. In some embodiments, the wireless communication system 146 can communicate directly with the device using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0125] The power source 110 can provide power to various components of the vehicle 100. In one embodiment, the power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to various components of the vehicle 100. In some embodiments, the power source 110 and the energy source 119 can be implemented together, such as in some all-electric vehicles.

[0126] Some or all functions of the vehicle 100 are controlled by a computer system 112. The computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium such as a data storage device 114. The computer system 112 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

[0127] Processor 113 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a dedicated device such as an ASIC or other hardware-based processor. Figure 1aThe processor, memory, and other elements of the computer 110 in the same block are functionally illustrated, but it will be appreciated by those of ordinary skill in the art that the processor, computer, or memory may actually include multiple processors, computers, or memories that may or may not be stored in the same physical housing. For example, the memory may be a hard drive or other storage medium located in a housing different from the computer 110. Therefore, references to a processor or computer will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components such as the steering assembly and the deceleration assembly may each have their own processor that performs only calculations related to the functions specific to the component.

[0128] The processor 113 obtains the current state information of the vehicle through the sensor obtained in the sensor system 104, and the processor 113 obtains multiple candidate tasks and determines the target task from the multiple candidate tasks, and jointly represents the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task, and inputs the task vector corresponding to the target task and the current state information into the decision model for processing to obtain a target action of the target task, and the control system 106 executes the target action to control the vehicle 100 to travel.

[0129] In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic) that may be executed by the processor 113 to perform various functions of the vehicle 100, including those described above. The data storage device 114 may also include additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the travel system 102, the sensor system 104, the control system 106, and the peripherals 108.

[0130] In addition to the instructions 115, the data storage device 114 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. Such information may be used by the vehicle 100 and the computer system 112 during operation of the vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0131] The user interface 116 is used to provide information to or receive information from a user of the vehicle 100. Optionally, the user interface 116 may include one or more input / output devices within the set of peripherals 108, such as a wireless communication system 146, an onboard computer 148, a microphone 150, and a speaker 152.

[0132] The computer system 112 may control functions of the vehicle 100 based on input received from various subsystems (e.g., the travel system 102, the sensor system 104, and the control system 106) and from the user interface 116. For example, the computer system 112 may utilize input from the control system 106 in order to control the steering unit 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, the computer system 112 may be operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0133] Alternatively, one or more of the above-mentioned components may be installed or associated separately from the vehicle 100. For example, the data storage device 114 may be partially or completely separate from the vehicle 1100. The above-mentioned components may be communicatively coupled together in a wired and / or wireless manner.

[0134] Optionally, the above components are only examples. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1a It should not be understood as limiting the embodiments of the present invention.

[0135] An autonomous vehicle traveling on a road, such as vehicle 100 above, can identify objects in its surrounding environment to determine adjustments to the current speed. The objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's respective characteristics, such as its current speed, acceleration, spacing from the vehicle, etc., can be used to determine the speed to be adjusted by the autonomous vehicle.

[0136] Optionally, the autonomous vehicle 100 or a computing device associated with the autonomous vehicle 100 (eg, Figure 1a The computer system 112, computer vision system 140, data storage device 114) can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified object. In other words, the self-driving car can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc.

[0137] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., cars in adjacent lanes on the road).

[0138] The vehicle 100 may be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, and cart, etc., and the embodiments of the present invention do not make any particular limitation.

[0139] according to Figure 1b , the computer system 101 includes a processor 103, which is coupled to a system bus 105. The processor 103 may be one or more processors, each of which may include one or more processor cores. A display adapter (video adapter) 107, which may drive a display 109, which is coupled to the system bus 105. The system bus 105 is coupled to an input / output (I / O) bus 113 via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with a variety of I / O devices, such as an input device 117 (e.g., keyboard, mouse, touch screen, etc.), a multimedia disk (media tray) 121, (e.g., CD-ROM, multimedia interface, etc.). A transceiver 123 (which may send and / or receive radio communication signals), a camera 155 (which may capture scene and dynamic digital video images) and an external USB interface 125. Optionally, the interface connected to the I / O interface 115 may be a USB interface.

[0140] The processor 103 may be any conventional processor, including a reduced instruction set computing ("RISC") processor, a complex instruction set computing ("CISC") processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an application specific integrated circuit ("ASIC"). Alternatively, the processor 103 may be a neural network processor or a combination of a neural network processor and the conventional processors described above.

[0141] Alternatively, in various embodiments described herein, the computer system 101 may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle 0. In other aspects, some of the processes described herein are performed on a processor disposed within the autonomous vehicle, and others are performed by a remote processor, including taking actions required to perform a single maneuver.

[0142] Computer 101 can communicate with software deployment server 149 via network interface 129. Network interface 129 is a hardware network interface, such as a network card. Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network, a cellular network, etc.

[0143] The hard disk drive interface is coupled to the system bus 105. The hard disk drive interface is connected to the hard disk drive. The system memory 135 is coupled to the system bus 105. The data running in the system memory 135 may include the operating system 137 and application programs 143 of the computer 101.

[0144] The operating system consists of a shell 139 and a kernel 141. The shell 139 is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: it waits for user input, interprets the user input to the operating system, and processes various operating system output results.

[0145] The kernel 141 consists of those parts of the operating system that manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel usually runs processes and provides communication between processes, provides CPU time slice management, interrupts, memory management, IO management, etc.

[0146] Application 143 includes programs related to controlling the automatic driving of the car, such as a program for managing the interaction between the automatic driving car and obstacles on the road, a program for controlling the route or speed of the automatic driving car, and a program for controlling the interaction between the automatic driving car and other automatic driving cars on the road. Application 143 also exists on the system of software deployment server 149. In one embodiment, when it is necessary to execute the automatic driving related program 147, computer system 101 can download application 143 from software deployment server 149.

[0147] When the processor 103 executes the application 143, the processor performs the following steps: obtaining a target task from multiple candidate tasks according to the navigation information, such as going straight at an intersection; and jointly representing the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task; and combining the task vector corresponding to the target task and the state information s of the target task obtained by the sensor 153. t The input is processed into the decision model to obtain the target action of the target task, such as acceleration, parking, etc., and then the target action is executed to control the vehicle driving.

[0148] Sensor 153 is associated with computer system 101. Sensor 153 is used to detect the environment around computer 101. For example, sensor 153 can detect animals, cars, obstacles and crosswalks, etc., and further, the sensor can also detect the environment around the above-mentioned animals, cars, obstacles and crosswalks, such as: the environment around the animals, for example, other animals around the animals, weather conditions, brightness of the surrounding environment, etc. Optionally, if computer 101 is located in an autonomous vehicle, the sensor can be a camera, an infrared sensor, a chemical detector, a microphone, etc.

[0149] Obtain a target task from multiple candidate tasks; and obtain the status information s of the target task according to the target task t The task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks of multiple candidate tasks, according to the state information s of the target task. t , the task vector and decision model corresponding to the target task update the sample database to obtain an updated sample database; randomly obtain multiple sample data from the updated sample database; the multiple sample data are sample data of some or all of the multiple candidate tasks; use the reinforcement learning method to adjust the parameters in the decision model according to the multiple sample data to obtain an adjusted decision model; determine whether the adjusted decision model converges; when the adjusted decision model converges, determine the adjusted decision model as the target decision model.

[0150] Among them, the decision model can be implemented based on a fully connected neural network, a convolutional neural network, a recurrent neural network or other neural networks.

[0151] The computer system 112 may also receive information from other computer systems or transfer information to other computer systems. Alternatively, sensor data collected from the sensor system 104 of the vehicle 100 may be transferred to another computer for processing. Figure 1c As shown, data from computer system 112 can be transmitted to cloud-side computer 720 via the network for further processing. The network and intermediate nodes can include various configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company's proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations of the foregoing. Such communication can be by any device capable of transmitting data to and from other computers, such as a modem and a wireless interface.

[0152] In one example, computer 720 may include a server having multiple computers, such as a load-balancing server farm, which exchanges information with different nodes of a network for the purpose of receiving, processing, and transmitting data from computer system 112. The server may be configured similarly to computer system 110, having a processor 730, memory 740, instructions 750, and data 760.

[0153] Optionally, the above data 760 includes the coordinates and speed of the vehicle in the world coordinate system, the coordinates, speed and heading angle of surrounding vehicles in the world coordinate system, etc.

[0154] When the processor 730 executes the instruction 750, it specifically implements the following steps:

[0155] According to the coordinates, speed of the self-vehicle in the world coordinate system and the coordinates, speed and heading angle of the surrounding social vehicles in the world coordinate system, the coordinates (x it ,y it ), speed v it and heading angle θ it .

[0156] After obtaining the speed of the vehicle and the coordinates of the surrounding vehicles in the vehicle coordinate system (x it ,y it ), speed v it and heading angle θ it , the speed of the vehicle and the coordinates of the surrounding vehicles in the vehicle coordinate system (x it ,y it ), speed v it and heading angle θ it Sent to the computer system 112 of the vehicle's computer 101 .

[0157] It should be noted here that the above instructions can be regarded as conversion instructions.

[0158] Figure 1d An example of an autonomous driving vehicle and a cloud service center according to an example embodiment is shown. The cloud service center 520 can receive information (such as data collected by vehicle sensors or other information) from autonomous driving vehicles 510, 512, and 514 within its operating environment 500 via a network 502 such as a wireless communication network.

[0159] The cloud service center 520 obtains information such as speed, coordinates, heading angle, etc. of the autonomous driving vehicles 510 , 512 , and 514 in the world coordinate system through the network 502 .

[0160] The cloud service center runs the stored program related to controlling the automatic driving of the car according to the received data to control the automatic driving vehicles 510, 512 and 514. The program related to controlling the automatic driving of the car can be a program for managing the interaction between the automatic driving car and obstacles on the road, a program for controlling the route or speed of the automatic driving car, and a program for controlling the interaction between the automatic driving car and other automatic driving cars on the road.

[0161] The cloud service center 520 obtains the state information of any vehicle A among the autonomous driving vehicles 510, 512 and 514, including the speed of the vehicle A, the coordinates of the surrounding vehicles in the coordinate system of the vehicle A, the speed and the heading angle; obtains the target task from multiple candidate tasks according to the navigation information of the vehicle A; jointly represents the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task; and inputs the state information and the task vector corresponding to the target task into the decision model for processing to obtain the target action of the target task.

[0162] After acquiring the target action, the cloud service center 520 sends the target action to vehicle A so that vehicle A travels according to the target action.

[0163] The network 502 provides portions of the map outwardly to autonomous vehicles 510, 512, or 514. In other examples, operations may be divided between different locations or centers. For example, multiple cloud service centers may receive, verify, combine, and / or send information reports. In some examples, information reports and / or sensor data may also be sent between autonomous vehicles. Other configurations are also possible.

[0164] Multiple cloud service centers can share information such as the speed, coordinates, and heading angle of vehicles in their service areas in the world coordinate system; and among multiple cloud service centers, when cloud service center 1 is unable to provide driving services for vehicle B within its service range, cloud service center 1 can send relevant information of vehicle B (such as vehicle B's status information, action space, and task vector corresponding to the task to be performed) to cloud service center 2; cloud service center 2 determines the target action of vehicle B based on vehicle B's status information, action space, and task vector corresponding to the task to be performed, and then sends the target action to cloud service center 1, which then sends the target action to vehicle B.

[0165] In some examples, the center sends the autonomous driving vehicle a solution for possible driving situations in the environment (e.g., informing the obstacle ahead and informing how to bypass it). For example, the cloud service center can assist the vehicle in determining how to proceed when facing a specific obstacle in the environment. The cloud service center sends a response to the autonomous driving vehicle indicating how the vehicle should proceed in a given scenario. For example, based on the collected sensor data, the cloud service center can confirm the presence of a temporary parking sign in front of the road, and also determine that the lane is closed due to construction based on the "lane closed" sign and the sensor data of the construction vehicle on the lane. Accordingly, the cloud service center sends a recommended operation mode for the autonomous driving vehicle to pass the obstacle (e.g., instructing the vehicle to change lanes to another road). When the cloud service center observes the video stream in its operating environment and has confirmed that the autonomous driving vehicle can safely and successfully pass through the obstacle, the operation steps used by the autonomous driving vehicle can be added to the driving information map. Accordingly, this information can be sent to other vehicles in the area that may encounter the same obstacle, so as to assist other vehicles not only to identify the closed lane but also to know how to pass.

[0166] See also Figure 2 , Figure 2 A flowchart of a decision-making method based on multi-task learning provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0167] S201, obtaining multiple candidate tasks, determining a target task from the multiple candidate tasks, and obtaining status information s of the target task according to the target task t .

[0168] Optionally, the above-mentioned multiple candidate tasks may be tasks in the same scenario. For example, in an intersection scenario, the above-mentioned multiple candidate tasks may include going straight at the intersection, turning left at the intersection, and turning right at the intersection; for example, in the scenario of a tactical competitive game, the above-mentioned multiple candidate tasks may include killing the most enemies, surviving the longest time, and dying the least.

[0169] Optionally, the plurality of candidate tasks may further include tasks in different scenarios, such as the tasks in the intersection scenario and the tasks in a tactical competitive game scenario.

[0170] For example, in the scene of a crossroads, since the ego vehicle and social vehicles have multiple interactive games, it is necessary to obtain the information of the ego vehicle and the surrounding social vehicles, so the state information of the target task s t Includes information about the vehicle and surrounding vehicles. Optionally, the information about the vehicle includes the speed v of the vehicle et The information of surrounding social vehicles includes the coordinates of surrounding social vehicles in the vehicle coordinate system (x it ,yit ), speed v it and heading angle θ it Assume that the surrounding social vehicles include the 5 vehicles closest to the vehicle, and the state information s t It can be expressed as s t =[v et ,x 1t ,y 1t ,v 1t ,θ 1t ,…,x 5t ,y 5t ,v 5t ,θ 5t ].

[0171] For example, in the scenario of competitive tactical games, the status information of the target task s t Including the coordinates of your character on the map (x pt ,y pt ) and forward speed v pt , and the coordinates of teammates in the character's coordinate system (x it ,y it )), forward speed v i and the advancing angle θ it Optionally, the status information of the target task s t It can also include your own and your teammates' health, firepower, number of kills, survival time, enemy location information, health and firepower, etc.

[0172] In a specific example, in a crossroads scenario, the target task can be determined from multiple candidate tasks based on the user's navigation information. For example, in a crossroads scenario, the multiple candidate tasks include turning left at the intersection, turning right at the intersection, and going straight at the intersection; if it is determined according to the navigation information that a right turn is required at the intersection, the target task determined from the multiple candidate tasks is turning right at the intersection.

[0173] In another specific example, in a tactical competitive game scenario, the target task can be determined from multiple candidate tasks according to the game settings. For example, in a tactical competitive game, the multiple candidate tasks include killing the most enemies and surviving the longest time. The game setting is that the winning condition is the longest survival time, so the target task determined from the multiple candidate tasks is the longest survival time.

[0174] S202 : performing task joint representation on the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task.

[0175] Among them, the task vector corresponding to each candidate task is obtained based on the characteristic subtasks and common subtasks of the above multiple candidate tasks.

[0176] Specifically, each of the multiple candidate tasks is decomposed according to prior knowledge to obtain subtasks corresponding to each candidate task; characteristic subtasks and common subtasks of the multiple candidate tasks are obtained according to the subtasks corresponding to each candidate task in the multiple candidate tasks; and a task vector corresponding to the target task is obtained according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0177] For example, in the scenario of an intersection, the above-mentioned multiple candidate tasks may include going straight at the intersection, turning left at the intersection, and turning right at the intersection; according to prior knowledge, going straight at the intersection is decomposed into two subtasks: collision or arrival at the intersection and improving traffic efficiency; turning left at the intersection is decomposed into two subtasks: collision or arrival at the intersection and improving traffic efficiency; turning right at the intersection is decomposed into two subtasks: collision or arrival at the intersection and improving traffic efficiency. Among them, collision or arrival at the intersection, collision or arrival at the intersection, collision or arrival at the intersection, and collision or arrival at the intersection are characteristic subtasks, and improving traffic efficiency is a common subtask.

[0178] After obtaining the characteristic subtasks and characteristic subtasks, each of the above multiple candidate tasks is jointly characterized according to the above characteristic subtasks and common subtasks to obtain the task vector corresponding to each candidate task. For example, for the scene of the intersection, the task vector corresponding to each candidate task is obtained based on the four subtasks of left turn collision or arrival at the intersection, straight collision or arrival at the intersection, right turn collision or arrival at the intersection and improving traffic efficiency.

[0179] Optionally, the task vector corresponding to each of the plurality of candidate tasks is composed of a plurality of elements, and the plurality of elements correspond one-to-one to the characteristic subtasks and common subtasks of the plurality of candidate tasks. The different values ​​of each element in the task vector corresponding to the candidate task indicate whether the candidate task contains the subtask corresponding to the element.

[0180] For example, for the three candidate tasks in the intersection scene (including left turn at intersection, straight ahead at intersection and right turn at intersection), the task vector corresponding to each candidate task is composed of four elements, wherein the four elements correspond to the above four subtasks: collision or arrival at left turn at intersection, collision or arrival at straight ahead at intersection, collision or arrival at right turn at intersection and improving traffic efficiency. The different values ​​of the elements in the task vector corresponding to each candidate task indicate whether the candidate task contains the subtask corresponding to the element.

[0181] Optionally, the task vector corresponding to the left turn at the intersection can be expressed as [1,0,0,1], the task vector corresponding to the straight-through at the intersection can be expressed as [0,1,0,1], and the task vector corresponding to the right turn at the intersection can be expressed as [0,0,1,1]; wherein, in the task vector [1,0,0,1] corresponding to the left turn at the intersection, the first "1" from left to right indicates that the candidate task "turn left at the intersection" includes the subtask "turn left at the intersection" The subtask is collided or arrived, the first "0" indicates that the candidate task "turn left at the intersection" does not include the subtask "go straight at the intersection" Collision or arrival", the second "0" indicates that the candidate task "turn left at the intersection" does not include the subtask "turn right at the intersection Collision or arrival", and the second "1" indicates that the candidate task "turn left at the intersection" includes the subtask "improve traffic efficiency"; the task vector [0,1,0,1] corresponding to the straight-through at the intersection The first "0" from left to right indicates that the candidate task "turn straight at the intersection" The first “1” indicates that the candidate task “going straight at the intersection” includes the subtask “collision or arrival at the intersection”, and the second “0” indicates that the candidate task “going straight at the intersection” does not include the subtask “collision or arrival at the intersection”, and the second “1” indicates that the candidate task “going straight at the intersection” includes the subtask “improving traffic efficiency”; in the task vector [0,0,1,1] corresponding to turning right at the intersection, the first “0” from left to right indicates that the candidate task “turning right at the intersection” does not include the subtask “collision or arrival at the intersection”, and the second “0” indicates that the candidate task “turning right at the intersection” does not include the subtask “collision or arrival at the intersection”, and the first “1” indicates that the candidate task “turning right at the intersection” includes the subtask “collision or arrival at the intersection”, and the second “1” indicates that the candidate task “turning right at the intersection” includes the subtask “improving traffic efficiency”.

[0182] Optionally, the task vector corresponding to turning left at the intersection can be expressed as [80,0,0,20], the task vector corresponding to going straight at the intersection can be expressed as [0,80,0,20], and the task vector corresponding to turning right at the intersection can be expressed as [0,0,80,20]; among which, in the task vector [80,0,0,20] corresponding to turning left at the intersection, "80" indicates that the candidate task "turn left at the intersection" includes the subtask "turn left at the intersection" and the first "0" from left to right indicates that the candidate task "turn left at the intersection" does not include the subtask "go straight at the intersection" and the second "0" indicates that the candidate task "turn left at the intersection" does not include the subtask "turn right at the intersection" and "20" indicates that the candidate task "turn left at the intersection" includes the subtask "improve traffic efficiency". In the task vector [0,80,0,20] corresponding to going straight at an intersection, the first "0" from left to right indicates that the candidate task "going straight at an intersection" does not include the subtask "collision or arrival at a left turn at an intersection", "80" indicates that the candidate task "going straight at an intersection" includes the subtask "collision or arrival at a straight-through intersection", and the second "0" indicates that the candidate task "going straight at an intersection" does not include the subtask "collision or arrival at a right turn at an intersection", and "20" indicates that the candidate task "going straight at an intersection" includes the subtask "improving traffic efficiency". This is equivalent to the task vector corresponding to the candidate task, which includes the subtasks corresponding to the larger element value but does not include the subtasks corresponding to the smaller element value. This is equivalent to the task vector corresponding to the candidate task, which includes the subtasks corresponding to the non-zero element value but does not include the subtasks corresponding to the zero element value. In other words, the elements in the task vector corresponding to the candidate task are the weights of the subtasks corresponding to the element; the more important the candidate task, the greater the corresponding weight of the subtask, and the less important the candidate task, the smaller the corresponding weight of the subtask.

[0183] It should be noted here that the order of subtasks in the task vector corresponding to the above-mentioned multiple candidate tasks includes but is not limited to the order described above ([collision or arrival at left turn at intersection, collision or arrival at straight-ahead intersection, collision or arrival at right turn at intersection, improving traffic efficiency]), and can also be other orderings.

[0184] S203, based on the task vector corresponding to the target task and the state information s of the target task t Determine the target action from the action space.

[0185] Specifically, the task vector corresponding to the target task and the state information s of the target task are t Input to the decision model M t The decision model is implemented by a neural network to obtain the target action of the target task.

[0186] Optionally, the neural network may be a fully connected neural network, a convolutional neural network, a recurrent neural network, or other types of neural networks.

[0187] In one embodiment, the task vector corresponding to the target task and the state information s of the target task are t Input to the decision model M t In order to obtain the target action of the target task, the target action includes:

[0188] According to the task vector corresponding to the target task and the state information s of the target task t Get the action value function of the target task, where the action value function in the action value function vector corresponds one-to-one to the subtask corresponding to the element in the task vector corresponding to the target task. The action value function vector can be expressed as R(s t ,a k ,g),a k is the action in the action space, and g is the task vector corresponding to the target task. In order to avoid the action value function vector R(s t ,a k , g) is related to the action value function of the subtask that is not related to the target task. According to the task vector g corresponding to the target task, the action value function R(s t ,a k ,g) delete the action value functions of the subtasks that are not related to the target task to obtain the value function of the target task, and then determine the target action from the action space based on the value function of the target task. The target action is the action in the action space that maximizes the value function of the target task.

[0189] Optionally, the above value function is a Q value function; wherein the Q value function of the above target task can be expressed as: Q(s t ,a k ,g)=g T R(s t ,a k ,g); the target action can be expressed as:

[0190] It should be noted that the above action space is composed of actions that can be executed when performing the above target tasks. For example, given navigation information, that is, given unmanned vehicle waypoints, the vehicle speed planning control is performed. In order to enhance the interactivity of the vehicle, the unmanned vehicle needs to have the ability to stop and wait, squeeze and accelerate. Therefore, the designed action space needs to cover a larger speed range. This embodiment uses a discrete action space [0, 3m / s, 6m / s, 9m / s].

[0191] In the target action a t After being executed, get the status information s of the target task t+1 , and according to the status information s of the target task t+1Determine whether the target task is completed. For example, if the target task is to go straight at the intersection, the state information s t+1 Determine whether the vehicle has passed the intersection. t+1 Determine that the target task has not been completed, and re-execute steps S202-S203; if according to the status information s t+1 When it is determined that the target task is completed, the next task is executed according to the above steps S201-S203.

[0192] It should be noted here that the status information s t is the state information at time t, state information s t+1 is the state information at time t+1. The two are the same type of information at different times. The target action a t is the action performed at time t. Target action a t The execution subject of the decision model may be the same as or different from the execution subject of the decision model. For example, after the decision device obtains the target action, it sends the target action to the car, and the control device of the car executes the target action to control the car.

[0193] It can be seen that in the scheme of the present application, by jointly representing the target task according to the subtasks of multiple candidate tasks, it is possible to use the same model to make decisions on multiple tasks and avoid mutual influence between multiple tasks; by obtaining the value function of the target task according to the task vector and the action value function vector corresponding to the target task, and then determining the target action based on the value function, it is avoided that the action value function of the subtask that is not related to the target task in the action value function vector affects the selection of the target action when making decisions on the target task, thereby improving the decision-making effect of the target task.

[0194] In a specific example, Figure 3 As shown, for the intersection interaction scenario, the driverless vehicle needs to be able to perform three tasks: turning left at the intersection, going straight at the intersection, and turning right at the intersection. In other words, the candidate tasks of the driverless vehicle include turning left at the intersection, going straight at the intersection, and turning right at the intersection. According to the navigation information, it is determined that it is necessary to go straight at the intersection, so the target task determined from the above candidate tasks is going straight at the intersection.

[0195] The above three candidate tasks are decomposed according to prior knowledge to obtain subtasks of each of the three candidate tasks; the subtask of turning left at the intersection includes collision or arrival at the left turn at the intersection and improving traffic efficiency, the subtask of going straight at the intersection includes collision or arrival at the straight-through intersection and improving traffic efficiency, and the subtask of turning right at the intersection includes collision or arrival at the right turn at the intersection and improving traffic efficiency; characteristic subtasks and common subtasks are extracted from the subtasks of the three candidate tasks; wherein the characteristic subtasks include collision or arrival at the left turn at the intersection, collision or arrival at the straight-through intersection and collision or arrival at the right turn at the intersection, and the common subtasks include improving traffic efficiency; and the task vector corresponding to each of the three candidate tasks is obtained according to the characteristic subtasks and the common subtasks; wherein the task vector g' corresponding to going straight at the intersection is [0,1,0,1];

[0196] In the scenario of a crossroads, since the ego vehicle needs to conduct multiple interactive games with the surrounding social vehicles, it is necessary to obtain complete information about the social vehicles around the ego vehicle. t , status information t Including the vehicle speed v e , the position coordinates of the social vehicle in the self-vehicle coordinate system (x i ,y i ), speed v i and heading angle θ i Assume that the surrounding social vehicles are the five vehicles closest to the vehicle, and the state information s t It can be expressed as: t =[v e ,x1,y1,v1,θ1,…,x5,y5,v5,θ5];

[0197] The task vector g and state information s corresponding to the straight-ahead intersection t Input into the decision model for processing to obtain the target action corresponding to the intersection straight, specifically, the task vector g and state information s corresponding to the intersection straight t Get the action value function vector R(s t ,a k ,g), action value function vector R(s t ,a k ,g) has a one-to-one correspondence with the subtasks corresponding to the elements in the task vector corresponding to going straight at the intersection; in order to exclude the action value function vector R(s t ,a k ,g), the impact of the action value function corresponding to the right turn collision or arrival at the intersection and the action value function corresponding to the left turn collision or arrival at the intersection, according to the task vector g corresponding to the straight-ahead intersection and the action value function vector R(s t ,a k,g) to obtain the Q value function corresponding to the straight-ahead intersection; the Q value function can be expressed as: Q(s t ,a k ,g)=g T R(s t ,a k ,g);

[0198] Finally, the Q value function Q(s t ,a k ,g) The action with the maximum value is determined as the target action a t , for example, accelerating straight ahead. t After being executed, get the status information s of the target task t+1 , through the status information s t+1 Determine whether the target task is completed. For example, based on the status information s t+1 The coordinates of the ego vehicle in the figure determine that the ego vehicle has passed the intersection, thereby determining that the straight-through task at the intersection is completed. If it is determined based on the status information that the ego vehicle has not passed the intersection, the target action corresponding to the straight-through task at the intersection is re-acquired according to the above method until the ego vehicle passes the intersection.

[0199] It can be seen that the use of this embodiment in the intersection scenario can achieve simultaneous learning of traffic strategies in three directions in one decision model, can successfully find the merging opportunity, improve traffic efficiency and avoid collisions through multiple interactions.

[0200] See also Figure 4 , Figure 4 A flowchart of a training method for a decision model based on multi-task learning provided in an embodiment of the present application. Figure 4 As shown, the method includes:

[0201] S401, randomly obtain multiple sample data from the first sample database, and use the reinforcement learning method to adjust the decision model M according to the multiple sample data t The parameters in the decision model M t+1 .

[0202] Among them, the first sample database includes sample data of multiple candidate tasks, and the sample data of the target task includes a task vector corresponding to the target task. The task vector corresponding to the target task is obtained based on common subtasks and characteristic subtasks in multiple candidate tasks. The target task is any one of the multiple candidate tasks.

[0203] Specifically, according to the loss function and the state information s of each sample data in the multiple sample data t , task vector, target action, state information s t+1 The loss value is calculated by the reward value vector r; the decision model M is adjusted according to the loss valuet The parameters in the decision model M t+1 .

[0204] Among them, the loss value can be expressed as:

[0205]

[0206] Among them, r in the above formula is the reward value vector in the above sample data, g is the task vector in the above sample data, and action a t is the action in the above sample data, and the discount factor γ is a constant.

[0207] In a feasible embodiment, before randomly acquiring a plurality of sample data from the first sample database, the method of this embodiment further includes:

[0208] Obtain a target task from multiple candidate tasks; and obtain the status information s of the target task according to the target task t And obtain the task vector corresponding to the target task;

[0209] According to the status information of the target task s t , the task vector corresponding to the target task and the decision model M t Generate sample data of the target task, and add the sample data of the target task to the preliminary sample database to obtain a first sample database.

[0210] Optionally, the above-mentioned multiple candidate tasks may be tasks in the same scenario. For example, in an intersection scenario, the above-mentioned multiple candidate tasks may include going straight at the intersection, turning left at the intersection, and turning right at the intersection; for example, in the scenario of a tactical competitive game, the above-mentioned multiple candidate tasks may include killing the most enemies, surviving the longest time, and dying the least.

[0211] Optionally, the plurality of candidate tasks may further include tasks in different scenarios, such as the tasks in the intersection scenario and the tasks in a tactical competitive game scenario.

[0212] For example, in the scene of a crossroads, since the ego vehicle and social vehicles have multiple interactive games, it is necessary to obtain the information of the ego vehicle and the surrounding social vehicles, so the state information of the target task s t Includes information about the vehicle and surrounding vehicles. Optionally, the information about the vehicle includes the speed v of the vehicle et The information of surrounding social vehicles includes the coordinates of surrounding social vehicles in the vehicle coordinate system (x it ,y it ), speed v it and heading angle θ it Assume that the surrounding social vehicles include the 5 vehicles closest to the vehicle, and the state information s tIt can be expressed as s t =[v et ,x 1t ,y 1t ,v 1t ,θ 1t ,…,x 5t ,y 5t ,v 5t ,θ 5t ].

[0213] For example, in the scenario of competitive tactical games, the status information of the target task s t Including the coordinates of your character on the map (x pt ,y pt ) and forward speed v pt , and the coordinates of teammates in the character's coordinate system (x it ,y it )), forward speed v i and the advancing angle θ it Optionally, the status information of the target task s t It can also include your own and your teammates' health, firepower, number of kills, survival time, enemy location information, health and firepower, etc.

[0214] Among them, the task vector corresponding to the target task is obtained based on the characteristic subtasks and common subtasks of multiple candidate tasks.

[0215] Decompose each of the multiple candidate tasks according to prior knowledge to obtain subtasks corresponding to each candidate task; obtain characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each candidate task in the multiple candidate tasks; obtain a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0216] It should be noted that the specific description of obtaining the task vector corresponding to the target task according to the target task can be found in the relevant description of the above step S202, which will not be described again here.

[0217] In a feasible embodiment, according to the state information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample data for the target task, including:

[0218] The status information of the target task s t The task vector corresponding to the target task is input into the decision model M t, to obtain the candidate actions of the target task; select the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to the preset probability, and the probability of the candidate action of the target task being selected is the preset probability; after the target action is executed, obtain the state information s of the target task t+1 , and according to the status information s of the target task t+1 Get the reward value vector of the target task; the reward value in the reward value vector corresponds to the subtask corresponding to the element in the task vector corresponding to the target task. The sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task. t , target action of target task, state information of target task s t+1 and the reward value vector of the target task.

[0219] In one example, the status information of the target task s t The task vector g corresponding to the target task is input into the decision model M t , to obtain the target action of the target task, including:

[0220] Decision Model M t According to the status information s of the target task t The task vector g corresponding to the target task obtains the action value function vector R(s t ,a k ,g), the action value function vector R(s) of the target task t ,a k ,g) has a one-to-one correspondence between the action value function and the subtask corresponding to the element in the task vector corresponding to the target task; the decision model M t According to the action value function vector R(s t ,a k ,g) and the task vector corresponding to the target task obtain the target action a of the target task from the action space t Since the action value function vector R(s t ,a k , g) contains the action value functions of subtasks that are not related to the target task. For example, the subtasks that are not related to the left turn at the intersection include right turn arrival or collision at the intersection and straight-through collision or arrival at the intersection. In order to avoid the influence of the action value functions of subtasks that are not related to the target task, the action value function R(s) of the target task is converted according to the task vector g corresponding to the target task. t ,a k ,g) delete the action value functions of the subtasks that are not related to the target task to obtain the value function of the target task; determine the candidate action from the action space based on the value function, and the candidate action is the action in the action space that maximizes the value function of the target task.

[0221] In a feasible embodiment, a target action a of the target task is selected from the candidate actions of the target task and the actions randomly obtained from the action space according to a preset probability. t ,include:

[0222] When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action a of the target task. t ; The first parameter is a random number in the range of [0,1]; when the first parameter is not greater than the preset probability, the action randomly obtained from the action space is determined as the target action a of the target task t .

[0223] It should be pointed out here that the initial value of the preset probability is 1 or a larger value close to 1; the preset probability gradually decreases with the increase in the number of training times; the purpose of setting the initial value of the preset probability to 1 or a larger value close to 1 is to explore new actions as much as possible in the early stage of training the decision model to avoid avoiding the optimal action during training.

[0224] Optionally, the maximum value of the preset probability is 1 and the minimum value is 0.1.

[0225] Optionally, the upper-value function is a Q-value function.

[0226] In the target action a t After being executed, get the status information s of the target task t+1 , and according to the status information s of the target task t+1 Get the reward value vector of the target task; the reward value in the reward value vector corresponds to the subtask corresponding to the element in the task vector corresponding to the target task. For example, for the task of turning left at the intersection, its reward value vector r(s t+1 ,a,g) can be expressed as [r l ,r s ,r r ,r c ], where r l is the reward value for the subtask left turn at the intersection, collision or arrival, r s is the reward value for the subtask intersection straight collision or arrival, r r is the reward value for the subtask right turn at the intersection, collision or arrival, r c The reward value for improving the efficiency of the subtask.

[0227] The reward value function can be expressed as:

[0228] According to the task vector g corresponding to the target task and the state information s of the target task t , target action a t , status informationt+1 And the reward value vector r obtains the sample data of the target task t ,a t ,r,s t+1 ,g>, and the sample data of the target task t ,a t ,r,s t+1 ,g>Save to the preliminary sample database to obtain the first sample database.

[0229] It should be noted that the target action execution subject may be the same as or different from the decision model execution subject. For example, after obtaining the target action, the decision model execution subject sends the target action to the target action execution subject to execute the target action.

[0230] S402, judgment decision model M t+1 Whether it converges; when the decision model M t+1 When convergence, determine the decision model M t+1 The target decision model.

[0231] In one example, the decision model M t+1 Whether it is convergent, including:

[0232] According to the status information s of the target task t+1 Determine whether the target task has been completed; when it is determined that the target task has been completed, determine the decision model M t+1 Whether to converge.

[0233] For example, for the intersection scene, the state information s t+1 Including the speed v of the vehicle e(t+1) , the coordinates of the surrounding vehicles in the vehicle coordinate system (x i(t+1) ,y i(t+1) ), speed V i(t+1) and heading angle θ i(t+1) , can be obtained by the vehicle's speed v e(t+1) , the coordinates of the surrounding vehicles in the vehicle coordinate system (x i(t+1) ,y i(t+1) ), speed V i(t+1) and heading angle θ i(t+1) Determine whether the vehicle has passed the intersection; if it has passed the intersection, it is determined that the target task is completed; if it has not passed the intersection, it is determined that the target task is not completed.

[0234] Further, if it is determined that the target task has not been completed, then t=t+1 is set and step S401 is repeated to execute "according to the status information s of the target task" t , the task vector corresponding to the target task and the decision model M t ​​Generate sample data of the target task, add the sample data of the target task to the preliminary sample database to obtain a first sample database” and S402 until the target task is completed.

[0235] After the target task is determined, the decision model M t+1 Whether to converge.

[0236] Judgment decision model M t+1 Whether it converges, if it converges, the decision model M t+1 Determine the target decision model. If it does not converge, repeat steps S401-S402 until the decision model M t+1 convergence.

[0237] Optionally, the decision model M is determined by judging whether the accumulated reward value converges. t+1 Specifically, the task vector g and state information s corresponding to the target task are t+1 Input to the decision model M t+1 In order to obtain a new reference action, specifically, according to the task vector g and state information s corresponding to the target task t+1 Get the action value function vector R(s t+1 ,a k ,g), and according to the action value function vector R(s t+1 ,a k ,g) and the task vector g corresponding to the target task determine the Q value function Q(s t+1 ,a k ; g), make the Q value function Q(s) in the action space t+1 ,a k ; g) The action with the largest value is determined as the new reference action; the target action a of the target task is selected from the new candidate actions and the actions randomly obtained from the action space according to the new preset probability t+1 , the probability of the new candidate action being selected is the new preset probability; when executing the target action a t+1 After that, get the status information s t+2 , according to the reward value function and state information s t+2 Determine the reward value vector r(s) of the target action task t+2 ,a t+1 ,g);

[0238] According to the current status information t+2 Determine whether the target task is completed; if it is determined that the target task is not completed, set t = t + 1, and repeat the above steps until the target task is completed. So far, multiple reward value vectors are obtained, and the reward values ​​corresponding to the subtasks related to the target task in each reward value vector in the multiple reward value vectors are accumulated to obtain the reward accumulation value.

[0239] If the reward accumulation value converges, the decision model M is determined t+1 Convergence; if the reward accumulation value does not converge, determine the decision model M t+1 Not convergent.

[0240] It should be pointed out here that in the above process, when obtaining the new target action a t+1 , status information t+2 and the reward value vector r(s t+2 ,a t+1 ,g), according to the task vector g and state information s corresponding to the target task t+1 , target action a t+1 , status information t+2 and the reward value vector r(s t+2 ,a t+1 ,g) Generate sample data and save the sample data into the first sample database to obtain a new sample database.

[0241] It should be noted that after the target task is completed, the target task is deleted from multiple candidate tasks to obtain the deleted candidate task; when step S401 needs to be executed again later, obtaining the target task from multiple candidate tasks specifically refers to obtaining a new target task from the deleted candidate task.

[0242] It can be seen that in the scheme of the present application, by jointly representing the target task, a task vector composed of characteristic subtasks and common subtasks is obtained, and the strategy of learning multiple candidate tasks through one model is realized, wherein the common subtasks of multiple candidate tasks are used to promote the learning of multiple candidate task strategies and improve the model convergence ability; the targeted learning of multiple candidate tasks by characteristic subtasks is carried out, which avoids the mutual influence between multiple tasks and the compromise of the model between multiple tasks, and realizes that the same model can achieve excellent results when making decisions on multiple tasks; by constructing a reward value vector, the execution status of each subtask in the target task is fed back to the decision model for learning; by obtaining the value function of the target task according to the task vector and the action value function vector corresponding to the target task, and then determining the target action based on the value function, it is avoided that the action value function of the subtask that is not related to the target task in the action value function vector affects the selection of the target action when making decisions on the target task.

[0243] See also Figure 5 , Figure 5 A flowchart of another decision model training method based on multi-task learning provided in an embodiment of the present application. Figure 5 As shown, the method includes:

[0244] S501: Determine a target task from multiple candidate tasks, and obtain a task vector g corresponding to the target task and state information s of the target task according to the target task. t .

[0245] The above multiple tasks are tasks in the same scenario. For example, in the scenario of an intersection, the above multiple candidate tasks include turning left at the intersection, going straight at the intersection, and turning right at the intersection. For another example, in the scenario of a tactical competitive game, the above multiple candidate tasks include the longest generation time, the least number of deaths, and the most number of enemies killed. Of course, the above multiple candidate tasks can also be tasks in different scenarios.

[0246] The target task is any one of the candidate tasks that is not used to train the decision model among the multiple candidate tasks.

[0247] Among them, the task vector corresponding to the above target task is obtained based on the characteristic subtasks and common subtasks of the above multiple candidate tasks.

[0248] Optionally, before determining the target task from a plurality of candidate tasks, a task joint representation is performed on each of the plurality of candidate tasks to obtain a task vector corresponding to each candidate task.

[0249] Specifically. Perform task decomposition on each of the above-mentioned multiple candidate tasks based on prior knowledge to obtain subtasks of each candidate task; obtain characteristic subtasks and common subtasks based on the subtasks of each candidate task in the multiple candidate tasks; perform task joint representation on each of the above-mentioned multiple candidate tasks based on the characteristic subtasks and common subtasks to obtain a task vector corresponding to each candidate task. Among them, the elements in the task vector corresponding to each candidate task correspond one-to-one to the above-mentioned characteristic subtasks and common subtasks. The different values ​​of each element in the task vector corresponding to the candidate task indicate whether the candidate task contains the subtask corresponding to the element.

[0250] For example, the above-mentioned multiple candidate tasks include turning left at an intersection, going straight at an intersection and turning right at an intersection; according to prior knowledge, turning left at an intersection can be decomposed into two sub-tasks: reaching or colliding with a left turn at an intersection and improving traffic efficiency; going straight at an intersection can be decomposed into two sub-tasks: reaching or colliding with a left turn at an intersection and improving traffic efficiency; turning right at an intersection can be decomposed into two sub-tasks: reaching or colliding with a right turn at an intersection and improving traffic efficiency; among them, reaching or colliding with a left turn at an intersection, colliding or reaching a straight-through at an intersection and colliding or reaching a right turn at an intersection are characteristic sub-tasks, and improving traffic efficiency is a common sub-task.

[0251] In one example, the task vector corresponding to turning left at an intersection can be expressed as [1,0,0,1], the task vector corresponding to going straight at an intersection can be expressed as [0,1,0,1], and the task vector corresponding to turning right at an intersection can be expressed as [0,0,1,1], where element 1 in the vector indicates that the subtask of the candidate task includes the subtask corresponding to the element, and element 0 in the vector indicates that the subtask of the candidate task does not include the subtask corresponding to the element.

[0252] For the intersection scenario, the state information of the target task s t Includes information about the vehicle and surrounding vehicles. Optionally, the information about the vehicle includes the speed v of the vehicle et The information of surrounding social vehicles includes the coordinates of surrounding social vehicles in the vehicle coordinate system (x it ,y it ), speed v it and heading angle θ it Assume that the surrounding social vehicles include the 5 vehicles closest to the self-vehicle, and the state information of the target task s t It can be expressed as s t =[v et ,x 1t ,y 1t ,v 1t ,θ 1t ,…,x 5t ,y 5t ,v 5t ,θ 5t ].

[0253] S502: The task vector g corresponding to the target task and the state information s of the target task are t Input to the decision model M t In order to obtain candidate actions, the target action a of the target task is selected from the candidate actions of the target task and the actions randomly obtained from the action space according to the preset probability. t .

[0254] Among them, the probability of the candidate action of the target task being selected is the preset probability.

[0255] The task vector g corresponding to the target task and the state information s of the target task t The action value function vector R(s) of the target task t ,a k ,g). Among them, the action value function vector R(s t ,a k ,g) has a one-to-one correspondence with the subtasks corresponding to the elements in the task vector corresponding to the target task. The action value function in the action value function vector is used to represent the execution of action a for the corresponding subtask. kFinally, we get the quality function of the result.

[0256] It should be noted that the above decision model M t It is implemented based on neural networks, such as fully connected neural networks, convolutional neural networks, pooling neural networks or other forms of neural networks.

[0257] According to the task vector g corresponding to the target task and the action value function vector R(s t ,a k ,g) Determine the Q value function Q(s) of the target task t ,a k ; g).

[0258] Since the action value function vector R(s t ,a k , g) contains the action value functions of subtasks that are not related to the target task. For example, the subtasks that are not related to the left turn at the intersection include right turn arrival or collision at the intersection and straight-through collision or arrival at the intersection. In order to avoid the influence of the action value functions of subtasks that are not related to the target task, the action value function R(s) of the target task is converted according to the task vector g corresponding to the target task. t ,a k ,g) The action value functions of subtasks that are not related to the target task are deleted to obtain the Q value function of the target task.

[0259] Among them, the Q value function of the target task can be expressed as:

[0260] Q(s t ,a k ; g) = g T R(s t ,a k ,g)

[0261] According to the Q value function Q(s t ,a k ; g) Determine a candidate action from the action space, the candidate action is the action space that makes the Q value function Q(s t ,a k ; g) The action with the largest value.

[0262] In a feasible embodiment, a target action a of the target task is selected from the candidate actions of the target task and the actions randomly obtained from the action space according to a preset probability. t ,include:

[0263] When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action a of the target task. t; The first parameter is a random number in the range of [0,1]; when the first parameter is not greater than the preset probability, the action randomly obtained from the action space is determined as the target action a of the target task t .

[0264] It should be noted here that the initial value of the preset probability is 1 or a larger value close to 1; the preset probability gradually decreases with the increase in the number of training times; optionally, the maximum value of the preset probability is 1 and the minimum value is 0.1.

[0265] S503, in target action a t After being executed, get the status information s of the target task t+1 , and according to the status information s t+1 And the reward value function obtains the reward value vector r(s) of the target task t+1 ,a t ,g).

[0266] Among them, the reward value vector (s t+1 ,a t ,g) has a one-to-one correspondence with the subtasks corresponding to the elements in the task vector corresponding to the target task. For example, for a left turn at an intersection, its reward value vector can be expressed as [r l ,r s ,r r ,r c ], where r l is the reward value for the subtask left turn at the intersection, collision or arrival, r s is the reward value for the subtask intersection straight collision or arrival, r r is the reward value for the subtask right turn at the intersection, collision or arrival, r c The reward value for improving the travel efficiency of the subtask.

[0267] The reward value function can be expressed as:

[0268] It should be noted here that the status information s t and status information t+1 The same type of information at different times.

[0269] S504: Corresponding task vector g and state information s according to the target task t , status information t+1 , target action a t and the reward value vector r(s t+1 ,a t ,g) Get sample data of the target task t ,a t ,r,s t+1 ,g>, and the sample data t ,a​​t ,r,s t+1 ,g>Save to the sample database.

[0270] It should be noted that the sample database includes sample data of multiple candidate tasks, and the same candidate task may include sample data acquired at different times.

[0271] S505, randomly obtain multiple sample data from the sample database, and calculate the loss value according to the loss function and the multiple sample data; adjust the decision model M according to the loss value t The parameters in the decision model M t+1 .

[0272] Among them, the loss value can be expressed as:

[0273]

[0274] Among them, r in the above formula is the reward value vector in the above sample data, g is the task vector in the above sample data, and a t is the target action in the above sample data, and the discount coefficient γ is a constant.

[0275] It should be noted here that the above-mentioned multiple pieces of sample data may be sample data of the same task or sample data of different tasks. Some of the sample data among the above-mentioned multiple pieces of sample data are of the same task, while some are of different tasks.

[0276] S506: According to the status information s t+1 Determine whether the target task is completed.

[0277] For example, for the intersection scene, the state information s t+1 Including the speed v of the vehicle e(t+1) , the coordinates of the surrounding vehicles in the vehicle coordinate system (x i(t+1) ,y i(t+1) ), speed V i(t+1) and heading angle θ i(t+1) , can be obtained by the vehicle's speed v e(t+1) , the coordinates of the surrounding vehicles in the vehicle coordinate system (x i(t+1) ,y i(t+1) ), speed V i(t+1) and heading angle θ i(t+1) Determine whether the vehicle has passed the intersection; if it has passed the intersection, it is determined that the target task is completed; if it has not passed the intersection, it is determined that the target task is not completed.

[0278] When it is determined that the target task is completed, step S507 is executed; if it is determined that the target task is not completed, t=t+1 is set and step S502 is executed.

[0279] S507, judgment decision model M t+1 Whether to converge.

[0280] Judgment decision model M t+1 Whether it converges, if it converges, execute step S508, if it does not converge, execute step S501.

[0281] Optionally, the decision model M is determined by judging whether the accumulated reward value converges. t+1 Specifically, the task vector g and state information s corresponding to the target task are t+1 Input to the decision model M t+1 In order to obtain a new reference action, specifically, according to the task vector g and state information s corresponding to the target task t+1 Get the action value function vector R(s t+1 ,a k ,g), and according to the action value function vector R(s t+1 ,a k ,g) and the task vector g corresponding to the target task determine the Q value function Q(s t+1 ,a k ; g), make the Q value function Q(s) in the action space t+1 ,a k ; g) The action with the largest value is determined as the new reference action, and a new target action a is selected from the new candidate actions and the actions randomly obtained from the action space according to the new preset probability t+1 , the probability of the new candidate action being selected is the new preset probability; in the new target action a t+1 After being executed, get the status information s t+2 , according to the reward value function and the current state information s t+2 Determine the reward value vector r(s) of the target action task t+2 ,a t+1 ,g); according to the status information s t+2 Determine whether the target task is completed; if it is determined that the target task is not completed, set t = t + 1, and repeat the above steps until the target task is completed. So far, multiple reward value vectors are obtained, and the reward values ​​corresponding to the subtasks related to the target task in each reward value vector in the multiple reward value vectors are accumulated to obtain the reward accumulation value.

[0282] If the reward accumulation value converges, the decision model M is determined t+1 Convergence; if the reward accumulation value does not converge, determine the decision model M t+1 Not convergent.

[0283] It should be pointed out here that in the above process, when obtaining the new target action a t+1, status information t+2 and the reward value vector r(s t+1 ,a t+1 ,g), according to the task vector g and state information s corresponding to the target task t+1 , target action a t+1 , status information t+2 and the reward value vector r(s t+2 ,a t+1 ,g) Generate sample data and save the sample data into the sample database.

[0284] It should be noted that after the target task is completed, the target task is deleted from multiple candidate tasks to obtain the deleted candidate task; when step S501 needs to be executed again later, obtaining the target task from multiple candidate tasks specifically refers to obtaining a new target task from the deleted candidate task.

[0285] S508, the decision model M t+1 Determine the target decision model and save it.

[0286] See also Figure 6 , Figure 6 A structural diagram of a decision-making device provided in an embodiment of the present application. Figure 6 As shown, the decision device 600 includes:

[0287] The task and status information acquisition module 601 is used to acquire the target task and its corresponding task vector from multiple candidate tasks, and acquire the status information of the target task during the process of training the decision model by the decision model training module 602 and during the process of making a decision based on the target decision model by the decision module 603, and send the target task and its task vector, and the status information of the target task to the decision model training module 602 and the decision module 603;

[0288] The decision model training module 602 is used to update the sample database according to the task vector corresponding to the target task and the state information of the target task; randomly obtain multiple sample data from the updated sample database, train the decision model according to the multiple sample data to obtain the target decision model; and send the target decision model to the decision module 603;

[0289] The decision module 603 is used to obtain the target action based on the trained decision model, the state information of the target task, and the task vector corresponding to the target task, and send the target action to the control module 604.

[0290] The control module 604 is used to execute the target action to complete the target task.

[0291] It should be noted here that the above-mentioned task and status information acquisition module 601 is specifically used to execute the relevant contents of steps S201, S401 and S501; the above-mentioned decision model training module 602 is used to execute the relevant contents of steps S402, S502-S508; the decision model 603 is used to execute the relevant contents of steps S202-S203, which will not be described in detail here.

[0292] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a vehicle-mounted device provided in an embodiment of the present application. Figure 7 As shown, the vehicle-mounted device 700 includes:

[0293] The environmental perception module 701 is used to perceive the surrounding environmental information of the vehicle, such as integrating the information obtained by various sensors to obtain the position, speed of the vehicle and the speed, position and heading angle of other vehicles around the vehicle; and send the information as status information to the decision module 703.

[0294] The navigation information module 702 is used to clarify the navigation information of the vehicle at the intersection, including the distance from the intersection, the traffic light information at the intersection and the direction of the vehicle, among which the turning directions of the vehicle are the above-mentioned multiple candidate tasks; and send the navigation information to the decision module 703.

[0295] The decision module 703 is used to determine the target task from the above-mentioned multiple candidate tasks according to the navigation information, such as turning left at an intersection; perform task joint characterization on the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task; input the task vector corresponding to the target task and the state information obtained by the environmental perception module 701 into the decision model for processing to obtain the target action of the target task, such as accelerating to turn left, decelerating to turn left, etc., and send the target action to the vehicle control module 704.

[0296] The vehicle control module 704 is used to control the vehicle driving according to the target action to complete the target task.

[0297] It should be noted that the specific process of the decision module 703 obtaining the target action based on the state information and the task vector corresponding to the target task can be found in the relevant description of steps S202-S203, which will not be described in detail here.

[0298] See also Figure 8 , Figure 8 Schematic diagram of a decision-making device based on multi-task learning. Figure 8 As shown, the decision device 800 includes:

[0299] The acquisition unit 801 is used to acquire multiple candidate tasks, acquire a target task from the multiple candidate tasks, and acquire the state information s of the target task from the target task.t ;

[0300] A joint characterization unit 802 is used to perform task joint characterization on a target task according to multiple candidate tasks to obtain a task vector corresponding to the target task, wherein the task vector corresponding to each candidate task is obtained based on characteristic subtasks and common subtasks of the multiple candidate tasks;

[0301] The determination unit 803 is used to determine the task vector corresponding to the target task and the state information s of the target task. t Determine the target action from the action space.

[0302] In a feasible embodiment, the joint characterization unit 802 is specifically configured to:

[0303] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0304] In a feasible embodiment, according to the task vector corresponding to the target task and the state information s of the target task t In terms of obtaining the action value function vector of the target task, the determination unit 803 is specifically used to:

[0305] The task vector corresponding to the target task and the state information s of the target task are t The input is processed into the target decision model to obtain the target action of the target task. The target decision model is implemented based on a neural network.

[0306] In a feasible embodiment, the determining unit 803 is specifically configured to:

[0307] According to the task vector corresponding to the target task and the state information s of the target task t Obtain an action value function vector of the target task, wherein the action value functions in the action value function vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; obtain the value function of the target task according to the task vector and the action value function vector corresponding to the target task; obtain the target action from the action space according to the value function of the target task, wherein the target action is the action in the action space that maximizes the value function of the target task.

[0308] It should be noted that the above-mentioned units (the acquisition unit 801 , the joint characterization unit 802 and the determination unit 803 ) are used to execute the relevant contents of the method shown in the above-mentioned steps S201 - S203 .

[0309] In this embodiment, the decision device 800 is presented in the form of a unit. The "unit" here may refer to a specific application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the acquisition unit 801, the joint characterization unit 802, and the determination unit 803 in the above decision device 800 can be Fig.10 The decision making device shown is implemented by the processor 1000.

[0310] See also Fig. 9 , Fig. 9 A structural diagram of a decision model training device provided in an embodiment of the present application. Fig. 9 As shown, the training device 900 includes:

[0311] An acquisition unit 901 randomly acquires a plurality of sample data from a first sample database; the first sample database includes sample data of a plurality of candidate tasks, the sample data of a target task includes a task vector corresponding to the target task, the task vector corresponding to the target task is obtained based on common subtasks and characteristic subtasks in the plurality of candidate tasks, and the target task is any one of the plurality of candidate tasks;

[0312] The adjustment unit 902 is used to adjust the decision model M according to the multiple sample data using the reinforcement learning method. t The parameters in the decision model M t+1 ;

[0313] Determining unit 903, used for determining the decision model M t+1 When convergence, determine the decision model M t+1 The target decision model.

[0314] In one possible embodiment,

[0315] The acquisition unit 901 is used to acquire a target task from multiple candidate tasks; and acquire the status information s of the target task. t And obtain the task vector corresponding to the target task,

[0316] The training device 900 further includes:

[0317] Update unit 904, used to update the state information s of the target task according to the state information s of the target task. t , the task vector corresponding to the target task and the decision model M t Generate sample data of the target task, and add the sample data of the target task to the preliminary sample database to obtain a first sample database.

[0318] In a feasible embodiment, in terms of acquiring the task vector corresponding to the target task, the acquiring unit 901 is specifically used to:

[0319] Decomposing each of the multiple candidate tasks to obtain subtasks corresponding to each candidate task; extracting characteristic subtasks and common subtasks of the multiple candidate tasks according to the subtasks corresponding to each of the multiple candidate tasks; and obtaining a task vector corresponding to the target task according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

[0320] In a feasible embodiment, according to the state information s of the target task t , the task vector corresponding to the target task and the decision model M t In terms of generating sample data of the target task, the updating unit 904 is specifically used for:

[0321] The status information of the target task s t The task vector corresponding to the target task is input into the decision model M t , to obtain the candidate actions of the target task; select the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to the preset probability, and the probability of the candidate action of the target task being selected is the preset probability; after the target action is executed, obtain the state information s of the target task t+1 , and according to the status information s of the target task t+1 Obtain a reward value vector for the target task; the reward values ​​in the reward value vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task;

[0322] The sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task. t , target action of target task, state information of target task s t+1 and the reward value vector of the target task.

[0323] In a feasible embodiment, when the state information s of the target task is t The task vector corresponding to the target task is input into the decision model M t In order to obtain aspects of candidate actions of the target task, the updating unit 904 is specifically used for:

[0324] Make the decision model M t According to the status information of the target task s tThe action value function vector of the target task is obtained from the task vector corresponding to the target task, and the action value function in the action value function vector of the target task corresponds one to one to the subtasks corresponding to the elements in the task vector corresponding to the target task; the value function of the target task is obtained according to the action value function vector of the target task and the task vector corresponding to the target task; the candidate action of the target task is obtained from the action space according to the value function of the target task, and the candidate action is the action in the action space that maximizes the value function of the target task.

[0325] In a feasible embodiment, in selecting the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to a preset probability, the updating unit 904 is specifically used to:

[0326] When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action of the target task; the first parameter is a random number in the value range [0,1]; when the first parameter is not greater than the preset probability, the action randomly obtained from the action space is determined as the target action of the target task.

[0327] In a feasible embodiment, the adjustment unit 902 is specifically configured to:

[0328] According to the loss function and the state information s of each sample data in multiple sample data t , task vector, target action, state information s t+1 The loss value is calculated by the reward value vector; the decision model M is adjusted according to the loss value t The parameters in the decision model M t+1 .

[0329] It should be noted that the above-mentioned units (acquisition unit 901, adjustment unit 902, determination unit 903 and update unit 904) are used to execute the relevant contents of the method shown in the above-mentioned steps S401, S402 and steps S501-S508.

[0330] In this embodiment, the training device 900 is presented in the form of a unit. Here, the "unit" may refer to a specific application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the acquisition unit 901, the adjustment unit 902, the determination unit 903, and the update unit 904 in the above training device 900 can be Fig.11 The training device shown is implemented by the processor 1100.

[0331] like Fig.10 The decision making device shown can be Fig.10 The decision-making device 1000 includes at least one processor 1001, at least one memory 1002 and at least one communication interface 1003. The processor 1001, the memory 1002 and the communication interface 1003 are connected through a communication bus and communicate with each other.

[0332] The communication interface 1003 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0333] The memory 1002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0334] The memory 1002 is used to store application code for executing the above solution, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1002.

[0335] The code stored in the memory 1002 can execute the decision-making method based on multi-task learning provided above.

[0336] The processor 1001 may also be used as one or more integrated circuits to execute relevant programs to implement the decision-making method based on multi-task joint learning in an embodiment of the present application.

[0337] The processor 1001 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the decision-making method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1001. In the implementation process, each step of the training method of the state generation model and selection strategy of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1001. The above-mentioned processor 1001 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and module block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1002, and the processor 1001 reads the information in the memory 1002 and completes the decision-making method of the embodiment of the present application in combination with its hardware.

[0338] The communication interface 1003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the decision-making device and other devices or a communication network. For example, the communication interface 1003 can be used to obtain state information or send a target action to an execution device (such as a vehicle control device).

[0339] The bus may include a path for transmitting information between various components of the device (e.g., memory 1002, processor 1001, communication interface 1003). In a possible embodiment, the processor 1001 specifically performs the following steps:

[0340] Get multiple candidate tasks, get the target task from the multiple candidate tasks; and get the status information of the target task according to the target task t ; Perform task joint characterization on the target task according to multiple candidate tasks to obtain the task vector corresponding to the target task. The task vector corresponding to each candidate task is obtained based on the characteristic subtasks and common subtasks of multiple candidate tasks; According to the task vector corresponding to the target task and the state information s of the target task t , determine the target action from the action space.

[0341] like Fig.11 The decision model training device shown can be used Fig.11 The training device 1100 includes at least one processor 1101, at least one memory 1102 and at least one communication interface 1103. The processor 1101, the memory 1102 and the communication interface 1103 are connected through a communication bus and communicate with each other.

[0342] The communication interface 1103 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0343] The memory 1102 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0344] The memory 1102 is used to store application code for executing the above solution, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the application code stored in the memory 1102.

[0345] The code stored in the memory 1102 can execute the decision model training method based on multi-task learning provided above.

[0346] The processor 1101 may also be used as one or more integrated circuits to execute relevant programs to implement the decision model training method based on multi-task learning in an embodiment of the present application.

[0347] The processor 1101 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the decision model training method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1101. In the implementation process, each step of the training method of the state generation model and selection strategy of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1101. The above-mentioned processor 1101 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and module block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1102, and the processor 1101 reads the information in the memory 1102 and completes the decision model training method of the embodiment of the present application in combination with its hardware.

[0348] The communication interface 1103 uses a transceiver such as, but not limited to, a transceiver to implement communication between the decision model training device and other devices or communication networks. For example, the communication interface 1103 can be used to obtain sample data used in model training, or to send a trained decision model to the decision device.

[0349] The bus may include a path for transmitting information between various components of the device (e.g., the memory 1102, the processor 1101, and the communication interface 1103). In a possible embodiment, the processor 1101 specifically performs the following steps:

[0350] Obtain the target task from multiple candidate tasks; and obtain the status information s of the target task t And obtain the task vector corresponding to the target task. The task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks of multiple candidate tasks, according to the state information s of the target task t , the task vector corresponding to the target task and the decision model M tGenerate sample data of the target task, add the sample data of the target task to the preliminary sample database to obtain a first sample database; randomly obtain multiple sample data from the first sample database; the multiple sample data are sample data of some or all of the candidate tasks in the multiple candidate tasks; use a reinforcement learning method to adjust the decision model M according to the multiple sample data t The parameters in the decision model M t+1 ; Judgment decision model M t+1 Whether it converges; when the decision model M t+1 When convergence, determine the decision model M t+1 The target decision model.

[0351] An embodiment of the present application provides a computer storage medium, which stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes part or all of the steps of any one of the decision-making methods based on multi-task learning or decision model training methods recorded in the above method embodiments.

[0352] In some embodiments, the disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture. Fig.12 Schematically illustrates a conceptual partial view of an example computer program product arranged according to at least some embodiments presented herein, the example computer program product comprising a computer program for executing a computer process on a computing device. In one embodiment, the example computer program product 1200 is provided using a signal bearing medium 1201. The signal bearing medium 1201 may include one or more program instructions 1202, which when executed by one or more processors may provide the above-described Figure 2 , Figure 4 and Figure 5 Thus, for example, reference to Figure 2 In the embodiment shown in , one or more features of steps S201-203 may be undertaken by one or more instructions associated with the signal bearing medium 1201. In addition, Fig.12 Program instructions 1202 in also describe example instructions.

[0353] In some examples, the signal bearing medium 1201 may include a computer readable medium 1203, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM) or a random access memory (RAM), etc. In some embodiments, the signal bearing medium 1201 may include a computer recordable medium 1204, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 1201 may include a communication medium 1205, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1201 may be communicated by a wireless form of communication medium 1205 (e.g., a wireless communication medium that complies with the IEEE802.11 standard or other transmission protocol). One or more program instructions 1202 may be, for example, computer executable instructions or logic implementation instructions. In some examples, such as for Figure 2 , Figure 4 and Figure 5 The computing device described can be configured to provide various operations, functions, or actions in response to program instructions 1202 communicated to the computing device by one or more of computer-readable media 1203, computer-recordable media 1204, and / or communication media 1205. It should be understood that the arrangement described here is only for illustrative purposes. Thus, it will be understood by those skilled in the art that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements can be omitted together according to the desired results. In addition, many of the elements described are functional entities that can be implemented as discrete or distributed components, or in any appropriate combination and position in combination with other components.

[0354] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0355] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0356] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0357] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0358] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0359] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a memory, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0360] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0361] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A decision model training method based on multi-task learning, characterized in that: include: Randomly obtain a plurality of sample data from a first sample database; The first sample database includes sample data of multiple candidate tasks, and the sample data of the target task includes the task vector corresponding to the target task and the state information s of the target task. t , the task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks in the multiple candidate tasks, wherein the target task is any one of the multiple candidate tasks; the multiple candidate tasks include going straight at an intersection, turning left at an intersection, and turning right at an intersection; the going straight at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the turning left at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the turning right at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the common subtask includes improving traffic efficiency; the characteristic subtasks include collision or arrival at an intersection, collision or arrival at an intersection, collision or arrival at a left turn, and collision or arrival at an intersection; the state information s of the target task t Including the information of the vehicle itself and surrounding vehicles; Using the reinforcement learning method, the decision model M is adjusted according to the multiple sample data t The parameters in the decision model M t+1 ; When the decision model M t+1 When convergence occurs, the decision model M is determined t+1 The target decision model.

2. The method according to claim 1, characterized in that: Before randomly acquiring a plurality of sample data from the first sample database, the method further includes: Acquire a target task from the plurality of candidate tasks; Acquire the status information s of the target task according to the target task t , and obtain the task vector corresponding to the target task according to the target task, According to the status information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample data of the target task, and add the sample data of the target task to a preliminary sample database to obtain the first sample database.

3. The method according to claim 2, characterized in that The acquiring, according to the target task, a task vector corresponding to the target task comprises: Decomposing each of the plurality of candidate tasks into subtasks corresponding to each of the candidate tasks; Extracting characteristic subtasks and common subtasks of the plurality of candidate tasks according to the subtasks corresponding to each candidate task in the plurality of candidate tasks; A task vector corresponding to the target task is obtained according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

4. The method according to claim 2 or 3, characterized in that: The state information s of the target task t , the task vector corresponding to the target task and the decision model M t Generate sample data for the target task, including: The state information s of the target task t The task vector corresponding to the target task is input into the decision model M t , so as to obtain candidate actions for the target task; Selecting a target action of the target task from candidate actions of the target task and actions randomly obtained from the action space according to a preset probability, wherein the probability of the candidate action being selected is the preset probability; After the target action is executed, the status information s of the target task is obtained t+1 , and according to the status information s of the target task t+1 Obtaining a reward value vector of the target task; wherein the reward values ​​in the reward value vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; The sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task. t , the target action of the target task, and the state information s of the target task t+1 and the reward value vector of the target task.

5. The method according to claim 4, characterized in that The state information s of the target task t The task vector corresponding to the target task is input into the decision model M t , to obtain candidate actions for the target task, including: The decision model M t According to the status information s of the target task t The task vector corresponding to the target task obtains the action value function vector of the target task; The decision model M t Acquire the value function of the target task according to the action value function vector of the target task and the task vector corresponding to the target task; The decision model M t A candidate action of the target task is obtained from the action space according to the value function of the target task, wherein the candidate action is an action in the action space that maximizes the value function of the target task.

6. The method according to claim 5, characterized in that The step of selecting a target action of the target task from candidate actions of the target task and actions randomly acquired from the action space according to a preset probability includes: When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action of the target task; wherein the first parameter is a random number in the value range of [0,1]; When the first parameter is not greater than the preset probability, an action randomly obtained from the action space is determined as a target action of the target task.

7. The method according to claim 6, characterized in that The decision model M is adjusted according to the plurality of sample data t The parameters in the decision model M t+1 include: According to the loss function and the state information s of each sample data in the plurality of sample data t , the task vector, the target action, the state information s t+1 The loss value is calculated by the reward value vector; Adjust the decision model M according to the loss value t The parameters in order to obtain the decision model M t+1 .

8. A decision-making method based on multi-task learning, characterized in that: include: Acquire multiple candidate tasks, and acquire a target task from the multiple candidate tasks; And obtain the status information s of the target task according to the target task t ; The status information s of the target task t Including the information of the vehicle and the information of surrounding social vehicles; the multiple candidate tasks include going straight at the intersection, turning left at the intersection and turning right at the intersection; the going straight at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; the turning left at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; the turning right at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; The target task is jointly characterized according to the multiple candidate tasks to obtain a task vector corresponding to the target task, wherein the task vector corresponding to each candidate task is obtained based on characteristic subtasks and common subtasks of the multiple candidate tasks; the common subtasks include improving traffic efficiency; the characteristic subtasks include straight-ahead collision or arrival at an intersection, left-turn collision or arrival at an intersection, and right-turn collision or arrival at an intersection; According to the task vector corresponding to the target task and the state information s of the target task t Determine the target action from the action space.

9. The method according to claim 8, characterized in that The performing task joint characterization on the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task includes: Decomposing each of the plurality of candidate tasks into subtasks corresponding to each of the candidate tasks; Extracting characteristic subtasks and common subtasks of the plurality of candidate tasks according to the subtasks corresponding to each candidate task in the plurality of candidate tasks; A task vector corresponding to the target task is obtained according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

10. The method according to claim 8 or 9, characterized in that: The task vector corresponding to the target task and the state information s of the target task t Determine the target action from the action space, including: The task vector corresponding to the target task and the state information s of the target task are t The input is processed into a target decision model to obtain the target action, and the target decision model is implemented based on a neural network.

11. The method according to claim 10, characterized in that The task vector corresponding to the target task and the state information s of the target task are t Input into the target decision model for processing to obtain the target action, including: The target decision model is based on the task vector corresponding to the target task and the state information s of the target task. t Acquire an action value function vector of the target task, wherein the action value functions in the action value function vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; The target decision model obtains the value function of the target task according to the task vector corresponding to the target task and the action value function vector; The target decision model obtains a target action from the action space according to a value function of the target task, and the target action is an action in the action space that maximizes the value function of the target task.

12. A decision model training device based on multi-task learning, characterized in that: include: An acquisition unit, used for acquiring a plurality of sample data from a first sample database; The first sample database includes sample data of multiple candidate tasks, and the sample data of the target task includes the task vector corresponding to the target task and the state information s of the target task. t , the task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks in the multiple candidate tasks, wherein the target task is any one of the multiple candidate tasks; the multiple candidate tasks include going straight at an intersection, turning left at an intersection, and turning right at an intersection; the going straight at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the turning left at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the turning right at an intersection includes two subtasks: collision or arrival at an intersection and improving traffic efficiency; the common subtask includes improving traffic efficiency; the characteristic subtasks include collision or arrival at an intersection, collision or arrival at an intersection, collision or arrival at a left turn, and collision or arrival at an intersection; the state information s of the target task t Including the information of the vehicle itself and surrounding vehicles; An adjustment unit, configured to adjust the decision model M according to the plurality of sample data using a reinforcement learning method t The parameters in the decision model M t+1 ; A determination unit is used for determining the decision model M t+1 When convergence occurs, the decision model M is determined t+1 The target decision model.

13. The device according to claim 12, characterized in that The acquisition unit is further used to acquire a target task from a plurality of candidate tasks; and acquire state information s of the target task according to the target task. t , obtaining a task vector corresponding to the target task according to the target task, wherein the task vector corresponding to the target task is obtained based on the common subtasks and characteristic subtasks of the plurality of candidate tasks, The device also includes: An updating unit is used to update the state information s of the target task according to the state information s of the target task. t , the task vector corresponding to the target task and the decision model M t Generate sample data of the target task, and add the sample data of the target task to a preliminary sample database to obtain the first sample database.

14. The device according to claim 13, characterized in that In the aspect of acquiring the task vector corresponding to the target task according to the target task, the acquiring unit is specifically used for: Decomposing each of the multiple candidate tasks into subtasks corresponding to each candidate task; Extracting characteristic subtasks and common subtasks of the plurality of candidate tasks according to the subtasks corresponding to each candidate task in the plurality of candidate tasks; A task vector corresponding to the target task is obtained according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

15. The device according to claim 13 or 14, characterized in that In the state information s of the target task t , the task vector corresponding to the target task and the decision model M t In terms of generating sample data of the target task, the updating unit is specifically used for: The state information s of the target task t The task vector corresponding to the target task is input into the decision model M t , so as to obtain candidate actions for the target task; Selecting a target action of the target task from candidate actions of the target task and actions randomly obtained from the action space according to a preset probability, wherein the probability of the candidate action being selected is the preset probability; After the target action is executed, the status information s of the target task is obtained t+1 , and according to the state information s of the target task t+1 Obtaining a reward value vector of the target task; wherein the reward values ​​in the reward value vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; The sample data of the target task includes the task vector corresponding to the target task and the status information s of the target task. t , the target action of the target task, and the state information s of the target task t+1 and the reward value vector of the target task.

16. The device according to claim 15, characterized in that In the state information s of the target task t The task vector corresponding to the target task is input into the decision model M t to obtain aspects of candidate actions of the target task, wherein the updating unit is specifically used for: Let the decision model M t According to the status information s of the target task t The action value function vector of the target task is obtained from the task vector corresponding to the target task; the value function of the target task is obtained according to the action value function vector of the target task and the task vector corresponding to the target task; and the candidate action of the target task is obtained from the action space according to the value function of the target task, wherein the candidate action is the action in the action space that maximizes the value function of the target task.

17. The device according to claim 16, characterized in that In the aspect of selecting the target action of the target task from the candidate actions of the target task and the actions randomly obtained from the action space according to a preset probability, the updating unit is specifically used to: When the first parameter is greater than the preset probability, the candidate action of the target task is determined as the target action of the target task; the first parameter is a random number in the range of [0,1]; When the first parameter is not greater than the preset probability, an action randomly obtained from the action space is determined as a target action of the target task.

18. The device according to claim 17, characterized in that The adjustment unit is specifically used for: According to the loss function and the state information s of each sample data in the plurality of sample data t , the task vector, the target action, the state information s t+1 The loss value is calculated by the reward value vector; Adjust the decision model M according to the loss value t The parameters in order to obtain the decision model M t+1 .

19. A decision-making device based on multi-task learning, characterized in that: include: An acquisition unit, used to acquire a plurality of candidate tasks, and acquire a target task from the plurality of candidate tasks; And obtain the status information s of the target task according to the target task t ; The status information s of the target task t Including the information of the vehicle and the information of surrounding social vehicles; the multiple candidate tasks include going straight at the intersection, turning left at the intersection and turning right at the intersection; the going straight at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; the turning left at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; the turning right at the intersection includes two subtasks of collision or arrival at the intersection and improving traffic efficiency; A joint characterization unit is used to perform task joint characterization on the target task according to the multiple candidate tasks to obtain a task vector corresponding to the target task, wherein the task vector corresponding to each candidate task is obtained based on characteristic subtasks and common subtasks of the multiple candidate tasks; the common subtasks include improving traffic efficiency; the characteristic subtasks include collision or arrival at a straight-ahead intersection, collision or arrival at a left-turn at an intersection, and collision or arrival at a right-turn at an intersection; The determination unit is further configured to determine the task vector corresponding to the target task and the state information s of the target task. t , determine the target action from the action space.

20. The device according to claim 19, characterized in that The joint characterization unit is specifically used for: Decomposing each of the plurality of candidate tasks into subtasks corresponding to each of the candidate tasks; Extracting characteristic subtasks and common subtasks of the plurality of candidate tasks according to the subtasks corresponding to each candidate task in the plurality of candidate tasks; A task vector corresponding to the target task is obtained according to the characteristic subtasks and common subtasks of the multiple candidate tasks.

21. The device according to claim 19 or 20, characterized in that The determining unit is specifically used for: The task vector corresponding to the target task and the state information s of the target task are t The input is processed into a target decision model to obtain the target action of the target task, and the target decision model is implemented based on a neural network.

22. The device according to claim 21, characterized in that The determining unit is specifically used for: According to the task vector corresponding to the target task and the state information s of the target task t Acquire an action value function vector of the target task, wherein the action value functions in the action value function vector correspond one-to-one to the subtasks corresponding to the elements in the task vector corresponding to the target task; Acquire the value function of the target task according to the task vector corresponding to the target task and the action value function vector; A target action is obtained from the action space according to the value function of the target task, wherein the target action is an action in the action space that maximizes the value function of the target task.

23. A prediction model training device, characterized in that: include: A memory for storing instructions; as well as a processor coupled to the memory; Wherein, when the processor executes the instruction, the method according to any one of claims 1-7 is performed.

24. A decision-making device based on multi-task learning, characterized in that: include: A memory for storing instructions; as well as a processor coupled to the memory; Wherein, when the processor executes the instruction, the method according to any one of claims 8 to 11 is performed.

25. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 11.

26. A computer storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 11.

27. A computer program product, characterized in that The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

Images