Automated driving reinforcement learning motion planning method and system considering uncertainty

Abstract

The present application relates to a kind of automatic driving reinforcement learning motion planning method and system considering uncertainty, comprising the following steps: based on the historical driving data obtained, using deep reinforcement learning method to train the multi-head value function network constructed in advance;Obtain the environment state information of current time, and respectively using the multi-head value function network trained and the automatic driving system of the automatic driving vehicle obtain the motion trajectory based on data fitting and the motion trajectory based on rule, obtain the optimal motion trajectory of the automatic driving vehicle after comparison.The uncertainty that may be caused by the black-box characteristic of reinforcement learning algorithm in the calculation process is considered by combining bootstrap uncertainty estimation, the dangerous decision that this uncertainty can cause is avoided by the method based on rule, which helps to improve the safety and reliability of this kind of method in intelligent vehicle.Therefore, the present application can be widely applied in intelligent vehicle field.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent vehicles, and particularly relates to an automatic driving reinforcement learning motion planning method and system considering uncertainty. BACKGROUND

[0002] In recent years, artificial intelligence technology has gradually begun to be applied in the field of intelligent transportation and vehicles, and intelligent networked vehicles have gradually entered people's field of vision. Generally, the automatic driving system of an intelligent vehicle can be divided into four modules: perception, positioning, decision-making and control. Among them, motion planning is one of the most critical parts of the decision-making module of the automatic driving system, and the result of the planning has a direct impact on the final driving decision result of the vehicle.

[0003] Existing motion planning methods can be roughly divided into rule-based methods and learning-based methods. Learning-based methods, especially reinforcement learning-based methods, can autonomously learn the optimal decision-making strategy from a large amount of driving data, thereby hoping to improve the decision-making ability of unmanned driving to a level beyond the current artificial design methods. However, such methods are generally based on deep neural networks and have black box characteristics, and their risks are difficult to estimate, which poses a huge challenge to the safety of unmanned driving. SUMMARY

[0004] To solve the above problems, the application provides an automatic driving reinforcement learning motion planning method and system considering uncertainty, which considers the uncertainty caused by the black box characteristics of the reinforcement learning algorithm in the calculation process by combining bootstrap uncertainty estimation, thereby improving the motion planning accuracy.

[0005] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0006] In a first aspect, the application provides an automatic driving reinforcement learning motion planning method considering uncertainty, comprising the following steps:

[0007] Based on the obtained historical driving data, a multi-head value function network is trained using a deep reinforcement learning method;

[0008] The environment state information at the current time is obtained, and the data-fitted motion trajectory and the rule-based motion trajectory are obtained by using the trained multi-head value function network and the automatic driving system of the vehicle to be planned, respectively, and the optimal motion trajectory of the vehicle to be planned is obtained after comparison.

[0009] Further, the multi-head value function network is trained based on the obtained historical driving data using a deep reinforcement learning method, comprising:

[0010] The multi-head value function network is constructed and randomly initialized;

[0011] collecting driving data;

[0012] bootstrap sampling is performed on the collected driving data, and each sub neural network in the multi-head value function network is trained respectively based on the sampled data using a deep reinforcement learning method.

[0013] Further, the multi-head value function network includes a plurality of parallel sub neural networks, the input of each sub neural network being a current environment state s and a desired decision action a, and the output being a decision value Q(s, a) corresponding to the environment state-desired decision action pair.

[0014] Further, the environment state information at the current time is obtained, and the data-fitted motion trajectory based on the trained multi-head value function network and the automatic driving system of the vehicle to be planned are obtained, and the rule-based motion trajectory is compared to obtain the optimal motion trajectory of the vehicle to be planned, comprising:

[0015] Obtain the environment state information at the current time;

[0016] Based on the environment state information at the current time, the global trajectory planning result of the vehicle to be planned itself is obtained using the automatic driving system, and the rule-based motion trajectory is calculated based on the cost function;

[0017] Based on the environment state information at the current time, the data-fitted motion trajectory is calculated using the multi-head value function network;

[0018] Based on bootstrap uncertainty calculation, the optimal motion trajectory is determined from the rule-based motion trajectory and the data-fitted motion trajectory and output.

[0019] Further, based on the environment state information at the current time, the global trajectory planning result of the vehicle to be planned itself is obtained using the automatic driving system, and the rule-based motion trajectory is calculated based on the cost function, comprising:

[0020] Based on the environment state information at the current time, m candidate trajectories a1, a2, a3, … a m are generated according to the global trajectory planning result of the automatic driving.

[0021] In each planning cycle, the trajectory with the minimum cost value is selected from the candidate trajectories as the rule-based motion trajectory a r using the cost function.

[0022] Further, based on the environment state information at the current time, the data-fitted motion trajectory is calculated using the trained multi-head value function network, comprising:

[0023] S231, the current environment state information s and the rule-based motion trajectory ar As input to the multi-head value function network, a rule-based motion trajectory a is obtained. r The corresponding series of decision values ​​Q1(s, a) r ), Q2(s, a r ), ..., Q n (s, a) r );

[0024] S232. From m candidate trajectories a1, a2, a3, ... a m A trajectory is randomly selected as the desired decision action 'a'. The desired decision action 'a' and the current environmental state information 's' are used as inputs to a multi-head value function network (MFC), resulting in a series of decision values ​​corresponding to the desired decision action 'a': Q1(s, a), Q2(s, a), ..., Q... n (s, a);

[0025] S233. Compare the series of decision values ​​corresponding to the desired decision action a with the rule-based motion trajectory a. r By comparing a series of corresponding decision values, we can obtain the desired decision action a that satisfies Q. i (s, a) > Q i (s, a) r The number of decision values ​​K, i = 1, 2, ..., n;

[0026] S234. Repeat steps S232 to S234, and take the expected decision action that satisfies the conditions the most times as the motion trajectory a based on data fitting. d .

[0027] Furthermore, the step of determining and outputting the optimal motion trajectory from rule-based and data-fit-based motion trajectories based on bootstrapping uncertainty calculation includes:

[0028] The motion trajectory a based on data fitting d In the middle, Q is satisfied i (s, a) > Q i (s, a) r The number of decision values ​​K is compared with the preset threshold K0:

[0029] If K is greater than K0, then the motion trajectory a will be based on the data fitting. d The optimal planned trajectory for the vehicle to be planned will be output; otherwise, the motion trajectory a based on rules will be used. r This serves as the output of the optimal planned trajectory for the vehicle to be planned.

[0030] Secondly, the present invention provides an autonomous driving reinforcement learning motion planning system that considers uncertainty, comprising:

[0031] An offline training module is configured to train a pre-constructed multi-head value function network based on acquired historical driving data using a deep reinforcement learning method.

[0032] An online planning module is configured to acquire environment state information at a current time, and obtain a data-fitting-based motion trajectory and a rule-based motion trajectory using the trained multi-head value function network and an automatic driving system of a vehicle to be planned respectively, and obtain an optimal motion trajectory of the vehicle to be planned after comparison.

[0033] In a third aspect, the present application provides a processing device, which comprises at least a processor and a memory, and the memory stores a computer program, and the processor executes the computer program to implement the steps of the automatic driving reinforcement learning motion planning method considering uncertainty.

[0034] In a fourth aspect, the present application provides a computer storage medium, which stores computer readable instructions, and the computer readable instructions can be executed by a processor to implement the steps of the automatic driving reinforcement learning motion planning method considering uncertainty.

[0035] The present application has the following advantages due to the above technical solutions: the automatic driving reinforcement learning motion planning based on bootstrap uncertainty estimation provided by the present application considers the uncertainty caused by the black-box characteristics of the reinforcement learning algorithm in the calculation process by combining bootstrap uncertainty estimation, avoids the dangerous decisions caused by the uncertainty through a rule-based method, and helps to improve the safety and reliability of the method in intelligent vehicles. Therefore, the present application can be widely applied in the field of intelligent vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0036] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not meant to limit the present application. Throughout the drawings, the same reference numerals are used for the same components. In the drawings:

[0037] Figure 1 is a flow chart of the automatic driving reinforcement learning motion planning method provided by the embodiments of the present application;

[0038] Figure 2 is a multi-head value function network model structure diagram provided by the embodiments of the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0040] It should be noted that the terms used herein are only intended to describe specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates otherwise, and it should also be understood that, when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component and / or a combination thereof.

[0041] In some embodiments of the present application, an automatic driving reinforcement learning motion planning method considering uncertainty is provided, which receives automatic driving perception results, i.e. information such as positions and speeds of obstacles in the surrounding environment, takes the current position of the ego vehicle as the starting point, and outputs a series of time sequence action sequences that can be executed by the intelligent vehicle. The generation of executable actions uses a planning method based on deep reinforcement learning, and the uncertainty of the neural network is estimated during the planning process.

[0042] Correspondingly, in some other embodiments of the present application, an automatic driving reinforcement learning motion planning method, device and medium considering uncertainty are provided.

[0043] Embodiment 1

[0044] As shown in Figure 1 , the present embodiment provides an automatic driving reinforcement learning motion planning method considering uncertainty, which includes two stages of offline training and online planning, and specifically includes the following steps:

[0045] Step S1, offline training: based on the obtained historical driving data, a multi-head value function network is trained using a deep reinforcement learning method;

[0046] Step S2, online planning: the environment state information at the current time is obtained, and the motion trajectory based on data fitting and the motion trajectory based on rules are obtained respectively by using the trained multi-head value function network and the automatic driving system of the automatic driving vehicle, and the optimal motion trajectory of the automatic driving vehicle is obtained after comparison.

[0047] Preferably, the above step S1 can be realized by the following steps:

[0048] S11, constructing a multi-head value function network and performing random initialization;

[0049] S12, collect historical driving data;

[0050] S13, bootstrap sampling on the collected historical driving data, and respectively train each sub neural network in the multi-headed value function network using a deep reinforcement learning method.

[0051] Preferably, as shown in the above step S11, the multi-headed value function network comprises a plurality of parallel sub neural networks (i.e. value functions), and the input of each sub neural network is the current environment state s and the expected decision action a, and the output is the decision value Q(s, a) corresponding to the environment state-expected decision action pair. Figure 2

[0052] Preferably, in the above step S11, the random initialization operation on the multi-headed value function network refers to the initialization operation on the weights in the plurality of parallel sub neural networks, wherein the initial value of each weight is sampled from a manually defined Gaussian distribution or other random distribution.

[0053] Preferably, in the above step S12, the collected driving data refers to the data generated during the driving of the autonomous vehicle, and the format of each piece of data is (s, a, r, s'), wherein s is the current environment state information, a is the expected decision action, r is the reward value obtained after executing the expected decision action a, and s' is the next environment state information. The reward value r is a function of the environment state information s and the expected decision action a, i.e. r(s, a).

[0054] Preferably, in the above step S13, the bootstrap sampling of the driving data from the replay storage refers to: randomly collecting a piece of driving data from the replay storage, and using the piece of driving data to train a part of the parallel sub neural networks in the multi-headed value function network. In order to determine which sub neural networks can be trained using the piece of driving data, for each sub neural network, a sampling based on a binomial distribution is performed, and if the sampling result is 1, the piece of driving data is used to train the sub neural network, otherwise the piece of driving data is not used to train the sub neural network.

[0055] Preferably, in the above step S13, when respectively training each sub neural network in the multi-headed value function network using a deep reinforcement learning method, a reinforcement learning module based on a deep Q learning algorithm is used. For the reinforcement learning algorithm, the state space and the action space need to be determined first. In the present application, the state space is the information of the obstacles in the environment, including the position, size and speed of the obstacles (x, y, l, w, vx, vy), x and y are the coordinates of the obstacles, l and w are the length and width of the obstacles, and vx and vy are the speeds of the obstacles in two directions; the action space is set as a discrete action space, and its length can be manually set according to actual needs. ​

[0056] In the offline training process, the reward function of the reinforcement learning module also needs to be determined:

[0057] R=r1+r2+r3

[0058] Wherein, r1 is the reward of reaching the end point, r2 is the collision penalty, and r1 is the penalty for long parking.

[0059] Preferably, the above step S2 can be realized by the following steps:

[0060] S21, obtaining the environment state information at the current time;

[0061] S22, based on the environment state information at the current time, using the global trajectory planning result of the autonomous vehicle itself and the cost function considering comfort, traffic efficiency and other factors, calculating the rule-based motion trajectory a r .

[0062] S23, based on the environment state information at the current time, using the trained multi-head value function network, calculating the data fitting-based motion trajectory;

[0063] S24, based on bootstrap uncertainty calculation, determining the optimal motion trajectory from the rule-based motion trajectory and the data fitting-based motion trajectory and outputting.

[0064] Preferably, the above step S22 can be realized by the following steps:

[0065] S221, based on the environment state information at the current time, generating m candidate trajectories a1, a2, a3,... a m of motion planning according to the global trajectory planning result of the autonomous vehicle.

[0066] S222, in each planning cycle, using the cost function considering comfort, traffic efficiency and other factors, selecting the trajectory with the minimum value from the candidate trajectories as the rule-based motion trajectory a r .

[0067] Preferably, in the above step S221, the global trajectory planning result is given by the autonomous driving system of the autonomous vehicle or manually specified.

[0068] Preferably, in the above step S222, the cost function is:

[0069] C=k j J t +k t g(T)+k p h(p1)

[0070] Wherein, J tThe time integral of the jerk of all points on the entire trajectory, g and h are arbitrary functions, T is the total expected time consumption of the trajectory, p1 is the lateral distance between the last point of the trajectory and the global trajectory planning result, k j , k t , k p is a weight, which can be manually adjusted and takes a value > 0.

[0071] Preferably, in the above step S23, as shown in Figure 2 , based on the environment state information at the current moment, the method for calculating the motion trajectory based on data fitting is obtained by using the trained multi-head value function network, and the method is specifically:

[0072] S231, the current environment state information s and the rule-based motion trajectory a r are input into the multi-head value function network to obtain a series of decision values Q1(s, a r ), Q2(s, a r ),..., Q r (s, a n ) corresponding to the rule-based motion trajectory a r ;

[0073] S232, a trajectory is randomly selected from m candidate trajectories a1, a2, a3,..., a m as the expected decision action a, and the expected decision action a and the current environment state information s are input into the multi-head value function network to obtain a series of decision values: Q1(s, a), Q2(s, a),..., Q n (s, a) corresponding to the expected decision action a;

[0074] S233, the series of decision values corresponding to the expected decision action a are compared with the series of decision values corresponding to the rule-based motion trajectory a r , and the number K of decision values in the expected decision action a that satisfy Q i (s, a) > Q i (s, a r ) is obtained, i = 1, 2,..., n;

[0075] S234, steps S232-S234 are repeated, and the expected decision action with the largest number of conditions is taken as the motion trajectory based on data fitting a d .

[0076] Preferably, in the above step S24, the method for outputting the optimal planning trajectory of the autonomous vehicle from the rule-based motion trajectory a r and the motion trajectory based on data fitting a d based on bootstrap uncertainty calculation includes:

[0077] the motion trajectory a based on data fitting is output as the optimal planning trajectory of the autonomous vehicle, otherwise, the motion trajectory a based on rules is output as the optimal planning trajectory of the autonomous vehicle. d Q is satisfied in the middle i (s, a) > Q i (s, a r ) is compared with a preset threshold K0, if K is greater than K0, the motion trajectory a based on data fitting is output as the optimal planning trajectory of the autonomous vehicle, otherwise, the motion trajectory a based on rules is output as the optimal planning trajectory of the autonomous vehicle. d (s, a) > Q r (s, a) > Q

[0078] Embodiment 2

[0079] The above embodiment 1 provides an autonomous driving reinforcement learning motion planning method considering uncertainty, and correspondingly, the present embodiment provides an autonomous driving reinforcement learning motion planning system considering uncertainty. The system provided by the present embodiment can implement the autonomous driving reinforcement learning motion planning method considering uncertainty of embodiment 1. The system can be realized by software, hardware or a combination of software and hardware. For example, the system can include integrated or separate functional modules or functional units to perform the corresponding steps in the methods of embodiment 1. Since the system of the present embodiment is basically similar to the method embodiment, the description process of the present embodiment is relatively simple, and the related parts can be referred to the part of the description of embodiment 1. The system provided by the present embodiment is only schematic.

[0080] The autonomous driving reinforcement learning motion planning system considering uncertainty provided by the present embodiment comprises:

[0081] An offline training module is configured to train the constructed multi-head value function network using a deep reinforcement learning method based on the acquired historical driving data;

[0082] An online planning module is configured to acquire the environment state information at the current time, and obtain the motion trajectory based on data fitting and the motion trajectory based on rules by using the trained multi-head value function network and the autonomous driving system of the autonomous vehicle respectively, and obtain the optimal motion trajectory of the autonomous vehicle after comparison.

[0083] Embodiment 3

[0084] The present embodiment provides a processing device corresponding to the autonomous driving reinforcement learning motion planning method considering uncertainty provided by embodiment 1. The processing device can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.

[0085] The processing device includes a processor, a memory, a communication interface and a bus, the processor, the memory and the communication interface are connected through the bus to complete the communication between each other. The memory stores a computer program which can run on the processor, and the processor executes the computer program to perform the method for autonomous driving reinforcement learning motion planning considering uncertainty provided in Embodiment 1.

[0086] In some embodiments, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, for example at least one disk memory.

[0087] In other embodiments, the processor can be a central processing unit (CPU), a digital signal processor (DSP) and various types of general-purpose processors, which are not limited here.

[0088] Embodiment 4

[0089] The method for autonomous driving reinforcement learning motion planning considering uncertainty of Embodiment 1 can be embodied as a computer program product, which can include a computer readable storage medium having computer readable program instructions loaded thereon for executing the method for autonomous driving reinforcement learning motion planning considering uncertainty described in Embodiment 1.

[0090] The computer readable storage medium can be a tangible device that keeps and stores instructions for use by an instruction execution device. The computer readable storage medium may, for example, be but is not limited to an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above.

[0091] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An automatic driving reinforcement learning motion planning method considering uncertainty, characterized in that, The method comprises the following steps: training a pre-constructed multi-head value function network using a deep reinforcement learning method based on acquired historical driving data; acquiring environment state information at the current moment, and obtaining a data-fitted motion trajectory and a rule-based motion trajectory using the trained multi-head value function network and an automatic driving system of the vehicle to be planned respectively, and comparing the two trajectories to obtain an optimal motion trajectory of the vehicle to be planned; the training of the pre-constructed multi-head value function network using the deep reinforcement learning method based on the acquired historical driving data comprises: constructing the multi-head value function network and performing random initialization; collecting driving data; performing bootstrap sampling on the collected driving data, and training each sub neural network in the multi-head value function network using a deep reinforcement learning method based on the sampled data; the acquisition of the environment state information at the current moment and the obtaining of the data-fitted motion trajectory and the rule-based motion trajectory using the trained multi-head value function network and the automatic driving system of the vehicle to be planned respectively to obtain the optimal motion trajectory of the vehicle to be planned after comparison comprises: acquiring the environment state information at the current moment; acquiring a global trajectory planning result using the automatic driving system of the vehicle to be planned based on the environment state information at the current moment, and calculating a rule-based motion trajectory based on a cost function; calculating a data-fitted motion trajectory using the multi-head value function network based on the environment state information at the current moment; determining an optimal motion trajectory from the rule-based motion trajectory and the data-fitted motion trajectory based on bootstrap uncertainty calculation and outputting the optimal motion trajectory; the acquisition of the environment state information at the current moment and the obtaining of the data-fitted motion trajectory using the trained multi-head value function network based on the environment state information at the current moment comprises: Based on the current moment environment state information, according to the global trajectory planning result of automatic driving, m candidate trajectories of motion planning are generated ; In each planning cycle, the trajectory with the lowest cost from the candidate trajectories is selected as the rule-based motion trajectory using a cost function ; the multi-head value function network comprises a plurality of parallel sub neural networks, and the input of each sub neural network is a current environment state s and an expected decision action a, and the output is a decision value Q(s, a) corresponding to the environment state-expected decision action pair. ① the current environment state information s and the rule-based motion trajectory as the input of the multi-head value function network, to obtain the rule-based motion trajectory corresponding to a series of decision values ; ② randomly sampling one trajectory from m candidate trajectories ;​ ③ compare the series of decision values corresponding to the desired decision action a with the series of decision values corresponding to the rule-based motion trajectory , to obtain the number K of decision values in the desired decision action a that satisfy ; ; ④ Repeat steps ②-③, and take the expected decision action with the largest number of conditions met as the motion trajectory based on data fitting .

2. The method of claim 1, wherein the method further comprises: the determination of the optimal motion trajectory from the rule-based motion trajectory and the data-fitted motion trajectory based on the bootstrap uncertainty calculation and the outputting of the optimal motion trajectory comprise:

3. The method of claim 1, wherein the method further comprises: comprise: Motion trajectory based on data fitting Satisfies The decision value quantity K and the preset threshold K0 are compared: If K is greater than K0, the motion trajectory based on the data fitting will be output as the optimal planning trajectory of the vehicle to be planned, otherwise, the motion trajectory based on the rule will be output as the optimal planning trajectory of the vehicle to be planned If K is greater than K0, the motion trajectory based on the data fitting will be output as the optimal planning trajectory of the vehicle to be planned, otherwise, the motion trajectory based on the rule will be output as the optimal planning trajectory of the vehicle to be planned If K is greater than K0, the motion trajectory based on the 4. An autonomous driving reinforcement learning motion planning system considering uncertainty, characterized in that, an offline training module configured to train a pre-constructed multi-head value function network using a deep reinforcement learning method based on acquired historical driving data; an online planning module configured to acquire environment state information at the current moment, and obtain a data-fitted motion trajectory and a rule-based motion trajectory using a trained multi-head value function network and an automatic driving system of a vehicle to be planned respectively, and compare the two trajectories to obtain an optimal motion trajectory of the vehicle to be planned; the training of the pre-constructed multi-head value function network using the deep reinforcement learning method based on the acquired historical driving data comprises: constructing the multi-head value function network and performing random initialization; collecting driving data; ​ The collected driving data is bootstrap sampled, and based on the sampled data, each sub neural network in the multi-head value function network is trained respectively using a deep reinforcement learning method; The environment state information at the current time is obtained, and the motion trajectory based on data fitting and the motion trajectory based on rules are obtained respectively by using the trained multi-head value function network and the automatic driving system of the vehicle to be planned, and after comparison, the optimal motion trajectory of the vehicle to be planned is obtained, including: Obtaining the environment state information at the current time; Based on the environment state information at the current time, the global trajectory planning result is obtained by using the automatic driving system of the vehicle to be planned, and the motion trajectory based on rules is calculated based on the cost function; Based on the environment state information at the current time, the motion trajectory based on data fitting is calculated by using the multi-head value function network; Based on the bootstrap uncertainty calculation, the optimal motion trajectory is determined from the motion trajectory based on rules and the motion trajectory based on data fitting and outputted; The environment state information at the current time is obtained, and the motion trajectory based on data fitting and the motion trajectory based on rules are obtained respectively by using the trained multi-head value function network and the automatic driving system of the vehicle to be planned, and after comparison, the optimal motion trajectory of the vehicle to be planned is obtained, including: Based on the current moment environment state information, according to the global trajectory planning result of automatic driving, m candidate trajectories of motion planning are generated ; In each planning cycle, the trajectory with the lowest cost from the candidate trajectories is selected as the rule-based motion trajectory using a cost function ; The environment state information at the current time is obtained, and the motion trajectory based on data fitting and the motion trajectory based on rules are obtained respectively by using the trained multi-head value function network and the automatic driving system of the vehicle to be planned, and after comparison, the optimal motion trajectory of the vehicle to be planned is obtained, including: ① the current environment state information s and the rule-based motion trajectory as the input of the multi-head value function network, to obtain the rule-based motion trajectory corresponding to a series of decision values ; ② randomly sampling one trajectory from m candidate trajectories The expected decision action a is obtained from the m candidate trajectories, and the expected decision action a and the current environment state information s are input into the multi-head value function network to obtain a series of decision values corresponding to the expected decision action a: ; ③ compare the series of decision values corresponding to the desired decision action a with the series of decision values corresponding to the rule-based motion trajectory , to obtain the number K of decision values in the desired decision action a that satisfy ; ; ④ Repeat steps ②-③, and take the expected decision action with the largest number of conditions met as the motion trajectory based on data fitting .

5. A processing device comprising at least a processor and a memory having stored thereon a computer program, characterized in that, The processor runs the computer program to implement the steps of the automatic driving reinforcement learning motion planning method considering uncertainty according to any one of claims 1 to 3.

6. A computer storage medium, characterized in that, There is computer readable instructions stored thereon, which can be executed by the processor to implement the steps of the automatic driving reinforcement learning motion planning method considering uncertainty according to any one of claims 1 to 3.

Images