Route planning method and device, electronic equipment and storage medium

Abstract

The invention provides a route planning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a state parameter of a vehicle and a target position parameter of a destination of the vehicle, and determining a reward function of an estimation neural network and an action space of the vehicle according to the state parameter and the target position parameter; inputting the state parameters and the action space into an estimation neural network to obtain a plurality of estimation values corresponding to the plurality of space directions; obtaining the number of repeated selection times of each space direction in the plurality of space directions in the estimation neural network, and performing parameter adjustment on the plurality of estimation values according to the number of repeated selection times to obtain a plurality of target values corresponding to the plurality of space directions; and if there is a target value greater than the threshold value in the plurality of target values, determining a target space direction corresponding to the target value greater than the threshold value, and performing route planning according to the target space direction and the speed parameter. The method can reduce the times of repeatedly adopting actions, and prevents the training of the estimated neural network from tending to a local optimal solution.

Description

Technical Field

[0001] The present application relates to the field of intelligent optimization technology, and more specifically, to a route planning method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, the application of assisted driving or intelligent driving of vehicles in people's daily life is becoming more and more extensive. The path planning problem of vehicles in assisted driving functions is also receiving more and more attention. Path planning technology can help vehicles avoid obstacles and plan an optimal movement route from the starting point to the target point under the condition of referring to a certain indicator. The current research on intelligent driving route planning is generally designed based on reinforcement learning algorithms. Reinforcement learning optimizes the reward equation through its own self-learning ability, and continuously iterates the neural network parameters until the correct intelligent driving route is output. However, in the existing deep reinforcement learning technology solutions, the neural network parameters used are different, the reinforcement learning algorithms selected are different, and the reward equations set are also different. The final effects of the models trained by different algorithm frameworks and reward equations are uneven. Therefore, how to avoid the training of the route planning neural network from tending to the local optimal solution and improve the learning efficiency of the neural network has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the embodiments of the present application propose a route planning method, device, electronic device and storage medium to improve the above-mentioned problems.

[0004] According to a first aspect of an embodiment of the present application, a route planning method is provided, the method comprising: obtaining state parameters of a vehicle and target position parameters of a destination of the vehicle, and determining a reward function of an estimation neural network and an action space of the vehicle based on the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions corresponding to different driving routes of the vehicle and a speed parameter of the vehicle; inputting the state parameters and the action space of the vehicle into an estimation neural network, obtaining multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network based on the reward function and the state parameters and the action space, the multiple estimated values ​​representing the confidence of selecting the corresponding multiple spatial directions for route planning of the vehicle; obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network for iteration, and adjusting the parameters of the multiple estimated values ​​based on the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions; if there is a target value greater than a threshold value among the multiple target values, determining the target spatial direction corresponding to the target value greater than the threshold value, and planning the vehicle route based on the target spatial direction and the speed parameter.

[0005] According to a second aspect of an embodiment of the present application, a route planning device is provided, the device comprising: a determination module for obtaining state parameters of a vehicle and target position parameters of a destination of the vehicle, and determining a reward function and an action space based on the state parameters and the target position parameters, wherein the action space includes multiple spatial directions and speed parameters; an estimated value determination module for inputting the state parameters, the action space and the reward function into an estimation neural network to obtain multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network; a target value determination module for obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and performing parameter adjustment on the multiple estimated values ​​based on the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions; a first route planning module for determining the target spatial direction corresponding to the target value greater than a threshold value if there is a target value greater than a threshold value among the multiple target values, and performing route planning based on the target spatial direction and the speed parameter.

[0006] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the route planning method as described above is implemented.

[0007] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the route planning method as described above is implemented.

[0008] In the scheme of the present application, the reward function and action space of the estimation neural network are first determined according to the state parameters of the vehicle and the target position parameters of the destination of the vehicle, and then the estimation neural network can determine multiple estimated values ​​corresponding to multiple spatial directions according to the state parameters, action space and reward function, and then determine multiple target values ​​corresponding to multiple control directions according to the number of times each spatial direction in the multiple spatial directions is repeatedly selected in the estimation neural network, thereby determining a target value greater than a threshold value among the multiple target values, and planning the route of the vehicle according to the target spatial direction and speed parameters corresponding to the target value greater than the threshold value. The present application determines the target value based on the number of times each spatial direction is repeatedly selected, thereby reducing the number of repeated actions, trying more possibilities, increasing randomness, avoiding the estimation neural network training from tending to local optimal solutions, improving the learning efficiency of the estimation neural network, and increasing the robustness of the estimation neural network, so that the vehicle has better stability in driving along the route planned based on the target spatial direction.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 It is a flowchart of a route planning method according to an embodiment of the present application.

[0012] Figure 2 It is a flowchart of a route planning method according to another embodiment of the present application.

[0013] Figure 3 It is a flowchart of a route planning method according to another embodiment of the present application.

[0014] Figure 4 It is a flowchart of a route planning method according to another embodiment of the present application.

[0015] Figure 5 It is a flowchart showing the specific steps of step 440 according to an embodiment of the present application.

[0016] Figure 6 It is a block diagram of a route planning device according to an embodiment of the present application.

[0017] Figure 7 It is a hardware structure diagram of an electronic device according to an embodiment of the present application.

[0018] The above-mentioned drawings have shown clear embodiments of the present invention, which will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention to computer technicians in this field through specific embodiments. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0023] See also Figure 1 , Figure 1 The route planning method provided by an embodiment of the present application is shown. In a specific embodiment, the route planning method can be applied to Figure 6 The route planning device 500 and the electronic device 600 ( Figure 7 ). The specific process of this embodiment will be described below. Of course, it can be understood that the method can be executed by a cloud server with computing processing capabilities. Figure 1 The process shown is described in detail, and the route planning method may specifically include the following steps:

[0024] Step 110, obtaining state parameters of the vehicle and target position parameters of the destination of the vehicle, and determining a reward function of an estimated neural network and an action space of the vehicle based on the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions corresponding to different driving routes of the vehicle and speed parameters of the vehicle.

[0025] As a method, state parameters refer to characteristic state information of a vehicle during driving, which may include state variables such as the current position information of the vehicle and the deflection angle of the destination. The state parameters of the vehicle may be recorded each time the vehicle performs an action, and the state space of the vehicle may be determined based on the recorded multiple state variables as a coordinate system. Optionally, the state variables of the vehicle may be recorded according to a preset period to determine the state parameters of the vehicle or the state space of the vehicle. Optionally, the state parameters of the vehicle may be obtained through the vehicle's assisted driving system, or may be obtained based on the vehicle's electronic control unit.

[0026] As a way, the action space refers to the set of all valid operations or choices available when the vehicle interacts with the environment in reinforcement learning. In this application, the action space refers to the set of data such as all driving routes or spatial directions that can reach the destination when the vehicle travels to the destination. Optionally, the action space can be a discrete action space or a continuous action space, which can be selected according to actual needs. Optionally, the speed parameter in the action space can be a constant speed parameter. Optionally, the vehicle's auxiliary intelligent driving system can be used to plan the vehicle's action space based on the vehicle's state parameters and the target position parameters of the destination.

[0027] As a method, the reward function is used to assist the intelligent driving system in route planning. The quality of route planning and the convergence effect of the neural network of the assisted intelligent driving system depend on whether the correct reward function is set. In reinforcement learning, the vehicle makes an action (which can be understood as movement) to interact with the environment, and the reward value is determined according to the result of the interaction, and then the network parameters of the neural network of the assisted intelligent driving system are updated according to the reward value. Optionally, the reward function may include positive reward values, negative reward values ​​and other reward values, wherein the positive reward value may be the reward value for the vehicle to reach the destination, the negative reward value may be the reward value for the vehicle to deviate from the destination, and other reward values ​​may be the corresponding reward values ​​of the vehicle in the process of searching for the destination. The other reward values ​​can be determined according to the actual driving state of the vehicle. The positive reward value and the negative reward value are fixed values ​​and can be set according to actual needs. No specific limitation is made here.

[0028] Step 120, input the state parameters and the action space of the vehicle into an estimation neural network, and obtain a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to the reward function and the state parameters and the action space, wherein the plurality of estimation values ​​represent the confidence of selecting the corresponding plurality of spatial directions for route planning of the vehicle.

[0029] As a method, when planning a route for a vehicle according to a destination, an estimated value (Q value) corresponding to each spatial direction in the action space can be determined by a Q learning algorithm, and then the target spatial direction of the vehicle is determined according to the estimated value corresponding to each spatial direction, and the route is planned according to the target spatial direction, wherein the estimated value is a plurality of estimated values ​​corresponding to a plurality of spatial directions in the action space determined by the algorithm's estimation neural network according to the vehicle's state parameters, action space, and reward function. The estimation neural network can be a recurrent neural network, a convolutional neural network, etc., and the type and structure of the estimation neural network can be set according to actual needs.

[0030] Step 130, obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and adjusting the parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions.

[0031] As a method, the number of times the vehicle selects each spatial direction in the action space under corresponding state parameters can be determined to determine the selection probability corresponding to each spatial direction, and then the parameters of multiple estimated values ​​can be adjusted according to the selection probability, so as to reduce the number of repeated spatial directions or repeated actions in the action space, increase the probability of other spatial directions and actions being adopted, improve the convergence speed of the network, and avoid convergence to the local optimum.

[0032] Optionally, a selection flag may be set for each spatial direction in the action space when the spatial direction is selected, and a different selection flag may be set for each selected spatial placement. The number of selection flags for each spatial direction may be used to obtain the number of repeated selections for each spatial direction in the estimation neural network. Optionally, a corresponding data set may be set for each spatial direction, the data set including a counter and the spatial direction. When the spatial direction is selected, the count of the corresponding counter is incremented by one. The number of repeated selections for each spatial direction in the estimation neural network may be obtained by determining the count value of the counter for each spatial direction.

[0033] Optionally, while determining multiple target values, the estimated neural network determines a reward value of the estimated neural network based on a reward function and state parameters, and updates network parameters of the estimated neural network based on the reward value and the multiple target values.

[0034] Step 140: If there is a target value greater than a threshold value among the multiple target values, determine the target space direction corresponding to the target value greater than the threshold value, and plan a route for the vehicle according to the target space direction and the speed parameter.

[0035] As a method, when it is determined that there is a target value greater than a threshold among multiple target values, it can be determined that the estimated neural network converges, and then it can be determined that route planning can be performed based on the target space direction corresponding to the target value greater than the threshold.

[0036] Optionally, the determined target spatial direction and speed parameters may be input into the vehicle's assisted intelligent driving system, which determines the vehicle's route and driving data based on the speed parameters and target spatial direction to the destination according to the target spatial direction and speed parameters, wherein the driving data includes steering wheel offset, etc.

[0037] In an embodiment of the present application, the reward function and action space of the estimation neural network are first determined based on the state parameters of the vehicle and the target position parameters of the destination of the vehicle, and then the estimation neural network can determine multiple estimated values ​​corresponding to multiple spatial directions based on the state parameters, action space and reward function, and then determine multiple target values ​​corresponding to multiple control directions based on the number of times each spatial direction in the multiple spatial directions is repeatedly selected in the estimation neural network, thereby determining a target value greater than a threshold value among the multiple target values, and planning the route of the vehicle based on the target spatial direction and speed parameters corresponding to the target value greater than the threshold value. The present application determines the target value based on the number of times each spatial direction is repeatedly selected, thereby reducing the number of repeated actions, trying more possibilities, increasing randomness, avoiding the estimation neural network training from tending to local optimal solutions, improving the learning efficiency of the estimation neural network, and increasing the robustness of the estimation neural network, so that the vehicle has better stability in driving along the route planned based on the target spatial direction.

[0038] See also Figure 2 , Figure 2 The route planning method provided by an embodiment of the present application is shown below. Figure 2 The process shown is described in detail, and the route planning method may specifically include the following steps:

[0039] Step 210, obtain the state parameters of the vehicle and the target position parameters of the destination of the vehicle, and determine the reward function of the estimated neural network and the action space of the vehicle based on the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions corresponding to different driving routes of the vehicle and the speed parameters of the vehicle.

[0040] Step 220, input the state parameters and the action space of the vehicle into an estimation neural network, and obtain a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to the reward function and the state parameters and the action space, wherein the plurality of estimation values ​​represent the confidence of selecting the corresponding plurality of spatial directions for route planning of the vehicle.

[0041] Step 230, obtain the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and adjust the parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions.

[0042] The specific step descriptions of step 210 to step 230 may refer to step 110 to step 130, which will not be repeated here.

[0043] Step 240, input the multiple target values ​​into the target neural network to obtain the loss value output by the target neural network.

[0044] As a method, after determining multiple target values, it is necessary to determine whether the estimated neural network has converged. The loss value can be calculated in conjunction with the target neural network, and whether the estimated neural network has converged is determined based on the loss value. The target neural network has the same network structure and network parameters as the estimated neural network. In order to avoid the situation where the estimated neural network calculates the loss value based on the estimated value, which may cause the estimated neural network to over-converge, the loss value can be calculated by coordinating with the target neural network having the same structure and the same network parameters, so that whether the estimated neural network has converged can be determined based on the calculated loss value. That is, if the loss value indicates that the target neural network has converged, it can be determined that the estimated neural network has converged.

[0045] As another method, a target neural network having the same structure and network parameters as the estimated neural network can be set, and the output value of the target neural network can be determined by the target neural network, and then the loss value can be determined according to the output value of the target neural network and the output value of the estimated neural network. Optionally, the formula: Loss = (Q (s t ,a t ;φ)―targetQ t ) 2 , where Loss is the loss value, Q(s t ,a t ; φ) is the output value of the target neural network, targetQ t To estimate the output value of the neural network, targetQ t =r t+1 +maxQ′(s t+1 ,a t+1 ; φ ― ), rt+1 is the reward function for estimating the reward value of the neural network, maxQ′(s t+1 ,a t+1 ;Q ― ) is the maximum target value among the multiple target values ​​calculated in the estimated neural network, Q(s t ,a t ; φ) represents the t Select the spatial direction a in the state t The corresponding estimated value at s t Select the spatial direction a in the state t is the network parameter of the target neural network. Similarly, Q′(s t+1 ,a t+1 ; φ ― ) represents t+1 Select the spatial direction a in the state t+1 The corresponding estimated value, φ ― For t+1 Select the spatial direction a in the state t+1 The network parameters of the neural network are estimated.

[0046] Step 250, determining whether the loss function of the target neural network converges according to the loss value.

[0047] As a method, the loss value can be compared with the loss threshold to determine the magnitude relationship between the loss value and the loss threshold, and based on the magnitude relationship, it is determined whether the loss function of the target neural network has converged. Optionally, if the magnitude relationship indicates that the loss value is greater than or equal to the loss threshold, it is determined that the loss function of the target neural network has not converged; if the magnitude relationship indicates that the loss value is less than or equal to the loss threshold, it is determined that the loss function of the target neural network has converged.

[0048] As another way, when training the target neural network and estimating the neural network, the number of iterations of the target neural network can also be used to determine whether the target neural network has converged. Optionally, a threshold of the number of iterations can be set in advance. When the number of iterations of the target neural network is equal to the threshold of the number of iterations, the target neural network is determined to have converged. Multiple target values ​​can be determined based on the network parameters corresponding to the last iteration and the state parameters and action space of the vehicle, and route planning can be performed based on the spatial direction corresponding to the largest target value among the multiple target values.

[0049] Step 260: If the loss function converges, determine the target space direction corresponding to the target value greater than the threshold among the multiple target values, and perform route planning based on the target space direction and the speed parameter.

[0050] As a method, when it is determined that the loss function converges, it can be determined that the estimated neural network and the target neural network converge, and then it can be determined that the route planning can be performed according to the target space direction corresponding to the target value greater than the threshold.

[0051] Optionally, the determined target spatial direction and speed parameters may be input into the vehicle's assisted intelligent driving system, which determines the vehicle's route and driving data based on the speed parameters and target spatial direction to the destination according to the target spatial direction and speed parameters, wherein the driving data includes steering wheel offset, etc.

[0052] Step 270: If the loss function does not converge, the estimated neural network and the target neural network are updated, and the loss value is re-determined according to the updated estimated neural network and the updated target neural network until the loss function converges.

[0053] As a method, when it is determined that the loss function does not converge, it is necessary to update the parameters of the estimated neural network and the target neural network according to the loss value, so that the loss value can be re-determined through the estimated neural network and the target neural network after the parameter update, and whether the loss function converges is determined again, and this step is repeated until the loss function converges. Optionally, the update of the network parameters of the estimated neural network and the target neural network can be terminated when the number of iterations of the estimated neural network and the target neural network reaches an iteration number threshold.

[0054] In some embodiments, step 270 includes: if the loss function does not converge, determining a reference value based on the loss value, and determining network parameters of the target neural network and the estimated neural network based on the reference value; updating the estimated neural network and the target neural network based on the network parameters, and re-determining the loss value based on the updated estimated neural network and the updated target neural network until the loss function converges.

[0055] As a method, the gradient descent method can be used to update the network parameters of the target neural network. Specifically, the formula After the updated network parameters are determined, the updated network parameters are copied to the estimated neural network to update the network parameters of the estimated neural network, so that the loss value can be re-determined according to the estimated neural network after the updated network parameters and the target neural network until the loss function converges.

[0056] In some embodiments, the route planning method further includes: obtaining a plurality of experience samples during the training of the estimation neural network and the target neural network through original training samples, wherein the experience samples are samples that have completed training in the empirical pool of the target neural network; and mixing the plurality of experience samples with the original training samples as training data for the estimation neural network and the target neural network.

[0057] As a method, there is an experience pool for storing data in the target neural network, which can store the data that have been trained in the target neural network.

[0058] As a method, there is an experience pool for storing data in the target neural network, and the experience pool can store training data (i.e., experience samples) that have completed training in the target neural network. Optionally, after the target neural network is trained according to the original training samples, a trained flag is set for the original training samples, and the original training samples with the added flag are stored in the experience pool to obtain experience samples. Optionally, experience samples can be obtained in the experience pool with a trained flag. Optionally, experience samples can be randomly and uniformly selected in the experience pool. Optionally, mixing multiple experience samples with the original training samples as training data for the estimated neural network and the target neural network can destroy the correlation of the training samples, thereby making the training of the estimated neural network and the target neural network more accurate.

[0059] In this embodiment, a loss value is determined by coordinating with a target neural network, and then whether the target neural network converges is determined based on the loss value, wherein the loss value is determined based on the target value, that is, whether the target neural network converges is determined based on the target value. When the target neural network converges, route planning is performed according to the determined target value. When the target neural network does not converge, network parameters of the target neural network and the estimated neural network are updated according to the target value until the target neural network and the estimated neural network converge, thereby improving the speed of neural network convergence and optimizing the effect of path planning.

[0060] See also Figure 3 , Figure 3 The route planning method provided by an embodiment of the present application is shown below. Figure 3 The process shown is described in detail, and the route planning method may specifically include the following steps:

[0061] Step 310, obtaining state parameters of the vehicle and target position parameters of the destination of the vehicle, and determining a reward function of the estimated neural network and an action space of the vehicle based on the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions corresponding to different driving routes of the vehicle and speed parameters of the vehicle.

[0062] Step 320, input the state parameters and the action space into an estimation neural network, and obtain a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to the reward function and the state parameters and the action space, wherein the plurality of estimation values ​​represent the confidence of selecting the corresponding plurality of spatial directions for route planning of the vehicle.

[0063] Step 330, obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and determining the selection probabilities corresponding to each of the multiple spatial directions according to the number of repeated selections.

[0064] As a method, after obtaining the number of repeated selections of each control direction in the estimation neural network in multiple spatial directions, the formula To determine the selection probability corresponding to each of the multiple spatial directions, where P(s,a) is the selection probability of spatial direction a among the multiple spatial directions when the state parameter is s, and M is the number of repeated selections of spatial direction a among the multiple spatial directions in the estimated neural network.

[0065] Step 340: Determine a plurality of expected values ​​corresponding to the plurality of spatial directions based on the selection probabilities corresponding to the plurality of spatial directions and the plurality of estimated values.

[0066] As a method, the expected value is used to indicate the value corresponding to each of the multiple spatial directions. Optionally, multiple expected values ​​corresponding to the multiple spatial directions can be determined by E(s,a)=Q(s,a)*P(s,a), where E(s,a) is the expected value corresponding to the a spatial direction among the multiple spatial directions, P(s,a) is the selection probability of the a spatial direction among the multiple spatial directions when the state parameter is s, and Q(s,a) is the estimated value of the a spatial direction among the multiple spatial directions when the state parameter is s.

[0067] Step 350, obtaining weights corresponding to the multiple estimated values, and determining multiple target values ​​corresponding to the multiple spatial directions based on the multiple expected values, the multiple estimated values ​​and the weights corresponding to the multiple estimated values.

[0068] As one method, the weight corresponding to the estimated value can be obtained first. The weight is a preset weight value, which can be set according to actual needs. The multiple target values ​​corresponding to the multiple spatial directions are determined based on the weight, multiple estimated values ​​and the selection probabilities corresponding to the multiple spatial directions. Optionally, the target value can be calculated by the formula Q″(s,a)=λQ(s,a)+(1-λ)E(s,a), where Q″(s,a) is the target value of spatial direction a among the multiple spatial directions when the state parameter is s, λ is the weight corresponding to the estimated value, which is a fixed value, and the weights corresponding to different estimated values ​​are the same, Q(s,a) is the estimated value of spatial direction a among the multiple spatial directions when the state parameter is s, and E(s,a) is the expected value corresponding to spatial direction a among the multiple spatial directions.

[0069] Step 360: If there is a target value greater than a threshold value among the multiple target values, determine the target space direction corresponding to the target value greater than the threshold value, and plan a route for the vehicle based on the target space direction and the speed parameter.

[0070] The specific step descriptions of step 310 to step 320 and step 360 may refer to step 110 to step 120 and step 140, which will not be repeated here.

[0071] In this embodiment, the selection probability corresponding to each spatial direction is determined according to the number of times each spatial direction in the action space is repeatedly selected, and then multiple expected values ​​corresponding to multiple spatial directions are determined based on the selection probability and multiple estimated values, and then multiple target values ​​are determined according to the weights corresponding to the multiple estimated values ​​and the multiple expected values, so that route planning can be performed according to the multiple target values. This embodiment reduces the number of times the vehicle enters a repeated state during training by modifying the calculation method of the Q value, so that it can better explore new states, while improving the convergence speed of the neural network and optimizing the effect of path planning.

[0072] See also Figure 4 , Figure 4 The route planning method provided by an embodiment of the present application is shown below. Figure 4 The process shown is described in detail, the state parameter includes the current position parameter; the route planning method may specifically include the following steps:

[0073] Step 410: Determine the distance between the vehicle and the destination and the angle between the direction vector of the vehicle and the destination according to the current position parameter and the target position parameter.

[0074] As a way, in order to avoid the problem of sparse rewards for the traditional reward function to the God General Network, the reward function of the vehicle during the movement can be determined according to the actual position of the vehicle and the target position of the destination, and the corresponding feedback value can be specifically determined according to the distance between the vehicle and the destination and the angle between the direction vector of the vehicle and the destination, so that the corresponding reward feedback value can be obtained during the movement of the vehicle, thereby improving the convergence speed of the estimated neural network and making the convergence direction approach the optimal solution.

[0075] Optionally, the distance between the vehicle and the destination and the angle between the direction vectors of the vehicle and the destination can be calculated based on the coordinates of the vehicle in the current position parameters of the vehicle and the coordinates of the destination in the target position of the destination. Optionally, the formula To determine the distance between the vehicle and the destination, (x1, y1, y1) is the current position parameter of the vehicle, (x2, y2, y2) is the target position parameter, and d is the distance between the vehicle and the destination. This can be determined by the formula To determine the direction vector angle between the vehicle and the destination, where cosθ is the direction vector angle between the vehicle and the destination.

[0076] Step 420: Determine a first reward feedback value according to the distance and the angle between the direction vectors.

[0077] As a method, a hyperparameter may be obtained first, which may be preset according to the demand, and then the first reward feedback value may be determined according to the direction vector angle between the vehicle and the destination, the distance between the vehicle and the destination, and the hyperparameter. Optionally, the first reward feedback value may be obtained by To determine, where R N is the first reward feedback value, K is a hyperparameter, cosθ N is the direction vector angle between the vehicle and the destination, d N is the distance between the vehicle and the destination.

[0078] Step 430, obtaining a positive reward feedback value and a negative reward feedback value, and determining the reward function estimation neural network according to the positive reward feedback value, the negative reward feedback value and the first reward feedback value, and determining the action space according to the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions corresponding to different driving routes of the vehicle and the speed parameters of the vehicle.

[0079] As a method, the reward feedback value indicates the quality of a certain action taken by the vehicle in a certain state, and the network parameters of the neural network are adjusted in a certain direction based on the reward feedback value, so that the neural network can be iterated again according to the adjusted network parameters, so that the convergence speed of the neural network is accelerated. Optionally, the positive reward feedback value refers to the reward value when the vehicle arrives at the destination, and the negative reward feedback value refers to the reward feedback value corresponding to the collision of the vehicle in the process of reaching the destination or the time length of the vehicle to reach the destination is greater than the time length threshold. In order to enable the vehicle to obtain appropriate reward feedback in the process of searching for the destination, reward feedback can be performed according to the first reward feedback value.

[0080] Optionally, a reward function may be determined based on the positive reward feedback value, the negative reward feedback value, and the first reward feedback value. The reward function may be: Among them, C G is the positive reward feedback value, C B is the negative reward feedback value, is the first reward feedback value. The setting of hyperparameters is continuously adjusted based on multiple experimental results and is set to 1 / 4C here. G Positive rewards need to be adjusted independently according to actual conditions in different environments.

[0081] Step 440, input the state parameters and the action space of the vehicle into the estimation neural network, and obtain multiple estimation values ​​corresponding to the multiple spatial directions output by the estimation neural network according to the reward function and the state parameters and the action space, wherein the multiple estimation values ​​represent the confidence of selecting the corresponding multiple spatial directions for route planning of the vehicle.

[0082] As a method, the reward feedback value of the estimated neural network can be determined based on whether the vehicle reaches the destination, whether the vehicle collides while searching for the destination, the specific circumstances of the vehicle reaching the destination, and the reward function.

[0083] In some embodiments, Figure 5 As shown, step 440 includes:

[0084] Step 441, input the state parameters and the action space into the estimation neural network to determine the action relationship between the current position and the destination of the vehicle.

[0085] As a method, the estimation neural network determines whether the vehicle is in the process of moving according to the state parameters of the vehicle, and determines whether the vehicle has reached the destination according to the action space and the state parameters, so as to determine the action relationship between the vehicle at the current position and the destination. Optionally, the action relationship between the vehicle at the current position and the destination can be that the vehicle has accurately reached the destination, the vehicle is far away from the destination, the vehicle is close to the destination, and the vehicle collides before reaching the destination, etc. The action relationship can be set according to actual needs.

[0086] Step 442, if the action relationship indicates that the vehicle arrives at the destination, the parameters of the estimation neural network are updated according to the positive reward feedback value corresponding to the reward function, and the estimation neural network after the parameter update outputs multiple estimation values ​​corresponding to the multiple spatial directions based on the state parameters and the action space of the vehicle.

[0087] As a method, when the action relationship indicates that the vehicle has reached the destination, a positive reward feedback value is given to the estimation neural network, so that when the estimation neural network determines the estimation value based on other spatial directions, the estimation neural network parameters can be updated according to the positive reward feedback value, so as to obtain multiple estimation values ​​corresponding to multiple spatial directions output by the estimation neural network.

[0088] Step 443, if the action relationship indicates that the vehicle collides with or exceeds the time limit when reaching the destination, the parameters of the estimation neural network are updated according to the negative reward feedback value corresponding to the reward function, and the estimation neural network after the parameter update outputs multiple estimation values ​​corresponding to the multiple spatial directions based on the state parameters and the action space of the vehicle.

[0089] As a method, when the action relationship indicates that a collision occurs during the process of the vehicle reaching the destination, or the time to reach the destination is greater than a time threshold (i.e., the destination is reached within the time limit), a negative reward feedback value is given to the estimation neural network, so that when the estimation neural network determines the estimation value based on other spatial directions, the parameters of the estimation neural network can be updated according to the negative reward feedback value, so as to obtain multiple estimation values ​​corresponding to multiple spatial directions output by the estimation neural network.

[0090] Step 444, if the action relationship indicates that the vehicle is approaching the destination or moving away from the destination, the parameters of the estimation neural network are updated according to the first reward feedback value corresponding to the reward function, and the estimation neural network after the parameter update outputs multiple estimation values ​​corresponding to the multiple spatial directions based on the state parameters and the action space of the vehicle.

[0091] As a method, when the action relationship indicates that the vehicle is approaching or moving away from the destination, a reward feedback value is given to the estimation neural network according to the first reward feedback value, so that when the estimation neural network determines the estimated value based on other spatial directions, the estimation neural network can be updated according to the first reward feedback value, so as to obtain multiple estimated values ​​corresponding to multiple spatial directions output by the estimation neural network. Optionally, when the first reward feedback value is greater than zero, it can be determined that the current direction vector of the vehicle and the direction vector of the destination are at an acute angle, that is, the vehicle is moving in the direction of the destination, that is, the vehicle is approaching the destination, and a negative reward is appropriately given to increase the probability of the vehicle going to the destination; when the first reward feedback value is less than zero, it can be determined that the current direction vector of the vehicle and the direction vector of the destination are at an obtuse angle, that is, the vehicle is moving away from the destination, that is, the vehicle is moving away from the destination, and a negative reward is appropriately given to reduce the probability of the vehicle moving away from the destination.

[0092] Please continue reading Figure 4 , step 450, obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and adjusting the parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions.

[0093] Step 460: If there is a target value greater than a threshold value among the multiple target values, determine the target space direction corresponding to the target value greater than the threshold value, and plan a route for the vehicle based on the target space direction and the speed parameter.

[0094] The specific step descriptions of step 450 to step 460 can refer to step 130 to step 140, which will not be repeated here.

[0095] In this embodiment, the action relationship between the current position of the vehicle and the destination is determined according to the current position of the vehicle and the destination, and then the reward value of the estimated neural network can be determined according to the action relationship and the reward function, so as to facilitate updating the network parameters of the estimated neural network according to the reward value. By resetting the reward function to reward the estimated neural network, the problem of sparse rewards is avoided, so that the vehicle can provide real-time feedback during the entire training process, correct actions in time, and optimize the planned route.

[0096] Figure 6 is a block diagram of a route planning device according to an embodiment of the present application, such as Figure 6 As shown, the route planning device 500 includes: a determination module 510 , an estimated value determination module 520 , a target value determination module 530 and a first route planning module 540 .

[0097] A determination module 510 is used to obtain the state parameters of the vehicle and the target position parameters of the destination of the vehicle, and determine the reward function and the action space based on the state parameters and the target position parameters, wherein the action space includes multiple spatial directions and speed parameters; an estimated value determination module 520 is used to input the state parameters, the action space and the reward function into the estimation neural network to obtain multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network; a target value determination module 530 is used to obtain the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and adjust the parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions; a first route planning module 540 is used to determine the target spatial direction corresponding to the target value greater than the threshold if there is a target value greater than the threshold among the multiple target values, and perform route planning based on the target spatial direction and the speed parameter.

[0098] In some embodiments, the route planning device 500 also includes: a loss value determination module, which is used to input the multiple target values ​​into the target neural network to obtain the loss value output by the target neural network; a judgment module, which is used to determine whether the loss function of the target neural network converges according to the loss value; a second route planning module, which is used to determine the target space direction corresponding to the target value greater than the threshold among the multiple target values ​​if the loss function converges, and perform route planning according to the target space direction and the speed parameter; or an update module, which is used to update the estimation neural network and the target neural network if the loss function does not converge, and re-determine the loss value according to the updated estimation neural network and the updated target neural network until the loss function converges.

[0099] In some embodiments, the update module includes: a network parameter determination unit, which is used to determine a reference value based on the loss value if the loss function does not converge, and determine the network parameters of the target neural network and the estimated neural network based on the reference value; an update unit, which is used to update the estimated neural network and the target neural network based on the network parameters, and redetermine the loss value based on the updated estimated neural network and the updated target neural network until the loss function converges.

[0100] In some embodiments, the update module also includes: an experience sample acquisition unit, used to acquire multiple experience samples during the training process of the estimated neural network and the target neural network through original training samples, and the experience samples are samples that have completed training in the empirical pool of the target neural network; a training data determination unit, used to mix the multiple experience samples with the original training samples as training data for the estimated neural network and the target neural network.

[0101] In some embodiments, the target value determination module 530 includes: a selection probability determination unit, used to obtain the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and determine the selection probabilities corresponding to each of the multiple spatial directions based on the number of repeated selections; an expected value determination unit, used to determine a plurality of expected values ​​corresponding to the multiple spatial directions based on the selection probabilities corresponding to each of the multiple spatial directions and the multiple estimated values; a target value determination unit, used to obtain weights corresponding to the multiple estimated values, and determine a plurality of target values ​​corresponding to the multiple spatial directions based on the multiple expected values, the multiple estimated values ​​and the weights corresponding to the multiple estimated values.

[0102] In some embodiments, the state parameter includes a current position parameter; the determination module 510 includes: a determination unit, used to determine the distance between the vehicle and the destination and the direction vector angle between the vehicle and the destination according to the current position parameter and the target position parameter; a first reward feedback value determination unit, used to determine the first reward feedback value according to the distance and the direction vector angle; a reward function determination unit, used to obtain a positive reward feedback value and a negative reward feedback value, and determine the reward function according to the positive reward feedback value, the negative reward feedback value and the first reward feedback value.

[0103] In some embodiments, the estimated value determination module 520 includes: an action relationship determination unit, which is used to input the state parameters, the action space and the reward function into the estimation neural network to determine the action relationship between the vehicle at the current position and the destination; an estimated value first determination unit, which is used to update the parameters of the estimation neural network according to the positive reward feedback value if the action relationship indicates that the vehicle has arrived at the destination, so as to obtain multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network; or an estimated value second determination unit, which is used to update the parameters of the estimation neural network according to the negative reward feedback value if the action relationship indicates that the vehicle collides or reaches the destination in the process of reaching the destination, so as to obtain multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network; or an estimated value third determination unit, which is used to update the parameters of the estimation neural network according to the first reward feedback value if the action relationship indicates that the vehicle is close to the destination or far away from the destination, so as to obtain multiple estimated values ​​corresponding to the multiple spatial directions output by the estimation neural network.

[0104] According to one aspect of the embodiments of the present application, an electronic device is also provided, such as Figure 7 As shown, the vehicle 600 includes a processor 610 and one or more memories 620. The one or more memories 620 are used to store program instructions executed by the processor 610. When the processor 610 executes the program instructions, the above-mentioned route planning method is implemented.

[0105] Furthermore, the processor 610 may include one or more processing cores. The processor 610 runs or executes instructions, programs, code sets or instruction sets stored in the memory 620, and calls data stored in the memory 620. Optionally, the processor 610 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 610 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a communication chip.

[0106] According to one aspect of the present application, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0107] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0108] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0109] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0111] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A route planning method, characterized in that: The method comprises: Acquire state parameters of a vehicle and target position parameters of a destination of the vehicle, and determine a reward function of an estimation neural network and an action space of the vehicle according to the state parameters and the target position parameters, wherein the action space of the vehicle includes a plurality of spatial directions corresponding to different driving routes of the vehicle and a speed parameter of the vehicle; Inputting the state parameters and the action space of the vehicle into an estimation neural network, obtaining a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to a reward function and the state parameters and the action space, wherein the plurality of estimation values ​​represent confidence levels in selecting the corresponding plurality of spatial directions for route planning of the vehicle; Obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and adjusting parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions; If there is a target value greater than a threshold value among the multiple target values, the target space direction corresponding to the target value greater than the threshold value is determined, and the route of the vehicle is planned according to the target space direction and the speed parameter.

2. The method according to claim 1, characterized in that After obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and performing parameter adjustment on the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions, the method further includes: Inputting the multiple target values ​​into a target neural network to obtain a loss value output by the target neural network; Determining whether the loss function of the target neural network converges according to the loss value; If the loss function converges, determining the target space direction corresponding to the target value greater than the threshold among the multiple target values, and performing route planning according to the target space direction and the speed parameter; or If the loss function does not converge, the estimated neural network and the target neural network are updated, and the loss value is re-determined according to the updated estimated neural network and the updated target neural network until the loss function converges.

3. The method according to claim 2, characterized in that If the loss function does not converge, the estimated neural network and the target neural network are updated, and the loss value is re-determined according to the updated estimated neural network and the updated target neural network until the loss function converges, including: If the loss function does not converge, determining a reference value according to the loss value, and determining network parameters of the target neural network and the estimation neural network according to the reference value; The estimated neural network and the target neural network are updated according to the network parameters, and the loss value is re-determined according to the updated estimated neural network and the updated target neural network until the loss function converges.

4. The method according to claim 2, characterized in that: The method further comprises: In the process of training the estimation neural network and the target neural network through the original training samples, a plurality of experience samples are obtained, where the experience samples are samples in the experience pool of the target neural network that have completed training; The plurality of experience samples are mixed with the original training samples as training data for the estimation neural network and the target neural network.

5. The method according to claim 1, characterized in that The obtaining of the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and performing parameter adjustment on the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions, includes: Obtaining the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and determining the selection probabilities corresponding to each of the multiple spatial directions according to the number of repeated selections; Determining a plurality of expected values ​​corresponding to the plurality of spatial directions according to the selection probabilities corresponding to the plurality of spatial directions and the plurality of estimated values; The weights corresponding to the multiple estimated values ​​are obtained, and the multiple target values ​​corresponding to the multiple spatial directions are determined according to the multiple expected values, the multiple estimated values, and the weights corresponding to the multiple estimated values.

6. The method according to claim 1, characterized in that The state parameter includes a current position parameter; and determining a reward function of the estimation neural network according to the state parameter and the target position parameter includes: Determining the distance between the vehicle and the destination and the direction vector angle between the vehicle and the destination according to the current position parameter and the target position parameter; Determine a first reward feedback value according to the distance and the angle between the direction vector; A positive reward feedback value and a negative reward feedback value are obtained, and a reward function of the estimation neural network is determined according to the positive reward feedback value, the negative reward feedback value and the first reward feedback value.

7. The method according to claim 6, characterized in that: The step of inputting the state parameter and the action space of the vehicle into an estimation neural network to obtain a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to a reward function and the state parameter and the action space comprises: Inputting the state parameter and the action space into the estimation neural network to determine the action relationship between the current position and the destination of the vehicle; If the action relationship indicates that the vehicle has arrived at the destination, updating the parameters of the estimation neural network according to the positive reward feedback value corresponding to the reward function, and outputting multiple estimation values ​​corresponding to the multiple spatial directions based on the state parameters and the action space of the vehicle according to the parameter-updated estimation neural network; or If the action relationship indicates that the vehicle collides with or reaches the destination in time during the process of reaching the destination, the estimation neural network is parameter updated according to the negative reward feedback value corresponding to the reward function, and multiple estimation values ​​corresponding to the multiple spatial directions output by the estimation neural network after the parameter update based on the state parameters and the action space of the vehicle are output; or If the action relationship indicates that the vehicle is approaching the destination or moving away from the destination, the parameters of the estimation neural network are updated according to the first reward feedback value corresponding to the reward function, and the estimation neural network after the parameter update outputs multiple estimation values ​​corresponding to the multiple spatial directions based on the state parameters and the action space of the vehicle.

8. A route planning device, characterized in that: The device comprises: A determination module, used to obtain state parameters of the vehicle and target position parameters of the destination of the vehicle, and determine a reward function of the estimation neural network and an action space of the vehicle according to the state parameters and the target position parameters, wherein the action space of the vehicle includes multiple spatial directions and speed parameters corresponding to different driving routes of the vehicle; An estimation value determination module, configured to input the state parameters and the action space of the vehicle into an estimation neural network, and obtain a plurality of estimation values ​​corresponding to the plurality of spatial directions output by the estimation neural network according to a reward function and the state parameters and the action space, wherein the plurality of estimation values ​​represent confidence levels in selecting the plurality of spatial directions corresponding to route planning for the vehicle; a target value determination module, used to obtain the number of repeated selections of each of the multiple spatial directions in the estimation neural network, and to adjust the parameters of the multiple estimated values ​​according to the number of repeated selections to obtain multiple target values ​​corresponding to the multiple spatial directions; The first route planning module is used to determine the target space direction corresponding to the target value greater than the threshold if there is a target value greater than the threshold among the multiple target values, and to plan the route of the vehicle according to the target space direction and the speed parameter.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images