Unmanned vehicle scheduling method, device and equipment and storage medium

Abstract

The invention discloses an unmanned vehicle scheduling method, device and equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring a pre-constructed connected graph of a specified area; the connected graph comprises nodes corresponding to the service sites and edges corresponding to actions transferred between the service sites; based on the initial action value function and the connected graph, utilizing a preset cross detection rule to obtain a cross condition between an edge corresponding to the current sampling node and an edge corresponding to at least one historical sampling node; determining an edge reward function corresponding to each current sampling node based on a preset edge weight reward function and the intersection condition; based on the reward function of the edge corresponding to each current sampling node, performing iterative optimization processing on the initial action value function to obtain target scheduling information; and obtaining service scheduling information of the unmanned vehicle based on the target scheduling information, and controlling the unmanned vehicle to serve in the designated area based on the service scheduling information.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as intelligent transportation and autonomous driving, and in particular to a method, device, equipment and storage medium for dispatching an unmanned vehicle. Background Art

[0002] Usually, when unmanned vehicles are used in logistics, security and other scenarios, they need to be dispatched and controlled to start from the transfer center, pass through various terminal service stations, and then return to the transfer center to complete related services. The unmanned vehicle scheduling problem can be summarized as the Traveling Salesman Problem (TSP), which is a classic combinatorial optimization problem.

[0003] At present, common methods for solving TSP problems include dynamic programming, genetic algorithms, etc. However, it is difficult for the solutions of related technologies to balance solution accuracy and computational efficiency, and thus it is impossible to effectively optimize the scheduling and control of unmanned vehicles to ensure the reliability of unmanned vehicle operation. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for dispatching an unmanned vehicle, which optimizes the dispatching control of the unmanned vehicle and ensures the reliability of the operation of the unmanned vehicle. The technical solution is as follows:

[0005] In a first aspect, a method for dispatching an unmanned vehicle is provided, the method comprising:

[0006] Acquire a pre-built connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to transfer actions between the service sites;

[0007] Based on the initial action value function and the connectivity graph, using a preset intersection detection rule, obtaining an intersection between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node;

[0008] Based on a preset edge weight reward function and the intersection situation, determine the reward function value of the edge corresponding to each current sampling node;

[0009] Based on the reward function value of the edge corresponding to each current sampling node, iteratively optimize the initial action value function to obtain target scheduling information;

[0010] Based on the target scheduling information, service scheduling information of the unmanned vehicle is obtained, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

[0011] In a possible implementation, before obtaining the pre-built connectivity graph of the designated area, the method includes:

[0012] Acquire map information and service sites of a specified area; wherein the map information includes road information and road condition information, and the service sites include a central site and a plurality of terminal sites;

[0013] Determine the weight values ​​of the nodes and the edges between the nodes based on the service sites, road information and road condition information;

[0014] A connectivity graph of the designated area is constructed based on the nodes and the weight values ​​of the edges between the nodes.

[0015] In a possible implementation, based on the initial action value function and the connectivity graph, using a preset intersection detection rule, obtaining an intersection between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node includes:

[0016] Based on the initial action value function and the connectivity graph, sampling the nodes of the connectivity graph according to a preset exploration rate to obtain a plurality of sampling nodes;

[0017] For each current sampling node, perform the following operations: determine an edge corresponding to the current sampling node and an edge corresponding to at least one historical sampling node before the current sampling node;

[0018] Using a preset cross detection rule, cross detection is performed on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node;

[0019] Based on the result of the intersection detection process, an intersection situation between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node is obtained.

[0020] In a possible implementation, using a preset intersection detection rule to perform intersection detection processing on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node respectively includes:

[0021] For each edge corresponding to a historical sampling node, perform the following operations:

[0022] Determine the edge vector corresponding to the current sampling node, the edge vector corresponding to the historical sampling node, and the edge vector from the current sampling node to the historical sampling node;

[0023] Performing a cross multiplication process on the edge vector corresponding to the current sampling node and the edge vector corresponding to the historical sampling node to obtain a first cross multiplication result;

[0024] Performing a cross multiplication process on the edge vector corresponding to the current sampling node and the edge vector from the current sampling node to the historical sampling node to obtain a second cross multiplication result;

[0025] Based on a preset cross condition, the first cross product result, and the second cross product result, a result of the cross detection process is obtained.

[0026] In a possible implementation, the intersection situation includes the presence of an intersection and the absence of an intersection, and determining the reward function value of the edge corresponding to each current sampling node based on a preset edge weight reward function and the intersection situation includes:

[0027] In response to the intersection situation being that there is an intersection, determining a first reward function value of the edge corresponding to the current sampling node based on a preset edge weight reward function and a preset edge intersection reward function;

[0028] In response to the intersection situation being that there is no intersection, determining a second reward function value of the edge corresponding to the current sampling node based on a preset edge weight reward function;

[0029] Based on the first reward function value and the second reward function value, a reward function value of an edge corresponding to each current sampling node is determined.

[0030] In a possible implementation, the iterative optimization process of the initial action value function based on the reward function value of the edge corresponding to each current sampling node to obtain the target scheduling information includes:

[0031] Iteratively optimizing the initial action value function based on the reward function value of the edge corresponding to each current sampling node and each current sampling node;

[0032] In response to the result of the iterative optimization processing satisfying a preset convergence condition, target scheduling information is obtained.

[0033] In a second aspect, a device for dispatching an unmanned vehicle is provided, the device comprising:

[0034] An acquisition unit, configured to acquire a pre-built connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to transfer actions between the service sites;

[0035] A detection unit, configured to obtain, based on the initial action value function and the connectivity graph, an intersection condition between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node using a preset intersection detection rule;

[0036] A determination unit, configured to determine a reward function of an edge corresponding to each current sampling node based on a preset edge weight reward function and the intersection condition;

[0037] An optimization unit, configured to iteratively optimize the initial action value function based on the reward function of the edge corresponding to each current sampling node to obtain target scheduling information;

[0038] The scheduling unit is used to obtain the service scheduling information of the unmanned vehicle based on the target scheduling information, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

[0039] In a third aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspect and any possible implementation manner.

[0040] In a fourth aspect, an electronic device is provided, including:

[0041] at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0044] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned aspects and any possible implementation method.

[0045] In a sixth aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0046] The beneficial effects of the technical solution provided by this application include at least:

[0047] It can be seen from the above technical scheme that the embodiment of the present application can obtain a pre-constructed connectivity graph of the designated area, and then based on the initial action value function and the connectivity graph, use the preset intersection detection rule to obtain the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node, determine the reward function value of the edge corresponding to each current sampling node based on the preset edge weight reward function and the intersection situation, and iteratively optimize the initial action value function based on the reward function value of the edge corresponding to each current sampling node to obtain target scheduling information, and obtain the service scheduling information of the unmanned vehicle based on the target scheduling information, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information. Since the detection of possible path intersections between service sites is taken into account in the process of searching for the optimal scheduling strategy based on the connectivity graph, the reliability of the optimal scheduling strategy can be ensured while ensuring the computational efficiency of the optimization process, and the optimization effect and time efficiency can be better coordinated, thereby ensuring the reliability of the unmanned vehicle service operation.

[0048] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a flowchart of a method for dispatching an unmanned vehicle provided by an embodiment of the present application;

[0051] Figure 2 is a flowchart of a method for dispatching an unmanned vehicle provided by another embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a connectivity graph of service sites of a method for dispatching an unmanned vehicle provided in another embodiment of the present application;

[0053] Figure 4 is a schematic diagram of edge intersection detection of any two edges provided by another embodiment of the present application;

[0054] Figure 5 is a schematic diagram of an optimization process based on a connectivity graph in a method for dispatching an unmanned vehicle provided in another embodiment of the present application;

[0055] Figure 6 is a schematic diagram of the architecture of a method for dispatching an unmanned vehicle provided by another embodiment of the present application;

[0056] Figure 7 is a schematic diagram of the topology of the shortest path in the method for dispatching an unmanned vehicle provided in another embodiment of the present application;

[0057] Figure 8 is a structural block diagram of an unmanned vehicle dispatching device provided in yet another embodiment of the present application;

[0058] Fig. 9 It is a block diagram of an electronic device used to implement the method for dispatching an unmanned vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0061] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0062] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0063] Please refer to Figure 1 , which shows a flow chart of a method for dispatching an unmanned vehicle provided by an embodiment of the present application. The method for dispatching an unmanned vehicle may specifically include:

[0064] Step 101: Obtain a pre-constructed connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to transfer actions between the service sites.

[0065] Step 102: Based on the initial action value function and the connectivity graph, using a preset intersection detection rule, obtain the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node.

[0066] Step 103: Based on the preset edge weight reward function and the intersection situation, determine the reward function value of the edge corresponding to each current sampling node.

[0067] Step 104: Based on the reward function value of the edge corresponding to each current sampling node, the initial action value function is iteratively optimized to obtain target scheduling information.

[0068] Step 105: Based on the target scheduling information, obtain the service scheduling information of the unmanned vehicle, and control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

[0069] It should be noted that the designated area may be an area where the unmanned vehicle provides operating services. For example, the designated area may include but is not limited to a delivery area, a safety inspection area, etc.

[0070] It should be noted that there may be multiple service sites. The service sites may include a central site and a terminal site. The central site may be a transfer central site. The central site may be a starting service site or a terminal service site. For example, a designated area may have one central site and multiple terminal sites.

[0071] It should be noted that a node in a connectivity graph may include a node state, and an edge in a connectivity graph may represent a transition from one state to another state. An edge may also correspond to an action of transferring between service sites.

[0072] It should be noted that part or all of the execution subject of steps 101 to 105 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in an unmanned vehicle dispatching platform on the network side, etc. This embodiment does not specifically limit this.

[0073] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0074] In this way, by taking into account the detection of possible path intersections between service sites in the process of searching for the optimal scheduling strategy based on the connectivity graph, the reliability of the optimal scheduling strategy can be guaranteed while ensuring the computational efficiency of the optimization process, thus better coordinating the optimization effect and time efficiency, thereby ensuring the reliability of the unmanned vehicle service operation.

[0075] Optionally, in a possible implementation of the present embodiment, before step 101, map information and service sites of the specified area can be obtained, and then based on the service sites, road information and road condition information, weight values ​​of nodes and edges between the nodes can be determined, and based on the weight values ​​of the nodes and edges between the nodes, a connectivity graph of the specified area can be constructed.

[0076] In this implementation, the map information may include road information and road condition information. The road information may include connection information of drivable roads. The road condition information may include road congestion information, etc.

[0077] In this implementation, the service site includes a central site and multiple terminal sites. The central site can be a transfer center site. The central site can be a starting service site or a terminal service site.

[0078] For example, in the planning of the shortest path in a logistics scenario, the unmanned vehicle starts from the transfer center, passes through each terminal station, and then returns to the transfer center to complete logistics-related services.

[0079] In this way, a connectivity graph of a specified area can be constructed based on map information and service sites, which can facilitate the subsequent calculation of the shortest path for unmanned vehicle services based on the connectivity graph, and can improve the reliability of the obtained scheduling information.

[0080] Optionally, in a possible implementation of this embodiment, in step 102, first, based on the initial action value function and the connected graph, the nodes of the connected graph may be sampled according to a preset exploration rate to obtain a plurality of sampling nodes. Secondly, for each current sampling node, the following operations may be performed: determining the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node before the current sampling node, using a preset cross detection rule, performing cross detection processing on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node, and obtaining the cross condition between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node based on the result of the cross detection processing.

[0081] In this implementation, the initial action-value function may be an initial Q-table based on a Q reinforcement learning algorithm.

[0082] In a specific implementation process of this implementation method, for each edge corresponding to a historical sampling node, the following operations are performed: first, the vector of the edge corresponding to the current sampling node, the vector of the edge corresponding to the historical sampling node, and the vector of the edge from the current sampling node to the historical sampling node can be determined. Secondly, the vector of the edge corresponding to the current sampling node and the vector of the edge corresponding to the historical sampling node are cross-multiplied to obtain a first cross-multiplication result. Thirdly, the vector of the edge corresponding to the current sampling node and the vector of the edge from the current sampling node to the historical sampling node are cross-multiplied to obtain a second cross-multiplication result. Thirdly, based on the preset cross condition, the first cross-multiplication result, and the second cross-multiplication result, the result of the cross detection process is obtained.

[0083] In a specific implementation process, the preset cross condition may include that both the first cross product result and the second cross product result are smaller than a preset threshold.

[0084] Here, for example, the preset threshold value may be 0.

[0085] It can be understood that, in the process of edge intersection detection, the vectors of the line segments of every two edges can be directly calculated to determine whether the line segments of every two edges intersect.

[0086] In a specific implementation process, when the first cross product result and the second cross product result are both smaller than a preset threshold, it can be determined that the result of the intersection detection process is that the two sides intersect, and further it can be determined that the intersection situation is that there is an intersection.

[0087] When the first cross product result or the second cross product result is greater than or equal to a preset threshold, it can be determined that the result of the intersection detection processing is that the two sides do not intersect, and further it can be determined that the intersection situation is that there is no intersection.

[0088] In this way, it is possible to determine whether there is an edge crossing based on the preset crossing condition, the first cross product result between each two edges, and the second cross product result, so as to facilitate the subsequent determination of the reward function between nodes based on the edge crossing situation, so as to avoid the occurrence of edge crossing in the final solution path, thereby further improving the reliability of the scheduling information.

[0089] It should be noted that the various specific implementation processes provided in this implementation can be combined with each other to implement the steps in the method for dispatching an unmanned vehicle in this embodiment. For a detailed description, please refer to the relevant content in this implementation, which will not be repeated here.

[0090] Optionally, in a possible implementation of this embodiment, the intersection situation may include the presence of an intersection and the absence of an intersection. In step 103, in response to the intersection situation being the presence of an intersection, based on a preset edge weight reward function and a preset edge intersection reward function, a first reward function value of the edge corresponding to the current sampling node is determined, and in response to the intersection situation being the absence of an intersection, based on a preset edge weight reward function, a second reward function value of the edge corresponding to the current sampling node is determined, and further based on the first reward function value and the second reward function value, a reward function value of the edge corresponding to each current sampling node can be determined.

[0091] In this implementation, the edge corresponding to the current sampling node may refer to the edge between the current sampling node and the next sampling node. The first reward function value of the edge corresponding to the current sampling node may represent the reward function value of the action between the current sampling node and the next sampling node.

[0092] In a specific implementation process of this implementation method, first, the edge weight reward function value can be calculated based on the weight value of the edge corresponding to the current sampling node, the average of the weight values ​​of the edge corresponding to each historical sampling node, and the preset edge weight reward function. Secondly, the edge cross reward function value can be calculated based on the average of the weight values ​​of the edge corresponding to each historical sampling node, the flag value, and the preset edge cross reward function. Thirdly, the difference between the edge weight reward function value and the edge cross reward function value is calculated to obtain the first reward function value.

[0093] In another specific implementation process of this implementation method, first, the edge weight reward function value can be calculated based on the weight value of the edge corresponding to the current sampling node, the average of the weight values ​​of the edges corresponding to each historical sampling node, and a preset edge weight reward function, and the edge weight reward function value is used as the second reward function value.

[0094] In this way, the method of calculating the reward function value of the action corresponding to the edge executed by the current sampling node can be determined according to the edge crossing between the edges corresponding to the current sampling node and each historical sampling node, which can avoid the occurrence of edge crossing in the final solution path, thereby further improving the reliability of the scheduling information.

[0095] It should be noted that the various specific implementation processes provided in this implementation can be combined with each other to implement the steps in the method for dispatching an unmanned vehicle in this embodiment. For a detailed description, please refer to the relevant content in this implementation, which will not be repeated here.

[0096] Optionally, in a possible implementation of this embodiment, in step 104, first, the initial action-value function may be iteratively optimized based on the reward function value of the edge corresponding to each current sampling node and each current sampling node. Secondly, in response to the result of the iterative optimization process satisfying a preset convergence condition, the target scheduling information may be obtained.

[0097] In this implementation, the preset convergence condition may include that the change in the action value function value obtained by the iterative optimization process is within a preset change threshold range.

[0098] It can be understood that, when the state information of the given input sampling node and the corresponding output action value change are small, it can be determined that the result of the iterative optimization process meets the preset convergence condition.

[0099] In this implementation, the result of the iterative optimization process that satisfies the preset convergence condition may be a target action-value function. The target action-value function may be a target Q table based on a Q reinforcement learning algorithm.

[0100] In a specific implementation process of this implementation method, first, based on each current sampling node, the state information of each sampling node and the action corresponding to each sampling node can be obtained. The action corresponding to the sampling node can be the state transfer from the current sampling node to the next sampling node, that is, the action corresponding to the edge. Secondly, based on the reward function value of the edge corresponding to each current sampling node, the state information of the sampling node, and the action corresponding to the sampling node, the initial action value function is iteratively optimized until the preset convergence condition is met to obtain the target action value function. Thirdly, based on the target action value function and the starting sampling node, the greedy strategy is used to obtain the optimal path through each service site. Thirdly, based on the optimal path, the target scheduling information is obtained, and the target scheduling information can be a service scheduling table.

[0101] In this specific implementation process, the greedy strategy may include an existing greedy algorithm or model for determining the optimal path, etc., which is not specifically limited here.

[0102] It should be noted that the various specific implementation processes provided in this implementation can be combined with each other to implement the steps in the method for dispatching an unmanned vehicle in this embodiment. For a detailed description, please refer to the relevant content in this implementation, which will not be repeated here.

[0103] In order to better understand the method of the embodiment of the present application, the method of the embodiment of the present application is described below in conjunction with the accompanying drawings and specific application scenarios.

[0104] Figure 2 FIG. 1 is a flow chart of a method for dispatching an unmanned vehicle provided by another embodiment of the present application. Figure 2 shown.

[0105] Step 201: Obtain map information and service sites of a specified area.

[0106] In this embodiment, the designated area may include a logistics distribution area, etc.

[0107] Step 202: construct a connectivity graph of the designated area based on the map information and service sites.

[0108] In this embodiment, the service site may include a central site and a terminal site. The central site may be a transfer central site. The service site may be a node of a connectivity graph.

[0109] For example, in the planning of the shortest path in a logistics scenario, the unmanned vehicle starts from the transfer center, passes through each terminal station, and then returns to the transfer center to complete logistics-related services.

[0110] In this embodiment, optionally, first, map information between service sites may be obtained, the map information including road information and traffic condition information. Secondly, based on the map information including road information and traffic condition information, the Manhattan distance weight and congestion efficiency weight of the connection edge between each two service sites, i.e., nodes, are calculated. Thirdly, the weight of the connection edge may be calculated based on the Manhattan distance weight and the congestion efficiency weight.

[0111] Specifically, the Manhattan distance weight of the connecting edge can be calculated using formulas (1) and (2): and congestion efficiency weight

[0112]

[0113] in, represents the Manhattan distance from node i to node j, represents the congestion efficiency weight from node i to node j, w congestion is the weight parameter, C represents the expected congestion time, (x i,y i ) can be the position of the i-node, (x j ,y j ) can be the position of node j.

[0114] Furthermore, the weight of the connecting edge can be calculated using formula (3):

[0115]

[0116] Here, the road information may be road connection, and the road condition information may be road congestion information.

[0117] In this implementation, Figure 3 FIG. 1 is a schematic diagram of a connectivity diagram of a service station of a method for dispatching an unmanned vehicle provided in another embodiment of the present application. Figure 3 As shown in Figure 1, node 0 can represent the central site of the service site, and nodes 1, 2, 3, and 4 can represent the end sites of the service site. The weights of the edges between nodes include l 0,1 , l 0,2 , l 0,3 , l 0,4 , l 1,2 , l 1,3 , l 1,4 , l 2,3 , l 2,4 , l 3 ,4 .

[0118] Here, a node may include state information. The state information may include location information of a service station corresponding to the node, traffic information, etc. An edge may represent a state transition from one node state to another node state. An edge may also represent the shortest path between nodes. An action may refer to an operation selected to be performed based on the current node state, with the goal of maximizing the total reward in the future.

[0119] It is understandable that for the connectivity graph of the service sites, the shortest path can be obtained by starting from the starting central node, visiting all the terminal sites and then returning to the starting central node, and making the total traversal path length the shortest.

[0120] Step 203: Using a preset reinforcement learning algorithm, based on the initial action value function and according to a preset exploration rate, the nodes of the connected graph are sampled to obtain a plurality of sampled nodes.

[0121] In this implementation, the preset reinforcement learning algorithm may be an improved Q reinforcement learning algorithm (eIQ-Learning). The initial action value function may be an initial Q table. Each Q value in the initial Q table may be an initial action value function value.

[0122] In this implementation, first, the node corresponding to the central site in the service site can be determined as the starting sampling node. Secondly, based on the starting sampling node, the nodes of the connected graph are sampled according to a preset exploration rate to obtain multiple sampling nodes.

[0123] It is understandable that by sampling the nodes of the connected graph, the edge from each current sampling node to the next sampling node and the action corresponding to the edge can be determined. Multiple edges can correspond to multiple sampling actions.

[0124] Step 204: Based on the edge between each current sampling node and the next sampling node of the current sampling node, a reward function of each current sampling node is determined using a preset intersection detection rule to calculate a reward function value.

[0125] In this implementation, for each current sampling node, first, the edge between the current sampling node and the next sampling node of the current sampling node can be determined as the edge corresponding to the current sampling node. Secondly, the edge between at least one historical sampling node before the current sampling node and the next sampling node of the historical sampling node can be determined as the edge corresponding to at least one historical sampling node.

[0126] In this implementation, for each edge corresponding to a historical sampling node, first, the vector of the edge corresponding to the current sampling node, the vector of the edge corresponding to the historical sampling node, and the vector of the edge from the current sampling node to the historical sampling node can be determined. Secondly, the vector of the edge corresponding to the current sampling node and the vector of the edge corresponding to the historical sampling node are cross-multiplied to obtain a first cross-multiplication result. Thirdly, the vector of the edge corresponding to the current sampling node and the vector of the edge from the current sampling node to the historical sampling node are cross-multiplied to obtain a second cross-multiplication result. Thirdly, based on the preset cross condition, the first cross-multiplication result, and the second cross-multiplication result, the result of the cross detection process is obtained.

[0127] In this embodiment, the preset cross condition may include that both the first cross product result and the second cross product result are smaller than a preset threshold value. Here, for example, the preset threshold value may be 0.

[0128] For example, Figure 4 FIG. 1 is a schematic diagram of edge intersection detection of any two edges provided by another embodiment of the present application. Figure 4 As shown, AB and CD can be line segments representing edges, and AC is the line connecting the starting points of the two line segments. If the two line segments AB and CD satisfy the preset intersection condition shown in formula (4), it can be determined that the two line segments AB and CD intersect:

[0129]

[0130] in, is the vector of line segment AB, that is, the vector of the edge corresponding to the current sampling node; is the vector of line segment CD, that is, the vector of the edge corresponding to the historical sampling node; is the vector of line segment AC, that is, the vector of the edge from the current sampling node to the historical sampling node.

[0131] It can be understood that, in the process of edge intersection detection, the vectors of the line segments of every two edges can be directly calculated to determine whether the line segments of every two edges intersect.

[0132] In this embodiment, when the first cross product result and the second cross product result are both smaller than a preset threshold, it can be determined that the result of the intersection detection process is that the two sides intersect, and further it can be determined that the intersection situation is that there is an intersection.

[0133] When the first cross product result or the second cross product result is greater than or equal to a preset threshold, it can be determined that the result of the intersection detection processing is that the two sides do not intersect, and further it can be determined that the intersection situation is that there is no intersection.

[0134] Furthermore, on the one hand, first, when the intersection situation is that there is an intersection, the first reward function value of the edge corresponding to the current sampling node can be determined based on the preset edge weight reward function and the preset edge intersection reward function, and the first reward function value can be used as the reward function value.

[0135] Secondly, here, the preset edge weight reward function C can be shown as the following formula (5):

[0136]

[0137] in, The average edge length determined by the N-step state transitions between historical sampling nodes, that is, the average weight value of the edge, C N+1 The state corresponding to the current sampling node can be obtained from s N to N+1 The determined edge length is the weight value of the edge. β can be a hyperparameter. c k is the weight value of the edge in the kth step, N is the number of nodes passed, and N+1 is the next node.

[0138] Here, the larger the C value is, the better the state transfer from the current sampling node to the next sampling node is, that is, the action selected in the current step is better than the average action selected in the past N steps, and vice versa.

[0139] Based on the weight value of the edge corresponding to the current sampling node and the weight value of the edge corresponding to the historical sampling node, the preset edge weight reward function C can be used to calculate the edge weight reward function value of the edge corresponding to the current sampling node, that is, the edge weight reward function value of the state transfer action corresponding to the current sampling node.

[0140] Again, the preset edge crossing reward function I can be shown as the following formula (6):

[0141]

[0142] Among them, i can be a flag bit, can be the mean edge length determined by the N-step state transitions between historical sampling nodes, γ can be a hyperparameter, c k is the weight of the edge at step k, and N is the number of nodes traversed.

[0143] Based on the weight value and flag value of the edge corresponding to the historical sampling node, the preset edge cross reward function I can be used to calculate the edge cross reward function value of the edge corresponding to the current sampling node, that is, the edge cross reward function value of the state transfer action from the current sampling node to the next sampling node.

[0144] It can be understood that here, the edge crossing detection is to check the current sampling node to the next sampling node, that is, the current step from s N to N+1 The determined edge l N+1 The edges {l1,l2,....,l N}intersection relationship between them, if there is an intersection, the flag bit i=1, otherwise, i=0.

[0145] Finally, the first reward function, i.e., the total reward function, can be determined based on the edge weight reward function and the edge cross reward function, so that the reward function value, i.e., the total reward function value, can be calculated based on the edge weight reward function value and the edge cross reward function value.

[0146] Here, the total reward function R can be expressed as follows:

[0147] R=CI (7)

[0148] Among them, C can be the edge weight reward function, and I can be the edge intersection reward function.

[0149] On the other hand, when the intersection situation is that there is no intersection, the second reward function of the edge corresponding to the current sampling node can be determined based on the preset edge weight reward function.

[0150] Here, when the intersection situation is that there is no intersection, the second reward function, that is, the total reward function, can be determined directly based on the function of the preset edge weight reward function to calculate the reward function value.

[0151] It can be understood that when the intersection situation is that there is no intersection, i=0, and using the above formulas (5) to (7), it can be determined that R=C, and the function value of the second reward function can be equal to the function value of the edge weight reward function.

[0152] Optionally, based on the first reward function and the second reward function, a reward function of an edge corresponding to each current sampling node is determined.

[0153] It can be understood that, for a current collection node with an intersection, the reward function value of the edge corresponding to the node can be the first reward function value; for a current collection node without an intersection, the reward function value of the edge corresponding to the node can be the second reward function value, and then the intersection situation of each current collection node can be used to obtain the corresponding reward function value of each current collection node to obtain the corresponding reward function value of all collection nodes.

[0154] Step 205: Based on the reward function value, the state information of the sampling node, and the action corresponding to the sampling node, the initial action-value function is iteratively optimized until the preset convergence condition is met to obtain the target action-value function.

[0155] In this embodiment, the state information of the sampling node and the action corresponding to the sampling node may be obtained. The action corresponding to the sampling node may be the action corresponding to the state transition from the current sampling node to the next sampling node.

[0156] In this embodiment, the action-value function may be updated using a temporal difference algorithm (TD error).

[0157] Here, exemplary, the initial action value function is iteratively optimized, which can be shown in the following formula (8):

[0158] Q(S t , A t )←Q(S t , A t )+α[r t+1 +γmax a Q(S t+1 , a)-Q(S t , A t )] (8)

[0159] Among them, α can be the learning rate (learning_rate), γ can be the reduction coefficient, which can be taken as 0.95. The Q(S t , A t ) is the new Q value of action a in the state s of the node. The Q(S t , A t ) is the original Q value, r t+1is the reward function value obtained after executing action a, max a Q(S t+1 , a) is the state S of the next node t+1 The maximum Q value among all the actions.

[0160] In this embodiment, illustratively, Figure 5 FIG. 1 is a schematic diagram of an optimization process based on a connectivity graph in a method for dispatching an unmanned vehicle provided in another embodiment of the present application. Figure 5 As shown, in graph search, state s is the state s of the node node , action a can be based on the action value function Q(S t , a) and the action selection strategy π are sampled. Action a is an implicit action, which only connects the transition process from state s to state s′. The action value function Q(S t+1 , a′) represents the next state S t+1 And the probability distribution of selecting action a' in this state. When transferring from state s to state s', you will get an immediate reward r t+1 . Q(S t+1 , a′) and Q(S t , a) can be transferred backward. Here, the action selection strategy π can include π→a1, π→a2, π→a3, π→a4.

[0161] Step 206: Based on the target action value function and the starting sampling node, the greedy strategy is used to obtain the optimal path through each service site.

[0162] Step 207: Obtain target scheduling information based on the optimal path.

[0163] At this point, the unmanned vehicle can be controlled to perform service tasks in a designated area based on the target scheduling information.

[0164] In this embodiment, the target scheduling information may be a service scheduling information table.

[0165] Exemplarily, the actions corresponding to the target action value function may be from 1→2, 2→3, 3→4, 4→1; the shortest path may refer to a continuous path, 1→2→3→4→1; and the service scheduling information table may refer to 1, 2, 3, 4, 1 of accessing the node table in sequence.

[0166] In this embodiment, Figure 6 FIG. 1 is a schematic diagram of the architecture of the method for dispatching an unmanned vehicle provided by another embodiment of the present application. Figure 6As shown, first, the service sites in the service application scenario, namely, service outlets and map information, namely, environment information, can be obtained. The service outlets are used as nodes and the map / environment is used as edges to construct a connected graph. Secondly, based on the connected graph, the eIQ-Learning algorithm is used to calculate the shortest path through each service outlet. Thirdly, based on the shortest path, the service scheduling table is obtained.

[0167] Here, in the eIQ-Learning algorithm, in the process of calculating the shortest path, for any current node, first, the action selection can be based on the initial Q table π to determine the action π→a corresponding to the current node, from the current node: s to the next node: s'. Secondly, combined with the edge intersection check rule, the corresponding reward r is calculated. Thirdly, based on the TD error algorithm, the action value function is iteratively optimized, and the target action value function π* is obtained through action sampling to obtain the shortest path.

[0168] In this embodiment, Figure 7 FIG. 1 is a schematic diagram of the topology of the shortest path in the method for dispatching an unmanned vehicle provided by another embodiment of the present application. Figure 7 As shown, the topological graph of the shortest path includes 10 nodes. The number of epochs of the eIQ-Learning algorithm is 100, the calculated path length is 2054.90, and the normalizer error is 7.918.

[0169] In this embodiment, the improved reinforcement learning (eIQ-Learning) algorithm is used to solve the shortest path for unmanned vehicle scheduling, which is significantly improved compared with the traditional algorithm. As shown in Table 1:

[0170] Table 1 Algorithm effect comparison table

[0171] Algorithm Name Total path length Solution time Improved accuracy Improved efficiency DP 2054.9 2.062 — — GA 2054.9 22.968 — — Q-Learning 2362.309 0.032 — — eIQ-Learning 2054.9 0.884 0 / 0 / 13.013 57.134 / 96.151 / —

[0172] Specifically, the eIQ-Learning algorithm has the same solution accuracy as the dynamic programming algorithm (DP) and the genetic algorithm (GA), but the solution efficiency of the eIQ-Learning algorithm has been improved by 57.13% and 96.15% respectively. Compared with the basic Q reinforcement learning algorithm (Q-Learning), the solution accuracy of the eIQ-Learning algorithm has been improved by 13.01%.

[0173] In this way, by adopting the technical solution in this embodiment, the TSP problem can be converted into a path planning problem, and the optimal scheduling strategy between sites can be learned by defining node states, collecting actions, executing rewards and state transition processes. Compared with the Q reinforcement learning algorithm, the learning efficiency is improved by adding edge cross-checks in the graph search process, and the accuracy of the solution can be improved by more than 10%. Moreover, compared with optimization algorithms such as dynamic programming / genetic algorithms, the computing efficiency is improved by more than 60% and 97% respectively, which can ensure the computing efficiency of the optimization process while ensuring the reliability of the optimal scheduling strategy, and better coordinate the optimization effect and time efficiency, thereby ensuring the reliability of the unmanned vehicle service operation.

[0174] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] Figure 8 A structural block diagram of an unmanned vehicle dispatching device provided by an embodiment of the present application is shown. Figure 8 As shown. The unmanned vehicle scheduling device 800 of this embodiment may include an acquisition unit 801, a detection unit 802, a determination unit 803, an optimization unit 804 and a scheduling unit 805. Among them, the acquisition unit 801 is used to acquire a pre-constructed connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to the actions of transfer between the service sites; the detection unit 802 is used to obtain the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node based on the initial action value function and the connectivity graph, using a preset intersection detection rule; the determination unit 803 is used to determine the reward function value of the edge corresponding to each current sampling node based on a preset edge weight reward function and the intersection; the optimization unit 804 is used to iteratively optimize the initial action value function based on the reward function value of the edge corresponding to each current sampling node to obtain target scheduling information; the scheduling unit 805 is used to obtain the service scheduling information of the unmanned vehicle based on the target scheduling information, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

[0177] It should be noted that part or all of the unmanned vehicle dispatching device of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in an unmanned vehicle dispatching platform on the network side, etc. This embodiment does not specifically limit this.

[0178] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0179] Optionally, in a possible implementation of this embodiment, the acquisition unit 801 can be specifically used to obtain map information and service sites of a specified area; wherein the map information includes road information and traffic condition information, and the service sites include a central site and multiple terminal sites; based on the service sites, road information and traffic condition information, the weight values ​​of the nodes and the edges between the nodes are determined; based on the weight values ​​of the nodes and the edges between the nodes, a connectivity graph of the specified area is constructed.

[0180] Optionally, in a possible implementation of this embodiment, the detection unit 802 can be used to sample the nodes of the connected graph according to a preset exploration rate based on the initial action value function and the connected graph to obtain multiple sampling nodes; for each current sampling node, perform the following operations: determine the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node before the current sampling node; use a preset intersection detection rule to perform intersection detection processing on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node; based on the result of the intersection detection processing, obtain the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node.

[0181] Optionally, in a possible implementation of this embodiment, the detection unit 802 can be used to perform the following operations for the edge corresponding to each historical sampling node: determine the vector of the edge corresponding to the current sampling node, the vector of the edge corresponding to the historical sampling node, and the vector of the edge from the current sampling node to the historical sampling node; perform cross multiplication processing on the vector of the edge corresponding to the current sampling node and the vector of the edge corresponding to the historical sampling node to obtain a first cross multiplication result; perform cross multiplication processing on the vector of the edge corresponding to the current sampling node and the vector of the edge from the current sampling node to the historical sampling node to obtain a second cross multiplication result; and obtain the result of the cross detection processing based on a preset cross condition, the first cross multiplication result, and the second cross multiplication result.

[0182] Optionally, in a possible implementation of this embodiment, the intersection situation includes the presence of an intersection and the absence of an intersection. The determination unit 803 can be used to determine, in response to the intersection situation being the presence of an intersection, a first reward function value of the edge corresponding to the current sampling node based on a preset edge weight reward function and a preset edge intersection reward function; in response to the intersection situation being the absence of an intersection, determine, based on a preset edge weight reward function, a second reward function value of the edge corresponding to the current sampling node; and determine, based on the first reward function value and the second reward function value, the reward function value of the edge corresponding to each current sampling node.

[0183] Optionally, in a possible implementation of this embodiment, the optimization unit 804 can be used to iteratively optimize the initial action-value function based on the reward function of the edge corresponding to each current sampling node and each current sampling node; in response to the result of the iterative optimization processing satisfying a preset convergence condition, the target scheduling information is obtained.

[0184] In this embodiment, a pre-constructed connectivity graph of a designated area can be acquired by an acquisition unit; the connectivity graph includes nodes corresponding to service sites and edges corresponding to actions of transfer between the service sites. Then, the detection unit can obtain the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node based on the initial action value function and the connectivity graph by using a preset intersection detection rule. The determination unit determines the reward function of the edge corresponding to each current sampling node based on a preset edge weight reward function and the intersection. The optimization unit iteratively optimizes the initial action value function based on the reward function of the edge corresponding to each current sampling node to obtain target scheduling information, so that the scheduling unit can obtain the service scheduling information of the unmanned vehicle based on the target scheduling information, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information. Since the detection of possible path intersections between service sites is taken into account in the process of searching for the optimal scheduling strategy based on the connectivity graph, the reliability of the optimal scheduling strategy can be ensured while the computational efficiency of the optimization process is ensured, and the optimization effect and time efficiency are well coordinated, thereby ensuring the reliability of the unmanned vehicle service operation.

[0185] In the technical solution of this application, the user personal information involved, such as the collection, storage, use, processing, transmission, provision and disclosure of user images and attribute data, etc., complies with the provisions of relevant laws and regulations and does not violate public order and good morals.

[0186] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0187] According to an embodiment of the present application, further, an unmanned vehicle including the provided electronic device is provided, and the unmanned vehicle may include an unmanned vehicle of level L2 or above. For example, the unmanned vehicle may include an unmanned logistics vehicle, an unmanned patrol vehicle, an unmanned delivery vehicle, etc.

[0188] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0189] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 906 is also connected to the bus 904.

[0190] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0191] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the method of unmanned vehicle scheduling. For example, in some embodiments, the method of unmanned vehicle scheduling may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the method of unmanned vehicle scheduling described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the method of unmanned vehicle scheduling in any other appropriate manner (e.g., by means of firmware).

[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0194] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0197] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0198] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0199] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for dispatching an unmanned vehicle, characterized in that: The method comprises: Acquire a pre-built connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to transfer actions between the service sites; Based on the initial action value function and the connectivity graph, using a preset intersection detection rule, obtaining an intersection between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node; Based on a preset edge weight reward function and the intersection situation, determine a reward function value of an edge corresponding to each current sampling node; Based on the reward function value of the edge corresponding to each current sampling node, iteratively optimize the initial action value function to obtain target scheduling information; Based on the target scheduling information, service scheduling information of the unmanned vehicle is obtained, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

2. The method according to claim 1, characterized in that Before obtaining the pre-built connectivity graph of the designated area, the method includes: Acquire map information and service sites of a specified area; wherein the map information includes road information and road condition information, and the service sites include a central site and a plurality of terminal sites; Determine the weight values ​​of the nodes and the edges between the nodes based on the service sites, road information and road condition information; A connectivity graph of the designated area is constructed based on the nodes and the weight values ​​of the edges between the nodes.

3. The method according to claim 1, characterized in that The method of obtaining the intersection between the edge corresponding to the current sampling node and the edge corresponding to at least one historical sampling node based on the initial action value function and the connectivity graph and using a preset intersection detection rule includes: Based on the initial action value function and the connectivity graph, sampling the nodes of the connectivity graph according to a preset exploration rate to obtain a plurality of sampling nodes; For each current sampling node, perform the following operations: determine an edge corresponding to the current sampling node and an edge corresponding to at least one historical sampling node before the current sampling node; Using a preset cross detection rule, cross detection is performed on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node; Based on the result of the intersection detection process, an intersection situation between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node is obtained.

4. The method according to claim 3, characterized in that The using a preset cross detection rule to perform cross detection processing on the edge corresponding to the current sampling node and the edge corresponding to each historical sampling node respectively includes: For each edge corresponding to a historical sampling node, perform the following operations: Determine the edge vector corresponding to the current sampling node, the edge vector corresponding to the historical sampling node, and the edge vector from the current sampling node to the historical sampling node; Performing a cross multiplication process on the edge vector corresponding to the current sampling node and the edge vector corresponding to the historical sampling node to obtain a first cross multiplication result; Performing a cross multiplication process on the edge vector corresponding to the current sampling node and the edge vector from the current sampling node to the historical sampling node to obtain a second cross multiplication result; Based on a preset cross condition, the first cross product result, and the second cross product result, a result of the cross detection process is obtained.

5. The method according to claim 1, characterized in that The intersection situation includes the presence of an intersection and the absence of an intersection, and the determining of the reward function value of the edge corresponding to each current sampling node based on the preset edge weight reward function and the intersection situation includes: In response to the intersection situation being that there is an intersection, determining a first reward function value of the edge corresponding to the current sampling node based on a preset edge weight reward function and a preset edge intersection reward function; In response to the intersection situation being that there is no intersection, determining a second reward function value of the edge corresponding to the current sampling node based on a preset edge weight reward function; Based on the first reward function value and the second reward function value, a reward function value of an edge corresponding to each current sampling node is determined.

6. The method according to claim 1, characterized in that The iterative optimization process of the initial action value function based on the reward function value of the edge corresponding to each current sampling node to obtain target scheduling information includes: Iteratively optimizing the initial action value function based on the reward function value of the edge corresponding to each current sampling node and each current sampling node; In response to the result of the iterative optimization processing satisfying a preset convergence condition, target scheduling information is obtained.

7. A device for dispatching unmanned vehicles, characterized in that: The device comprises: An acquisition unit, configured to acquire a pre-built connectivity graph of a designated area; the connectivity graph includes nodes corresponding to service sites and edges corresponding to transfer actions between the service sites; A detection unit, configured to obtain, based on the initial action value function and the connectivity graph, an intersection condition between an edge corresponding to a current sampling node and an edge corresponding to at least one historical sampling node using a preset intersection detection rule; A determination unit, configured to determine a reward function value of an edge corresponding to each current sampling node based on a preset edge weight reward function and the intersection condition; An optimization unit, configured to iteratively optimize the initial action value function based on the reward function value of the edge corresponding to each current sampling node to obtain target scheduling information; The scheduling unit is used to obtain the service scheduling information of the unmanned vehicle based on the target scheduling information, so as to control the unmanned vehicle to provide service in the designated area based on the service scheduling information.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images