Systems and methods for learning to selectively optimize a determined path

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining a path. One example method includes inputting a plurality of path solution candidates to be optimized to a twin neural network, the twin neural network including a plurality of value prediction networks, each trained to predict a cost related to the path solution candidates to be optimized, identifying one or more path solution candidates to be optimized from the plurality of path solution candidates to be optimized based on an output of the twin neural network, inputting the one or more identified path solution candidates to be optimized to a path optimizer to obtain one or more optimized path solution candidates, and determining an optimized path solution having a lowest cost from the one or more optimized path solution candidates.

Description

Technical Field

[0001] This application generally relates to systems and methods for determining paths, and in particular, to systems and methods for determining paths by identifying promising path solution candidates and selectively optimizing the identified path solution candidates. Background Technology

[0002] Route optimization is the process of determining the optimal cost route solution from a finite set of candidate route solutions. The classic Traveling Salesman Problem (TSP) and Vehicle Routing Problem (VRP) are some exemplary variations of route optimization. Practical applications of route optimization can be found in fields such as telecommunications network design, task scheduling, transportation system planning, energy, finance, and supply chain management. Route optimization problems involving finding efficient routes for vehicles are generally referred to as VRPs. Several variations of VRP include Pickup and Delivery VRP (VRPPD), Last-In-First-Out VRP (VRP with Last-In-First-Out), VRP with Time Windows (VRPTW), and Capacitated VRP (CVRP).

[0003] In typical path optimization scenarios, the optimal path solution can comprise multiple paths through N given locations under various constraints. Finding the optimal path solution is challenging because the total number of candidate paths is enormous, even when N is small. It is generally accepted that determining the optimal solution to a VRP is an NP-hard (non-deterministic polynomial-time hardness) problem. In practice, some path solution candidates (e.g., randomly selected paths) can serve as a starting point, and a path optimizer can be used to optimize them to obtain several optimized path solutions from which the path solution with the lowest cost can be identified. Since each path solution candidate may only guide the optimizer to explore a small portion of the search space (e.g., the solution space), it is necessary to apply the optimizer to a large number of path solution candidates to find the optimal path solution. However, applying the optimizer typically consumes significant time and computational resources. Therefore, a method is needed to determine the optimal path by intelligently identifying promising path solution candidates for optimization. Summary of the Invention

[0004] The various embodiments described in this specification may include, but are not limited to, systems, methods, and non-transitory computer-readable media for determining paths.

[0005] According to some embodiments, a computer-executed method for determining a path may include: inputting a plurality of candidate path solutions to be optimized into a Siamese neural network, the Siamese neural network including a plurality of value prediction networks, each of the value prediction networks being trained to predict costs associated with the candidate path solutions to be optimized; identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the output of the Siamese neural network; inputting the identified one or more candidate path solutions to be optimized into a path optimizer to obtain one or more optimized candidate path solutions; and determining the optimized path solution with the lowest cost from the one or more optimized candidate path solutions.

[0006] In some embodiments, each of the plurality of candidate routes to be optimized comprises one or more routes that allow one or more vehicles to pass through multiple locations, and the candidate route to be optimized is subject to one or more constraints, including one or more of the following: time constraints; travel distance constraints; vehicle capacity constraints; and power cost constraints.

[0007] In some embodiments, the path optimizer includes a set of improved operators learned based on a reinforcement learning algorithm, and the set of improved operators performs one or more of the following operations: changing the order of at least two of the plurality of positions, the at least two positions being on one of the one or more paths; and moving a position on one of the one or more paths to another of the one or more paths.

[0008] In some embodiments, the plurality of value prediction networks in the Siamese neural network are identical.

[0009] In some embodiments, the Siamese neural network includes two value prediction networks, and the method further includes: training the two value prediction networks by performing one or more iterations of an adjustment process, wherein performing one or more iterations of the adjustment process includes: obtaining a training set including a third path solution candidate to be optimized and a fourth path solution candidate to be optimized; inputting the training set into the path optimizer to obtain a third score of the third path solution candidate to be optimized and a fourth score of the fourth path solution candidate to be optimized; inputting the training set into the two value prediction networks respectively to obtain a fifth score of the third path solution candidate to be optimized and a sixth score of the fourth path solution candidate to be optimized; and adjusting the weights of the two value prediction networks based at least on the third score, the fourth score, the fifth score, and the sixth score.

[0010] In some embodiments, before inputting the training set into the two value prediction networks, the adjustment process further includes: determining whether the difference between the third score and the fourth score is greater than a preset threshold; and if the difference is not greater than the preset threshold, discarding the third and fourth path solution candidates to be optimized.

[0011] In some embodiments, adjusting the weights of the two value prediction networks includes: determining the label of the training set based on the third score and the fourth score; converting the fifth score and the sixth score into a fifth logit value and a sixth logit value; determining a cross-entropy loss function based on the label, the fifth logit value, and the sixth logit value; and adjusting the weights of the two value prediction networks based on the cross-entropy loss function.

[0012] In some embodiments, each of the plurality of value prediction networks includes: a bidirectional long short-term memory (LSTM) layer comprising a plurality of LSTM units; an attention layer for embedding the outputs from the plurality of LSTM units; and an output layer for generating a score based on the outputs from the attention layer and a plurality of features associated with the candidate path solution to be optimized.

[0013] In some embodiments, the candidate path solution to be optimized includes multiple paths, each associated with a distance, and the multiple features associated with the candidate path solution to be optimized include: the sum of the distances of the multiple paths in the candidate path solution to be optimized; and the standard deviation of the distances of the multiple paths in the candidate path solution to be optimized.

[0014] In some embodiments, the Siamese neural network includes two value prediction networks, and the input of a plurality of path solution candidates to the Siamese neural network includes: for each of the plurality of path solution candidates to be optimized: pairing the path solution candidate to be optimized with each other path solution candidate to be optimized that is different from the path solution candidate to be optimized; and inputting the path solution candidate to be optimized and each of the paired other candidates into the two value prediction networks to determine a separate score for the path solution candidate to be optimized.

[0015] In some embodiments, the step of inputting the candidate path solution to be optimized and each of the paired other candidate path solutions to be optimized into the two value prediction networks to determine a separate score for the candidate path solution to be optimized includes: obtaining a seventh score for the candidate path solution to be optimized and an eighth score for each of the paired other candidate path solutions to be optimized; determining a positive score for the candidate path solution to be optimized if the seventh score is greater than the eighth score; and determining a non-positive score for the candidate path solution to be optimized if the seventh score is not greater than the eighth score.

[0016] In some embodiments, identifying one or more path solution candidates to be optimized from the plurality of path solution candidates to be optimized based on the Siamese neural network includes: for each path solution candidate to be optimized in the plurality of path solution candidates to be optimized: determining a total score for the path solution candidate to be optimized, the total score including the sum of the individual scores of the path solution candidate to be optimized; and identifying the path solution candidate to be optimized with the highest total score from the plurality of path solution candidates to be optimized.

[0017] According to other embodiments, a system for determining a path includes one or more processors and one or more computer-readable storage devices coupled to the one or more processors and having instructions stored thereon, the instructions being executable by the one or more processors to perform operations including: inputting a plurality of candidate path solutions to be optimized into a Siamese neural network, the Siamese neural network including a plurality of value prediction networks, each of the value prediction networks being trained to predict costs associated with the candidate path solutions to be optimized; identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the output of the Siamese neural network; inputting the identified one or more candidate path solutions to be optimized into a path optimizer to obtain one or more optimized candidate path solutions; and determining the optimized path solution with the lowest cost from the one or more optimized candidate path solutions.

[0018] According to another embodiment, a non-transitory computer-readable storage medium for determining a path is configured with instructions executable by one or more processors to cause the one or more processors to perform operations, the operations including: inputting a plurality of candidate path solutions to be optimized into a Siamese neural network, the Siamese neural network including a plurality of value prediction networks, each of the value prediction networks being trained to predict costs associated with the candidate path solutions to be optimized; identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the output of the Siamese neural network; inputting the identified one or more candidate path solutions to be optimized into a path optimizer to obtain one or more optimized candidate path solutions; and determining the optimized path solution with the lowest cost from the one or more optimized candidate path solutions.

[0019] The embodiments disclosed herein have one or more technical effects. In some embodiments, the method and system include a Learn to Restart (L2R) module to filter multiple path solution candidates, helping the Black-box Optimization Agent (BOA) initialize or restart, thereby finding an ideal solution in the search space. Based on the filtering results, the BOA can selectively optimize (e.g., choose to optimize) a smaller number of path solution candidates and avoid wasting computational resources and time optimizing those unpromising candidates. In some embodiments, the L2R module includes two identical trained value prediction networks, each providing an approximate value surface (e.g., similar to an approximate value surface for a continuous optimization problem) on the search space implicitly defined by the BOA. In other words, assuming the BOA has a certain value prediction capability for a given path solution candidate X to be optimized... 0 After optimization, the optimized path solution X was returned. * (represented as X) * |X 0 Then, the L2R module can be trained to use the value prediction network to predict X. * |X 0 The function can be used without actually running BOA. In some embodiments, the trained value prediction network can be used as a weak learner to build the L2R module through mechanisms such as boosting to accurately identify “promising” candidates for optimization. For example, by applying the trained value prediction network multiple times over various combinations of path solution candidates, the L2R module can fairly accurately determine the most “promising” path solution candidate for BOA optimization.

[0020] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the functionality of the elements of the operating methods and related structures, and the economics of the combination of components and manufacture, will become more apparent when considered in conjunction with the accompanying drawings and the appended claims. All these drawings form part of this specification, wherein like reference numerals denote corresponding parts in the respective drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not restrictive. Attached Figure Description

[0021] Figure 1 An environment associated with a path problem or a variant instance thereof, according to some embodiments, is shown.

[0022] Figure 2 Examples of path solutions for path problems or variant instances thereof, according to some embodiments, are shown.

[0023] Figure 3Exemplary operations for optimizing path solutions are shown according to some embodiments.

[0024] Figure 4 An exemplary system diagram for determining a path is shown according to some embodiments.

[0025] Figure 5 An exemplary value prediction network for determining a path is shown according to some embodiments.

[0026] Figure 6 An exemplary application of a trained value prediction network for path determination, according to some embodiments, is shown.

[0027] Figure 7 An exemplary method for determining a path is shown according to some embodiments.

[0028] Figure 8 A block diagram of a computer system for determining a path is shown according to some embodiments.

[0029] Figure 9 A block diagram of a computer system that can implement any of the embodiments described herein is shown. Detailed Implementation

[0030] Capacitated Vehicle Routing Problem (CVRP) and its variants can be formulated as combinatorial problems (NP-hard problems). A key challenge in solving combinatorial optimization problems is guiding the solution-searching agent (e.g., a solver) to efficiently explore the vast search space. Traditional methods typically rely on enumeration (e.g., exhaustive search, randomized, or tabu search) or iterative processes (e.g., iteratively exploring the search space from a given starting point). These methods are far from satisfactory because they are either enumerable, and therefore can only explore a small fraction of the space within feasible time and computational budgets, or they must restrict the exploration to a very limited region.

[0031] The embodiments described herein provide methods, systems, and apparatus for efficiently determining paths. In some embodiments, a value prediction network can be trained to filter path solution candidates before actually exploring the search space. The value prediction network can provide an approximate surface on the search space (e.g., a solution search space), which is implicitly defined by a black-box optimizing agent used in a given CVRP problem. From a plurality of path solution candidates, the value prediction network can select “promising” path solution candidates and discard those that are “unpromising.” A “promising” path solution candidate may refer to the starting point of an ideal solution actually found by the BOA in the search space and has a higher probability of being optimized to a point with optimal cost (e.g., a cost below a preset value). In this specification, BOA and path optimizer are used interchangeably.

[0032] Optimal route solutions can be provided to navigate one or more vehicles to multiple locations within an area. Vehicles may include cars, trucks, trains, ships, drones, robots, airplanes, etc. Vehicles may access these locations for various purposes, such as picking up and / or transporting goods and / or passengers. These locations can be connected by roads, highways, air routes, waterways, etc., to provide routes for vehicles. In some embodiments, optimal route solutions can be obtained for instances (VRPs or variants thereof) within an environment. For example, this environment could be a community, city, state, country, or other suitable geographic region. Route solutions in such an environment may include a set of routes for vehicles to travel. Traveling along each route may incur travel costs.

[0033] Route solutions may be limited by cost thresholds (e.g., budget) and may be subject to constraints such as time, distance traveled, vehicle weight, size, range of travel, other constraints, or any combination thereof. Optimizing route solutions can involve minimizing costs while adhering to constraints. Both travel costs and constraints can be instance-specific. In a business environment, a service provider may plan anticipated delivery times, dispatch service vehicles, or assign routes to delivery vehicles accordingly, based on the optimal route solution corresponding to that environment. For example, a logistics company may need to transport goods from one or more pickup locations to one or more delivery locations, thus having the objective of finding the optimal route for its fleet to access the pickup and unloading locations. In another example, a delivery company may only be able to deliver goods using a fleet with limited loading capacity. In yet another example, a delivery company may have time constraints, requiring goods to be delivered to delivery locations within a specified time. It may be necessary to identify the optimal route solution within a preset time window to ensure adherence to time constraints. In some cases, this time window may be seconds or even milliseconds, allowing the fastest route to be determined immediately after a user enters a navigation request into a map application. In some other cases, this time window may be minutes or even hours, allowing the planning of the optimal delivery route for the following day's delivery. Alternatively, in some embodiments, the environment may include an integrated circuit on which the determination of the optimal wiring solution is similar to that of the path solution.

[0034] Figure 1 An environment associated with a routing problem or a variant thereof, according to some embodiments, is illustrated. In some embodiments, instance 110 includes one or more warehouses 101, multiple locations 180 (e.g., houses waiting for delivery from the aforementioned one or more warehouses, or houses requesting pickup from the aforementioned one or more warehouses), and one or more vehicles 102 accessible to the multiple locations via various routes. Each route may navigate along one or more roads, highways, air routes, waterways, other suitable passages connecting at least two locations, or any combination thereof. The objective of route optimization may be to find an optimal route solution to minimize the vehicle's travel costs while satisfying various constraints such as time, travel distance, vehicle weight, size, range of travel, power (e.g., gasoline, electricity) costs, other constraints, or any combination thereof. Travel costs and these constraints can be instance-specific.

[0035] In some embodiments, the environment may include a computing system 120 containing one or more computing devices, such as servers in a server cluster. The computing system 120 may acquire instance 110 and represent it as a computer-readable data structure 130. Data structure 130 may include a depot node 150 corresponding to a warehouse, multiple locations 160 corresponding to multiple sites 180, and multiple routes 170 between locations 160. The multiple routes 170 may be associated with features related to travel costs (e.g., distance, tolls, traffic), road conditions, speed limits, and the number of traffic lights and stop signs. Paths in the routing solution may begin from a starting point such as depot 150, include one or more of the multiple routes 170 connected to each other, and cover one or more of the multiple locations 160. Each route 170 may allow one-way or two-way traffic. Each location 160 may be associated with some features including static and dynamic characteristics. Static characteristics can be independent of the route solution and include, for example, location information (e.g., map coordinates or address of location 160), demand information (e.g., the number of packages to be delivered or picked up, the total size or weight of the packages, and the lenient or strict estimated arrival time for delivery or pickup), other suitable information, or any combination thereof. Dynamic characteristics can be specific to the route solution and include, for example, location information associated with one or more adjacent locations along the route in the current route solution, the remaining capacity of a vehicle when leaving a location, other suitable information, or any combination thereof.

[0036] In some embodiments, the computing system 120 may determine one or more path solutions (e.g., optimal paths) based on data structure 130 and / or other information. (See below for reference.) Figure 2An example describing a path solution is provided. Computing system 120 can provide one or more path solutions to one or more vehicles 102 (e.g., an onboard computer), one or more computing devices (e.g., a computer in a path control center, a mobile phone of a vehicle driver), etc., to perform path planning. In this environment, each system and device may be equipped with suitable software (e.g., an application programming interface) and / or hardware (e.g., wired, wireless connectivity) to access other devices in the environment. Generally, systems and devices are able to communicate with each other via one or more wired or wireless networks (e.g., the Internet), through which data can be transferred. Each system and device may include one or more processors and one or more memories coupled to the processors. The memories may be non-transitory and computer-readable, and configured with instructions executable by one or more processors to cause the processors to perform the operations described herein. These instructions may be stored in the memories or may be downloaded via a communication network without needing to be stored in the memories. Although systems and devices are shown as separate components in this figure, it should be understood that these devices may be implemented as a single device or multiple devices coupled together.

[0037] Figure 2 Examples of path solutions for path problems or variant instances thereof, according to some embodiments, are shown. Figure 1 For example, an instance (represented by computer-readable data structure 130) may require a corresponding routing solution 220 to satisfy one or more instance-specific constraints, such as time constraints, travel distance constraints, vehicle capacity constraints, and power cost constraints. For instance, a company might require service to some users before the end of the day (e.g., buyers have booked same-day delivery), then the routing solution can be determined to include one or more routes that allow vehicle 240 to visit all these users before the end of the day. As shown, three vehicles 240 can cover all locations via routes 230a, 230b, and 230c, respectively.

[0038] When determining one or more routes, user location, traffic information, vehicle capacity (size limitations, weight limitations, travel limitations), characteristics of the packages to be served (such as frozen items, fragile items, size, weight), other suitable information, or any combination thereof may be considered. In another example, the service provider may require that the cost of providing services to all users be lower than a preset budget. This cost may include power costs (such as gasoline costs, electricity costs, other types of energy costs, or any combination thereof), labor costs (such as driver wages, food, accommodation, other types of costs, or any combination thereof), tolls, other types of costs, or any combination thereof. Such a route solution for the service provider may be determined to include one or more routes (230a, 230b, 230c) such that the total cost of the solution (the sum of the costs associated with each route) is lower than the budget. The cost associated with each route may have other constraints, such as time, distance traveled, vehicle weight, size, travel range, power (e.g., gasoline, electricity) costs, other constraints, or any combination thereof. In some embodiments, vehicle 240 is consistent in terms of vehicle capacity, condition, and fuel efficiency. In other embodiments, vehicle 240 may be diverse in terms of vehicle capacity, condition, and fuel efficiency. In some embodiments, instance 110 may have multiple constraints to be satisfied. For example, instance 110 may have a first budget for the number of vehicles to be used, a second budget for service time, and a third budget for total monetary expenditure. For example, the cost constraint is to deliver 100 packages by the end of the day using no more than 5 vehicles with minimal fuel and labor costs.

[0039] Figure 3 Exemplary operations for optimizing path solutions are illustrated according to some embodiments. For example... Figure 3 As shown, these operations can include changing the order of at least two positions among multiple locations on one of the aforementioned paths; and moving a position on one of the paths to another of the paths. For example, after applying the "swap" action 312, path 1 in box 310 is updated to path 1' in box 320. In this way, two positions among multiple locations swap their positions (service order) in the path. In another example, after applying the "move" action 332, paths 2 and 3 in box 330 are updated to paths 2' and 3' in box 340. In this way, two positions on path 2 are repositioned to path 3. Although only one action is shown for each improvement action here, an improvement action can include one or more "swap" actions, one or more "move" actions, other suitable actions, or any combination thereof.

[0040] Figure 3The two operations shown are exemplary operations that BOA can choose to optimize a given pool of candidate path solutions. In some embodiments, BOA may include a state-action policy trained on a reinforcement learning (RL) algorithm to identify and execute the optimal action from the action pool in response to a given state of the environment. In the context of the VRP problem, the action pool may include multiple optimization actions (e.g., including...). Figure 3 The two actions shown and other suitable actions); the state of the environment may include the current path solution, some historical data (such as X improvement actions that led to the current path solution and their corresponding effects), other suitable information, or any combination thereof. The trained policy can guide BOA to continuously optimize a given path solution candidate until convergence (e.g., the solution cannot be further optimized). It can be noted that this convergence may refer to "local" convergence within the search space.

[0041] Figure 4 An exemplary system diagram for determining a path is shown according to some embodiments. Figure 4 The components of system 400 may include a black-box optimization agent (BOA) module 410, a learn to restart (L2R) module 420 (hereinafter referred to as L2R 420), and an exploration strategy module 430. Depending on the implementation, system 400 may have more than Figure 4 The more, fewer, or alternative components shown.

[0042] In some embodiments, the BOA module 410 may be assumed to be a classical optimizer, such as an RL-trained agent (a trained reinforcement learning agent) that takes the path solution candidate X to be optimized from the VRP instance. 0 Optimize it and return X * (For example, the optimized path can be represented as X) * |X 0 This indicates that the solution is based on X. 0 (Optimized). This manual does not restrict the training method for BOA 410. Figure 5 An exemplary training method is described.

[0043] In some embodiments, L2R 420 can be configured to approximate the behavior of BOA 410 for a given path solution candidate. For example, for a path solution candidate X to be optimized for a VRP instance. 0BOA 410 can implicitly define the search space; an L2R 420 can be trained to approximate the value surface within this search space and a value prediction network can be used to predict X. * |X 0 The objective function value is obtained without actually running BOA. This predicted value can be expressed as:

[0044]

[0045] For multiple candidate path solutions to be optimized, It is possible to predict which of these paths will likely lead to a better outcome path solution for BOA 410 (e.g., the optimized version provides the optimal cost after being optimized by BOA). In some embodiments, L2R 420 can be trained offline and used to screen a large number of path solution candidates before feeding “promising” path solution candidates into BOA 410 for actual optimization.

[0046] L2R 420 can be implemented in various ways. In some embodiments, in response to a given path solution candidate, L2R 420 can be trained to predict the final path cost (e.g., distance, time) associated with the path solution candidate after it has been optimized by BOA (but BOA has not actually been run). In some embodiments, L2R 420 can be trained to identify the most “promising” one from multiple path solution candidates without needing to accurately predict the optimized path cost of multiple path solution candidates. For example, L2R 420 can simply be trained to identify a path solution candidate that is better than another path solution candidate (e.g., more likely to be optimized into a better solution).

[0047] In some embodiments, Figure 4The exploration strategy module 430 coordinates the interaction between BOA 410 and L2R 420. For example, the exploration strategy module 430 may first obtain a set of candidate path solutions to be optimized corresponding to a path instance, and then send them to L2R 420 for screening in step 432. In some embodiments, L2R 420 may include a value prediction network to approximate the search behavior of BOA 410 and to approximately predict how well a given candidate path solution can be optimized and / or estimate how "promising" each candidate path solution is. Subsequently, the exploration strategy module 430 may identify a subset of the set of candidate path solutions to be optimized based on the screening results from L2R 420, for BOA 410 to perform actual optimization. For example, L2R 420 may directly return the subset in step 434, or it may return the scores of the set of candidate path solutions to be optimized, for the exploration strategy module 430 to determine the subset to be sent to BOA 410 in step 436. After optimizing the candidate path solutions selected in BOA 410, an optimized path solution with the lowest cost can be identified in step 437.

[0048] In some embodiments, the exploration strategy module 430 may also be configured to explore some candidate path solutions to be optimized before performing the above-described process. For example, the exploration strategy module 430 may first generate one or more random path solutions, which may result in completely unpredictable costs. These random path solutions may be sent to the BOA 410 for optimization. After the BOA 410 returns an optimized version of the converged random path solution, the exploration strategy module 430 may perturb these optimized versions in various ways to generate a set of candidate path solutions to be optimized (e.g., after each perturbation, the optimized version may become a candidate to be optimized). For simplicity, the candidate path solutions to be optimized generated by perturbing the optimized path solutions may be referred to as the "descendants" of the optimized path solutions. These "descendants" may guide the BOA 410 to explore different regions in the search space to explore ideal path solutions.

[0049] In some embodiments, such as Figure 5 As shown, the L2R 420 module may include a value prediction network in the form of a Siamese neural network.

[0050] Figure 5 An exemplary value prediction network for determining a path is shown according to some embodiments. Figure 5The value prediction network in BOA 410 can refer to a Siamese neural network that includes two identical neural networks, 502 and 505 (e.g., sharing the same weights / parameters / structure). For each candidate path solution to be optimized, the Siamese neural network can be trained to predict which candidate path solution to be optimized will have a lower cost (e.g., shorter travel distance) after being optimized by BOA 410.

[0051] In some embodiments, each of neural networks 502 and 505 may include a bidirectional LSTM layer with multiple LSTM (Long Short-Term Memory) units 510, followed by a fully connected layer. LSTM-based networks are well-suited for classification, processing, and prediction based on time-series data. In the context of VRP, a path solution may include multiple paths, each path comprising a time series corresponding to a travel plan (e.g., a vehicle visiting multiple locations within a time window). In some embodiments, each path in the path solution may be represented as a sequence of nodes corresponding to multiple customers. Each node may include customer-specific characteristics of the corresponding customer, such as location, demand, and characteristics of adjacent customers (e.g., distance to the previous node and distance to the next node). Figure 5 As shown, for a given path solution, each path can first be fed into LSTM block 510 for embedding.

[0052] Subsequently, the LSTM embeddings of each path can be concatenated and fed into attention layer 520, which may include several attention stacks. In some embodiments, the output of attention block 520 (e.g., the embeddings) can be sent to multilayer perceptron (MLP) layer 550 to generate a score for the input path solution (e.g., a candidate path solution to be optimized). In some embodiments, in addition to the output of attention layer 520, MLP 550 may also consider multiple solution-specific features 530 associated with the input path solution, such as the sum and standard deviation of path distances in the path solution. These solution-specific features can be appended to the embeddings generated by attention layer 520 so that MLP 550 generates a score for the input path solution. Here, the score may be a representation of the expected cost associated with a hypothetical optimized version of the input path solution.

[0053] Training data for training neural networks 502 and 505 can be obtained in various ways. In some embodiments, because different VRP instances can have different instance-specific features, some VRP instances can be sampled so that the trained neural network can automatically adapt to different VRP instances. For each VRP instance, at least two path solutions to be optimized can be randomly sampled as a training sample and then sent individually to BOA 410 for optimization. The resulting cost can be used to label the training sample (e.g., a path solution to be optimized). For example, if the first path solution to be optimized produces a lower cost, the training sample can be labeled as 1; otherwise, it can be labeled as 0.

[0054] In some embodiments, to obtain high-quality (e.g., more representative) training samples, a path solution to be optimized can be abandoned if the difference between the corresponding outcome costs is less than a preset threshold. In some embodiments, the process of acquiring training data can be performed offline and in parallel. For example, if BOA does not require a GPU, multiple CPUs can be used (e.g., BOA can be hosted by each CPU or each core on a CPU).

[0055] In some embodiments, the training process may be as follows: obtaining a training set including a third routing solution candidate and a fourth routing solution candidate to be optimized; inputting the training set into a path optimizer (e.g., BOA) to obtain a third score for the third routing solution candidate and a fourth score for the fourth routing solution candidate; inputting the training set into two value prediction networks respectively to obtain a fifth score for the third routing solution candidate and a sixth score for the fourth routing solution candidate; and adjusting the weights of the two value prediction networks based at least on the third, fourth, fifth, and sixth scores.

[0056] During training, each training sample can be fed into neural networks 502 and 505 to obtain corresponding scores (e.g., the fifth and sixth scores). Based on the obtained scores, the probability that the first path solution in the training sample is more "promising" than another path solution can be obtained. This probability can be compared with the aforementioned labels associated with that training sample to determine the loss. For example, the obtained scores can be converted into two logistic regression (logit) values, and the cross-entropy loss function can be defined as:

[0057]

[0058] Where S is the training sample set, y sp is the label of the training sample s. s It is the available predicted probability that the first path solution is more "promising" than the other path solution in the training samples. In some embodiments, each of neural networks 502 and 505 can be trained using adaptive moment estimation (ADAM) with a learning rate of 0.0005.

[0059] Understandably, despite Figure 5 The Siamese neural network described includes two separate networks, 502 and 505, but it can include three or more networks, depending on implementation preferences and application scenarios. Assuming the Siamese neural network contains three networks, the training samples to be collected may need to contain three path solutions to be optimized, and the label of each training sample can be adjusted accordingly. For example, if the first solution in a sample achieves a score greater than the scores of the other two solutions, the training sample can be labeled as 2; if the first solution in a sample achieves a score greater than only one of the scores of the other two solutions, the training sample can be labeled as 1; otherwise, it may be labeled as 0.

[0060] Figure 6 An exemplary application of a trained value prediction network for path determination, according to some embodiments, is shown. The trained value prediction network 610 can refer to... Figure 5 The twin neural network described in the text comprises two trained neural networks. Figure 6 An exemplary application of this can begin by inputting multiple candidate path solutions to be optimized into a Siamese network 610 using the following method: for each candidate among the multiple candidate path solutions to be optimized, pairing the candidate with each other candidate among the multiple candidate path solutions to be optimized that is different from the candidate; and inputting the pairing of the candidate with each other candidate into two value prediction networks to determine the individual score of the candidate.

[0061] like Figure 6As shown, given a set of four path solution candidates, candidate #1, candidate #2, candidate #3, and candidate #4, each candidate can be paired with another candidate to form a pair 600. For example, candidate #1 can be paired with the other three candidates to form three pairs. Each of the three pairs can then be fed into a Siamese network 610, which can generate two scores for each pair. For example, when the pair of candidate #1 and candidate #2 is fed into the Siamese network 610, two scores can be generated for candidate #1 and candidate #2 respectively. If the score of candidate #1 is greater than the score of candidate #2, then candidate #1 can receive a positive individual score for “winning” the comparison (e.g., it is more “promising” than candidate #2 according to the trained Siamese network 610). For a simple example, a positive individual score could be 1. If candidate #1 loses the comparison because it is “less promising” than candidate #2, it may receive a non-positive individual score, such as 0 or -1. In this example, because candidate #1 participates in three comparisons (e.g., in three pairs), it can get three separate scores.

[0062] Subsequently, each of the multiple path solution candidates can obtain a total score 620 based on its individual scores. For example, the total score 620 of candidate #1 could be the sum of its three individual scores obtained from the Siamese network 610. Finally, the path solution candidate 630 with the highest total score can be selected and fed into the path optimizer (e.g., BOA) for actual optimization. The optimized version of the selected path solution candidate can be determined as the optimal path solution to be executed.

[0063] In some embodiments, instead of selecting the path solution candidate with the highest total score, a preset number of path solution candidates with high total scores can be selected for optimization by the path optimizer. The final optimal path solution can be identified from the preset number of optimized path solutions.

[0064] Figure 7 An exemplary method 700 for determining a path is illustrated according to some embodiments. Method 700 can be derived from... Figure 1 The computing system 120 shown is Figure 4 The system implementation shown, and corresponding to Figure 1-6 The illustrated embodiment. Depending on the implementation, the method may include additional, fewer, or alternative steps.

[0065] Block 710 includes inputting a plurality of candidate path solutions to be optimized into a Siamese neural network, the Siamese neural network including a plurality of value prediction networks, each of which is trained to predict costs associated with the candidate path solutions to be optimized. In some embodiments, each candidate path solution to be optimized comprises: traversing one or more paths through multiple locations along a route, and the candidate path solution to be optimized is subject to one or more constraints, including one or more of the following: time constraints; travel distance constraints; vehicle capacity constraints; and power cost constraints. In some embodiments, the path optimizer includes a set of improved operators learned based on a reinforcement learning algorithm, and the set of improved operators performs one or more of the following operations: changing the order of at least two locations among the multiple locations in one of the one or more paths; and moving a location on one of the one or more paths to another of the one or more paths. In some embodiments, the plurality of value prediction networks in the Siamese neural network are identical.

[0066] Block 720 includes identifying one or more candidate paths for optimization from the plurality of candidate paths for optimization based on the output of the Siamese neural network. In some embodiments, the Siamese neural network includes two value prediction networks, and inputting the plurality of candidate paths for optimization into the Siamese neural network includes: for each candidate path for optimization in the plurality of candidate paths for optimization: pairing the candidate path for optimization with each other candidate path for optimization in the plurality of candidate paths for optimization that is different from the candidate path for optimization; and inputting the candidate path for optimization and each other candidate pairing into the two value prediction networks to determine a separate score for the candidate path for optimization. In some embodiments, inputting the candidate path solution to be optimized and each of the paired other candidate path solutions to be optimized into the two value prediction networks to determine an individual score for the candidate path solution to be optimized includes: obtaining a seventh score for the candidate path solution to be optimized and an eighth score for each of the paired other candidate path solutions to be optimized; determining a positive score for the candidate path solution to be optimized if the seventh score is greater than the eighth score; and determining a non-positive score for the candidate path solution to be optimized if the seventh score is not greater than the eighth score. In some embodiments, identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the Siamese neural network includes: for each candidate path solution to be optimized: determining a total score for the candidate path solution to be optimized, the total score including the sum of the individual scores of the candidate path solution to be optimized; and identifying the candidate path solution to be optimized with the highest total score from the plurality of candidate path solutions to be optimized.

[0067] Block 730 includes inputting the identified one or more path solution candidates to be optimized into a path optimizer to obtain one or more optimized path solution candidates.

[0068] Block 740 includes determining an optimized path solution with the lowest cost from the one or more optimized path solution candidates.

[0069] In some embodiments, the Siamese neural network includes two value prediction networks, and the method 700 further includes: training the two value prediction networks by performing one or more iterations of an adjustment process, wherein performing one or more iterations of the adjustment process includes: obtaining a training set including a third path solution candidate to be optimized and a fourth path solution candidate to be optimized; inputting the training set into the path optimizer to obtain a third score of the third path solution candidate to be optimized and a fourth score of the fourth path solution candidate to be optimized; inputting the training set into the two value prediction networks respectively to obtain a fifth score of the third path solution candidate to be optimized and a sixth score of the fourth path solution candidate to be optimized; and adjusting the weights of the two value prediction networks based at least on the third score, the fourth score, the fifth score, and the sixth score. In some embodiments, before inputting the training set into the two value prediction networks, the adjustment process further includes: determining whether the difference between the third score and the fourth score is greater than a preset threshold; and if the difference is not greater than the preset threshold, then discarding the third path solution candidate to be optimized and the fourth path solution candidate to be optimized. In some embodiments, adjusting the weights of the two value prediction networks includes: determining a label for the training set based on the third score and the fourth score; converting the fifth score and the sixth score into a fifth logistic regression value and a sixth logistic regression value; determining a cross-entropy loss function based on the label, the fifth logistic regression value, and the sixth logistic regression value; and adjusting the weights of the two value prediction networks based on the cross-entropy loss function. In some embodiments, each of the plurality of value prediction networks includes: a bidirectional long short-term memory (LSTM) layer comprising a plurality of LSTM units; an attention layer for embedding the output from the plurality of LSTM units; and an output layer for generating a score based on the output from the attention layer and a plurality of features associated with a candidate path solution to be optimized. In some embodiments, the candidate path solution to be optimized includes a plurality of paths, each path associated with a distance, and the plurality of features associated with the candidate path solution to be optimized includes: the sum of the distances of the plurality of paths in the candidate path solution to be optimized; and the standard deviation of the distances of the plurality of paths in the candidate path solution to be optimized.

[0070] Figure 8 A block diagram of a computer system 800 for determining a path is shown according to some embodiments. The components of the computer system 800 presented below are intended for illustration. Depending on the implementation, the computer system 800 may include additional, fewer, or alternative components.

[0071] Computer systems can provide Figure 1-7 Exemplary implementations of the systems, operations, and methods shown are provided. Computer system 800 may include one or more processors and one or more non-transitory computer-readable storage media (e.g., one or more memories) coupled to the one or more processors and configured with instructions executable by the one or more processors to cause a system or device (e.g., a processor) to perform the methods described above, such as method 700. Computer system 800 may include various units / modules corresponding to the instructions (e.g., software instructions).

[0072] In some embodiments, the computer system 800 may be referred to as an apparatus for determining a path. The apparatus may include an input module 810, an identification module 820, an optimization module 830, and a determination module 840. In some embodiments, the input module 810 may input a plurality of candidate path solutions to be optimized into a Siamese neural network, the Siamese neural network including a plurality of value prediction networks, each of which is trained to predict the cost associated with the candidate path solutions to be optimized. In some embodiments, the identification module 820 may identify one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the output of the Siamese neural network. In some embodiments, the optimization module 830 may input the identified one or more candidate path solutions to be optimized into a path optimizer to obtain one or more optimized candidate path solutions. In some embodiments, the determination module 840 may determine the optimized path solution with the lowest cost from the one or more optimized candidate path solutions.

[0073] Figure 9 A block diagram of a computer system 900 that can implement any of the embodiments described herein is shown. The computer system 900 can... Figure 1-8 Implemented in any component of the environment and system shown. Figure 1-8 The one or more example methods shown can be performed by one or more implementations of the computer system 900.

[0074] Computer system 900 may include bus 902 or other communication mechanisms for communicating information, and one or more hardware processors 904 coupled to bus 902 for processing information. Hardware processor 904 may be, for example, one or more general-purpose microprocessors.

[0075] Computer system 900 may also include main memory 906, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 902 for storing information and instructions executable by processor 904. Main memory 906 may also be used to store temporary variables or other intermediate information during the execution of instructions executable by processor(s) 904. When stored in a storage medium accessible by processor(s) 904, such instructions render computer system 900 to a dedicated machine tailored to perform the operations specified in the instructions. Computer system 900 may also include read-only memory (ROM) 908 or other static devices coupled to bus 902 for storing information and instructions executable by processor 904. Storage device 910, such as a disk, optical disk, or USB thumb drive (flash drive), may be provided, and storage device 910 may be coupled to bus 902 to store information and instructions.

[0076] Computer system 900 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic, which, when combined with the computer system, enables computer system 900 to be a special-purpose machine or to be programmed as such. According to one embodiment, computer system 900 performs the operations, methods, and processes described herein in response to processor 904 executing one or more sequences of one or more instructions contained in main memory 906. These instructions may be read into main memory 906 from another storage medium, such as storage device 910. Executing the sequence of instructions contained in main memory 906 causes processor 904 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0077] Main memory 906, ROM 908, and / or storage device 910 may include non-transitory storage media. The term "non-transitory media" and similar terms, as used herein, refer to a medium that stores data and / or instructions that cause a machine to operate in a particular manner, excluding transient signals. The non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 910. Volatile media include dynamic memory, such as main memory 906. Common forms of non-transitory media may include, for example, floppy disks, floppy disk drives, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or memory cartridges, and their network versions.

[0078] Computer system 900 may include a network interface 918 coupled to bus 902. Network interface 918 may provide bidirectional data communication coupling for one or more network links connected to one or more local networks. For example, network interface 918 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connections to a corresponding type of telephone line. As another example, network interface 918 may be a Local Area Network (LAN) card to provide data communication connections to a compatible LAN (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, network interface 918 may transmit and receive electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0079] Computer system 900 can send messages and receive data, including program code, through (multiple) networks, network links, and network interface 918. In the Internet example, the server can transmit request code for the application through the Internet, an ISP (Internet Service Provider), a local network, and network interface 918.

[0080] The received code can be executed by processor 904 upon receipt and / or stored in storage device 910 or other non-volatile memory for subsequent execution.

[0081] Each process, method, and algorithm described in the foregoing sections can be implemented in a code module executed by one or more computer systems or a computer processor including computer hardware, and is fully or partially automated. The processes and algorithms can be implemented partially or wholly in dedicated circuitry.

[0082] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this application. Furthermore, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the blocks or states associated with them may be executed in other suitable orders. For example, the described blocks or states may be executed in a different order than specifically disclosed, or multiple blocks or states may be combined in a single block or state. Examples of blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. Examples of systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged compared to the disclosed embodiments.

[0083] The various operations of the methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation that runs to perform one or more of the operations or functions described herein.

[0084] Similarly, the methods described herein can be implemented at least in part by a processor, where one or more specific processors are examples of hardware. For example, at least some operations of a method can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also operate to support the performance of the relevant operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a set of computers (as an example of a machine including processors) that are accessible via a network (e.g., the Internet) and one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0085] The performance of certain operations can be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0086] Throughout this specification, multiple instances can implement components, operations, or structures described as single instances. Although the individual operations of one or more methods are shown and described as separate operations, one or more of these single operations may be executed concurrently, and these operations are not required to be executed in the order shown. Structures and functionalities presented as separate components in a configuration can be implemented as composite structures or components. Similarly, structures and functionalities presented as single components can be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0087] Although an overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the embodiments of this application. The detailed description should not be construed as limiting, and the scope of the various embodiments is defined only by the appended claims together with the full scope of the subject matter equivalent to those claims. Furthermore, the related terms used herein (such as “first,” “second,” and “third,” etc.) do not indicate any order, hierarchy, or importance, but are used to distinguish one element from another. Additionally, the terms “a,” “an,” and “a plurality” do not indicate a limitation on quantity, but rather indicate the presence of at least one of the mentioned items.

Claims

1. A computer-executed method for determining a path, comprising: Multiple candidate path solutions to be optimized are input into a Siamese neural network, which includes multiple value prediction networks, each of which is trained to predict the cost associated with a candidate path solution to be optimized, as well as the degree to which a candidate path solution to be optimized is optimized and / or the probability that a candidate path solution to be optimized is optimized into the optimal path solution. Based on the output of the Siamese neural network, one or more path solution candidates to be optimized are identified from the plurality of path solution candidates to be optimized; The identified one or more candidate path solutions to be optimized are input into the path optimizer to obtain one or more optimized candidate path solutions. as well as From the one or more optimized path solution candidates, determine the optimized path solution with the lowest cost.

2. The method according to claim 1, wherein, Each of the plurality of candidate route solutions to be optimized comprises one or more routes that allow one or more vehicles to pass through multiple locations, and the candidate route solution to be optimized is subject to one or more constraints, the constraints including one or more of the following: Time constraints; Driving distance constraints; Vehicle capacity constraints; and Energy cost constraints.

3. The method according to claim 2, wherein, The path optimizer includes a set of improved operators learned based on a reinforcement learning algorithm, and the set of improved operators performs one or more of the following operations: Changing the order of at least two of the plurality of positions, wherein the at least two positions are on one of the one or more paths; and Move the position on one of the one or more paths to another of the one or more paths.

4. The method according to claim 1, wherein, The multiple value prediction networks in the twin neural network are identical.

5. The method according to claim 1, wherein, The twin neural network comprises two value prediction networks, and the method further includes: The two value prediction networks are trained by performing one or more iterations of the adjustment process, wherein the one or more iterations of the adjustment process include: Obtain a training set that includes the third and fourth candidate path solutions to be optimized; The training set is input into the path optimizer to obtain the third score of the third candidate path solution to be optimized and the fourth score of the fourth candidate path solution to be optimized. The training set is input into the two value prediction networks respectively to obtain the fifth score of the third candidate path solution to be optimized and the sixth score of the fourth candidate path solution to be optimized; and The weights of the two value prediction networks are adjusted based on at least the third score, the fourth score, the fifth score, and the sixth score.

6. The method according to claim 5, wherein the adjustment process further comprises: before inputting the training set into the two value prediction networks: Determine whether the difference between the third score and the fourth score is greater than a preset threshold; as well as If the difference is not greater than the preset threshold, then the third and fourth candidate path solutions to be optimized are discarded.

7. The method according to claim 5, wherein, The adjustment of the weights of the two value prediction networks includes: The label of the training set is determined based on the third score and the fourth score; The fifth score and the sixth score are converted into a fifth logistic regression value and a sixth logistic regression value; The cross-entropy loss function is determined based on the label, the fifth logistic regression value, and the sixth logistic regression value; and The weights of the two value prediction networks are adjusted based on the cross-entropy loss function.

8. The method according to claim 4, wherein, Each of the plurality of value prediction networks includes: A bidirectional long short-term memory (LSTM) layer, comprising multiple LSTM units; An attention layer is used to embed the outputs from the plurality of LSTM units; and An output layer is used to generate a score based on the output from the attention layer and multiple features associated with the candidate path solution to be optimized.

9. The method according to claim 8, wherein, The candidate path solutions to be optimized include multiple paths, each associated with a distance. The multiple features associated with the candidate path solutions to be optimized include: The sum of the distances of the multiple paths in the candidate path solutions to be optimized; and The standard deviation of the distances of the multiple paths in the candidate path solutions to be optimized.

10. The method according to claim 1, wherein, The Siamese neural network includes two value prediction networks, and the input of multiple candidate path solutions to be optimized into the Siamese neural network includes: For each of the plurality of candidate path solutions to be optimized: The candidate path solution to be optimized is paired with each of the plurality of candidate path solutions that is different from the candidate path solution to be optimized; and The candidate path solution to be optimized and each of the other paired candidates are input into the two value prediction networks to determine the individual score of the candidate path solution to be optimized.

11. The method of claim 10, wherein inputting the candidate path solution to be optimized and each of the paired other candidates path solutions to be optimized into the two value prediction networks to determine a separate score for the candidate path solution to be optimized comprises: Obtain the seventh score of the candidate path solution to be optimized and the eighth score of each of the other candidate path solutions to be optimized in the pair; If the seventh score is greater than the eighth score, then a positive score is determined for the candidate path solution to be optimized; as well as If the seventh score is not greater than the eighth score, then a non-positive score is determined for the candidate path solution to be optimized.

12. The method according to claim 10, wherein, The step of identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the Siamese neural network includes: For each of the plurality of candidate path solutions to be optimized: Determine the total score of the candidate path solution to be optimized, the total score comprising the sum of the individual scores of the candidate path solution to be optimized; and From the multiple candidate path solutions to be optimized, the candidate path solution with the highest total score is identified.

13. A system for determining a path, comprising one or more processors and one or more non-transitory computer-readable memories, the memories being coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations including: Multiple candidate path solutions to be optimized are input into a Siamese neural network, which includes multiple value prediction networks, each of which is trained to predict the cost associated with a candidate path solution to be optimized, as well as the degree to which a candidate path solution to be optimized is optimized and / or the probability that a candidate path solution to be optimized is optimized into the optimal path solution. Based on the output of the Siamese neural network, one or more path solution candidates to be optimized are identified from the plurality of path solution candidates to be optimized; The identified one or more candidate path solutions to be optimized are input into the path optimizer to obtain one or more optimized candidate path solutions. as well as From the one or more optimized path solution candidates, determine the optimized path solution with the lowest cost.

14. The system according to claim 13, wherein, The twin neural network comprises two value prediction networks, and the operation further includes: The two value prediction networks are trained by performing one or more iterations of the adjustment process, wherein the one or more iterations of the adjustment process include: Obtain a training set that includes the third and fourth candidate path solutions to be optimized; The training set is input into the path optimizer to obtain the third score of the third candidate path solution to be optimized and the fourth score of the fourth candidate path solution to be optimized. The training set is input into the two value prediction networks respectively to obtain the fifth score of the third candidate path solution to be optimized and the sixth score of the fourth candidate path solution to be optimized; and The weights of the two value prediction networks are adjusted based on at least the third score, the fourth score, the fifth score, and the sixth score.

15. The system of claim 14, wherein the adjustment process further comprises, before inputting the training set into the two value prediction networks: Determine whether the difference between the third score and the fourth score is greater than a preset threshold; as well as If the difference is not greater than the preset threshold, then the third and fourth candidate path solutions to be optimized are discarded.

16. The system according to claim 14, wherein, The Siamese neural network includes two value prediction networks, and the input of multiple candidate path solutions to be optimized into the Siamese neural network includes: For each of the plurality of candidate path solutions to be optimized: The candidate path solution to be optimized is paired with each of the plurality of candidate path solutions that is different from the candidate path solution to be optimized; and The candidate path solution to be optimized and each of the other paired candidates are input into the two value prediction networks to determine the individual score of the candidate path solution to be optimized.

17. The system according to claim 16, wherein, The step of identifying one or more candidate path solutions to be optimized from the plurality of candidate path solutions to be optimized based on the Siamese neural network includes: For each of the plurality of candidate path solutions to be optimized: Determine the total score of the candidate path solution to be optimized, the total score comprising the sum of the individual scores of the candidate path solution to be optimized; and From the multiple candidate path solutions to be optimized, the candidate path solution with the highest total score is identified.

18. A non-transitory computer-readable storage medium for determining a path, configured with instructions executable by one or more processors to cause the one or more processors to perform operations, the operations including: Multiple candidate path solutions to be optimized are input into a Siamese neural network, which includes multiple value prediction networks, each of which is trained to predict the cost associated with a candidate path solution to be optimized, as well as the degree to which a candidate path solution to be optimized is optimized and / or the probability that a candidate path solution to be optimized is optimized into the optimal path solution. Based on the output of the Siamese neural network, one or more path solution candidates to be optimized are identified from the plurality of path solution candidates to be optimized; The identified one or more candidate path solutions to be optimized are input into the path optimizer to obtain one or more optimized candidate path solutions. as well as From the one or more optimized path solution candidates, determine the optimized path solution with the lowest cost.

19. The storage medium according to claim 18, wherein, The twin neural network comprises two value prediction networks, and the operation further includes: The two value prediction networks are trained by performing one or more iterations of the adjustment process, wherein the one or more iterations of the adjustment process include: Obtain a training set that includes the third and fourth candidate path solutions to be optimized; The training set is input into the path optimizer to obtain the third score of the third candidate path solution to be optimized and the fourth score of the fourth candidate path solution to be optimized. The training set is input into the two value prediction networks respectively to obtain the fifth score of the third candidate path solution to be optimized and the sixth score of the fourth candidate path solution to be optimized; and The weights of the two value prediction networks are adjusted based on at least the third score, the fourth score, the fifth score, and the sixth score.

20. The storage medium of claim 19, wherein the adjustment process further comprises, before inputting the training set into the two value prediction networks: Determine whether the difference between the third score and the fourth score is greater than a preset threshold; as well as If the difference is not greater than the preset threshold, then the third and fourth candidate path solutions to be optimized are discarded.

Images