A controller for optimizing motion trajectories to control the motion of one or more devices.

Abstract

To provide a controller, a method, and a storage medium for optimizing a motion trajectory to control motion of devices.SOLUTION: A method comprises: a step 110 for inputting a task parameter including at least one device state into a neural network trained to output an estimated motion trajectory for performing a task; a step 120 for extracting at least some of integer values of a solution to a mixed integer optimization problem for planning execution of the task resulting in the estimated motion trajectory; a step 130 for solving the mixed integer optimization problem for the task parameter with the corresponding integer values fixed to the extracted integer values; a step 135 for generating a constrained optimized motion trajectory thereby; and a step 140 for changing at least one device state to track the optimized motion trajectory.SELECTED DRAWING: Figure 1B

Description

Technical Field

[0001] The present disclosure relates to a control system for planning the motion of a plurality of moving devices, and more particularly to a controller for optimizing a motion trajectory to control the motion of one or more devices.

Background Art

[0002] By optimally planning the motion of a plurality of devices moving within a work space having obstacles, there are a plurality of applications such as a wheeled ground robot, a ground aircraft, and a flying drone, for reaching a specific position while avoiding collisions with obstacles and collisions between devices. As an example, an aerial drone can be adjusted to reach a position and inspect a structure while avoiding collisions with other structures and collisions between drones. An aircraft can be adjusted to move on the ground of an airport and reach its assigned runway or gate while avoiding other aircraft, carts, and structures. A mobile robot on a factory floor can be adjusted to reach its work position or a position where unloading is required while avoiding work areas, human workers, and other robots. However, such planning is difficult. This is because the presence of obstacles makes the accessible work space and the configurations that the moving devices can adopt without collision non-convex, and non-convex problems require a large amount of time and computational resources to solve.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Generally, mixed-integer programming (MIP) is a method that can solve the above planning problems. MIP methods involve solving constrained optimization problems, where the cost function is minimized / maximized under several constraints, some optimization variables are real-valued (continuous), and others are integers or binary values ​​(discrete). In particular, mixed-integer linear / quadratic programming (MILP / MIQP) can be applied when the device motion can be explained by linear mechanics, and obstacles can sometimes be represented by polygons (i.e., by linear inequalities) in a traditional way. MILP / MIQP is known to complete in a finite time, thus allowing for the derivation of a time range for computing the solution. However, this computation is still very complex (NP-hard) and requires a vast amount of computational resources, memory, and operations. Instead, a central control system to tune the device often needs to be implemented in an embedded processing unit (EPU) with limited memory and computational power. As a result, a solution is needed that can optimize the motion planning of multiple devices in an obstacle-filled workspace while utilizing the limited computational and memory resources specific to the EPU, which means reducing the computational and memory requirements for solving MIP.

[0004] Some existing solutions to reduce the computational requirements for solving MIPs involve using machine learning (ML) to learn how to approximate MIP solutions. In such cases, training data is generated by solving several MIPs, and then this is used to train an ML method, for example, a neural network (NN). During execution, instead of solving the MIP problem, the trained NN is evaluated to obtain an approximate solution. However, this motion planning method can result in collisions between one or more devices, because machine learning approximates actions for obstacle avoidance. Avoiding obstacles requires discontinuous trajectories relative to the initial device positions, because from some device positions, the obstacle is avoided by circling it from the left, and from different device positions, it is avoided by circling it from the right. From a region between such two device positions, a small perturbation to the left causes the device to follow a path circling from the left, and a small perturbation to the right causes the device to follow a path circling from the right. On the other hand, general ML architectures generate continuous signals. This means that small deviations from the input vector, for example, from the device and obstacle positions, will result in small changes in the output vector and device trajectory. Therefore, since general ML architectures perform trajectory interpolation used for training, they cannot always achieve a trajectory that avoids obstacles. Interpolating a trajectory around the obstacle from the left and other trajectories around the obstacle from the right will always result in a trajectory that goes straight through the obstacle from a certain initial position of the device, and thus a trajectory that causes a collision. Thus, embodiments of this disclosure are based on the recognition that collisions will occur under certain initial conditions when ML is used alone.

[0005] Other existing solutions involve using the trajectory obtained from ML through a so-called warm start of the MIP solving algorithm as an initial guess for solving the MIP problem. For example, the solution of the MIP problem is based on a branch-bound optimization algorithm, and the initial solution guess can be used to obtain an upper bound of the optimal value, which allows avoiding exhaustive searching of the branch-bound tree. Thus, the number of real-valued problem relaxations in the branch-bound algorithm is reduced, and therefore the computation and memory required to solve the MIP are also reduced. However, the warm start is effective in restricting the optimal MIP solution only if the ML solution is feasible, and this does not apply to obstacle-colliding solutions where the obstacle-colliding solution can be generated by applying ML to the discontinuous problem. Furthermore, even with a warm start, it is still necessary to solve the original MIP. Thus, while obtaining the solution may be faster to some extent, in some cases it may not be faster, or the solver may take longer, for example, if the warm start suggests searching branches of a different tree than the branch where the solution lies, and a general heuristic excludes such branches in the root.

[0006] Therefore, embodiments of this disclosure are based on the recognition that even with a warm start, solving the MIP problem often still requires a large amount of computing resources, memory, and operations performed.

[0007] Therefore, there is still a need to solve motion planning problems with reliable, low computational and memory requirements for various configurations of devices, obstacles, and destinations. [Means for solving the problem]

[0008] Therefore, the objective of some embodiments is to solve the optimization problem of the motion trajectory of one or more devices. In particular, the objective of some embodiments is to optimize the motion trajectory of one or more constrained devices by solving the MIP problem.

[0009] Some embodiments are based on the understanding that machine learning (ML) algorithms can be used to obtain fast and reliable solutions to MIP problems for planning the motion of one or more devices so that obstacles are avoided and they reach a desired destination.

[0010] Some embodiments are based on the understanding that trajectory predictions can be obtained for each constrained device by training an ML module, for example a neural network, on a selected set of training data. Rather than using the trajectory integrally (for example, in ML-only methods, or in methods that use predictions as initial guesses for the MIP problem, or in warm-start methods), some embodiments are based on the understanding that predicted values ​​can be extracted for specific variables (in particular, all or at least some of the integer variables). Furthermore, the specific variables may be fixed to such predicted values ​​in the MIP problem. As a result, a reduced problem can be obtained in which some variables are already determined. Furthermore, by solving the reduced problem with a simpler optimization method, a complete solution to the multi-device motion planning problem can be obtained, which can then be used directly or improved upon to create a complete solution to the MIP problem. In this way, some embodiments transform a mixed-integer optimization problem into a real-valued optimization problem by fixing the integer variables of the MIP problem to extracted integer values.

[0011] In some embodiments, the MIP problem is formulated to find a global optimal solution within a search space determined by constraints. Furthermore, the admissibility of values ​​for at least some integer variables that partition the search space of the mixed-integer optimization is evaluated based on the predicted trajectory for performing the task. In some embodiments, at least some of the integer variables are binary. In some cases, at least some of the integer variables are associated with a group of separation constraints for real-valued variables to avoid collisions between devices and obstacles and collisions between multiple devices. If one of the constraints for the real-valued variables within the group of separation constraints is satisfied at a particular time, then a collision with the corresponding obstacle or device at that particular time is avoided.

[0012] In some embodiments, the tolerance of values ​​for at least some integer variables in the MIP problem is tested by determining the membership of at least some portions of the predicted trajectory into a region determined by constraints of real-valued variables that define obstacles or other devices among a plurality of devices.

[0013] Some embodiments are based on the understanding that integer values ​​can be extracted when the ML module estimates feasible trajectories for each controlled device. Therefore, it is necessary to improve the likelihood of feasible predicted trajectories by following the training data, training loss function, and variable fixing.

[0014] In some embodiments, a mixed-integer optimization problem is warm-started to update the optimized trajectory based on the optimized motion trajectory. The optimized motion trajectory is updated in response to detection of time availability before the point in time when the calculation of the optimized trajectory needs to be completed.

[0015] Some embodiments are based on the understanding that, in order for the ML module to be more accurate when the probability of collision is higher, more information about the vicinity of obstacles can be obtained by using focused sampling instead of uniform sampling in the selection of training data. Some embodiments are based on the understanding that, for training the ML module, the loss function may include additional terms that support the training error deviating from the constraints rather than being neutral with respect to the constraints. If the predicted trajectory is not feasible (i.e., no collision occurs), a feasible trajectory can be reconstructed by moving according to a portion of the predicted trajectory planned before the collision. Furthermore, the ML module may be re-evaluated from a new position for each device, and in some cases, this process may be continued for each device until the resulting trajectory is feasible, ultimately concatenating a contiguous portion of the predicted trajectory.

[0016] Accordingly, one embodiment discloses a controller for controlling the motion of at least one device, wherein the at least one device performs a task of changing the state of at least one device, the state of at least one device includes at least one position of at least one device that is constrained to the motion of at least one device, and the controller comprises a processor and a memory storing instructions, which, when executed by the processor, causes the controller to input the parameters of the task, including the state of at least one device, into a neural network trained to output an estimated motion trajectory for performing the task; extract at least some integer values ​​of a solution to a mixed-integer optimization problem to plan the execution of the task resulting in an estimated motion trajectory; generate a constrained optimized motion trajectory by solving the mixed-integer optimization problem for the parameters of the task using corresponding integer values ​​fixed to the extracted integer values; and change the state of at least one device to track the optimized motion trajectory. [Brief explanation of the drawing]

[0017] [Figure 1A] This figure shows a controller for controlling a device according to some embodiments of the present disclosure. [Figure 1B] A block diagram of a method for controlling the motion of a device, according to some embodiments of the present disclosure, is shown. [Figure 2] This figure shows a multi-device motion planning problem according to some embodiments of the present disclosure. [Figure 3] A schematic diagram of components included in multi-device motion planning according to some embodiments of this disclosure is shown. [Figure 4] A schematic diagram of a multi-device planning system for controlling one or more constrained devices, according to some embodiments of this disclosure, is shown. [Figure 5] This figure shows collision avoidance constraints for the motion of one or more devices according to some embodiments of the present disclosure. [Figure 6] A schematic diagram of a solution method for a mixed integer programming (MIP) problem via a branch and bound method for planning multi-device motion, according to some embodiments of this disclosure, is shown. [Figure 7] The following are functional block diagrams for solving a multi-device motion planning problem based on a machine learning (ML) module, according to some embodiments of this disclosure. [Figure 8] This figure shows a collision between multiple devices caused by the approximate behavior of an ML module when predicting discontinuous trajectories using a continuous prediction model in a collision avoidance mode, according to some embodiments of the present disclosure. [Figure 9] This figure shows an approximation of a predictive model that predicts the continuous variable of a discontinuous function in a collision avoidance mode, according to some embodiments of this disclosure. [Figure 10]A flowchart for solving a multi-device planning problem by warm-starting a MIP solver module using an ML module according to some embodiments of the present disclosure is shown. [Figure 11] A flowchart for solving a multi-device planning problem by predicting a trajectory using an ML module according to some embodiments of the present disclosure is shown. [Figure 12A] A schematic diagram of data collection using centralized sampling according to some embodiments of the present disclosure is shown. [Figure 12B] A diagram showing various centralized sampling methods according to an embodiment of the present disclosure. [Figure 13A] A diagram showing a learning algorithm for a prediction model according to some embodiments of the present disclosure. [Figure 13B] A schematic diagram showing layers and signals included in a deep neural network of a prediction model according to some embodiments of the present disclosure is shown. [Figure 14] A diagram showing an error of a predicted trajectory regarding collision avoidance according to some embodiments of the present disclosure. [Figure 15] A diagram showing a barrier function for collision avoidance according to some embodiments of the present disclosure. [Figure 16] A flowchart of an algorithm for updating prediction model parameters when a loss function includes a barrier function term according to some embodiments of the present disclosure is shown. [Figure 17] A flowchart of an algorithm for predicting a trajectory in an ML module according to an embodiment of the present disclosure is shown. [Figure 18] A flowchart of an algorithm for obtaining values of integer variables from a trajectory predicted by an ML module according to some embodiments of the present disclosure is shown.

MODE FOR CARRYING OUT THE INVENTION

[0018] Embodiments of this disclosure disclose a system for generating motion plans for one or more autonomous devices such that each device reaches its corresponding destination, represented by a target location. Furthermore, the motion plan must avoid collisions between each device and obstacles in the environment, as well as collisions between any two different devices, at all points in time within a future planning time interval. Examples of devices include autonomous ground vehicles such as robots or automobiles in factory automation, aircraft on airport surfaces, and unmanned aerial vehicles such as drones for monitoring or inspecting infrastructure.

[0019] Figure 1A shows a schematic diagram of a controller 111 for controlling device 100 according to an embodiment of the present disclosure. Examples of device 100 may include, but are not limited to, autonomous vehicles, mobile robots, aerial drones, ground vehicles, aerial vehicles, water vehicles, and underwater vehicles. Figure 1 shows a schematic diagram of a quadcopter drone as an example of device 100 in an embodiment of the present disclosure. Device 100 includes actuators that cause motion of device 100 and sensors for sensing the environment and location of device 100. Hereinafter, device 100 will also be referred to as drone 100. As shown in Figure 1, the rotor 101 may be an actuator, and the sensor for sensing the environment may include a LiDAR (light detection and ranging) 102 and a camera 103. Furthermore, sensors for localization may include a GPS or indoor GPS 104. Such sensors may be integrated with an inertial measurement unit (IMU). The drone 100 also includes a communication transceiver 105 for sending and receiving information, and a control unit 106 for processing data acquired from sensors and the transceiver 105, calculating commands for actuators 101, and calculating data to be transmitted via the transceiver 105.

[0020] Furthermore, the controller 111 is configured to control the motion of the drone 100 by calculating a motion plan for the drone 100 based on information transmitted from the drone 100. The motion plan for the drone 100 may include one or more trajectories in which the drone moves. In some embodiments, there exists one or more devices (such as the drone 100 shown in Figure 1) whose motion is coordinated and controlled by the controller 111. Controlling and coordinating the motion of one or more devices is equivalent to solving a mixed-integer optimization problem.

[0021] The controller 111 controls the motion of the drone 100 to perform a task by changing the state of the drone 100, and the state of the drone 100 may include the position of the drone 100, which is constrained on the motion of the drone 100. According to some embodiments, the controller 111 controls the motion of the drone 100 by using a neural network 112 that limits the search space for finding the motion trajectory. The neural network 112 may be a probabilistic or deterministic neural network.

[0022] In different embodiments, the controller 111 obtains task parameters from the drone 100 and / or a remote server (not shown). The task parameters include, but may include, the state of the drone 100. In some embodiments, the parameters may include the initial position of the drone 100, the target position of the drone 100, one or more geometric configurations of one or more stationary obstacles defining at least part of the constraints, the geometric configurations of moving obstacles defining at least part of the constraints, and their motions. The parameters are given to the neural network 112 to obtain an estimated motion trajectory for performing the task, and the neural network 112 is trained to output an estimated motion trajectory for performing the task.

[0023] Furthermore, the neural network 112 may extract at least some integer values ​​of a solution to a mixed-integer optimization problem for planning the execution of a task that will result in an estimated motion trajectory, based on the input parameters. The mixed-integer optimization problem is solved for the task parameters using the corresponding integer values ​​fixed to the extracted integer values ​​to generate a constrained optimized motion trajectory. Thus, tasks that change the state of the drone 100 are performed to track the optimized motion trajectory. In this way, the neural network 112 limits the search space for discovering the motion trajectory, while the optimal motion trajectory is determined by the controller 111. As a result, such a combination allows for the discovery of a viable and / or optimal motion trajectory for the drone 100 with reduced computational and memory resources.

[0024] Figure 1B shows a block diagram of a method 199 performed by a controller 111 to control the motion of at least one device, such as device 100, according to some embodiments. Device 100 performs a task that changes its state, including at least its position. The performance of this task is constrained by the motion of device 100. The controller 111 includes a processor and a memory that stores instructions, which, when executed by the processor, cause the controller to implement method 199.

[0025] Method 199, upon receiving task parameters including the state of the device, inputs these parameters into a neural network 112 trained to output an estimated motion trajectory 115 for performing the task (110). Method 199 extracts at least some of the integer values ​​125 of the solution to a mixed integer optimization problem to plan the execution of the task that yields the estimated motion trajectory (120), solves the mixed integer optimization problem for the task parameters with the corresponding integer values ​​fixed to the extracted integer values ​​125 (130), thereby generating a constrained optimized motion trajectory (135).

[0026] Next, the controller 111 performs the task by instructing the device 100 to change its state according to the optimized motion trajectory 135 (140). For example, the controller may determine the control inputs to the actuators of the device 100 in order to track the motion trajectory. Examples of controllers that generate control inputs may include proportional-integral-derivative controllers, model-predictive controllers, reinforcement learning controllers, and so on.

[0027] In particular, some embodiments train the neural network to estimate the trajectory 115 rather than the integer value 125. Some embodiments are based on the understanding that doing so simplifies the training process to produce viable results.

[0028] Multi-device motion planning Figure 2 illustrates a multi-device motion planning problem according to some embodiments of this disclosure. Figure 2 shows several devices (drones 201a, 202b, 202c, and 202d, etc.) that are required to reach their assigned final positions 202a, 202b, 202c, and 202d. Furthermore, obstacles 203a, 203b, 203c, 203d, 203e, and 203f in the surrounding environment of drones 201a-201d are shown. Drones 201a-201d are required to reach their assigned final positions 202a-202d while avoiding obstacles 203a-203f in the surrounding environment. Simple trajectories (such as trajectory 204 shown in Figure 2) may cause collisions. Accordingly, embodiments of the present disclosure calculate a trajectory 205 that avoids obstacles 203a to 203f and collisions between drones 201a to 201d, which can be achieved by avoiding trajectory overlaps, or, if multiple trajectories overlap (206), by ensuring that the drones corresponding to the overlapping points reach a point in a future planning time interval that is sufficiently separated.

[0029] Figure 3 shows a schematic diagram of components included in multi-device motion planning according to some embodiments of the present disclosure. Figure 3 is a schematic diagram of a system for coordinating the motion of multiple devices 302. The multi-device planning system 301 may correspond to the controller 111 shown in Figure 1. The multi-device planning system 301 receives information from at least one of the multiple devices 302 and from one or more remote sensors 305 via its corresponding communication transceiver. Based on the acquired information, the multi-device planning system 301 calculates a motion plan for each device 302. The multi-device planning system 301 transmits the motion plan for each device 302 via the communication transceiver. The device control 304 of each device 302 receives the information and uses it to control the corresponding device hardware 303.

[0030] Figure 4 shows a schematic diagram of a multi-device planning system 301 for controlling one or more devices according to an embodiment of the present disclosure. The multi-device planning system 301 includes a hardware processor 401 connected to a communication transceiver 402 and a memory 403. In some embodiments, the hardware processor 401 is an embedded processing unit (EPU). The memory 403 may include a non-temporary computer-readable medium.

[0031] In some implementations, the memory 403 may include multiple sections: a first section that stores data about each of the devices 302; a second section that stores one or more computer-readable programs for calculating a motion plan for each device 302; a third section that stores parameters for a machine learning (ML) module for predicting a solution to a motion planning problem; and a fourth section that stores data about the environment surrounding one or more devices 302. In some embodiments, the memory 403 may store algorithms associated with the neural network 112. The hardware processor 401 and the memory 403 form a controller (e.g., controller 111) that controls the motion of one or more devices and performs the task of changing the state of one or more devices. The state of at least one device includes the position of that one or more devices, which is constrained to the motion of that one or more devices.

[0032] Typical Formulation

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038] Collision avoidance constraints between device 502 and obstacle 501, and between device 502 and other devices, consist of a group of real-valued constraints that determine the region of obstacle 501 or other devices as the intersection of the corresponding region 504 where the constraint is satisfied. Furthermore, each constraint in the group of real-valued constraints is associated with at least one binary variable indicating whether the constraint is satisfied at a particular point in time.

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043] The MIP problem (8) may also be solved by the branch and bound method shown in Figure 6. The branch and bound method can obtain the optimal solution to the MIP problem (8) by solving a mixed integer optimization problem. However, the branch and bound method solves multiple optimization problems for different values ​​of integer variables, which consumes the computational resources of the controller 111.

[0044] Figure 6 shows a solution to a mixed integer programming (MIP) problem via a branch-bound tree method for planning multi-device motion, which creates a problem addressed by some embodiments of this disclosure, where the integer variables are binary, each of which is an integer variable less than or equal to 1, and therefore can only be either 0 or 1. At the root node 601 of the branch-bound tree, the solution to the MIP problem is V bIt begins by relaxing all components of ξ that have indices within the given range to real values ​​rather than binary values. The solution to the relaxation problem can be obtained using standard methods for real-valued mathematical programming, such as the interior-point method, the effective constraint method, or the simplex method. Then, if this solution satisfies the integer constraint (i.e., V b If all components of ξ with indices within are binary, this can be stored as the optimal integer solution 602. Otherwise, V b One component of ξ having an index within is selected and assigned to the value 0 of the child node of the first branch 603 of the branch-bound tree, and the same component is assigned to the value 1 of the child node of another branch 604, thereby creating two new problems with fewer relaxed variables, and all unassigned variables can be made real.

[0045] If an integer feasible solution is found at a node (for example, node 605), it is compared to the stored integer solutions. If it is found to be a better solution, i.e., with a lower cost, the new solution can replace the stored solution; otherwise, it is discarded. In either case, further searching from this node is stopped (606). If a solution to the relaxation problem is found to be inferior to the stored integer solutions, i.e., with a higher cost, at any node (for example, node 607), further searching from this solution is stopped (608).

[0046] Therefore, the solution to the MIP problem (8) obtained by the branch-and-bound algorithm shown in Figure 6 as an example generates the optimal trajectory for the multi-device motion planning problem. However, Figure 6 shows that the number of relaxation problems to be solved is combinatorial with respect to integer (or binary) variables, and thus the computational burden for solving the MIP problem (8) can be very large in terms of both time and memory requirements. This is especially true when the number of devices increases, because an increase in devices leads to a combinatorial increase in the number of relaxation problems to be solved; that is, the addition of one device can more than double the number of problems to be solved. The latter finding makes such a branch-and-bound method unsuitable for embedded control units with limited computational and memory resources, particularly when the solution must be quickly recalculated due to changes in some of the obstacle locations, device states, or destinations.

[0047] To eliminate such problems, some embodiments are based on ML techniques, which will be further described with reference to Figure 7. Figure 7 shows a functional block diagram for solving a multi-device motion planning problem based on a machine learning (ML) module 704 according to an embodiment of the present disclosure. As shown in Figure 7, data generation may be performed by solving the multi-device motion planning problem in simulations of different scenario states, including at least some of the initial device position, destination, and obstacle centers and / or dimensions (701). Furthermore, this data is used to train the ML module 704, which includes, for example, a predictive model based on a neural network (702). Furthermore, receiving the scenario states (703), the ML module 704 calculates the ML trajectory 705 of the devices to solve the multi-device motion planning problem accordingly.

[0048] Some embodiments are based on the recognition that there are several cases in which the strategies described in Figure 7 become difficult. This is because predictive models such as neural networks typically generate continuous functions, whereas collision avoidance requires discontinuous functions. Continuous functions cause small deviations in the data to result in small deviations in the trajectory. However, in the case of collision avoidance, for example, it is necessary to change the path to avoid an obstacle from a path that goes around the obstacle from the left to a path that goes around the obstacle from the right, so small deviations in the data may need to result in large deviations in the trajectory. In addition, embodiments of this disclosure are based on the recognition that training 702 of ML modules that generate discontinuous functions (ML module 704, etc.) may be extremely difficult and may require a large amount of data 701.

[0049] Figure 8 shows a collision between multiple devices caused by the approximate behavior of the ML module when a continuous prediction model is used to predict discontinuous trajectories in a collision avoidance mode according to an embodiment of the present disclosure.

[0050] Figure 8 shows that, for a device at the first position 801, trajectory 811, which avoids collision with obstacle 820 and reaches the destination 821, circles obstacle 820 from left to right, whereas, for the second initial state 802, trajectory 812 circles obstacle 820 from right to left. As shown in Figure 8, for certain intermediate states 803 and 804, a continuous function with a small deviation from the trajectory obtained for initial states 801 and 802 can still avoid obstacle 820 if the circling behavior is maintained in trajectories 813 and 814. However, for further initial states 805 and 806, small modifications to trajectories 811 and 812 obtained for initial states 801 and 802 may cause trajectories 815 and 816 to collide with obstacle 820.

[0051] Figure 9 shows an approximate prediction model for predicting the continuous variable of a discontinuous function in a collision avoidance mode according to an embodiment of the present disclosure. Figure 9 shows an obstacle 906 and the destination position 907 of one or more devices.

[0052] During operation, the expected behavior of the continuous prediction model can be obtained using a neural network via an ML module (e.g., ML module 704). For the initial state, with a large distance to the left 901 and a large distance to the right 902 from the center of the obstacle 906, the ML module can produce the correct result of turning left 903 and right 904 towards the destination position 907. However, in the intermediate region 905, the resulting trajectory may not curve sufficiently. Therefore, the resulting trajectory will pass through the obstacle and cause a collision. Thus, the solution of the policy described with reference to Figure 7 cannot guarantee the provision of a viable trajectory for the multi-device motion planning problem.

[0053] Some embodiments are based on the further recognition that existing alternatives only involve initializing the MIP solver using the ML module to achieve a warm start. A schematic diagram of such a strategy is shown in Figure 10.

[0054] Figure 10 shows a flowchart for solving a multi-device planning problem based on warm-starting a MIP solver module using an ML module, according to an embodiment of the present disclosure. As shown in Figure 10, data is acquired by simulating and solving a multi-device motion planning problem for different scenario states that define task parameters including, for example, initial device positions, destinations, and one or a combination of obstacle centers and / or obstacle dimensions (1001). Furthermore, the acquired data is used to train an ML module that includes a predictive model based on a neural network (1002). During operation, receiving the scenario states (1003), the ML module 1004 calculates a warm-start candidate solution 1005 for the MIP solver 1006. The MIP solver 1006 uses the warm-start candidate solution 1005 to speed up the calculation of the trajectory 1007 for the multi-device motion planning problem. Thus, the trajectory is correctly calculated to be both feasible and optimal. Since the trajectory is the solution to the MIP problem, this policy of using the computed trajectory for a warm start can often reduce the computational burden. However, such reductions, though not always, are based on obtaining integer feasible solutions from the ML module, for the reason being the continuity of the general ML architecture mentioned earlier. A warm start requires an integer feasible solution to have all values ​​of a binary variable that is either 0 or 1, while a continuous approximation solution includes intermediate values ​​between 0 and 1. Applying quantization or rounding schemes based on selecting integer values ​​from real values ​​does not always provide a useful warm start candidate solution because the solution may be rendered unfeasible for one or more constraints. In addition, the computational burden on the MIP solver 1006 after a warm start may remain the same or even increase because the candidate solution forces the exploration of parts of the branched-bound tree shown in Figure 6 that are not explored in a cold start, i.e., without input from the ML module.Furthermore, since some embodiments require the full MIP solver 1006, the reduction in computational and memory requirements is generally insufficient for multi-device motion planning.

[0055] Fast computation of solutions to multi-device motion planning problems with fixed subvariables In this disclosure, several embodiments are based on the understanding that a trajectory of a multi-device motion planning problem with limited computational and memory burden is obtained by enhancing the ML module to fix only some of the variables in a mixed-integer optimization problem. Specifically, in the branched-limited tree of Figure 6, the number of relaxation problems that need to be solved decreases exponentially with decreasing numbers of integer variables in the MIP problem, so that only some or all of the integer variables are fixed. In fact, if all integer variables are fixed, there is only one real-valued optimization problem that needs to be solved, and therefore computational and memory requirements can be greatly reduced. In some embodiments, some or all of the integer variables in the MIP problem are binary variables.

[0056] In some embodiments, some or all of the integer variables are extracted from an ML module that predicts the trajectory of one or more devices, although the ML module does not explicitly predict the integer variables. In some embodiments, after the integer variables are fixed, the desired trajectory for the multi-device motion planning problem is obtained by solving a single real-valued optimization problem. The solution to the real-valued optimization problem can be obtained using standard methods for real-valued mathematical programming, such as the interior-point method, gradient-based method, alternating direction method of multipliers (ADMM), effective constraint method, or simplex method. In some embodiments, a fixed number of real-valued optimization problems are solved for different future planning time partition lengths to compute the time-optimal trajectory for each device. In some embodiments, only some of the integer variables are fixed, and therefore, the trajectory for each device needs to be computed by solving partitioned MIP problems, resulting in a significant reduction in the number of solutions to the relaxation problem, thus greatly reducing the computational and memory requirements of multi-device motion planning. Finally, in this disclosure, some embodiments are based on the understanding that the success rate of the proposed method can be increased by training the ML module to prioritize the generation of feasible rather than near-optimal trajectories achievable by concentrated sampling, and by modifying the loss function used for training to include a barrier term for collision avoidance.

[0057] In addition, several embodiments are based on the understanding that iterative evaluation of ML modules initialized in different prediction steps is likely to provide a viable solution to the multi-device motion planning problem. The latter is due to a prediction model that generates an approximate trajectory, and therefore, if a collision is predicted to occur in such an approximate trajectory at some future stage, re-evaluating the ML module from a future prediction step before the collision will return a differently approximated trajectory that is likely to avoid one or more of the previously predicted collisions. Furthermore, the operation of a system based on the understanding of this disclosure is shown in Figure 11.

[0058] Figure 11 shows a flowchart for predicting trajectories by solving a multi-device planning problem using an ML module according to an embodiment of the present disclosure. In some embodiments, the partitioned problem is solved by using the predicted trajectories when integer variables are fixed. During configuration, a data generation module 1101 selects the data to be acquired using concentrated sampling, which is then used in a training module 1102 to train a predictive model using a loss function that includes barrier terms to improve feasibility, resulting in an ML module 1104 (also called the “ML Predictor”).

[0059] Some embodiments are based on the understanding that this configuration, namely the data generation module 1101 and the training module 1102, can be executed offline and do not need to be implemented in an embedded processing unit (EPU), and therefore these components do not affect the computational and memory resources required for multi-device motion planning. During operation, upon receiving the scenario state 1103, the ML module 1104 calculates the predicted trajectory 1105, from which the problem reduction module 1106 extracts variables, so that some or all of the integer variables are fixed. Based on the predicted trajectory, the motion of one or more devices is planned so that this one or more devices do not collide with obstacles or with each other.

[0060] To extract at least some of the integer values, the problem partitioning module 1106 may formulate a mixed-integer optimization problem that searches for a global optimal solution in a search space determined by the constraints defined by equations (2) to (6). The mixed-integer optimization problem can then be transformed into a real-valued optimization problem by fixing the integer variables of the mixed-integer optimization problem to the integer values ​​extracted by the problem partitioning module 1106. Furthermore, the solution to the mixed-integer optimization problem can be obtained by adding the fixed integer values ​​to the solution of the real-valued optimization problem.

[0061] In the formulation of a mixed-integer optimization problem, at least some of the integer variables are binary. Furthermore, at least some of the integer variables are associated with a group of real-valued separation constraints to avoid collisions between one or more devices and obstacles, and collisions between one or more devices. If one of the constraints on the real-valued variables within the group of separation constraints is satisfied at a given time, then a collision with the corresponding obstacle or with other devices among one or more devices at that time is avoided.

[0062] Furthermore, the acceptable values ​​of at least some integer variables that partition the search space of the mixed-integer optimization are evaluated based on the predicted (or estimated) motion trajectory 1105 to perform the task of changing the state of one or more devices. The acceptable values ​​of at least some integer variables in the mixed-integer optimization problem are tested by determining the membership of at least some portions of the estimated motion trajectory 1105 into a region determined by constraints on real-valued variables. The constraints on real-valued variables define obstacles or other devices among one or more devices.

[0063] In some embodiments, the area of ​​the obstacle and one or more devices are defined by one or more constraints of real-valued variables and task parameters. The one or more constraints of real-valued variables are linear in state variables. Furthermore, each of the one or more constraints of real-valued variables defining the area of ​​the obstacle is associated with at least one binary variable, which indicates whether the constraint is satisfied at a particular point in time when the acceptability of the value of at least one binary variable associated with this constraint is tested. To this end, it is determined whether the estimated motion trajectory 1105 satisfies the constraint to which an integer variable is associated at a particular point in time.

[0064] Continuing with Figure 11, the reduced solver module 1107 for the partition problem computes a trajectory that solves the multi-device motion planning problem. Such a trajectory is called the optimized motion trajectory, and the optimized motion trajectory 1108 may be used immediately or may be used as a viable warm start for the entire solver module 1109, which can solve the entire MIP problem for a fixed number of nodes and produce a further improved solution 1110. Furthermore, the optimized motion trajectory 1108 is updated or improved based on the solution of the MIP warm start based on the optimized motion trajectory 1108. In some embodiments, depending on whether it is detected that there is available time before the time when the computation of the optimized motion trajectory must be completed, the optimized motion trajectory is updated by solving or improving a mixed integer optimization problem that is warm-started based on the optimized motion trajectory. Furthermore, the optimized motion trajectory 1108 is generated from a solution that can be computed at the lowest cost by the time when the computation of the optimized motion trajectory must be completed (also called the "available period").

[0065] Furthermore, depending on whether improvements are made, trajectory 1111 is assigned to either the optimized trajectory 1108 obtained from the segmented solver module 1107, or to trajectory 1110 from the entire MIP solver module 1109.

[0066] Some embodiments of this disclosure are based on the understanding that the operation, namely the ML prediction 1104, problem partitioning 1106, and partition solver 1107, needs to be executed online and implemented on the EPU for multi-device motion planning, but the amount of computational and memory resources required for multi-device motion planning by these components is significantly reduced. In some embodiments, the ML module 1104, problem partitioning module 1106, partition solver module 1107, and all solver module 1109 are included in the controller 111 (Figure 1).

[0067] In some embodiments, the entire solver module 1109 may be part of the online operation of the multi-device motion planning system, and the computation and memory resources of the entire solver module 1109 may be limited by limiting the number of iterations in the optimization algorithm, for example, by limiting the number of nodes in the branch-limited tree.

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072]

number

[0073]

number

[0074] Another embodiment of concentrated sampling involves active learning 1265, where an initial set of data is used to construct a learner, which then selects the next batch of samples containing the most useful information (e.g., according to entropy measurement). By repeating the learning and batch selection process, the collected data will contain concentrated samples around the most informative region of the state space.

[0075]

number

[0076] Figure 13A shows a training algorithm for a predictive model according to an embodiment of the present disclosure. In particular, Figure 13A summarizes various predictive models 1301 that can be trained to generate trajectories. The predictive model may be deterministic 1311 or probabilistic 1313, and is based on a neural architecture 1321 / 1331 or kernel representation 1323 / 1333. Examples of deterministic predictive models include multilayer perceptrons, convolutional neural networks 1321, kernel regression and support vector machines 1323, etc. Examples of probabilistic predictive models include Bayesian neural networks, neural processes 1331, Gaussian processes, clinging interpolation 1333, etc. One embodiment described in Figure 13B uses a deep neural network as the predictive model.

[0077]

number

[0078]

number

[0079] While general training methods for predictive models aim to minimize the training error of the predictive model equally in all directions, in this disclosure, it is recognized that in the case of multi-device motion planning, the training error must be minimized primarily in the direction that causes a violation of the constraints determining collision avoidance, because violating these constraints causes significant damage to the device, while other errors only cause performance degradation.

[0080] Figure 14 shows the predicted trajectory error for collision avoidance according to an embodiment of the present disclosure. Figure 14 shows the optimal trajectory 1401 for training and the learned trajectory 1411 which has errors and moves away from the obstacle 1402. Thus, the learned trajectory 1411 only results in performance degradation. Figure 14 further shows the learned trajectory 1412 which has errors and approaches the constraint and actually violates the constraint. Thus, the learned trajectory 1412 causes a collision between the device 1403 and the obstacle 1402.

[0081]

number

[0082] Figure 15 shows a barrier function 1501 for collision avoidance according to an embodiment of the present disclosure. Figure 15 will be described in relation to Figure 11. In Figure 15, the barrier function 1501 for an obstacle 1502 is shown, based on an ellipse 1503 containing the obstacle 1502. The barrier function 1501 has a small value where the distance from the area where the constraint is violated is large, and a large value within the area where the constraint is violated. The barrier function 1501 grows rapidly from a small value to a large value in the area where the distance from the area where the constraint is violated is small, and the barrier function 1501 does not decrease with respect to the distance of the area where the constraint is violated.

[0083]

number

[0084] Figure 16 shows a flowchart of an algorithm for updating predictive model parameters when the loss function includes a barrier function term, according to an embodiment of this disclosure. Figure 16 will be described in relation to Figure 11. This algorithm begins at step 1601.

[0085] In step 1601, the current prediction of the prediction model is evaluated. Furthermore, in step 1602, the MSE cost is calculated. In step 1603, the barrier cost is calculated, and in step 1604, the total loss function is calculated. In step 1605, the termination conditions are checked. If the termination conditions, such as the total number of updates or progress obtained by the updates, are met, in step 1606, the training module terminates and returns the current optimal prediction model parameters. Otherwise, in step 1607, the parameters are updated. In some embodiments, each update of the model parameters 1607 is performed by a stochastic gradient descent algorithm or by any variation thereof such as the Adam optimization algorithm.

[0086]

number

[0087] Some embodiments are based on the recognition that predicting device state trajectories is easier, even if the objective is to predict the values ​​of some or all integer variables in problem partitioning module 1106. This is because integer variables can be reconstructed from device state trajectories, and state trajectories are perceived to be smaller in size and have fewer constraints between their vector components. Instead, the set of integer variables is much larger, and instead of a state vector existing for each device and each time step, there is one integer variable for each device, each time step, each obstacle, and each side. Also, because integer variables are discrete values, they are not easily predicted by continuous real-valued prediction models and are subject to relative constraints that cannot be easily reinforced within the prediction model. Therefore, less data is required to predict real-valued state trajectories, and the prediction model is simpler; that is, there are fewer equations, and significant prediction errors are less and smaller.

[0088] Despite the use of a training loss function with additional barrier terms, collisions can still occur in the predicted trajectory. However, in this disclosure, some embodiments are based on the understanding that collisions occurring in the predicted trajectory still result from the approximation of the continuous prediction model in predicting discontinuous solutions. Because the region where discontinuity occurs is limited, small perturbations to the state from which the predicted trajectory is calculated may trigger recalculations to avoid collisions. Therefore, in some embodiments, the ML module functions as a realization of the receding horizon, as shown in Figure 17.

[0089]

number

[0090]

number

[0091]

number

[0092] Furthermore, in step 1704, the ML module checks whether a collision between a device and an obstacle or between two devices occurs in the candidate trajectory by evaluating whether inequalities (4) and (5a) are satisfied at any given time step. If no collision is detected, the ML module proceeds to step 1705.

[0093] In step 1705, the counter is set to S=T, a candidate trajectory is accepted, and it is generated as the output of the ML module. If a collision is detected otherwise, the ML module proceeds to step 1706.

[0094]

number

[0095]

number

[0096] Figure 18 shows a flowchart of an algorithm for obtaining integer variable values ​​from trajectories predicted by an ML module, according to an embodiment of this disclosure. Figure 18 will be explained in relation to Figure 11.

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102] In some embodiments, collision avoidance constraints or destination constraints may be defined in a different manner from the mixed integer constraints of (3), (5), and (6), respectively, but some or all of the integer or binary variables in the MIP problem of the multi-device motion planning system can be fixed by implementing the problem partitioning module 1106 as described in Figure 18.

[0103]

number

[0104] In some embodiments, the partition solver module 1107 requires a solution to one or more real-valued optimization problems using standard methods for real-valued mathematical programming, such as the interior-point method, gradient-based methods, ADMM, effective constraint method, or simplex method. In some embodiments, the partition solver module 1107 requires a solution to a MIP problem with a significantly smaller number of integer or binary variables, and can compute a feasible and near-optimal solution to a multi-device motion planning problem using heuristic or approximate optimization algorithms, for example, by rounding techniques or feasibility pumps. In other embodiments of the present disclosure, the partition solver module 1107 computes a solution to a partitioned MIP problem based on a global optimization algorithm, which includes, for example, a branch-and-bound method, branch-and-cut method, branch-and-price method, or similar methods based on branch and / or section methods.

[0105] As shown in Figure 11, the output of the partitioned solver module 1107 may be used directly, or it may be used to warm-start the whole solver module 1109. In the latter case, the whole solver module 1109 warm-starts the MIP solver by using the real-valued component of solution (20) and the integer-valued component of solution (21) to compute the optimal or near-optimal solution to the MIP problem (8). Since solutions (20) and (21) provide a viable solution, the MIP solver in the whole solver module has a viable warm-start solution, which is therefore effective in reducing the computation time and memory required to find the optimal or near-optimal MIP solution.

[0106] Some embodiments are based on the understanding that the whole solver module 1109 uses an iterative procedure of mixed integer programming that can be stopped at any time and always provides a viable solution that is at least as optimal as the warm-start solution from the partitioned solver module 1107. In particular, the whole solver module may explore only a relatively small number of fixed nodes in the branch-bound tree in Figure 6, following a warm start, which limits its computation time and memory requirements to a known number of multi-device motion plans for the EPU.

[0107] If a failure is returned by the problem partitioning module 1106 or the partitioning solver module 1107, each device is stopped and moved slowly away from one or more of the nearest obstacles in a random and / or safe direction, and this method is repeated. Since the impossibilities are limited to a relatively small area and are most likely due to approximations of discontinuities close to obstacles, the failure recovery mechanism based on new predictions from the ML module using the updated scenario state is likely to provide a viable trajectory for each device in future time steps.

[0108] Embodiment This specification provides only specific examples of embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of specific examples of embodiments will provide a description that will enable the implementation of one or more specific examples of embodiments for those skilled in the art. Various modifications are intended to be made to the function and configuration of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims. Specific details are given in the following description for a full understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be carried out without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other examples, well-known processes, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments. Furthermore, similar reference numbers and names in the various drawings refer to similar elements.

[0109] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Moreover, the order of operations is interchangeable. A process may terminate when its operations are complete, but it may have additional steps that are not discussed or included in the diagrams. Furthermore, not all operations in any specifically described process can occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, the termination of the function may correspond to returning the function to the calling function or the main function.

[0110] Furthermore, embodiments of the disclosed subject matter can be implemented either manually or automatically, at least partially. Manual or automatic implementation may be performed using, or at least assisted by, a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. If implemented with software, firmware, middleware, or microcode, the program code or code segments that perform the required tasks may be stored on a machine-readable medium. A processor can then perform the required tasks.

[0111] Furthermore, embodiments of the present disclosure and the functional operations described herein can be implemented in digital electronic circuits, in tangible computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Furthermore, some embodiments of the present disclosure can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-temporary program carrier for execution by a data processing device or for controlling the operation of a data processing device. Furthermore, program instructions can be encoded on artificially generated propagating signals, for example, on machine-generated electrical, optical, or electromagnetic signals generated to encode information to be transmitted to a suitable receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof.

[0112] Computer programs (sometimes called, or described as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, as standalone programs, or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may, but may not, correspond to files in a file system. A program can be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or a coordinated set of files, for example, a file that stores one or more modules, subprograms, or parts of code. Computer programs can be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a communication network.

[0113] A computer suitable for running computer programs may, for example, be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or is operationally coupled to such disks to receive data from them, transfer data to them, or both. However, a computer does not have to have such devices. Furthermore, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0114] To provide user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. User interaction may be provided using other types of devices. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer may implement user interaction by sending documents to and receiving documents from a device used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0115] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser that allows a user to interact with an implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0116] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically communicate through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship.

[0117] While this disclosure has been described using several preferred embodiments, it should be understood that various other adaptations and modifications can be carried out within the spirit and scope of this disclosure. Therefore, it is the aspect of the following claims to cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. A controller for controlling the motion of at least one device, wherein the at least one device performs a task of changing the state of the at least one device, the state of the at least one device includes at least one position of the at least one device, which is subject to constraints on the motion of the at least one device, and the controller, Processor and The system comprises a memory in which instructions are stored, and when an instruction is executed by the processor, the controller, The parameters of the task, including the state of at least one device, are input to a neural network trained to output an estimated motion trajectory for performing the task. As a result, in order to plan the execution of the task that yields the estimated motion trajectory, extract at least some of the integer values ​​of the solution to the mixed-integer optimization problem, Using the integer values ​​fixed to the extracted integer values, the mixed-integer optimization problem is solved for the parameters of the task to generate the optimized motion trajectory subject to the constraints. To track the optimized motion trajectory, the state of the at least one device is changed. The controller that executes the command.

2. The controller according to claim 1, wherein the parameters for the task include one or a combination of the initial position of the at least one device, the target position of the at least one device, the geometric configuration of at least one stationary obstacle defining at least part of the constraints, and the geometric configuration and motion of a moving obstacle defining at least part of the constraints.

3. The controller according to claim 1, wherein the at least one device is at least one of an autonomous vehicle, a mobile robot, an aerial drone, a ground vehicle, an aerial vehicle, a water vehicle, or an underwater vehicle.

4. In order to extract at least some of the aforementioned integer values, the processor: To find the global optimal solution within the search space determined by the aforementioned constraints, we formulate a mixed-integer optimization problem. Based on the estimated motion trajectory for performing the task, which is the output of the neural network, the acceptableness of values ​​for at least some integer variables that partition the search space of the mixed integer optimization is evaluated. The controller according to claim 1, which is configured as follows.

5. The processor is further configured to formulate a mixed-integer optimization problem, wherein at least some of the integer variables are binary, At least some of the integer variables are associated with a group of real-valued variables that are tangent to or tangent to avoid collisions between at least one device and an obstacle and between multiple devices including at least one device. The controller according to claim 4, wherein if one of the constraints of a real-valued variable within the group of separation constraints is satisfied at a particular time, a collision with the corresponding obstacle or with another device among the plurality of devices is avoided at that particular time.

6. The controller according to claim 5, wherein the processor is further configured to test the acceptability of values ​​for at least some integer variables in the mixed integer optimization problem by determining the membership of at least some portion of the estimated motion trajectory into a region determined by the obstacle or constraints of real-valued variables defining other devices among the plurality of devices.

7. The area of ​​the obstacle and the at least one device are defined by one or more constraints of linear real-valued variables in the state variables and the parameters of the task, Each of the one or more constraints of real-valued variables defining the area of ​​the obstacle is associated with at least one binary variable indicating whether the constraint is satisfied at a particular time point. The controller according to claim 6, wherein the processor is further configured to test the acceptability of the value of the at least one binary variable associated with the constraint by determining whether the estimated motion trajectory satisfies the constraint to which the integer variable is associated at a particular point in time.

8. The aforementioned processor further, By fixing the integer variables of the mixed integer optimization problem to the extracted integer values, the mixed integer optimization problem is transformed into a real-valued optimization problem. The solution to the mixed-integer optimization problem is obtained by adding a fixed value to the integer variable from the solution to the real-valued optimization problem. The controller according to claim 1, which is configured as follows.

9. The controller according to claim 1, wherein the processor is further configured to update the optimized motion trajectory by solving the mixed integer optimization problem which is warm-started based on the optimized motion trajectory.

10. The controller according to claim 1, wherein the processor is further configured to update the optimized motion trajectory by solving the mixed integer optimization problem, which is warm-started based on the optimized motion trajectory, in response to detection of time availability prior to the time at which the calculation of the optimized motion trajectory should be completed.

11. The processor is further configured to update the optimized motion trajectory by improving the mixed-integer optimization problem from a warm start based on the optimized motion trajectory, and the optimized motion trajectory is updated within the available period prior to the point in time when the calculation of the optimized motion trajectory needs to be completed. The controller according to claim 1, wherein the processor is further configured to generate the optimized motion trajectory from the lowest cost solution calculated within the available period.

12. The aforementioned processor further, The impossibility of the estimated motion trajectory is detected as an impossibly achievable estimated motion trajectory. The parameters of the task are updated by changing the state of the at least one device according to a portion of the unexecutable estimated motion trajectory. With respect to the updated parameters of the task, estimate the optimized motion trajectory. The controller according to claim 1, which is configured as follows.

13. The controller according to claim 12, wherein the impracticality of the estimated motion trajectory is detected by evaluating whether the estimated motion trajectory causes at least one collision between a plurality of devices including the at least one device, or between one of the at least one devices and an obstacle.

14. The controller according to claim 1, wherein the neural network is trained using a reconstruction loss function modified with a barrier function, and the value of the reconstruction loss function is increased based on the distance of the state of the at least one device to a region where the constraint on the motion of the at least one device is violated.

15. The aforementioned barrier function is, Where the distance from the region where the constraint is violated is large, the value is small. Within the region where the aforementioned constraint is violated, the value is large. The barrier function grows rapidly from a small value to a large value in regions where the distance from the region where the constraint is violated is small. The controller according to claim 14, wherein the barrier function does not decrease with respect to the distance of the region in which the constraint is violated.

16. The controller according to claim 12, wherein the neural network is trained using training data which includes samples in the search space of the parameters of the task labeled with the optimal solution of the mixed integer optimization problem, the samples which include a set of regular samples and a set of biased samples, the set of biased samples being biased toward the constraints.

17. The controller according to claim 1, wherein the neural network is a probabilistic neural network.

18. The controller according to claim 1, wherein the processor is an embedded processing unit (EPU).

19. A method for controlling the motion of at least one device, wherein the at least one device performs a task that changes the state of the at least one device, which is subject to constraints on the motion of the at least one device, the method uses a processor together with stored instructions that implement the method, the instructions, when executed by the processor, perform a step of the method, the step is The steps include inputting the parameters of the task, including the state of at least one device, into a neural network trained to output an estimated motion trajectory for performing the task, The steps include extracting at least some integer values ​​of the solution to a mixed-integer optimization problem in order to plan the execution of the task which results in the estimated motion trajectory, The steps include: generating the optimized motion trajectory subject to the constraints by solving the mixed-integer optimization problem for the parameters of the task using the corresponding integer values ​​fixed to the extracted integer values; A method comprising the step of changing the state of the at least one device in order to track the optimized motion trajectory.

20. A non-temporary computer-readable storage medium on which a processor-executable program is implemented to perform a method for controlling the motion of at least one device, wherein the at least one device performs a task that changes the state of the at least one device, subject to constraints on the motion of the at least one device, and the method is The steps include inputting the parameters of the task, including the state of at least one device, into a neural network trained to output an estimated motion trajectory for performing the task, The steps include extracting at least some integer values ​​of the solution to a mixed-integer optimization problem in order to plan the execution of the task which results in the estimated motion trajectory, The steps include: generating the optimized motion trajectory subject to the constraints by solving the mixed-integer optimization problem for the parameters of the task using the corresponding integer values ​​fixed to the extracted integer values; A non-temporary computer-readable storage medium, comprising the step of changing the state of at least one device in order to track the optimized motion trajectory.

Images