A method for controlling ordered operations of multiple mobile carriers

Abstract

The present invention relates to the field of intelligent mobile carriers, specifically a method for orderly operation control of multiple mobile carriers, comprising the following steps: simulating a real environment to construct a simulation environment and a mobile carrier model, setting a reinforcement learning network framework, and defining the observation space and action space of each mobile carrier; designing a timing reward function in stages according to the mobile carrier formation and comfort tasks, wherein the reward function includes a movement reward, a formation reward, a state reward, an acceleration reward, and an angular acceleration reward. The present invention adopts a staged idea, starting from a simple obstacle-free scene and gradually transitioning to a complex scene containing fixed obstacles and dynamic obstacles, ensuring the adaptability and stability of the model in different environments, and by designing the reward functions of movement reward, formation reward, state reward, acceleration reward, and angular acceleration reward, it is possible to guide the mobile carrier to move toward the target point, and improve the comfort of passengers while ensuring efficiency.

Description

Technical Field

[0001] The invention relates to the field of intelligent mobile carriers, and in particular to a method for controlling orderly operations of multiple mobile carriers. Background Art

[0002] In large public places such as airport terminals, mobile carriers need to provide convenient ride services for passengers. In a multi-mobile carrier system, mobile carriers can efficiently perform tasks that are difficult for a single mobile carrier to complete through collaborative formation.

[0003] However, when the existing technology is actually used, during the operation and movement of multi-mobile carrier systems, traditional collaborative formation control and obstacle avoidance algorithms perform well when dealing with known environments or static obstacles, but have limitations when facing dynamic, unknown or complex environments. Summary of the invention

[0004] The purpose of the present invention is to provide a method for orderly operation control of multiple mobile carriers to solve the problem that traditional cooperative formation control and obstacle avoidance algorithms have limitations when dealing with complex and dynamic environments.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for controlling orderly operation of multiple mobile carriers, comprising the following steps:

[0006] S1. Simulate the real environment to build a simulation environment and mobile carrier model, and define the observation space and action space of each mobile carrier;

[0007] S2. Designing a timing reward function in stages according to the mobile carrier formation and comfort tasks, wherein the reward function includes movement reward, formation reward, state reward, acceleration reward and angular acceleration reward;

[0008] S3. Construct a reinforcement learning network framework including an actor network and a critic network, wherein the actor network is used to output the speed and angular velocity information of the mobile carrier, and the critic network is used to output the reward data obtained in the current state;

[0009] S4, performing formation simulation training in the simulation environment in step S1, sampling the interaction data between the simulation environment and the mobile carrier according to the reinforcement learning network framework in step S3, making decisions on the actions of the mobile carrier, and collecting experience data;

[0010] S5. When the value of the experience memory for collecting the experience data in step S4 reaches the preset value, start updating the optimization strategy. Based on the MADDPG algorithm, use the data in the experience pool to update the actor network and the critic network in step S3, and deploy the new network parameters to the mobile carrier to ensure that multiple mobile carriers adopt the latest strategy.

[0011] S6. In an environment with no obstacles, fixed obstacles, and randomly generated obstacle positions, the ideal formation model obtained in step S5 is introduced to continue training until the overall reward function converges and the formation task is completed. After all phased training is completed, steps S5 and S6 are repeated until the ideal formation model is obtained.

[0012] S7. In the actual environment, the formation model trained in step S6 is deployed to each mobile carrier, the point cloud information and odometer information of the lidar are called, and the corresponding strategies and action instructions are output by calling the model according to the position and posture of each mobile carrier to complete the formation task.

[0013] Preferably, the simulation environment in step S1 is built based on the real environment inside the terminal, and is combined with fixed obstacles and some random obstacles inside the terminal to generate a simulation environment that simulates reality.

[0014] Preferably, in step S2:

[0015] The movement reward represents the change in the distance between the mobile carrier and the target point, ensuring that the mobile carrier moves toward the target point;

[0016] The formation reward indicates whether the position relationship between the mobile carriers is the set value;

[0017] The state reward indicates that the simulation ends when the mobile carrier reaches the target point or a collision occurs;

[0018] The acceleration bonus and angular acceleration bonus impose comfort limits on the speed and angular velocity of the mobile carrier respectively.

[0019] Preferably, the reinforcement learning network framework including the actor network and the critic network is constructed in step S3 as follows:

[0020] Establish an actor network to obtain lidar point cloud data, odometer information, and output the speed and angular velocity information of the mobile carrier;

[0021] A critic network is established to obtain the relative position and speed information of the current mobile carrier, and output the reward data obtained in the current state.

[0022] Preferably, the sampling process in step S4 includes sampling the interaction between multiple mobile carriers and the simulation environment, each mobile carrier inputs its own observation data into its own actor network to generate the next action, and uses the critic network to evaluate the value of the state of the mobile carrier to obtain reward data, and saves the experience of all mobile carriers into the experience pool.

[0023] Preferably, the MADDPG algorithm in step S5 is expanded from a single agent to multiple agents on the basis of the DDPG algorithm, and incorporates the ideas of centralized training and distributed execution, evaluates the overall reward data of all mobile carriers, and limits the sampled motion speed and angular velocity values ​​to a certain range.

[0024] Preferably, the step S6 imports the ideal formation model obtained in step S5 to continue training specifically as follows:

[0025] S61: Training is performed in an obstacle-free scene. The reward function at this stage only includes movement reward and formation reward. The next stage of training begins after the reward converges and the formation task is completed.

[0026] S62, training in a fixed obstacle scene. The reward function of this stage includes movement reward, formation reward, state reward, acceleration reward and angular acceleration reward. After the overall reward function converges and the formation task is completed, the next stage of training begins.

[0027] S63. Add multiple randomly generated dynamic obstacles to the fixed obstacle scene for training. The reward function at this stage is the same as that in step S63. The training ends when the overall reward function converges and the formation task is completed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention constructs a simulation environment and a mobile carrier model by simulating the internal environment of the terminal, and combines it with a reinforcement learning framework, so that the mobile carrier can be trained in an environment close to the real scene. Different ordered reward functions are designed, and the positions of obstacles are randomly generated for training, and finally a neural network model of each mobile carrier is obtained. It not only effectively solves the problem of slow convergence speed of the ordered algorithm of multiple mobile carriers in complex environments, but also significantly improves the robustness and safety of the ordered order of multiple mobile carriers. It adopts a phased idea, starting from a simple obstacle-free scene and gradually transitioning to a complex scene containing fixed obstacles and dynamic obstacles, ensuring the adaptability and stability of the model in different environments. At the same time, by designing a reward function including movement reward, formation reward, state reward, acceleration reward and angular acceleration reward, it can guide the mobile carrier to move towards the target point, maintain formation, avoid collision, and improve passenger comfort while ensuring efficiency.

[0030] The present invention also adopts the idea of ​​centralized training and distributed execution, and uses the MADDPG algorithm to evaluate the overall reward data of all mobile carriers to update the network parameters. This strategy avoids the local convergence problem and improves the stability of the algorithm. At the same time, the idea of ​​phased training also enables the formation model to gradually adapt to the training environment from simple to complex, and ultimately achieve an ideal formation effect in an environment with randomly generated obstacle positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the overall flow chart of the autonomous formation of the present invention;

[0032] Figure 2 It is a flowchart of step 6 of the present invention;

[0033] Figure 3 Schematic diagram of the MADDPG algorithm of the present invention;

[0034] Figure 4 This is a diagram of the operation process of multiple mobile carriers of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Traditional obstacle avoidance algorithms mainly include geometry-based methods such as visibility graph method, grid method, free space method, etc. and potential field-based methods such as artificial potential field method. These methods usually perform well when dealing with known environments or static obstacles, but when facing dynamic, unknown or complex environments, they have limitations such as slow convergence speed in complex environments. Reinforcement learning, as a machine learning method that learns the optimal strategy by interacting with the environment, has been widely used in the field of obstacle avoidance in recent years. The reinforcement learning obstacle avoidance algorithm enables the intelligent agent to find the optimal obstacle avoidance strategy in an uncertain environment through continuous trial and error and feedback mechanism.

[0037] See also Figure 1-4 The present invention provides a technical solution: a method for controlling orderly operations of multiple mobile carriers, comprising the following steps:

[0038] S1. Simulate the real environment to build a simulation environment and mobile carrier model, and define the observation space and action space of each mobile carrier, specifically:

[0039] A multi-mobile carrier formation simulation environment is built based on the real environment inside the terminal, and a simulation environment simulating reality is generated by combining fixed obstacles and some random obstacles inside the terminal;

[0040] The formation decision problem of multiple mobile carriers is defined as a partially observable Markov decision process.

[0041] S2. Design the temporal reward function in stages according to the mobile carrier formation and comfort tasks, where:

[0042] The reward function is the reward obtained by the i-th mobile carrier at time step t, which includes two parts: formation reward function and comfort reward function. The formation reward function includes: movement reward, formation reward and state reward, and the comfort reward function includes: acceleration reward and angular acceleration reward.

[0043] The mobile reward in the formation reward can represent the change in the distance between the robot and the target point, ensuring that the mobile carrier moves towards the target point. Indicates the position of the mobile carrier at the last moment, using ( represents the position of the mobile carrier at this moment, ( Indicates the coordinates of the target point, movement reward for:

[0044]

[0045] The formation reward indicates whether the position relationship between the mobile carriers is the set value, expressed as ( Represents the position coordinates of the leader, using ( Indicates The position coordinates of the followers, formation rewards for:

[0046]

[0047] in, Indicates The formation error between the mobile carrier and the leader;

[0048] The state reward indicates that the simulation ends when the mobile carrier reaches the target point or collides with it. for:

[0049]

[0050] Among them, when the mobile carrier reaches the target point, the state reward Assigned to , when the moving carrier collides Assigned to , if the distance between the mobile carrier and the obstacle is less than 1 Assigned to ,otherwise The value is assigned to 0; and is a custom constant, It can be expressed as:

[0051]

[0052] in Indicates the distance between the mobile carrier and the obstacle.

[0053] The final moving vector at each time step The reward is obtained by weighting each part, and the final reward function is for:

[0054]

[0055] in, A custom constant.

[0056] The acceleration bonus in the comfort bonus means that the acceleration of the mobile carrier is restricted by comfort, where Indicates the speed output of the mobile carrier at the last moment, Represents the speed output at the current moment. Since the speed output frequency is fixed, the acceleration of the mobile carrier can be expressed as , acceleration reward for:

[0057]

[0058] in, A custom constant.

[0059] The angular acceleration bonus in the comfort bonus indicates that the angular acceleration of the mobile carrier is subject to comfort restrictions, where Indicates the angular velocity output of the mobile carrier at the last moment, Represents the angular velocity output at the current moment. Since the angular velocity output frequency is fixed, the angular acceleration of the mobile carrier can be expressed as , angular acceleration reward for:

[0060]

[0061] in, A custom constant.

[0062] S3. Construct a reinforcement learning network framework including an actor network and a critic network, input the odometer information and lidar information of multiple mobile carriers as states into the reinforcement learning network, and output the speed and angular velocity of multiple mobile carriers in real time after reinforcement learning network training. The actor network is used to output the speed and angular velocity information of the mobile carrier, and the critic network is used to output the reward data obtained in the current state, specifically:

[0063] Establish an actor network to obtain lidar point cloud data, odometer information, and output the speed and angular velocity information of the mobile carrier;

[0064] Establish a critic network to obtain the relative position and speed information of the current mobile carrier, and output the reward data obtained in the current state;

[0065] S4, performing formation simulation training in the simulation environment in step S1, sampling the interaction between multiple mobile carriers and the simulation environment according to the reinforcement learning network framework in step S3, making decisions on the actions of the mobile carriers, each mobile carrier inputting its own observation data into its own actor network to generate the next action, and using the critic network to evaluate the value of the state of the mobile carrier to obtain reward data, and saving the experience of all mobile carriers into the experience pool, which includes the observation, action and reward of each mobile carrier;

[0066] S5. When the value of the experience memory for collecting the experience data in step S4 reaches the preset value, start updating the optimization strategy. Based on the MADDPG algorithm, use the data in the experience pool to update the actor network and the critic network in step S3, and deploy the new network parameters to the mobile carrier to ensure that multiple mobile carriers adopt the latest strategy. The MADDPG algorithm is based on the DDPG algorithm and is extended from a single agent to multiple agents. It also adds the idea of ​​centralized training and distributed execution, evaluates the overall reward data of all mobile carriers, and limits the sampled action speed and angular velocity values ​​to a certain range.

[0067] S6. In an environment with no obstacles, fixed obstacles, and randomly generated obstacle positions, the ideal formation model obtained in step S5 is introduced to continue training until the overall reward function converges and the formation task is completed, and all phased training is completed, specifically:

[0068] S61, the first stage of training is in an obstacle-free scene, and the initial position and target point of the mobile carrier are randomly generated; after each mobile carrier reaches the target point, the state is set to reach the target point and the starting point and target point are directly reset to perform a new round of simulation; the reward function of this stage only includes movement reward and formation reward, until the reward converges and the formation task is completed, and then enter the next stage of training;

[0069] S62, the second stage of training is training in a fixed obstacle scene, and the simulation environment includes multiple mobile carriers and fixed obstacles; the reward function of this stage includes the three reward functions involved, and in each training round, multiple mobile carriers generate fixed initial positions and target points. When the status of all mobile carriers is arrival or collision, the next round of training begins; when the overall reward function converges and the formation task is completed, the next stage of training begins;

[0070] S63, in the third stage of training, multiple randomly generated dynamic obstacles are added to the fixed obstacle scene, and the simulation environment includes multiple mobile carriers, fixed obstacles and randomly generated dynamic obstacles; the reward function of this stage includes the three reward functions involved, and when the status of all mobile carriers in each environment in each training round is arrival or collision, the next round of training begins; when the overall reward function converges and the formation task is completed, the training ends;

[0071] Repeat steps S5 and S6 until an ideal formation model is obtained.

[0072] S7. In the actual environment, deploy the formation model trained in step S6 to each mobile carrier, and let each mobile carrier call the point cloud information and odometer information of the lidar, and output the corresponding strategy and action instructions by calling the model according to the position and posture of each mobile carrier, and add a smoothing function to the speed output to ensure the smooth start and stop of the mobile carrier to complete the formation task.

[0073] The present invention aims at the autonomous formation of multiple mobile carriers, adopts the MADDPG algorithm for centralized training and distributed execution, then constructs a simulation environment and a multi-mobile carrier model that simulates the interior of an airport, uses the lidar point cloud data and odometer information of the mobile carriers as state inputs and generates different strategies for each mobile carrier, generates different actions according to different strategies and environmental states, and designs a reward function that can realize autonomous formation. Then, the overall reward value of multiple mobile carriers is evaluated through a critic network, and finally a model of autonomous formation of multiple mobile carriers is obtained after multiple training updates and iterations.

[0074] The present invention constructs a simulation environment and a mobile carrier model by simulating the internal environment of the terminal, and combines it with a reinforcement learning framework, so that the mobile carrier can be trained in an environment close to the real scene. Different ordering reward functions are designed to randomly generate obstacle positions for training, and finally a neural network model for each mobile carrier is obtained. This not only effectively solves the problem of slow convergence of the multi-mobile carrier ordering algorithm in a complex environment, but also significantly improves the robustness and safety of the multi-mobile carrier ordering. It adopts a phased idea, starting from a simple obstacle-free scene and gradually transitioning to a complex scene containing fixed obstacles and dynamic obstacles, ensuring the adaptability and stability of the model in different environments. At the same time, by designing a reward function including movement rewards, formation rewards, state rewards, acceleration rewards, and angular acceleration rewards, the mobile carrier can be guided to move toward the target point, maintain formation, avoid collisions, and improve passenger comfort while ensuring efficiency;

[0075] In addition, this method also adopts the idea of ​​centralized training and distributed execution, and uses the MADDPG algorithm to evaluate the overall reward data of all mobile carriers to update the network parameters. This strategy avoids local convergence problems and improves the stability of the algorithm. At the same time, the idea of ​​phased training also enables the formation model to gradually adapt to the training environment from simple to complex, and finally achieves the ideal formation effect in an environment with randomly generated obstacle positions.

[0076] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for controlling the orderly operation of multiple mobile carriers, characterized in that: The following steps are involved: S1. Simulate the real environment to build a simulation environment and mobile carrier model, and define the observation space and action space of each mobile carrier. The simulation environment is built based on the real environment inside the terminal, and combines fixed obstacles and some random obstacles inside the terminal to generate a simulation environment that simulates reality; S2. Design a time series reward function in stages according to the mobile carrier formation and comfort tasks. The reward function includes movement reward, formation reward, state reward, acceleration reward and angular acceleration reward, where: The movement reward represents the change in the distance between the mobile carrier and the target point, ensuring that the mobile carrier moves toward the target point; The formation reward indicates whether the position relationship between the mobile carriers is the set value; The state reward indicates that the simulation ends when the mobile carrier reaches the target point or a collision occurs; The acceleration bonus and angular acceleration bonus respectively impose comfort limits on the speed and angular velocity of the mobile carrier; S3. Construct a reinforcement learning network framework including an actor network and a critic network, wherein the actor network is used to output the speed and angular velocity information of the mobile carrier, and the critic network is used to output the reward data obtained in the current state; S4, performing formation simulation training in the simulation environment in step S1, sampling the interaction data between the simulation environment and the mobile carrier according to the reinforcement learning network framework in step S3, making decisions on the actions of the mobile carrier, and collecting experience data; S5. When the value of the experience memory for collecting the experience data in step S4 reaches the preset value, start updating the optimization strategy. Based on the MADDPG algorithm, use the data in the experience pool to update the actor network and the critic network in step S3, and deploy the new network parameters to the mobile carrier to ensure that multiple mobile carriers adopt the latest strategy. S6. In an environment with no obstacles, fixed obstacles, and randomly generated obstacle positions, the ideal formation model obtained by step S5 is imported to continue training until the overall reward function converges and the formation task is completed. After all phased training is completed, steps S5 and S6 are repeated until the ideal formation model is obtained. The specific steps of importing step S5 to obtain the ideal formation model and continuing training are as follows: S61: Training is performed in an obstacle-free scene. The reward function at this stage only includes movement reward and formation reward. The next stage of training begins after the reward converges and the formation task is completed. S62, training in a fixed obstacle scene. The reward function of this stage includes movement reward, formation reward, state reward, acceleration reward and angular acceleration reward. After the overall reward function converges and the formation task is completed, the next stage of training begins. S63, adding multiple randomly generated dynamic obstacles to perform training on the basis of the fixed obstacle scene. The reward function at this stage is the same as that in step S63. The training ends when the overall reward function converges and the formation task is completed. S7. In the actual environment, the formation model trained in step S6 is deployed to each mobile carrier, the point cloud information and odometer information of the lidar are called, and the corresponding strategies and action instructions are output by calling the model according to the position and posture of each mobile carrier to complete the formation task.

2. A method for controlling ordered operations of multiple mobile carriers according to claim 1, characterized in that: The reinforcement learning network framework including the actor network and the critic network constructed in step S3 is specifically as follows: Establish an actor network to obtain lidar point cloud data, odometer information, and output the speed and angular velocity information of the mobile carrier; A critic network is established to obtain the relative position and speed information of the current mobile carrier, and output the reward data obtained in the current state.

3. A method for controlling ordered operations of multiple mobile carriers according to claim 2, characterized in that: The sampling process in step S4 includes sampling the interaction between multiple mobile carriers and the simulation environment. Each mobile carrier inputs its own observation data into its own actor network to generate the next action, and uses the critic network to evaluate the value of the mobile carrier's state to obtain reward data, and saves the experience of all mobile carriers into the experience pool.

4. A method for controlling ordered operations of multiple mobile carriers according to claim 3, characterized in that: The MADDPG algorithm in step S5 is based on the DDPG algorithm and is expanded from a single agent to multiple agents. It incorporates the ideas of centralized training and distributed execution, evaluates the overall reward data of all mobile carriers, and limits the sampled motion speed and angular velocity values ​​to a certain range.

Images