Obstacle avoidance control method for mobile robots and control method for multiple mobile robots

Through the combination of deep reinforcement learning and PID control, the problem of poor sensor accuracy and insufficient processing capabilities of high-dimensional state space when avoiding obstacles in complex environments is solved, and more efficient and stable obstacle avoidance performance is achieved.

CN118938931APending Publication Date: 2024-11-12GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411168010.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When existing mobile robots avoid obstacles in complex environments, they have poor sensor accuracy and strong environmental interference, and cannot effectively deal with multiple obstacles and high-dimensional state spaces. The traditional Q-learning algorithm lacks the ability to handle complex obstacle avoidance tasks.

Method used

Adopt the mobile robot trajectory tracking and obstacle avoidance control method based on deep reinforcement learning, combined with DQN and PID control, and by building a deep Q network and reward mechanism, robots are encouraged to take the best actions to achieve obstacle avoidance, using ε-greedy strategy and PID control The device improves the robot's obstacle avoidance accuracy and stability.

Benefits of technology

Improve the obstacle avoidance ability of mobile robots, enhance their independent learning and adaptability in complex environments, make up for the poor processing capabilities of traditional Q-learning in high-dimensional state space, and achieve more efficient and stable obstacle avoidance performance. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938931A_ABST
    Figure CN118938931A_ABST
Patent Text Reader

Abstract

The invention aims to provide a mobile robot obstacle avoidance control method and a control method for multiple mobile robots. The method comprises the following steps: constructing a robot model and a closed space of at least two obstacles; constructing reinforcement learning training, and obtaining an optimal action in the reinforcement learning model; rewards are obtained through the difference between the optimal action and the current action, and the mobile robot is encouraged to take the action closer to the optimal action, so that the robot model is controlled to perform obstacle avoidance and navigation; according to the method provided by the invention, the defects that the traditional Q-learning cannot adapt to a complex obstacle avoidance environment and the high-dimensional state space processing capability is poor can be overcome.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of mobile robot obstacle avoidance, and in particular, relates to a mobile robot trajectory tracking and obstacle avoidance control method based on deep reinforcement learning and an obstacle avoidance control method for multiple mobile robots. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly, and mobile robots have been widely used in various fields, especially in complex environments. The demand for mobile robots is increasing. The use of mobile robots can be seen everywhere in life and work, such as home sweeping robots, food delivery robots in the express delivery and takeaway industries, and food delivery robots in the catering industry. However, most of these common mobile robots require human intervention. Sweeping robots need to collide multiple times to avoid obstacles. Food delivery and food delivery robots often fail to avoid obstacles during their tasks due to the complex environment. Faced with these situations, people hope that mobile robots can achieve autonomous navigation and real-time obstacle avoidance to minimize human intervention.

[0003] Mobile robots need to achieve obstacle avoidance tasks mainly by using laser radar or ultrasonic, infrared and other technologies. However, the above methods have the disadvantages of poor accuracy of obstacle avoidance sensors, strong interference from strong light or strong wind, inability to face multiple obstacles, and high cost of high-performance sensors.

[0004] Reinforcement learning technology is a good way to achieve real-time obstacle avoidance for mobile robots. Reinforcement learning technology is to solve the problem of maximizing rewards or achieving specific goals through learning strategies during the interaction between intelligent agents and the environment. Therefore, applying reinforcement learning technology to the obstacle avoidance system of mobile robots can enhance the autonomous learning ability, real-time performance, robustness and adaptability of mobile robots. It can enable mobile robots to autonomously learn effective obstacle avoidance strategies through interaction with the environment without pre-programming specific rules, and can adjust their behavior strategies according to different environments and obstacle layouts, so that mobile robots can avoid obstacles more intelligently and efficiently. The Q-learning algorithm is a good reinforcement learning technology, but Q-learning only considers single obstacle avoidance and does not consider long-term dynamic avoidance of multiple targets. The Q-learning algorithm has a relatively weak processing capability for high-dimensional state space and is not suitable for complex obstacle avoidance tasks.

[0005] Based on this, a mobile robot trajectory tracking and obstacle avoidance control method based on deep reinforcement learning needs to be developed urgently. Summary of the invention

[0006] The purpose of the present application is to provide a mobile robot obstacle avoidance control method and a multi-mobile robot control method, so as to at least solve one technical problem in the prior art.

[0007] The technical solution of this application is:

[0008] A mobile robot obstacle avoidance control method based on deep reinforcement learning and PID control for mobile robot trajectory tracking and obstacle avoidance, comprising:

[0009] Construct a robot model and a closed space with at least 2 obstacles;

[0010] Constructing reinforcement learning training to obtain the optimal action in the reinforcement learning model;

[0011] The reward is obtained by the difference between the optimal action and the current action, and the mobile robot is encouraged to take actions closer to the optimal action to control the robot model to avoid obstacles and navigate.

[0012] The step of constructing reinforcement learning training and obtaining the optimal action in the reinforcement learning model includes:

[0013] Build a deep Q network;

[0014] Obtaining the action with the largest Q value through the deep Q network, and taking the largest action as the optimal action;

[0015] The weights of the deep Q network are updated according to the Q value.

[0016] The construction of the deep Q network includes, in sequence:

[0017] Sequential model, through a linear stack of multiple network layers arranged in sequence;

[0018] The first fully connected layer, whose input is the size of the state space, is used to extract preliminary nonlinear features from the input state;

[0019] The second fully connected layer, whose input is determined by the output shape of the first fully connected layer;

[0020] The third fully connected layer, as the output layer, has the output of the second fully connected layer as its input, and the number of its output neurons is the size of the action space;

[0021] The state space is determined by the data of the mobile robot's built-in laser radar scanning the surrounding environment and the action value; the action space is the Actionspace of Turtlebot3, which contains all possible actions. The actions represent the basic operations that the robot can perform in different states, such as forward, backward, left turn, right turn, etc.

[0022] The constructing of the deep Q network also includes:

[0023] The Dropout layer is set between the second fully connected layer and the third fully connected layer to prevent overfitting.

[0024] The step of obtaining the action with the maximum Q value through the deep Q network and taking the maximum action as the optimal action includes:

[0025] The action selection adopts the ε-greedy strategy, where ε (epsilon) is a parameter between 0 and 1, indicating the probability of exploration;

[0026] When the generated random number is less than ε, a random action is selected;

[0027] When the generated random number is greater than ε, select the action with the largest estimated Q value under the current strategy;

[0028] The action with the largest estimated Q value is selected as the optimal action.

[0029] The updating of the weight of the deep Q network according to the Q value comprises:

[0030] The Q value is obtained by the following formula:

[0031] Q(s,a)=r+γmax a′ Q target (s′, a′);

[0032] Among them, Q(s,a) refers to the Q value of performing action a in state s; r is the action reward obtained by performing action a; γ is the discount factor indicating the importance of future rewards; max a′ Q target (s′, a′) refers to the maximum Q value of all possible actions a′ in the new state s′.

[0033] The reward is obtained by the difference between the optimal action and the current action, and the mobile robot is encouraged to take an action closer to the optimal action, so as to control the robot model to avoid obstacles and navigate, including:

[0034] Set up a reward function, where the reward is determined by the current action and the best action;

[0035] Add a PID controller to control the linear speed and angular velocity of the mobile robot, so that it can better achieve obstacle avoidance and training tasks;

[0036] Adjust the reference trajectory coordinates of the robot's movement to determine the target position and direction of the robot's movement.

[0037] The reward function is set, and the reward is determined by the current action and the best action, including:

[0038] The reward is obtained by: the best action and the current action, and the reward is obtained by the difference between the best action and the current action, including:

[0039]

[0040] reward-=-|action|*1;

[0041] Among them, reward represents reward, action represents the current action, and best_action represents the best action; reward-=-|action|*1, the larger the action value, the greater the reward, which can encourage the robot to take bigger actions;

[0042] The formula can reflect the gap between the mobile robot's action and the ideal action, and encourage the mobile robot to take actions closer to the optimal one.

[0043] The best action best_action is input by the data scanned by the mobile robot's laser radar and the constraints of its processing of the scanned data; according to the left force of the mobile robot leftforce: And the right force rightforce: Compare to determine the best action best_action for the mobile robot;

[0044] Among them, limit[1] and limit[0] are the upper and lower limits of the constraint conditions.

[0045] The method of adding a PID controller to control the linear velocity and angular velocity of the mobile robot includes:

[0046] The formula of the PID controller is expressed as:

[0047] output=K p *error+K i *integral+K d *derivative;

[0048] Among them, K p is the proportionality coefficient; K i is the integration coefficient; K d is the differential coefficient; error is the error between the current state and the target value; integral is the accumulated value of the error; derivative is the rate of change of the error;

[0049] By formula To adjust the reference trajectory coordinates;

[0050] Where (x, y) is the original coordinate of the reference trajectory, θ is the angle of the reference trajectory, and Δl is the adjustment length;

[0051] Get the target distance d:

[0052] Among them, (x0, y0) is the current position of the mobile robot;

[0053] Get the target angle α:

[0054] Among them, θ0 is the current robot's orientation angle, α is the target angle, and the angle difference is normalized within the range of [-π, π]: Since the angle difference of α needs to be normalized to [-π, π], the target angle α is subtracted by 2π or added by 2π to keep α in [-π, π].

[0055] A control method for multiple mobile robots, comprising:

[0056] Create at least two mobile robots, assign a corresponding position and an independent space and at least two trajectories to each mobile robot, and make the mobile robots follow their own trajectories respectively;

[0057] The mobile robots share the same deep reinforcement learning neural network model, so that any mobile robot executes the mobile robot obstacle avoidance control method as described above, so that any mobile robot has the same obstacle avoidance capability and obstacle avoidance strategy.

[0058] The beneficial effects of this application include at least:

[0059] The obstacle avoidance control method for a mobile robot described in the present application first constructs a robot model and a closed space with at least two obstacles; then constructs reinforcement learning training to obtain the optimal action in the reinforcement learning model; finally, rewards are obtained by comparing the difference between the optimal action and the current action, encouraging the mobile robot to take actions closer to the optimal action to control the robot model for obstacle avoidance and navigation. The method described in the present application uses the DQN algorithm to make up for the shortcomings of the traditional Q-learning algorithm that cannot adapt to complex obstacle avoidance environments and poor processing capabilities of high-dimensional state spaces, while improving the stability and convergence of the mobile robot; at the same time, by combining deep reinforcement learning with PID, the linear velocity and angular velocity of the mobile robot can be controlled by the PID controller to improve the accuracy and stability of the mobile robot in performing obstacle avoidance tasks; and the potential field method can enable the robot to automatically avoid obstacles, dynamically avoid obstacles and move toward the target point. This method is simple to calculate and is suitable for real-time control. Finally, the method adopts a reward mechanism based on the potential field method to calculate the optimal action, and calculates the reward by comparing the difference between the actual action and the optimal action; the method described in the present application can make up for the shortcomings of the traditional Q-learning algorithm that cannot adapt to complex obstacle avoidance environments and poor processing capabilities of high-dimensional state spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the obstacle avoidance control method for a mobile robot described in the present application;

[0061] Figure 2 This is a diagram of using Gazebo to construct a closed space with multiple obstacles and mobile robot trajectory tracking and obstacle avoidance;

[0062] Figure 3 This is a control block diagram of a mobile robot obstacle avoidance control method based on deep reinforcement learning and PID;

[0063] Figure 4 It is a schematic diagram of trajectory tracking and obstacle avoidance of multiple mobile robots. DETAILED DESCRIPTION

[0064] The present application is further described below in conjunction with the accompanying drawings.

[0065] The purpose of the present application is to provide a method for controlling obstacle avoidance of a mobile robot based on deep reinforcement learning and PID. The present invention uses DQN (Deep Q-Network) technology to replace traditional Q-learning. Compared with traditional Q-learning, DQN can use deep neural networks to approximate the Q-value function so that it can process high-dimensional state space; at the same time, adding a PID controller can control the linear speed and angular velocity of the mobile robot, allowing it to better achieve obstacle avoidance tasks and training tasks. Based on this concept, the present application provides the following embodiments to explain the technical solution of the present application in detail.

[0066] Specific embodiment 1:

[0067] This application provides an embodiment:

[0068] The present invention proposes a mobile robot obstacle avoidance control method based on deep reinforcement learning, the specific characteristics of which are as follows: Figure 1 , follow the steps below:

[0069] Step 1: Create a simulated mobile robot:

[0070] According to the effect of the invention, the mobile robot can avoid obstacles in an unknown space, so the simulated mobile robot is preferably Turtlebot3-Burger.

[0071] Step 2: Build a closed space simulation environment:

[0072] like Figure 2 , use Gazebo to build a closed space with various obstacles to simulate the real environment, and call the Turtlebot3-Burger mobile robot described in step 1 to perform obstacle avoidance tasks in the simulation environment.

[0073] Step 3: Build DQN reinforcement learning training and control the mobile robot:

[0074] Step 3.1: Save each simulation experiment. Each simulation experiment will be stored in JSON and h5 format files. Before starting the experiment, you can decide whether to load the weights and parameters of the pre-trained model according to your own needs. If you need to load, it will load the weights of the model from the specified path and read the corresponding JSON file to obtain and set the epsilon value. In this way, you can use the saved model status and parameters when continuing training or testing.

[0075] Step 3.2: Build a Deep Q Network (DQN):

[0076] First, create a sequential model model = Sequential(). The sequential model is a linear stack of multiple network layers arranged in sequence.

[0077] Then add the first fully connected layer:

[0078] model.add(Dense(64,input_shape=(self.state_size,),activation='relu',kernel_initializer='lecun_uniform')), this fully connected layer contains 64 neurons, and its input is the size of the state space, which is determined by the data of the mobile robot's built-in laser radar scanning the surrounding environment and the action value. Its activation function is the RELU function: Relu(x)=max(0,x). The nonlinear transformation is performed through the RELU function. In this way, the model can extract preliminary nonlinear features from the input state.

[0079] Add a second fully connected layer. The second fully connected layer also has 64 neurons. Its input is determined by the output shape of the previous fully connected layer, and the activation function is also the RELU function.

[0080] Then add a Dropout layer with a dropout rate of 0.2, which can be used to prevent overfitting.

[0081] Finally, a third fully connected layer is added as the output layer. The input is the output of the second fully connected layer. The number of its output neurons is the size of the action space, expressed as output = [o1, o2, ..., o n ], where n is equal to the size of the action space. That is, if the action space size is 4, then the output vector may be [Q(s,a_1),Q(s,a_2),Q(s,a_3),Q(s,a_4)], where Q(s,a_i) represents the estimated Q value of executing action a_i in the current state s. The final decision is usually to choose the action with the largest Q value.

[0082]

[0083] Step 3.3: Select Action:

[0084] The action selection of the mobile robot adopts the ε-greedy strategy. ε (epsilon) is a parameter between 0 and 1, which represents the probability of exploration. When the generated random number is less than ε, a random action is selected. This allows the exploration of new states and actions and prevents the model from falling into a local optimal solution. When the generated random number is greater than ε, the action with the largest estimated Q value under the current strategy is selected. This allows the current knowledge of the model to be used to select the optimal action.

[0085] Step 3.4: According to step 3.2, the choice of the mobile robot's action is related to the Q value, so the Q value needs to be calculated to update the weight of the Q network. The calculation of the Q value is based on the Bellman equation: Q(s,a)=r+γmax a′ Q target (s′, a′). Where Q(s,a) refers to the Q value of executing action a in state s; r is the action reward obtained by executing action a; γ is the discount factor indicating the importance of future rewards; max a′ Q target (s′, a′) refers to the maximum Q value of all possible actions a′ in the new state s′.

[0086] In this embodiment, DQN can use the experience replay method to store and reuse previous experience data, which can make the training samples more efficiently used, reduce the correlation between samples, and improve the stability and convergence of training. In contrast, traditional Q-learning usually adopts online learning, which is easily affected by the correlation between samples and timing deviation. Therefore, DQN can make up for the shortcomings of traditional Q-learning that cannot adapt to complex obstacle avoidance environments and has poor processing capabilities of high-dimensional state spaces.

[0087] Step 4: Control the obstacle avoidance and navigation of the mobile robot, such as Figure 3 :

[0088] Step 4.1: Set up the reward function:

[0089] The calculation of reward depends on two parts, the best action and the current action. The reward is obtained by the difference between the best action and the current action, and the reward is subtracted from the absolute value of the action, which can be expressed by the formula:

[0090]

[0091] reward-=-|action|*1;

[0092] The formula can reflect the gap between the mobile robot's action and the ideal action, and encourage the mobile robot to take actions closer to the optimal one.

[0093] Furthermore, the best action (best_action) is a potential field calculated by the mobile robot's built-in laser radar according to the surrounding environment. The potential field is usually used to guide the mobile robot to take action so that it moves in the direction with the minimum "potential energy". Specifically: best_action is formed by the scandata scanned by the mobile robot's laser radar and the limit condition limit for processing the scan data as function input, and then the left force (leftforce) of the mobile robot is calculated: And the right force: Where limit[1] and limit[0] are the upper and lower limits of the constraint. The best_action of the mobile robot is determined by comparing the left and right side forces.

[0094] The reward mechanism in this embodiment uses a reward mechanism based on the artificial potential field method to calculate the best action, which allows the mobile robot to quickly respond to environmental changes and more effectively avoid collisions between robots and obstacles. The artificial potential field method uses the superposition of attractive and repulsive forces to generate a potential field. This method has a small amount of calculation, is easy to apply in real time, can quickly generate the path of robot movement, and is suitable for obstacle avoidance tasks with high real-time requirements. Moreover, for multiple mobile robots, the artificial potential field method can achieve collision avoidance and collaboration between robots by setting the potential field of each robot.

[0095] Step 4.2: Control the movement of the mobile robot:

[0096] According to step 4.1, we know that the reward is determined by the current action and the best action, so we need to control the movement of the mobile robot. The mobile robot performs obstacle avoidance and navigation tasks in a confined space, so it needs to calculate the target position based on the reference trajectory and finally reach the target position. In order to enable the mobile robot to better perform obstacle avoidance tasks and get closer to the target position faster, a PID controller is added here. The PID controller can control the linear velocity and angular velocity of the mobile robot, allowing it to better achieve obstacle avoidance tasks and training tasks.

[0097] Furthermore, the main function of the PID controller is to calculate the output value of the controller based on the difference between the target value and the actual value, so as to adjust the system behavior and make the mobile robot move closer to the target state. The formula is: output = K p *error+K i *integral+K d *derivative where K p is the proportionality coefficient; K i is the integration coefficient; K dis the differential coefficient; error is the error between the current state and the target value; integral is the accumulated value of the error; derivative is the rate of change of the error. This embodiment adds a PID controller, which can provide dynamic adjustment of the linear speed and angular speed control of the mobile robot. When the mobile robot performs an obstacle avoidance task, it can ensure that the linear speed and angular speed of the robot are within a safe and reasonable range, which helps to prevent the mobile robot from rotating too much or too little, thereby better achieving the obstacle avoidance task and improving the accuracy and stability of the mobile robot in the obstacle avoidance process.

[0098] Step 4.3: According to step 4.2, we know that the mobile robot motion determines the target position and direction according to the reference trajectory. Therefore, we can use the formula To adjust the reference trajectory coordinates, where (x, y) is the original coordinates of the reference trajectory, θ is the angle of the reference trajectory, and Δl is the adjustment length. Then calculate the target distance: (x0, y0) is the current position of the mobile robot. Then calculate the target angle: θ0 is the current robot's orientation angle, α is the target angle, and the angle difference is normalized within the range of [-π, π]:

[0099] Specific embodiment II:

[0100] The present invention provides an embodiment:

[0101] In daily life, more than one mobile robot is used. In order to improve efficiency, multiple mobile robots are generally used. This requires that commands can be issued to multiple mobile robots at the same time, and they can all avoid obstacles while avoiding the running robots. At the same time, Q-learning is not suitable for the control tasks of multiple mobile robots. To achieve multiple mobile robots receiving commands at the same time and having the same obstacle avoidance capabilities, better algorithms are needed, and distributed control of multiple mobile robots is required.

[0102] Based on the above discussion, this embodiment discloses a control method for multiple mobile robots, including:

[0103] Create at least two mobile robots, assign a corresponding position and an independent space and at least two trajectories to any mobile robot, and make the mobile robots track their respective trajectories; make the mobile robots share the same deep reinforcement learning neural network model, and make any mobile robot execute the mobile robot obstacle avoidance control method as described in specific embodiment I, so that any mobile robot has the same obstacle avoidance capability and obstacle avoidance strategy.

[0104] According to steps 1-4 in specific embodiment I, obstacle avoidance control of a single mobile robot can be achieved. To achieve control of multiple mobile robots, improvements need to be made on the control method of a single mobile robot.

[0105] Specifically, Figure 4 , to achieve multi-mobile robot control, you need to create multiple mobile robots, that is, add multiple mobile robots to the original single mobile robot file, and name them respectively and write their corresponding positions and independent spaces, so that there will be no errors during control. In order to achieve the control of multiple mobile robots for obstacle avoidance, each robot needs to be given the same obstacle avoidance capability. Therefore, multiple mobile robot instances need to be added in the main loop so that all robots share the same deep reinforcement learning neural network model. This ensures that each robot has the same obstacle avoidance capability and strategy without the need to train a separate model for each robot. This not only simplifies the training process, but also improves the overall efficiency of the system. Further, increase from one trajectory to multiple trajectories, and let the mobile robots track their own trajectories respectively.

[0106] Specific embodiment III:

[0107] This application provides an embodiment:

[0108] An electronic device for obstacle avoidance control of a mobile robot comprises: a storage medium and a processing unit; wherein the storage medium is used to store a computer program, and the processing unit exchanges data with the storage medium, and is used to execute the computer program through the processing unit when performing obstacle avoidance control of the mobile robot, and perform the steps of the obstacle avoidance control method of the mobile robot as described in specific embodiment I.

[0109] A computer-readable storage medium having a computer program stored therein; when the computer program is run, the steps of the mobile robot obstacle avoidance control method described in specific embodiment I are executed.

[0110] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0111] The above disclosures are only several specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be thought of by a person skilled in the art should fall within the protection scope of the present application. The above application numbers are only for description and do not represent the advantages and disadvantages of the implementation scenarios.

Claims

1. A mobile robot obstacle avoidance control method based on deep reinforcement learning and PID control for mobile robot trajectory tracking and obstacle avoidance, characterized in that: include: Construct a robot model and a closed space with at least 2 obstacles; Constructing reinforcement learning training to obtain the optimal action in the reinforcement learning model; The reward is obtained by the difference between the optimal action and the current action, and the mobile robot is encouraged to take actions closer to the optimal action to control the robot model to avoid obstacles and navigate.

2. The mobile robot obstacle avoidance control method according to claim 1, characterized in that: The step of constructing reinforcement learning training and obtaining the optimal action in the reinforcement learning model includes: Build a deep Q network; Obtaining the action with the largest Q value through the deep Q network, and taking the largest action as the optimal action; The weights of the deep Q network are updated according to the Q value.

3. The mobile robot obstacle avoidance control method according to claim 2, characterized in that: The construction of the deep Q network includes, in sequence: Sequential model, through a linear stack of multiple network layers arranged in sequence; The first fully connected layer, whose input is the size of the state space, is used to extract preliminary nonlinear features from the input state; The second fully connected layer, whose input is determined by the output shape of the first fully connected layer; The third fully connected layer, as the output layer, has the output of the second fully connected layer as its input, and the number of its output neurons is the size of the action space; The state space is determined by the data of the mobile robot's own laser radar scanning the surrounding environment and the action value.

4. The mobile robot obstacle avoidance control method according to claim 3, characterized in that: The constructing of the deep Q network also includes: The Dropout layer is set between the second fully connected layer and the third fully connected layer to prevent overfitting.

5. The mobile robot obstacle avoidance control method according to claim 2, characterized in that: The step of obtaining the action with the maximum Q value through the deep Q network and taking the maximum action as the optimal action includes: The action selection adopts the ε-greedy strategy, where ε (epsilon) is a parameter between 0 and 1, indicating the probability of exploration; When the generated random number is less than ε, a random action is selected; When the generated random number is greater than ε, select the action with the largest estimated Q value under the current strategy; The action with the largest estimated Q value is selected as the optimal action.

6. The mobile robot obstacle avoidance control method according to claim 2, characterized in that: The updating of the weight of the deep Q network according to the Q value comprises: The Q value is obtained by the following formula: Q(s,a)=r+γmax a′ Q target (s′,a′); Among them, Q(s,a) refers to the Q value of performing action a in state s; r is the action reward obtained by performing action a; γ is the discount factor indicating the importance of future rewards; max a′ Q target (s′,a′) refers to the maximum Q value of all possible actions a′ in the new state s′.

7. The mobile robot obstacle avoidance control method according to claim 1, characterized in that: The reward is obtained by the difference between the optimal action and the current action, and the mobile robot is encouraged to take an action closer to the optimal action, so as to control the robot model to avoid obstacles and navigate, including: Set up a reward function, where the reward is determined by the current action and the best action; Add a PID controller to control the linear speed and angular velocity of the mobile robot, so that it can better achieve obstacle avoidance and training tasks; Adjust the reference trajectory coordinates of the robot's movement to determine the target position and direction of the robot's movement.

8. The mobile robot obstacle avoidance control method according to claim 7, characterized in that: The reward function is set, and the reward is determined by the current action and the best action, including: The reward is obtained by: the best action and the current action, and the reward is obtained by the difference between the best action and the current action, including: reward-=-|action|*1; The formula can reflect the gap between the mobile robot's action and the ideal action, and encourage the mobile robot to take actions closer to the optimal one. Among them, reward means reward; action means current action; best_action means best action; The best action best_action is input by the data scanned by the mobile robot's laser radar and the constraints of its processing of the scanned data; according to the left force of the mobile robot leftforce: And the right force rightforce: Compare to determine the best action best_action for the mobile robot; Among them, limit[1] and limit[0] are the upper and lower limits of the constraint conditions.

9. The mobile robot obstacle avoidance control method according to claim 7, characterized in that: The method of adding a PID controller to control the linear velocity and angular velocity of the mobile robot includes: The formula of the PID controller is: output = K p *error+K i *integral+K d *derivative; Among them, K p is the proportionality coefficient; K i is the integration coefficient; K d is the differential coefficient; error is the error between the current state and the target value; integral is the accumulated value of the error; derivative is the rate of change of the error; By formula To adjust the reference trajectory coordinates; Where (x, y) is the original coordinate of the reference trajectory, θ is the angle of the reference trajectory, and Δl is the adjustment length; Get the target distance d: Among them, (x0, y0) is the current position of the mobile robot; Get the target angle α: Among them, θ0 is the current robot's orientation angle, α is the target angle, and the angle difference is normalized within the range of [-π, π]:

10. A control method for multiple mobile robots, characterized in that: Create at least two mobile robots, assign a corresponding position and an independent space and at least two trajectories to each mobile robot, and make the mobile robots follow their own trajectories respectively; The mobile robots share the same deep reinforcement learning neural network model, so that any mobile robot executes the mobile robot obstacle avoidance control method as described in any one of claims 1 to 9, so that any mobile robot has the same obstacle avoidance capability and obstacle avoidance strategy.