Method for controlling cooperative motion of multiple intelligent wheelchairs based on reinforcement learning

Abstract

The invention discloses a multi-intelligent wheelchair cooperative motion control method based on reinforcement learning, and the method comprises the steps: 1, obtaining data from a digital twin system of a health-care community, transmitting the data to a trained reinforcement learning network, and transmitting an obtained final decision to a wheelchair of a user; step 2, a training strategy based on an MADDPG reinforcement learning control algorithm; and step 3, the system finally issues an instruction, and selects to execute proper path planning and obstacle avoidance control according to the current road condition. According to the scheme, effective communication and cooperation are established among the multiple intelligent wheelchairs, the operation efficiency of a single wheelchair is improved, meanwhile, the overall energy consumption of the system is reduced, and the operation safety in the dynamic complex environment is improved. A reliable and stable travel mode is provided, and the life quality of the old people is improved.

Description

Technical Field

[0001] The present invention relates to intelligent agent technology, and in particular to a method for collaborative motion control of multiple intelligent wheelchairs based on reinforcement learning. Background Art

[0002] Multi-agent technology is an important branch in the field of artificial intelligence. It realizes collaborative work and collective intelligence by combining multiple agents. Multi-agent systems have a wide range of applications, including robot collaboration, unmanned vehicles, logistics planning, social networks, and power grids. In multi-agent systems, interaction and cooperation between agents are key. Agents can achieve cooperation by transmitting information, sharing knowledge, and coordinating actions. In order to achieve good collaborative effects, multi-agent systems need to solve challenges such as communication, cooperation strategies, conflict resolution, and balance between individual goals. When designing and developing multi-agent systems, it is necessary to consider the capabilities and characteristics of each agent, as well as the interactions between them. Commonly used multi-agent technologies include game theory, collaborative algorithms, cooperative learning, and distributed optimization. These technologies can help agents make decisions under incomplete information and dynamic environments and maximize the utility of the overall system.

[0003] As the problem of population aging becomes increasingly prominent, smart wheelchair technology has received extensive attention and development. The goal of smart wheelchairs is to use advanced sensors, navigation systems and autonomous control algorithms to provide users with safer, more convenient and personalized mobility solutions. The technical background of smart wheelchairs involves the integration of knowledge and technology from multiple fields. These include robotics, perception and navigation technology, human-computer interaction technology and electric vehicle technology. By integrating these key technologies, smart wheelchairs can achieve functions such as autonomous obstacle avoidance, path planning, positioning navigation and intelligent control. Sensors play a key role in the design and development of smart wheelchairs. Smart wheelchairs are usually equipped with a variety of sensors, such as lidar, cameras, ultrasonic sensors and inertial measurement units. These sensors can sense the surrounding environment and user status, identify obstacles, judge road conditions and monitor user posture and psychological needs. Navigation and path planning are one of the core technologies of smart wheelchairs. By combining map data, user needs and environmental information, smart wheelchairs can plan the best movement path while considering obstacle avoidance and safety. The design of navigation and path planning algorithms needs to take into account many factors, such as road conditions, traffic conditions, user preferences and obstacle recognition.

[0004] As an important branch of machine learning, reinforcement learning is committed to solving the problem of learning and decision-making of intelligent agents in unknown environments. Compared with traditional supervised learning or unsupervised learning, reinforcement learning has stronger autonomy and self-learning ability. The technical background of reinforcement learning comes from the comprehensive application of multiple disciplines such as mathematics, cybernetics and psychology. The theoretical framework based on Markov decision process (MDP) is the core foundation of reinforcement learning. The MDP model describes the interaction process between an intelligent agent and the environment. By defining elements such as state, action, reward function and state transition probability, as well as an optimization goal, reinforcement learning can achieve optimal decision-making by learning and optimizing strategies. One of the key technologies of reinforcement learning is the modeling and optimization of value functions. The value function is used to evaluate the long-term reward expectation obtained by the intelligent agent under a specific state or action, including state value function and action value function. Through learning and optimizing the value function, reinforcement learning can discover the optimal decision-making strategy and obtain the maximum cumulative reward in a complex environment.

[0005] The existing technology has the following shortcomings: on the one hand, the current mainstream society invests too little in health care communities, and most of the investment is concentrated on public welfare elderly care services such as nursing homes; on the other hand, there are currently few multi-agent control methods for the simultaneous travel of multiple smart wheelchairs. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a method for collaborative motion control of multiple intelligent wheelchairs based on reinforcement learning.

[0007] Technical solution: The method for collaborative motion control of multiple intelligent wheelchairs based on reinforcement learning described in the present invention comprises the following steps:

[0008] Step 1: Obtain data from the digital twin system of the health care community, then feed the data into the trained reinforcement learning network, and the final decision is then sent to the user's wheelchair;

[0009] Step 2: Training strategy based on MADDPG reinforcement learning control algorithm;

[0010] Step 3: The system finally issues instructions to select and execute appropriate path planning and obstacle avoidance control based on the current road conditions.

[0011] Furthermore, the step 1 acquires data through a wireless network or a cellular data network.

[0012] Furthermore, the method of step 2 is:

[0013] (1) Initialize the Actor network u of agent i i , target Actor network u′ i , Critic Network Target Critic Network

[0014] (2) Initialize the attenuation factor γ and the soft update coefficient τ;

[0015] (3) For episode = 1 to Mdo;

[0016] (4) Initialize the environment and state set S of all agents;

[0017] (5) Perform the following loop for t = 1 to episode_maxstep and for each agent i in the environment:

[0018] (6) Agent i selects an action based on the current Actor network and exploration noise.

[0019] (7) Execute action Get the next state And get instant returns

[0020] (8) Training experience Deposit into the land;

[0021] (9) For each agent i=1,…,N, execute the following loop:

[0022] (10) From D i Randomly extract batcH_size empirical data;

[0023] (11) Calculate the expected return according to formula (2);

[0024] (12) According to formula (5), minimize the loss function to update the Critic network;

[0025] (13) Update the Actor network using the gradient descent method according to formula (4);

[0026] (14) Update the target network parameters of each agent i;

[0027] (15) End the loop for each agent i at time t.

[0028] (16) End the loop of the episode.

[0029] (17) End the update loop for each agent i.

[0030] Explanation of relevant terms:

[0031] Multi-agent refers to a system composed of multiple independent individuals, each of which has its own knowledge, ability and decision-making ability, and can complete a task or achieve a goal through mutual interaction and collaboration. Multi-agent systems can be applied in many fields, such as robot collaboration, intelligent transportation systems, distributed sensor networks, collaborative decision-making, etc. In these systems, multiple intelligent individuals collaborate with each other to complete complex tasks. Each intelligent individual can make decisions based on changes in the environment and the behavior of other individuals, and achieve the overall goal through mutual communication and cooperation. Smart wheelchairs are assistive tools that use artificial intelligence and robotics technology to provide a higher degree of autonomy and convenience to people with limited mobility, including people with disabilities or elderly people with limited mobility. Smart wheelchairs are usually equipped with various sensors and navigation systems to automatically sense the surrounding environment and navigate. They can autonomously plan paths and avoid obstacles based on user instructions or preset destinations by using global positioning systems (GPS) or other positioning technologies. Reinforcement learning is a machine learning method that enables intelligent systems to gradually improve and optimize their behavior through a process of trial and error and feedback by imitating the way humans learn. In reinforcement learning, intelligent systems are called agents, which learn by interacting with the environment. The agent can observe the state of the environment and choose actions based on these states. After performing an action, it receives a feedback signal, called a reward, which is used to evaluate whether the action is good or bad. The goal of the agent is to maximize the cumulative reward through continuous trial and learning.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The solution of the present invention not only improves the operating efficiency of a single wheelchair by establishing effective communication and collaboration between multiple smart wheelchairs, but also reduces the overall energy consumption of the system and improves the operating safety in dynamic and complex environments.

[0034] 2. The present invention provides a reliable and stable travel method for elderly people who have difficulty in traveling, thereby improving the quality of life of the elderly population. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The overall process of the collaborative motion control method for multiple intelligent wheelchairs based on reinforcement learning;

[0036] Figure 2 Schematic diagram of obstacle avoidance strategy. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0038] In response to the two issues mentioned in the fourth section, this embodiment proposes the following Figure 1 The overall process shown is:

[0039] Step 1: Obtain data from the digital twin system of the health care community through a wireless network or cellular data network, and then send the real-time data to the trained reinforcement learning network. The final decision is then sent to the user's wheelchair.

[0040] Step 2: Training strategy based on MADDPG reinforcement learning control algorithm:

[0041] (1) Initialize the Actor network u of agent i i , target Actor network u′ i , Critic Network Target Critic Network

[0042] (2) Initialize the attenuation factor γ and the soft update coefficient τ

[0043] (3) For episode = 1 to M do:

[0044] (4) Initialize the environment and state set S of all agents;

[0045] (5) Perform the following loop for t = 1 to episode_maxstep and for each agent i in the environment:

[0046] (6) Agent i selects actions based on the current Actor network and exploration noise

[0047] (7) Execute action Get the next state And get instant returns

[0048] (8) Training experience Deposit D i ; In the algorithm, t represents the current time step, traversing from t=1 to episode_maxstep; i represents the index of the agent, traversing each agent; is the observed state of the ith agent at time step t; is the action of the ith agent at time step t; is the reward obtained by the ith agent after performing the action at time step t; is the observed state of the ith agent at time step t+1.

[0049] (9) For each agent i=1,…,N, execute the following loop:

[0050] (10) From D i Randomly extract batch_size empirical data from;

[0051] (11) Calculate the expected return according to formula (2);

[0052] (12) According to formula (5), minimize the loss function to update the Critic network;

[0053] (13) Update the Actor network using the gradient descent method according to formula (4);

[0054] (14) Update the target network parameters of each agent i;

[0055] (15) End the loop for each agent i at time t.

[0056] (16) End the loop of the episode.

[0057] (17) End the update loop for each agent i.

[0058] Step 3: The system finally issues instructions to select and execute appropriate path planning and obstacle avoidance control based on the current road conditions.

[0059] In order to solve the problems that the smart wheelchair may roll over due to high speed and large turning angle when automatically avoiding obstacles, and may fail to avoid obstacles in time due to slow speed and small turning angle, this study analyzed the relationship between the steering angle and speed of the smart wheelchair, and used the MADDPG reinforcement learning control algorithm to achieve speed control during automatic obstacle avoidance, thereby improving the safety of the wheelchair's automatic obstacle avoidance.

[0060] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for collaborative motion control of multiple intelligent wheelchairs based on reinforcement learning, characterized in that: The steps include: Step 1: Obtain data from the digital twin system of the health care community, then feed the data into the trained reinforcement learning network, and the final decision is then sent to the user's wheelchair; Step 2: Training strategy based on MADDPG reinforcement learning control algorithm; Step 3: The system finally issues instructions to select and execute appropriate path planning and obstacle avoidance control based on the current road conditions.

2. The method for controlling the coordinated motion of multiple intelligent wheelchairs based on reinforcement learning according to claim 1, characterized in that: The step 1 uses a wireless network or a cellular data network to acquire data.

3. The method for controlling the coordinated motion of multiple intelligent wheelchairs based on reinforcement learning according to claim 1, characterized in that: The method of step 2 is: (1) Initialize the Actor network u of agent i i , target Actor network u′ i , Critic Network Target Critic Network (2) Initialize the attenuation factor γ and the soft update coefficient τ; (3) For episode = 1 to M do; (4) Initialize the environment and state set S of all agents; (5) Perform the following loop for t = 1 to episode_maxstep and for each agent i in the environment: (6) Agent i selects actions based on the current Actor network and exploration noise (7) Execute action Get the next state And get instant returns (8) Training experience Deposit into the land; (9) For each agent i=1,…,N, execute the following loop: (10) From D i Randomly extract batch_size empirical data from; (11) Calculate the expected return according to formula (2); (12) According to formula (5), minimize the loss function to update the Critic network; (13) Update the Actor network using the gradient descent method according to formula (4); (14) Update the target network parameters of each agent i; (15) End the loop for each agent i at time t. (16) End the loop of the episode. (17) End the update loop for each agent i.

Images