Robot control method, device, medium and system

Abstract

The invention provides a robot control method, device, medium and system. According to the method, a strategy value function, a state value function and an action value function are generated by adopting a Markov decision process technology, and actions are one of normal movement, successful obstacle avoidance, slipping, moving, overturning and collision, so that complex working conditions are considered, a Bellman equation is iteratively updated, and the method is suitable for a large-scale industrial robot. And determining an optimal control strategy until the Bellman equation converges or until the current number of iterations is equal to the preset number of iterations, and controlling the robot to move by adopting the optimal control strategy, so that compared with an existing scheme, the robot can be effectively controlled aiming at complex working conditions, and the working efficiency of the robot is improved. Therefore, the problem that the control effect on the robot under the complex working conditions is poor due to the fact that control over the robot in an existing scheme cannot be applied to the complex working conditions is solved.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular, to a robot control method, device, medium and system. Background Art

[0002] State perception has always been a key and difficult issue in the research of the mobile robot industry. Mobile robots in complex environments will encounter various abnormal states such as slipping, moving, overturning and collision, and in some special environments there will be more complex states, so the robot is required to be able to accurately identify and judge these special abnormal states in a timely manner and make corresponding decisions. Traditional rule-based methods require modeling in advance, so during their application, they can only make judgments based on existing models, with poor generalization ability and inability to handle new situations; although methods based on neural network machine learning have made certain progress in pattern recognition, their recognition range is limited to neural networks, training models and preset models, and their generalization is poor.

[0003] That is, the existing solutions for controlling robots cannot be applied to more complex working conditions, resulting in poor control effects on robots under complex working conditions. Summary of the invention

[0004] The main purpose of the present application is to provide a robot control method, device, medium and system, so as to at least solve the problem that the existing solutions for robot control cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0005] In order to achieve the above object, according to one aspect of the present application, a robot control method is provided, the method comprising:

[0006] The Markov decision process technology is used to generate a strategy value function, a state value function and an action value function, wherein the strategy value function represents the probability of selecting an action in a given state, the state value function represents the cumulative reward expected to be obtained from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained from taking the action in the state under the strategy, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over;

[0007] Generate a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function;

[0008] The Bellman equation is iteratively updated until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine an optimal control strategy, and the optimal control strategy is used to control the movement of the robot.

[0009] According to another aspect of the present application, a control device for a robot is provided, the device comprising:

[0010] A first processing unit is used to generate a strategy value function, a state value function and an action value function by using Markov decision process technology, wherein the strategy value function represents the probability of selecting an action in a given state, the state value function represents the cumulative reward expected to be obtained from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained from taking the action in the state under the strategy, wherein the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over;

[0011] A second processing unit is used to generate a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function;

[0012] The third processing unit is used to iteratively update the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine an optimal control strategy, and adopt the optimal control strategy to control the movement of the robot.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0014] According to another aspect of the present application, a control system of a robot is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0015] By applying the technical solution of the present application, a strategy value function, a state value function and an action value function are generated by adopting Markov decision process technology, where the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning and collision, thereby taking complex working conditions into consideration, iteratively updating the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to the preset number of iterations, so as to determine the optimal control strategy, and adopt the optimal control strategy to control the movement of the robot, so that the present application can effectively control the robot under complex working conditions compared with the existing solutions, thereby solving the problem that the control of the robot under the existing solutions cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A schematic flow chart of a robot control method provided according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram showing a reinforcement learning algorithm flow in a robot control method provided in an embodiment of the present application;

[0019] Figure 3 A structural block diagram of a robot control device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0024] Markov Decision Process (MDP) is a mathematical framework for describing random decision problems. In MDP, the system is modeled as a mathematical model with state, action and reward. The system changes its state by selecting actions at each time step and receives a reward at each time step. Markov decision process technology refers to methods and techniques that use MDP to solve decision problems, such as algorithms based on value functions or policies, which are used to find the optimal strategy to maximize long-term rewards. MDP technology has a wide range of applications in reinforcement learning, control theory, artificial intelligence and other fields.

[0025] Ground sensors can detect the material of the ground by measuring parameters such as the reflectivity, conductivity, and speed of sound wave propagation of the ground. Specifically, ground sensors can use different technologies to identify the material of the ground, such as identifying the reflective properties of the ground material through the electromagnetic wave spectrum reflected by the ground, distinguishing different materials through the conductivity of the ground, and judging the hardness of the ground through the speed of sound wave propagation. In addition, ground sensors can also use machine learning algorithms to analyze and identify ground materials, and identify the characteristics of different ground materials through training models, thereby realizing the detection and identification of ground materials. Through these methods, ground sensors can accurately detect different ground materials and provide support for applications related to ground materials.

[0026] As introduced in the background technology, state perception has always been a key and difficult issue in the research of the mobile robot industry. Mobile robots in complex environments will encounter various abnormal states such as slipping, moving, overturning and collision, and in some special environments there will be more complex states, so the robot is required to be able to timely and accurately identify and judge these special abnormal states and make corresponding decisions. Traditional rule-based methods require modeling in advance, so they can only be judged based on existing models during their application, with poor generalization ability and inability to handle new situations; although methods based on neural network machine learning have made certain progress in pattern recognition and other aspects, their recognition range is limited to neural networks, training models and preset models, and their generalization is poor. In order to solve the problem that the existing solutions for robot control cannot be applied to more complex working conditions, resulting in poor control effects on robots under complex working conditions, the embodiments of the present application provide a robot control method, device, medium and system.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] In this embodiment, a robot control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 FIG. 1 is a flow chart of a robot control method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, using Markov decision process technology to generate a strategy value function, a state value function and an action value function, wherein the strategy value function represents the probability of selecting an action under a given state, the state value function represents the cumulative reward expected to be obtained starting from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained starting from the state under the strategy, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over;

[0031] Markov decision process is a mathematical modeling method for reinforcement learning problems. MDP consists of the following components:

[0032] State set S: all possible environment states.

[0033] Action set A: all possible actions.

[0034] Transition probability P(s'|s,a): The probability of transitioning to state s' after executing action a in state s.

[0035] Reward function R(s,a,s'): The immediate reward obtained when the state s is transferred to the state s' after executing action a in the state s.

[0036] Discount factor γ (usually (0\leq\gamma<1)): A discount on future rewards, used to balance immediate rewards and long-term rewards.

[0037] In one embodiment of the present application, a Markov decision process technique is used to generate a policy value function, including:

[0038] The above strategy value function is constructed as:

[0039] π(a|s)=P(\text{select action}a\text{in state}s);

[0040] Among them, π(a|s) is the probability that the above robot chooses action a in a given state s, and P is the probability.

[0041] State value function V π (s) represents the expected cumulative reward starting from state s under strategy π:

[0042]

[0043] E π is the expected value under strategy π, which is used to calculate the average value of the cumulative reward starting from state s when following strategy π;

[0044] It is the accumulation of time steps from time step t = 0 to infinity, which is used to consider the cumulative rewards of all future time steps starting from the current state;

[0045] γ t The discount factor γ is raised to the power of t, which reduces the weight of future rewards, making recent rewards more important than distant rewards;

[0046] R(s t ,a t ,s t+1 ) is the time step t, from state s t Execute action a t After that, transfer to state s t+1 The immediate rewards obtained when the time comes reduce the weight of future rewards, making the recent rewards more important than the long-term rewards.

[0047] s 0 =s is s 0 The initial state is equal to the given state s.

[0048] Action-value function Q π (s,a) represents the expected cumulative reward under strategy π, starting from state s and taking action a:

[0049]

[0050] Step S102, generating a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function;

[0051] Specifically, the Bellman equation for constructing the above state-value function is:

[0052] V π (s)=∑ a π(as)∑ s',γ p(s',γs,a)[γ+γV π (s')];

[0053] The Bellman equation for constructing the above action-value function is:

[0054] Q π (s,a)=Σ s',γ p(s',γ|s,a)[γ+γΣ a' π(a'|s')Q π (s',a')];

[0055] Among them, V π (s) is the state value of the robot in state s, π(a|s) is the probability that the robot selects action a in a given state s, p(s',γ|s,a) is the probability that the robot transfers to state s' after executing action a in state s under the influence of discount factor γ, V π (s') is the state value of the robot in state s', Q π (s,a) is the action value of the robot starting from the state s by taking the action a, Q π (s',a') is the action value of the robot taking the action a' starting from the state s', and π(a'|s') is the probability of the robot choosing action a' in a given state s'.

[0056] The use of Bellman equations in the field of robot control has the following benefits: Helping robots achieve optimal control: The Bellman equation provides a mathematical framework that can help robots find the optimal control strategy to achieve the best behavior in a given environment. Realizing dynamic planning of state space: The Bellman equation can help robots dynamically plan the state space, thereby finding the optimal action plan and improving the robot's decision-making efficiency and performance. Improving the learning efficiency of robots: Through the Bellman equation, robots can continuously adjust their behavior based on environmental feedback and rewards, thereby achieving faster learning and adaptation. Optimizing path planning and control strategies: The Bellman equation can help robots find the best path planning and control strategies in complex environments, improving the execution efficiency and success rate of robots in various tasks. Improving the robustness and adaptability of robots: By using the Bellman equation, robots can dynamically adjust control strategies based on changes in the current state and environment, improving the robustness and adaptability of robots in complex environments.

[0057] Step S103, iteratively updating the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine an optimal control strategy, and adopt the optimal control strategy to control the movement of the robot.

[0058] In the above steps, the strategy value function, state value function and action value function are generated by adopting Markov decision process technology, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, so as to take complex working conditions into consideration, and iteratively update the above Bellman equation until the above Bellman equation converges or until the current number of iterations is equal to the preset number of iterations to determine the optimal control strategy, and adopt the above optimal control strategy to control the movement of the robot, so that compared with the existing scheme, the present application can effectively control the robot under complex working conditions, thereby solving the problem that the control of the robot under the existing scheme cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0059] The Q-learning algorithm can be used in the process of iteratively updating the above Bellman equation. The Q-learning algorithm is a reinforcement learning algorithm based on value iteration, which is used to solve problems in Markov decision processes (MDPs). In the Q-learning algorithm, the agent selects actions by learning a Q-value function, which represents the estimated value of the long-term reward obtained by performing a specific action in a specific state. The agent continuously updates the Q-value function by interacting with the environment until it converges to the optimal policy. This algorithm is a model-free method, so it does not need to model the environment. It is suitable for exploring unknown environments and solving problems with large state spaces.

[0060] Optimal strategy and optimal value function:

[0061] Optimal state value function V * (s): represents the maximum cumulative reward that can be achieved starting from state s:

[0062]

[0063] Optimal action-value function Q * (s,a): represents the maximum cumulative reward that can be achieved by taking action a from state s:

[0064]

[0065] Optimal strategy π * is the optimal strategy corresponding to the optimal state value function or the optimal action value function:

[0066]

[0067] Iteration process:

[0068] Value Iteration:

[0069] Initialize V(s) to an arbitrary value.

[0070] Repeat the update until convergence:

[0071]

[0072] Policy Iteration:

[0073] Initialize the policy π to be an arbitrary policy.

[0074] Repeat the following steps until convergence:

[0075] Policy evaluation: Calculate Vπ(s).

[0076] Policy improvement: Update policy π:

[0077]

[0078] Q-Learning:

[0079] Initialize Q(s,a) to any value.

[0080] Repeat the update until convergence:

[0081] Q π (s,a)=Q(s,a)+a(γ+γ(max Q(s',a'))-Q(s,a));

[0082] In one embodiment of the present application, the above-mentioned optimal control strategy is used to control the movement of the robot, including:

[0083] When it is determined that the robot needs to avoid obstacles, a corresponding target action is determined according to the optimal control strategy;

[0084] Control the robot to perform the target action to successfully avoid obstacles.

[0085] Specifically, applying the optimal control strategy to the robot obstacle avoidance scenario can cope with more complex working conditions than existing solutions.

[0086] In one embodiment of the present application, in the process of controlling the movement of the robot using the above optimal control strategy, the above method further includes:

[0087] A heuristic search algorithm is used to calculate the cost value for each motion path corresponding to the above optimal control strategy;

[0088] The motion path with the lowest cost value is used as the intermediate target point;

[0089] The above heuristic search algorithm is used to generate an optimal motion path based on the above robot to the above intermediate target point, and from the above intermediate target point to the final target point.

[0090] Specifically, the A* algorithm (i.e., heuristic search algorithm) is a heuristic search algorithm used to solve path planning problems. It is based on the Dijkstra algorithm and the idea of ​​heuristic search. It selects the next node to explore by estimating the cost from the starting point to the target node, so as to find the optimal solution as quickly as possible during the search process. The core idea of ​​the A* algorithm is to maintain two lists: an open list and a closed list. At the beginning, the starting point is added to the open list and its cost is set to 0. Then, the following steps are repeated until the target node is found or the open list is empty:

[0091] Select the node with the smallest cost from the open list as the current node;

[0092] Remove the current node from the open list and add it to the closed list;

[0093] Traverse the neighbor nodes of the current node, calculate their costs, and update their parent nodes to the current node;

[0094] Add neighbor nodes to the open list.

[0095] In each iteration, the A* algorithm selects the next node to explore based on the estimated cost from the current node to the target node (usually the Euclidean distance or Manhattan distance) and the actual cost of the current node (the cost from the starting point to the current node). This can reduce the time and space complexity of the search while ensuring that the optimal solution is found.

[0096] In one embodiment of the present application, before using the Markov decision process technology to generate the strategy value function, the state value function and the action value function, the method further includes:

[0097] A variety of acquisition devices are used to acquire robot data, including odometer, IMU, collision sensor, ground sensor, dirt detection sensor, temperature sensor, humidity sensor and audio sensor.

[0098] Odometer data: the distance and direction of the robot's movement; IMU data: acceleration, angular velocity and gyroscope data; collision sensor: detects whether a collision has occurred; ground sensor: detects the ground material; other sensor data: dirt detection sensor, temperature and humidity sensor, audio sensor;

[0099] Action Space: Drive motor instructions: forward, backward, turn left, turn right; auxiliary actions: stop, restart;

[0100] Data preprocessing: Normalize sensor data to ensure input consistency and stability;

[0101] Anomaly detection: Add anomaly detection mechanism to timely detect and handle anomalies in sensor data;

[0102] Multi-sensor fusion: Combining multiple sensor data to improve the accuracy and robustness of state detection;

[0103] Reward Function:

[0104] In one embodiment of the present application, in the process of iteratively updating the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations to determine the optimal control strategy, the method further includes:

[0105] In a case where it is determined that the robot moves normally, determining a first positive reward value;

[0106] When it is determined that the robot successfully avoids the obstacle, determining a second positive reward value, wherein the second positive reward value is greater than the first positive reward value;

[0107] In a case where it is determined that the robot is slipping or colliding, determining a first negative reward value;

[0108] In the case where it is determined that the robot is moving, a second negative reward value is determined, and the second negative reward value is less than the first negative reward value;

[0109] When it is determined that the robot overturns, a third negative reward value is determined, and the third negative reward value is greater than the first negative reward value.

[0110] Positive Rewards:

[0111] When the robot moves normally, the first positive reward value is given;

[0112] When the robot successfully avoids an obstacle, a second positive reward value is given;

[0113] Negative rewards:

[0114] When the robot slips, the first negative reward value is given;

[0115] When the robot moves, a second negative reward value is given;

[0116] When the robot capsizes, a third negative reward value is given;

[0117] When the robot collides, the first negative reward value is given.

[0118] By using reinforcement learning methods and integrating data from multiple sensors such as odometers, IMUs, and collision sensors, various abnormal states of the sweeping robot can be effectively judged.

[0119] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the robot control method of the present application will be described in detail below in conjunction with specific embodiments.

[0120] This embodiment relates to a reinforcement learning algorithm flow in a specific robot control method, such as Figure 2 As shown, including:

[0121] Initialize the neural network;

[0122] Initialize the state and action space:

[0123] According to the current state, the action is selected as one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning and collision, and the values ​​are +1, +10, -10, -5, -20, -10 respectively;

[0124] Update the Q value and strategy, and determine whether the current number of training steps has reached the preset number of training steps;

[0125] When the current number of training steps reaches the preset number of training steps, determining that the training is finished;

[0126] When the current number of training steps does not reach the preset number of training steps, the action is reselected according to the current state as one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning and collision, and the values ​​are +1, +10, -10, -5, -20, -10 respectively; the Q value and strategy are updated until the current number of training steps reaches the preset number of training steps.

[0127] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0128] The embodiment of the present application also provides a control device for a robot. It should be noted that the control device for the robot in the embodiment of the present application can be used to execute the control method for the robot provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0129] The following is an introduction to the control device of the robot provided in the embodiment of the present application.

[0130] Figure 3 1 is a block diagram of a robot control device according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0131] The first processing unit 31 is used to generate a strategy value function, a state value function and an action value function by using the Markov decision process technology, wherein the strategy value function represents the probability of selecting an action under a given state, the state value function represents the cumulative reward expected to be obtained starting from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained starting from the state by taking the action under the strategy, wherein the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over;

[0132] A second processing unit 32 is used to generate a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function;

[0133] The third processing unit 33 is used to iteratively update the above Bellman equation until the above Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine the best control strategy, and use the above best control strategy to control the movement of the robot.

[0134] In the above steps, the strategy value function, state value function and action value function are generated by adopting Markov decision process technology, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, so as to take complex working conditions into consideration, and iteratively update the above Bellman equation until the above Bellman equation converges or until the current number of iterations is equal to the preset number of iterations to determine the optimal control strategy, and adopt the above optimal control strategy to control the movement of the robot, so that compared with the existing scheme, the present application can effectively control the robot under complex working conditions, thereby solving the problem that the control of the robot under the existing scheme cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0135] In one embodiment of the present application, the first processing unit includes a first building module,

[0136] The first building block is used to construct the above strategy value function:

[0137] π(a|s)=P(\text{select action}a\text{in state}s);

[0138] Among them, π(a|s) is the probability that the above robot chooses action a in a given state s, and P is the probability.

[0139] In one embodiment of the present application, the third processing unit includes a first processing module and a second processing module.

[0140] The first processing module is used to determine the corresponding target action according to the above-mentioned optimal control strategy when it is determined that the above-mentioned robot needs to avoid obstacles;

[0141] The second processing module is used to control the robot to perform the target action to successfully avoid obstacles.

[0142] In one embodiment of the present application, the third processing unit includes a third processing module, a fourth processing module and a fifth processing module.

[0143] The third processing module is used to use a heuristic search algorithm to calculate the cost value for each motion path corresponding to the above-mentioned optimal control strategy in the process of controlling the movement of the robot by using the above-mentioned optimal control strategy;

[0144] The fourth processing module is used for adopting the motion path with the lowest cost value as the intermediate target point;

[0145] The fifth processing module is used to adopt the above-mentioned heuristic search algorithm to generate an optimal motion path based on the above-mentioned robot to the above-mentioned intermediate target point, and from the above-mentioned intermediate target point to the final target point.

[0146] In one embodiment of the present application, the second processing unit includes a second building block and a third building block.

[0147] The second building block is used to construct the Bellman equation of the above state value function:

[0148] V π (s)=∑ a π(as)∑ s',γ p(s',γs,a)[γ+γV π (s')];

[0149] The third building block is used to construct the Bellman equation of the above action-value function:

[0150] Qπ(s,a)=∑ s',γ p(s',γ|s,a)[γ+γΣ a' π(a'|s')Q π (s',a')];

[0151] Among them, V π (s) is the state value of the robot in state s, π(a|s) is the probability that the robot selects action a in a given state s, p(s',γs,a) is the probability that the robot transfers to state s' after executing action a in state s under the influence of discount factor γ, V π (s') is the state value of the robot in state s', Q π (s,a) is the action value of the robot starting from the state s by taking the action a, Q π (s',a') is the action value of the robot taking the action a' starting from the state s', and π(a'|s') is the probability of the robot choosing action a' in a given state s'.

[0152] In one embodiment of the present application, the third processing unit includes a first determining module, a second determining module, a third determining module, a fourth determining module and a fifth determining module.

[0153] The first determination module is used to determine a first positive reward value when it is determined that the robot moves normally during a process of iteratively updating the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations to determine the optimal control strategy;

[0154] The second determination module is used to determine a second positive reward value when it is determined that the robot successfully avoids the obstacle, and the second positive reward value is greater than the first positive reward value;

[0155] The third determination module is used to determine a first negative reward value when it is determined that the robot is slipping or colliding;

[0156] The fourth determination module is used to determine a second negative reward value when it is determined that the robot is moving, and the second negative reward value is less than the first negative reward value;

[0157] The fifth determination module is used to determine a third negative reward value when it is determined that the robot is overturned, and the third negative reward value is greater than the first negative reward value.

[0158] In one embodiment of the present application, the above-mentioned device also includes a fourth processing unit, which is used to obtain robot data using multiple collection devices before using Markov decision process technology to generate strategy value function, state value function and action value function, and the above-mentioned collection devices include odometer, IMU, collision sensor, ground sensor, dirt detection sensor, temperature sensor, humidity sensor and audio sensor.

[0159] The control device of the robot includes a processor and a memory. The first processing unit, the second processing unit and the third processing unit are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0160] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing solution cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0162] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the control method of the robot.

[0163] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the control method of the robot is executed when the program is run.

[0164] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and can be run on the processor, and the processor implements at least the following steps when executing the program: using Markov decision process technology to generate a strategy value function, a state value function and an action value function, the above strategy value function represents the probability of selecting an action in a given state, the above state value function represents the cumulative reward expected to be obtained under the strategy, starting from the above state, the above action value function represents the cumulative reward expected to be obtained starting from the above state taking the above action under the above strategy, the above action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, the above overturning represents that the above robot flips; according to the above strategy value function, the above state value function and the above action value function, generate a Bellman equation, the above Bellman equation represents the relationship between the above state value function and the above action value function; iteratively update the above Bellman equation until the above Bellman equation converges or until the current number of iterations is equal to the preset number of iterations, so as to determine the optimal control strategy, and use the above optimal control strategy to control the movement of the robot. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0165] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: using Markov decision process technology to generate a strategy value function, a state value function and an action value function, the above strategy value function represents the probability of selecting an action in a given state, the above state value function represents the cumulative reward expected to be obtained under the strategy, starting from the above state, the above action value function represents the cumulative reward expected to be obtained under the above strategy, starting from the above state taking the above action, the above action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, the above overturning represents that the above robot flips over; generating a Bellman equation based on the above strategy value function, the above state value function and the above action value function, the above Bellman equation represents the relationship between the above state value function and the above action value function; iteratively updating the above Bellman equation until the above Bellman equation converges or until the current number of iterations is equal to the preset number of iterations, so as to determine the optimal control strategy, and use the above optimal control strategy to control the movement of the robot.

[0166] The present application also provides a control system for a robot, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include means for executing any one of the methods described above.

[0167] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0174] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0177] 1) The robot control method of the present application generates a strategy value function, a state value function and an action value function by adopting Markov decision process technology, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, so as to take complex working conditions into consideration, and iteratively update the above-mentioned Bellman equation until the above-mentioned Bellman equation converges or until the current number of iterations is equal to the preset number of iterations to determine the optimal control strategy, and adopt the above-mentioned optimal control strategy to control the movement of the robot, so that the present application can effectively control the robot under complex working conditions compared with the existing scheme, thereby solving the problem that the control of the robot under the existing scheme cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0178] 2) The control device of the robot of the present application generates a strategy value function, a state value function and an action value function by adopting the Markov decision process technology, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, so as to take complex working conditions into consideration, and iteratively update the above-mentioned Bellman equation until the above-mentioned Bellman equation converges or until the current number of iterations is equal to the preset number of iterations to determine the optimal control strategy, and adopt the above-mentioned optimal control strategy to control the movement of the robot, so that compared with the existing scheme, the present application can effectively control the robot under complex working conditions, thereby solving the problem that the control of the robot under the existing scheme cannot be applied to more complex working conditions, resulting in poor control effect of the robot under complex working conditions.

[0179] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method, characterized in that: include: The Markov decision process technology is used to generate a strategy value function, a state value function and an action value function, wherein the strategy value function represents the probability of selecting an action in a given state, the state value function represents the cumulative reward expected to be obtained from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained from taking the action in the state under the strategy, and the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over; Generate a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function; The Bellman equation is iteratively updated until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine an optimal control strategy, and the optimal control strategy is used to control the movement of the robot.

2. The method according to claim 1, characterized in that The Markov decision process technique is used to generate the policy value function, including: The strategy value function is constructed as: π(as)=P(\text{select action}a\text{in state}s); Among them, π(a | s) is the probability that the robot selects action a in a given state s, and P is the probability.

3. The method according to claim 1, characterized in that: The optimal control strategy is used to control the movement of the robot, including: When it is determined that the robot needs to avoid obstacles, determining a corresponding target action according to the optimal control strategy; The robot is controlled to perform the target action to successfully avoid obstacles.

4. The method according to claim 1, characterized in that In the process of controlling the movement of the robot by adopting the optimal control strategy, the method further comprises: A heuristic search algorithm is used to calculate the cost value for each motion path corresponding to the optimal control strategy; adopting the motion path with the lowest cost value as the intermediate target point; The heuristic search algorithm is used to generate an optimal motion path based on the robot to the intermediate target point and from the intermediate target point to the final target point.

5. The method according to claim 1, characterized in that According to the strategy value function, the state value function and the action value function, a Bellman equation is generated, including: The Bellman equation for constructing the state value function is: V π (s)= ∑ a p(as) ∑ s',γ p(s',γs,a)[γ+γV π (s')]; The Bellman equation for constructing the action-value function is: Qπ(s,a)=∑s',γp(s',γs,a)[γ+γ∑a'π(a's')Qπ(s',a')]; Among them, Vπ ( s ) is the state value of the robot in state s, π ( a|s ) is the probability that the robot selects action a in a given state s, p(s',γs,a) is the probability that the robot transfers to state s' after executing action a in state s under the influence of the discount factor γ, V π (s') is the robot in state s 'The state value, Q π (s,a) is the action value of the robot starting from the state s by taking the action a, Q π (s',a') is the action value of the robot starting from the state s' by taking the action a', and π(a's') is the probability of the robot choosing action a' in a given state s'.

6. The method according to claim 1, characterized in that In the process of iteratively updating the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations to determine the optimal control strategy, the method further includes: In a case where it is determined that the robot is moving normally, determining a first positive reward value; In a case where it is determined that the robot successfully avoids the obstacle, determining a second positive reward value, wherein the second positive reward value is greater than the first positive reward value; In a case where it is determined that the robot is slipping or colliding, determining a first negative reward value; In the case where it is determined that the robot is to move, a second negative reward value is determined, wherein the second negative reward value is less than the first negative reward value; In the case where it is determined that the robot is tipping over, a third negative reward value is determined, the third negative reward value being greater than the first negative reward value.

7. The method according to any one of claims 1 to 6, characterized in that Before using the Markov decision process technology to generate the policy value function, the state value function and the action value function, the method further includes: A variety of acquisition devices are used to acquire robot data, including an odometer, an IMU, a collision sensor, a ground sensor, a dirt detection sensor, a temperature sensor, a humidity sensor, and an audio sensor.

8. A robot control device, characterized in that: include: A first processing unit is used to generate a strategy value function, a state value function and an action value function by using Markov decision process technology, wherein the strategy value function represents the probability of selecting an action in a given state, the state value function represents the cumulative reward expected to be obtained from the state under the strategy, and the action value function represents the cumulative reward expected to be obtained from taking the action in the state under the strategy, wherein the action is one of the following: normal movement, successful obstacle avoidance, slipping, moving, overturning, collision, and the overturning represents that the robot flips over; A second processing unit, configured to generate a Bellman equation according to the strategy value function, the state value function and the action value function, wherein the Bellman equation represents the relationship between the state value function and the action value function; The third processing unit is used to iteratively update the Bellman equation until the Bellman equation converges or until the current number of iterations is equal to a preset number of iterations, so as to determine an optimal control strategy, and adopt the optimal control strategy to control the movement of the robot.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A robot control system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images