Dynamic Programming Monitoring Method Based on Markov Model

Abstract

The present invention discloses a dynamic planning monitoring method based on a Markov model, which specifically relates to the technical field of robot path planning, including: modeling the robot's operating environment, obtaining the position, shape and speed information of obstacles through an onboard image acquisition device; discretizing the state space of the robot when performing a task, dividing it into a finite number of state points, and matching control instructions for each state point; based on real-time environment modeling, obtaining a forward-looking state point set, obtaining a forward-looking state point set and corresponding control instructions; if the obstacle position changes or a new obstacle appears, outputting the final updated control instruction based on obstacle update information; outputting the final updated time series control instruction to the robot execution system to realize path planning and obstacle avoidance; and effectively solving the problems of insufficient intelligence level and low operating efficiency of robots in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and more specifically, to a dynamic planning monitoring method based on a Markov model. Background Art

[0002] With the rapid development of robotics technology, robots are increasingly used in warehousing, logistics, manufacturing and other fields.

[0003] Complex operating environments and diverse task requirements place higher demands on the robot's path planning and obstacle avoidance capabilities. Currently, robots mainly use local path planning methods based on sensor information fusion and global path planning methods based on global maps in path planning and obstacle avoidance. Although they perform well in many scenarios, they still have problems with insufficient intelligence and low operating efficiency when facing complex and changing environments. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a dynamic programming monitoring method based on the Markov model. By introducing the Markov model and the dynamic programming algorithm, an efficient and stable path planning and obstacle avoidance strategy is provided for the robot to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a dynamic programming monitoring method based on a Markov model, comprising the following steps:

[0006] Step S001, robot state modeling: the state of the robot at time t+1 is determined by the state at time t and the input control command, and is also affected by the range of motion and motion noise;

[0007] Step S002, real-time environment modeling: modeling the robot's working environment, obtaining the position, shape and speed information of obstacles through the onboard image acquisition device; constructing a state space based on the acquired information, the state space includes the robot's position, direction, target position, obstacle position, door and key status;

[0008] Step S003, task discretization: discretize the state space of the robot in executing the task, divide it into a finite number of state points, and match control instructions for each state point;

[0009] Step S004, obtaining control instructions: based on real-time environment modeling, obtaining a forward-looking state point set, obtaining a forward-looking state point set and corresponding control instructions;

[0010] Step S005, real-time monitoring feedback: during the execution process, the obstacle information is monitored in real time to determine whether the obstacle information has changed; if the obstacle position has changed or a new obstacle has appeared, the final updated control instruction is output based on the obstacle update information; otherwise, the process directly goes to step S006;

[0011] Step S006, execution of control instructions: output the final updated time series control instructions to the robot execution system to achieve path planning and obstacle avoidance; repeat steps S002-S006 until the robot completes the task.

[0012] Preferably, the robot state modeling process includes a real-time position update mechanism,

[0013] Establish a rectangular coordinate system on the two-dimensional plane, and use the coordinates and orientation of the robot as the state. represents the position of the robot at time t, represents the direction of the robot at time t, and the state of the robot at time t is recorded as , It indicates the movable range of the robot, that is, the free movable position after removing obstacles;

[0014] The state of the robot at the next moment Satisfies the following formula:

[0015]

[0016]

[0017] in, Indicates the control instructions input to the robot. Indicates the running line speed, represents the running angular velocity, Indicates action instructions;

[0018] Represents the position change vector and direction change vector after the robot input command is executed;

[0019] Represents motion noise, which is used to describe the external interference to the robot during its motion. It is the fluctuation range of position coordinates and direction. The value of motion noise is the average value under historical samples.

[0020] Preferably, the control instructions include the robot moving forward, rotating, picking up a key, and unlocking a door.

[0021] Preferably, the control instructions are obtained in the following manner: based on iterative learning, the robot uses the updated Q value table to plan the optimal path and avoid obstacles according to the current state and environmental information, and outputs the final updated time series control instructions, which specifically includes the following steps:

[0022] Step S41: according to the result of real-time environment modeling, set the initial state space and control instructions, initialize the Q value table and action strategy;

[0023] Step S42, output the converged state Q value table: in the discretized state space and control instructions, use the Q value iteration algorithm to iteratively update the Q value table, and update the robot's Q value table through continuous iterative learning. In each iteration, the robot needs to execute the action under the current strategy and observe the environmental feedback. After multiple iterations, the Q value table will gradually converge to a stable state, and the converged state Q value table is output;

[0024] Step S43: extract the optimal action selection in each state from the converged state Q value table, and output a series of time series control instructions to guide the robot's actions. In each iteration, after the robot executes the current strategy action, it obtains environmental feedback through sensors, updates the reward value and adjusts the action strategy, and finally obtains the optimized Q value table.

[0025] Preferably, in step S42, the iterative learning process is as follows:

[0026] For the robot’s current state s, consider all possible actions;

[0027] After executing action a, observe the environmental feedback, which includes the reward value and the next state s';

[0028] The Q value iteration algorithm satisfies the following formula:

[0029]

[0030] in, represents the Q value corresponding to executing action a in state s, Represents the reward value obtained by executing action a in state s and transferring to state s'; represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a;

[0031] γ is a discount factor used to represent the present value of future rewards;

[0032] max_a'Q(s', a') represents the maximum Q value obtained by selecting action a' that maximizes the Q value in state s'.

[0033] Preferably, the reward value The way to obtain is:

[0034] Analyze the characteristic parameters obtained when state s executes action a and transfers to state s';

[0035] The characteristic parameters include: the remaining path to the destination, the time consumption, the number of collisions with obstacles, and the number of visits to new areas, which are recorded as: L c , T s , P c , t n ;

[0036] Input the collected feature parameters into the reward value calculation function, and calculate the corresponding reward value using the following formula;

[0037]

[0038] in, Indicates the maximum time limit, represents the shortest path length, k1 and k2 are weight coefficients for adjusting the path length and time consumption respectively; represents the exploration reward, which in the embodiment is a piecewise function set based on the number of visits to the new area, Indicates the operation penalty value, which is set according to the situation where the robot violates the task rules and fails to achieve the specified goal;

[0039] Furthermore, the exploration reward is a piecewise function based on the number of visits to the new area n, expressed as g(tn); the operation penalty value is set according to whether the robot violates the task rules or fails to achieve the specified goal. =-k (k is a constant), otherwise =0.

[0040] Preferably, the execution step of the control instruction includes a turning optimization control operation. When the control instruction is identified as a rotation, a smoothing term is introduced to optimize the turning, and the angular velocity change rate of the robot is adjusted within a preset range to avoid sharp turns.

[0041] Preferably, the real-time monitoring and feedback step includes a robot operation monitoring step, including:

[0042] Step S51, collecting robot status information and environment status information, using sensors to collect the robot's current status information in real time, and outputting the robot's environmental perception information, including perception speed and perception accuracy; calculating the environmental perception quality index based on the perception speed and perception accuracy; if the environmental perception quality index meets the requirements, proceed to the next step, otherwise, issue an early warning;

[0043] Step S52, record each action performed by the robot and the corresponding timestamp, output the response speed deviation and the response action deviation; calculate the robot state controllability index based on the robot's response speed deviation and the response action deviation; if the robot state controllability index meets the requirements, proceed to the next step, otherwise, issue an early warning;

[0044] Step S53, monitoring the running status and output results of the path planning algorithm, obtaining convergence accuracy, smoothness, and obstacle avoidance capability indicators, measuring the path planning quality based on the convergence accuracy, smoothness, and obstacle avoidance capability indicators, and outputting the path planning quality index; if the path planning quality index meets the requirements, proceed to the next step, otherwise, an early warning is issued;

[0045] Step S54: Combine the environment perception quality index HG, the robot state controllability index JZ and the path planning quality index LK to obtain the following formula: The dynamic programming quality evaluation coefficient Gz is calculated; where tha, thb, and thc represent the corresponding thresholds of each item respectively;

[0046] Step S55: taking measures based on the changing trend of the dynamic programming quality evaluation coefficient Gz.

[0047] Preferably, the environmental perception quality index HG is obtained in the following manner:

[0048] The perception speed and perception accuracy are input into the following formula to calculate the environmental perception quality index HG.

[0049]

[0050] The perception speed V_per is the reciprocal of the time required for the robot to obtain obstacle position information from the environment; the perception accuracy A_per is quantified by comparing the difference between the environmental information perceived by the robot and the actual environmental information;

[0051]

[0052] The mean square error refers to the mean square error between the perceived obstacle position and the actual obstacle position.

[0053] Preferably, the robot state controllability index JZ is obtained in the following manner:

[0054] Input the response speed deviation Pt and the response action deviation Pd into the following formula to output the robot state controllability index JZ;

[0055]

[0056] The response speed deviation Pt is obtained as follows:

[0057] Get the average response time t_avg from when the robot receives the control command to when it executes the control command, and use the ratio of the average response time to the maximum response time t_max to represent the response speed deviation Pt, using the following formula:

[0058]

[0059] The response speed deviation Pt is calculated, and max (·) means taking the maximum value of the value in brackets;

[0060] The response action deviation Pd is obtained as follows:

[0061] Get the target position range (x tar ±x tar* Δσ_max,y tar ±y tar* Δσ_max) and the actual position is (x act ,y act ), Δσ_max represents the maximum fluctuation amplitude of motion noise, which is expressed by the following formula:

[0062]

[0063] The response action deviation Pd is calculated.

[0064] Preferably, the path planning quality index is obtained in the following manner:

[0065] The obstacle avoidance success frequency is expressed as the ratio of the number of successful avoidance attempts by the robot to the total number of obstacle avoidance attempts made by the robot;

[0066] Collect the angular velocity data of the robot at each time point during the movement process. There are n time points, i represents the sequence number of the time points, and the angular velocity of the i-th time point is recorded as represents the average angular velocity;

[0067] The path planning quality index LK is calculated by the following formula

[0068]

[0069]

[0070] in, represents the smoothness coefficient, represents the successful obstacle avoidance frequency, and Ca represents the convergence accuracy.

[0071] Technical effects and advantages of the present invention:

[0072] (1) The present invention provides a dynamic programming monitoring method based on the Markov model. By introducing the Markov model and the dynamic programming algorithm, efficient path planning and obstacle avoidance in complex environments are achieved. The method has higher adaptability and robustness, provides strong support for the industrial application of technology, and effectively solves the problems of insufficient intelligence level and low operation efficiency of robots in the prior art. In the field of robot navigation, it can enable robots to achieve autonomous navigation in unknown or dynamic environments and complete complex tasks. In addition, the method can also be applied to game AI, drone control and other fields to enhance the autonomous decision-making ability of intelligent bodies.

[0073] (2) The present invention provides a dynamic planning monitoring method based on a Markov model, including environmental perception quality monitoring, robot state controllability monitoring, path planning quality monitoring and dynamic planning quality evaluation, which helps to timely discover and deal with potential problems, thereby improving the stability and reliability of the robot; a multi-level early warning mechanism is set up, which can trigger early warnings in time according to the real-time data of the environmental perception quality index HG, the robot state controllability index JZ and the path planning quality index LK, and provide valuable reference information for operators to make correct decisions quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flow chart of the dynamic planning monitoring method of the present invention.

[0075] Figure 2 This is a flowchart of the robot operation status analysis and early warning of the present invention. DETAILED DESCRIPTION

[0076] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0078] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0079] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0080] Example 1, see Figure 1The present invention provides a flow chart of a dynamic planning monitoring method. Figure 1 The dynamic programming monitoring method based on the Markov model shown includes the following steps:

[0081] Step S001, robot state modeling: the state of the robot at time t+1 is determined by the state at time t and the input control command, and is also affected by the range of motion and motion noise;

[0082] Step S002, real-time environment modeling: modeling the robot's working environment, obtaining the position, shape and speed information of obstacles through the onboard image acquisition device, and constructing a state space based on the acquired information. The state space includes the robot's position, direction, target position, obstacle position, and the state of doors and keys;

[0083] Explanation: The image acquisition device is one or more of a laser radar, a camera sensor, an inertial navigation system, and an ultrasonic sensor;

[0084] If the image acquisition device is not unique, Kalman filtering and particle filtering algorithms are used to fuse data from different sources to improve the accuracy and robustness of the environmental model;

[0085] Based on the collected and fused data, a three-dimensional model or two-dimensional grid map of the robot's operating environment is constructed to provide a basis for subsequent path planning and obstacle avoidance;

[0086] Step S003, task discretization: discretize the state space of the robot in executing the task, divide it into a finite number of state points, and match control instructions for each state point;

[0087] Explanation: Finite state points refer to dividing the robot's continuous state space into a finite number of discrete, distinguishable state sets. Each state point represents a specific state that the robot may be in during the execution of a task. These states are obtained by discretizing the continuous state space.

[0088] Explanation: The state space includes the position, speed, and posture parameters of the robot. The parameters are divided into a series of discrete points, simplifying the originally complex continuous state space into a finite set of discrete states. One or more possible control instructions are specified for each discrete state point. The control instructions guide the robot to transfer from the current state to the next state.

[0089] Furthermore, the degree of discretization is adjusted according to the robot's accuracy requirements and computing power, including:

[0090] Determine the granularity of state space discretization based on task requirements, including path planning accuracy and obstacle avoidance sensitivity. If the task requires high-precision path planning, the state space needs to be divided into finer discrete points.

[0091] Analyze the complexity of the path planning algorithm and control algorithm to ensure that the robot can complete calculations and decisions within an acceptable time in the discretized state space;

[0092] If the computing power is strong and the accuracy requirement is high, increase the number of discrete points; otherwise, reduce the number of discrete points; verify the performance and effect of different discretization levels through experiments or simulations to determine the optimal discretization scheme;

[0093] Step S004, obtaining control instructions: based on real-time environment modeling, obtaining a forward-looking state point set, obtaining a forward-looking state point set and corresponding control instructions;

[0094] Step S005, real-time monitoring feedback: during the execution process, the obstacle information is monitored in real time to determine whether the obstacle information has changed; if the obstacle position changes or a new obstacle appears, the final update time series is output based on the obstacle update information; otherwise, the process directly goes to step S006;

[0095] Step S006, execution of control instructions: output the final updated time series control instructions to the robot execution system to achieve efficient path planning and obstacle avoidance; repeat steps S002-S006 until the robot completes the task.

[0096] It needs to be further explained in the embodiment of the present invention that the robot state modeling process includes a real-time position update mechanism;

[0097] Establish a rectangular coordinate system on the two-dimensional plane, and use the coordinates and orientation of the robot as the state. represents the position of the robot at time t, represents the direction of the robot at time t, and the state of the robot at time t is recorded as , It indicates the movable range of the robot, that is, the free movable position after removing obstacles;

[0098] The state of the robot at the next moment Satisfies the following formula:

[0099]

[0100]

[0101] in, Indicates the control instructions input to the robot. Indicates the running line speed, represents the running angular velocity, Indicates action instructions;

[0102] Represents the position change vector and direction change vector after the robot input command is executed;

[0103] Represents motion noise, which is used to describe the external interference to the robot during its motion. It is the fluctuation range of position coordinates and direction. The value of motion noise is the average value under historical samples.

[0104] What needs to be further explained in the embodiment of the present invention is that the control instructions include the robot executing forward movement, rotation, picking up a key, and unlocking a door; among which, picking up a key and unlocking a door are action instructions.

[0105] It is necessary to further explain in the embodiment of the present invention that the control instruction is obtained in the following manner: on the basis of iterative learning, the robot uses the updated Q value table to plan the optimal path and avoid obstacles according to the current state and environmental information, and outputs the final updated time series control instruction, which specifically includes the following steps:

[0106] Step S41: according to the result of real-time environment modeling, set the initial state space and control instructions, initialize the Q value table and action strategy;

[0107] Explanation: The Q value table is used to record the expected rewards that can be obtained by selecting different actions under different states; initializing the robot's action strategy refers to using a genetic algorithm or an ant colony algorithm to generate a preliminary path planning and control sequence, the purpose of which is to find a suboptimal but feasible path in the simplified deterministic problem;

[0108] Step S42, output the converged state Q value table: in the discretized state space and control instructions, use the Q value iteration algorithm to iteratively update the Q value table; through continuous iterative learning, update the robot's Q value table. In each iteration, the robot needs to execute the action under the current strategy and observe the environmental feedback. After multiple iterations, the Q value table will gradually converge to a stable state, and output the converged state Q value table;

[0109] Explanation: Using PyTorch to implement Q-value iteration, introducing accelerated convergence strategies such as momentum terms and learning rate adjustment into the Q-value iteration algorithm to improve the learning speed and convergence of the algorithm;

[0110] Step S43: extract the optimal action selection in each state from the converged state Q value table, and output a series of time series control instructions to guide the robot's actions. In each iteration, after the robot executes the current strategy action, it obtains environmental feedback through sensors, updates the reward value and adjusts the action strategy, and finally obtains the optimized Q value table.

[0111] It needs to be further explained in the embodiment of the present invention that in step S42, the iterative learning process is as follows:

[0112] For the robot’s current state s, consider all possible actions;

[0113] After executing action a, observe the environmental feedback, which includes the reward value and the next state s';

[0114] The Q value iteration algorithm satisfies the following formula:

[0115]

[0116] in, represents the Q value corresponding to executing action a in state s, Represents the reward value obtained by executing action a in state s and transferring to state s'; represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a;

[0117] γ is a discount factor used to represent the present value of future rewards;

[0118] max_a'Q(s', a') represents the maximum Q value obtained by selecting action a' that maximizes the Q value in state s'.

[0119] It is necessary to further explain in the embodiment of the present invention that the reward value The way to obtain is:

[0120] Analyze the characteristic parameters obtained when state s executes action a and transfers to state s';

[0121] The characteristic parameters include: the remaining path to the destination, the time consumption, the number of collisions with obstacles, and the number of visits to new areas, which are recorded as: L c , T s , P c , t n ;

[0122] Input the collected feature parameters into the reward value calculation function, and calculate the corresponding reward value using the following formula;

[0123]

[0124] in, Indicates the maximum time limit, represents the shortest path length, k1 and k2 are weight coefficients for adjusting the path length and time consumption respectively; represents the exploration reward, which in the embodiment is a piecewise function set based on the number of visits to the new area, Indicates the operation penalty value, which is set according to the situation where the robot violates the task rules and fails to achieve the specified goal;

[0125] Furthermore, the exploration reward is a piecewise function based on the number of visits to the new area n, expressed as g(tn); the operation penalty value is set according to whether the robot violates the task rules or fails to achieve the specified goal. =-k (k is a constant), otherwise =0.

[0126] It needs to be further explained in the embodiment of the present invention that the real-time monitoring feedback step includes the following contents:

[0127] Monitor robot status: Continuously monitor the robot's status (such as position, speed, and posture) during the entire execution process;

[0128] Receive feedback information: Receive feedback information from the environment (such as changes in obstacle positions, new obstacles);

[0129] Dynamically adjust strategies: Based on monitoring and feedback information, dynamically adjust control strategies to ensure that the robot can successfully complete the task.

[0130] It needs to be further explained in the embodiment of the present invention that the execution step of the control instruction includes a turning optimization control operation. When the control instruction is identified as a rotation, a smoothing term is introduced to optimize the turning, and the angular velocity change rate of the robot is adjusted within a preset range to avoid sharp turns.

[0131] By penalizing rapid changes in angular velocity, the robot is encouraged to complete turns in a smoother manner.

[0132] For ease of understanding, the embodiments of the present invention are described as follows:

[0133] Suppose a robot moves in a two-dimensional grid environment. The robot has at least four possible actions: move up, move down, move left, and move right. In more complex scenarios, the control instructions may include more action options, such as moving at different speeds, rotating at different speeds, picking up keys, and unlocking doors.

[0134] State definition: define the robot's position in the grid as the current state s;

[0135] Control instructions / action space: The control instructions or action space includes at least four possible actions: up (U), down (D), left (L), right (R); in complex scenarios, there are possible actions;

[0136] When the robot is in a certain state s, “all possible” actions are moves in these four directions: U, D, L, R;

[0137] The robot uses the Q-value table to evaluate the possible rewards of performing each action in each state, and selects the optimal action based on the Q-value table;

[0138] Iterative learning process: During the iterative learning phase, the robot will try to perform these actions, observe the environmental feedback (reward value and next state), and update the Q-value table and action strategy; for example, if the robot chooses to move up (U) when in state s=(3, 4) and successfully reaches state s'=(3, 5), it will update the Q-value table related to state s and action U based on the environmental feedback (reward value and state transition probability);

[0139] Once the Q-value table converges to a stable state, the robot can extract the optimal action selection in each state from the converged state Q-value table, thereby planning the optimal path from the starting point to the end point;

[0140] In a possible embodiment, obtaining a convergence state Q value table based on deep learning includes the following steps:

[0141] Step 1: Implement the Q-value network using a deep learning framework such as TensorFlow or PyTorch.

[0142] Explain that the construction of the network that defines the Q value includes an input layer, a hidden layer, and an output layer;

[0143] Input layer: receives the robot's current state information, such as location coordinates, items held (such as keys), and areas visited;

[0144] Hidden layer: contains multiple neurons, which are used to extract features of state information. These features will be used to evaluate the value of different actions.

[0145] Output layer: Each neuron corresponds to the Q value of a possible action, which indicates the expected reward that the robot can obtain by performing the action in the current state. If the robot has N possible actions, the output layer has N neurons.

[0146] Step 2: Randomly initialize the parameters of the Q-value network: After defining the Q-value network, use a uniform distribution or a normal distribution to initialize the parameters, such as weights and biases;

[0147] Step 3: Let the robot perform actions in the environment and record information about each state, action, reward value, and next state. Initialize the data structure: create an empty data structure to store experience.

[0148] Explanation: Initialize the robot's state and select an action based on the current strategy (e.g., ε-greedy strategy), which involves a trade-off between exploration (randomly selecting an action) and exploitation (selecting the action with the highest Q value); execute the selected action and record the current state, action, reward value, and next state; if the robot reaches the terminal state, end the current storage experience; otherwise, update the robot's state to the next state and continue to select actions;

[0149] Step 4: Use the error between the target Q value and the current Q value to update the parameters of the Q value network, calculate the gradient based on the error, and update the network parameters in the opposite direction of the gradient; repeat multiple times until the Q value network converges to a stable state;

[0150] The explanation is that, during the iteration process, the proportion of exploration is gradually reduced so that the robot relies more on the learned policy to choose actions.

[0151] In summary, the technical solution provided by the embodiment of the present invention improves the quality of path planning by designing a reward function that reflects the robot's task requirements and obstacle avoidance requirements; in the iterative learning process, it considers the optimization of multiple objectives (such as path length, obstacle avoidance effect, and time efficiency) to achieve more comprehensive path planning; uses PyTorch to implement Q value iteration, which greatly improves the calculation speed and convergence of the algorithm; and by introducing a smoothing term, optimizes the turning strategy, further enhancing the stability and adaptability of the algorithm.

[0152] Example 2, see Figure 2 The robot operation status analysis and early warning flow chart of the embodiment 1 is different from that of the embodiment 1 in that the real-time monitoring and feedback step includes a robot operation status analysis and early warning step, including:

[0153] Step S51, collecting robot status information and environment status information, using sensors to collect the robot's current status information in real time, and outputting the robot's environmental perception information, including perception speed and perception accuracy; calculating the environmental perception quality index based on the perception speed and perception accuracy; if the environmental perception quality index meets the requirements, proceed to the next step, otherwise, issue an early warning;

[0154] Step S52, record each action performed by the robot and the corresponding timestamp, output the response speed deviation and the response action deviation; calculate the robot state controllability index based on the robot's response speed deviation and the response action deviation; if the robot state controllability index meets the requirements, proceed to the next step, otherwise, issue an early warning;

[0155] Step S53, monitoring the running status and output results of the path planning algorithm, obtaining convergence accuracy, smoothness, and obstacle avoidance capability indicators, measuring the path planning quality based on the convergence accuracy, smoothness, and obstacle avoidance capability indicators, and outputting the path planning quality index; if the path planning quality index meets the requirements, proceed to the next step, otherwise, an early warning is issued;

[0156] Step S54, combining the environment perception quality index HG, the robot state controllability index JZ and the path planning quality index LK, and calculating the dynamic planning quality evaluation coefficient Gz through a formula;

[0157] Step S55, taking measures based on the changing trend of the dynamic programming quality evaluation coefficient Gz;

[0158] When Gz shows a downward trend, it indicates that the robot is operating abnormally and more monitoring resources need to be allocated, such as increasing the sensor sampling frequency, strengthening data analysis and processing, etc., to more accurately understand the robot's operating status.

[0159] Explanation, through the formula The dynamic programming quality evaluation coefficient Gz is calculated, where tha, thb, and thc represent the corresponding thresholds of each item.

[0160] What needs to be further explained in the embodiment of the present invention is the method of obtaining the environment perception quality index HG:

[0161] The perception speed and perception accuracy are input into the following formula to calculate the environmental perception quality index HG.

[0162]

[0163] The perception speed V_per is the reciprocal of the time required for the robot to obtain obstacle position information from the environment; the perception accuracy A_per is quantified by comparing the difference between the environmental information perceived by the robot and the actual environmental information;

[0164]

[0165] The mean square error refers to the mean square error between the perceived obstacle position and the actual obstacle position.

[0166] It needs to be further explained in the embodiment of the present invention that the robot state controllability index JZ is obtained in the following manner:

[0167] Input the response speed deviation Pt and the response action deviation Pd into the following formula to output the robot state controllability index JZ;

[0168]

[0169] The response speed deviation Pt is obtained as follows:

[0170] Get the average response time t_avg from when the robot receives the control command to when it executes the control command, and use the ratio of the average response time to the maximum response time t_max to represent the response speed deviation Pt, using the following formula:

[0171]

[0172] The response speed deviation Pt is calculated, and max (·) means taking the maximum value of the value in brackets;

[0173] The response action deviation Pd is obtained as follows:

[0174] Get the target position range (x tar ±x tar* Δσ_max,y tar ±y tar* Δσ_max) and the actual position is (x act ,y act ), Δσ_max represents the maximum fluctuation amplitude of motion noise, which is expressed by the following formula:

[0175]

[0176] The response action deviation Pd is calculated.

[0177] It needs to be further explained in the embodiment of the present invention that the path planning quality index is obtained in the following manner:

[0178] The obstacle avoidance success frequency is expressed as the ratio of the number of successful avoidance attempts by the robot to the total number of obstacle avoidance attempts made by the robot;

[0179] Collect the angular velocity data of the robot at each time point during the movement process. There are n time points, i represents the sequence number of the time points, and the angular velocity of the i-th time point is recorded as represents the average angular velocity;

[0180] The path planning quality index LK is calculated by the following formula

[0181]

[0182]

[0183] in, represents the smoothness coefficient, represents the successful obstacle avoidance frequency, and Ca represents the convergence accuracy.

[0184] Summary: This technical solution covers several key links of robot operation monitoring, including environmental perception quality monitoring, robot state controllability monitoring, path planning quality monitoring and dynamic planning quality evaluation, which helps to timely discover and deal with potential problems, thereby improving the stability and reliability of the robot; a multi-level early warning mechanism is set up, which can trigger early warnings in time according to the real-time data of the environmental perception quality index HG, the robot state controllability index JZ and the path planning quality index LK, which not only improves the speed of problem discovery, but also can accurately judge the nature and severity of the problem according to the changing trend of specific indicators, and provide valuable reference information for operators to make correct decisions quickly.

[0185] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic programming monitoring method based on a Markov model, characterized in that: The following steps are involved: Step S001, robot state modeling: the state of the robot at time t+1 is determined by the state at time t and the input control command, and is also affected by the range of motion and motion noise; Step S002, real-time environment modeling: modeling the robot's working environment, obtaining the position, shape and speed information of obstacles through the onboard image acquisition device, and constructing a state space based on the acquired information; Step S003, task discretization: discretize the state space of the robot in executing the task, divide it into a finite number of state points, and match control instructions for each state point; Step S004, obtaining control instructions: based on real-time environment modeling, obtaining a forward-looking state point set, obtaining a forward-looking state point set and corresponding control instructions; Step S005, real-time monitoring feedback: during the execution process, the obstacle information is monitored in real time to determine whether the obstacle information has changed; if the obstacle position has changed or a new obstacle has appeared, the final updated control instruction is output based on the obstacle update information; otherwise, the process directly goes to step S006; Step S006, execution of control instructions: output the final updated time series control instructions to the robot execution system to achieve path planning and obstacle avoidance; repeat steps S002-S006 until the robot completes the task; The real-time monitoring feedback step includes the robot operation monitoring step including: Step S51, collecting robot status information and environment status information, using sensors to collect the robot's current status information in real time, and outputting the robot's environmental perception information, including perception speed and perception accuracy; calculating the environmental perception quality index based on the perception speed and perception accuracy; if the environmental perception quality index meets the requirements, proceed to the next step, otherwise, issue an early warning; Step S52, record each action performed by the robot and the corresponding timestamp, output the response speed deviation and the response action deviation; calculate the robot state controllability index based on the robot's response speed deviation and the response action deviation; if the robot state controllability index meets the requirements, proceed to the next step, otherwise, issue an early warning; Step S53, monitoring the running status and output results of the path planning algorithm, obtaining convergence accuracy, smoothness, and obstacle avoidance capability indicators, measuring the path planning quality based on the convergence accuracy, smoothness, and obstacle avoidance capability indicators, and outputting the path planning quality index; if the path planning quality index meets the requirements, proceed to the next step, otherwise, an early warning is issued; Step S54: Combine the environment perception quality index HG, the robot state controllability index JZ and the path planning quality index LK to obtain the following formula: The dynamic programming quality evaluation coefficient Gz is calculated; where tha, thb, and thc represent the corresponding thresholds of each item respectively; Step S55: taking measures based on the changing trend of the dynamic programming quality evaluation coefficient Gz.

2. The dynamic programming monitoring method based on the Markov model according to claim 1 is characterized in that: The robot state modeling process includes a real-time position update mechanism, Establish a rectangular coordinate system on the two-dimensional plane, take the coordinates and orientation of the robot as the state, and use w represents the position of the robot at time t, represents the direction of the robot at time t, and the state of the robot at time t is recorded as , Indicates the robot's movable range; The state of the robot at the next moment Satisfies the following formula: ; ;in, Indicates the control instructions input to the robot. Indicates the running line speed, represents the running angular velocity, Indicates action instructions; Represents the position change vector and direction change vector after the robot input command is executed; Represents motion noise, which is used to describe the external interference to the robot during its motion. It is the fluctuation range of position coordinates and direction. The value of motion noise is the average value under historical samples.

3. The dynamic programming monitoring method based on the Markov model according to claim 2 is characterized in that: The control instructions are obtained in the following manner: based on iterative learning, the robot uses the updated Q value table to plan the optimal path and avoid obstacles according to the current state and environmental information, and outputs the final updated time series control instructions, which specifically includes the following steps: Step S41: according to the result of real-time environment modeling, set the initial state space and control instructions, initialize the Q value table and action strategy; Step S42, output the converged state Q value table: in the discretized state space and control instructions, use the Q value iteration algorithm to iteratively update the Q value table, and update the robot's Q value table through continuous iterative learning. In each iteration, the robot needs to execute the action under the current strategy and observe the environmental feedback. After multiple iterations, the Q value table will gradually converge to a stable state, and the converged state Q value table is output; Step S43: extract the optimal action selection in each state from the converged state Q value table, and output a series of time series control instructions to guide the robot's actions. In each iteration, after the robot executes the current strategy action, it obtains environmental feedback through sensors, updates the reward value and adjusts the action strategy, and finally obtains the optimized Q value table.

4. The dynamic programming monitoring method based on the Markov model according to claim 3 is characterized in that: In step S42, the iterative learning process is as follows: For the robot’s current state s, consider all possible actions; After executing action a, observe the environmental feedback, which includes the reward value and the next state s'; The Q value iteration algorithm satisfies the following formula: ;in, represents the Q value corresponding to executing action a in state s, Represents the reward value obtained by executing action a in state s and transferring to state s'; represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a; γ is a discount factor used to represent the present value of future rewards; max_a'Q(s', a') represents the maximum Q value obtained by selecting action a' that maximizes the Q value in state s'.

5. The dynamic programming monitoring method based on the Markov model according to claim 4 is characterized in that: The reward value The way to obtain is: Analyze the characteristic parameters obtained when state s executes action a and transfers to state s'; The characteristic parameters include: the remaining path to the destination, the time consumption, the number of collisions with obstacles, and the number of visits to new areas, which are recorded as: L c , T s , P c , t n ; Input the collected feature parameters into the reward value calculation function, and calculate the corresponding reward value using the following formula; ;in, Indicates the maximum time limit, represents the maximum path length, k1 and k2 are weight coefficients for adjusting the path length and time consumption respectively; Represents exploration reward, Represents the operation penalty value, which is set based on the robot's violation of task rules and failure to achieve the specified goal.

6. The dynamic programming monitoring method based on Markov model according to claim 1 is characterized in that: The execution steps of the control instructions include turning optimization control operations. When the control instruction is identified as rotation, a smoothing term is introduced to optimize the turning and adjust the robot's angular velocity change rate within a preset range to avoid sharp turns.

7. The dynamic programming monitoring method based on Markov model according to claim 1 is characterized in that: The environmental perception quality index HG is obtained in the following way: Input the perception speed and perception accuracy into the following formula to calculate the environmental perception quality index HG: ; Wherein, the perception speed V_per is the reciprocal of the time required for the robot to obtain obstacle position information from the environment; the perception accuracy A_per is quantified by comparing the difference between the environmental information perceived by the robot and the actual environmental information; ; The mean square error refers to the mean square error between the perceived obstacle position and the actual obstacle position.

8. The dynamic programming monitoring method based on Markov model according to claim 1 is characterized in that: The robot state controllable index JZ is obtained as follows: Input the response speed deviation Pt and the response action deviation Pd into the following formula to output the robot state controllability index JZ; ; Among them, the response speed deviation Pt is obtained as follows: Get the average response time t_avg from when the robot receives the control command to when it executes the control command, and use the ratio of the average response time to the maximum response time t_max to represent the response speed deviation Pt, using the following formula: ; The response speed deviation Pt is calculated, and max (·) means taking the maximum value of the value in brackets; The response action deviation Pd is obtained as follows: Get the target position range (x tar ±x tar* Δσ_max,y tar ±y tar* Δσ_max) and the actual position is (x act ,y act ), Δσ_max represents the maximum fluctuation amplitude of motion noise, which is expressed by the following formula: The response action deviation Pd is calculated.

9. The dynamic programming monitoring method based on Markov model according to claim 1 is characterized in that: The path planning quality index is obtained as follows: The obstacle avoidance success frequency is expressed as the ratio of the number of successful avoidance attempts by the robot to the total number of obstacle avoidance attempts made by the robot; Collect the angular velocity data of the robot at each time point during the movement process. There are n time points, i represents the sequence number of the time points, and the angular velocity of the i-th time point is recorded as represents the average angular velocity; The path planning quality index LK is calculated by the following formula: ; ;in, represents the smoothness coefficient, represents the successful obstacle avoidance frequency, and Ca represents the convergence accuracy.

Images