A Lyapunov-optimized method for mobile robot inspection and collaborative unloading
By employing Lyapunov optimization and multi-agent reinforcement learning methods, a data backlog and energy deficit queue are constructed to achieve coupled decision-making between paths and resources in mobile robot inspection. This solves the problem of the separation between path planning and resource allocation in existing technologies, and improves the long-term stability and resource utilization of the system.
Patent Information
- Application Number
- CN202611076971.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-07-20
AI Technical Summary
Existing technologies for mobile robot inspection suffer from problems such as disconnect between path planning and resource allocation, insufficient system coordination, and poor long-term operational stability. In particular, they have low computational resource utilization in complex environments and lack rigorous theoretical guarantees, leading to the risk of unlimited data accumulation or energy depletion.
A Lyapunov optimization-based approach is adopted. By constructing a data backlog queue and an energy deficit virtual queue, and combining them with a multi-agent proximal policy optimization framework, a joint optimization objective function is generated. This enables coupled decision-making of movement path, data acquisition rate, and computation offloading strategy. Furthermore, the policy network is updated through reinforcement learning to form a closed-loop optimization.
It significantly improves the sustainability and reliability of multi-robot inspection, reduces data retention time, enhances system throughput, improves resource utilization, and ensures long-term stable operation.
Smart Images

Figure CN122573089B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial internet technology, and in particular to a mobile robot inspection and collaborative unloading method based on Lyapunov optimization. Background Technology
[0002] With the growing demand for intelligent inspection, mobile robots are widely used in facilities inspection, security monitoring, and other fields. However, when performing computationally intensive tasks such as environmental perception and image recognition, they often face performance bottlenecks due to their limited battery capacity and computing power. Therefore, task offloading technology has become crucial for improving their battery life and computing power.
[0003] Existing technical solutions mainly revolve around two core approaches: one is based on optimization theory, which constructs an objective function and uses mathematical tools to optimize it, decomposing the long-term stochastic optimization problem into solvable subproblems in each time slot, thereby jointly optimizing offloading decisions, transmission power, and computational resource allocation; the other is based on artificial intelligence, which trains an intelligent agent model through deep reinforcement learning algorithms, enabling it to adaptively generate offloading decisions according to the system state. Despite progress, the following major shortcomings remain:
[0004] 1) In terms of system collaboration, existing solutions generally treat path planning and resource allocation separately. Path planning does not fully consider the spatiotemporal availability of communication conditions and computing resources, and the unloading decision does not actively optimize the movement trajectory to improve communication quality or approach computing nodes. This hierarchical separation leads to a disconnect between physical space movement and information processing, limiting overall scheduling efficiency. 2) In terms of resource utilization, existing multi-user collaboration mainly revolves around fixed hierarchical infrastructure. Its unloading scheme is often fixed or only collaborative when encountering each other. In complex inspection environments, when fixed infrastructure is overloaded, malfunctions, or has poor signal coverage, it will severely limit the system's robustness and overall computing throughput. 3) Finally, existing methods focus on single time slot or short-term optimization, lacking rigorous theoretical guarantees. It is difficult to ensure that the task queue and energy queue of mobile robots are bounded in the long term in power plant environments with long cycles, dispersed locations, and sparse charging piles. There is a risk of data accumulation indefinitely or equipment interruption due to energy depletion. Therefore, there is an urgent need for a mobile robot inspection method that deeply integrates physical movement and information processing and provides theoretical guarantees for long-term stability. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a mobile robot inspection and collaborative unloading method based on Lyapunov optimization, which solves the problems of lack of data validity guarantee, fragmented path planning and resource allocation, and rigid resource scheduling in existing technologies under long-term operating conditions.
[0006] According to an embodiment of the present invention, a mobile robot inspection and collaborative unloading method based on Lyapunov optimization includes the following steps:
[0007] S1: Each mobile robot perceives the status of the inspection task and its own energy consumption status in real time, builds a data backlog queue based on the amount of unprocessed tasks at each inspection point, and builds an energy deficit virtual queue based on the difference between real-time energy consumption and energy threshold.
[0008] S2: Based on the data backlog queue and the energy deficit virtual queue, construct a Lyapunov function, and obtain a joint optimization objective function that includes data acquisition, mobile energy consumption and computational offloading cost by minimizing the upper bound of the drift plus penalty term;
[0009] S3: Based on the current dual queue state, environmental perception information and neighbor node resources, a multi-agent near-end strategy optimization framework is used to solve the joint optimization objective function and generate a joint decision for global collaborative optimization. This joint decision includes at least mutually coupled movement path decision, data acquisition rate allocation and computation offloading strategy.
[0010] S4: The mobile robot moves to the target location according to the decision path, collects data according to the allocation rate, and dynamically allocates tasks between edge servers according to the unloading strategy, either locally, through direct communication between devices to perform collaborative calculations with neighboring robots, or by unloading to edge servers, to obtain the actual task processing volume and energy consumption data.
[0011] S5: Update the data backlog queue based on the actual task processing volume and energy consumption data, and update the energy deficit queue based on the robot's total energy consumption in the corresponding time slot and the energy replenishment amount at the charging station. Use the updated queue state as the input for the next step, and update the parameters of the policy network by learning the reward value corresponding to the historical decision and the history of queue state changes to achieve closed loop.
[0012] Furthermore, the Lyapunov function is composed of the sum of squares of the data backlog queue and the sum of squares of the energy deficit virtual queue, and a non-negative weight parameter is introduced to weight the data backlog term;
[0013] The weighting parameter is used to adjust the weighting relationship between the data backlog queue length and system energy consumption.
[0014] Furthermore, the joint optimization objective function includes:
[0015] The system data backlog weighting term is the product of the weighting parameter and the sum of the total data backlog of all current inspection points. It is used to adjust the weight relationship between the data backlog queue length and system energy consumption in the optimization objective.
[0016] The data queue change term is a single-step conditional expectation of the current data backlog queue state and energy deficit queue state, used to characterize the single-step change of the data backlog queue.
[0017] The energy queue change term is the product of the energy deficit queue and the difference between the actual energy consumption and the amount of charging replenishment within a single time slot. It is used to characterize the single-step change of the virtual energy deficit queue.
[0018] The constraints include robot movement distance constraints, data acquisition volume constraints, computing resource capacity constraints, and unloading ratio allocation constraints.
[0019] Furthermore, the solution of the joint optimization objective function adopts an algorithm framework based on multi-agent reinforcement learning, specifically implemented as follows:
[0020] The multi-robot inspection system is modeled as a multi-agent Markov decision process, where the global state includes the position distribution of all robots, the energy deficit virtual queue state, and the data backlog queue state of the inspection points.
[0021] Each robot acts as an independent intelligent agent, selecting actions based on local observation states, which include the robot's own position, energy deficit level, data backlog at its location, and the status of neighboring robots within its communication range.
[0022] The policy network parameters are updated using a proximal policy optimization algorithm. By maximizing the cumulative reward function constructed based on Lyapunov drift, a joint decision is made on the movement path, data acquisition rate, and computation unloading strategy.
[0023] Furthermore, in the multi-agent reinforcement learning framework, the action space includes position movement decisions, data acquisition volume decisions, and unloading mode selection:
[0024] The location movement decision is to select the target inspection point at the next moment while satisfying the maximum movement distance constraint;
[0025] The data collection volume decision is to select the amount of data to be collected from the backlog queue of the current location, and the amount of data collected shall not exceed the current total backlog of that location;
[0026] The unloading mode selection includes local calculation, unloading to a nearby robot via D2D communication, or unloading to an edge server, and determines the unloading ratio for each path;
[0027] The unloading mode selection introduces a collaborative selection variable, which restricts the task to select at most one collaborative robot for unloading within each time slot.
[0028] Furthermore, the data acquisition rate allocation is limited by the data backlog status at the current location, specifically:
[0029] When the robot is at an inspection point, the upper limit of the data acquisition rate is determined by the length of the data backlog queue at that point.
[0030] The data acquisition rate is configured to zero when the robot leaves the current location or when the data backlog at the current location is zero.
[0031] Furthermore, the computation offloading strategy constructs a unified collaborative model that includes local computing resources, nearby robot computing resources, and edge server computing resources;
[0032] Based on real-time channel status information, load status, and energy status, calculate the latency and energy consumption costs of local processing, direct communication transmission between devices, and edge offloading.
[0033] Based on the cost calculation results, a proportional allocation is made between local computing, direct communication offloading between devices, or edge server offloading.
[0034] Furthermore, the method for updating the energy deficit virtual queue is as follows:
[0035] The robot's computing energy consumption, transmission energy consumption, and movement energy consumption are monitored in real time at each time slot, and the total energy consumption is obtained by summing them up.
[0036] When the robot is located at a charging station, the charging mechanism is triggered, and the energy replenishment value is updated according to the charging power.
[0037] Based on the difference between the total energy consumption and the energy replenishment, the energy deficit virtual queue is updated using the following formula:
[0038] ;
[0039] in, for Time queue value, Total energy consumption, This refers to the amount of charge. This is an indicator function; it takes a value of 1 when the robot is at a charging station, and 0 otherwise.
[0040] Furthermore, the distributed reinforcement learning method employs a multi-agent proximal policy optimization algorithm, including:
[0041] Build independent execution and evaluation networks for each robot;
[0042] The execution network is used to output an action probability distribution based on the local observation state and to limit the policy update magnitude by a pruning objective function. The pruning objective function is determined based on the product of the policy update ratio and the dominance function, wherein the dominance function is constructed according to Lyapunov drift.
[0043] The evaluation network is used to evaluate the global state value, and its parameter updates are achieved by minimizing the loss function calculated based on the reward value, which is constructed by inverting the Lyapunov drift plus penalty term;
[0044] During training, global state information is used to update the evaluation network parameters, while local observation information is used to update the execution network parameters.
[0045] Furthermore, the method also includes a periodic global optimization step:
[0046] At the end of each training round, state transition samples from all time slots are collected to form an experience replay pool;
[0047] Based on the replay pool data, the parameters of the execution network and the evaluation network are updated iteratively multiple times;
[0048] By periodically updating parameters, the multi-robot system can adjust its strategy for balancing data backlog reduction and energy consumption in an inspection environment.
[0049] The technical principle of this invention is as follows: by constructing a data backlog queue at inspection points and a virtual queue for robot energy deficit, the congestion of the system is measured by a Lyapunov function, transforming long-term stochastic optimization into single-slot deterministic optimization; the function computation adopts the Multi-Agent Proximal Policy Optimization (MAPPO) framework, and the robot autonomously decides on movement, data collection, and unloading strategies based on local observations (position, queue, and neighbor status); finally, the policy network is updated through a reward mechanism (penalizing backlog and encouraging energy conservation), realizing coupled scheduling of "movement-data collection-computation", which can ensure long-term system stability without future prediction.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] 1. This invention adopts a unified modeling method for data backlog queues and energy deficit virtual queues, and uses the Lyapunov function to measure system congestion. It solves the problems of existing technologies lacking strict theoretical guarantees and being unable to avoid unbounded data accumulation or robot energy depletion during long-term operation, and significantly improves the sustainability and reliability of multi-robot inspection.
[0052] 2. This invention enables the robot to dynamically adjust its path and task processing strategies based on data backlog distribution and resource status through a coupled optimization mechanism of mobile decision-making, data acquisition and computational unloading. This solves the technical problem that traditional path planning and computational unloading are independent of each other, resulting in a disconnect between physical space movement and information processing, and limited overall efficiency. It significantly reduces data dwell time and improves the overall throughput of the system.
[0053] 3. This invention adopts a heterogeneous resource on-demand scheduling strategy based on multi-agent reinforcement learning, which solves the technical problem that existing offloading schemes rely on a single computing mode or passively triggered cooperation, resulting in the inability to utilize idle computing power and low resource utilization. It realizes flexible switching and dynamic scheduling of local, D2D and edge computing resources, fully taps the idle computing power of the system, alleviates local computing bottlenecks, and enhances the robustness of the system. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention.
[0055] Figure 2 This is a schematic diagram of the power plant topology and multi-robot collaboration model according to an embodiment of the present invention.
[0056] Figure 3 This is a flowchart of the MAPPO collaborative inspection and unloading algorithm according to an embodiment of the present invention.
[0057] Figure 4 This is a data queueing curve diagram of the MAPPO method according to an embodiment of the present invention.
[0058] Figure 5 This is a data queue curve diagram of the PSO method in an embodiment of the present invention.
[0059] Figure 6 This is a graph showing the energy consumption queue of the MAPPO method according to an embodiment of the present invention.
[0060] Figure 7 This is a graph showing the energy consumption queue of the PSO method according to an embodiment of the present invention. Detailed Implementation
[0061] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0062] like Figure 1 As shown in the figure, this invention proposes a mobile robot inspection and collaborative unloading method based on Lyapunov optimization, including the following steps:
[0063] S1: Each mobile robot perceives the status of the inspection task and its own energy consumption status in real time, builds a data backlog queue based on the amount of unprocessed tasks at each inspection point, and builds an energy deficit virtual queue based on the difference between real-time energy consumption and energy threshold.
[0064] S2: Based on the data backlog queue and the energy deficit virtual queue, a Lyapunov function is constructed. By minimizing the upper bound of the drift plus penalty term, a joint optimization objective function including data acquisition, mobile energy consumption and computational offloading cost is obtained.
[0065] S3: Based on the current dual queue state, environmental perception information and neighbor node resources, a multi-agent near-end strategy optimization framework is used to solve the joint optimization objective function and generate a joint decision for global collaborative optimization. This joint decision includes at least the mutually coupled movement path decision, data acquisition rate allocation and computation offloading strategy.
[0066] S4: The mobile robot moves to the target location according to the decision path, collects data according to the allocation rate, and obtains the actual task processing volume and energy consumption data by calculating locally, cooperating with neighboring robots through direct communication between devices, or dynamically allocating tasks between edge servers according to the unloading strategy.
[0067] S5: Update the data backlog queue based on the actual task processing volume and energy consumption data, and update the energy deficit queue based on the robot's total energy consumption in the corresponding time slot and the energy replenishment amount at the charging station. Use the updated queue state as the input for the next step, and update the parameters of the policy network by learning the reward value corresponding to the historical decision and the history of queue state changes to achieve closed loop.
[0068] The detailed working process of this embodiment is as follows:
[0069] In step S1, the task and energy consumption are sensed, a data backlog queue and an energy deficit virtual queue are constructed, and a power plant topology model is established.
[0070] To address the lack of a unified modeling and long-term stability guarantee mechanism for data backlog and energy consumption in existing technologies, this invention constructs a dual-queue model to transform the dynamically changing inspection task volume and energy status into quantifiable queue states, providing a definite mathematical basis for subsequent joint optimization, thereby theoretically avoiding the risks of infinite data accumulation and robot energy depletion.
[0071] 1. Power plant topology and robot modeling:
[0072] Robot ensemble: In a power plant environment, robots can continuously move between inspection points and collect and process information from these points. Define an ensemble of m mobile robots. Each robot is equipped with maximum computing resources and maximum transmission power .
[0073] Divide time into discrete time slots The length of each time slot is All decisions and state updates are executed in fixed time slots.
[0074] Inspection points and charging stations: After being detected by robots, inspection points and charging stations in power plants can be abstracted into a diagram. To display the connectivity between points, where Indicates the inspection points and charging station collection points. ; Represents the set of edges connecting vertices, with their weights. This can be expressed as the distance between points, and can be achieved using an adjacency matrix. express:
[0075] ;
[0076] in, Indicates the connection of inspection points and The distance of the robot's movement route. Specifically, when... hour, .
[0077] 2. Robot position distribution model:
[0078] Location representation: using express The distribution of [something] at various inspection points, when =0 means Not at the inspection point. =1, At the inspection point. Therefore In the time slot The location can be represented as:
[0079] ;
[0080] when When, it means In the time slot It was moved.
[0081] Motion Constraints: Given the limited mobility of the robot within a single time slot, the robot's positional changes between adjacent time slots are constrained by the maximum movable distance, which leads to:
[0082] ;
[0083] in, This indicates the maximum distance the robot can move within a time slot.
[0084] 3. Data backlog queue model:
[0085] Queue Definition: Considering the continuous generation of inspection point data and the asynchronous access by the robot, this invention defines a queue for each inspection point. Establish a data backlog queue , indicating in time slot The initial total amount of unprocessed data accumulated at this point, in bits.
[0086] Data generation: per time slot Each inspection point Independently generate new data, with a generation volume of [number]. After the data is generated, it is added to the queue for that location.
[0087] Data Acquisition: When the time slot robot Located at the point At that time, it can collect data from the queue at that location for processing. In the time slot The amount of data collected is denoted as This is a decision variable that must satisfy:
[0088] .
[0089] Queue Update: The data queue for each inspection point is updated according to the following rules:
[0090] ;
[0091] in, Indicates in time slot All located at the point The total amount of data collected by the robot from this location.
[0092] 4. Energy Deficit Virtual Queue Model:
[0093] Queue definition: for each robot Building a virtual queue for energy deficit It is used to quantify the difference between energy consumption and energy threshold, reflecting long-term energy constraints.
[0094] Energy consumption components: Total energy consumption of the robot It consists of three parts:
[0095] 1) Calculate energy consumption:
[0096] ;
[0097] Among them, power calculation , Energy consumption coefficient These are architecture-related constants, typically ranging from 2 to 3. For local calculation latency, For unloading indicator variables (1 indicates) Towards uninstall).
[0098] 2) Transmission power consumption: , For transmission power, This refers to transmission delay.
[0099] 3) Mobile energy consumption: ;in, The distance energy consumption coefficient, Starting from, The endpoint is the path; if there is no direct path, the distances of all adjacent points on the path must be accumulated.
[0100] Charging mechanism: Define indicator function When the robot is at the charging station Otherwise, it is 0; the robot replenishes a fixed amount of energy per time slot while at the charging station. .
[0101] The queue update formula is:
[0102] ;
[0103] in, for Time queue value, for Energy consumption always exists. for The fixed amount of energy that can be replenished when located at a charging station in each time slot. This is an indicator function; it takes a value of 1 when the robot is at a charging station, and 0 otherwise.
[0104] This invention transforms the originally random and dynamic inspection process into a deterministic queue evolution process by establishing a data backlog queue and an energy deficit virtual queue, providing a rigorous mathematical basis for the subsequent optimization framework and ensuring the long-term stability of the system.
[0105] In step S2, a Lyapunov function is constructed based on the dual queues, the drift plus penalty term is derived, and a joint optimization objective function is generated.
[0106] Considering that existing technologies have problems with path planning and computational unloading decisions being independent, leading to low scheduling efficiency, this invention uses Lyapunov drift theory to transform long-term stochastic optimization problems into deterministic optimization problems based on the current queue state, forcibly incorporating the three dimensions of "movement-collection-computation" into the same mathematical framework for coupled optimization.
[0107] Aggregate the queue states of the entire system into a vector:
[0108] ;
[0109] Lyapunov function definition: Define the function The degree of "congestion" across all queues in the system is measured as a weighted linear combination of the sum of squares of the data backlog queues and the sum of squares of the energy deficit virtual queues:
[0110] ;
[0111] The larger the function value, the more severe the overall system congestion.
[0112] Single-step drift: Define single-step conditional Lyapunov drift as the expected change from the current queue state to the next state.
[0113] ;
[0114] To ensure queue stability and minimize long-term average data backlog, a drift penalty term is constructed:
[0115] ;
[0116] in, >0 is a control parameter used to adjust the weighting relationship between queue length and system performance.
[0117] Objective function generation: By performing upper bound analysis on the drift term, in each time slot The system decision objective is transformed into minimizing this upper bound, which can be expressed as the following single-slot deterministic optimization problem based on the current queue state, ultimately generating a joint optimization objective function. for:
[0118] ;
[0119] in, For local calculation ratios, To unload to the robot proportion, The proportion of data offloaded to edge servers; For binary unloading indicator variables (1 indicates) Towards uninstall), The number of CPU cycles required to process 1 bit of data.
[0120] The first term of this objective function is the system data backlog weighting term, which is used to directly penalize the current total amount of system data backlog and drive the system to reduce data backlog; the second term is the data queue change term, which is used to encourage the robot to collect data to reduce the data queue length; and the third term is the energy queue change term, which is used to encourage the robot to reduce energy consumption and recharge in a timely manner to maintain energy balance.
[0121] Delay and Rate Model:
[0122] D2D transmission rate: Robot Towards The maximum transfer rate during unloading is:
[0123] ;
[0124] in, express and The wireless channel bandwidth between It is the noise power spectral density. Represents free space path loss, and and The distance between them is inversely proportional.
[0125] Response latency: Robot The total response latency is taken as the maximum value of the three modes:
[0126] ;
[0127] Local computation latency: .
[0128] Edge unloading delay: ; This provides the maximum computing resources for edge servers.
[0129] D2D unloading delay: .
[0130] This invention transforms a long-term stochastic optimization problem into a deterministic optimization problem. The robot can make decisions based solely on the current queue state, without needing to predict future tasks or energy consumption, thus reducing decision complexity. The objective function simultaneously couples movement, acquisition, and unloading decisions, achieving three-dimensional collaborative optimization and providing a clear mathematical objective for subsequent algorithm solutions.
[0131] In step S3, the multi-robot system is modeled as a multi-agent Markov decision process (MAMDP), and the multi-agent proximal policy optimization (MAPPO) algorithm is used to solve the joint optimization objective to generate movement, acquisition, and unloading strategies.
[0132] To address the problem that existing offloading solutions rely on a single computing model or static collaboration and are difficult to handle high-dimensional dynamic decision-making, this invention uses a multi-agent reinforcement learning framework to fit a high-dimensional state space using neural networks, enabling on-demand dynamic scheduling of heterogeneous computing resources, while improving system robustness through distributed decision-making.
[0133] This invention models the solution process as a multi-agent Markov decision process (MAMDP), denoted as... ;in, Represents the global state space of the environment; For the joint actions of all robots (Agents), where Indicates the first An agent in a time slot The action; Indicates the reward for performing the action; The state space observed by each robot (Agent); For state transition probabilities; in each time slot, the Agent determines the probability based on the policy. Select an action; all agents executing that action will cause the environment to change from its current state. Transition to the next state It follows the state transition probability. Each robot makes independent decisions as an intelligent agent, but they are coordinated through a global value function, with their common goal being to maximize the cumulative reward.
[0134] 1. The basic elements of MAMDP are defined as follows:
[0135] 1) Global environmental status :
[0136] For any time slot The global state of the system Composed of the physical and queue states of all robots within the power plant, it reflects the overall real-time operation of the system:
[0137] ;
[0138] in, This indicates the location distribution of all robots; This indicates the virtual queue status of all robots' energy deficit. This indicates the data backlog queue status for all inspection points.
[0139] 2) Local observation :
[0140] Due to limitations in communication range and perception capabilities, each robot can only acquire local information in the observation state space. We consider the robot's own physical state, the backlog at its location, and the states of neighboring robots as the agent's observations. The observed state of the Agent it represents It can be represented as:
[0141] ;
[0142] in, Number the inspection point where the robot is currently located; This represents the robot's current energy deficit level. The amount of data to be processed at the current location; This is the set of states of neighboring robots within the communication range.
[0143] 3) Action Space :
[0144] In the time slot Each agent needs to simultaneously decide where to go and how to process the data. Actions Including location relocation decisions, data collection volume decisions, and offloading mode selection, it can be represented as:
[0145] ;
[0146] in, The target movement position for the next time slot must meet the following conditions. Neighborhood constraints; This represents the amount of data collected from the current location. To calculate the unloading ratio decision, .
[0147] 4) Reward Function :
[0148] To transform the Lyapunov optimization objective into a reward maximization objective in reinforcement learning, we invert the objective function in the optimization problem and use it as the reward function to drive the agent to maximize the cumulative reward. Simultaneously, we introduce a quadratic term for the energy consumption queue to define the stable energy consumption queue. of The instant rewards are as follows:
[0149] .
[0150] This reward mechanism design based on Lyapunov drift has a dual regulation mechanism:
[0151] When data backlog at a certain point When the value is higher, the weight of the first item increases, incentivizing the robot to collect more data. Alternatively, it can be moved to that high backlog point to maximize system throughput.
[0152] When the robot has an energy deficit At higher levels, the penalty weights for the second and fourth items increase significantly, forcing the agent to actively move to the charging station using low-energy-consumption actions. This results in positive energy rewards, thereby ensuring the energy stability of the system's long-term operation.
[0153] By maximizing this cumulative reward, the agent can automatically learn the optimal dynamic balance strategy between clearing data backlogs and maintaining its own power.
[0154] 2. MAPPO Algorithm Design:
[0155] MAPPO combines the policy stability of PPO with the advantages of global coordination among multi-agent systems, comprising an Actor (policy network) and a Critic (value network). PPO is an improved version of the policy gradient method, which improves training stability by limiting the magnitude of policy updates, preventing drastic policy changes during updates that could lead to learning instability or performance degradation. However, in multi-agent environments, learning a local value function for each agent individually may result in inadequate consideration of inter-agent interactions. To coordinate the behavior of multiple agents and improve the overall system performance, MAPPO introduces a global value function.
[0156] Actor Networks: Using Representing the The Actor network parameters of each Agent approximate the policy function, and the input local observations are used as inputs. Output action probability distribution To limit the magnitude of policy updates and improve training stability, MAPPO introduces a Clip objective function for the Actor network to avoid training oscillations. Its loss function is as follows:
[0157] ;
[0158] in, The strategy update ratio. Used to limit the policy update rate, if Exceed It will then be cropped to or . It is the dominance function.
[0159] Critic network: Each robot i has an independent Critic network, using... The parameters of the Critic network are represented by an approximate state-value function. The input is the global state. Output state value The loss function is the mean squared error:
[0160] ;
[0161] in, This is the discount factor.
[0162] Dominance function calculation: Generalized dominance estimation (GAE) is used.
[0163] ;
[0164] in, , It is a parameter among the advantageous parameters.
[0165] In the MAPPO algorithm, the agent The parameters of the Actor network and Critic network can be updated using the following formula:
[0166] ;
[0167] ;
[0168] In some examples, as shown in Table 1, the steps of the MAPPO algorithm based on Lyapunov optimization are as follows:
[0169]
[0170] Table 1
[0171] This invention achieves distributed decision-making through multi-agent reinforcement learning. The robot only needs local observation to collaboratively optimize movement, collection, and unloading strategies. The MAPPO algorithm ensures training stability, and the reward function is directly linked to Lyapunov drift, ensuring that the decision-making simultaneously satisfies data backlog reduction and energy constraints, thereby improving the utilization rate of heterogeneous resources.
[0172] In steps S4 and S5, the robot performs movement, collection, and unloading operations according to the decision, updates the dual queues based on the actual results, and adjusts the policy network parameters through reinforcement learning feedback to form a closed-loop optimization.
[0173] Since existing static scheduling cannot adapt to dynamic environments and seriously affects the accuracy of the model, this invention updates the queue and policy network in real time, enabling the system to dynamically respond to fluctuations in data generation, changes in communication conditions and energy state transitions, thus achieving long-term stable operation.
[0174] 1. Decision Implementation:
[0175] Mobile execution: Robot press When moving to the target location, if there are intermediate nodes along the path, move along the shortest path and accumulate the movement energy consumption. , This represents the number of nodes in the path.
[0176] Data Acquisition: Data collected from the current location. Data, if (Abnormal situation), forced truncation .
[0177] Task Uninstallation:
[0178] Local calculation: by The ratio is processed locally, reducing computational latency. Energy consumption is:
[0179] ;
[0180] D2D Uninstall: Press =1 Select Collaborative Robot Transmission rate Transmission delay The transmission energy consumption is:
[0181] ;
[0182] Edge offloading: Uploads to the edge server via base station, at a speed of... The calculation of latency and energy consumption is the same as that of D2D unloading.
[0183] 2. Queue Update:
[0184] Data queue update:
[0185] ,like This indicates that the location data has been cleared.
[0186] Energy queue update:
[0187] ,like This indicates that there is sufficient energy.
[0188] 3. Strategy Update:
[0189] Experience replay: After each round, the stored samples are used for training, and the parameters of the Actor and Critic networks are updated through multiple iterations.
[0190] Exploration and Utilization: Employing methods in the initial stages of training Greedy strategies increase exploration, which gradually decreases in the later stages. This makes the strategy more stable.
[0191] Stability proof:
[0192] Lyapunov drift analysis can prove that: if the objective function If minimized, then there exists a constant. >0, making it possible for any time slot have:
[0193] ;
[0194] That is, the long-term average queue length is bounded, and the risks of unbounded data backlog and energy depletion are eliminated.
[0195] Periodic global optimization:
[0196] In some examples, each Perform a global optimization once every 100 rounds:
[0197] Collect all historical trajectory data of the robot and analyze data backlog hotspots and energy consumption bottlenecks;
[0198] Adjusting weight parameters (e.g., hotspot areas increase) The energy-stressed area is reduced. );
[0199] Reinitialize some network parameters to avoid local optima.
[0200] like Figure 3 As shown, the algorithm flow of this invention mainly includes three stages: First, system initialization is performed, network parameters and environmental state are configured, and data queues and energy queues are established; second, the time slot loop is entered, the robot perceives the local environment and outputs actions, performs movement, collection and unloading operations, the system updates the queue state in real time and calculates Lyapunov rewards; finally, the network parameters are updated using cache pool data, and the strategy is optimized through multiple rounds of iteration until the termination condition is met, so as to realize closed-loop collaborative optimization in dynamic environment.
[0201] This invention enables the system to dynamically adapt to environmental changes through a closed loop of "execution-feedback-update", with dual queues reflecting the system status in real time and a policy network continuously optimizing decision quality, ultimately achieving long-term stable operation and efficient resource utilization.
[0202] To verify the practicality of the proposed method, a comparative experiment was conducted with the Particle Swarm Optimization (PSO) algorithm. The simulation platform configuration was as follows: dual Intel Xeon Platinum 8470Q CPU, NVIDIA RTX 5090 D GPU, Ubuntu 22.04 operating system, Python 3.12 programming language, and PyTorch 2.10 deep learning framework. To ensure fairness in the comparison, all algorithms were run under the same initial conditions, and the environment was not reset throughout the experiment. Each algorithm was executed for 100 episodes, with each episode containing 100 time steps.
[0203] Experimental results are as follows Figure 4 , 5 As shown in Figures 6 and 7, the MAPPO method significantly outperforms PSO in terms of data queue length, consistently maintaining a shorter queue backlog. Furthermore, MAPPO not only effectively stabilizes the energy deficit queue but also maintains stable energy consumption levels even when the energy queue experiences a sharp increase. Figures 4-7The study demonstrates the queue changes of MAPPO and PSO in 100 episodes, showing that MAPPO has a significant advantage in queue stability, further validating its effectiveness and robustness in real-world scenarios.
[0204] To more intuitively demonstrate MAPPO's advantages in queue control, the specific queue data for MAPPO and PSO in the last episode were summarized. As shown in Tables 2 and 3, the data queues for each inspection point and the energy deficit queues for each inspection robot indicate that MAPPO's data queues remained around 200, and its energy deficit queues also remained at a low level. In contrast, PSO's inspection point data queues continued to grow, and all energy deficit queues accumulated to tens of thousands, further confirming MAPPO's superiority in maintaining queue stability.
[0205]
[0206] Table 2
[0207]
[0208] Table 3
[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A mobile robot inspection and collaborative unloading method based on Lyapunov optimization, characterized in that: Includes the following steps: S1: Each mobile robot perceives the status of the inspection task and its own energy consumption status in real time, builds a data backlog queue based on the amount of unprocessed tasks at each inspection point, and builds an energy deficit virtual queue based on the difference between real-time energy consumption and energy threshold. S2: Based on the data backlog queue and the energy deficit virtual queue, a Lyapunov function is constructed. By minimizing the upper bound of the drift plus penalty term, a joint optimization objective function is obtained, which includes data acquisition, mobile energy consumption, and computational offloading costs. The joint optimization objective function includes: The system data backlog weighting term is the product of the weighting parameter and the total amount of data backlog at all current inspection points. It is used to adjust the weighting relationship between the data backlog queue length and system energy consumption in the optimization objective. The data queue change term is a single-step conditional expectation of the current data backlog queue state and energy deficit queue state, used to characterize the single-step change of the data backlog queue. The energy queue change term is the product of the energy deficit queue and the difference between the actual energy consumption and the amount of charging replenishment within a single time slot. It is used to characterize the single-step change of the virtual energy deficit queue. The constraints include robot travel distance constraints, data acquisition volume constraints, computing resource capacity constraints, and unloading ratio allocation constraints. S3: Based on the current dual-queue state, environmental awareness information, and neighbor node resources, a multi-agent near-end policy optimization algorithm is used to solve the joint optimization objective function, generating a joint decision for global collaborative optimization. This joint decision includes at least mutually coupled movement path decisions, data acquisition rate allocation, and computation offloading strategies. Specifically, the solution to the joint optimization objective function uses a multi-agent near-end policy optimization algorithm, implemented as follows: The multi-robot inspection system is modeled as a multi-agent Markov decision process, where the global state includes the position distribution of all robots, the energy deficit virtual queue state, and the data backlog queue state of the inspection points. Each robot acts as an independent intelligent agent, selecting actions based on local observation states, which include the robot's own position, energy deficit level, data backlog at its location, and the status of neighboring robots within its communication range. The policy network parameters are updated using a proximal policy optimization algorithm. By maximizing the cumulative reward function constructed based on Lyapunov drift, a joint decision is made on the movement path, data acquisition rate, and computation unloading strategy. S4: The mobile robot moves to the target location according to the decision path, collects data according to the allocation rate, and dynamically allocates tasks between local calculations, direct communication between devices to nearby robots, or to edge servers according to the unloading strategy, so as to obtain the actual task processing volume and energy consumption data. S5: Update the data backlog queue based on the actual task processing volume and energy consumption data, and update the energy deficit virtual queue based on the robot's total energy consumption in the corresponding time slot and the energy replenishment amount at the charging pile location. Use the updated queue state as the input for the next step, and update the parameters of the policy network by learning the reward value corresponding to the historical decision and the history of queue state changes to achieve closed loop.
2. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: The Lyapunov function is composed of the sum of squares of the data backlog queue and the sum of squares of the energy deficit virtual queue, and a non-negative weight parameter is introduced to weight the data backlog term. The weighting parameter is used to adjust the weighting relationship between the data backlog queue length and system energy consumption.
3. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: In the multi-agent Markov decision-making process, the action space includes position movement decisions, data acquisition volume decisions, and unloading mode selection. The location movement decision is to select the target inspection point at the next moment while satisfying the maximum movement distance constraint; The data collection volume decision is to select the amount of data to be collected from the backlog queue of the current location, and the amount of data collected shall not exceed the current total backlog of that location; The unloading mode selection includes local calculation, unloading to a nearby robot via direct communication between devices, or unloading to an edge server, and determines the unloading ratio for each path; The unloading mode selection introduces a collaborative selection variable, which restricts the task to select at most one collaborative robot for unloading within each time slot.
4. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: The data acquisition rate allocation is limited by the data backlog status at the current location, specifically: When the robot is at an inspection point, the upper limit of the data acquisition rate is determined by the length of the data backlog queue at that point. The data acquisition rate is configured to zero when the robot leaves the current location or when the data backlog at the current location is zero.
5. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: The computation offloading strategy constructs a unified collaborative model that includes local computing resources, nearby robot computing resources, and edge server computing resources. Based on real-time channel status information, load status, and energy status, calculate the latency and energy consumption costs of local processing, direct communication transmission between devices, and edge offloading. Based on the cost calculation results, a proportional allocation is made between local computing, direct communication offloading between devices, or edge server offloading.
6. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: The method for updating the energy deficit virtual queue is as follows: The robot's computing energy consumption, transmission energy consumption, and movement energy consumption are monitored in real time at each time slot, and the total energy consumption is obtained by summing them up. When the robot is located at a charging station, the charging mechanism is triggered, and the energy replenishment value is updated according to the charging power. Based on the difference between the total energy consumption and the energy replenishment, the energy deficit virtual queue is updated using the following formula: ; in, for Time queue value, Total energy consumption, This refers to the amount of charge. This is an indicator function; it takes a value of 1 when the robot is at a charging station, and 0 otherwise.
7. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 1, characterized in that: The parameter updates of the policy network include: Build independent execution and evaluation networks for each robot; The execution network is used to output an action probability distribution based on the local observation state and to limit the policy update magnitude by a pruning objective function. The pruning objective function is determined based on the product of the policy update ratio and the dominance function, wherein the dominance function is constructed according to Lyapunov drift. The evaluation network is used to evaluate the global state value, and its parameter updates are achieved by minimizing the loss function calculated based on the reward value, which is constructed by inverting the Lyapunov drift plus penalty term; During training, global state information is used to update the evaluation network parameters, while local observation information is used to update the execution network parameters.
8. The mobile robot inspection and collaborative unloading method based on Lyapunov optimization as described in claim 7, characterized in that: The method also includes a periodic global optimization step: At the end of each training round, state transition samples from all time slots are collected to form an experience replay pool; Based on the replay pool data, the parameters of the execution network and the evaluation network are updated iteratively multiple times; By periodically updating parameters, the multi-robot system can adjust its strategy for balancing data backlog reduction and energy consumption in an inspection environment.
Citation Information
Patent Citations
Wind power plant inspection method and system fused with mobile edge computing
CN119105858A
Multi-queue task distributed unloading method and device in edge cloud Internet of Vehicles and medium
CN120034908A
