A cloud-edge collaborative robot cluster simulation training and optimization system

Abstract

The present invention discloses a cloud-edge collaborative robot cluster simulation training and optimization system, comprising: a node labeling unit, a node classification unit, a dual-model construction unit, a reinforcement learning unit, and a periodic parameter collection unit for periodically collecting real-time cloud model parameters and local model parameters; a parameter fusion unit for unifying the format and aligning the timestamps of cloud model parameters and local model parameters, and fusing them into global parameters; a global model application unit for constructing the global parameters into a global strategy model under a reinforcement learning framework for use in simulation optimization training of the target system; the present invention improves the clarity of computing resource configuration, realizes the unified expression of cross-node task objectives, provides an input basis for centralized strategy optimization, and improves the consistency of cloud-edge collaboration.

Description

Technical Field

[0001] The present invention relates to the field of cloud-edge collaboration, and specifically to a cloud-edge collaborative robot cluster simulation training and optimization system. Background Art

[0002] With the widespread deployment of robot clusters in scenarios such as intelligent manufacturing and urban distribution, the system has put forward higher requirements for multi-objective optimization such as path planning, task scheduling, and energy consumption control. To improve the overall collaborative efficiency, more and more robot systems are introducing cloud computing and edge computing resources to build a "cloud-edge-end" collaborative control architecture. In recent years, some studies have attempted to adopt a cloud-edge collaborative reinforcement learning framework to achieve multi-level collaborative decision-making by optimizing global strategies in the cloud and training local strategies on edge nodes. For example, patent document CN113067873B proposes a cloud-edge collaborative optimization method based on deep reinforcement learning, which has achieved initial results in delay control and energy consumption optimization.

[0003] However, existing technologies still have key bottlenecks: the current method cannot dynamically adjust the role division of edge nodes according to the complexity of the task, resulting in unreasonable allocation of cloud-edge computing resources; at the same time, the sub-task goals lack systematic indicator aggregation, making it difficult to build a complete and clear global optimization goal; in addition, the cloud and local policy model training processes are independent of each other, lacking a unified parameter fusion and synchronization mechanism, which affects the overall policy consistency of the system. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a cloud-edge collaborative robot cluster simulation training and optimization system, which solves the technical problems raised in the background technology by introducing dynamic calculation of task complexity and subtask indicator aggregation.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A cloud-edge collaborative robot cluster simulation training and optimization system, including:

[0007] A node marking unit, used to mark K edge nodes of the target system;

[0008] A node classification unit, used to classify K edge nodes into N cloud nodes and M local nodes;

[0009] A dual-model construction unit, used to respectively construct a cloud policy model for the cloud node and a local policy model for the local node;

[0010] A reinforcement learning unit is used to build a reinforcement learning framework and update the model parameters of the cloud policy model and the local policy model based on the reinforcement learning framework to form real-time cloud model parameters and local model parameters;

[0011] Periodic parameter collection unit, used to periodically collect real-time cloud model parameters and local model parameters;

[0012] The parameter fusion unit is used to unify the format and timestamp of the cloud model parameters and local model parameters, and fuse them into global parameters;

[0013] The global model application unit is used to construct the global parameters into a global strategy model under the reinforcement learning framework for use in the simulation optimization training of the target system.

[0014] In some embodiments, the classification step of the node classification unit includes:

[0015] S2-1. Obtain the pre-assigned node indexes of K edge nodes;

[0016] S2-2. Match the node subtask of the edge node in the target system according to the node index

[0017] S2-3, calculate the task complexity of the node subtask;

[0018] S2-4, comparing the task complexity of the node subtask with a preset threshold to classify the K edge nodes into N cloud nodes and M local nodes;

[0019] In some embodiments, the task complexity of computing a node subtask includes:

[0020] S2-3-1. Obtain the number of robot terminals connected to the edge node, bandwidth utilization, resource utilization, and average task queue length;

[0021] S2-3-2. Perform a normalized weighted sum of the number of robot terminals connected to the edge node, bandwidth utilization, resource utilization, and average task queue length to generate the task complexity of the edge node;

[0022] In some embodiments, the step of constructing the cloud policy model includes:

[0023] S3-A1, obtain node subtasks and subtask indicators of cloud nodes;

[0024] S3-A2, aggregating the subtask indicators to generate a cloud-based task target representing the system-level target;

[0025] S3-A3. Define the optimization process of the cloud task objectives as centralized reinforcement learning under the reinforcement learning framework to build a cloud strategy model.

[0026] In some embodiments, the subtask indicators are aggregated to generate a cloud-based task target representing a system-level target, including:

[0027] S3-A2-1. Obtain several target features for each subtask indicator;

[0028] S3-A2-2. Standardize a number of target features and combine the standardized target features into a task indicator vector representing the subtask indicator;

[0029] S3-A2-3. Concatenate the task indicator vectors corresponding to the subtask indicators to generate the cloud task target.

[0030] In some embodiments, the optimization process of the cloud task goal is defined as centralized reinforcement learning under the reinforcement learning framework, including:

[0031] S3-A3-1. Define the cloud state space and cloud action space of the cloud task goal;

[0032] Among them, the cloud state space is used to represent the environmental state of the target system during the training process of the cloud task target, which includes multi-dimensional state features;

[0033] S3-A3-2. Define an immediate reward for a cloud task target; the immediate reward for the cloud task target is based on a difference representation between state features of adjacent time steps in the cloud state space;

[0034] S3-A3-3. Calculate the long-term cumulative reward of the cloud task target based on the immediate reward of the cloud task target.

[0035] In some embodiments, the step of constructing the local policy model includes:

[0036] S3-B1: For the local node, obtain its node subtasks and subtask indicators;

[0037] S3-B2, using the subtask indicators as distributed task indicators of respective local nodes;

[0038] S3-B3. Define the optimization process of M distributed task indicators as distributed reinforcement learning under the reinforcement learning framework to construct M local strategy models.

[0039] In some embodiments, the optimization process of the M distributed task indicators is defined as distributed reinforcement learning under the reinforcement learning framework, including:

[0040] S3-B3-1. Define the distributed state space and distributed action space of M distributed task indicators

[0041] S3-B3-2. Define instant rewards for M distributed task indicators; the instant rewards for the distributed task indicators are based on the difference representation between state features of adjacent time steps in the distributed state space;

[0042] S3-B3-3. Calculate the long-term cumulative rewards of distributed task indicators based on their immediate rewards.

[0043] In some embodiments, the step of the reinforcement learning unit updating the cloud model parameters includes:

[0044] S4-1. Obtain status characteristics of N cloud nodes in real time , action characteristics , next state characteristics and instant rewards ;

[0045] S4-2, the real-time state characteristics of each time step , action characteristics , instant rewards and next state features Constructed as experience tuple ;

[0046] S4-3. Real-time state features in the prediction network of the cloud strategy model Perform forward propagation and calculate action features The predicted Q value of ;in, represents the model parameters;

[0047] S4-4. Next state characteristics in the target network of the cloud strategy model Perform forward propagation and calculate the maximum Q value , and calculate the target Q value;

[0048] The target Q value is calculated as follows:

[0049] ;

[0050] in, represents the target Q value, represents the discount factor for the predicted Q value at each time step;

[0051] S4-5, calculate the mean square error loss function between the target Q value and the predicted Q value;

[0052] S4-6. Calculate the gradient of the mean square error loss function with the current parameters, and update the parameters of the cloud strategy model based on the gradient of the current parameters. .

[0053] In some embodiments, the fusion step of the parameter fusion unit includes:

[0054] S6-1. Collect the updated model parameters of N cloud nodes and M local nodes in the current training cycle respectively;

[0055] S6-2. Unify the formats and align the timestamps of the collected cloud model parameters and local model parameters to generate aligned parameters;

[0056] S6-3. Normalize the aligned parameters to generate normalized local model parameters and normalized cloud model parameters;

[0057] S6-4. Perform weighted fusion on the normalized local model parameters and the normalized cloud model parameters to generate unified global parameters.

[0058] The present invention provides a cloud-edge collaborative robot cluster simulation training and optimization system, which has the following beneficial effects:

[0059] The present invention introduces quantitative calculation of task complexity and node classification, so that edge nodes can be adaptively classified as cloud nodes or local nodes under different load conditions, thereby improving the clarity of computing resource configuration. By constructing a vector expression and standardized aggregation method for subtask indicators, a unified expression of cross-node task objectives is achieved, providing an input basis for centralized policy optimization. The centralized and distributed training structures in reinforcement learning are adopted to construct cloud and local policy models respectively, ensuring that complex decisions have a global perspective while retaining the response efficiency of local nodes. Furthermore, through periodic parameter collection, alignment and fusion, synchronous updates between cloud-edge models are established, which solves the problem of training path fragmentation and improves the consistency of cloud-edge collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a structural block diagram of a cloud-edge collaborative robot cluster simulation training and optimization system of the present invention;

[0061] Figure 2 This is a schematic diagram of the optimization process of a cloud-edge collaborative robot cluster simulation training and optimization system of the present invention;

[0062] Figure 3 Schematic diagram of the learning process of the cloud strategy model of the present invention;

[0063] Figure 4 Schematic diagram of the global parameter fusion process of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] Example 1: Please refer to Figures 1 to 4 ,The present invention provides a cloud-edge collaborative robot cluster simulation training and optimization system, comprising: a node marking unit, for marking K edge nodes of a target system;

[0066] The target system refers to a distributed task execution platform with multi-robot collaboration capabilities, which usually includes a cloud computing center, multiple edge nodes and the robot terminals connected to them.

[0067] For example, in an intelligent manufacturing scenario, the target system can be used for the collaborative operation of multi-station robotic arms in a workshop: the edge nodes control the robot action execution, visual inspection, etc. of specific stations respectively, and the cloud centrally optimizes and schedules the production rhythm, path allocation and energy consumption.

[0068] For example, in an unmanned delivery system, the target system includes multiple mobile delivery robots and edge nodes for local path obstacle avoidance, communication coordination and other task processing. The cloud is responsible for global optimization of the overall delivery path planning, scheduling allocation and task load balancing.

[0069] In this process, K edge nodes are the logical control units in the target system. Each node is connected to several robot terminals and can be numbered and marked based on the actual deployment location.

[0070] A node classification unit, used to classify K edge nodes into N cloud nodes and M local nodes;

[0071] A dual-model construction unit, used to respectively construct a cloud policy model for the cloud node and a local policy model for the local node;

[0072] A reinforcement learning unit is used to build a reinforcement learning framework and update the model parameters of the cloud policy model and the local policy model based on the reinforcement learning framework to form real-time cloud model parameters and local model parameters;

[0073] Periodic parameter collection unit, used to periodically collect real-time cloud model parameters and local model parameters;

[0074] The parameter fusion unit is used to unify the format and timestamp of the cloud model parameters and local model parameters, and fuse them into global parameters;

[0075] The global model application unit is used to construct the global parameters into a global strategy model under the reinforcement learning framework for use in the simulation optimization training of the target system.

[0076] Specifically, the global parameters include the fused policy network weight parameters, target Q-value network parameters, policy gradients, objective function configuration parameters, etc. The global parameters are periodically collected from the cloud and local policy models over multiple cycles and obtained through timestamp alignment, normalization, and weighted fusion. Furthermore, the global policy model can be constructed based on a reinforcement learning framework, such as a distributed actor-critic framework, in which the global actor model is managed by the cloud and each node uses a local critic model to perform action evaluation.

[0077] After training, the global policy model can be periodically distributed to K edge nodes to achieve unified behavioral guidance for all sub-nodes in the target system in the direction of system-level optimization, ensuring that the training strategy takes into account both global consistency and local adaptability.

[0078] Exemplarily, the classification step of the node classification unit includes:

[0079] S2-1. Obtain the pre-assigned node indexes of K edge nodes;

[0080] The node index is represented by the node number and the corresponding node subtask and subtask indicator data set. Furthermore, each edge node is connected to several robot terminals, which can collect multi-dimensional state characteristics of the target system. These multi-dimensional state characteristics are defined based on the specific task indicators and operating environment of the target system.

[0081] S2-2. Match the node subtask of the edge node in the target system according to the node index

[0082] It should be noted that the node subtask is a task type or priority pre-assigned according to the node index.

[0083] S2-3, calculate the task complexity of the node subtask;

[0084] S2-4, comparing the task complexity of the node subtask with a preset threshold to classify the K edge nodes into N cloud nodes and M local nodes;

[0085] For example, if the task complexity is greater than a preset threshold, it is classified as a cloud node; otherwise, it is classified as a local node.

[0086] In this embodiment, the core basis for node classification is the comparison of "task complexity" with a preset threshold. By introducing a node index structure, each edge node is not only identified by a number but also carries semantic information about the task metrics it is responsible for. Different nodes have different real-time processing capacity requirements depending on the number of connected terminals, task load, and resource bottlenecks. By mapping the complexity of node subtasks into quantifiable judgment criteria, node role classification is achieved.

[0087] Furthermore, the step S2-3 specifically includes:

[0088] S2-3-1. Obtain the number of robot terminals connected to the edge node, bandwidth usage, resource usage (CPU / memory / GPU, etc.), and average task queue length;

[0089] For example, the bandwidth utilization is the ratio of the current uplink / downlink bandwidth occupancy to the total bandwidth, the resource utilization is the weighted average of the CPU utilization, memory utilization and GPU utilization of the edge node, and the average task queue length is the average number of tasks to be processed per unit time, reflecting the degree of task backlog.

[0090] S2-3-2. Perform a normalized weighted sum of the number of robot terminals connected to the edge node, bandwidth utilization, resource utilization, and average task queue length to generate the task complexity of the edge node;

[0091] Specifically, since the above execution parameters have different dimensions and value ranges, they must first be normalized to map them to a uniform range. For example, if the number of robot terminals is 8 and the maximum supported number is 10, the normalization parameter is 0.8; if the bandwidth utilization rate and resource utilization rate are 75% and 60%, respectively, the normalization parameters are defined as 0.75 and 0.6; if the average task queue length is 5 tasks and the maximum tolerated length is 10, the normalization parameter is 0.5.

[0092] The above normalized parameters are then weighted and summed to obtain the task complexity. The weight of each normalized parameter can be set based on historical data or empirical data so that the maximum task complexity is 1.

[0093] In this embodiment, the calculation of task complexity is based on the normalized weighted sum of the number of robot terminals, bandwidth utilization, resource utilization, and average task queue length. The differences in the value ranges of different indicators are resolved through standardization, and the indicators are fused using empirical weights to generate a single complexity value. The above four types of parameters jointly reflect the computing pressure and task load status of the edge node during the current operating cycle. The number of terminals characterizes the task input scale, the bandwidth and resource utilization reflect the occupancy of data interaction and computing power, and the task queue length directly indicates the current degree of task accumulation. In essence, they can all be mapped to "resource occupancy density" or "load intensity" per unit time, with consistent expression capabilities.

[0094] Exemplarily, the steps of constructing the cloud policy model include:

[0095] S3-A1, obtain node subtasks and subtask indicators of cloud nodes;

[0096] S3-A2, aggregating the subtask indicators to generate a cloud-based task target representing the system-level target;

[0097] S3-A3. Define the optimization process of the cloud task objectives as centralized reinforcement learning under the reinforcement learning framework to build a cloud strategy model.

[0098] In this embodiment, the subtask indicators of multiple cloud nodes are integrated into a unified optimization goal, which provides a clear optimization direction for centralized reinforcement learning.

[0099] Furthermore, the step S3-A2 specifically includes:

[0100] S3-A2-1. Obtain several target features for each subtask indicator;

[0101] For example, for the target system, the subtask indicators of different cloud nodes may correspond to optimization directions such as shortest path planning, highest obstacle avoidance success rate, minimum task response time, and optimal energy consumption.

[0102] Therefore, the corresponding target features should be specific and quantifiable indicators, such as path length (in meters), obstacle avoidance success rate (expressed as a percentage), response time (in seconds), energy consumption (in joules), etc. Multiple target features can together constitute a feature vector used to represent the indicators of this subtask.

[0103] S3-A2-2. Standardize a number of target features and combine the standardized target features into a task indicator vector representing the subtask indicator;

[0104] Given that the dimensions, scales, and value ranges of different target features are different, standardization or normalization methods (such as maximum and minimum normalization) are required.

[0105] S3-A2-3. Concatenate the task indicator vectors corresponding to the subtask indicators to generate the cloud task target.

[0106] For example, the weight of the path length can be preset to 0.4, the obstacle avoidance success rate to 0.3, the response time to 0.2, and the energy consumption to 0.1, and then a composite cloud task target can be generated through fusion methods such as weighted summation.

[0107] That is to say, after being standardized and spliced, the indicator features of each subtask are uniformly stored as cloud-based task targets in vector form.

[0108] In this embodiment, the cloud-based task objective is constructed using a process called "feature extraction, normalization, and vector fusion," transforming multiple subtask metrics into a numerical vector with a unified structure. By compressing the metrics of multiple subtasks into a single continuous vector, a globally expressive optimization objective is provided for the reinforcement learning model.

[0109] Furthermore, the step S3-A3 specifically includes:

[0110] S3-A3-1. Define the cloud state space and cloud action space of the cloud task goal;

[0111] The cloud state space is used to represent the environmental state of the target system during the cloud mission training process and includes multi-dimensional state features. For example, these multi-dimensional state features may include the robot's position coordinates, velocity vector, remaining energy, mission completion progress, and communication channel status. In this embodiment, the cloud state space is represented as a vector.

[0112] The cloud action space is used to represent the behavioral instructions that the cloud policy model can output in the current state, such as path adjustment, task reallocation, priority switching, and global coordination instruction issuance. The cloud action space is usually a discrete set or a continuous control variable.

[0113] S3-A3-2. Define an immediate reward for a cloud task target; the immediate reward for the cloud task target is based on a difference representation between state features of adjacent time steps in a cloud state space.

[0114] That is, the difference between two state features (such as path position, task progress, collision event, etc.) that have adjacent temporal relationships in the cloud state space is usually selected as the calculation basis.

[0115] For example, the immediate reward of the cloud task goal can be calculated based on the effect of the action performed by each cloud node in the current state, such as the amount of distance shortened or the number of collisions avoided.

[0116] Therefore, the calculation formula for the immediate reward of the cloud task target can be:

[0117] ;

[0118] in, represents the immediate reward of the cloud task target, represents the amount of reduction in the distance the robot moves at time step t (positive reward); Indicates whether a collision occurs at time step t (negative reward), which can usually be defined as a binary variable (0 or 1); and is the weight coefficient, which is used to adjust the impact of different factors on immediate rewards.

[0119] S3-A3-3. Calculate the long-term cumulative reward of the cloud task target based on the immediate reward of the cloud task target;

[0120] The long-term cumulative reward of the cloud task goal is calculated by weighting the immediate rewards over multiple time steps and is used to evaluate the overall performance of the cloud task. Generally speaking, the weights over multiple time steps are gradually reduced in proportion to the future weights.

[0121] For example, the long-term cumulative reward of the cloud task goal is calculated as follows:

[0122] ;

[0123] in, represents the long-term cumulative reward of the cloud task goal starting from time step t, Indicates that at time step Instant rewards for cloud mission goals, is the discount factor, and its value range is usually [0, 1 The closer it is to 1, the higher the weight of future rewards; the closer it is to 0, the higher the weight of current rewards.

[0124] In this embodiment, the defined cloud state space covers key intermediate variables in task execution, such as path, energy consumption, and progress. The action space incorporates specific cloud-controllable behaviors, such as task allocation and coordinated scheduling, while meeting the executable constraints of the policy output. The immediate reward function, measured by state changes (e.g., path shortening, collision events), directly reflects the performance impact of actions on the target system. The long-term reward construct emphasizes the delayed effects of state responses on the timeline, facilitating cross-timestep reinforcement learning of the cloud policy model.

[0125] Exemplarily, the steps of constructing the local policy model include:

[0126] S3-B1: For the local node, obtain its node subtasks and subtask indicators;

[0127] S3-B2, using the subtask indicators as distributed task indicators of respective local nodes;

[0128] S3-B3. Define the optimization process of M distributed task indicators as distributed reinforcement learning under the reinforcement learning framework to construct M local strategy models.

[0129] In this embodiment, the local policy model focuses on the individual goals of each edge node and is independently trained using distributed reinforcement learning. Each local node uses only its own subtask metrics as the optimization target, without relying on the state or action information of other nodes. This makes the local policy model more suitable for tasks that are sensitive to response delays or have limited edge computing power.

[0130] Furthermore, the step S3-B3 specifically includes:

[0131] S3-B3-1. Define the distributed state space and distributed action space of M distributed task indicators

[0132] S3-B3-2. Define instant rewards for M distributed task indicators; the instant rewards for the distributed task indicators are based on the difference representation between state features of adjacent time steps in the distributed state space;

[0133] The immediate reward of distributed task indicators can be calculated based on the immediate benefits brought by the local node completing the local task in the current state, such as the number of effective subtasks completed per unit time, the energy consumption control level, the communication success rate with neighboring nodes, or the task completion rate.

[0134] S3-B3-3. Calculate the long-term cumulative rewards of distributed task indicators based on their immediate rewards.

[0135] In this embodiment, the state and action spaces of distributed reinforcement learning are confined to the local node to avoid cross-node dependencies. The design of the immediate reward for distributed task indicators emphasizes the efficiency of local task completion, and the indicators are set to align with real-time perceptible indicators such as task completion rate and communication success rate in edge scenarios. The long-term cumulative reward for distributed task indicators continues this design approach, measuring the cumulative benefits of local actions over a longer time scale.

[0136] Example 2: This example 2 differs from Example 1 in that it discloses a process for updating the cloud model parameters. Specifically, in this example, the step of the reinforcement learning unit updating the cloud model parameters includes:

[0137] S4-1. Obtain status characteristics of N cloud nodes in real time , action characteristics , next state characteristics and instant rewards ;

[0138] Exemplarily, the state characteristics may include multi-dimensional parameters such as robot position, speed, energy consumption, and target distance;

[0139] The action characteristics may include steering angle, speed adjustment value, path switching signal, etc.

[0140] The immediate reward is assigned based on the contribution of the current strategy action to the task indicators, such as path shortening, collision penalty, etc.

[0141] S4-2, the real-time state characteristics of each time step , action characteristics , instant rewards and next state features Constructed as experience tuple ;

[0142] S4-3. Real-time state features in the prediction network of the cloud strategy model Perform forward propagation and calculate action features The predicted Q value of ,in, represents the model parameters;

[0143] S4-4. Next state characteristics in the target network of the cloud strategy model Perform forward propagation and calculate the maximum Q value , and calculate the target Q value;

[0144] The target Q value is calculated as follows:

[0145] ;

[0146] in, represents the target Q value, represents the discount factor for the predicted Q value at each time step;

[0147] S4-5, calculate the mean square error loss function between the target Q value and the predicted Q value;

[0148] S4-6. Calculate the gradient of the mean square error loss function with the current parameters, and update the parameters of the cloud strategy model based on the gradient of the current parameters. .

[0149] Furthermore, in this embodiment, the steps for updating local model parameters differ in that the cloud-based policy model optimizes system-level task metrics, focusing on the global optimization effect of multi-node collaboration (e.g., minimizing overall path length, optimizing resource utilization, and balancing task scheduling). In contrast, the local policy model prioritizes node-level immediate task efficiency, placing greater emphasis on the real-time task performance and resource consumption control of individual local robots. Consequently, the structures of the corresponding immediate reward function and long-term objective function may differ.

[0150] Exemplarily, the fusion step of the parameter fusion unit in this embodiment includes:

[0151] S6-1. Collect the updated model parameters of N cloud nodes and M local nodes in the current training cycle respectively;

[0152] For example, the model parameters may include weight parameters of the policy network, Q-value function parameters or gradient vectors in the value network, etc.

[0153] S6-2. Unify the formats and align the timestamps of the collected cloud model parameters and local model parameters to generate aligned parameters;

[0154] Specifically, during the acquisition process, local model parameters and cloud model parameters can be assigned the same time index. This allows them to be matched with the same time index when timestamps are aligned. If some nodes experience communication delays or missing data, parameter compensation or backfilling from the previous cycle can be used to synchronize data.

[0155] S6-3. Normalize the aligned parameters to generate normalized local model parameters and normalized cloud model parameters;

[0156] Specifically, when normalizing each local model parameter or cloud model parameter, the following processing method can be adopted: the maximum and minimum values ​​of the historical training cycle are used as the normalization upper and lower limits.

[0157] S6-4, performing weighted fusion on the normalized local model parameters and the normalized cloud model parameters to generate unified global parameters;

[0158] For example, methods such as weighted averaging strategies and parameter fusion functions based on task importance can be used to ensure that the fusion results balance global optimality and local adaptability. For example, if the strategy success rate of a task executed by a local node is higher than the global average, its parameters can be given a higher fusion weight (such as 0.3), while nodes with large strategy fluctuations can have a lower weight (such as 0.1). The weight of cloud nodes can be set to 0.6 to enhance control stability.

[0159] For example, assuming there are 3 local nodes and 1 cloud node, with weights of [0.2, 0.2, 0.2, 0.4], the global parameters for each round can be expressed as: ;

[0160] in, Represents global parameters, , , Represent the number of local model adoptions, Represents cloud-based model parameters.

[0161] In this embodiment, global parameters combine the centralized optimization of cloud-based nodes with the individual policy corrections made by local nodes based on actual execution feedback, forming unified policy update parameters. This global parameter serves as the basis for synchronizing node models, maintaining the consistency of the target system's policies while preserving the local adaptability of each node. This allows the target system to balance global efficiency and local execution when collaborating on complex tasks.

[0162] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (e.g., infrared, wireless, microwave, etc.).

[0163] The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that contains one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A cloud-edge collaborative robot cluster simulation training and optimization system, characterized by: include: A node marking unit, used to mark K edge nodes of the target system; A node classification unit, used to classify K edge nodes into N cloud nodes and M local nodes; The classification step of the node classification unit includes: S2-1. Obtain the pre-assigned node indexes of K edge nodes; S2-2. Match the node subtask of the edge node in the target system according to the node index S2-3, calculate the task complexity of the node subtask; S2-4, comparing the task complexity of the node subtask with a preset threshold to classify the K edge nodes into N cloud nodes and M local nodes; The task complexity of the computing node subtask includes: S2-3-1. Obtain the number of robot terminals connected to the edge node, bandwidth utilization, resource utilization, and average task queue length; S2-3-2. Perform a normalized weighted sum of the number of robot terminals connected to the edge node, bandwidth utilization, resource utilization, and average task queue length to generate the task complexity of the edge node; A dual-model construction unit, used to respectively construct a cloud policy model for the cloud node and a local policy model for the local node; The steps of constructing the cloud policy model include: S3-A1, obtain node subtasks and subtask indicators of cloud nodes; S3-A2, aggregating the subtask indicators to generate a cloud-based task target representing the system-level target; S3-A3, define the optimization process of cloud task objectives as centralized reinforcement learning under the reinforcement learning framework to build a cloud strategy model; The step of aggregating the subtask indicators to generate a cloud-based task target representing a system-level target includes: S3-A2-1. Obtain several target features for each subtask indicator; S3-A2-2. Standardize a number of target features and combine the standardized target features into a task indicator vector representing the subtask indicator; S3-A2-3. Concatenate the task indicator vectors corresponding to the subtask indicators to generate the cloud task target; A reinforcement learning unit is used to build a reinforcement learning framework and update the model parameters of the cloud policy model and the local policy model based on the reinforcement learning framework to form real-time cloud model parameters and local model parameters; Periodic parameter collection unit, used to periodically collect real-time cloud model parameters and local model parameters; The parameter fusion unit is used to unify the format and timestamp of the cloud model parameters and local model parameters, and fuse them into global parameters; The global model application unit is used to construct the global parameters into a global strategy model under the reinforcement learning framework for use in the simulation optimization training of the target system.

2. A cloud-edge collaborative robot cluster simulation training and optimization system according to claim 1, characterized in that: The optimization process of the cloud task goal is defined as centralized reinforcement learning under the reinforcement learning framework, including: S3-A3-1. Define the cloud state space and cloud action space of the cloud task goal; Among them, the cloud state space is used to represent the environmental state of the target system during the training process of the cloud task target, which includes multi-dimensional state features; S3-A3-2. Define an immediate reward for a cloud task target; the immediate reward for the cloud task target is based on a difference representation between state features of adjacent time steps in the cloud state space; S3-A3-3. Calculate the long-term cumulative reward of the cloud task target based on the immediate reward of the cloud task target.

3. The cloud-edge collaborative robot cluster simulation training and optimization system according to claim 1 is characterized in that: The steps of constructing the local policy model include: S3-B1: For the local node, obtain its node subtasks and subtask indicators; S3-B2, using the subtask indicators as distributed task indicators of respective local nodes; S3-B3. Define the optimization process of M distributed task indicators as distributed reinforcement learning under the reinforcement learning framework to construct M local strategy models.

4. A cloud-edge collaborative robot cluster simulation training and optimization system according to claim 3, characterized in that: The optimization process of M distributed task indicators is defined as distributed reinforcement learning under the reinforcement learning framework, including: S3-B3-1. Define the distributed state space and distributed action space of M distributed task indicators S3-B3-2. Define instant rewards for M distributed task indicators; the instant rewards for the distributed task indicators are based on the difference representation between state features of adjacent time steps in the distributed state space; S3-B3-3. Calculate the long-term cumulative rewards of distributed task indicators based on their immediate rewards.

5. The cloud-edge collaborative robot cluster simulation training and optimization system according to claim 1 is characterized in that: The step of the reinforcement learning unit updating the cloud model parameters includes: S4-1. Obtain status characteristics of N cloud nodes in real time , action characteristics , next state characteristics and instant rewards ; S4-2, the real-time state characteristics of each time step , action characteristics , instant rewards and next state features Constructed as experience tuple ; S4-3. Real-time state features in the prediction network of the cloud strategy model Perform forward propagation and calculate action features The predicted Q value of ;in, represents the model parameters; S4-4. Next state characteristics in the target network of the cloud strategy model Perform forward propagation and calculate the maximum Q value , and calculate the target Q value; The target Q value is calculated as follows: ; in, represents the target Q value, represents the discount factor for the predicted Q value at each time step; S4-5, calculate the mean square error loss function between the target Q value and the predicted Q value; S4-6. Calculate the gradient of the mean square error loss function with the current parameters, and update the parameters of the cloud strategy model based on the gradient of the current parameters. .

6. The cloud-edge collaborative robot cluster simulation training and optimization system according to claim 1 is characterized in that: The fusion step of the parameter fusion unit includes: S6-1. Collect the updated model parameters of N cloud nodes and M local nodes in the current training cycle respectively; S6-2. Unify the formats and align the timestamps of the collected cloud model parameters and local model parameters to generate aligned parameters; S6-3. Normalize the aligned parameters to generate normalized local model parameters and normalized cloud model parameters; S6-4. Perform weighted fusion on the normalized local model parameters and the normalized cloud model parameters to generate unified global parameters.

Images