Robot control method, device and robot control system based on metaverse

Abstract

The application discloses a robot control method and device based on a meta universe, a robot control system, a storage medium and a computer program product, and the method comprises the following steps: collecting characteristic data of a physical environment and a robot cluster, and constructing a meta universe virtual environment based on the characteristic data; establishing a virtual-real bidirectional communication link between the meta universe virtual environment and the robot cluster; synchronizing a real-time state of the robot cluster and a working environment to the meta universe virtual environment, performing collaborative strategy deduction in the meta universe virtual environment, and obtaining a target control strategy; and generating a control instruction according to the target control strategy and delivering the control instruction to the robot cluster for execution. According to the scheme, the safety and working efficiency of the robot cluster are improved.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, specifically relating to a robot control method, device, robot control system, storage medium, and computer program product based on the metaverse. Background Technology

[0002] Multi-robot collaborative operations have been widely applied in scenarios such as logistics warehousing, flexible manufacturing, and hazardous environment detection. Currently, the mainstream robot collaborative control solutions in the industry rely on traditional communication networks combined with fixed rule bases to achieve task allocation and path planning, while using digital twin technology to complete status display.

[0003] While existing solutions can achieve basic robot swarm collaboration, they generally suffer from a disconnect between virtual and physical data interaction and control links. Specifically, there is a significant data delay between the physical robot and the digital virtual body, making it impossible to achieve millisecond-level real-time bidirectional mapping; virtual simulation is only used for offline analysis or post-event review, making it difficult to perform path pre-simulation and conflict prediction in advance; and the control system cannot adaptively adjust the control logic according to dynamic changes such as communication fluctuations and sudden environmental changes. Ultimately, this results in poor robot swarm collaboration flexibility, high collision risk, and low operational efficiency.

[0004] Therefore, how to connect the virtual and real links and build an integrated robot control mode that can be mapped in real time, simulated in advance, and dynamically adapted is a technical problem that urgently needs to be solved.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The purpose of this invention is to provide a robot control method, device, robot control system, storage medium, and computer program product based on the metaverse, in order to solve the problem that the virtual and real links are separated in the robot swarm collaborative control schemes of related solutions, which cannot achieve low-latency virtual and real synchronization, pre-deduction and dynamic adaptive control, resulting in insufficient safety and efficiency of swarm operation, and thus achieve the effect of improving the safety and operation efficiency of robot swarm operation.

[0007] This invention provides a robot control method based on a metaverse, comprising: collecting feature data of the physical environment and a robot cluster; constructing a metaverse virtual environment based on the feature data; establishing a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster; synchronizing the real-time status of the robot cluster and the working environment to the metaverse virtual environment; performing collaborative strategy deduction in the metaverse virtual environment to obtain a target control strategy; and generating control commands according to the target control strategy and issuing them to the robot cluster for execution.

[0008] In some implementations, establishing a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster includes: performing timing calibration processing on the transmitted data between the metaverse virtual environment and the robot cluster to synchronize the data between the metaverse virtual environment and the robot cluster.

[0009] In some implementations, performing collaborative strategy deduction in the metaverse virtual environment includes: generating at least two virtual motion paths and arranging and combining the at least two virtual motion paths to obtain a set of candidate motion paths; evaluating each set of candidate motion paths according to preset evaluation rules; and selecting the set of candidate motion paths that meets preset conditions as the target control strategy.

[0010] In some implementations, the method further includes: weighted fusion of the robot cluster's local preset control strategy with the target control strategy to obtain a hybrid control strategy; generating control commands according to the hybrid control strategy and sending them to the robot cluster for execution.

[0011] In some embodiments, the method further includes: real-time monitoring of the communication latency between the metaverse virtual environment and the robot cluster, as well as abnormal operating conditions in the physical environment; when the communication latency is greater than a preset threshold, and / or the abnormal operating conditions meet preset operating condition judgment conditions, reducing the weight of the target control strategy in the weighted fusion process, and increasing the weight of the preset control strategy.

[0012] In some implementations, the method further includes: training a collaborative control model based on reinforcement learning in the metaverse virtual environment, and deploying the trained collaborative control model to the edge for use by the robot cluster. In line with the above method, another aspect of the present invention provides a robot control device based on a metaverse, comprising: a construction unit configured to collect feature data of the physical environment and a robot cluster, and construct a metaverse virtual environment based on the feature data; a communication unit configured to establish a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster; a deduction unit configured to synchronize the real-time status of the robot cluster and the working environment to the metaverse virtual environment, and perform collaborative strategy deduction in the metaverse virtual environment to obtain a target control strategy; and an execution unit configured to generate control commands according to the target control strategy and issue them to the robot cluster for execution.

[0013] In some implementations, the communication unit establishes a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster, including: performing timing calibration processing on the transmitted data between the metaverse virtual environment and the robot cluster to synchronize the data between the metaverse virtual environment and the robot cluster.

[0014] In some implementations, the deduction unit performs collaborative strategy deduction in the metaverse virtual environment, including: generating at least two virtual motion paths and arranging and combining the at least two virtual motion paths to obtain a set of candidate motion paths; evaluating each set of candidate motion paths according to preset evaluation rules; and selecting the set of candidate motion paths that meets preset conditions as the target control strategy.

[0015] In some implementations, the execution unit is further configured to: weight and fuse the preset control strategy of the robot cluster with the target control strategy to obtain a hybrid control strategy; generate control commands according to the hybrid control strategy and send them to the robot cluster for execution.

[0016] In some implementations, the execution unit is further configured to: monitor in real time the communication delay between the metaverse virtual environment and the robot cluster, as well as abnormal operating conditions in the physical environment; when the communication delay is greater than a preset threshold, and / or the abnormal operating conditions meet preset operating condition judgment conditions, reduce the weight of the target control strategy in the weighted fusion process, and increase the weight of the preset control strategy.

[0017] In some implementations, the execution unit is further configured to: train a cooperative control model based on reinforcement learning in the metaverse virtual environment, and deploy the trained cooperative control model to the edge for use by the robot cluster. In conjunction with the aforementioned device, the present invention further provides a robot control system, comprising: the aforementioned metaverse-based robot control device.

[0018] In conjunction with the above method, the present invention further provides a storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes the above-described metaverse-based robot control method.

[0019] In conjunction with the above method, the present invention further provides a computer program product comprising a computer program that, when processed and executed, implements the steps of the above-described metaverse-based robot control method.

[0020] The present invention first collects characteristic data of the physical environment and robot swarm to construct a corresponding metaverse virtual environment. Then, it establishes a two-way virtual-physical communication link between the virtual environment and the robot swarm. Subsequently, the real-time status of the robot swarm and the working environment is synchronized to the virtual environment. Cooperative strategy deduction is performed in the virtual environment to obtain the target control strategy. Finally, the target control strategy is converted into control commands and issued to the robot swarm for execution. This achieves low-latency virtual-physical synchronization and pre-emptive strategy deduction, effectively solving the problems of fragmented virtual-physical links, lack of pre-emptive deduction, and dynamic control capabilities in traditional solutions. This improves the safety of robot swarm collaborative operations and also increases overall operational efficiency.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating an embodiment of the robot control method based on the metaverse of the present invention; Figure 2 This is a schematic diagram of an embodiment of the robot control device based on the metaverse of the present invention; Figure 3 This is a flowchart illustrating a robot digital collaborative control method based on a metaverse architecture. Figure 4 This is a flowchart illustrating the process of virtual simulation and collaborative decision-making. Figure 5 A flowchart illustrating the adaptive virtual-real fusion control process; Figure 6 This is a flowchart illustrating a decentralized collaborative control mechanism based on a consortium blockchain. Figure 7 A schematic diagram illustrating the self-evolution process of collaborative strategies combined with deep reinforcement learning.

[0024] Referring to the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows: 101-Construction unit; 102-Communication unit; 103-Deduction unit; 104-Execution unit. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] According to embodiments of the present invention, a metaverse-based robot control method is provided, applied to a collaborative system consisting of N mobile robots (N≥2), the system including a physical robot cluster, an edge computing gateway, a cloud server, and a user terminal (VR / AR device). Figure 1 The flowchart of an embodiment of the method of the present invention is shown. The robot control method based on the metaverse may include steps S110 to S140.

[0027] In step S110, feature data of the physical environment and robot cluster are collected, and a metaverse virtual environment is constructed based on the feature data.

[0028] A robot swarm refers to a group of multiple robots that cooperate to complete a task. The physical environment is the actual working space where the robot swarm operates, including physical elements such as the site, fixed facilities, and natural obstacles. Feature data characterizes the spatial form and layout of the physical environment, as well as information such as the robot swarm's shape, position, and operational status. The metaverse virtual environment is a high-fidelity digital virtual scene built upon real data from the physical world, enabling data interaction and simulation calculations with physical devices.

[0029] The metaverse virtual environment needs to be consistent with the real physical scene. Only by building it based on real feature data can the virtual scene accurately replicate the on-site situation, providing a reliable digital carrier for subsequent simulation and state synchronization. Specifically, after the operation starts, data collection is first carried out on the physical environment where the robot is located and the robot cluster on site. The collected feature data is integrated and processed, and then the corresponding metaverse virtual environment is built based on the integrated data.

[0030] In some embodiments, in addition to collecting data once to build a basic virtual environment, incremental feature data can be collected periodically to dynamically update the established metaverse virtual environment and continuously ensure the consistency between the virtual scene and the physical scene.

[0031] In step S120, a virtual-physical bidirectional communication link is established between the metaverse virtual environment and the robot cluster.

[0032] The virtual-physical bidirectional communication link is a data transmission channel connecting the metaverse virtual environment and the robot swarm, supporting the mutual transmission of data and commands between the two sides. Physical state data needs to be transmitted to the virtual environment for synchronization, and control commands generated by the virtual environment also need to be transmitted to the physical robots for execution. The bidirectional communication link ensures the normal flow of data and commands between the two ends. Specifically, after the metaverse virtual environment is built, a dedicated data transmission channel is established, allowing the metaverse virtual environment and the robot swarm in the physical environment to exchange data and commands.

[0033] In some embodiments, different types of communication methods can be selected to build links according to the network conditions of the work scenario. High-frequency communication mode is used in scenarios with good network conditions, while anti-interference communication mode is switched in complex interference scenarios.

[0034] In some implementations, step S120, establishing a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster, includes: performing timing calibration processing on the transmitted data between the metaverse virtual environment and the robot cluster to synchronize the data between the metaverse virtual environment and the robot cluster.

[0035] Specifically, during the data transmission between the metaverse virtual environment and the robot cluster, time-series calibration is performed on each piece of transmitted data. This process eliminates time deviations during data transmission, ultimately ensuring that the data in the metaverse virtual environment and the data on the robot cluster are consistent.

[0036] Timing calibration is a process that matches and corrects for time discrepancies in data sent and received by different nodes according to a unified time base. Data transmission in network links is affected by factors such as network jitter, transmission routing, and device response speed, leading to time misalignment and causing a disconnect between the virtual scene and the physical device's status display. Performing timing calibration unifies the time base of data at both ends, compensating for timing deviations introduced during transmission.

[0037] In step S130, the real-time status of the robot cluster and the working environment is synchronized to the metaverse virtual environment, and the collaborative strategy is deduced in the metaverse virtual environment to obtain the target control strategy.

[0038] Real-time status includes the robot swarm's current position, operating speed, working status, and dynamic information such as current scene changes and obstacle distribution in the operating environment. Collaborative strategy simulation is the process of simulating different cooperative modes of the robot swarm in a virtual scenario and selecting an operational plan suitable for the actual operation.

[0039] The physical scene is constantly changing, and real-time state synchronization allows the virtual scene to keep up with the changes on-site, avoiding discrepancies between the virtual and reality. Utilizing the virtual environment for strategy simulation allows for the prediction of the operational effects of different work plans before the robot actually takes action, proactively mitigating operational risks and optimizing work plans. Specifically, real-time dynamic information about the robot swarm and the working environment is continuously collected and transmitted to the metaverse virtual environment, allowing the virtual scene to update its state synchronously. Subsequently, multiple sets of work plan simulations are conducted within the updated virtual environment, and the final target control strategy is determined from these simulations.

[0040] In some embodiments, when conducting collaborative strategy simulations, the focus of the simulation can be adjusted according to the priority of the tasks, with priority given to simulating the corresponding task plans for high-priority tasks.

[0041] In some implementations, step S130 involves performing a collaborative strategy deduction in the metaverse virtual environment, including: generating at least two virtual motion paths and arranging and combining the at least two virtual motion paths to obtain a set of candidate motion paths; evaluating each set of candidate motion paths according to a preset evaluation rule; and selecting the set of candidate motion paths that meets the preset conditions as the target control strategy.

[0042] A virtual motion path is a simulated robot trajectory for actual movement and operation, planned within a metaverse virtual environment. The candidate motion path set is a set of alternative solutions for multi-robot collaborative operations, formed by combining multiple virtual motion paths.

[0043] Specifically, within the metaverse virtual environment, based on the current physical environment and the real-time status of the robot swarm, two or more independent virtual motion paths are generated. These multiple virtual motion paths are then arranged and combined in different ways to form multiple sets of candidate motion paths. Pre-defined evaluation rules are invoked to comprehensively evaluate each set of candidate motion paths, analyzing the performance of each scheme. Based on the analysis results, the optimal path combination that meets all preset conditions is selected from all candidate motion path sets and determined as the final target control strategy.

[0044] The virtual motion path generation rules are as follows: an initial path baseline is generated on a rasterized map using a preset algorithm; a time dimension is introduced to expand it into a spatiotemporal trajectory (x, y, t), with multiple trajectories not overlapping in the spatiotemporal domain; priority constraints are applied, with lower priority paths planned in the remaining spatiotemporal region after high-priority paths are fixed; and kinematic feasibility is verified (velocity / acceleration / turning radius). The final result is a virtual motion path that satisfies both collision-free and kinematic constraints.

[0045] The preset evaluation rule is: if a collision risk or inefficiency is detected in the virtual simulation, the candidate motion path set is removed and replanned. The process of identifying collision risks or inefficiencies is as follows: the algorithm calculates a dynamic safety corridor for each candidate path, which is the three-dimensional spatial safety area around the path that changes over time; during physical execution, LiDAR and depth cameras scan the environment, and the point cloud data is compared with the virtual pre-stored expected environment model. If a difference exceeding the safety threshold is detected, it is judged as a sudden anomaly, triggering weight adjustment; confidence is confirmed through multi-frame accumulation to avoid false alarms due to noise in a single frame. The corresponding preset evaluation rule can be switched according to different task types, focusing on safety indicators for high-risk operations and efficiency indicators for routine transportation operations.

[0046] The preset and optimal evaluation criteria are: zero collisions in virtual simulation; shortest task completion time; lowest energy consumption with a time difference of <10%; shortest total path length; and lowest load coefficient. When multiple candidate solutions simultaneously meet the preset conditions, the optimal solution can be selected according to priority, while the second-best solution is reserved as an emergency backup.

[0047] Figure 4 This is a flowchart illustrating the virtual simulation and collaborative decision-making process, which includes: First, acquiring real-time status data of the physical environment and the robot, and synchronizing it to the metaverse virtual scene to construct a digital mirror consistent with the real world; then setting simulation duration, safety constraints, and other conditions, and rapidly generating multiple candidate paths for robot collaborative movement through parallel simulation; subsequently, performing virtual collision detection and operational efficiency evaluation on these paths, and if the scheme does not meet the preset safety and optimal indicators, it is eliminated and the path is replanned; if the scheme meets the standards, it is determined as the optimal collaborative strategy, and then the control commands are sent to the edge gateway and the robot for execution; the robot continuously transmits its own status data during the operation, updating the metaverse virtual model.

[0048] By generating, combining, evaluating, and screening multiple paths, multiple collaborative schemes can be simulated and optimized in a virtual environment. This allows for the early detection of issues such as path conflicts and unreasonable schemes, effectively solving the shortcomings of traditional robot collaborative schemes, such as rigidity and susceptibility to operational anomalies on-site. It also improves the rationality and reliability of robot cluster collaborative operation schemes.

[0049] In step S140, control commands are generated according to the target control strategy and sent to the robot cluster for execution.

[0050] Specifically, based on the determined target control strategy, it is converted into control instructions that the robot swarm can recognize and execute. These control instructions are then sent to the robot swarm on site, and the robot swarm carries out the operation according to the instructions.

[0051] This invention builds a virtual scene based on real data and establishes a virtual-real communication channel. It uses the virtual environment to complete the preliminary strategy deduction and then transforms the mature solution into instructions to drive the robot operation. This effectively solves the problems of the traditional solution, such as the disconnect between the virtual and real scenes, the inability to predict the operation risk in advance, and the poor adaptability of the collaborative solution. It ensures the orderliness, safety and overall operation efficiency of the robot cluster collaborative operation.

[0052] In some implementations, the method further includes: weighted fusion of the robot cluster's local preset control strategy with the target control strategy to obtain a hybrid control strategy; generating control commands based on the hybrid control strategy and sending them to the robot cluster for execution.

[0053] Specifically, the preset control strategy stored locally in the robot cluster is retrieved, and the target control strategy derived from the virtual environment is combined with the target control strategy. The two strategies are then weighted and fused to generate a hybrid control strategy. The hybrid control strategy is then converted into control instructions that the robot cluster can recognize and execute. These control instructions are then sent to the robot cluster, which completes the operation according to the instructions.

[0054] Relying solely on virtual simulation-based target control strategies or using only locally preset control strategies is insufficient to adapt to complex and ever-changing on-site conditions. Weighted fusion of these two approaches can balance virtual global planning capabilities with local autonomous operation capabilities, thereby improving the adaptability of control scenarios.

[0055] In some embodiments, fusion modes can be pre-defined according to the type of job task, and fixed weight combinations can be configured for the two sets of control strategies under different modes; alternatively, the instruction formats of the two sets of strategies can be uniformly converted before fusion to ensure a smooth fusion process. Furthermore, the integrity of the generated control instructions can be verified, and the issuance operation can be performed only after confirming that the instructions are correct; alternatively, a batch issuance method can be adopted, pushing the corresponding instructions sequentially according to the job time sequence.

[0056] In some implementations, the method further includes: real-time monitoring of the communication latency between the metaverse virtual environment and the robot cluster, as well as abnormal operating conditions in the physical environment; when the communication latency is greater than a preset threshold, and / or the abnormal operating conditions meet preset operating condition judgment conditions, reducing the weight of the target control strategy in the weighted fusion process, and increasing the weight of the preset control strategy.

[0057] Specifically, throughout the entire process of the robot swarm performing its tasks, the system continuously collects and monitors the data transmission time between the metaverse virtual environment and the robot swarm, while simultaneously identifying various changes in working conditions within the physical working environment. When the monitored communication latency exceeds a pre-set threshold, or when abnormal working conditions in the physical environment meet predetermined judgment criteria, or when both situations occur simultaneously, the weight of the target control strategy is reduced during the strategy weighting and fusion phase, while the weight of the robot swarm's locally preset control strategy is increased. Abnormal working conditions refer to scenarios in the physical working environment that deviate from normal operating conditions, such as slippery ground causing slippage.

[0058] The specific process of adjusting the weights is as follows: if the communication delay is >50ms (normally 5-10ms), the virtual pre-simulation weight is linearly reduced from 90% to 30%; if more than 5 new obstacle points are detected and the virtual model does not cover them, the weight is reduced to 40%; if the odometry displacement deviates from the virtual pre-simulation by more than 5cm, the weight is reduced to 50%.

[0059] Figure 5 The flowchart illustrates the adaptive virtual-real fusion control process, which includes: When robots perform collaborative operations, the system continuously monitors communication latency and sudden changes in the physical environment in real time. If both the communication link and the physical environment are normal, the global strategy generated by the virtual pre-simulation of the metaverse takes the lead and is weighted and fused with the local control scheme to execute virtual-real fusion command control. If communication latency exceeds the standard or abnormal changes occur in the environment, the control weight of the virtual pre-simulation strategy is reduced, and the weight of the robot's local autonomous navigation strategy is increased, allowing the robot to rely more on local autonomous control for operation. Then, virtual-real fusion command control is executed, and the entire process forms a closed loop. The weight ratio is continuously and dynamically adjusted according to the working conditions, thereby ensuring operational efficiency while effectively responding to communication fluctuations and environmental changes, and improving the stability and safety of robot swarm operations.

[0060] Excessive communication latency can cause delays in strategies and instructions issued in the virtual environment, and sudden abnormal conditions on-site can render the target control strategy simulated globally unsuitable. Reducing the weight of virtual strategies and increasing the weight of local strategies allows robots to rely more on local autonomous strategies, reducing the impact of abnormal situations on the operation process and improving the stability and safety of robot clusters in various abnormal scenarios.

[0061] In some implementations, the method further includes: training a cooperative control model based on reinforcement learning in the metaverse virtual environment, and deploying the trained cooperative control model to the edge for use by the robot cluster.

[0062] Specifically, relying on the established metaverse virtual environment, diverse work scenarios and operating states are simulated. Reinforcement learning is used to repeatedly train the collaborative control model, continuously refining its computational logic. Once the collaborative control model has completed all training and meets usage standards, it is migrated and installed on edge devices at the work site. The robot cluster can then access and use this model at any time during operation. Edge devices deployed at the robot's work site can perform data processing and command scheduling locally, shortening data transmission distances.

[0063] The metaverse virtual environment can replicate massive amounts of real-world operational conditions without requiring physical equipment or on-site space, enabling safe and efficient model training. Reinforcement learning allows models to learn from experience during continuous simulations, constantly improving the rationality of robot swarm collaborative scheduling and path planning. Deploying models at the edge reduces latency caused by data transmission to remote platforms, ensuring that model outputs can be quickly applied to the robot swarm; calling models locally also improves control response speed and operational reliability.

[0064] Figure 7 To illustrate the process of self-evolution of collaborative strategies using deep reinforcement learning, this method includes: First, constructing a high-fidelity, large-scale training scenario based on the metaverse, and conducting deep reinforcement learning (DRL) training of robot swarms under the premise of no physical equipment risk; During the training process, the DRL agent learns and masters the optimal robot collaborative control strategy through continuous trial and error iteration; After training convergence, the trained neural network model is exported and deployed to the edge computing gateway at the work site to reduce the latency caused by cloud calls; During actual operation, the robot swarm directly calls the model from the edge to achieve low-latency real-time collaborative control; Simultaneously, the operational data generated by the robots performing operations on-site flows back to the cloud, and the cloud continuously iterates and optimizes the model based on this real data, ultimately forming a closed loop of "virtual simulation training - edge real-time control - real data feedback - continuous model optimization," which improves the intelligence level and response speed of robot swarm collaborative control while ensuring the long-term adaptability and operational stability of the model.

[0065] In some implementations, the method further includes: visually monitoring the metaverse virtual environment and the operational status of the robot swarm using VR or AR devices.

[0066] Throughout the operation of the robot swarm, staff can use VR or AR devices to view the scene in the metaverse virtual environment in real time, and intuitively grasp the various operational statuses of the robot swarm. This achieves immersive visual monitoring, allowing staff to have a three-dimensional and intuitive understanding of the real-time situation of the metaverse virtual environment and the robot swarm, improving the convenience and comprehensiveness of on-site supervision.

[0067] Figure 3 This is a flowchart illustrating a robot digital collaborative control method based on a metaverse architecture. The method specifically includes steps 1 to 5.

[0068] Step 1: Constructing the Metaverse Virtual Mapping Environment: First, LiDAR, depth cameras, and inertial measurement units (IMUs) are used to collect 3D point cloud data of the physical environment. Combined with historical map data, a high-fidelity 3D virtual scene model is constructed on a cloud server. This model not only includes static environmental features (such as walls and shelves) but also maps the physical robot's geometric dimensions, joint angles, and current pose in real time. To reduce latency, an edge computing architecture is adopted, offloading some rendering and data processing tasks to edge computing gateways deployed on-site in the factory. Only key status data and final control commands are uploaded to the cloud or sent to the terminal.

[0069] Step 2: Establish a real-time bidirectional communication channel between the virtual and physical worlds: The physical robot cluster sends raw sensor data and its own status information (position, speed, battery level, etc.) to the edge computing gateway at a high frequency (e.g., 100Hz-500Hz) via a 5G or industrial Wi-Fi 6 network. The edge gateway cleans, aligns, and timestamps this data, driving the corresponding virtual robot model in the virtual scene to update its motion in real time. Conversely, when a user issues control commands in the VR terminal or completes path planning in the virtual environment, the commands are encrypted and transmitted to the edge gateway, then distributed to the corresponding physical robots for execution. During this process, a timestamp compensation algorithm is introduced to eliminate timing deviations caused by network jitter, ensuring strict synchronization between virtual operations and physical actions. Step 3, Generation of Collaborative Strategies Based on Virtual Simulation: When the system detects that multiple robots need to collaborate to complete a task (such as jointly moving a large object), it does not immediately execute physical actions, but first performs a pre-simulation in the metaverse virtual space. The system maps the current physical state to a virtual initial state and uses parallel computing technology to simulate all possible motion trajectories within the next T seconds in the virtual environment. Specifically, the system simulates various possible path combinations, speed changes, and sudden obstacle situations. If a path is detected to have a collision risk or low efficiency in the virtual simulation, the algorithm will automatically eliminate the scheme and replan. Only when the virtual simulation confirms that a collaborative strategy is within a safety threshold and has optimal efficiency will the strategy be converted into a specific sequence of control commands.

[0070] Step 4, Dynamic Weight Adjustment and Closed-Loop Execution: During physical execution, the system monitors the communication link quality and changes in the physical environment in real time. If a sudden anomaly occurs in the physical environment that is not covered by the virtual model, or if the communication delay exceeds a set threshold, the system will automatically switch control modes: reducing the weight of dependence on the virtual pre-simulation results and increasing the weight of local autonomous navigation algorithms (such as SLAM local replanning), forming a hybrid control mode of "virtual-real fusion". Once the anomaly is eliminated, the system automatically switches back to the control mode dominated by virtual pre-simulation.

[0071] Step 5, Visual Monitoring and Feedback: The user's VR / AR terminal receives rendered images from the virtual environment in real time. The operator can observe the robot swarm's operation process from a first-person perspective, and even drag and drop robots directly in the virtual space to adjust task points using gestures. The system will overlay key data from the physical execution process (such as torque and temperature) onto the virtual interface.

[0072] In some implementations, the method further includes: combining blockchain to complete data verification and data storage during the collaborative strategy deduction process.

[0073] Figure 6 This is a flowchart illustrating a decentralized collaborative control mechanism based on a consortium blockchain. Specifically, it includes: first, generating a robot collaborative control strategy in the metaverse virtual space; then submitting the strategy to a smart contract for rule verification, with consensus verification conducted by each node of the consortium blockchain; if the verification fails, the strategy is regenerated; if the verification passes, the control instructions are signed and stored on the blockchain, and then the encrypted instructions are broadcast to the physical robot network; after receiving the instructions, the robot executes the task and uploads the execution results to the blockchain, achieving end-to-end data storage from strategy generation, verification, distribution to execution. Ultimately, this achieves the goal of full traceability and accountability for all operations, thereby ensuring the authenticity, integrity, and security of the robot collaborative control instructions.

[0074] Throughout the entire phase of collaborative strategy simulation within the metaverse virtual environment, blockchain technology is used simultaneously to verify all types of data generated during the simulation, and all relevant data is uniformly stored and preserved. The data generated by the collaborative strategy simulation directly impacts the operational plans of the robot swarm; data distortion or tampering will cause operational anomalies. Blockchain, with its tamper-proof and distributed recording characteristics, can both verify data validity in real time and permanently retain data records, ensuring the authenticity and reliability of the simulation data.

[0075] The technical solution of this embodiment collects feature data of the physical environment and robot cluster to construct a metaverse virtual environment, establishes a virtual-physical two-way communication link and synchronizes real-time status, and then deduces the target control strategy in the virtual environment and sends it to the robot for execution. This opens up the virtual-physical interaction channel, realizes low-latency virtual-physical synchronization and pre-deduction of strategy, solves the problems of virtual-physical link separation and lack of pre-deduction, and improves the safety and efficiency of robot cluster operation. According to embodiments of the present invention, a metaverse-based robot control device corresponding to the metaverse-based robot control method is also provided. See also Figure 2 The schematic diagram of an embodiment of the device of the present invention shown herein may include: a construction unit 101, a communication unit 102, a deduction unit 103, and an execution unit 104.

[0076] The construction unit 101 is configured to collect feature data of the physical environment and robot cluster, and construct a metaverse virtual environment based on the feature data.

[0077] A robot swarm refers to a group of multiple robots that cooperate to complete a task. The physical environment is the actual working space where the robot swarm operates, including physical elements such as the site, fixed facilities, and natural obstacles. Feature data characterizes the spatial form and layout of the physical environment, as well as information such as the robot swarm's shape, position, and operational status. The metaverse virtual environment is a high-fidelity digital virtual scene built upon real data from the physical world, enabling data interaction and simulation calculations with physical devices.

[0078] The metaverse virtual environment needs to be consistent with the real physical scene. Only by building it based on real feature data can the virtual scene accurately replicate the on-site situation, providing a reliable digital carrier for subsequent simulation and state synchronization. Specifically, after the operation starts, data collection is first carried out on the physical environment where the robot is located and the robot cluster on site. The collected feature data is integrated and processed, and then the corresponding metaverse virtual environment is built based on the integrated data.

[0079] Communication unit 102 is configured to establish a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster.

[0080] The virtual-physical bidirectional communication link is a data transmission channel connecting the metaverse virtual environment and the robot swarm, supporting the mutual transmission of data and commands between the two sides. Physical state data needs to be transmitted to the virtual environment for synchronization, and control commands generated by the virtual environment also need to be transmitted to the physical robots for execution. The bidirectional communication link ensures the normal flow of data and commands between the two ends. Specifically, after the metaverse virtual environment is built, a dedicated data transmission channel is established, allowing the metaverse virtual environment and the robot swarm in the physical environment to exchange data and commands.

[0081] In some implementations, the communication unit 102 establishes a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster, including: performing timing calibration processing on the transmitted data between the metaverse virtual environment and the robot cluster to synchronize the data between the metaverse virtual environment and the robot cluster.

[0082] Specifically, during the data transmission between the metaverse virtual environment and the robot cluster, time-series calibration is performed on each piece of transmitted data. This process eliminates time deviations during data transmission, ultimately ensuring that the data in the metaverse virtual environment and the data on the robot cluster are consistent.

[0083] Timing calibration is a process that matches and corrects for time discrepancies in data sent and received by different nodes according to a unified time base. Data transmission in network links is affected by factors such as network jitter, transmission routing, and device response speed, leading to time misalignment and causing a disconnect between the virtual scene and the physical device's status display. Performing timing calibration unifies the time base of data at both ends, compensating for timing deviations introduced during transmission.

[0084] The deduction unit 103 is configured to synchronize the real-time status of the robot cluster and the working environment to the metaverse virtual environment, and to perform collaborative strategy deduction in the metaverse virtual environment to obtain the target control strategy.

[0085] Real-time status includes the robot swarm's current position, operating speed, working status, and dynamic information such as current scene changes and obstacle distribution in the operating environment. Collaborative strategy simulation is the process of simulating different cooperative modes of the robot swarm in a virtual scenario and selecting an operational plan suitable for the actual operation.

[0086] The physical scene is constantly changing, and real-time state synchronization allows the virtual scene to keep up with the changes on-site, avoiding discrepancies between the virtual and reality. Utilizing the virtual environment for strategy simulation allows for the prediction of the operational effects of different work plans before the robot actually takes action, proactively mitigating operational risks and optimizing work plans. Specifically, real-time dynamic information about the robot swarm and the working environment is continuously collected and transmitted to the metaverse virtual environment, allowing the virtual scene to update its state synchronously. Subsequently, multiple sets of work plan simulations are conducted within the updated virtual environment, and the final target control strategy is determined from these simulations.

[0087] In some implementations, the deduction unit 103 performs collaborative strategy deduction in the metaverse virtual environment, including: generating at least two virtual motion paths and arranging and combining the at least two virtual motion paths to obtain a set of candidate motion paths; evaluating each set of candidate motion paths according to preset evaluation rules; and selecting the set of candidate motion paths that meets preset conditions as the target control strategy.

[0088] A virtual motion path is a simulated robot trajectory for actual movement and operation, planned within a metaverse virtual environment. The candidate motion path set is a set of alternative solutions for multi-robot collaborative operations, formed by combining multiple virtual motion paths.

[0089] Specifically, within the metaverse virtual environment, based on the current physical environment and the real-time status of the robot swarm, two or more independent virtual motion paths are generated. These multiple virtual motion paths are then arranged and combined in different ways to form multiple sets of candidate motion paths. Pre-defined evaluation rules are invoked to comprehensively evaluate each set of candidate motion paths, analyzing the performance of each scheme. Based on the analysis results, the optimal path combination that meets all preset conditions is selected from all candidate motion path sets and determined as the final target control strategy.

[0090] The virtual motion path generation rules are as follows: an initial path baseline is generated on a rasterized map using a preset algorithm; a time dimension is introduced to expand it into a spatiotemporal trajectory (x, y, t), with multiple trajectories not overlapping in the spatiotemporal domain; priority constraints are applied, with lower priority paths planned in the remaining spatiotemporal region after high-priority paths are fixed; and kinematic feasibility is verified (velocity / acceleration / turning radius). The final result is a virtual motion path that satisfies both collision-free and kinematic constraints.

[0091] The preset evaluation rule is: if a collision risk or inefficiency is detected in the virtual simulation, the candidate motion path set is removed and replanned. The process of identifying collision risks or inefficiencies is as follows: the algorithm calculates a dynamic safety corridor for each candidate path, which is the three-dimensional spatial safety area around the path that changes over time; during physical execution, LiDAR and depth cameras scan the environment, and the point cloud data is compared with the virtual pre-stored expected environment model. If a difference exceeding the safety threshold is detected, it is judged as a sudden anomaly, triggering weight adjustment; confidence is confirmed through multi-frame accumulation to avoid false alarms due to noise in a single frame. The corresponding preset evaluation rule can be switched according to different task types, focusing on safety indicators for high-risk operations and efficiency indicators for routine transportation operations.

[0092] The preset and optimal evaluation criteria are: zero collisions in virtual simulation; shortest task completion time; lowest energy consumption with a time difference of <10%; shortest total path length; and lowest load coefficient. When multiple candidate solutions simultaneously meet the preset conditions, the optimal solution can be selected according to priority, while the second-best solution is reserved as an emergency backup.

[0093] By generating, combining, evaluating, and screening multiple paths, multiple collaborative schemes can be simulated and optimized in a virtual environment. This allows for the early detection of issues such as path conflicts and unreasonable schemes, effectively solving the shortcomings of traditional robot collaborative schemes, such as rigidity and susceptibility to operational anomalies on-site. It also improves the rationality and reliability of robot cluster collaborative operation schemes.

[0094] The execution unit 104 is configured to generate control commands according to the target control strategy and send them to the robot cluster for execution.

[0095] Specifically, based on the determined target control strategy, it is converted into control instructions that the robot swarm can recognize and execute. These control instructions are then sent to the robot swarm on site, and the robot swarm carries out the operation according to the instructions.

[0096] This invention builds a virtual scene based on real data and establishes a virtual-real communication channel. It uses the virtual environment to complete the preliminary strategy deduction and then transforms the mature solution into instructions to drive the robot operation. This effectively solves the problems of the traditional solution, such as the disconnect between the virtual and real scenes, the inability to predict the operation risk in advance, and the poor adaptability of the collaborative solution. It ensures the orderliness, safety and overall operation efficiency of the robot cluster collaborative operation.

[0097] In some implementations, the execution unit 104 is further configured to: weightedly fuse the preset control strategy of the robot cluster with the target control strategy to obtain a hybrid control strategy; generate control instructions according to the hybrid control strategy and send them to the robot cluster for execution.

[0098] Specifically, the preset control strategy stored locally in the robot cluster is retrieved, and the target control strategy derived from the virtual environment is combined with the target control strategy. The two strategies are then weighted and fused to generate a hybrid control strategy. The hybrid control strategy is then converted into control instructions that the robot cluster can recognize and execute. These control instructions are then sent to the robot cluster, which completes the operation according to the instructions.

[0099] Relying solely on virtual simulation-based target control strategies or using only locally preset control strategies is insufficient to adapt to complex and ever-changing on-site conditions. Weighted fusion of these two approaches can balance virtual global planning capabilities with local autonomous operation capabilities, thereby improving the adaptability of control scenarios.

[0100] In some embodiments, fusion modes can be pre-defined according to the type of job task, and fixed weight combinations can be configured for the two sets of control strategies under different modes; alternatively, the instruction formats of the two sets of strategies can be uniformly converted before fusion to ensure a smooth fusion process. Furthermore, the integrity of the generated control instructions can be verified, and the issuance operation can be performed only after confirming that the instructions are correct; alternatively, a batch issuance method can be adopted, pushing the corresponding instructions sequentially according to the job time sequence.

[0101] In some implementations, the execution unit 104 is further configured to: monitor in real time the communication delay between the metaverse virtual environment and the robot cluster, as well as abnormal operating conditions in the physical environment; when the communication delay is greater than a preset threshold, and / or the abnormal operating conditions meet preset operating condition judgment conditions, reduce the weight of the target control strategy in the weighted fusion process, and increase the weight of the preset control strategy.

[0102] Specifically, throughout the entire process of the robot swarm performing its tasks, the system continuously collects and monitors the data transmission time between the metaverse virtual environment and the robot swarm, while simultaneously identifying various changes in working conditions within the physical working environment. When the monitored communication latency exceeds a pre-set threshold, or when abnormal working conditions in the physical environment meet predetermined judgment criteria, or when both situations occur simultaneously, the weight of the target control strategy is reduced during the strategy weighting and fusion phase, while the weight of the robot swarm's locally preset control strategy is increased. Abnormal working conditions refer to scenarios in the physical working environment that deviate from normal operating conditions, such as slippery ground causing slippage.

[0103] The specific process of adjusting the weights is as follows: if the communication delay is >50ms (normally 5-10ms), the virtual pre-simulation weight is linearly reduced from 90% to 30%; if more than 5 new obstacle points are detected and the virtual model does not cover them, the weight is reduced to 40%; if the odometry displacement deviates from the virtual pre-simulation by more than 5cm, the weight is reduced to 50%.

[0104] Excessive communication latency can cause delays in strategies and instructions issued in the virtual environment, and sudden abnormal conditions on-site can render the target control strategy simulated globally unsuitable. Reducing the weight of virtual strategies and increasing the weight of local strategies allows robots to rely more on local autonomous strategies, reducing the impact of abnormal situations on the operation process and improving the stability and safety of robot clusters in various abnormal scenarios.

[0105] In some implementations, the execution unit 104 is further configured to: train a cooperative control model based on reinforcement learning in the metaverse virtual environment, and deploy the trained cooperative control model to the edge for use by the robot cluster.

[0106] Specifically, relying on the established metaverse virtual environment, diverse work scenarios and operating states are simulated. Reinforcement learning is used to repeatedly train the collaborative control model, continuously refining its computational logic. Once the collaborative control model has completed all training and meets usage standards, it is migrated and installed on edge devices at the work site. The robot cluster can then access and use this model at any time during operation. Edge devices deployed at the robot's work site can perform data processing and command scheduling locally, shortening data transmission distances.

[0107] The metaverse virtual environment can replicate massive amounts of real-world operational conditions without requiring physical equipment or on-site space, enabling safe and efficient model training. Reinforcement learning allows models to learn from experience during continuous simulations, constantly improving the rationality of robot swarm collaborative scheduling and path planning. Deploying models at the edge reduces latency caused by data transmission to remote platforms, ensuring that model outputs can be quickly applied to the robot swarm; calling models locally also improves control response speed and operational reliability.

[0108] In some implementations, the execution unit 104 is also configured to: perform visual monitoring of the metaverse virtual environment and the robot cluster operation status via VR or AR devices.

[0109] Throughout the operation of the robot swarm, staff can use VR or AR devices to view the scene in the metaverse virtual environment in real time, and intuitively grasp the various operational statuses of the robot swarm. This achieves immersive visual monitoring, allowing staff to have a three-dimensional and intuitive understanding of the real-time situation of the metaverse virtual environment and the robot swarm, improving the convenience and comprehensiveness of on-site supervision.

[0110] In some implementations, the execution unit 104 is also configured to: combine blockchain to complete data verification and data storage during the collaborative strategy deduction process.

[0111] Throughout the entire phase of collaborative strategy simulation within the metaverse virtual environment, blockchain technology is used simultaneously to verify all types of data generated during the simulation, and all relevant data is uniformly stored and preserved. The data generated by the collaborative strategy simulation directly impacts the operational plans of the robot swarm; data distortion or tampering will cause operational anomalies. Blockchain, with its tamper-proof and distributed recording characteristics, can both verify data validity in real time and permanently retain data records, ensuring the authenticity and reliability of the simulation data.

[0112] Since the processing and functions implemented by the device in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0113] By adopting the technical solution of this invention, a metaverse virtual environment is built based on real scene data, a two-way communication channel is established with the robot cluster and dynamic information is synchronized, and the optimal collaborative control strategy is generated through virtual simulation, which is then converted into instructions to drive robot operation. This effectively makes up for the shortcomings of traditional collaborative control, such as the disconnect between virtual and real and the lack of prior simulation, enhances the safety of cluster operation, and improves the overall operating efficiency. According to embodiments of the present invention, a robot control system corresponding to a metaverse-based robot control device is also provided. This robot control system may include the metaverse-based robot control device described above.

[0114] Since the processing and functions implemented by the robot control system in this embodiment are basically the same as those in the embodiments, principles and examples of the aforementioned device, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0115] By adopting the technical solution of this invention, data is collected from the physical environment and robot clusters to construct a metaverse virtual scene, establishes a virtual-physical data transmission link and synchronizes real-time status, completes collaborative strategy deduction in the virtual environment, and then issues the determined plan to the robots for execution. This solves the problems of virtual-physical link separation, inability to achieve low-latency synchronization and pre-planning, and improves the stability and efficiency of robot cluster operations. According to an embodiment of the present invention, a storage medium corresponding to the metaverse-based robot control method is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the metaverse-based robot control method described above during runtime.

[0116] Since the processing and functions implemented by the storage medium in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0117] By adopting the technical solution of this invention, a metaverse virtual environment is built based on the feature data collected on site, a two-way communication link with the robot cluster is established, the real-time dynamics of the equipment and the environment are synchronized, the target control strategy is deduced in the virtual scene, and then converted into instructions to drive the robot to operate. This eliminates the defects of the virtual-real link being disconnected and the lack of prior risk prediction, improves the safety of cluster operation, and optimizes the overall operation efficiency. According to an embodiment of the present invention, a computer program product corresponding to the metaverse-based robot control method is also provided. The computer program product includes a computer program that, when processed and executed, implements the steps of the metaverse-based robot control method described above.

[0118] Since the processing and functions implemented by the computer program product in this embodiment are basically corresponding to the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0119] By adopting the technical solution of this invention, the physical environment and robot state data are collected to construct a metaverse virtual scene, a virtual-real two-way communication channel is built and the state is synchronized, the optimal collaborative solution is generated in the virtual environment, and then instructions are issued to the robot cluster for execution. This solves the problems of virtual-real separation, lack of pre-planning and dynamic adaptation capabilities in traditional collaborative control, and improves the reliability and operating efficiency of robot cluster operation.

[0120] In summary, it is readily understood by those skilled in the art that, without conflict, the aforementioned advantageous methods can be freely combined and superimposed.

[0121] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A robot control method based on the metaverse, characterized in that, The method includes: Collect feature data of the physical environment and robot swarm, and construct a metaverse virtual environment based on the feature data; Establish a virtual-physical bidirectional communication link between the aforementioned metaverse virtual environment and the robot cluster; The real-time status of the robot cluster and its operating environment is synchronized to the metaverse virtual environment, and a collaborative strategy is deduced in the metaverse virtual environment to obtain the target control strategy. Control commands are generated based on the target control strategy and sent to the robot cluster for execution.

2. The robot control method based on the metaverse according to claim 1, characterized in that, Establishing a virtual-physical bidirectional communication link between the aforementioned metaverse virtual environment and the robot swarm includes: The data transmitted between the metaverse virtual environment and the robot cluster is time-series calibrated to synchronize the data between the metaverse virtual environment and the robot cluster.

3. The robot control method based on the metaverse according to claim 1, characterized in that, Performing collaborative strategy deduction within the aforementioned metaverse virtual environment includes: Generate at least two virtual motion paths, and arrange and combine the at least two virtual motion paths to obtain a set of candidate motion paths; The candidate motion path sets of each group are evaluated according to the preset evaluation rules; The set of candidate motion paths that meet the preset conditions is selected as the target control strategy.

4. The robot control method based on the metaverse according to any one of claims 1 to 3, characterized in that, The method further includes: The robot cluster's local preset control strategy is weighted and fused with the target control strategy to obtain a hybrid control strategy; Control commands are generated based on the hybrid control strategy and sent to the robot cluster for execution.

5. The robot control method based on the metaverse according to claim 4, characterized in that, The method further includes: Real-time monitoring of communication latency between the metaverse virtual environment and the robot cluster, as well as abnormal operating conditions in the physical environment; When the communication delay exceeds a preset threshold, and / or the abnormal operating condition meets the preset operating condition judgment conditions, the weight of the target control strategy in the weighted fusion process is reduced, and the weight of the preset control strategy is increased.

6. The robot control method based on the metaverse according to any one of claims 1 to 3, characterized in that, The method further includes: In the metaverse virtual environment, a collaborative control model is trained based on reinforcement learning, and the trained collaborative control model is deployed to the edge for the robot cluster to use.

7. A robot control device based on a metaverse, characterized in that, include: The construction unit is configured to collect feature data of the physical environment and robot clusters, and construct a metaverse virtual environment based on the feature data. The communication unit is configured to establish a virtual-physical bidirectional communication link between the metaverse virtual environment and the robot cluster; The simulation unit is configured to synchronize the real-time status of the robot cluster and the working environment to the metaverse virtual environment, and to perform collaborative strategy simulation in the metaverse virtual environment to obtain the target control strategy. The execution unit is configured to generate control commands according to the target control strategy and send them to the robot cluster for execution.

8. A robot control system, characterized in that, include: The robot control device based on the metaverse as described in claim 7.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the device containing the storage medium is controlled to perform the robot control method based on the metaverse as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot control method based on the metaverse as described in any one of claims 1 to 6.

Images