Digital twinning-based heterogeneous body-equipped intelligent equipment collaborative management system and digital twinning-based heterogeneous body-equipped intelligent equipment collaborative management method

Abstract

The invention discloses a digital twinning-based heterogeneous body-equipped intelligent equipment collaborative management system and method, and belongs to the technical field of digital twinning and body-equipped intelligent systems. The system comprises a digital twin platform, a communication module, a task scheduling module, a cooperative control module and an edge computing node. The digital twinborn platform constructs a high-fidelity three-dimensional model of a physical environment and digital twinborn bodies of the intelligent devices with the bodies; the communication module is used for realizing bidirectional data interaction between the platform and the intelligent equipment; the task scheduling module performs task allocation and path planning by adopting a reinforcement learning algorithm based on the global dynamic situation map; the cooperative control module models an equipment space-time relationship through a graph neural network, predicts conflicts in real time and generates a dynamic avoidance strategy; and the edge computing node carries out real-time preprocessing on the equipment sensing data. According to the method, global optimization scheduling and multi-device intelligent cooperation are realized, and the overall efficiency, robustness and adaptability of the system are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital twinning and embodied intelligence system, and particularly relates to a heterogeneous embodied intelligent device collaborative management system and method based on digital twinning. BACKGROUND

[0002] With the deep integration of artificial intelligence, robotics and the Internet of Things (IoT), embodied intelligent devices (Embodied AI) represented by drones, unmanned vehicles, service robots and bionic robot dogs are gradually applied to complex physical scenarios such as smart restaurants, intelligent warehouses, modern parks and smart hospitals to replace or assist human beings in completing repetitive and high-intensity tasks such as delivery, cleaning, inspection and security. The traditional operation mode relying on manpower has inherent problems such as high labor cost, limited work efficiency by individual ability, fluctuation of service quality due to fatigue or negligence, and long personnel training period. The introduction of automated equipment is considered an effective way to solve these pain points, and various commercial robot systems for single scenarios (such as food delivery and floor cleaning) have appeared in the market. These systems usually navigate and avoid obstacles based on pre-stored maps and local sensors (such as laser radars and visual cameras), and can perform preset tasks in structured or simple dynamic environments.

[0003] However, the existing technical solutions generally expose three core defects at three levels when dealing with real-world complex, dynamic and multi-agent coexisting environments: First, at the perception and interaction level, existing devices have a serious "perception island" problem. Most commercial robots use a self-centered perception mode, and their decisions only rely on real-time data from their own local sensors. This mode leads to a lack of dynamic understanding of the global environment, and the device cannot perceive key information such as the density distribution of people in public areas, the real-time state and intention of other devices, and temporary dynamic obstacles (such as randomly parked trolleys and unexpected spills on the ground). For example, during the peak period in a restaurant, a food delivery robot cannot predict congested areas and can only rely on local sensors to stop or detour when encountering obstacles, resulting in low running efficiency, unpredictable task execution time, and even safety risks in mixed environments. This isolated and passive perception method makes the behavior pattern of intelligent devices appear stereotyped and clumsy, making it difficult to adapt to high-dynamic and human-machine integrated complex environments.

[0004] Secondly, at the scheduling and planning level, the existing system lacks global optimization and forward-looking decision-making capabilities. Traditional multi-robot scheduling systems mostly use scheduling strategies based on static rules (such as first-come-first-served, fixed priority). These strategies fail to fully consider the complex spatial characteristics of the environment (such as changes in passage width, tidal flow congestion, and dynamic occupancy status of functional areas) and real-time changes in task flow, making it impossible to perform unified and collaborative optimization across devices. The direct consequences are: path conflicts during multi-device operation, congestion at bottleneck sections, uneven task allocation (some devices are overloaded while others are idle), and low utilization efficiency of overall resources (such as power and time). For example, in a warehouse scenario, multiple AMRs (autonomous mobile robots) may block each other due to lack of coordination, or concentrate on the same area to pick up goods, resulting in a decrease in overall throughput. Existing path planning algorithms are mostly limited to static or short-term dynamic environments and lack the ability to use historical data and real-time information for predictive and adaptive planning.

[0005] Thirdly, at the system architecture and coordination level, existing solutions have a serious "system island" problem. In actual deployment, robots of different functions and different brands often come from multiple manufacturers, with different underlying control systems, communication protocols, and data interfaces, forming independent "information islands". This makes complex collaborative operations across devices (such as cleaning robots giving way to delivery robots, or multiple devices collaborating to handle the same task) extremely difficult or even impossible to implement. The system lacks a centralized platform that can accommodate heterogeneous devices and implement information sharing and unified scheduling, thus failing to form an organic whole with complementary functions and coordinated actions to flexibly respond to the ever-increasing complex task demands in reality. This fragmented technical status greatly limits the upper limit of the overall efficiency of the automation system.

[0006] Therefore, there is an urgent need for a multi-agent system with global real-time perception, dynamic intelligent decision-making, and deep integration of heterogeneous devices to solve the technical problems in environmental adaptability, scheduling efficiency, and coordination capabilities. SUMMARY

[0007] Embodiments of the present application provide a heterogeneous embodied intelligent device collaborative management system based on digital twinning: comprising: a digital twinning platform for building and maintaining a real-time high-fidelity three-dimensional model of the physical environment and digital twinning bodies of multiple embodied intelligent devices connected to the system; a communication module based on wireless communication networks and Internet of Things communication protocols for realizing bidirectional data interaction between the digital twinning platform and multiple embodied intelligent devices for issuing instructions and feedback; The task scheduling module is configured to receive an external task request, and perform task allocation and path planning for the plurality of embodied intelligent devices based on a global environment state provided by the digital twin platform and states of the plurality of embodied intelligent devices. The cooperative control module is configured to model the plurality of embodied intelligent devices and their spatio-temporal relationships as nodes and edges in a graph neural network, to predict path conflicts between the plurality of embodied intelligent devices in real time, and to generate a dynamic avoidance strategy based on the prediction results and task priorities of the plurality of embodied intelligent devices, the strategy including at least one of instructions for a specific device to perform waiting, deceleration, or path re-planning. The edge computing node is deployed in the physical scene and is configured to perform real-time preprocessing on sensor data uploaded by the plurality of embodied intelligent devices, and upload structured data to the digital twin platform.

[0008] In some embodiments, the digital twin platform includes: The environment modeling unit is configured to construct a high-fidelity three-dimensional geometric model of the physical environment by means of a laser radar, a visual SLAM, and a high-definition camera, and to perform semantic information labeling on the three-dimensional geometric model, the semantic information including functional area division, key coordinate points, and a spatial business grid. The device modeling unit is configured to construct a digital twin for the plurality of embodied intelligent devices, the digital twin including static attributes and dynamic attributes, the static attributes including physical dimensions, kinematic models, sensor parameters, actuator capabilities, and rated loads of the embodied intelligent devices, and the dynamic attributes including high-precision poses, speeds, remaining power, current task states, and fault alarm information uploaded by the embodied intelligent devices in real time. The data fusion unit is configured to integrate multi-source sensor data to generate a global dynamic situation map, the global dynamic situation map including at least a global passenger flow heat map, a position distribution map of the embodied intelligent devices, and channel congestion level information.

[0009] In some embodiments, the multi-source sensor data includes environment sensor data, data uploaded by the plurality of embodied intelligent devices, and external business system data. The multi-source sensor data is preprocessed by the edge computing node, wherein: laser radar point clouds are denoised using a voxel filtering algorithm, ground segmentation is performed using a RANSAC algorithm, camera images are subjected to target detection using a YOLOv5 algorithm, and unstructured data is converted into structured data.

[0010] In some embodiments, the task scheduling module performs dynamic path planning using a reinforcement learning algorithm based on the global dynamic situation map, and generates an optimal path in combination with real-time updated environment information.

[0011] In some embodiments, the IoT communication protocol is the MQTT protocol, the wireless communication network is a 5G or Wi-Fi6 network, and the data of the state feedback is uploaded to the digital twin platform by the communication module after being preprocessed by the edge computing node.

[0012] In another aspect, a method for collaborative management of heterogeneous embodied intelligent devices based on digital twin includes the following steps: Step S1: constructing and maintaining a real-time high-fidelity three-dimensional model of the physical environment through the digital twin platform, and constructing digital twins containing static attributes and dynamic attributes for multiple embodied intelligent devices accessing the system; Step S2: based on the wireless communication network and the IoT communication protocol, establishing a two-way data interaction channel between the digital twin platform and the multiple embodied intelligent devices, realizing real-time synchronization of instruction issuing and state feedback; Step S3: receiving external task requests, based on the global dynamic situation map and the state of each device provided by the digital twin platform, using a reinforcement learning algorithm to assign tasks and dynamically plan paths for the multiple embodied intelligent devices; Step S4: during the execution of tasks, the multiple embodied intelligent devices continuously receive environmental update information issued by the digital twin platform, and after real-time preprocessing of their own sensor data through the edge computing node, they feed back structured state data to the digital twin platform; Step S5: modeling the multiple embodied intelligent devices and their spatio-temporal relationships as nodes and edges in a graph through a graph neural network to predict path conflicts in real time; and based on the prediction results and task priorities, generating a dynamic avoidance strategy to dynamically reschedule and collaboratively control scenes with conflicts or environmental emergencies.

[0013] In some embodiments, in step S3, the reinforcement learning algorithm is a Q-Learning algorithm, and the optimization goal is to minimize the total task completion time; the output of the dynamic path planning includes the optimal navigation path point sequence issued to the embodied intelligent devices.

[0014] In some embodiments, in step S5, the dynamic avoidance strategy includes instructing the embodied intelligent devices to perform at least one of the following operations: waiting, slowing down, or re-planning the path. In another aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory, and the processor implements any of the above-mentioned methods for collaborative management of heterogeneous embodied intelligent devices based on digital twin when executing the program.

[0015] In yet another aspect, a computer-readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement any of the above-mentioned methods for collaborative management of heterogeneous embodied intelligent devices based on digital twin.

[0016] The above technical solution has the following technical effects: The digital-twin-based heterogeneous embodied intelligent device collaborative management system and method of the present application solves the problems of "blind behavior" and "response lag" of devices in the traditional scheme caused by relying on local perception, significantly enhances the forward-looking understanding and adaptive ability of a single device in a complex dynamic environment. In addition, at the scheduling and decision-making level, the system uses the global real-time information provided by the digital twin model to perform task allocation and path planning using data-driven intelligent algorithms (such as reinforcement learning), so that the scheduling strategy changes from a rigid mode based on static rules to a dynamic optimization mode that combines real-time spatial characteristics and task flow changes, fundamentally avoiding path conflicts, uneven resource allocation, and low overall efficiency among multiple devices, and achieving a qualitative leap in system overall throughput and resource utilization efficiency. At the system collaboration and robustness level, the bidirectional interaction mechanism based on the unified digital twin platform breaks down the "information silos" between different brands and types of heterogeneous devices, enabling cross-device instruction collaboration and data sharing; at the same time, advanced technologies such as graph neural networks are used to model the spatio-temporal relationships of multiple devices, enabling predictive resolution of potential conflicts and global collaborative response to sudden environmental events, so that the entire system can operate efficiently and reliably as a functionally complementary and action-consistent organic whole, greatly improving the overall intelligence, job flexibility, and operational robustness of the system in real complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a structural schematic diagram of a digital-twin-based heterogeneous embodied intelligent device collaborative management system according to an embodiment of the present application; Figure 2 FIG. 2 is a flowchart of a digital-twin-based heterogeneous embodied intelligent device collaborative management method according to another embodiment of the present application; Figure 3 FIG. 3 is a structural schematic diagram of another embodiment of the present application. DETAILED DESCRIPTION

[0018] To further illustrate the embodiments, the present application provides accompanying drawings. These drawings are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should be able to understand other possible implementations and advantages of the present application. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0019] Currently, in various labor-intensive scenarios such as restaurants, modern parks, large warehouses, hospitals, etc., a large number of service and management tasks, such as accurate material distribution, timed and on-demand cleaning of the environment, periodic inspection of key facilities, etc., still largely rely on traditional manual mode. This mode not only brings high labor costs and management overhead, but also has a series of inherent problems such as limited work efficiency due to individual ability, fluctuating service quality due to human factors (such as fatigue, negligence), long personnel training period, etc. In order to break through these bottlenecks, some advanced scenarios begin to introduce robots, unmanned vehicles and other automated equipment to replace or assist manual work. However, this part of the improvement still has many problems, such as serious lack of perception and interaction ability; rigid scheduling strategy and lack of foresight; and system isolation, lack of synergy, etc.

[0020] In view of this, the present application proposes a heterogeneous embodied intelligent device collaborative management system and method based on digital twinning, aiming to overcome the limitations of the prior art in multi-agent system collaborative management. The present application positions the digital twinning platform as the "intelligent cloud brain" of the entire collaborative system, which has a global view and strong computing and decision-making ability, responsible for providing macro and real-time updated environmental information and well-thought-out task instructions for the devices. Each embodied intelligent device acts as the "sensing tentacle" and "flexible hand and foot" of the system, faithfully executing instructions in the physical world while feeding back real-time and local environmental data to the "brain" through its rich sensors. This continuous "perception-decision-execution-feedback" loop enables the system to integrate scattered local information into a coherent global situation, thereby realizing system-level intelligent emergence that individual devices do not possess.

[0021] As shown in Figure 1 An embodiment of the present application provides a heterogeneous embodied intelligent device collaborative management system based on digital twinning: comprising: A digital twinning platform for building and maintaining a real-time high-fidelity three-dimensional model of the physical environment and digital twinning bodies of a plurality of embodied intelligent devices connected to the system; A communication module based on wireless communication networks and Internet of Things communication protocols for realizing bidirectional data interaction between the digital twinning platform and the plurality of embodied intelligent devices for issuing instructions and feeding back states; A task scheduling module for receiving external task requests, based on the global environmental state provided by the digital twinning platform and the states of the plurality of embodied intelligent devices, to allocate tasks and plan paths for the plurality of embodied intelligent devices; The cooperative control module is configured to model the plurality of embodied intelligent devices and their spatiotemporal relationship as nodes and edges in a graph neural network, to predict path conflicts between the plurality of embodied intelligent devices in real time, and to generate a dynamic avoidance strategy based on the prediction result and the task priorities of the plurality of embodied intelligent devices, the strategy including at least one of an instruction for a specific device to perform waiting, deceleration, or path re-planning. The edge computing node is deployed in the physical scene and is configured to pre-process sensor data uploaded by the plurality of embodied intelligent devices in real time, and upload structured data to the digital twin platform.

[0022] In the above embodiments, a high-fidelity virtual mapping of the physical environment and the plurality of embodied intelligent devices is constructed by the digital twin platform, and the virtual mapping is used as a “cloud brain” to realize bidirectional real-time data interaction with the embodied intelligent device cluster in the physical world through the communication module; the task scheduling module makes optimized decisions based on the global perspective provided by the twin data, and the cooperative control module predicts and resolves spatiotemporal conflicts between multiple devices in advance through the graph neural network; finally, the edge computing node ensures the real-time processing and low latency of the perception data. Thus, the three core problems of “perception island”, “rigid scheduling”, and “lack of cooperation” in the multi-agent system are fundamentally solved, and the traditional independent devices are transformed into an organic whole with global intelligence, dynamic optimization, and predictive cooperation capability, thereby significantly improving the overall operating efficiency, safety, and robustness of the system in a complex dynamic environment.

[0023] In the above embodiments, the graph neural network (GNN) is a neural network that directly operates on graph structure data. It can capture complex dependency relationships transmitted between nodes (entities) through edges (relationships). The cooperative control module models each robot as a node in the graph and models the spatiotemporal relationship between them as an edge through the graph neural network. GNN analyzes this dynamically changing graph to predict potential path conflicts between multiple devices and provide decision basis for intelligent avoidance.

[0024] In some embodiments, the digital twin platform includes: The environment modeling unit is configured to construct a high-fidelity three-dimensional geometric model of the physical environment through a laser radar, a visual SLAM, and a high-definition camera, and to perform semantic information labeling on the three-dimensional geometric model, the semantic information including functional area division, key coordinate points, and a spatial business grid. The device modeling unit is configured to construct a digital twin for the plurality of embodied intelligent devices, the digital twin including static attributes and dynamic attributes, the static attributes including physical dimensions, kinematic models, sensor parameters, actuator capabilities, and rated loads of the embodied intelligent devices, and the dynamic attributes including high-precision poses, speeds, remaining power, current task states, and fault alarm information uploaded by the embodied intelligent devices in real time. A data fusion unit is configured to integrate multi-source sensor data and generate a global dynamic situation map, which at least includes a global passenger flow heat map, a position distribution map of the embodied intelligent device, and channel congestion level information.

[0025] In the above embodiment, by decoupling and refining the functions of the digital twin platform, it is divided into three core components: an environment modeling unit, a device modeling unit, and a data fusion unit, which together build a decision base that surpasses traditional static maps and is rich in semantics and real-time data. In specific implementation, the environment modeling unit first constructs a high-fidelity three-dimensional geometric model of the physical environment through multi-sensor fusion technologies such as laser radar, visual SLAM, and high-definition cameras. Then, through artificial or AI-assisted methods, the model is finely annotated with semantics, including functional areas (such as pickup areas, charging areas), key coordinate points (such as precise coordinates of pickup points), and spatial business grids (such as table number Grid_A1), providing a directly understandable business operation interface for task scheduling. The device modeling unit constructs a digital twin for each embodied intelligent device connected to the system. This twin not only contains static attributes such as physical dimensions, kinematic models, and sensor parameters, but also receives and updates dynamic attributes such as high-precision pose, speed, remaining power, task status, and fault alarms uploaded by the device in real time, achieving a holographic mirror and precise tracking of the physical device's state. The data fusion unit integrates multi-source heterogeneous data from environmental sensors, device self-reporting, and external business systems (such as restaurant POS systems). After standardized processing (such as point cloud denoising, image recognition, and unstructured data conversion) by edge computing nodes, it generates a global dynamic situation map commonly used by the system, which at least includes a real-time updated global passenger flow heat map, a device position distribution map, and channel congestion level information. The technical effect lies in the collaborative work of the three units, which provides a data highly unified, information extremely rich, and real-time updated decision base for upper-layer applications, enabling the decision-making of task scheduling and collaborative control modules to be based on a precise digital model that deeply integrates physical space structure, business semantic rules, and real-time dynamic information, greatly improving the accuracy, reliability of subsequent intelligent decision-making, and the intelligent level of the overall system.

[0026] In the above embodiment, Visual Simultaneous Localization and Mapping (Visual SLAM) is a cutting-edge technology in the intersection of computer vision and robotics, allowing machines (such as robots, drones, autonomous vehicles, AR / VR devices) to use one or more cameras as the main sensor to achieve the following two core functions: Localization: estimating one's own position and pose in an unknown environment in real time (i.e., "where am I? Which direction am I facing?").

[0027] Mapping: constructing a map of the surrounding environment in real time based on observed data (i.e., "what does the world around me look like?").

[0028] Most importantly, "synchronization" means that the two processes are carried out simultaneously, online. The device explores the environment while using the constantly incoming new visual information to both update its own position and refine its understanding of the environment map, with the two processes relying on and complementing each other.

[0029] In some embodiments, the multi-source sensor data includes: environment sensor data, multi-robotic intelligent device upload data, and external business system data; The multi-source sensor data is preprocessed by an edge computing node, wherein: the laser radar point cloud is denoised by a voxel filtering algorithm, the ground segmentation is performed by a RANSAC algorithm, the camera image is detected by a YOLOv5 algorithm, and the unstructured data is converted into structured data.

[0030] In the above embodiment, by establishing a layered and modular data preprocessing architecture, the unified access specification of three types of heterogeneous data sources, i.e., environment sensor data, device state data, and external business system data, is clearly divided. The technical implementation principle is that the massive raw data processing task originally borne by the central platform is sunk to the edge computing node deployed on the scene site, and an algorithm pipeline optimized for real-time is configured for it: first, for the dense point cloud data generated by the laser radar, a voxel grid filtering algorithm is used for downsampling and noise removal, which compresses the data volume by more than 90% while ensuring the integrity of the key environmental geometric features; then, the RANSAC (Random Sample Consensus) algorithm is applied to robustly segment the ground plane from the filtered point cloud, providing a stable reference coordinate system for the navigation planning of all robotic intelligent devices; at the same time, the YOLOv5 deep learning object detection model is integrated for the high-definition video stream collected by the camera, which can identify dynamic obstacles (such as pedestrians and temporarily placed objects) in the image and output their categories and pixel coordinates, realizing the semantic analysis of visual information; finally, all these processing results and the original state data uploaded by the device, together with the business system instructions, are packaged into lightweight structured JSON data packets.

[0031] The above embodiments not only offload computation through edge side, greatly alleviate the bandwidth pressure of the upstream network and the computing load of the central platform, ensuring that the system responds to environmental changes within hundreds of milliseconds; more importantly, it provides high-quality data sources with high standardization, clear semantics and low noise for all upstream functional modules, making the state synchronization of the digital twin more accurate and the generation of the global dynamic situation map more reliable, thereby directly improving the accuracy of advanced decisions such as task scheduling and collaborative control and the robustness of the overall system operation.

[0032] Among the algorithms used in the above embodiments, Voxel Filtering is an algorithm for downsampling and denoising three-dimensional point cloud data. It divides the point cloud space into tiny cubes (voxels) and uses the center of gravity or center point of all points in each cube to represent the cube, thereby greatly reducing the data volume while preserving shape features. Its use in the system is: on the edge computing node, pre-process the original point cloud data collected by the laser radar, reduce the transmission bandwidth and computing burden. In addition, the ANSAC (Random Sample Consensus) algorithm is a robust model fitting algorithm. It estimates the mathematical model parameters (such as plane, straight line) by randomly sampling a small amount of data points, and counts how many data points conform to the model (consensus set). Through multiple iterations, find the model with the largest consensus set. Its use in this embodiment is mainly reflected in: from the pre-processed point cloud, quickly and accurately segment the ground plane to provide a stable reference plane for robot navigation. There is also the YOLOv5 (You Only Look Once, Version 5) algorithm, which is an advanced and efficient deep learning target detection algorithm. Its feature is "look once", that is, a single forward propagation can predict the bounding box and class of multiple targets in the image, which is very fast and suitable for real-time applications. Its use in the system of the present application is: on the edge computing node, real-time analysis of camera images, identification of dynamic obstacles (such as pedestrians, trolleys), specific objects (such as dining tables, take-out entrances), etc., and conversion of pixel information into structured data with semantics.

[0033] In some embodiments, the task scheduling module uses a reinforcement learning algorithm for dynamic path planning based on the global dynamic situation map, and generates an optimal path in combination with real-time updated environmental information.

[0034] In this embodiment, the task scheduling module models the path planning problem of the entire system as a continuously evolving reinforcement learning environment. It uses a global dynamic situation map that integrates multi-source information as the state space for reinforcement learning, which contains high-dimensional and dynamic environmental information such as real-time passenger flow heat map, device location distribution, and channel congestion level. Each movement instruction sent to the embodied intelligent device is considered as an action, and the minimization of overall task completion time is set as the core optimization goal to design the reward function. By continuously trying and evaluating the long-term cumulative reward brought by different actions in a specific environment state, the algorithm (such as Q-Learning) can autonomously learn an optimal strategy. The advantage of this strategy is dynamic and forward-looking: instead of calculating the shortest geometric path, it can actively avoid temporarily congested areas (such as densely populated areas) and balance the load of the entire system, generating a truly optimal path in the time dimension in a complex and variable environment, ultimately maximizing the overall throughput of the system and fundamentally improving the efficiency of task execution.

[0035] In the above embodiment, the reinforcement learning (RL) algorithm is a machine learning paradigm in which the agent learns the optimal decision-making strategy by interacting with the environment and learning from the rewards (positive / negative) obtained. It is used for dynamic path planning in the task scheduling module. The algorithm aims to minimize the overall task completion time and models the robot's movement decision-making process as a reinforcement learning problem, calculating the optimal path (action) based on the global dynamic situation map (state).

[0036] In some embodiments, the Internet of Things communication protocol is the MQTT protocol, the wireless communication network is a 5G or Wi-Fi6 network, and the data for state feedback is preprocessed by the edge computing node and uploaded to the digital twin platform by the communication module.

[0037] This embodiment builds a hierarchical and collaborative data processing and transmission system: first, at the network layer, 5G or Wi-Fi6 high-performance wireless communication technologies are selected to provide high-bandwidth, low-latency, and wide-connection basic communication capabilities for the system, ensuring high-speed and stable transmission of massive data; second, at the application layer, the MQTT protocol designed for the Internet of Things is used, which is based on the publish / subscribe mode, allowing the digital twin platform as the central node to efficiently issue batch instructions (such as path points) to specific devices or device groups, while devices can asynchronously publish their state messages, achieving communication decoupling and scalability; most importantly, edge computing nodes are introduced in the data uplink for pre-processing, converting raw and unstructured sensor data (such as images, point clouds) into lightweight structured information (such as "target position coordinates", "obstacle type") on site, greatly reducing the amount of data to be uploaded.

[0038] The embodiment greatly improves the real-time performance and reliability of the system. The edge preprocessing combined with the high-speed network controls the data delay within milliseconds, meeting the stringent timing requirements of multi-agent collaboration. The MQTT protocol can effectively cope with network fluctuations. At the same time, the resource utilization is optimized, avoiding the impact of raw data flooding on the central platform and network bandwidth, enabling the system to support larger-scale device clusters. On the other hand, the embodiment enhances the clarity and maintainability of the system architecture, with clear separation of communication and computing responsibilities, providing flexibility for continuous evolution and deployment of the system.

[0039] In the above embodiment, the MQTT (Message Queuing Telemetry Transport) protocol is a lightweight, publish / subscribe-based IoT message transmission protocol designed for scenarios with limited network bandwidth and limited device resources.

[0040] The digital twin-based heterogeneous embodied intelligent device collaboration management system of the present application has technical advantages that are not limited to the theoretical level. Through testing in real scenarios, the technical advantages of the system have been verified. For example, the application of the system in a university student cafeteria has produced significant and quantifiable economic and experience benefits, as follows: Efficiency is greatly improved, and operational capacity is enhanced: In the peak period (12:00-13:00) of a university student cafeteria, after applying the system, the order processing volume increased by 27%, and the comprehensive efficiency of robot task execution improved by up to 35%. This means that with the same hardware investment, the system's service throughput capacity and operational upper limit have been significantly enhanced, especially during peak business periods.

[0041] Conflict is significantly reduced, and the running process is smooth: Due to the global collaborative scheduling and conflict prediction capabilities, the path conflict occurrence rate between devices is reduced from 22 times per hour under the traditional scheme to only 3 times per hour. This not only means that the time wasted by robots in avoiding and waiting is greatly reduced, but also represents that the running process of the entire automation system has become highly smooth and safe from frequent stalls.

[0042] Cost is effectively reduced, and cost reduction and efficiency improvement are achieved: Energy cost: By optimizing the standby position and movement path of the device through intelligent scheduling, the system effectively eliminates a large amount of invalid movement and unnecessary acceleration and deceleration, reducing the average energy consumption of a single device by 15% to 18%.

[0043] Labor cost: Due to the improvement in automation level and reliability, a single store can reduce about 30% of service personnel. This allows human employees to be freed from tedious and repetitive physical labor and focus on more valuable customer communication and service innovation.

[0044] User experience improvement, service quality enhancement: The system significantly improves the stability and predictability of the service. For example, the standard deviation of the delivery time distribution is greatly reduced from 4.2 minutes to 2.8 minutes. This means that the waiting time of most customers is stable within a predictable range, and the uniformity of the service is greatly improved, effectively improving customer satisfaction and loyalty.

[0045] In addition to the aforementioned smart restaurant application scenarios, the system of the present application can be applied to various fields requiring heterogeneous device collaboration after adaptive adjustment as a general collaborative management framework. The following are several typical embodiments: Embodiment one: intelligent warehousing and logistics system In a large high-density warehouse environment, the system is formed into an integrated intelligent warehouse operation platform for collaborative scheduling of a heterogeneous device group composed of autonomous mobile robots (AMRs), unmanned forklifts, conveyor belt systems, and aerial inventory inspection drones. In this application scenario, the digital twin model not only includes the physical layout of the shelves, storage locations, conveyors, and charging areas, but also integrates warehouse management system (WMS) and order management system (OMS) data streams, and maps real-time inventory status. When collaborating, the system responds to outbound order requests, the digital twin platform performs global task decomposition and optimization, plans the optimal "goods-to-person" or "robot-to-goods" picking path, assigns AMRs to perform collaborative picking tasks, and synchronously schedules unmanned forklifts for bulk cargo handling and dynamic restocking. For inventory checking tasks, the system schedules drones to follow pre-calculated paths to conduct image acquisition and inventory checking on high-level shelves, achieving unmanned and accurate inventory checking. The system monitors the power and task load of each device in real time, performs dynamic charging scheduling and task redistribution, thereby achieving full-process automation and maximum efficiency of warehouse operations.

[0046] Embodiment two: intelligent hospital logistics and service management system In the complex indoor environment of a hospital, the system is implemented as a collaborative control platform for in-hospital intelligent logistics and service robots, which uniformly schedules medicine delivery robots, medical waste transfer robots, ultraviolet disinfection robots, and mobile guidance robots. The digital twin model of this scenario integrates the physical layout of the floor structure, patient rooms, operating rooms, pharmacies, and elevators, and fuses real-time crowd density information. The key feature is deep integration with the hospital information system (HIS), which provides real-time access to surgery schedules, medical orders, and patient demand data. The system dynamically assigns priorities based on the urgency of tasks in the HIS, such as assigning the highest priority to emergency surgery medicine delivery tasks, planning barrier-free paths, and coordinating elevator services; for routine disinfection tasks, it plans paths based on crowd heat maps during low-traffic periods; and the guidance robot provides optimal guidance services for patients based on real-time registration and department queue data.

[0047] Embodiment three: smart park comprehensive operation and maintenance system In the park and campus semi-open environment, the system serves as the intelligent operation and maintenance hub, coordinating and scheduling security patrol robot dogs, unmanned express delivery vehicles, road cleaning robots, and environmental monitoring drones. The digital twin model integrates static geographic information such as buildings, roads, and greenery, and accesses real-time data from multiple sources such as surveillance cameras, environmental sensors (such as air quality and noise monitoring), and Internet of Things garbage cans, forming a dynamic multi-dimensional park status view. The security patrol robot dogs execute dynamic patrol routes based on pedestrian density and historical security event data; the cleaning robots respond to scheduling instructions when the Internet of Things garbage can filling rate exceeds the threshold; the unmanned express vehicle path planning integrates real-time pedestrian and vehicle flow data; and the environmental monitoring drone collects environmental data according to the preset strategy, providing data support for management decisions.

[0048] Embodiment four: intelligent agricultural precision operation system In a large modern agricultural scenario, the system is implemented as a precision agriculture collaborative operation platform, which uniformly schedules autonomous tractors, precision pesticide unmanned aerial vehicles, irrigation robots, and vision recognition picking robots. The digital twin model grids the work area and integrates meteorological data, soil sensor humidity and nutrient data, and multispectral unmanned aerial vehicle image data. Based on the analysis results of the growth of crops in different grids, the system determines the water and fertilizer requirements, schedules irrigation robots or pesticide unmanned aerial vehicles to perform on-demand irrigation or variable-rate pesticide application operations; according to the visual recognition and maturity model, the picking robot is scheduled to collaboratively perform the harvesting operation, and the harvesting order and transportation path are optimized to achieve the harvesting of agricultural products at the optimal maturity.

[0049] It should be understood that although the application scenarios and optimization objectives of the above embodiments are different, they all share the core architecture of the digital twin body construction specification, the physical-virtual bidirectional interaction protocol, and the dynamic scheduling algorithm based on reinforcement learning. The main difference between the scenarios is the adaptability of the optimization objective function in the semantic information contained in the digital twin model and the task scheduling strategy, which fully demonstrates the high scalability and practical value of the present application as a general technical framework.

[0050] On the other hand, as Figure 2 The present application also provides a heterogeneous embodied intelligent device collaborative management method based on digital twinning, comprising the following steps: Step S1: constructing and maintaining a real-time high-fidelity three-dimensional model of the physical environment through a digital twinning platform, and constructing a digital twin body containing static attributes and dynamic attributes for multiple embodied intelligent devices connected to the system; Step S2: based on wireless communication networks and Internet of Things communication protocols, establishing a bidirectional data interaction channel between the digital twinning platform and the multiple embodied intelligent devices, realizing real-time synchronization of command issuance and state feedback; Step S3: receiving an external task request, based on the global dynamic situation map and the state of each device provided by the digital twin platform, using a reinforcement learning algorithm to assign tasks and dynamically plan paths for multiple embodied intelligent devices; Step S4: during the execution of the task, the multiple embodied intelligent devices continuously receive the environment update information issued by the digital twin platform, and after real-time preprocessing of the sensor data by the edge computing node, the structured state data is fed back to the digital twin platform; Step S5: modeling the multiple embodied intelligent devices and their spatio-temporal relationships as nodes and edges in a graph through a graph neural network to predict path conflicts in real time; and based on the prediction results and task priorities, generating a dynamic avoidance strategy for dynamic rescheduling and collaborative control of scenes with conflicts or environmental emergencies.

[0051] The above embodiment constructs a "digital twin driven, edge collaborative, and globally optimized" closed-loop control paradigm: through steps S1 and S2, a high-fidelity model that accurately maps and is in real-time synchronization with the physical world is first created in the digital space, and a reliable two-way data channel is established; then, in step S3, the global perspective provided by the digital twin and the decision-making ability of reinforcement learning are used to allocate tasks and plan paths from the perspective of system-level efficiency, achieving optimal scheduling; step S4 converts local perception data into effective information through edge computing, ensuring the real-time of the feedback loop and the quality of the decision-making data of the central platform; finally, step S5 uses a graph neural network to model complex spatio-temporal relationships, achieving predictive perception of potential conflicts and intelligent resolution based on priorities. This embodiment transforms the traditional isolated, passive, and rule-based device control mode into a globally collaborative, proactive intervention, and dynamically adaptive system-level intelligent management mode, thereby significantly improving the overall operational efficiency, operational safety, and resource utilization of the multi-agent system in a complex and dynamic environment.

[0052] In some embodiments, in step S3, the reinforcement learning algorithm is a Q-Learning algorithm, and the optimization goal is to minimize the total task completion time; the output of the dynamic path planning includes the optimal navigation path point sequence issued to the embodied intelligent device.

[0053] In the above embodiment, the Q-Learning algorithm is a classic algorithm of reinforcement learning, which evaluates the long-term return of taking a certain action in a specific state by maintaining a Q-value table. The Q-Learning algorithm takes "minimizing the total task completion time" as the goal, models the mobile robot's movement decision-making process as a reinforcement learning problem, and calculates the optimal path (action) based on the global dynamic situation map (state).

[0054] The above embodiment converts the multi-device cooperative path planning problem into a sequence decision optimization problem with the global goal of minimizing the overall task completion time. The Q-Learning algorithm maps the environment state (defined by the global dynamic situation map) and possible actions (moving direction) to a Q-value table, enabling the system to autonomously learn the action strategy that can achieve the maximum long-term reward under a specific environment state. The technical effect is twofold: first, the optimal navigation path point sequence generated is no longer an isolated geometric path, but a dynamic decision that integrates real-time environmental information (such as congestion and pedestrian flow) to actively guide the device to choose the most efficient route; second, the algorithm optimizes from the overall system efficiency, avoiding local optimal solutions, thereby significantly reducing the waiting and conflict time between devices, and fundamentally improving the overall throughput and cooperative work efficiency of the multi-agent system.

[0055] In some embodiments, in step S5, the dynamic avoidance strategy includes instructions for the specific embodied intelligent device to perform at least one of waiting, slowing down, or re-planning a path.

[0056] This embodiment converts abstract conflict prediction into specific and executable control instructions. Based on the prediction results of the spatio-temporal relationship between devices by the graph neural network, and combined with the pre-set task priority rules, the potential path conflict resolution strategy is specified into three gradient-based basic operations: waiting, slowing down, or re-planning a path. This dynamic avoidance strategy realizes seamless connection between decision-making and execution, enabling the intelligent decision of the cooperative control module to be accurately and immediately executed by physical devices. It not only avoids the system shock and efficiency loss caused by traditional emergency stop strategies, but also adopts the most reasonable response according to the severity of the conflict and the urgency of the task, thereby maintaining the smoothness and efficiency of the overall system operation to the greatest extent under the premise of ensuring absolute safety.

[0057] As Figure 3 The present application also provides a digital twin-based heterogeneous embodied intelligent device cooperative management device, which is in a computing device including a processor 301, a memory 302, and a bus 303. The memory stores a computer program that, when executed by the processor, implements the above-mentioned digital twin-based heterogeneous embodied intelligent device cooperative management method. The device can be a server, a cloud computing platform, or an edge computing device, supporting the use of educational institutions or online learning platforms.

[0058] Further, as an implementable solution, the above-mentioned heterogeneous embodiment-intelligent device collaborative management apparatus based on digital twinning can be a computer unit, which can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer unit can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above-mentioned constituent structure of the computer unit is merely an example of the computer unit, and does not constitute a limitation on the computer unit, and can include more or fewer components than the above, or combine certain components, or different components. For example, the computer unit can also include an input / output device, a network access device, a bus, and the like, and the embodiments of the present application do not limit this.

[0059] Further, as an implementable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The processor is the control center of the computer unit, and connects various parts of the computer unit through various interfaces and lines.

[0060] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer unit by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the mobile phone, and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0061] In some embodiments, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the above method according to the embodiments of the present application.

[0062] The modules / units integrated with the computer unit, if realized in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the present application can also be implemented by a computer program to instruct related hardware to complete all or part of the above-mentioned method embodiments. The computer program can be stored in a computer readable storage medium, and the computer program, when executed by a processor, can implement the steps of the above-mentioned method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction.

[0063] In some embodiments, the present application also provides a computer program product, which includes a computer program, and the computer program, when executed by a processor, implements the steps of the above method.

[0064] Although the present application is specifically shown and described in connection with the preferred embodiments, those skilled in the art should understand that various changes in form and details can be made to the present application without departing from the spirit and scope of the present application as defined in the appended claims.

Claims

1. A heterogeneous embodied intelligent device collaborative management system based on digital twinning, characterized in that, Comprise: A digital twin platform for building and maintaining a real-time high-fidelity three-dimensional model of a physical environment and digital twins of a plurality of embodied intelligent devices connected to the system; A communication module based on a wireless communication network and an Internet of Things communication protocol for realizing bidirectional data interaction between the digital twin platform and the plurality of embodied intelligent devices in terms of instruction issuance and state feedback; A task scheduling module for receiving external task requests, based on the global environment state provided by the digital twin platform and the state of the plurality of embodied intelligent devices, to assign tasks and plan paths for the plurality of embodied intelligent devices; A collaborative control module for modeling the plurality of embodied intelligent devices and their spatio-temporal relationships as nodes and edges in a graph neural network to predict path conflicts between the plurality of embodied intelligent devices in real time, and based on the prediction results and the task priorities of the plurality of embodied intelligent devices, to generate dynamic avoidance strategies, including at least one of instructions for specific devices to execute waiting, slowing down, or re-planning paths; An edge computing node deployed in a physical scene for real-time preprocessing of sensor data uploaded by the plurality of embodied intelligent devices and uploading structured data to the digital twin platform.

2. The system of claim 1, wherein, The digital twin platform comprises: An environment modeling unit for building a high-fidelity three-dimensional geometric model of a physical environment through laser radar, visual SLAM, and high-definition cameras, and labeling the three-dimensional geometric model with semantic information, including functional area division, key coordinate points, and spatial business grid; A device modeling unit for building digital twins of the plurality of embodied intelligent devices, including static attributes and dynamic attributes, the static attributes including physical dimensions, kinematic models, sensor parameters, actuator capabilities, and rated loads of the embodied intelligent devices, and the dynamic attributes including high-precision pose, speed, remaining power, current task state, and fault alarm information uploaded by the embodied intelligent devices in real time; A data fusion unit for integrating multi-source sensor data to generate a global dynamic situation map, which at least includes a global passenger flow heat map, a location distribution map of embodied intelligent devices, and channel congestion level information.

3. The system of claim 2, wherein, The multi-source sensor data includes environment sensor data, data uploaded by the plurality of embodied intelligent devices, and external business system data; The multi-source sensor data is preprocessed by an edge computing node, wherein: laser radar point cloud is denoised using a voxel filtering algorithm, ground segmentation uses the RANSAC algorithm, camera images use the YOLOv5 algorithm for object detection, and unstructured data is converted into structured data.

4. The system of claim 2, wherein, The task scheduling module uses a reinforcement learning algorithm to perform dynamic path planning based on the global dynamic situation map, and generates an optimal path in combination with real-time updated environment information.

5. The system of claim 1, wherein, The Internet of Things communication protocol is the MQTT protocol, the wireless communication network is 5G or Wi-Fi6 network, and the data of the state feedback is uploaded to the digital twin platform by the communication module after preprocessing by the edge computing node. 6.A method for collaborative management of heterogeneous embodied intelligent devices based on digital twinning, characterized in that, Comprise the following steps: Step S1: constructing and maintaining a real-time high-fidelity three-dimensional model of the physical environment through the digital twin platform, and constructing digital twins containing static attributes and dynamic attributes for multiple embodied intelligent devices accessing the system; Step S2: based on wireless communication networks and Internet of Things communication protocols, establishing a two-way data interaction channel between the digital twin platform and the multiple embodied intelligent devices, realizing real-time synchronization of instruction issuance and state feedback; Step S3: receiving external task requests, based on the global dynamic situation map and the state of each device provided by the digital twin platform, using a reinforcement learning algorithm to assign tasks and dynamically plan paths for the multiple embodied intelligent devices; Step S4: during the execution of tasks, the multiple embodied intelligent devices continuously receive environmental update information issued by the digital twin platform, and after real-time preprocessing of their own sensor data through edge computing nodes, they feed back structured state data to the digital twin platform; Step S5: modeling the multiple embodied intelligent devices and their spatio-temporal relationships as nodes and edges in a graph through a graph neural network to predict path conflicts in real time; and based on the prediction results and task priorities, generating a dynamic avoidance strategy to dynamically reschedule and cooperatively control scenarios with conflicts or environmental emergencies.

7. The method of claim 6, wherein, In step S3, the reinforcement learning algorithm is a Q-Learning algorithm, and the optimization goal is to minimize the total task completion time; the output of the dynamic path planning includes the optimal navigation path point sequence issued to the embodied intelligent devices.

8. The method of claim 6, wherein, In step S5, the dynamic avoidance strategy includes instructing the embodied intelligent devices to perform at least one of the following operations: waiting, slowing down, or re-planning the path.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 8. The processor executes the program to implement the method of any one of claims 6 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 6 to 8.

Images