Group robot collaborative operation management and predictive maintenance system based on internet of things

Abstract

The application discloses a group robot collaborative operation management and predictive maintenance system based on Internet of Things, comprising a physical layer, a data layer, a digital twin layer, a collaborative operation management layer, a predictive maintenance layer connected with the data layer and the digital twin layer respectively, and an application service layer, wherein the collaborative operation management layer is constructed by integrating global state perception, multi-agent deep reinforcement learning task allocation and space-time collaborative path planning, so that the system can dynamically respond to changing tourist demand and environmental conditions, realize optimal assignment of tasks, conflict-free path planning and behavior coordination, and improve the overall task throughput, response speed and resource utilization of the robot cluster.

Description

Technical Field

[0001] This application belongs to the field of robot collaborative control and intelligent operation and maintenance technology, specifically involving a group robot collaborative operation management and predictive maintenance system based on the Internet of Things. Background Technology

[0002] With the rapid development and intelligent transformation of the cultural tourism industry, service robots are increasingly widely used in theme parks, museums, scenic spots, and other scenarios. Their functions have expanded from simple guided tours to multiple dimensions such as delivery, cleaning, and interactive performances, forming complex multi-robot cluster operation environments. However, the current operation and management of such robot systems mainly face the following challenges: In terms of multi-robot collaboration, existing systems mostly adopt pre-programmed or simple centralized scheduling strategies, which are difficult to adapt to the characteristics of tourist flow distribution and the strong spatiotemporal dynamics of task requests in cultural and tourism scenarios. The lack of efficient collaboration mechanisms between robots can easily lead to uneven task allocation, path conflicts, resource idleness or congestion, and the overall operational efficiency and service response speed need to be improved.

[0003] In terms of equipment maintenance, current methods mainly rely on fixed-cycle manual inspections and post-fault repairs. This approach cannot accurately and in real time perceive the health status of the internal mechanical and electrical components of the equipment, making it difficult to detect early performance degradation or potential faults in a timely manner. This leads to frequent unplanned downtime, affecting the visitor experience, and also incurs high maintenance costs.

[0004] Digital twin technology has been applied in the industrial field, but in the management of cultural and tourism robot clusters, its application is often limited to static 3D visualization, failing to be deeply integrated with real-time IoT data, artificial intelligence algorithms, and business operations. The lack of high-fidelity, two-way dynamic synchronization and simulation capabilities between physical entities and digital models limits its potential in task pre-playing, algorithm testing, fault pre-diagnosis, and collaborative optimization.

[0005] Therefore, there is an urgent need for a comprehensive management system that can deeply integrate IoT, digital twin and artificial intelligence technologies to achieve intelligent collaborative operation of group robots, real-time and accurate status perception, early fault prediction and maintenance, and dynamic interaction optimization, so as to improve the overall service reliability, operational efficiency and tourist satisfaction of robot clusters in cultural and tourism scenarios. Summary of the Invention

[0006] This application provides an IoT-based collaborative operation management and predictive maintenance system for group robots, aiming to solve the problems existing in the prior art.

[0007] An IoT-based collaborative operation management and predictive maintenance system for group robots includes: The physical layer includes multiple robots and multi-source sensors and IoT communication modules mounted on them, which collect the robots' own status data, environmental data and task interaction data. The data layer communicates with the physical layer to receive, clean, fuse, and store real-time and historical operational data from the physical layer. A digital twin layer, connected to the data layer, constructs and maintains a digital twin model synchronized with the physical robot based on the data processed by the data layer. The digital twin model includes a geometric model, a behavioral model, and a fault model. The collaborative operation management layer is connected to the data layer and the digital twin layer respectively, and performs dynamic task allocation, path collaborative planning and behavior coordination for the robot cluster based on the digital twin model and / or the data of the data layer. The predictive maintenance layer is connected to the data layer and the digital twin layer respectively. By analyzing the data of the data layer and / or the model output of the digital twin layer, it realizes fault warning, health assessment and remaining life prediction of robot components. The application service layer connects to the data layer, digital twin layer, collaborative operation management layer, and predictive maintenance layer, respectively, and provides functions such as visual monitoring, mobile management, maintenance work orders, and data analysis.

[0008] Optionally, the data layer includes an IoT gateway cluster, a streaming data processing engine, and a hierarchical storage module; The IoT gateway cluster receives data from the physical layer and performs protocol parsing, preliminary filtering, and aggregation. The streaming data processing engine connects to the IoT gateway cluster to clean, align, extract and fuse data to form a comprehensive state feature vector of the robot. The hierarchical storage module is connected to the streaming data processing engine and stores the processed data into time-series databases, relational databases, and object storage according to access frequency and value density.

[0009] Optionally, the construction and synchronization of the digital twin layer includes: the geometric model constructs its three-dimensional model based on the robot's original design data, and performs kinematic modeling and lightweight processing according to the operating state; The behavior model integrates the robot's kinematics, dynamics, and perception and decision-making algorithms to simulate its dynamic behavior and intelligent responses in a virtual environment. The fault model is based on a fault mode and effects analysis library and parametric fault simulation to simulate equipment performance degradation and fault propagation. The digital twin layer achieves synchronization with the physical robot through real-time data-driven operation and supports historical playback and future projection through a simulated clock.

[0010] Optionally, the digital twin layer further includes a parameter self-learning and calibration module; When a physical robot experiences a real malfunction and is repaired, the parameter self-learning and calibration module extracts the full-cycle data of the malfunction and replays it in the digital twin environment. By comparing the real fault feature sequences with the fault model simulation output sequences, an optimization algorithm is used to automatically adjust the parameters of the fault model, so that the simulation data matches the real data, and the optimized parameters are updated to the fault model parameter library.

[0011] Optionally, the collaborative operation management layer includes a task allocation module and a path planning module; The task allocation module adopts a multi-agent deep reinforcement learning architecture. Based on the global state vector including the robot state matrix, task queue feature matrix and environmental context tensor, it outputs task allocation decisions to optimize task completion rate, response time, load balancing and energy efficiency. The path planning module performs path search based on a spatiotemporal map, and achieves collision-free collaborative movement of multiple robots by reserving spatiotemporal paths with the broadcast robot and using a distributed conflict detection and negotiation mechanism.

[0012] Optionally, the collaborative operation management layer further includes an interaction strategy generation module; The interaction strategy generation module generates personalized interaction strategies based on real-time multimodal tourist context data, matching from a preset interaction strategy knowledge base or generating them through a large language model. After the generated strategy is filtered for security and compliance and verified in a digital twin simulation environment, it is compiled into executable instructions for the robot, and the knowledge base and the generated model are optimized based on the evaluation feedback of the interaction effect.

[0013] Optionally, the predictive maintenance layer includes: a real-time anomaly detection module that uses an unsupervised learning algorithm to perform early anomaly detection on the sensor data stream; The fault mode identification module uses a multi-classification model to identify the fault modes of confirmed anomalies. The remaining useful life prediction module uses a sequence learning model to predict the remaining useful life of components based on time-series data of component health indicators, and provides the probability distribution and confidence interval of the prediction results.

[0014] Optionally, the remaining useful life prediction module adopts a regression prediction model based on bidirectional long short-term memory network and attention mechanism; The module enables Monte Carlo Dropout during the inference phase, obtaining the distribution of predicted samples through multiple forward propagations to quantify prediction uncertainty. The module supports incremental learning. When real-world failures occur, new data is used to perform safety verification and fine-tuning updates on the model, and model version management is implemented.

[0015] Optionally, the application service layer includes: a visualization monitoring platform, used to display the real-time location, status, alarms, and fused data layers of the robot cluster based on a 3D map; Mobile management terminals are used to push maintenance work orders to on-site personnel, provide navigation and augmented reality assistance, and support on-site data collection and transmission. The maintenance work order system is used to automatically generate work orders based on the output of the predictive maintenance layer or manual alarms, and to realize intelligent scheduling, closed-loop management and knowledge accumulation of work orders; The Data Analysis Report Center provides templated and customizable data analysis reports, interactive data exploration, and automated report delivery.

[0016] Optionally, the system further includes a unified spatiotemporal data model for defining and managing robot entities, sensor entities, observation data entities, task entities, and event entities; All time- and space-related data are attached with uniform time labels and spatial coordinate labels; The unified spatiotemporal data model serves as the semantic basis for data exchange and association among the data layer, digital twin layer, collaborative operation management layer, and predictive maintenance layer.

[0017] Compared with the prior art, this application has at least the following beneficial effects: This application constructs a collaborative operation management layer that integrates global state perception, multi-agent deep reinforcement learning task allocation, and spatiotemporal collaborative path planning. The system can dynamically respond to changing tourist demands and environmental conditions, achieve optimal task assignment, conflict-free path planning, and behavior coordination, and improve the overall task throughput, response speed, and resource utilization of the robot swarm.

[0018] This application establishes a digital twin model integrating geometry, behavior, and faults, and deeply binds it with real-time IoT data. The system not only achieves millisecond-level mirroring of the physical robot state, but also supports task pre-playing, algorithm verification, and fault simulation in a virtual environment. Based on a fault model parameter self-learning mechanism based on real cases, it continuously improves the mapping accuracy of the digital model to the physical failure process, providing a reliable foundation for predictive maintenance.

[0019] This application achieves early anomaly detection, accurate fault mode identification, and probabilistic prediction of remaining service life of robot key components by extracting and fusing features from multi-source sensor data in real time through the data layer and combining machine learning models. This effectively reduces unplanned downtime, extends equipment life, and lowers maintenance costs. Attached Figure Description

[0020] Figure 1 A schematic diagram of the module connections for the IoT-based group robot collaborative operation management and predictive maintenance system provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0022] The IoT-based collaborative operation management and predictive maintenance system for group robots provided in this application includes: The physical layer includes multiple service robots deployed in cultural and tourism scenarios, each equipped with multi-source sensors for monitoring its own status and the environment, as well as an IoT communication module; Specifically, the physical layer comprises one or more robot clusters deployed within amusement parks or cultural and tourism scenic areas. Each cluster is professionally configured according to its core service functions, such as including: The tour guide robot is equipped with a highly mobile chassis, a multimodal interactive screen, a directional broadcasting system, and a etiquette and posture simulation mechanism. It is used to undertake tasks such as area tours, scenic spot explanations, Q&A interactions, and welcoming and seeing off guests. The delivery and patrol robots are equipped with sealed storage compartments with temperature control or humidity control functions, multi-compartment shelves, and intelligent storage and retrieval robotic arms, and are used to undertake fixed-point delivery of souvenirs, beverages and food, mobile retail and park patrol and display tasks. The cleaning and disinfection robot integrates sweeping, vacuuming, water spraying, disinfection atomization modules and path coverage algorithms to undertake automated cleaning and public health maintenance tasks in specific time periods or areas; The interactive performance robot has multi-degree-of-freedom bionic joints, flexible end effectors and a high-precision motion control system, which are used to perform complex anthropomorphic behaviors such as troupe dance, plot performance, tourist follow-and-take photos and simple game interaction. Each robot is equipped with an integrated multi-source heterogeneous sensor system to perceive its own status, environmental information, and task interaction data in real time, including but not limited to: Condition monitoring sensor group: Used to collect mechanical and electrical health status data of the robot body. This includes vibration sensors installed on drive motors and joint reducers to monitor abnormal vibration spectra; temperature sensors installed on motor drivers, battery packs, and the main control unit; and current / voltage sensors integrated into the power supply circuit to monitor power consumption and electrical anomalies. The selection and signal characteristics of these sensors (such as vibration signal sampling frequency and temperature monitoring point layout) can refer to mature solutions for industrial equipment condition monitoring, and be adaptively optimized according to the robot's lightweight and mobile characteristics. Environmental and Task Sensor Group: Used to perceive the robot's external environment and the information required to perform tasks. It includes multi-view vision sensors (RGB camera, depth camera) for simultaneous localization and mapping, obstacle recognition, and visitor face and expression recognition; a microphone array for collecting ambient sound, recognizing voice commands, and detecting abnormal noises (such as mechanical grotesque sounds); a global navigation satellite system receiver and inertial measurement unit for fusing and providing real-time pose data with centimeter-level accuracy; and ultrasonic / infrared proximity sensors for near-range obstacle avoidance. Specialized functional sensors are selected according to the type of robot. For example, temperature and humidity sensors in the storage compartment of a delivery robot, dust concentration sensors in a cleaning robot, and six-dimensional force / torque sensors at the joint ends of a performance robot. The data layer, which connects to the physical layer, includes an IoT gateway cluster, a streaming data processing engine, and a hierarchical storage module, used to aggregate, clean, fuse, and store real-time and historical operational data from the robot. The multi-source heterogeneous data from each robot is transmitted via its onboard communication module to an IoT gateway cluster deployed at the edge of the park. The gateway hardware adopts an industrial-grade design and features multiple network interfaces. Its main functions include: Protocol Adaptation and Parsing: The gateway has a built-in protocol stack that supports parsing IoT protocols such as MQTT and CoAP from robots, as well as industrial protocols such as ModbusTCP from traditional devices, and converts unstructured or semi-structured data into standard JSON or Protocol Buffers format. Preliminary data filtering and aggregation: Primary calculations are performed at the edge, such as real-time FFT transformation of raw high-frequency vibration data (e.g., 10kHz sampling) to extract amplitude features of specific frequency bands (e.g., RMS values ​​of 0-1kHz and 1-5kHz) before uploading, significantly reducing network bandwidth pressure. Simultaneously, dead-zone filtering is applied to state data, ensuring that data is only uploaded when changes exceed a threshold. Connection Management and Security: Responsible for maintaining long-lived connections with all online bots, implementing heartbeat detection and reconnection after disconnection. Also responsible for implementing the first line of security measures, such as device authentication (based on digital certificates or keys), data link encryption (e.g., TLS / DTLS), and DDoS attack protection. The data stream, initially processed by the gateway, is then rapidly connected to a streaming data processing engine (such as Apache Flink or Spark Streaming) via a message queue (such as Apache Kafka). This engine performs the core data cleaning and real-time fusion tasks, and the execution flow includes the following: Data cleaning: Automatically filters obviously erroneous sensor readings (such as temperature values ​​exceeding the range, abnormal positional changes in a static state) based on preset rules (such as physical limits, signal rationality). For time series data loss caused by brief communication interruptions, intelligent filling is performed using time series prediction-based methods (such as linear interpolation or previous valid value preservation) to ensure data continuity. For data containing high-frequency noise (such as raw current readings), online smoothing is performed using sliding window mean filtering or low-pass digital filters. Multi-source data spatiotemporal fusion: To address the timestamp discrepancies between different sensors and robots, the system injects high-precision timing signals (such as PTP) into the data stream. Based on this benchmark, the fusion engine performs timestamp alignment and resampling on the associated data streams to ensure that subsequent analysis is based on the same time benchmark. It transforms the position and environmental perception data of all robots (such as the position of objects recognized by vision) into a unified global coordinate system of the park, providing a consistent spatiotemporal view for collaborative tasks. It also associates and packages the vibration characteristic values, motor current values, and infrared temperature measurement points of the robot at the same moment to form a "feature vector" describing the comprehensive state of the device at that moment. The processed data is stored in storage media with different characteristics according to its access frequency and value density, forming a hierarchical storage architecture: Real-time / hot data storage: Time-series databases (such as InfluxDB and TDengine) are used to store recent (e.g., within 7 days) high-frequency sensor time-series data, real-time status, and alarm events. Its time-series optimization features support fast writing of massive amounts of data and efficient querying of recent historical data. Business / Warm data storage: Use relational databases (such as PostgreSQL) or document databases (such as MongoDB) to store structured business data, including robot profiles, task work orders, maintenance records, visitor interaction logs, task execution results, etc. Historical / Cold Data Archiving: For detailed time-series data that exceeds a certain time limit (such as 3 months), it is compressed and dumped to an object storage service (such as an object storage service compatible with the S3 protocol) for low-cost long-term archiving, which is used to support long-term trend analysis, model retraining and other batch offline calculations. The streaming data processing engine has built a real-time, automated feature extraction pipeline for the continuous raw signals uploaded by vibration sensors. This pipeline is designed to complete the conversion from raw waveforms to diagnostic feature vectors before the data stream reaches the storage system. The specific process is as follows: In the pipeline processing stage, the pipeline sequentially performs the following processing stages: First, for the original discrete vibration signal sequence input at a fixed sampling frequency (e.g., 10kHz), a digital low-pass filter (e.g., a finite impulse response filter) with a cutoff frequency lower than the Nyquist frequency is applied to eliminate high-frequency noise and prevent spectral aliasing in subsequent processing. The filtered continuous data stream is then sequentially divided into a series of continuous, possibly partially overlapping data segments, called analysis windows. The window length is not fixed; its determination strategy is detailed below. For the discrete amplitude data within each analysis window, a set of time-domain statistical characteristics are calculated, including at least: peak value, average value, root mean square value, standard deviation, and kurtosis. These characteristics are used to quickly reflect the overall energy level and impulse characteristics of the signal. For the data in the same analysis window, a fast Fourier transform is applied to convert the signal from the time domain to the frequency domain, obtaining its amplitude spectrum. Further, for a specific frequency band of interest in the spectrum (the method for determining it is described below), envelope demodulation analysis is performed: first, the signal in that frequency band is extracted by bandpass filtering, then its envelope is obtained by Hilbert transform, and finally, the envelope signal is subjected to FFT to obtain the envelope spectrum. Envelope spectrum can effectively highlight periodic impact characteristics, and is particularly sensitive to early failures of bearings and gears. The large number of time-domain and frequency-domain features (which may be up to tens of dimensions) obtained by calculation are reduced in dimensionality by principal component analysis or feature selection methods based on domain knowledge, forming a low-dimensional, information-condensed comprehensive feature vector for use in subsequent fault diagnosis models. An adaptive adjustment strategy for the sliding window length is employed to capture fault characteristics synchronized with the equipment's rotation cycle. The analysis window length needs to dynamically adapt to the equipment's rotational speed. Specifically, the strategy is as follows: The system receives speed feedback signals from the robot's servo motors in real time; Calculate the equipment's rotation period based on the current rotation speed (unit: revolutions per second). =1 / rotation speed; Determine the duration of the target window This duration is set to an integer multiple of the rotation period, i.e. , where N is a positive integer (e.g., N=10, representing coverage of 10 complete rotation cycles), ensuring that the window contains an integer multiple of the fault characteristic cycles, thus avoiding energy leakage in the spectrum analysis; Based on the set sampling frequency Fs, the time length is... Convert to the corresponding window length (Number of data points), i.e. The aforementioned As an up-rounding function, this strategy dynamically adjusts the physical time length of the analysis window as the robot accelerates or decelerates, always ensuring that a preset number of complete rotation cycles are included, thereby optimizing the accuracy of the spectrum analysis. The algorithm for selecting the center frequency and bandwidth of envelope demodulation is designed to accurately extract the resonant frequency band caused by the fault. The center frequency and bandwidth of the envelope demodulation are adaptively determined according to the following steps: Baseline spectrum acquisition and resonant band identification: Under the initial healthy condition of the equipment without faults, a segment of vibration data is collected, and its long-term average amplitude spectrum is calculated as the baseline spectrum. By analyzing the baseline spectrum, several high-frequency resonant peaks inherent in the equipment structure (usually located above 1kHz) are identified. These resonant peak frequencies are recorded as candidate center frequencies. ; Fault characteristic frequency calculation and monitoring frequency band mapping: Based on the equipment model (such as bearing model, number of gear teeth) and real-time speed, calculate the theoretical fault characteristic frequency in real time, such as bearing outer ring fault frequency, gear meshing frequency, etc. Optimal center frequency selection: During runtime, the real-time spectrum is continuously calculated. When a candidate resonance band is detected... When the energy in the vicinity is significantly higher than the baseline level, it indicates that the frequency band may be excited by a fault. The system selects the center frequencies of the 1-2 resonant frequency bands with the most significant energy increase as the actual center frequency Fc of the current envelope demodulation; Bandwidth determination: Set a symmetrical bandwidth B around the selected center frequency Fc. The width of bandwidth B is usually designed to cover the -3dB attenuation width of the resonance peak, or set to a fixed value (such as 500Hz) based on experience, to ensure that the modulated fault impulse information is included while excluding noise in irrelevant frequency bands; Through the aforementioned pipeline and its core adaptive strategy, the system can extract fault-sensitive and high-quality diagnostic features from the continuous vibration data stream in real time and automatically, providing reliable input for subsequent intelligent diagnosis and predictive maintenance decisions.

[0023] Furthermore, in order to achieve accurate correlation and unified management of multi-source heterogeneous data such as robot body state, environmental perception, task execution and system events in the spatiotemporal dimension, a structured unified spatiotemporal data model is constructed and adopted. This model defines the core semantic entities of the system, the relationships between entities and the data attachment specifications, which constitute the cornerstone of the data fusion and exchange of the entire system. The core entity types of the unified spatiotemporal data model include at least: Robot Entity: Represents each individual robot in the physical world. Its attributes include metadata such as unique identifier, model, cluster affiliation, current logical state, and power status. This entity serves as an anchor point for other dynamic data. Sensor Entity: Represents each sensor instance installed on a robot or in the environment. Its attributes are associated with its corresponding robot entity or fixed location and include static metadata such as sensor type, range, accuracy, and sampling rate. One robot entity can be associated with multiple sensor entities; Observational data entity: Characterizes a specific reading or processed feature generated by a sensor entity at a specific moment. This is the most active data entity. Its attributes must include not only numerical values, but also a reference to the sensor entity that generated the data, a high-precision timestamp, a data quality identifier, and a reference to the original data or feature vector; Task Entity: Represents a specific task assigned by the system to a robot or robot swarm. Its attributes include a unique task ID, task type, priority, target status, planned path point sequence, creation time, deadline, and execution status. Event Entities: Represent semantically meaningful state changes or alarms occurring internally or externally to the system. Their attributes include event type, severity level, trigger source, detailed description, occurrence time, acknowledgment status, and references to associated robot or task entities. All time- and space-related data must be attached in accordance with a unified labeling standard: Time stamping: Coordinated Universal Time (UTC) is used as the sole reference. All timestamps for observation data, events, and robot status updates must be synchronized to a high-precision clock source provided by a precision time protocol network deployed throughout the park. For non-real-time data (such as logs imported afterward), the time zone information must be provided for conversion. Spatial Labeling: A predefined global Cartesian coordinate system covering the entire park is used as the spatial reference. All location data, whether from GNSS positioning of the robot (which requires coordinate transformation), UWB local positioning, or map points constructed by visual SLAM, must be transformed into this global coordinate system for representation. Location data is stored in the form of (x,y,z) triples, along with coordinate system identification and estimated positioning accuracy. The data model is managed in the form of a schema definition file, which is described using a language such as AvroIDL or Protocol Buffers. Each addition, deletion, or modification of an entity or attribute corresponds to a schema version number. The stream processing engine and storage system must declare the schema version they use when reading and writing data. The system supports backward-compatible schema evolution; for example, adding optional fields does not affect the reading of older versions of data. Within the stream processing engine, the entities of the data model are primarily represented as messages in the streaming data. Adding a new observation involves encapsulating a message conforming to the "observation data entity" pattern and injecting it into the message queue. Modifying the state of a robot entity (e.g., changing it from "idle" to "performing a task") generates an "event entity" message (state change event) and may trigger an update to the robot entity's state copy. In storage systems (such as relational databases or document databases), core entities are typically materialized as tables or collections; Spatiotemporal index construction: To support efficient spatiotemporal range queries (such as "querying the location of all robots in a certain area within a certain time period"), the system builds a composite index for fields containing timestamps and location coordinates, such as using PostGIS spatial index and B-tree time index, or using the time-series database's native time partition and label index. Relationship query optimization: Clearly represent the relationships between entities through foreign keys or embedded documents (e.g., storing "sensor IDs" in "observation data"). For frequent join queries (e.g., "query the latest status of all robots under a certain task"), materialized views or periodic pre-aggregation can be used to transform join queries into queries on a single table, thereby improving response speed; A digital twin layer, connected to the data layer, is used to construct and maintain a three-dimensional visual digital twin model synchronized with the physical robot. The model includes a geometric model, a behavioral model, and a fault model. The geometric model forms the morphological basis of the digital twin. For the cultural tourism robot, its construction employs multi-source data fusion and layered detail technology, as detailed below: Basic structural modeling: Based on the robot's original computer-aided design drawings, import them into professional 3D modeling software to reconstruct its precise assembly structure, including 3D mesh models, materials, and textures of components such as the shell, chassis, robotic arm, and sensor housing; The model is rich in operation: For components whose shape changes during operation (such as robotic arm joints and wheels), their kinematic chains are established, and the rotation axis, translational degree of freedom and their range are defined; Lightweighting and Layered Detail Processing: To meet the performance requirements of real-time rendering on web or mobile devices, high-precision CAD models undergo mesh simplification, texture compression, and layered detail processing. Multiple model versions with different levels of detail are generated and dynamically switched based on viewing distance, ensuring reduced computational load while maintaining visual appeal. The construction of the behavioral model endows the digital twin with the ability to simulate the dynamic behavior and intelligent responses of a physical entity. Its core lies in creating and integrating mathematical models and algorithms that describe the robot's motion laws, environmental interactions, and decision-making logic, as detailed below: The kinematic and dynamic models were established, and core kinematic mathematical models were developed for the mechanical structures of different types of cultural and tourism robots: Kinematic Model of Mobile Platform: For wheeled mobile robots, an accurate kinematic model is established based on their chassis configuration. For differential drive robots, a mathematical model is established based on the left and right wheel speeds ωL and ωR, wheel radii r, and wheel track L. The instantaneous linear velocity v and angular velocity ω are calculated, and the pose (x, y, ...) in the global coordinate system is updated through integration. For omnidirectional mobile robots (such as those using Mecanum wheels or omnidirectional wheels), establish the wheel velocity vector to the robot body velocity vector. The mapping matrix, this model is the basis for virtual robot path tracking simulation; Robotic Arm Kinematic Model: For robots equipped with multi-joint robotic arms (such as those used for delivery or performance), the Denavit-Hartenberg parametric method is used to establish their kinematic model. This model defines the homogeneous transformation matrix from the base to the end effector, thereby enabling accurate calculation of forward kinematics (calculating the end effector pose from joint angles) and inverse kinematics (inversely solving the joint angles from the desired end effector pose). The model parameters are calibrated based on the precise dimensions and joint configuration of the robotic arm. Multibody Dynamics Model: To simulate more complex force interactions (such as load changes when grasping objects or the bumps and jolting of driving on rough roads), a simplified multibody dynamics model is introduced based on the kinematic model. This model is established based on the Lagrange equations or the Newton-Euler method, calculating the required driving torque of each joint under a given motion state and external forces. This model can be used to simulate current changes in motors under load and to assess the potential risks to mechanical structures under abnormal stress. To test and validate the robot's intelligent algorithms in a virtual environment, a simulation module of its perception and decision-making capabilities needs to be built within a twin: Virtual sensor simulation: In a 3D virtual scene, a virtual camera is set up at the camera mounting position of the twin robot model. Using a graphics rendering engine, RGB images and depth maps from the camera's perspective are rendered in real time, and effects such as lens distortion and noise can be simulated to generate a simulated image stream consistent with the data format of a real camera. A series of detection rays are emitted from the virtual LiDAR installation point into the scene. Based on the collision detection results of the 3D scene, the distance of each ray is calculated, thereby generating simulated LiDAR point cloud data. Parameters such as the number of scan lines, angular resolution, and maximum and minimum detection distance can be configured to match real sensors. Inertial Measurement Unit Simulation: Based on the linear acceleration and angular velocity calculated from the kinematic and dynamic model of the virtual robot, the simulated noise and zero bias are superimposed to generate the IMU data stream; The navigation planning algorithms, obstacle avoidance algorithms, and task scheduling logic actually used by the physical robot are integrated into the digital twin in the form of a software library or executable script. This decision logic model receives simulated data from virtual sensors as input. In the virtual environment, the digital twin autonomously runs its core algorithms for navigation, obstacle avoidance, and task execution using its kinematic model and virtual sensor data. For example, the virtual robot can plan obstacle avoidance paths in real time and drive the kinematic model to move based on simulated laser point clouds, thereby verifying the robustness and safety of the algorithm in new maps or extreme scenarios before deployment. This process constitutes a complete "perception-decision-control" closed-loop simulation. The fault model is the core of predictive maintenance, mapping the physical mechanisms of equipment failure to the digital space: Failure Mode and Effects Analysis (FMEA) Library: Based on the Failure Mode and Effects Analysis (FMEA) methodology, a failure library is established for various robot components. The library defines failure types, root causes, development processes, observable symptoms, and effects. Parametric fault simulation: Adjustable "health parameters" are reserved in the behavioral or dynamic models of the digital twin. For example, an "efficiency decay coefficient" is introduced in the motor model, and a "backlash increase parameter" or "abnormal friction coefficient" is introduced in the gear model. By dynamically adjusting these parameters, fault states such as equipment performance degradation, response hysteresis, abnormal noise and vibration can be simulated. Fault propagation simulation: Establishing fault logic dependencies between components. When simulating a fault in a component (such as a bearing), the system can automatically trigger the simulation of abnormal states in related components (such as increased motor current or intensified drive shaft vibration) based on physical logic, thereby deduce the propagation path and final impact of the fault in a virtual environment; The digital twin layer has a virtual-real synchronization and simulation deduction mechanism, realizing dynamic, bidirectional synchronization between physical entities and digital twins: Real-time data-driven synchronization: The real-time status data stream processed by the data layer continuously drives the digital twin via protocols such as WebSocket or MQTT. The robot's position, posture, joint angles, sensor readings, etc., are mirrored and updated in the virtual model with millisecond-level latency. Reverse control and "test-before-real": It supports sending control commands (such as moving to a point or performing an action) to the physical robot through the digital twin interface, realizing "virtual control of the physical". More importantly, it supports the "test-before-real" mode: new task paths, complex collaborative actions or maintenance operation procedures can be simulated and optimized in the digital twin environment for multiple rounds. After verification, the generated optimized command sequence is then sent to the physical robot for execution, greatly reducing the risk and cost of trial and error on site. Simulation clock management: For fault prediction or historical playback, the digital twin system has an independent simulation clock. It can accelerate operation and quickly extrapolate the long-term operating status of equipment in the future; it can also freeze or replay to analyze the complete process of a historical fault event in detail. Furthermore, in order to achieve a high-fidelity mapping of the digital twin fault simulation model to the failure process in the physical world, and to overcome the problem of inaccurate model parameters based on prior knowledge, a closed-loop parameter self-learning and calibration mechanism based on real fault case data was designed and implemented. Once the physical robot experiences a real malfunction and completes repairs, the system automatically triggers a learning cycle: Full-cycle data archiving: Extract all relevant multimodal time-series data of the robot from the data storage system for a preset period before the failure (e.g., 24 hours), during the failure and its duration, and during the post-repair verification phase, forming a complete failure case data package. This data package includes raw sensor data / features, control commands, status logs, maintenance records (e.g., replaced parts), and the final failure diagnosis conclusion; Alignment of key states and feature sequences: The case data package is preprocessed and its time axis is aligned with the digital twin simulation clock. Key observation sequences directly related to the target fault model are extracted. For example, for motor bearing wear faults, the amplitude evolution sequence of fault characteristic frequencies in the envelope spectrum of vibration acceleration is extracted. Replay this failure case in a digital twin environment: Set the initial conditions for simulation: Set the complete state of the robot (pose, sensor readings, and task status) at a healthy moment before the failure occurs as the initial state for the digital twin simulation. Perform parametric fault simulation: In the simulation, activate the fault model corresponding to the real fault type and load its current parameter set (such as the "equivalent clearance increment" parameter for bearing wear). Reverse simulation and data comparison: Run a digital twin simulation, advancing the simulation timeline to cover the entire fault development period. Simultaneously record virtual observation data generated in the simulation model that correspond to the key feature sequences extracted in step 1; Error and sensitivity calculation: The virtual feature sequence generated by the simulation is compared point by point with the actual feature sequence in the real case, and the overall error (such as root mean square error) is calculated. By fine-tuning the fault model parameters and repeating the simulation, the sensitivity of each parameter to the simulation output error is analyzed to determine which parameters play a dominant role in fitting the real data. The system transforms the parameter calibration problem into an optimization problem: Define the objective function: use the difference between the simulated output feature sequence and the actual feature sequence (such as the aforementioned root mean square error) as the objective function; Constructing an optimization solver: An iterative optimization algorithm is used to automatically find the optimal fault model parameters that minimize the objective function. For cases with a relatively smooth parameter space, gradient descent or its variants can be used; for cases where multiple local optima may exist, global optimization methods such as genetic algorithms or particle swarm optimization can be employed. Automatic calibration is performed: the optimizer repeatedly calls the digital twin fault simulation within a reasonable range of set parameters, evaluates the objective function and adjusts the parameters based on the results of each simulation, until the parameter combination that best matches the simulation data with the real data is found; Finally, the optimized new parameter set is updated in the model parameter library for this type of fault, and associated with the corresponding fault mode, robot model, and operating conditions. At the same time, the old parameter version is retained to form a model evolution history. The complete fault case, the corresponding optimal simulation parameters, and the parameter-feature correlation are structured and stored in the fault diagnosis knowledge base as a reference for future rapid diagnosis and prediction of similar faults. The collaborative operation management layer connects the data layer and the digital twin layer. Based on the twin data and artificial intelligence algorithms, it is used to realize the dynamic allocation of tasks, collaborative path planning, behavior coordination and tourist interaction strategy generation of robot clusters. The collaborative operation management layer includes a task allocation module and a path planning module. The collaborative operation management layer, as the intelligent decision-making hub connecting the digital twin model and the physical execution terminal, achieves global optimization of task scheduling, spatial planning, behavioral coordination and human-computer interaction of the robot cluster based on real-time twin data and artificial intelligence algorithms. Specifically, to address the strong spatiotemporal dynamics of visitor flow distribution and task requests in amusement park scenarios, the task allocation module of the system described in this invention adopts a multi-agent deep reinforcement learning architecture with centralized training and distributed execution to achieve online learning and optimization of task assignment strategies for robot clusters.

[0024] Construct a reinforcement learning simulation environment connected to a digital twin environment. In this environment, each robot is treated as an executive agent, while a central task scheduler acts as a coordinating agent. The coordinating agent is responsible for observing the global state and deciding which executive agent to assign a task to. The executive agents are responsible for receiving tasks and reporting their execution status. The state vector input to the coordinating agent policy network It is a structured splicing tensor containing the following components: Robot state matrix: one * The matrix, where The total number of robots, This describes the feature dimensions of a single robot. Each row contains: the robot's real-time pose data in the global coordinate system (x, y, ...). The current battery percentage, a binary-coded skill vector (indicating whether it has the ability to guide, deliver, perform, etc.), and a current task status code (such as idle, in progress, charging, or faulty). Task queue feature matrix: one * The matrix, where The number of tasks currently pending assignment. This describes the feature dimensions of a single task. Each row contains: task location coordinates, task type code, numeric priority weight, creation timestamp, and a flexible deadline. Context Tensor: The park map is discretized into a grid, and a visitor distribution heatmap is generated based on real-time visitor flow monitoring data (such as camera counts and Wi-Fi probes). After normalization, it is used as a two-dimensional channel input. Simultaneously, time information (such as time of day and whether it is a holiday) can be encoded as additional features. Action space and policy execution coordinate the agent's output actions at each decision step. Defined as a multi-discrete action space. Specifically, for each task in the current task queue, the policy network needs to output an action, which is derived from {robot1ID, robot2ID, ..., robot...}. Choose one item from the set of "delayed allocation". Choosing "delayed allocation" means that the task will not be dispatched at present, and will wait for subsequent state changes. Policy networks typically use attention mechanisms or graph neural networks to process variable-length task queue inputs and output the allocation probability distribution for each task, determining the final allocation through sampling or greedy selection; The reward function R is designed to be calculated after each round of task allocation and execution, aiming to guide the policy to learn the global optimization objective. This function is a weighted sum of multiple sub-reward items: Task completion reward For each successfully completed task, a positive reward is given, the size of which is proportional to the task priority; Response time penalty For each task, a negative, time-increasing penalty is imposed on the waiting time from its creation to when the robot begins execution. Load balancing rewards Calculate the standard deviation of the cumulative task execution time of all robots within a cycle. The smaller the standard deviation, the higher the reward, encouraging an even distribution of workload. Energy efficiency penalty Estimate total energy consumption based on the total distance all robots travel and standby power consumption within a cycle, and apply a negative reward to encourage energy conservation; Task Expiration Penalty For tasks that miss the flexible deadline due to improper allocation, apply a large fixed negative reward; Total Rewards The weighting coefficient Determined during system calibration.

[0025] The training and deployment process is as follows: Offline pre-training: Using a digital twin environment, a large number of training scenarios containing different tourist distributions and task burst patterns are simulated and generated to train the MA-DRL model offline on a large scale, so that it learns the basic allocation rules. Online fine-tuning and deployment: Deploying the pre-trained model to a real system. In the initial stages of operation, [the following steps are taken]. - A greedy strategy is used to randomly explore different allocation methods with a certain probability, while recording the actual transition states and rewards. This real-world interaction data is then used to continuously fine-tune the model online, adapting it to the dynamic characteristics of the real environment. Safety policy fallback: The output of the reinforcement learning policy module must pass through a rule-based safety verification layer. For example, it ensures that tasks requiring robotic arm operation are not assigned to robots lacking the necessary skills, or that robots with battery levels below a threshold are not assigned long-distance tasks. Verification failure triggers alternative assignment schemes in the rule base. Through the above specific design, the task dynamic allocation module can autonomously learn and execute a complex task scheduling strategy that comprehensively considers efficiency, fairness and energy consumption based on high-dimensional, multimodal real-time state information, thereby significantly improving the overall operational efficiency of robot clusters in dynamic cultural and tourism environments. To achieve collision-free and highly efficient collaborative movement of multiple robot systems in a shared dynamic environment, the path planning module of this invention constructs a collaborative path planning method with conflict prediction and resolution capabilities based on spatiotemporal maps and a distributed negotiation mechanism.

[0026] The system occupies the map in a static two-dimensional grid. Based on this, a discretized time dimension is introduced to construct a three-dimensional spatiotemporal planning map. The map is represented by a three-dimensional array Grid[x][y][t], where (x, y) are the spatial raster coordinates and t is the discretized time slice index. Each Grid[x][y][t] cell stores the following state information: Occupancy status: Indicates whether the spatiotemporal unit is currently reserved by a robot, permanently occupied by a static obstacle, or temporarily occupied by a dynamic obstacle (based on prediction); Occupant ID: If reserved, record the unique identifier of the robot that reserved the unit; Reservation types: divided into hard reservations (confirmed, non-negotiable) and soft reservations (pending confirmation, negotiable); Duration of time slice t Set according to system accuracy requirements (e.g., 0.1 seconds). Map depth on the timeline. Determined based on the estimated time of the longest planned path; When the robot Received to proceed to the target point After receiving the move command, its local planner initiates the following process: Spatiotemporal A* Search: The planner uses the robot's current position as the spatial starting point and the current moment as the temporal starting point, and... As the spatial endpoint (temporal endpoint is free), in The spatiotemporal A* algorithm is performed on the map. When expanding a node (x, y, t), the algorithm checks its adjacent spatiotemporal successor node (x', y', t+1). The occupancy status in the system. Only when all successor nodes are unoccupied (or only one is occupied)... The path branch is considered feasible only when it is reserved by itself. Path reservation and broadcasting: The planner finds the optimal path Each spatiotemporal node (x, y, t) generates a reserved declaration. This declaration includes: Robot ID. Path ID, list of involved spatiotemporal units, and time window occupied by each unit. Subsequently, This reservation statement is broadcast to all other robots that may be affected (usually spatially adjacent robots) through a communication network based on a distributed spatiotemporal conflict detection and resolution protocol. Conflict Detection and Negotiation: Robots Upon receiving from After the reservation statement is made, it is immediately maintained locally. Conflict detection is performed on the copy. If a spacetime unit in the declaration is found to be conflicting... If there is overlap between units that have been submitted or planned to be reserved, it is determined to be a spatiotemporal conflict; like If the reservation priority is lower (based on preset rules, such as low task urgency or late release time), then... It needs to revoke its own conflict reservation, release the corresponding unit in its reservation declaration, and then trigger local replanning; If both parties have the same priority or cannot be compared, a round of negotiation will be initiated. Negotiation can adopt a simple "first-come, first-served" principle, or a bidding process based on a contract network protocol: the conflicting unit will issue the task. and The incremental path cost resulting from abandoning a unit is calculated as a "bid". The robot with the smaller incremental cost wins the right to occupy the unit, while the other robot needs to replan. During the execution of the reserved path, the robot uses onboard sensors to detect any deviations in real time. Pre-marked dynamic obstacles (such as moving groups of tourists); When an obstacle is detected encroaching on the spatiotemporal region of its currently reserved path, the robot first attempts to make local trajectory adjustments, such as fine-tuning on the velocity-time plane to briefly pause or detour, thereby avoiding a collision. This adjustment generates a new local trajectory; The robot immediately broadcasts the incremental spatiotemporal reservation corresponding to the adjusted local trajectory. Simultaneously, if the local adjustment cannot avoid a collision, the robot will send an emergency avoidance request to the cooperative planning system, which includes its current precise position, velocity, and suggested avoidance direction. Upon receiving an incremental reservation or emergency request, a neighboring robot immediately checks its local plan to see if it is affected. If affected, it pauses its current movement and may trigger a localized, rapid, collaborative replanning limited to directly related robots to collectively generate a conflict-free temporary solution until the dynamic obstacle leaves the critical area. Through the aforementioned mechanism based on explicit spatiotemporal reservation and distributed negotiation, robot swarms can plan and execute movement tasks in a highly collaborative manner within a shared workspace, effectively avoiding deadlocks and collisions, and maintaining the overall operational order and efficiency of the system under dynamic disturbances. This method is the core technological guarantee for the safe and smooth operation of large-scale robot swarms in complex human environments. To accomplish complex collaborative performances or joint operations, the system uses a modular behavior tree to define and coordinate cluster behavior, as follows; High-level behavior decomposition: Decompose a macro-level collaborative task (such as "perform a welcome dance") into a series of sub-task sequences, and define the preconditions, execution logic and success criteria for each sub-task (such as "formation movement to position A" or "synchronous execution of action sequence B"). Distributed Behavior Tree Execution: The overall behavior tree instance is distributed to each robot participating in the collaboration. Each robot is responsible for executing its assigned behavior branch and continuously synchronizing its execution status (e.g., "Ready," "Executing," "Completed") via the communication network. A master coordinating node monitors the progress of the entire behavior tree or uses distributed consensus to trigger branch switching or error recovery when necessary. Real-time parameter adjustment: The parameters of the action nodes in the behavior tree (such as formation and movement speed) can be dynamically fine-tuned according to the real-time environment (such as the size of the performance area and the distance between the audience) to ensure the best display effect; To enhance the interactive experience, the system builds an engine that analyzes visitor context in real time and generates personalized interaction strategies; Multimodal context awareness: The engine accesses and analyzes multimodal data from robots and environmental sensors in real time, including: facial expression analysis results of tourists, voice emotion recognition results, rough age and gender estimates, duration of stay in front of attractions, and historical interaction records with robots; Strategy matching and generation: Based on the aforementioned contextual features, the engine matches or generates the optimal interaction solution in real time from a pre-defined interaction strategy knowledge base. The strategies in the knowledge base define the voice content and tone, screen display content, body movements (such as waving and nodding), and subsequent task suggestions (such as recommending nearby attractions or asking if photos are needed). Reinforcement learning optimization: The system records the explicit feedback (such as ratings and changes in stay time) and implicit feedback (such as whether to follow the recommendation) of tourists after each interaction, and uses them as reward signals to continuously optimize the strategy generation model through online reinforcement learning, so that the interaction strategy becomes more and more in line with the preferences of different tourist groups. Furthermore, to address the diversity and unpredictability of tourist interactions in cultural and tourism scenarios, an interaction strategy knowledge base was constructed to achieve highly personalized and context-adaptive robot interaction behavior. Specifically, the interaction strategy knowledge base adopts a multi-level structure to achieve a mapping from abstract intentions to specific actions: Basic motion element library: Defines the smallest interactive motion unit that the robot can execute. Each motion element contains its unique identifier, execution parameters, and physical constraints; Scene rule base: Based on domain knowledge, common interaction scenarios are decomposed into a set of decision trees or generative rules. Each rule is in the form of IF <condition> THEN <action sequence>. The condition is based on multimodal context features, and the action sequence is composed of basic action elements. Interaction Case Library: Stores a complete record of historical interaction instances, including the original contextual feature vectors, the strategies employed (rule ID or generation strategy), multi-dimensional evaluations of the execution results (such as task completion rate, visitor dwell time, and facial expression changes), and possible optimization suggestions. The case library supports case-based reasoning and the evaluation of new strategies. When the scenario rule base cannot match the current context (confidence score below the threshold) or creative interaction is required, the system activates the policy generation module based on a large language model. This module employs the following steps: The real-time perceived multimodal context information is converted into a structured natural language description according to a preset template, serving as an input prompt for the LLM. This prompt template is carefully designed and includes: System role definition: The LLM is explicitly instructed to act as a "cultural tourism robot interaction strategy designer"; Current status description: Key information is listed in key-value pairs, such as time: 2 PM; location: in front of the fairytale castle; visitor characteristics: a family of three, child about 5 years old, parents about 30 years old, looking happy; recent interactions: none; robot status: idle, fully charged; Interaction goals and constraints: Specify the core goals of this interaction (such as "enhancing visitor immersion") and the constraints that must be followed (such as "actions must be safe, voice must be concise, and total duration must not exceed 60 seconds"). Output format specifications: Strictly require LLM to output using a specified JSON Schema strategy, which defines the structure of the action sequence, the type of each action, and parameter fields; The constructed prompts are input into a large language model (such as a dedicated model that has been fine-tuned with instructions). Based on its internal knowledge and understanding of the context, the LLM generates a policy JSON object that meets the required format. The generated strategy first passes through a security and compliance filter to check for inappropriate information. Then, it undergoes rapid feasibility verification in a digital twin simulation environment, simulating the execution of the action sequence to check for violations of physical constraints (such as joint limits and collision risks) or logical errors. Once verification is successful, the strategy is marked as usable. The validated strategy JSON is sent to the instruction compiler, which maps and compiles each abstract action object into a specific instruction sequence that can be recognized by the underlying robot control platform, and assigns it to the corresponding execution module (speech synthesis, motion control, screen rendering, etc.). Simultaneously, the system employs a strategy arbitrator to dynamically select between rule base matching and LLM generation based on the certainty of the current scenario. The effectiveness of each interaction is automatically evaluated using metrics such as visitor feedback and task completion rate. The evaluation results are used for: Rule base optimization: Highly effective generation strategies can be abstracted, generalized, and transformed into new rules to be added to the scenario rule base; LLM model fine-tuning: Using high-quality interaction cases (context + success strategy) as training data, the LLM is continuously fine-tuned under supervision to make its output more accurate and better meet business needs; Prompt Template Iteration: Based on the quality analysis of the generation strategy, continuously optimize the prompt engineering template to improve instruction compliance; Through the aforementioned hybrid architecture, the system ensures both efficient and reliable responses in common scenarios, while also providing creative capabilities for open and novel situations. Among the key technical aspects for achieving stable application of this module are designing effective prompt templates to guide the LLM output of structured, safe, and feasible strategies, and constructing an efficient simulation verification environment to quickly assess the physical feasibility of the generated strategies. This design enables robot interaction to evolve from a pre-programmed, fixed pattern into a continuously learning and dynamically evolving intelligent service process. The predictive maintenance layer connects the data layer and the digital twin layer. It analyzes data through machine learning models to enable fault warning, health assessment, remaining life prediction and maintenance strategy recommendation for robot components. The predictive maintenance layer employs a hybrid approach based on machine learning and statistical processes to achieve real-time quantitative assessment of the health status of robot components, specifically including the following methods: Real-time anomaly detection: For high-frequency sensor data streams (such as vibration and current), unsupervised learning algorithms (such as isolated forest, single-class support vector machine, or autoencoder) are used to establish a feature baseline model under normal operating conditions. After feature extraction, real-time data is input into this model. If the reconstruction error or deviation score exceeds the dynamic threshold, an early anomaly alarm is triggered, indicating potential degradation, earlier than traditional threshold alarms. Comprehensive Health Index Calculation: By integrating multiple relevant but heterogeneous sensor indicators (such as motor three-phase current balance, gearbox vibration kurtosis, and bearing temperature trend), a comprehensive health index ranging from 0 to 100 is calculated using principal component analysis or a domain-knowledge-based weighted fusion algorithm. This index intuitively reflects the overall health status of a component or subsystem, and its slow downward trend is used to assess progressive degradation. Fault mode identification: Once an anomaly is confirmed, the system calls a multi-class fault diagnosis model (such as processing vibration spectrum based on convolutional neural network, or processing multi-dimensional feature vector based on gradient boosting tree) to compare the current data mode with the known fault database (such as bearing inner ring spalling, gear tooth breakage, poor lubrication), and outputs the most likely fault mode and confidence level, providing direction for precise maintenance. For critical wear-prone components, the system establishes a performance degradation trajectory model and predicts their remaining service life, as follows: Degradation Feature Extraction and Tracking: One or more features sensitive to the degradation of specific components are selected as health indicators (such as the energy of the fault frequency amplitude in the vibration envelope spectrum, the slow decay rate of motor efficiency). Noisy HI measurements are smoothed using state estimation techniques such as Kalman filters or particle filters, and their evolution trajectory is tracked in real time. The RUL prediction model achieves probabilistic inference of the failure time of key components by building and continuously optimizing a prediction model based on sequence learning. To address the temporal evolution characteristics of component health indicators, a regression prediction model based on the fusion of bidirectional long short-term memory network and attention mechanism is constructed. The construction process is as follows: Data preparation and sequence construction: Collect complete lifecycle data of historical, similar components from commissioning to complete failure. Extract their health indicator sequences. The primary input feature is T, where T represents the failure time. Simultaneously, the component's cumulative operating time, load cycles, and other operating condition information are used as auxiliary feature sequences. The complete HI sequence is divided into multiple fixed-length sliding window samples, each labeled with the remaining time (in hours or cycles) between the end of the window and the component's failure time, and then normalized. The model input layer receives a normalized HI sequence window. Then, a two-layer Bi-LSTM network encodes the sequence, capturing its long-term forward and backward dependencies. The output of the Bi-LSTM is fed into a multi-head self-attention layer, which automatically learns the importance weights of the HI values ​​at different time points in the sequence to the final RUL prediction, thereby focusing on key stages of accelerated degradation or mutation. Finally, the output of the attention layer is regressed through a fully connected layer to output a scalar, namely the normalized RUL prediction value. Mean squared error is used as the loss function, and early stopping is employed to prevent overfitting. Dropout regularization is applied to the model during training, and time-series cross-validation is used to ensure the model's robustness. When performing RUL prediction for a new component, the system performs the following steps to provide a point estimate and its confidence interval: Sequence Input and Point Estimation: Construct a sequence of all HI observations of the target component from the initial operation to the present, using the same length as during training (padding is used if necessary). Input this sequence into the trained prediction model to obtain the normalized RUL point estimate. Then, it is reversed and normalized to the actual unit of time. Probability distribution estimation: To quantify the uncertainty of the prediction, the system employs a Bayesian deep learning method. Specifically, Monte Carlo Dropout is enabled on the aforementioned neural network during the inference phase. That is, the same input sequence is propagated forward multiple times, with a portion of neurons randomly discarded each time (Dropout enabled), thereby obtaining a set of RUL prediction samples. The mean of this sample set is the final point estimate, and its distribution approximately reflects the uncertainty of the prediction. By fitting this distribution to a two-parameter Weibull distribution and outputting its scale and shape parameters, the RUL prediction interval at a given confidence level (e.g., 90%) can be calculated. ; To adapt to batch variations in component manufacturing, changes in the operational environment, and new failure modes, the prediction model possesses online evolution capabilities, specifically including: New failure cases trigger incremental learning: When any component of the same type in the system experiences a real failure, its complete, tagged HI sequence will be automatically added to the incremental learning queue. Safety Validation and Model Update: The system initiates a model update process periodically (or when the queue reaches a certain size). First, the performance of the current model is evaluated on an independent historical validation set. Then, the model is fine-tuned with a small learning rate using data from the incremental learning queue. After fine-tuning, it is evaluated again on the validation set and a new subset of data. The new model is deployed to replace the old model only if the fine-tuned model is significantly better than or at least no worse than the original model in key metrics; otherwise, the original model is retained and the data anomaly is recorded.

[0027] Model version management and rollback: All deployed models are versioned. Each update saves complete model parameters, training data summaries, and performance reports. If a new model exhibits systematic prediction bias during online monitoring, the system supports automatic rollback to the previous stable version. Through the aforementioned model construction, uncertainty quantification, and adaptive update process, the RUL prediction module can provide a forward-looking remaining lifetime assessment that includes confidence information, and it can continuously improve itself as system operating data accumulates, thus providing a core basis for dynamic and scientific maintenance decisions. Among these, the key directions for further improvement in enhancing the module's performance and reliability are: how to efficiently integrate attention mechanisms to capture key degradation inflection points, the Monte Carlo Dropout repetition count and uncertainty interval calibration method, and strategies to prevent catastrophic forgetting in incremental learning. The application service layer connects the above layers and provides functions such as a visual monitoring platform, mobile management terminal, maintenance work order system and data analysis report; The application service layer includes a visual monitoring platform, mobile management terminals, a maintenance work order system, and a data analysis and reporting center; The visualization monitoring platform, developed based on WebGL or a similar graphics engine, provides managers with a comprehensive situational awareness of the entire robot cluster, from macro to micro levels, and has the following functions: Multi-layered visual interaction, providing the following views: Park-level macro view: Using a 3D high-precision map as the base map, dynamically display the real-time location distribution, movement trajectory, global task hotspots, and critical infrastructure status (such as charging station utilization rate) of all robots. Robot-level microscopic view: Select any robot and its digital twin model will be highlighted, and its real-time status dashboard will be displayed simultaneously, including: core sensor reading curves (such as current, temperature), health index trends, current task details, and first-person real-time video stream transmitted from the airborne camera; It also provides alarm and event visualization functions, which can intuitively locate faulty robots, congested areas or abnormal environmental events through icon flashing, color changes (such as red warning) and animation effects in three-dimensional space, and support clicking to view details; It also supports cross-dimensional data fusion and display, allowing business data layers to be overlaid on a 3D scene. For example, the "tourist heat map" can be overlaid with the "robot task distribution map" to help analyze the service supply and demand matching; or the historical trajectory playback and vibration spectrum changes can be played synchronously for fault review. The mobile management terminal is designed for on-site inspection and emergency response personnel, providing a dedicated application based on smartphones or tablets, primarily offering: Task reception and execution: Receives work orders from the collaborative operations management layer in real time (e.g., "Go to point A to handle robot R001 being stuck"), with built-in navigation guiding personnel to the site. Provides standardized on-site checklists and handling instructions (which can be linked to maintenance step animations in the digital twin model); Real-time data acquisition and reporting: The terminal integrates QR code / NFC scanning functionality for rapid identification of robots or components. It supports on-site photography, audio recording, and form filling, and allows manual input of simple inspection data (such as visual oil level measurement and manual temperature measurement). All information is transmitted back to the system in real time, updating the corresponding work order status and equipment electronic history. Augmented Reality Assistance: On devices with AR capabilities, applications can use the camera to overlay virtual information onto the real robot, such as highlighting the location of screws to be operated, dynamically displaying the internal cable routing, or displaying key parameters in the form of a floating window, greatly improving the accuracy and efficiency of maintenance operations. The maintenance work order system is a core component for end-to-end, closed-loop maintenance operation management, deeply integrated with the predictive maintenance layer, and has the following functions: Intelligent work order generation: The system can automatically create preventative maintenance work orders based on the output of the predictive maintenance layer (fault warnings, RUL prediction results); or corrective maintenance work orders can be created from manual alarms from the monitoring platform and mobile terminals. Work orders automatically associate with equipment information, historical records, recommended maintenance plans, and a list of required spare parts. Work order scheduling and closed-loop management: Based on the urgency of the work order, required skills, spare parts inventory, and personnel location, combined with the scheduling algorithm of the collaborative operations management layer, the system intelligently assigns work orders to suitable maintenance personnel or teams. Work order status (pending assignment, in progress, awaiting acceptance, completed) is fully traceable, and mandatory information entry is required for key nodes (such as fault cause confirmation and barcode scanning of replaced parts) to ensure process closure and data integrity. Knowledge Accumulation and Analysis: Completed work orders, along with all process data, are automatically archived to form an electronic maintenance history for equipment. The system periodically analyzes work order data, compiling statistics on indicators such as average repair time, spare parts consumption rate, and repeat failure rate for various types of faults, providing data support for optimizing maintenance strategies and improving spare parts inventory models. The data analysis and reporting center provides management with flexible self-service data analysis and visualization report generation capabilities, as detailed below: Templated and Custom Reports: Includes various built-in business analysis report templates, such as "Daily / Monthly Robot Operation Performance Report," "Fault Statistics and TOP Analysis," and "Maintenance Cost Analysis Report." It also provides a drag-and-drop report designer, allowing users to generate personalized reports based on thematic tables in the data warehouse, customizing dimensions, metrics, and chart types. Interactive data exploration: Supports drill-down analysis of key indicators (such as overall equipment efficiency). For example, clicking on a declining OEE value for a certain month allows you to drill down to view the details of robot downtime for that month, and further drill down to the complete sensor data sequence of a specific failure to achieve root cause tracing. Automatic push and alert reports: Configurable automatic generation and push of reports on a regular basis (such as daily morning reports) or triggered by events (such as a surge in a certain type of failure). Reports not only present data, but can also generate concise text analysis summaries based on built-in rules, pointing out key changes, potential risks, and improvement suggestions.

[0028] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A collaborative operation management and predictive maintenance system for group robots based on the Internet of Things, characterized in that, include: The physical layer includes multiple robots and multi-source sensors and IoT communication modules mounted on them, which collect the robots' own status data, environmental data and task interaction data. The data layer communicates with the physical layer to receive, clean, fuse, and store real-time and historical operational data from the physical layer. A digital twin layer, connected to the data layer, constructs and maintains a digital twin model synchronized with the physical robot based on the data processed by the data layer. The digital twin model includes a geometric model, a behavioral model, and a fault model. The collaborative operation management layer is connected to the data layer and the digital twin layer respectively, and performs dynamic task allocation, path collaborative planning and behavior coordination for the robot cluster based on the digital twin model and / or the data of the data layer. The predictive maintenance layer is connected to the data layer and the digital twin layer respectively. By analyzing the data of the data layer and / or the model output of the digital twin layer, it realizes fault warning, health assessment and remaining life prediction of robot components. The application service layer connects to the data layer, digital twin layer, collaborative operation management layer, and predictive maintenance layer, respectively, and provides functions such as visual monitoring, mobile management, maintenance work orders, and data analysis.

2. The IoT-based group robot collaborative operation management and predictive maintenance system according to claim 1, characterized in that, The data layer includes an IoT gateway cluster, a streaming data processing engine, and a hierarchical storage module. The IoT gateway cluster receives data from the physical layer and performs protocol parsing, preliminary filtering, and aggregation. The streaming data processing engine connects to the IoT gateway cluster to clean, align, extract and fuse data to form a comprehensive state feature vector of the robot. The hierarchical storage module is connected to the streaming data processing engine and stores the processed data into time-series databases, relational databases, and object storage according to access frequency and value density.

3. The IoT-based group robot collaborative operation management and predictive maintenance system according to claim 2, characterized in that, The construction and synchronization of the digital twin layer includes: the geometric model is constructed based on the robot's original design data to create its three-dimensional model, and kinematic modeling and lightweight processing are performed according to the operating state; The behavior model integrates the robot's kinematics, dynamics, and perception and decision-making algorithms to simulate its dynamic behavior and intelligent responses in a virtual environment. The fault model is based on a fault mode and effects analysis library and parametric fault simulation to simulate equipment performance degradation and fault propagation. The digital twin layer achieves synchronization with the physical robot through real-time data-driven operation and supports historical playback and future projection through a simulated clock.

4. The IoT-based group robot collaborative operation management and predictive maintenance system according to claim 3, characterized in that, The digital twin layer also includes a parameter self-learning and calibration module; When a physical robot experiences a real malfunction and is repaired, the parameter self-learning and calibration module extracts the full-cycle data of the malfunction and replays it in the digital twin environment. By comparing the real fault feature sequences with the fault model simulation output sequences, an optimization algorithm is used to automatically adjust the parameters of the fault model, so that the simulation data matches the real data, and the optimized parameters are updated to the fault model parameter library.

5. The IoT-based collaborative operation management and predictive maintenance system for group robots according to claim 1, characterized in that, The collaborative operation management layer includes a task allocation module and a path planning module; The task allocation module adopts a multi-agent deep reinforcement learning architecture. Based on the global state vector including the robot state matrix, task queue feature matrix and environmental context tensor, it outputs task allocation decisions to optimize task completion rate, response time, load balancing and energy efficiency. The path planning module performs path search based on a spatiotemporal map, and achieves collision-free collaborative movement of multiple robots by reserving spatiotemporal paths with the broadcast robot and using a distributed conflict detection and negotiation mechanism.

6. The IoT-based collaborative operation management and predictive maintenance system for group robots according to claim 1, characterized in that, The collaborative operation management layer also includes an interaction strategy generation module; The interaction strategy generation module generates personalized interaction strategies based on real-time multimodal tourist context data, matching from a preset interaction strategy knowledge base or generating them through a large language model. After the generated strategy is filtered for security and compliance and verified in a digital twin simulation environment, it is compiled into executable instructions for the robot, and the knowledge base and the generated model are optimized based on the evaluation feedback of the interaction effect.

7. The IoT-based group robot collaborative operation management and predictive maintenance system according to claim 1, characterized in that, The predictive maintenance layer includes: a real-time anomaly detection module that uses an unsupervised learning algorithm to perform early anomaly detection on the sensor data stream; The fault mode identification module uses a multi-classification model to identify the fault modes of confirmed anomalies. The remaining useful life prediction module uses a sequence learning model to predict the remaining useful life of components based on time-series data of component health indicators, and provides the probability distribution and confidence interval of the prediction results.

8. The IoT-based collaborative operation management and predictive maintenance system for group robots according to claim 7, characterized in that, The remaining useful life prediction module adopts a regression prediction model based on bidirectional long short-term memory network and attention mechanism; The module enables Monte Carlo Dropout during the inference phase, obtaining the distribution of predicted samples through multiple forward propagations to quantify prediction uncertainty. The module supports incremental learning. When real-world failures occur, new data is used to perform safety verification and fine-tuning updates on the model, and model version management is implemented.

9. The IoT-based collaborative operation management and predictive maintenance system for group robots according to claim 1, characterized in that, The application service layer includes: a visualization monitoring platform, used to display the real-time location, status, alarms, and fused data layers of the robot cluster based on a 3D map; Mobile management terminals are used to push maintenance work orders to on-site personnel, provide navigation and augmented reality assistance, and support on-site data collection and transmission. The maintenance work order system is used to automatically generate work orders based on the output of the predictive maintenance layer or manual alarms, and to realize intelligent scheduling, closed-loop management and knowledge accumulation of work orders; The Data Analysis Report Center provides templated and customizable data analysis reports, interactive data exploration, and automated report delivery.

10. The IoT-based group robot collaborative operation management and predictive maintenance system according to claim 1, characterized in that, The system also includes a unified spatiotemporal data model for defining and managing robot entities, sensor entities, observation data entities, task entities, and event entities; All time- and space-related data are attached with uniform time labels and spatial coordinate labels; The unified spatiotemporal data model serves as the semantic basis for data exchange and association among the data layer, digital twin layer, collaborative operation management layer, and predictive maintenance layer.

Images