A method and system for constructing a metaverse digital twin supporting real-time interaction

Abstract

The application discloses a kind of meta-universe digital twin construction method and system supporting real-time interaction, belong to computer technology field.The method includes: through the original data stream of multiple sensors deployed in physical entity acquisition;In edge side, semantic event is identified based on kinematics parameter calculation and multi-condition joint determination, and key parameter is extracted to generate lightweight semantic instruction package;After instruction package is transmitted to cloud, corresponding virtual model in parameterization model library is executed according to its driving parameter, and the hierarchical relationship of associated object is updated to realize visual follow-up;Motion interpolation and smoothing processing are carried out between instructions based on physical constraint prediction algorithm, and forward prediction is carried out using timestamp to compensate network delay.The system includes multi-source data acquisition, edge semantic extraction, instruction transmission, virtual reconstruction and physical calibration and other modules.The application reduces data transmission load, improves the real-time performance and motion fluency of virtual-real synchronization, and enhances the adaptive ability and interaction security of system.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a method and system for constructing a metaverse digital twin that supports real-time interaction. Background Technology

[0002] With the development of information technology, the metaverse, as a new form of internet that integrates the virtual and the real, has significant value in fields such as the industrial internet and smart cities. Digital twin technology, by constructing virtual mappings of physical entities, enables the monitoring, simulation, and interaction of the real world, and is a key means to support the virtual-real integration of the metaverse. Specifically, a digital twin system that achieves real-time interaction needs to complete multi-source data acquisition, 3D reconstruction, and dynamic presentation to ensure that the virtual model can quickly and accurately respond to changes in physical entities, providing users with an immersive experience.

[0003] Currently, most common methods for constructing digital twins employ a full data synchronization mechanism. This involves frequently collecting raw data such as point clouds and videos of physical entities and uploading them to the cloud for 3D reconstruction and rendering. However, this approach faces significant limitations in practical applications: First, the massive amount of raw data puts enormous pressure on network bandwidth, making it difficult to support concurrent multi-device scenarios. Second, the data transmission and processing introduce high latency, preventing accurate synchronization between the virtual world and the physical entity. Furthermore, the cloud or client needs to process complex unstructured data, resulting in high computational and rendering overhead, which is particularly detrimental to stable operation on resource-constrained mobile or edge devices.

[0004] Therefore, how to reduce data transmission volume and computational load while ensuring real-time interaction and visual realism has become a technical problem that the Metaverse digital twin system urgently needs to solve. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for constructing a metaverse digital twin that supports real-time interaction, employing the following technical solution.

[0006] Firstly, a method for constructing a metaverse digital twin that supports real-time interaction includes:

[0007] S1 collects raw data streams containing motion state and environmental information in real time through multiple types of sensors deployed on physical entities;

[0008] S2, the original data stream is processed at the edge, and based on the real-time calculation of kinematic parameters and the joint determination of multiple conditions, the semantic events executed by the physical entity are identified, and the key physical parameters associated with the semantic events are extracted to generate a structured lightweight semantic instruction package.

[0009] S3, transmit the lightweight semantic instruction package to the cloud server;

[0010] S4, the cloud server parses the lightweight semantic instruction package, drives the corresponding virtual model in the parameterized model library to perform the corresponding action according to the device identifier and action type contained therein, and updates the hierarchical relationship of related objects in the virtual scene to achieve visual follow-up;

[0011] S5, the cloud server, uses a physical constraint prediction algorithm to interpolate and smooth the motion of the virtual model between continuously received semantic instruction packets, and performs forward prediction based on the timestamps in the instruction packets to compensate for network transmission delay.

[0012] Preferably, in step S2, semantic events are identified based on real-time calculation of kinematic parameters and joint determination of multiple conditions, specifically including:

[0013] The first and / or second derivative operations are performed on the collected joint angle or displacement data to calculate the velocity and acceleration parameters in real time.

[0014] Simultaneously monitor the rate of change of force / torque sensor data stream;

[0015] When the calculated kinematic parameters conform to the predefined change model of a specific action within a preset time window, and the relevant force / torque change rate simultaneously meets the preset threshold, they are jointly determined as the corresponding semantic event.

[0016] Before generating the final instruction, the confidence level of the identified semantic events is assessed. The assessment factors include whether the movement trajectory is located in a compliant operating area and whether it conforms to the preset process flow sequence.

[0017] Preferably, in step S2, generating a structured lightweight semantic instruction package specifically includes:

[0018] The semantic event type identifier, event start timestamp, target object identifier, and spatial coordinates and attitude parameters extracted from the current data are encapsulated into a binary data packet according to a predefined fixed field format. The total number of bytes in the data packet is in the tens to hundreds of bytes range.

[0019] Preferably, in step S4, the corresponding virtual model in the parameterized model library is driven to perform a corresponding action, specifically including:

[0020] Based on the parsed device identifier, the corresponding virtual model is located in the parametric model library. The parametric model library includes at least a geometry layer that stores the geometric appearance of the model, an animation layer that predefines standard motion skeletal animation sequences, and an attribute layer that manages dynamic parameters.

[0021] Based on the parsed action type, control the animation state machine of the virtual model to switch to the corresponding state;

[0022] Using the parsed target position and posture parameters as constraints, the joint motion parameters required to drive the skeletal animation sequence are calculated through inverse kinematics algorithm, thereby driving the model in the geometry layer to produce continuous movements.

[0023] Preferably, in step S4, the hierarchical relationship of related objects in the virtual scene is updated to achieve visual tracking, specifically as follows:

[0024] Based on the target object identifier in the semantic instruction package, the corresponding virtual object model is found in the virtual scene, and the virtual object model is set as a child node of the coordinate system of the virtual model end of the action in the scene graph data structure, so that the virtual object model can move synchronously with the virtual model end.

[0025] Preferably, in step S5, interpolation and smoothing processing based on the physical constraint prediction algorithm is performed, specifically including:

[0026] Based on the pose parameters in the most recently received semantic instruction packet and the historical motion state of the virtual model, the predicted motion trajectory of the virtual model within the instruction interval is calculated based on the established simplified dynamic model.

[0027] In a virtual environment, a collision model is established for the virtual model and key obstacles, and collision detection is performed on the predicted motion trajectory.

[0028] If a potential collision is detected, the virtual model's trajectory is automatically corrected, and the collision warning information is encapsulated as a feedback command and sent to the edge.

[0029] Preferably, in step S5, forward prediction based on the timestamp in the instruction packet is performed to compensate for network transmission delay, specifically including:

[0030] The difference between the generation timestamp of the semantic instruction packet and the current rendering time in the cloud is used to calculate the end-to-end latency;

[0031] When the delay exceeds a preset threshold, the pose of the virtual model from the moment the instruction was generated to the current rendering moment is predicted based on the simplified dynamics model and the most recent instruction parameters.

[0032] The current display pose of the virtual model is transiently adjusted to the predicted pose.

[0033] Preferred options also include:

[0034] The network status is monitored in real time through the global status monitoring module;

[0035] When network performance indicators are detected to be deteriorating, a control command is sent to the edge side, triggering the edge side to merge multiple atomic semantic events with continuous logical relationships identified within a preset time window into a high-order composite semantic event, and generate a corresponding composite semantic command package for uploading.

[0036] Preferably, the physical entity is an industrial robotic arm or an automated guided vehicle; the multiple types of sensors include at least two or more of the following: encoders, force / torque sensors, vision sensors, and positioning sensors.

[0037] Secondly, a metaverse digital twin construction system supporting real-time interaction includes:

[0038] A multi-source data acquisition module, deployed on the physical entity side, is used to acquire raw data streams in real time;

[0039] An edge semantic extraction module, deployed on an edge computing node, is used to perform the kinematic parameter calculation, multi-condition joint determination and confidence assessment as described in claim 2, and to generate a structured lightweight semantic instruction package;

[0040] The instruction transmission module is used to reliably transmit the lightweight semantic instruction package to the cloud.

[0041] The virtual reconstruction module, deployed on a cloud server, includes a parameterized model library and is used to perform the model-driven, inverse kinematics solution as described in claim 4 and the scene hierarchy update as described in claim 5.

[0042] A physical calibration module, deployed on a cloud server, is used to perform the physical constraint prediction, collision detection and trajectory correction as described in claim 6, and the delay compensation prediction as described in claim 7.

[0043] The global status monitoring module is used to execute the network monitoring and command merging strategy triggering described in claim 8.

[0044] In summary, this application includes at least one of the following beneficial technical effects:

[0045] 1. This application uses edge semantic extraction technology to identify and abstract the continuous raw motion data stream of physical entities into discrete high-level semantic events in real time. Only a structured lightweight instruction package containing action type and key pose parameters is uploaded. This reduces the amount of data to be transmitted from megabytes of raw sensor data to hundreds of bytes, fundamentally alleviating network bandwidth pressure. This enables high-frequency and stable virtual-real data synchronization even in narrowband networks or large-scale device concurrency scenarios, laying the foundation for the large-scale deployment of metaverse digital twins.

[0046] 2. By introducing a physical constraint prediction algorithm based on a dynamic model and a forward delay compensation mechanism, this application enables the cloud system to perform continuous and smooth motion interpolation between received discrete semantic instructions and actively offset the impact of network transmission delay. This solves the problem of "jumping" or "ghosting" in virtual model motion caused by instruction interruption and network jitter, improves the visual coherence and real-time synchronization accuracy of the virtual world's reproduction of physical entity motion, and enhances the user's immersive interactive experience.

[0047] 3. This application designs a virtual-real two-way closed-loop calibration logic that includes real-time collision detection and trajectory correction, as well as an adaptive mechanism that dynamically adjusts the semantic abstraction level according to network conditions. The collision warning in the virtual space can be fed back to the physical side for intervention, ensuring operational safety. The system can intelligently merge fine-grained instructions when the network is congested, maintain the operation of the core logic, and jointly improve the reliability, safety and overall stability of the system in complex industrial environments, achieving an adaptive balance between performance and resource consumption. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a method for constructing a metaverse digital twin that supports real-time interaction, according to the present invention.

[0049] Figure 2 This is a schematic diagram of the physical constraint prediction and real-time interactive calibration process in this invention;

[0050] Figure 3 This is a schematic diagram of the multi-level interaction relationship and data flow between physical entities and the metaverse virtual space in this invention. Detailed Implementation

[0051] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of specific embodiments based on the present invention is provided in conjunction with the accompanying drawings and preferred embodiments.

[0052] Example 1

[0053] This embodiment takes the remote monitoring and real-time digital twin construction of a six-axis industrial robotic arm in an industrial automated production line as the application scenario. In this scenario, the physical entity is the industrial robotic arm that performs precision assembly tasks, and the metaverse environment is a virtual factory workshop located on a cloud server.

[0054] In terms of system composition, the metaverse digital twin construction system supporting real-time interaction constructed in this embodiment includes a multi-source data acquisition module, an edge semantic extraction module, an instruction transmission module, a virtual reconstruction module, a physical calibration module, and a global status monitoring module.

[0055] The multi-source data acquisition module is connected to the underlying controller of the robotic arm via an industrial bus interface to acquire the robotic arm's raw state data in real time. This data mainly includes the following four categories:

[0056] 1. Joint angle data: It comes from 16-bit absolute encoders distributed at the 6 joints of the robotic arm. The encoder outputs pulse signals, which directly reflect the real-time rotation angle of each joint. The system collects the pulse signals at a sampling frequency of 1000 Hz.

[0057] 2. End-effector force and torque data: Acquired by a six-axis torque sensor deployed on the end effector of the robotic arm. This sensor measures the force components and torque components in three directions in real time, which is used to monitor the force state of the robotic arm during the assembly process.

[0058] 3. Visual perception data: Depth cameras are placed above the workspace to collect a stream of depth images, including the robotic arm itself and the surrounding parts to be assembled, at a rate of 60 frames per second, forming continuous environmental perception information.

[0059] 4. Global positioning data: Using a laser tracker mounted on the robotic arm base, the precise spatial positioning coordinates of the robotic arm in the factory's global coordinate system are obtained, thereby determining its initial pose and providing a reference for subsequent position mapping in virtual space.

[0060] All collected raw data is aligned with timestamps and transmitted to the edge for further processing.

[0061] The above four types of data serve different semantic extraction and reconstruction needs at the edge: joint angle and global positioning data are mainly used to reconstruct the overall pose and motion trajectory of the robotic arm; end force and torque data are used to identify the interaction state between the robotic arm and the environment, such as grasping, assembly, and abnormal force; and visual perception data are used to assist in environmental understanding and action verification, such as confirming whether the position of the target part matches the robotic arm's motion intention.

[0062] This multi-source complementary data system enables more accurate identification of high-level semantic events at the edge, providing a reliable basis for subsequent lightweight instruction generation and cloud-based parameterized reconstruction, and effectively avoiding misidentification or information loss due to the limitations of single-type data.

[0063] The edge semantic extraction module is deployed in an edge computing gateway located near the industrial production line. This gateway is equipped with a high-performance graphics processor and tensor processing unit to provide computing power support for real-time data processing.

[0064] The edge semantic extraction module runs an edge semantic engine, whose core task is to transform the continuous physical state changes of the robotic arm into discrete action semantic instructions with clear meanings.

[0065] In the specific working process, the edge semantic extraction module first performs first and second derivative operations on the pulse signals collected by the joint encoder, thereby calculating the angular velocity and angular acceleration of each joint of the robotic arm in real time.

[0066] The system continuously monitors these motion parameters. When it detects that the angular velocity change of the second and third joints of the robotic arm is within a 100-millisecond time window and conforms to the acceleration curve model preset by the system, and at the same time, it determines that the displacement vector of the end effector is pointing to the part storage area through pose calculation, the edge semantic engine determines that the robotic arm is performing a grasping operation.

[0067] It should be noted that the acceleration curve model is derived from the statistical analysis of historical motion data of the robotic arm under typical grasping actions, reflecting the relationship between the angular velocity change threshold and time of the second and third joints during the acceleration phase; the pose calculation is performed by using the forward kinematics model of the robotic arm to calculate the position and pointing vector of the end effector in the global coordinate system in real time based on the joint angles.

[0068] Furthermore, the aforementioned parts storage area is predefined in the system as a cubic region in three-dimensional space, and its coordinate range is determined by visual perception data or production system configuration. In addition, during the judgment process, the system will also refer to whether the reading of the end torque sensor is within the typical force range of the grasping action, and whether the target parts in the visual data are in an operable state, thereby comprehensively improving the confidence of semantic recognition.

[0069] At this point, the system no longer transmits all the raw data generated by the joint encoder within those 100 milliseconds to the cloud, but instead performs key information extraction.

[0070] The extracted information includes the start timestamp of this grasping action, the unique identification code of the target part, the three-dimensional spatial coordinates of the grasping point at the end of the robotic arm, and the quaternion of the attitude rotation of the end effector at the moment of grasping.

[0071] The above information is packaged together to generate a structured lightweight data packet, which mainly contains an action type identifier and the aforementioned key pose parameters.

[0072] By identifying and extracting semantic events from continuous raw data streams, data volume compression is achieved. Raw sensor data that originally needed to be transmitted in megabytes is compressed into semantic instruction packets of only a few hundred bytes.

[0073] Its core lies in understanding and abstracting the physical motion intent at the edge, and only uploading the understood high-level semantics and key parameters, thereby laying a solid foundation for subsequent low-bandwidth transmission and efficient cloud reconstruction.

[0074] The instruction transmission module performs serialization and integrity verification on the lightweight data packets generated by the edge semantic extraction module, and is responsible for their reliable transmission to the server.

[0075] The instruction transmission module encapsulates data packets in a binary stream format. The structure of the data packets is strictly defined, with a 2-byte header start identifier followed by a 4-byte device unique identifier, used to distinguish different physical entities in the cloud.

[0076] Next is a 1-byte semantic instruction encoding, representing the specific action type, such as grab or place; followed by a 1-byte parameter field length indicator, which indicates the number of bytes occupied by subsequent key parameters.

[0077] The list of key pose parameters is allocated a fixed length of 24 bytes to carry core information such as timestamps, spatial coordinates, and rotation quaternions; at the same time, a 2-byte cyclic redundancy check code is appended to the end of the data packet so that the receiver can verify whether the data has been corrupted during transmission.

[0078] After encapsulation, the instruction transmission module sends the data packets to the remote server via a low-latency 5G industrial private network.

[0079] This highly structured, lightweight data packet design directly serves the core advantage of this invention. Fixed-length field definitions and a compact binary format result in extremely small data packet sizes, typically only tens of bytes, which fundamentally alleviates the network bandwidth pressure caused by traditional full data transmission.

[0080] Meanwhile, clearly defined fields, such as the device's unique identifier and semantic instruction encoding, enable the cloud-based virtual reconstruction module to perform rapid parsing and accurate addressing; key pose parameters are stored centrally in a fixed format, making it easy for the virtual reconstruction module to directly extract and drive the parameterized model, greatly reducing the computational overhead and latency of cloud data processing.

[0081] Therefore, the instruction transmission module is not only a data transport channel, but also a key link to realize edge semantic abstraction and cloud parameterized driven collaborative mode, ensuring that the high-level semantic intent extracted from the edge can be accurately received and executed by the cloud system with the lowest network burden and the highest processing efficiency.

[0082] To ensure the orderliness and reliability of transmission, the instruction transmission module adds a sequential number to each data packet before sending it.

[0083] At the receiving end, the system detects whether data packets are lost based on this sequence number. Once a packet loss is detected, the receiving end will initiate a retransmission request to the sending end, and the instruction transmission module will be responsible for retransmitting the corresponding data packets until the transmission is successful.

[0084] In summary, the instruction transmission module ensures that semantic instruction data can be delivered to the cloud processing platform efficiently, reliably, and with low latency by defining a well-defined data packet structure, utilizing a high-performance industrial network for transmission, and supplementing it with sequential numbering and packet loss retransmission mechanisms. This provides a crucial data pathway guarantee for the accurate and real-time reconstruction of virtual space.

[0085] The virtual reconfiguration module is deployed on the Metaverse cloud processing platform and is responsible for managing a pre-built parametric model library that covers digital descriptions of all equipment in the factory.

[0086] The parametric model library is organized using a layered abstract architecture, specifically divided into a bottom layer, a middle layer, and a top layer. Specifically:

[0087] 1. The bottom layer is the geometric topology layer, which mainly stores the high-precision 3D mesh model of the storage device, defining the model's visual shape and surface details;

[0088] 2. The middle layer is the skeletal animation layer, where the system predefines animation state machines for various standard actions of movable devices. For example, for a robotic arm, it predefines skeletal animation sequences and their triggering logic for a series of typical actions such as grasping, placing, rotating, and resetting.

[0089] 3. The top layer is the attribute control layer, which manages various variable attributes and parameters related to the device and provides an interface for dynamic status updates.

[0090] This layered architecture enables the virtual reconstruction module to efficiently process semantic instructions. The geometric topology layer provides the foundation for visual presentation; the skeletal animation layer contains the logic and data of the actions. When an instruction triggers a specific action state machine, this layer outputs the corresponding skeletal transformation sequence; and the attribute control layer binds and updates dynamic parameters such as motion speed and force magnitude in real time.

[0091] The three layers of data are interconnected with spatiotemporal information through shared model identifiers, ensuring that instructions can simultaneously drive consistent changes in the model's form, actions, and attributes.

[0092] When the virtual reconstruction module receives the "execute capture" semantic instruction packet from the network, its processing flow starts immediately. First, it parses the unique device identification code in the data packet and quickly locates the corresponding robotic arm digital model in the virtual factory scene based on the identification code.

[0093] Next, the virtual reconstruction module manipulates the animation controller of the digital model to switch its internal state machine from the current "idle" state to the "grab" state, indicating that the model is ready to execute the grab animation.

[0094] Subsequently, the virtual reconstruction module extracts key pose parameters from the semantic instruction package, including the target position and end effector posture. Using these parameters, the module performs calculations through inverse kinematics algorithms to deduce the precise angles that each joint of the virtual robotic arm needs to achieve.

[0095] It should be noted that the above "solution" uses the target position and end pose in the instruction package as constraints, and combines them with the joint motion range contained in the predefined grabbing motion skeletal animation sequence to solve the problem. The calculated accurate joint angle dataset will be directly used to drive the corresponding grabbing animation state machine in the skeletal animation layer, thereby causing the 3D mesh model in the geometric topology layer to make corresponding and continuous motion deformations.

[0096] Finally, the virtual reconstruction module drives the 3D model, causing each joint to move according to the calculated angles, thereby reproducing the grasping action in the virtual space that is synchronized with the physical world.

[0097] In summary, the virtual reconstruction module, through a structured parameter library and a clear command response mechanism, efficiently transforms the received lightweight semantic commands into continuous and accurate action representations in the virtual model, realizing the real-time mapping and reconstruction of physical entity states in the metaverse, and providing a core virtual driving foundation for subsequent physical calibration and interactive feedback.

[0098] The physical calibration module is responsible for performing high-precision simulation and synchronous correction of the virtual robotic arm's motion in the cloud. Its core function is to compensate for the gaps in motion performance caused by the discreteness of semantic commands.

[0099] The physics calibration module calls the physics engine integrated in the server to apply real physical law calculations to the virtual model. Since the semantic instructions received from the edge are discontinuous in time, the physics calibration module uses a physical constraint prediction algorithm to perform dynamic motion interpolation between these discrete instruction points.

[0100] To achieve prediction, the system needs to establish a simplified dynamic model for the virtual robotic arm in advance, which includes basic physical properties such as mass and inertia.

[0101] Before the actual prediction loop begins, the physical calibration module needs to initialize the motion state. When a new semantic instruction packet is received, it first uses the timestamp, spatial coordinates, and rotation quaternions in the instruction packet, combined with the virtual model pose corresponding to the previous instruction packet, to differentially calculate the average velocity and acceleration of the virtual robotic arm's end effector within the current instruction interval, as the initial dynamic model. and Input; if it is the first command after system startup, use zero value or preset safe value for initialization.

[0102] During computation, the module acquires the motion state of the virtual robotic arm's end effector at the previous moment, including its velocity vector. and acceleration .

[0103] Using these parameters, the physics calibration module calculates the predicted pose for the current simulation cycle based on classical kinematic formulas. The calculation formula is clearly defined as follows:

[0104]

[0105] in This represents the system's preset fixed simulation time step and is a configurable parameter.

[0106] In addition to motion prediction, the physical calibration module also performs collision detection in the virtual environment, creating simplified bounding box collision models for the virtual robotic arm and key objects around it, such as material racks and conveyor belts.

[0107] In each frame of the simulation calculation, the physics calibration module checks whether the robot arm's predicted motion trajectory will cause a penetrating collision with these bounding boxes.

[0108] It should be noted that the detected predicted motion trajectory is a sequence of continuous poses in the next few frames calculated in real time by the above-mentioned physical constraint prediction algorithm based on the dynamic model and the constantly updated end motion state. This trajectory reflects the most likely motion path of the virtual robotic arm from the current state under the constraints of physical laws.

[0109] If a potential collision is detected, such as when the predicted trajectory indicates that the robotic arm will penetrate into the material rack, the physical calibration module will automatically trigger the trajectory correction mechanism.

[0110] Trajectory correction can be achieved by adjusting the target point of the inverse kinematics solution, inserting obstacle avoidance path points, or fine-tuning the joint movement sequence. The corrected, collision-free virtual trajectory will ultimately be used to drive model rendering.

[0111] Meanwhile, the collision warning information will be packaged into a lightweight feedback command and transmitted back to the edge via the network, thereby alerting the physical system to environmental risks.

[0112] In summary, by combining kinematic prediction and real-time collision detection, the physics calibration module not only smooths the motion of the virtual model during command interruptions, but more importantly, ensures the rationality and safety of virtual operations in terms of physical logic.

[0113] Regarding the methodology, this embodiment strictly follows these steps:

[0114] Step S1 involves real-time acquisition of multi-source raw data using encoders, torque sensors, and depth cameras deployed on the side of the robotic arm.

[0115] In practice, this step is broken down into the following sequential operations:

[0116] S101 deploys and enables multiple types of sensors for synchronous data acquisition.

[0117] Six 16-bit absolute encoders are installed at the six joints of the robotic arm to acquire the rotation angle pulse signals of each joint in real time at a sampling frequency of 1000 Hz; a six-axis torque sensor is integrated on the end effector of the robotic arm to continuously measure the force and torque components of the end effector in three-dimensional space; at the same time, a depth camera is fixedly installed above the workspace to acquire a depth image stream containing the robotic arm body and the surrounding environment at a rate of 60 frames per second.

[0118] S102 performs real-time filtering and noise reduction on the encoder signal.

[0119] The edge computing gateway receives the raw pulse signal from the encoder and processes it using a mean filtering algorithm. This algorithm can effectively identify and eliminate instantaneous abnormal jump values ​​caused by electromagnetic interference on site, thereby obtaining stable and reliable joint angle data.

[0120] S103 performs time synchronization and alignment of all sensor data streams.

[0121] The system assigns a high-precision timestamp to each data sampling point and uses this as a benchmark to time-align sensor data with different sampling frequencies. For example, it aligns torque sensor data and encoder data on a unified time axis to ensure the consistency of multi-source data status in subsequent processing.

[0122] S104 performs preprocessing and data simplification on the point cloud stream acquired by the depth camera.

[0123] The system processes depth images in real time and filters out static environmental background point clouds through a background culling algorithm, retaining only the dynamic region data covered by the robotic arm's motion envelope, thus significantly reducing the amount of visual data that needs to be processed.

[0124] In step S2, the edge semantic engine identifies and abstracts the preprocessed data, with the aim of converting the continuous physical state data stream into discrete high-level semantic instructions.

[0125] Specifically, perform the following operations in sequence:

[0126] S201 monitors the motion parameters of specific joints in real time.

[0127] The edge semantic engine continuously calculates the rate of change of angle of each joint of the robotic arm. For example, when it detects that the cumulative change of the rotation angle of the 5th joint exceeds 30 degrees, the first judgment condition will be triggered.

[0128] S202, synchronously monitors the state changes of the end-effector interaction force.

[0129] The edge semantic engine simultaneously reads the real-time data stream from the six-axis torque sensor and calculates the rate of change of torque or force components. If it finds that the rate of change of a certain component suddenly increases in a short period of time and exceeds the system's preset threshold, it will trigger the second judgment condition.

[0130] S203, semantic events based on multi-condition joint determination.

[0131] When the two conditions S201 and S202 above are met simultaneously within the set time tolerance window, the edge semantic engine determines that the robotic arm has triggered a "rotation tightening" semantic event.

[0132] S204, extract the key physical parameters associated with the event.

[0133] Once an event is determined, the edge semantic engine will immediately extract the core feature parameters of the event from the current data snapshot and historical cache. The parameters mainly include the rotation axis vector calculated based on the rotation direction of the 5th joint, the target torque stability value extracted from the torque sensor reading, and the time spent from the start to the end of the entire event.

[0134] S205, perform confidence assessment and verification on the identified semantic events.

[0135] Before generating the final instruction, the edge semantic engine performs a logical verification, which checks whether the motion trajectory of the robotic arm's end effector falls entirely within the preset compliant operating area. It also determines whether the "rotation tightening" event conforms to the established process flow sequence, such as whether it immediately follows "grabbing parts". Only when all verifications pass will a valid semantic instruction be generated.

[0136] Step S3: Package the semantic events and related key parameters identified on the edge side and send them to the cloud server.

[0137] Specifically, it will proceed in the following steps:

[0138] S301, prepare to generate the necessary elements for semantic instructions.

[0139] The system will obtain the confirmed valid semantic event type identifier and all important physical parameters associated with the event from the results of step S2. The parameters include at least the timestamp of the event start, the unique identification code of the target object, and the spatial position coordinates and attitude quaternions of the robotic arm end.

[0140] S302 encapsulates binary data packets according to a predefined structure.

[0141] The system packages all the above elements according to a pre-designed fixed format. The number of bytes occupied by each field in the data packet format, the data type stored, and the order of arrangement all follow uniform rules.

[0142] For example, one byte can be used to encode the action type, such as grab or place; a fixed-length field, such as 24 bytes, can be used to store key position and posture parameters.

[0143] After being packaged according to the rules, a lightweight semantic instruction package with a total length of approximately 64 bytes will be generated.

[0144] S303 sends data packets via a reliable transport layer protocol.

[0145] After encapsulation, the system calls the transport layer network protocol to send the 64-byte semantic instruction packet to the specified cloud server address. Since the transmitted content is only structured lightweight instruction data rather than the original video stream, the bandwidth usage of this step is kept at a low level of less than 10 kilobytes per second.

[0146] In step S4, the Metaverse cloud processing platform receives semantic instructions and drives the virtual model to complete reconstruction and scene updates.

[0147] Specifically, it mainly includes the following steps:

[0148] S401 receives and parses semantic instruction packets.

[0149] Once the cloud platform's network interface receives a binary semantic instruction packet from the edge, it will first check the integrity of the instruction packet, such as verifying the cyclic redundancy check code at the end of the packet.

[0150] After confirming that everything is correct, the data packet is unpacked according to the pre-agreed format specifications, and the key fields are extracted, mainly the device's unique identification code, the semantic command code representing the action type, and the parameter list containing information such as position and attitude.

[0151] S402, retrieve the corresponding digital model based on the identification code.

[0152] Using the unique device identification code parsed in the previous step, the system searches and matches it in the pre-loaded parametric model library, enabling it to accurately locate the digital twin of the industrial robotic arm that needs updating in the virtual factory scenario.

[0153] S403 drives the state machine switching of the virtual model.

[0154] The system controls the animation controller of this virtual robotic arm, specifying the action to be performed based on semantic instruction encoding, and switches the model's predefined state machine from the current state to the target state. For example, switching from the "idle" state to the "grasp" state indicates that the model is preparing to execute the grasping animation.

[0155] S404 calculates the motion details of the model based on command parameters.

[0156] The virtual reconstruction module then extracts key data such as the target position coordinates and end-effector quaternions from the parsed parameter list. It then calls the inverse kinematics algorithm to use these parameters as constraints to calculate the precise angles that each joint of the virtual robotic arm needs to rotate to, thereby driving the model's skeletal animation system to produce continuous and smooth movements that meet the instructions.

[0157] S405, Update object relationships in the virtual scene.

[0158] The system will also process other virtual objects involved in the instructions, such as the parts being manipulated. Based on the target object identification code contained in the instruction package, it will find the corresponding part model in the scene, and then modify the hierarchical relationship of the part model in the data structure of the virtual scene graph, setting it as a child node of the coordinate system of the robotic arm end effector. In this way, the part model can move together with the end effector of the robotic arm, achieving a visual binding effect.

[0159] Step S5: Achieve motion smoothing and delay cancellation of the virtual model through physical constraint prediction and compensation.

[0160] This step is responsible for dynamically interpolating the motion of the virtual model in the cloud under the constraints of physical laws, and actively compensating for network transmission latency. This is achieved through the following sequential operations:

[0161] S501 records and associates the spatiotemporal information of the instruction packet.

[0162] After receiving and parsing the semantic instruction packet, the cloud system extracts the timestamp of instruction generation. This timestamp is a precise marker added by the edge system when generating the instruction packet. Simultaneously, the system also records the current rendering loop time on the cloud side.

[0163] S502 calculates end-to-end transmission and processing latency.

[0164] The system subtracts the current rendering time from the timestamp of the command generation, and the difference is the end-to-end latency. This latency is usually the total time it takes for data to travel from the physical world to the cloud to be ready for rendering, mainly including the time spent on network transmission and the time spent on cloud processing.

[0165] S503, determine whether the delay exceeds the preset threshold and make a decision.

[0166] The system will compare the calculated total delay value with a pre-set threshold.

[0167] For example, if the threshold is set to 50 milliseconds, and the actual delay exceeds this threshold, it is considered that the network delay has significantly affected the real-time performance of virtual-real synchronization, and a forward prediction compensation process will be initiated.

[0168] S504 performs forward pose prediction based on a dynamic model.

[0169] Once the decision to compensate is made, the physics calibration module will immediately begin working. First, it will take the pose parameters from the most recently received semantic instruction packet, as well as the motion state of the virtual model at the end of the previous simulation cycle, such as the position, velocity, and acceleration of the end effector. Then, it will input this data into a simplified dynamic model that has been pre-built for the virtual robotic arm, calculate it according to classical kinematic formulas, and then predict the pose that the end effector of the virtual model should theoretically reach from the moment the instruction was generated to the current rendering moment.

[0170] S505 transiently adjusts the virtual model to the predicted pose.

[0171] The system directly manipulates the digital model of the virtual robotic arm, updating the pose of its end effector from the originally calculated position in the current rendering frame to the pose predicted in the previous step. At the same time, in order to drive the movement of the entire model, it is also necessary to synchronously update the angles of all joints through inverse kinematics calculation, thereby visually eliminating the motion lag caused by network latency.

[0172] Furthermore, this system also includes a global status monitoring module, which continuously counts and monitors the frequency of commands sent by each communication node on the production line.

[0173] The global status monitoring module will synchronously monitor the network operation status of the entire workshop. Once it detects that the packet loss rate of data transmission has increased due to Wi-Fi signal interference or other reasons and exceeds the 5% threshold value preset by the system, the module will immediately start an adaptive adjustment mechanism.

[0174] Specifically, the global state monitoring module sends a control command to the semantic extraction module located at the edge, requiring the edge to execute a specific command merging strategy.

[0175] Upon receiving this control command, the edge semantic extraction module will change its working mode and merge multiple consecutive and logically related micro-action semantics identified in a short period of time.

[0176] For example, the "grab" action command identified successively and the "lift" action command that follows can be merged into a higher-level "composite material picking" semantic command.

[0177] By aggregating and abstracting action commands at the edge, the system can reduce the number of data packets that need to be sent to the cloud. Therefore, under unstable network conditions, the frequency of data packet transmission and the potential risk of packet loss are reduced.

[0178] In summary, the global status monitoring module dynamically adjusts the fineness of semantic abstraction at the edge by real-time perception of network conditions and command flow, enabling the entire system to adapt to changes in the network environment; at the same time, it ensures the reliability of command transmission and the overall stability and durability of the system under complex operating conditions such as wireless interference.

[0179] Example 2

[0180] This embodiment sets up a new application scenario, namely a large-scale automated logistics warehousing center. In this scenario, the physical entities are hundreds of automated guided vehicles that autonomously drive within the warehouse and collaboratively complete the transfer of goods. The corresponding metaverse environment is constructed as a virtual warehousing digital twin platform that can reflect the global logistics progress and details in real time.

[0181] To effectively support this large-scale, high-concurrency scenario, this embodiment enhances the large-scale concurrent processing capability and visualization rendering efficiency based on Embodiment 1, mainly by introducing a distributed rendering architecture and expanding the data acquisition end.

[0182] Specifically, at the data acquisition level, multi-source data acquisition modules are deployed on each automated guided vehicle (AGV), forming a distributed front-end perception network. Each acquisition module continuously acquires various types of sensor data from its own vehicle. This data forms the basis for perceiving the physical world and mainly includes the following categories:

[0183] The first type is the pulse signal generated by the wheel speed encoder, which is used to calculate the basic displacement and speed of the vehicle.

[0184] The second type is scanning data acquired by vehicle-mounted LiDAR, which provides the outline and distance information of the surrounding environment.

[0185] The third category is the angular velocity and acceleration data output in real time by the inertial measurement unit, which is used to sense the vehicle's attitude changes and motion dynamics.

[0186] The fourth type involves capturing images through vehicle-mounted cameras and processing them with integrated visual algorithms to output obstacle recognition results, which are used to supplement the ability to perceive dynamic or specific obstacles.

[0187] These multi-source data, distributed across hundreds of mobile nodes, collectively constitute the real-time nerve endings of the virtual digital twin platform for perceiving the physical world.

[0188] The edge semantic extraction module is directly integrated into the onboard controller of each automated guided vehicle.

[0189] To cope with the complex and dynamic environment inside the warehouse, the core of the edge semantic extraction module adopts a semantic recognition mechanism based on state space partitioning to understand vehicle behavior.

[0190] Specifically, the system pre-divides the digital map of the entire warehouse into several logical areas with different functional attributes, mainly including the driving area, loading area, charging area, and obstacle avoidance area.

[0191] Each region in the system has a clearly defined coordinate boundary. The edge semantic extraction module receives the vehicle's positioning data in real time, determines which functional region its current coordinates are located in, and triggers the corresponding semantic understanding by combining real-time sensor information.

[0192] For example, when the system determines the vehicle's coordinates to enter the loading area through positioning data, and at the same time identifies specific shelf outline features by processing the scanning data of the vehicle-mounted LiDAR, the edge semantic engine will determine that the vehicle intends to perform a loading operation.

[0193] At this point, the edge semantic engine no longer uploads the original radar point cloud or continuous coordinates, but generates an "execute loading" semantic instruction. The instruction contains the target shelf number and the end position offset calculated based on the radar data for precise docking.

[0194] Another common scenario is that when a vehicle is in the driving area, and it detects a dynamic obstacle ahead that has not been path-planned by fusing LiDAR and camera data, the edge semantic engine will immediately generate an "emergency obstacle avoidance" command. This command includes a suggested obstacle avoidance steering angle calculated based on the environmental model and vehicle dynamics, as well as the deceleration value required to ensure safety.

[0195] In this embodiment, the instruction transmission module uses the message queue telemetry transmission protocol as the basis for data interaction. This protocol is suitable for IoT scenarios with numerous devices and loose connections, and can effectively manage the massive connections and message streams generated by hundreds of automated guided vehicles online simultaneously.

[0196] To address the potential network congestion and processing delays caused by large-scale concurrency, the instruction transmission module implements a semantic instruction packet priority marking mechanism. This mechanism assigns a priority identifier to each sent data packet based on the urgency and importance of the action represented by the instruction.

[0197] For example, "emergency obstacle avoidance" commands generated from the edge are marked as having the highest priority by the system because they correspond to safety events requiring immediate response. In the network transmission queue, data packets with the highest priority identifier are scheduled for priority transmission; similarly, they are prioritized for retrieval and processing in the server's receive and processing queue.

[0198] Through this business logic-based priority scheduling, the system ensures that even during periods of limited network bandwidth or high server load, the most critical control commands can achieve the lowest transmission and processing latency, thereby guaranteeing overall security and real-time response in large-scale collaborative operations.

[0199] The virtual reconstruction module adopts a distributed management architecture in the metaverse cloud processing platform to cope with the data processing and rendering pressure brought by hundreds of automated guided vehicles operating concurrently.

[0200] The parametric model library managed by the virtual reconstruction module pre-stores digital models of all different types of automated guided vehicles that may appear in the warehouse. Each model not only includes the geometric appearance, but also integrates complete kinematic constraint parameters. These parameters are used to define the physical behavior boundaries of the vehicle in the virtual environment, such as its maximum turning radius, braking distance under different loads, and other key attributes.

[0201] To achieve efficient parallel rendering, the server logically divides the entire virtual warehouse scene into multiple spatial units, specifically into a regular grid of 20 meters by 20 meters, with each such grid area serving as an independent spatial slice.

[0202] Each spatial segment is assigned an independent rendering unit, which is responsible for the real-time status updates, animation driving, and image generation of all virtual models within its corresponding segment.

[0203] When semantic command packets from multiple robots arrive at the cloud simultaneously, the load balancer will intervene to intelligently distribute them, parse the real-time coordinate information of the robot carried in each command packet, determine the spatial segment to which it belongs based on the coordinates, and then quickly route the command to the corresponding rendering unit responsible for that segment for processing.

[0204] The physical calibration module is specifically designed for scenarios where hundreds of automated guided vehicles (AGVs) are working together. It expands the scope and logic of its collision detection and performs a global collision detection.

[0205] This detection not only includes collisions between each robot and static environments such as virtual warehouse walls and shelves, but more importantly, it calculates in real time the shortest distance between any two mobile robots for their predicted trajectories.

[0206] The system continuously predicts the motion path of each robot within a certain period of time based on the dynamic model of each robot and the received semantic instructions. When it is detected that the predicted trajectories of two robots will intersect in 3 seconds, or that the shortest distance between them is lower than the set safety threshold, the physical calibration module determines that there is a potential risk of path conflict.

[0207] Once a conflict is predicted, the physical calibration module will initiate a local path planning simulation in the virtual digital twin space. Using a preset obstacle avoidance algorithm, such as a path replanning algorithm combined with a time window, it will calculate one or more collision-free optimized obstacle avoidance paths for the robot involved in the conflict.

[0208] After the calculation is completed, the physical calibration module will encapsulate the key parameters of the new path simulated and adjusted in the virtual space, such as new turning points and speed adjustment suggestions, into lightweight feedback instructions. These instructions are transmitted back to the physical entity side via the network, that is, sent to the corresponding automated guided vehicle onboard controller, thereby guiding the robot to adjust its behavior in advance in the physical world and avoiding actual collisions.

[0209] Regarding the method and process, the specific steps of this embodiment are as follows:

[0210] Step S1 involves synchronously acquiring raw data through multiple onboard sensors and performing data preprocessing for semantic understanding at the edge. Specifically, this includes:

[0211] S101: Each automated guided vehicle (AGV) collects wheel speed encoder pulses, raw data from the inertial measurement unit, lidar point clouds, and camera images in parallel, providing synchronous raw input for subsequent fusion and feature extraction.

[0212] S102, in order to obtain the vehicle pose that meets the semantic recognition accuracy requirements, the edge gateway integrates an extended Kalman filter that estimates the vehicle's planar motion state. This filter uses the speed information calculated by the wheel speed encoder as the process input and integrates the heading angle change observations provided by the inertial measurement unit.

[0213] When the lidar scanning data is successfully matched with the preset warehouse digital map, the matching result will be used as a high-confidence global position observation to further correct the pose estimation.

[0214] Through this fusion process, the system outputs stable real-time pose information that can be used to accurately determine the functional area (such as the loading area or the driving area) where the vehicle is located.

[0215] S103, in order to efficiently extract the geometric features that are key to semantic judgment, the system performs two-level processing on the original lidar point cloud.

[0216] First, voxel mesh downsampling is performed to reduce the data size. Then, based on the curvature and normal distribution characteristics of the point cloud, edge feature points representing the environmental contour (such as shelf corners) and planar feature points representing flat surfaces (such as walls and floors) are extracted. The unstructured original point cloud is transformed into a lightweight set of feature points that characterize the key geometric structure of the environment.

[0217] Through the above preprocessing, the system generates accurate vehicle pose and structured environmental feature data at the edge.

[0218] Step S2: The edge semantic engine performs real-time parsing of the continuous vehicle pose stream. When it identifies regular movements that conform to preset rules, it abstracts these movements into compact semantic descriptors and uploads them. The specific implementation process is as follows:

[0219] S201, the edge semantic engine maintains a fixed-length time window, continuously analyzes the position and speed sequence of vehicles within the window, calculates the overall direction angle change rate of the trajectory within the window, and calculates the average value and standard deviation of the speed sequence.

[0220] S202, the system presets a set of quantitative rules for identifying straight-line cruise state, which requires that the trajectory direction angle change rate be continuously less than 5 degrees per second, while the average speed is higher than 0.5 meters per second and the speed standard deviation is less than 0.2 meters per second. These rules together define the feature template of the straight-line cruise mode.

[0221] S203, when the feature data calculated in real time simultaneously meets all the above rules, the edge semantic engine determines that the vehicle is in a straight-line cruise state; then the edge semantic engine stops sending the high-frequency pose points within the window to the cloud, and instead extracts and encapsulates the starting point coordinates, ending point coordinates, and average speed of this cruise to generate a straight-line cruise semantic identifier.

[0222] S204, the straight-line cruise semantic identifier, together with the timestamp, constitutes a minimal semantic instruction that transforms the unstructured pose data stream that originally needed to be continuously transmitted into a structured, lightweight semantic event report that is triggered occasionally, fundamentally reducing the total amount of data transmitted over the network.

[0223] Step S3 involves encapsulating semantic events into structured network packets and transmitting them via a priority-based wireless network. Specifically, this includes the following operations:

[0224] S301, the edge system encapsulates the semantic identifier and related parameters generated in step S2 into a data packet according to a predefined binary format. The header of the data packet contains a unique device identifier and an instruction type code, the body carries key parameters, and the tail contains a check code.

[0225] S302, to differentiate the urgency of commands, the transmission module assigns transmission priorities to data packets based on the command type. For example, emergency obstacle avoidance commands are assigned the highest priority, while regular cruise commands are assigned a normal priority. The network transmission queue schedules data according to this priority to ensure that high-priority commands are sent first.

[0226] S303 Finally, the data packets are sent to the cloud server via an industrial wireless network deployed in the warehouse. This network supports a quality of service (QoS) guarantee mechanism, can identify the priority of data packets, and provide a low-latency, highly reliable transmission channel for high-priority data streams.

[0227] Through the above process, semantic instructions are transformed into network packets with clear format and controllable transmission, providing clear and timely data input for the cloud-based virtual reconstruction module.

[0228] In step S4, the Metaverse cloud processing platform parses semantic instructions and drives the corresponding models and associated states in the distributed virtual scene to be updated.

[0229] After receiving the data packet, the cloud server first verifies and parses its structure, extracting the device's unique identification code, instruction type, and key parameters.

[0230] S402, the system uses the device's unique identifier to query the global mapping table, determine the corresponding virtual robot instance and its logical rendering segment, and route the instruction to the rendering server responsible for that segment.

[0231] S403, the rendering server calls a predefined behavior parsing template according to the instruction type. The template converts the parameters in the instruction, such as coordinates and velocity, into specific control parameters of the virtual robot animation controller, such as path point sequences or motion speed values, thereby driving the model to move.

[0232] S404: For instructions involving the transfer of goods, the system synchronously updates the virtual inventory database and modifies the container field of the relevant goods.

[0233] The scene graph engine adjusts the hierarchical binding of the virtual cargo model based on the new ownership relationship to achieve a visual transfer effect.

[0234] Through the above process, abstract semantic instructions are accurately interpreted and executed as specific model movements and state changes in the virtual environment, thus completing the core state synchronization of the digital twin.

[0235] Step S5: Perform continuous interpolation of the virtual robot's motion in the cloud based on physical laws to fill the action gaps between semantic instructions.

[0236] To ensure continuity of movement within the virtual monitoring interface, the system performs the following operations within the time interval between two consecutive semantic commands:

[0237] S501, the physics engine, uses the pose and historical velocity provided by the most recent semantic instruction to build short-term motion extrapolation for each virtual robot.

[0238] S502's physics engine uses a uniformly accelerated motion model, based on the position at the previous moment. ,speed Based on this, the predicted pose sequence for a future period of time is calculated.

[0239] S503, when a new semantic instruction arrives, the system acquires its specified target pose. Subsequently, the system will predict the end point of the sequence and... As two key control points, a cubic spline interpolation curve is constructed between these two points.

[0240] This curve ensures the continuity of position and velocity, generating a complete motion trajectory that smoothly transitions from the current predicted state to the new command target.

[0241] S504, ultimately, this smooth trajectory is converted into path data that drives the virtual robot animation controller, enabling it to present a continuous, non-jumping motion effect even during command interruptions.

[0242] The global status monitoring module continuously evaluates the system's operating status and dynamically adjusts the command generation strategy on the edge side when the network load is too high.

[0243] The global status monitoring module periodically collects performance data of communication links in each area, focusing on monitoring data packet transmission latency and packet loss rate. Each logical area has a predefined load threshold.

[0244] When the number of active devices in a certain area exceeds the threshold and the network performance indicators of that area continue to deteriorate, such as the average packet loss rate being higher than 2% for several consecutive sampling periods, the global status monitoring module determines that network congestion has occurred in that area.

[0245] Subsequently, the module sends instructions to the edge semantic extraction engines of all automated guided vehicles in the area to activate their high-order semantic merging function. In this mode, the edge semantic engine will extend the analysis window and merge multiple atomic actions with continuous logical relationships identified within the window.

[0246] For example, the sequentially identified left-turn, straight-ahead, and right-turn intentions are merged into a single path navigation semantic command pointing to the final destination. By reducing the number of command packets that need to be uploaded, the system effectively alleviates local network congestion and ensures that the overall load remains at a stable operating level.

[0247] This adaptive mechanism enables the system to intelligently balance the restoration accuracy and transmission efficiency of the digital twin based on real-time network conditions, ensuring overall reliability in large-scale concurrent scenarios.

[0248] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0249] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for constructing a metaverse digital twin that supports real-time interaction, characterized in that, include: S1 collects raw data streams containing motion state and environmental information in real time through multiple types of sensors deployed on physical entities; S2, the original data stream is processed at the edge, and based on the real-time calculation of kinematic parameters and the joint determination of multiple conditions, the semantic events executed by the physical entity are identified, and the key physical parameters associated with the semantic events are extracted to generate a structured lightweight semantic instruction package. S3, transmit the lightweight semantic instruction package to the cloud server; S4, the cloud server parses the lightweight semantic instruction package, drives the corresponding virtual model in the parameterized model library to perform the corresponding action according to the device identifier and action type contained therein, and updates the hierarchical relationship of related objects in the virtual scene to achieve visual follow-up; S5, the cloud server, uses a physical constraint prediction algorithm to interpolate and smooth the motion of the virtual model between continuously received semantic instruction packets, and performs forward prediction based on the timestamps in the instruction packets to compensate for network transmission delay.

2. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S2, semantic events are identified based on real-time calculation of kinematic parameters and joint determination of multiple conditions, specifically including: The first and / or second derivative operations are performed on the collected joint angle or displacement data to calculate the velocity and acceleration parameters in real time. Simultaneously monitor the rate of change of force / torque sensor data stream; When the calculated kinematic parameters conform to the predefined change model of a specific action within a preset time window, and the relevant force / torque change rate simultaneously meets the preset threshold, they are jointly determined as the corresponding semantic event. Before generating the final instruction, the confidence level of the identified semantic events is assessed. The assessment factors include whether the movement trajectory is located in a compliant operating area and whether it conforms to the preset process flow sequence.

3. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S2, a structured, lightweight semantic instruction package is generated, specifically including: The semantic event type identifier, event start timestamp, target object identifier, and spatial coordinates and attitude parameters extracted from the current data are encapsulated into a binary data packet according to a predefined fixed field format. The total number of bytes in the data packet is in the tens to hundreds of bytes range.

4. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S4, the corresponding virtual model in the parameterized model library is driven to perform corresponding actions, specifically including: Based on the parsed device identifier, the corresponding virtual model is located in the parametric model library. The parametric model library includes at least a geometry layer that stores the geometric appearance of the model, an animation layer that predefines standard motion skeletal animation sequences, and an attribute layer that manages dynamic parameters. Based on the parsed action type, control the animation state machine of the virtual model to switch to the corresponding state; Using the parsed target position and posture parameters as constraints, the joint motion parameters required to drive the skeletal animation sequence are calculated through inverse kinematics algorithm, thereby driving the model in the geometry layer to produce continuous movements.

5. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S4, the hierarchical relationship of related objects in the virtual scene is updated to achieve visual tracking, specifically as follows: Based on the target object identifier in the semantic instruction package, the corresponding virtual object model is found in the virtual scene, and the virtual object model is set as a child node of the coordinate system of the virtual model end of the action in the scene graph data structure, so that the virtual object model can move synchronously with the virtual model end.

6. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S5, interpolation and smoothing are performed based on the physical constraint prediction algorithm, specifically including: Based on the pose parameters in the most recently received semantic instruction packet and the historical motion state of the virtual model, the predicted motion trajectory of the virtual model within the instruction interval is calculated based on the established simplified dynamic model. In a virtual environment, a collision model is established for the virtual model and key obstacles, and collision detection is performed on the predicted motion trajectory. If a potential collision is detected, the virtual model's trajectory is automatically corrected, and the collision warning information is encapsulated as a feedback command and sent to the edge.

7. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, In step S5, forward prediction is performed based on the timestamp in the instruction packet to compensate for network transmission delay, specifically including: The difference between the generation timestamp of the semantic instruction packet and the current rendering time in the cloud is used to calculate the end-to-end latency; When the delay exceeds a preset threshold, the pose of the virtual model from the moment the instruction was generated to the current rendering moment is predicted based on the simplified dynamics model and the most recent instruction parameters. The current display pose of the virtual model is transiently adjusted to the predicted pose.

8. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, Also includes: The network status is monitored in real time through the global status monitoring module; When network performance indicators are detected to be deteriorating, a control command is sent to the edge side, triggering the edge side to merge multiple atomic semantic events with continuous logical relationships identified within a preset time window into a high-order composite semantic event, and generate a corresponding composite semantic command package for uploading.

9. The method for constructing a metaverse digital twin supporting real-time interaction according to claim 1, characterized in that, The physical entity is an industrial robotic arm or an automated guided vehicle; the multiple types of sensors include at least two or more of the following: encoders, force / torque sensors, vision sensors, and positioning sensors.

10. A metaverse digital twin construction system supporting real-time interaction, used to implement the metaverse digital twin construction method supporting real-time interaction as described in any one of claims 1-9, characterized in that, include: A multi-source data acquisition module, deployed on the physical entity side, is used to acquire raw data streams in real time; An edge semantic extraction module, deployed on an edge computing node, is used to perform the kinematic parameter calculation, multi-condition joint determination and confidence assessment as described in claim 2, and to generate a structured lightweight semantic instruction package; The instruction transmission module is used to reliably transmit the lightweight semantic instruction package to the cloud. The virtual reconstruction module, deployed on a cloud server, includes a parameterized model library and is used to perform the model-driven, inverse kinematics solution as described in claim 4 and the scene hierarchy update as described in claim 5. A physical calibration module, deployed on a cloud server, is used to perform the physical constraint prediction, collision detection and trajectory correction as described in claim 6, and the delay compensation prediction as described in claim 7. The global status monitoring module is used to execute the network monitoring and command merging strategy triggering described in claim 8.

Images