Implementation method of digital twin data modular interface

Abstract

The invention relates to the technical field of digital twinning, in particular to an implementation method of a digital twinning data modular interface, which comprises the following steps of: constructing a unified data abstraction layer and defining an access descriptor; dynamically loading the analysis module and standardizing the data flow; data distribution based on a standardized bus; and carrying out space-time alignment and fusion mapping on the multi-source heterogeneous data. The unified data abstraction layer and the pluggable analysis module are constructed, plug and play of the sensor and standardized processing of heterogeneous data are achieved, hardware strong coupling is thoroughly eliminated, and the development threshold is greatly reduced. Meanwhile, in combination with a space-time alignment and cascade fusion strategy, the problem of accurate matching of multi-modal data is solved, and the mapping accuracy and real-time performance of the twin are remarkably improved. In addition, the bus-based asynchronous decoupling architecture effectively enhances the expansibility of the system.

Description

Technical Field

[0001] This invention relates to the field of digital twin technology, and in particular to a method for implementing a modular interface for digital twin data. Background Technology

[0002] Digital twin technology constructs a virtual mirror image of a physical entity using digital means. Its core lies in the bidirectional data interaction and mapping between the virtual and real worlds. In practical industrial applications, digital twin systems need to access various sensor data from the physical world, such as visual images from cameras, 3D point clouds from LiDAR, and traditional sensor data like temperature, pressure, and vibration. This data exhibits typical multi-source heterogeneity, specifically manifested in diverse data formats (e.g., pixel matrices, 3D point clouds, time-series values), inconsistent communication protocols (e.g., RTSP, HTTP, MQTT, TCP / IP custom protocols), and significant differences in data sampling frequency and semantics.

[0003] Most existing digital twin data access solutions adopt a hard-coded or customized development model, that is, writing dedicated data parsing and access code for each specific model or type of sensor. This traditional approach has obvious drawbacks.

[0004] First, the system suffers from poor compatibility and scalability. Adding or changing sensor types often requires modifying, recompiling, and deploying the underlying code, resulting in long development cycles and a heavy workload. Second, the high degree of code coupling leads to low maintainability; a change in one part affects the entire system, and developers must be proficient in both low-level sensor protocols and high-level digital twin modeling, significantly raising the technical barrier. Furthermore, when facing the fusion requirements of complex data such as visual and radar data, the lack of a unified time synchronization and spatial alignment mechanism makes it difficult to guarantee accurate mapping and real-time driving of multimodal data on the digital twin. Summary of the Invention

[0005] To address the technical problems of existing digital twin systems when accessing multi-source heterogeneous sensor data, such as inconsistent interfaces, poor compatibility, complex data processing procedures, and strong coupling between code and hardware, this invention proposes a method for implementing a modular interface for digital twin data.

[0006] This invention is achieved through the following technical solution: A method for implementing a modular interface for digital twin data includes the following steps: S1. Construct a unified data abstraction layer and define access descriptors. Read configuration files from the system's predefined directory and map different types of sensor hardware information to sensor description objects in memory. The configuration files adopt a structured text format and define the sensor's identity, communication connection configuration, data specification description, and semantic and processing configuration. S2. Dynamic loading of parsing modules and standardization of data streams: Based on the field information in the sensor description object, the system matches the corresponding parser module in the preset pluggable parsing module pool. After the parsing module is instantiated, it establishes a connection with the physical sensor using the connection information in the descriptor and converts the received raw heterogeneous data into an intermediate data object with a unified structure. The intermediate data object contains at least a unique sensor identifier, a unified timestamp, and a data payload that conforms to a predefined architecture. S3. Data distribution based on a standardized bus: The converted intermediate data objects are published to the standardized data bus, and the backend processing module obtains data by subscribing to topics with independent identifiers. S4. Spatiotemporal alignment and fusion mapping of multi-source heterogeneous data: The data mapping and fusion module subscribes to standardized image and point cloud data on the data bus, and sequentially performs spatiotemporal alignment processing, independent perceptual feature extraction, multidimensional data association and fusion, and entity state generation. Finally, it outputs a list of entity states with unique IDs to drive the digital twin.

[0007] Furthermore, the access descriptor in S1 is specifically: Identification and basic attributes, including unique identifiers within the system, sensor category and model; Connection and communication configuration, including communication protocol type, access address, and protocol-specific parameters; Data specification description, including the original data format, data encoding method, data frequency, and data architecture definition of key fields; Semantics and processing configuration, including physical location coordinates, installation orientation, recommended parsing module name, and preprocessing steps.

[0008] Furthermore, the step of matching the corresponding parser module in S2 is specifically as follows: The system first checks whether the expected parsing module name is explicitly specified in the sensor description object. If it exists, the specified module is loaded directly. If it is not specified, the system generates a composite key value based on the communication protocol field and data format field in the sensor description object, and automatically matches and loads the corresponding general parsing module by querying the parser registry.

[0009] Furthermore, the spatiotemporal alignment process in S4 includes time synchronization and spatial alignment; The time synchronization is based on high-precision hardware timestamps. In the data cache queue, the frame timestamps of low-frequency sensors are used as a reference. The two frames with the closest timestamps are found in the image queue. Virtual image frames at the corresponding time are generated by linear interpolation or minimum time difference. The spatial alignment is based on a pre-calibrated extrinsic matrix, which transforms the three-dimensional points in the radar point cloud coordinate system to the camera coordinate system. The camera intrinsic matrix is ​​then used to project the three-dimensional points onto the two-dimensional image pixel coordinate system, thus establishing a spatial mapping relationship between the three-dimensional points and the two-dimensional pixels.

[0010] Furthermore, the independent perceptual feature extraction in S4 specifically includes: The visual perception pathway inputs the aligned image into the two-dimensional object detection model and outputs a list of two-dimensional bounding boxes containing category ID, confidence score, and center point coordinates. The radar perception pathway inputs the aligned point cloud into the 3D target detection network and outputs a list of 3D bounding boxes containing category ID, confidence level, 3D center position, size, and orientation angle.

[0011] Furthermore, the multidimensional data association and fusion in S5 adopts a cascading association strategy, specifically as follows: Based on the coarse correlation stage of projection, the transformation matrix calculated in the spatial alignment stage is used to project the vertices of the 3D detection box sensed by the radar onto the image plane to generate a 2D projection box. Calculate the intersection-union ratio between the two-dimensional projection frame and the two-dimensional bounding box output by the visual perception path; When the crossover ratio is greater than the preset threshold, the corresponding 3D detection results and 2D detection results are used to form a candidate matching pair.

[0012] Furthermore, the cascaded association strategy also includes an optimal matching stage based on metric learning, specifically: Construct a cost matrix and calculate the comprehensive matching cost for each candidate matching pair. The calculation of the comprehensive matching cost is based on the two-dimensional intersection-union ratio cost, the semantic similarity cost of the classification results, and the Euclidean distance cost between the back projection ray of the visual detection box and the center of the radar three-dimensional box. The cost matrix is ​​input into a global optimization algorithm to find the one-to-one matching result that minimizes the total matching cost. Matches with a matching cost higher than a set threshold are discarded and considered invalid matches.

[0013] Furthermore, the entity state generation in S4 includes an attribute fusion step, specifically: For successfully matched detection pairs, their attributes are weighted and fused; the position coordinates and size attributes of the entities are mainly based on radar detection data. The category attribute of an entity is selected from the category with higher confidence or a weighted vote is applied; The final confidence level of an entity is calculated jointly based on visual confidence level and radar confidence level.

[0014] Furthermore, the entity state generation in S4 also includes cross-frame tracking and state output steps, specifically: Initialize a tracker and assign a globally unique ID to each newly emerging fused entity; The state of the tracker in the current frame is predicted using Kalman filtering, and the fused entity in the current frame is updated with the active tracker using an optimization algorithm. The output includes a standardized state list containing timestamps, entity IDs, semantic categories, fused 3D state data, and metadata, which is published to a dedicated data bus for the digital twin rendering engine to subscribe to.

[0015] The beneficial effects of this invention are: (1) By constructing a unified data abstraction layer and defining structured access descriptors, this invention separates the physical attributes and communication configuration of the sensor from the underlying code. Unlike the existing technology that hard-codes specific sensors, this invention can complete the driver matching and connection establishment by modifying or adding configuration files when facing changes or additions to the sensor hardware. There is no need to rewrite or compile the core code, which significantly reduces the coupling of the system, realizes true plug-and-play, and greatly shortens the system deployment and maintenance cycle. (2) Through the pre-set pluggable parsing module pool and standardized data flow design, the present invention can dynamically load parsers that adapt to different protocols such as RTSP, MQTT and formats such as H.264 and point cloud, and automatically convert the original heterogeneous data into intermediate data objects containing unified timestamps and architecture. This process shields the upper layer application from the complex communication protocols and data format differences at the lower layer, so that developers do not need to be proficient in the underlying hardware protocols to focus on business logic, effectively reducing the technical threshold and cost of developing digital twin systems. (3) To address the temporal and spatial inconsistencies between visual and radar data, this invention proposes a preprocessing mechanism that includes high-precision timestamp interpolation synchronization and spatial alignment of extrinsic parameter matrices, ensuring accurate correspondence of multi-source data under the same spatiotemporal reference. Based on this, a cascaded strategy combining projection-based coarse association and metric learning-based fine association is adopted to effectively solve the target matching problem in complex scenarios. By fusing the high-precision positional attributes of radar with the rich semantic attributes of vision, and combining Kalman filtering for state tracking, entity state data far superior to that of a single sensor is generated, significantly improving the real-time performance and accuracy of motion and state mapping of digital twins in virtual space. (4) This invention adopts a publish / subscribe model based on a standardized data bus, realizing asynchronous decoupling between the data acquisition and parsing end and the back-end processing end. The back-end module only needs to subscribe to a specific identifier to obtain data without interfering with each other. This not only facilitates the horizontal expansion of the system, but also prevents the entire system from crashing when a single parsing module fails, thereby improving the overall robustness and stability of the system in industrial application scenarios. In summary, this invention constructs a unified data abstraction layer and a pluggable parsing module, achieving plug-and-play functionality for sensors and standardized processing of heterogeneous data, completely eliminating strong hardware coupling and significantly lowering the development threshold. Simultaneously, by combining spatiotemporal alignment and cascaded fusion strategies, it solves the problem of accurate multimodal data matching, significantly improving the mapping accuracy and real-time performance of twins. Furthermore, the bus-based asynchronous decoupling architecture effectively enhances the system's scalability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the implementation method of a modular digital twin data interface proposed in this invention. Figure 2 This is a schematic diagram of a terminal device for implementing a modular digital twin data interface proposed in this invention. Figure 3 This is a schematic diagram of a readable storage medium for implementing a modular interface for digital twin data proposed in this invention. In the diagram, 200 is the terminal device, 210 is the memory, 211 is the RAM, 212 is the cache, 213 is the ROM, 214 is the program / utility, 215 is the program module, 220 is the processor, 230 is the bus, 240 is the external device, 250 is the I / O interface, 260 is the network adapter, and 300 is the program product. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention. Example

[0019] This embodiment provides a method for implementing a modular interface for digital twin data. This method is applied to a digital twin monitoring system in a smart manufacturing workshop, aiming to achieve real-time, high-precision mapping of the status of mobile robots, operators, and goods in the workshop.

[0020] In the physical scenario of this embodiment, a high-definition security camera is deployed at the south entrance of the workshop to collect video stream data; simultaneously, a 16-line LiDAR is installed at the center of the workshop ceiling to collect 3D point cloud data. To integrate this data from two sensors with different communication protocols, data formats, and sampling frequencies into a unified digital twin system, this solution employs a layered, modular interface architecture.

[0021] The system's operation begins with the construction of the unified data abstraction layer and the definition of access descriptors. During system startup initialization or hot deployment updates, the system's main control process first scans the predefined sensor configuration directory in the server's file system. Within this directory, an independent structured text configuration file is created for each physical sensor. These files act as passports for physical devices entering the digital world. For the aforementioned high-definition security camera, the system reads the file named "South Gate Entrance Camera Configuration." This file first defines the device's identity and basic attributes in detail through text descriptions. This includes setting the system's unique identifier field to `camera_entrance_01` as the unique index key for internal data routing; setting the sensor category field to "vision," clearly indicating that it belongs to the visual perception device category; and recording the specific model in the manufacturer / model field for subsequent driver matching. Next, the configuration file details the connection and communication configurations, specifying the communication protocol type as RTSP (Real-Time Streaming Protocol) and providing a specific access address, such as `rtsp: / / 192.168.1.100...`. Simultaneously, the protocol-specific parameter fields configure the TCP transmission mode and encrypted access credentials, ensuring the security of the data link. In the data specification section, the document specifies that the original data format is an H.264 encoded video stream with a data frequency of 25 Hz. It also defines fields in the data architecture to ensure that the subsequently parsed data must include key metadata such as the video stream's resolution and color space. More importantly, the semantics and processing configuration precisely records the camera's physical position coordinates (including X, Y, and Z coordinates) in the workshop's global coordinate system, as well as the Euler angles (yaw, pitch, and roll) of its installation orientation. It explicitly specifies the recommended parsing module as a camera-specific advanced parser and defines a preprocessing step of horizontal flipping before the data enters the parsing process. Similarly, for LiDAR, the system reads the corresponding configuration file, identifies its communication protocol type as TCP, the original data format as binary point cloud packets, the data frequency as 10 Hz, and records its corresponding physical position and calibration parameters.

[0022] After completing the reading and parsing of the configuration file, the system enters the execution phase of dynamically loading the parsing module and standardizing the data flow. Internally, the system maintains a pluggable parsing module pool containing various general and specialized drivers. For each sensor description object, the system executes intelligent matching logic: First, the system checks if the description object contains a user-specified suggested parsing module name. For the aforementioned camera, since it is specified in the configuration file, the system directly loads the corresponding dedicated parsing module. For general sensors without a specified dedicated module, the system extracts its communication protocol type field and raw data format field, combines them to generate a composite key value, and retrieves and loads the matching general parser from the registry. After the parsing module is instantiated, it immediately uses the access address and protocol-specific parameters in the description object to automatically initiate a connection request to the physical sensor, such as establishing an RTSP session with the camera or a TCP socket connection with the radar. After the connection is successfully established, the parsing module begins to receive raw data streams in real time. The camera's parsing module decodes the received H.264 bitstream into frames of raw RGB image data, while the radar's parsing module unpacks the binary data packets into a point cloud list containing three-dimensional coordinates and reflection intensity.

[0023] At this point, the parsing module performs a crucial standardization operation, encapsulating these heterogeneous raw data into intermediate data objects with a unified structure. This intermediate object strictly contains three core fields: a unique sensor identifier directly inherited from the configuration file; a unified timestamp in a standardized format generated by the parsing module based on the reception time or hardware timestamp; and a data payload conforming to a predefined architecture. This standardization process ensures that regardless of the brand of sensor connected upstream, the data output downstream has a consistent interface, achieving deep decoupling between software and hardware.

[0024] Subsequently, the system executes a data distribution step based on a standardized bus. To handle high-concurrency data throughput, this system uses a message queue middleware as the standardized data bus. The parsing module serializes the encapsulated intermediate data objects and publishes them to predefined logical topics. For example, visual data is published to the visual data topic, and radar data is published to the radar data topic. This publish / subscribe model achieves asynchronous decoupling between the data production and consumption ends. The backend processing module does not need to know which specific process generated the data; it only needs to subscribe to the independent identifier of the corresponding topic to obtain the data stream. In this embodiment, the core data mapping and fusion module, as the main consumer, subscribes to both the visual and radar topics, thereby acquiring full-scale perception data of the workshop environment.

[0025] After the data mapping and fusion module receives standardized image and point cloud data from the bus, it executes four core processing stages in sequence to achieve deep fusion of multi-source heterogeneous data. The first stage is spatiotemporal alignment preprocessing. Due to the different sampling frequencies of the camera and radar, the system first performs time synchronization. The fusion module maintains its own data cache queue. Using the low-frequency radar frame timestamp as the trigger reference, when each radar data frame arrives, the system retrieves the two frames before and after the radar frame whose timestamps are closest to the radar frame time from the image cache queue. A virtual image frame that is perfectly aligned with the radar frame in time is generated through a linear interpolation algorithm. Next, spatial alignment is performed. The system reads the physical location coordinates and installation orientation recorded in the configuration file and constructs a rigid body transformation matrix (extrinsic parameter matrix) from the radar coordinate system to the camera coordinate system. Using this matrix, the system transforms each 3D point in the radar point cloud to the camera coordinate system and further uses the camera's intrinsic parameter matrix to project these 3D points onto the 2D image pixel coordinate system, establishing a spatial mapping relationship between 3D points and 2D pixels.

[0026] The second stage involves independent perception and feature extraction. The system operates two perception pathways in parallel: vision and radar. In the vision pathway, the aligned image is input into a pre-trained 2D target detection model. The model identifies targets such as workers and AGVs in the image and outputs a list of 2D bounding boxes, each containing a category ID, confidence score, and center point coordinates. In the radar pathway, the aligned point cloud is input into a 3D target detection network. The network extracts spatial features and outputs a list of 3D bounding boxes, including the target's 3D center position, size, and orientation angle. At this point, the system has two independent detection results, but the correspondence between them has not yet been determined.

[0027] The third stage is multi-dimensional data association and fusion. The system adopts a cascaded association strategy to solve the cross-modal target matching problem. First, a coarse association based on projection is performed. The system projects the vertices of the 3D detection box output by the radar onto the image plane using the aforementioned transformation matrix to generate a 2D projection box. The intersection-union ratio (IU) between this projection box and the visual 2D bounding box is calculated. If the IU is greater than a preset threshold, the two are listed as candidate matching pairs. Subsequently, optimal matching based on metric learning is performed. The system constructs a cost matrix and comprehensively calculates the comprehensive matching cost of each candidate pair. This cost is calculated based on the 2D IU cost, the semantic similarity cost of the classification result, and the Euclidean distance cost between the back-projection ray of the visual detection box and the center of the radar 3D box. The cost matrix is ​​solved by a global optimization algorithm to find the one-to-one matching result with the minimum total cost and to eliminate invalid matches with excessively high costs. For successfully matched detection pairs, the system performs attribute fusion, prioritizing the radar's position and size attributes, combining them with the visual category attributes, and jointly calculating the final confidence of the entity.

[0028] The fourth stage is entity state generation and output. To ensure data continuity, the system introduces a cross-frame tracking mechanism. The system initializes a tracker and assigns a globally unique ID to each newly appearing fused entity, and uses Kalman filtering to predict its state in subsequent frames, updating the new observations with active trackers. Finally, the system serializes an entity state list containing timestamps, entity IDs, semantic categories, fused 3D state data, and metadata, and publishes it through a dedicated data bus. After subscribing to this data, the digital twin rendering engine drives the digital model in the virtual workshop to move synchronously, thus achieving a full-process, modular, and high-precision real-time mapping from the physical workshop to the digital twin. This complete processing flow, without modifying any underlying code, can adapt to changes in different sensor hardware simply by adjusting configuration files, fully demonstrating the beneficial effects of the present invention.

[0029] It should be clarified that the digital twin data modular interface implementation method described in this invention operates based on a specific computer system architecture, which includes a memory, a processor, and several functional software modules running on the processor. The pluggable parsing module pool physically manifests as a set of dynamic link library files or containerized microservice image repositories stored in a specific directory on the server's non-volatile storage medium; logically, it is a registry structure that records the index information, version number, and supported protocol type key-value pairs of all available parser programs. The parser module refers to an executable program code segment or independent process loaded from the above module pool into the computer's random access memory (RAM) and instantiated for execution. Structurally, each parser module encapsulates two core sub-units: one is a network communication interface unit, containing TCP / UDP sockets or HTTP / MQTT client code, used to physically connect to sensors according to instructions in the configuration file; the other is a decoding algorithm unit, containing a specific format decoding library, used to convert binary streams into readable data.

[0030] The system processes raw heterogeneous data, which refers to the initial data stream directly acquired by physical sensor hardware and transmitted to the system via the network. Its physical form is an unprocessed binary bitstream or raw text message. Specifically, for vision sensors, raw heterogeneous data manifests as compressed video frames in H.264 or H.265 format transmitted according to the RTSP protocol; for LiDAR, it manifests as hexadecimal data packets containing distance and reflectivity information transmitted according to the UDP protocol; and for traditional IoT devices, it manifests as Modbus register values ​​or key-value pairs in JSON format. Before being processed by the parser module, these data have different data structures, byte order, and encoding methods, making them unreadable by general-purpose programs.

[0031] The standardized data bus connecting the parsing end and the processing end is manifested in the system as a message queue middleware service (such as Apache Kafka, RabbitMQ or MQTT Broker) deployed on the server. It allocates several memory buffers with independent identifiers (Topics) for temporarily storing and forwarding standardized intermediate data objects.

[0032] The data mapping and fusion module is a core data processing software service running on a high-performance computing unit (usually equipped with a GPU accelerator card). It consists of three logical sub-units: First, the spatiotemporal alignment unit stores pre-calibrated sensor extrinsic and intrinsic parameter matrices for performing coordinate transformation operations; second, the feature extraction unit loads pre-trained deep neural network model files (such as YOLO weight files or PointPillars network structures) to infer target features from images and point clouds; and finally, the association and fusion unit embeds the logic of Kalman filtering and Hungarian matching algorithms to generate the final entity state.

[0033] The back-end processing module for final consumer data typically refers to a digital twin rendering engine or a Supervisory Control and Data Acquisition (SCADA) system. This module connects to a standardized data bus via network sockets, subscribes to the fused entity state list, and maps this data to transformation matrices or animation state machine parameters of the 3D model in the virtual scene, thereby presenting a dynamic picture synchronized with the physical world on the display terminal. Example

[0034] This embodiment, based on embodiment 1, proposes a specific process for achieving deep fusion of multi-source heterogeneous data by executing four stages in the data mapping and fusion module.

[0035] After the data mapping and fusion module receives multi-source heterogeneous data from the standardized data bus, it first enters the spatiotemporal alignment preprocessing process. This process aims to solve the problems of inconsistent sampling frequencies and independent spatial coordinate systems between different sensors. The system maintains first-in-first-out (FIFO) cache queues for image data and point cloud data in memory respectively. Given that the sampling frequency of radar is usually lower than that of cameras (e.g., radar 10Hz, camera 30Hz), the system adopts a time synchronization strategy based on low-frequency radar frames. Whenever a frame of radar data with a specific hardware timestamp arrives, the system immediately locks the timestamp and traverses the image cache queue to retrieve the two nearest neighbor images whose timestamps are located before and after the radar frame time. Subsequently, the system executes a linear interpolation algorithm to perform weighted calculations on the image pixels based on the time difference ratio between the radar frame timestamp and the timestamps of the two preceding and following image frames, thereby synthesizing a virtual image frame that is completely synchronized with the current radar frame in the time dimension. After completing time synchronization, the system immediately executes a spatial alignment algorithm, reads the preset sensor physical location coordinates and installation orientation Euler angles from the configuration file, and constructs a rigid body transformation matrix (i.e., extrinsic parameter matrix) to transform the radar coordinate system to the camera coordinate system. Using this matrix, the system multiplies the coordinates of each 3D spatial point in the radar point cloud data by the transformation matrix, uniformly transforming them to the camera coordinate system. Then, the system calls the camera's intrinsic parameter matrix (containing parameters such as focal length and principal point coordinates), and uses perspective projection to project the transformed 3D points onto a 2D image plane, calculating the corresponding 2D pixel coordinates of each 3D point on the image, thus establishing a one-to-one mapping relationship between sparse point clouds and dense image pixels.

[0036] After spatiotemporal alignment is completed, the system enters an independent perception and feature extraction process, simultaneously activating the visual perception pathway and the radar perception pathway. In the visual perception pathway, the system inputs the spatiotemporally aligned, standardized image into a pre-trained deep learning 2D object detection model (such as the YOLOv8 architecture). The model extracts image features through a convolutional neural network and outputs a list of 2D bounding boxes containing various types of targets (workers, AGVs). The data structure of each bounding box includes a category ID, a confidence score, and the pixel coordinates of the box center. Simultaneously, in the radar perception pathway, the system inputs aligned point cloud data into a 3D object detection network. The network performs voxelization feature extraction on the point cloud and regresses a list of 3D bounding boxes for obstacles in the scene. This list includes the 3D coordinates of the geometric center of the target, its length, width, and height dimensions, and its orientation angle around the Z-axis. At this point, although the system has obtained 2D and 3D detection results respectively, these two result sets are independent and have not yet established a connection between corresponding entities.

[0037] To unify the mapping of independent perception results to physical entities, the system then executes a multi-dimensional data association and fusion process. This process employs a coarse-to-fine cascade association strategy. First, a projection-based coarse association algorithm is executed. Using the transformation matrix calculated in the aforementioned spatial alignment stage, the system projects the eight vertices of each 3D detection box sensed by the radar onto the 2D image plane, generating the corresponding 2D projected convex hull or minimum bounding rectangle. Subsequently, the system calculates the intersection-over-union (IoU) ratio between this 2D projected box and all 2D bounding boxes output by the visual perception path. When the calculated IoU value is greater than a preset threshold, it is determined that the 3D target and the corresponding 2D target have a potential association and are listed as candidate matching pairs. To address the mismatch problem in complex scenarios, the system further executes an optimal matching algorithm based on metric learning. The system constructs a cost matrix with the number of rows equal to the number of visual targets and the number of columns equal to the number of radar targets. Each element in the matrix represents the comprehensive matching cost of a candidate pair. The cost calculation formula integrates three weights: first, the geometric cost based on the two-dimensional intersection-union ratio; second, the semantic similarity cost based on the classification results; and third, the cost based on spatial distance (calculating the Euclidean distance between the back-projection ray of the visual detection box center and the center of the radar 3D box).

[0038] After construction, the system uses the Hungarian algorithm to solve the global cost matrix, seeking the one-to-one allocation scheme that minimizes the total matching cost, and filtering out matching pairs whose final cost still exceeds a set safety threshold. For successfully matched target pairs, the system performs weighted attribute fusion: location and size attributes directly use higher-precision radar measurements, while category attributes prioritize visual classification results or undergo weighted voting. The final confidence score is calculated using a joint probability formula based on dual resetting confidence scores from both visual and radar measurements.

[0039] Finally, the system executes the entity state generation and output process to generate continuous and stable digital twin-driven data. The system introduces a cross-frame tracking mechanism, initializing a Kalman filter for each newly appearing fused entity and assigning it a globally unique tracking ID. In subsequent data frames, the system first uses the state prediction equation of the Kalman filter to calculate the prior state of the current frame based on the posterior estimate of the previous frame; then, it uses an association algorithm to match the newly generated fused observations of the current frame with the active trackers. For trackers that are successfully matched, the prior state is corrected using the measurement update equation combined with the current observations to obtain the optimal posterior estimate.

[0040] Finally, the system encapsulates the processed data into a standardized list of entity states. This list includes a unified timestamp, a unique entity ID, a semantic category, a fused and corrected high-precision 3D position, velocity vector, attitude angle, and metadata information. After serialization, this list is published through a dedicated data bus topic for downstream digital twin rendering engines to subscribe to and parse, thereby driving the digital model and physical entity in the virtual workshop to perform millisecond-level synchronous movement, achieving real-time mapping throughout the entire process. Example

[0041] refer to Figure 2 Based on Embodiment 1, this embodiment proposes a terminal device for implementing a modular interface for digital twin data. The terminal device 200 includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0042] The memory 210 may include a readable medium in the form of volatile memory, such as RAM 211 and / or cache memory 212, and may further include ROM 213.

[0043] The memory 210 also stores a computer program that can be executed by the processor 220, causing the processor 220 to perform an application of any of the above-described implementation methods of a digital twin data modular interface in this application embodiment. The specific implementation method and the achieved technical effects are consistent with those described in the above-described application embodiments, and some details will not be repeated here. The memory 210 may also include a program / utility 214 having a set (at least one) of program modules 215. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0044] Accordingly, processor 220 can execute the aforementioned computer program, as well as executable program / utility 214.

[0045] Bus 230 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0046] Terminal device 200 can also communicate with one or more external devices 240, such as keyboards, pointing devices, Bluetooth devices, etc., and with one or more devices capable of interacting with it, and / or with any device that enables it to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via I / O interface 250. Furthermore, terminal device 200 can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of terminal device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms. Example

[0047] This embodiment proposes a readable storage medium for implementing a digital twin data modular interface. The computer-readable storage medium stores instructions that, when executed by a processor, implement any of the above-described methods for implementing a digital twin data modular interface. The specific implementation method and the technical effects achieved are consistent with those described in the embodiments of the above applications, and some details will not be repeated.

[0048] Figure 3 The present embodiment illustrates a program product 300 for implementing the above-described applications. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product 300 of the present invention is not limited thereto. In this embodiment, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. The program product 300 may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0049] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0050] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing a modular interface for digital twin data, characterized in that, Includes the following steps: S1. Construct a unified data abstraction layer and define access descriptors. Read configuration files from the system's predefined directory and map different types of sensor hardware information to sensor description objects in memory. The configuration files adopt a structured text format and define the sensor's identity, communication connection configuration, data specification description, and semantic and processing configuration. S2. Dynamic loading of parsing modules and standardization of data streams: Based on the field information in the sensor description object, the system matches the corresponding parser module in the preset pluggable parsing module pool. After the parsing module is instantiated, it establishes a connection with the physical sensor using the connection information in the descriptor and converts the received raw heterogeneous data into an intermediate data object with a unified structure. The intermediate data object contains at least a unique sensor identifier, a unified timestamp, and a data payload that conforms to a predefined architecture. S3. Data distribution based on a standardized bus: The converted intermediate data objects are published to the standardized data bus, and the backend processing module obtains data by subscribing to topics with independent identifiers. S4. Spatiotemporal alignment and fusion mapping of multi-source heterogeneous data: The data mapping and fusion module subscribes to standardized image and point cloud data on the data bus, and sequentially performs spatiotemporal alignment processing, independent perceptual feature extraction, multidimensional data association and fusion, and entity state generation. Finally, it outputs a list of entity states with unique IDs to drive the digital twin.

2. The method for implementing a modular interface for digital twin data according to claim 1, characterized in that, The access descriptor in S1 is specifically: Identification and basic attributes, including unique identifiers within the system, sensor category and model; Connection and communication configuration, including communication protocol type, access address, and protocol-specific parameters; Data specification description, including the original data format, data encoding method, data frequency, and data architecture definition of key fields; Semantics and processing configuration, including physical location coordinates, installation orientation, recommended parsing module name, and preprocessing steps.

3. The method for implementing a modular interface for digital twin data according to claim 1, characterized in that, The step of matching the corresponding parser module in S2 is as follows: The system first checks whether the expected parsing module name is explicitly specified in the sensor description object. If it exists, the specified module is loaded directly. If it is not specified, the system generates a composite key value based on the communication protocol field and data format field in the sensor description object, and automatically matches and loads the corresponding general parsing module by querying the parser registry.

4. The method for implementing a modular interface for digital twin data according to claim 1, characterized in that, The spatiotemporal alignment process in S4 includes time synchronization and spatial alignment; The time synchronization is based on high-precision hardware timestamps. In the data cache queue, the frame timestamps of low-frequency sensors are used as a reference. The two frames with the closest timestamps are found in the image queue. Virtual image frames at the corresponding time are generated by linear interpolation or minimum time difference. The spatial alignment is based on a pre-calibrated extrinsic matrix, which transforms the three-dimensional points in the radar point cloud coordinate system to the camera coordinate system. The camera intrinsic matrix is ​​then used to project the three-dimensional points onto the two-dimensional image pixel coordinate system, thus establishing a spatial mapping relationship between the three-dimensional points and the two-dimensional pixels.

5. The method for implementing a modular interface for digital twin data according to claim 4, characterized in that, The independent perceptual feature extraction in S4 specifically includes: The visual perception pathway inputs the aligned image into the two-dimensional object detection model and outputs a list of two-dimensional bounding boxes containing category ID, confidence score, and center point coordinates. The radar perception pathway inputs the aligned point cloud into the 3D target detection network and outputs a list of 3D bounding boxes containing category ID, confidence level, 3D center position, size, and orientation angle.

6. The method for implementing a modular interface for digital twin data according to claim 5, characterized in that, The multidimensional data association and fusion in S5 adopts a cascading association strategy, specifically as follows: Based on the coarse correlation stage of projection, the transformation matrix calculated in the spatial alignment stage is used to project the vertices of the 3D detection box sensed by the radar onto the image plane to generate a 2D projection box. Calculate the intersection-union ratio between the two-dimensional projection frame and the two-dimensional bounding box output by the visual perception path; When the crossover ratio is greater than the preset threshold, the corresponding 3D detection results and 2D detection results are used to form a candidate matching pair.

7. The method for implementing a modular interface for digital twin data according to claim 6, characterized in that, The cascaded association strategy also includes an optimal matching stage based on metric learning, specifically: Construct a cost matrix and calculate the comprehensive matching cost for each candidate matching pair. The calculation of the comprehensive matching cost is based on the two-dimensional intersection-union ratio cost, the semantic similarity cost of the classification results, and the Euclidean distance cost between the back projection ray of the visual detection box and the center of the radar three-dimensional box. The cost matrix is ​​input into a global optimization algorithm to find the one-to-one matching result that minimizes the total matching cost. Matches with a matching cost higher than a set threshold are discarded and considered invalid matches.

8. The method for implementing a modular interface for digital twin data according to claim 7, characterized in that, The entity state generation in S4 includes an attribute fusion step, specifically: For successfully matched detection pairs, their attributes are weighted and fused; the position coordinates and size attributes of the entities are mainly based on radar detection data. The category attribute of an entity is selected from the category with higher confidence or a weighted vote is applied; The final confidence level of an entity is calculated jointly based on visual confidence level and radar confidence level.

9. The method for implementing a modular interface for digital twin data according to claim 8, characterized in that, The entity state generation in S4 also includes cross-frame tracking and state output steps, specifically: Initialize a tracker and assign a globally unique ID to each newly emerging fused entity; The state of the tracker in the current frame is predicted using Kalman filtering, and the fused entity in the current frame is updated with the active tracker using an optimization algorithm. The output includes a standardized state list containing timestamps, entity IDs, semantic categories, fused 3D state data, and metadata, which is published to a dedicated data bus for the digital twin rendering engine to subscribe to.

Images