A method of managing complex data sets in a digital twin interactive application

Abstract

The application relates to the technical field of digital twinning, in particular to a method for managing complex data sets in a digital twinning interactive application, which comprises the following steps: step 1, creating a main data storage; step 2, configuring an index structure; step 3, establishing an index association relationship, defining the association relationship between indexes, forming an association graph, and maintaining each index and its associated index set through a mapping table, wherein the mapping table is a key-value pair structure, the key is an index identifier, and the value is an associated index identifier set; step 4, data object adding operation; step 5, data object query operation; step 6, data object modification operation; step 7, data object deletion operation; and step 8, dynamic index expansion. Through the accurate index configuration and data operation mechanism, the complex data sets in the digital twinning interactive application can be efficiently stored, queried and updated. Especially when the data volume is large, the system can provide fast query and efficient dynamic expansion capability.

Description

Technical Field

[0001] This invention relates to the field of digital twin technology, and in particular to a method for managing complex datasets in digital twin interactive applications. Background Technology

[0002] Digital twin interactive applications are virtual models based on physical entities or processes, used for real-time simulation, monitoring, and optimization of actual systems. In such applications, the datasets are typically dynamic, interconnected, and complex, containing multi-dimensional information such as the attributes, installation location, status, and type of smart devices. This data needs to support efficient real-time querying, updating, and range operations to meet the performance requirements of interactive applications.

[0003] In existing technologies, managing such datasets often employs traditional data structures such as arrays, hash tables, or various tree structures. Arrays have a contiguous memory layout and fast traversal speed, but querying a specific value requires a linear scan, deletion operations are inefficient, and range queries require traversing the entire array. Hash tables support fast single-point queries and deletions, but their fragmented memory leads to inefficient traversal, and range queries also require a full table traversal. Tree structures (such as B-trees or red-black trees) perform well in range queries, but insertion and deletion speeds are slower than hash tables, and they suffer from poor memory access locality and low traversal efficiency.

[0004] In digital twin development, to support multiple query patterns (such as by device ID, name, status, etc.), developers typically need to manually maintain multiple containers (such as multiple hash tables or arrays) to index the same dataset. This approach leads to code redundancy, maintenance difficulties, and is prone to introducing data inconsistency issues. Furthermore, when business requirements change (such as changing the query dimension from building to floor), the existing structure lacks flexibility and often requires extensive code refactoring.

[0005] While relational databases offer indexing capabilities, they involve disk I / O operations, which cannot meet the low-latency requirements of real-time interactive applications. Pure in-memory data structures, on the other hand, lack advanced indexing mechanisms, making it difficult to support multi-dimensional and highly correlated query needs. Therefore, existing technologies suffer from performance bottlenecks, insufficient flexibility, and difficulties in guaranteeing data consistency in digital twin scenarios with high real-time requirements, multi-dimensional queries, and dynamic updates. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides a method for managing complex datasets in digital twin interactive applications, comprising the following steps:

[0007] Step 1: Create the main data storage. Initialize the main data storage area in memory to store data objects, which are entities in the digital twin application. The underlying structure of the main data storage area is selected as an array or a hash table according to the application requirements, and a unique identifier is assigned to each data object. The unique identifier is a 32-bit unsigned integer.

[0008] Step 2: Configure the index structure and create multiple indexes for the main data storage area. Each index corresponds to a query dimension. Extract key values ​​from data objects based on the key extraction function. The type of each index is independently configured as a hash index or a sorted index. The hash index uses a hash table to store the mapping between key values ​​and data object identifiers, and the sorted index uses an ordered array to store the mapping between key values ​​and data object identifiers.

[0009] Step 3: Establish index relationships, define the relationships between indexes to form a relationship graph, and maintain each index and its associated index set through a mapping table. The mapping table is a key-value pair structure, where the key is the index identifier and the value is a set of associated index identifiers.

[0010] Step 4: Data object addition operation. When a new data object is added to the main data storage area, a unique identifier is generated and the data object is stored in the main data storage area. All configured indexes are traversed. For each index, the key extraction function is used to extract the key value from the data object and the key value and data object identifier are inserted into the index's mapping structure.

[0011] Step 5: Data object query operation. Receive the query request, specify the target index and query conditions, execute the query according to the index type. For a hash index, locate the identifier set corresponding to the key value through hash lookup. For a sorted index, select linear scan or binary search algorithm to determine the query range based on the data size, and return the identifier set.

[0012] Step 6: Data object modification operation. When the attribute of a data object is modified, locate the identifier of the data object in the main data storage area. For all indexes affected by the attribute modification, call the change function to update the attribute value in the data object, remove the old key value and identifier mapping from the index, insert the new key value and identifier mapping, and automatically trigger the linkage update of all related indexes according to the association relationship.

[0013] Step 7: Data object deletion operation. When deleting a data object, the data object and its identifier are removed from the main data storage area. All indices are traversed, and the key-value entry corresponding to the identifier is removed from the mapping structure of each index. If a custom destructor is configured, the custom destructor is called to release the resources occupied by the data object.

[0014] Step 8: Dynamic index expansion. When a new query dimension is added, a new index is dynamically added. The key extraction function and type of the new index are defined. All existing data objects in the main data storage area are traversed to extract key values ​​and build the mapping structure of the new index. The new index is then incorporated into the relationship management.

[0015] Preferably, in step 2, the configuration index structure further includes:

[0016] A change function is set for each index, which is called when the key value of the data object is modified to perform the actual key value modification operation;

[0017] The definition of the change function includes: receiving a data object and a new key value as input, modifying the value of the corresponding attribute in the data object, and returning the modification result;

[0018] When the index is created, the change function is registered to the index management unit via parameter passing to ensure that it is automatically called in the data object modification operation in step 6.

[0019] The modified function supports type safety checks, which verify the type matching of input parameters and return values ​​at compile time to avoid runtime errors.

[0020] Preferably, in step 3, establishing the index association further includes:

[0021] The association graph uses a directed graph structure to represent the dependencies between indexes, where nodes are indexes and edges represent the linkage relationships between indexes;

[0022] The maintenance of the mapping table includes: when a new index is added, checking the association requirements of existing indexes and dynamically updating the mapping table;

[0023] In the data object modification operation in step 6, all related indexes are queried according to the mapping table, and update operations are performed in batches to ensure data consistency;

[0024] The association supports multi-level nesting, meaning that a modification to one index can trigger recursive updates to multiple levels of associated indexes.

[0025] Preferably, in step 5, the data object query operation further includes:

[0026] For range queries on sorted indexes, the query algorithm is intelligently selected based on the size of the dataset. When the data volume is less than a preset threshold, a linear scan is used, and when the data volume is greater than or equal to the preset threshold, a binary search is used.

[0027] The preset threshold is dynamically adjusted based on historical query performance data and is obtained by monitoring query response time statistics.

[0028] The implementation of the binary search algorithm includes: first, determining the upper and lower bound key values ​​of the query range, using binary search to locate the lower bound position, and then linearly traversing to the upper bound position to collect all identifiers within the range;

[0029] After the query results are returned, secondary filtering is supported, allowing you to filter the set of identifiers based on additional conditions.

[0030] Preferably, in step 1, creating the master data storage further includes:

[0031] The selection of the underlying structure of the main data storage area is based on data operation frequency and query mode. Among them, array structure is suitable for traversal-intensive scenarios, and hash table structure is suitable for addition and deletion-intensive scenarios.

[0032] The array structure is implemented using contiguous memory allocation, supporting fast sequential access;

[0033] The hash table structure is implemented using the open-chain method to resolve hash collisions and dynamically adjusts the hash bucket size to maintain the load factor.

[0034] The allocation of unique identifiers is achieved through a global counter, ensuring the uniqueness and incrementing nature of the identifiers.

[0035] Preferably, in step 2, the configuration index structure further includes:

[0036] The definition of the key extraction function supports multi-attribute composite keys, that is, extracting multiple attribute values ​​from a data object and combining them into a composite key;

[0037] Composite keys can be generated in various ways, including tuples, custom structures, string concatenation, numerical encoding, or hash fusion.

[0038] For sorted indexes, the comparison rules for composite keys are based on lexicographical order or a custom sorting function;

[0039] Preferably, in step 4, the data object addition operation further includes:

[0040] Perform data validation before adding operations, including checking the integrity of data object attributes and the validity of key values;

[0041] If verification fails, refuse to add and return an error message;

[0042] The add operation supports batch processing, which means adding multiple data objects at once. First, the integrity and validity of all objects are checked, then the information needed to build the index is collected in a centralized manner, and finally the objects are added in a unified manner, which improves efficiency.

[0043] Preferably, in step 6, the data object modification operation further includes:

[0044] Before making any changes, check the validity of the attribute modification, including key-value type matching and range constraints.

[0045] Preferably, in step 7, the data object deletion operation further includes:

[0046] Calls to custom destructors support resource cleanup hooks, allowing users to define cleanup logic, such as closing network connections or releasing file handles;

[0047] Preferably, in step 8, the dynamic index expansion further includes:

[0048] The addition of new indexes supports hot deployment, meaning that index building can be completed without stopping system services;

[0049] During hot deployment, data modification operations are suspended, and only query operations are allowed to ensure data consistency;

[0050] Before activating a new index, a consistency check is performed, comparing the new index with existing data. Only after the check passes can the index be included in the query process.

[0051] The beneficial effects of this invention are:

[0052] 1. By configuring multiple independent indexes for each data object and dynamically expanding the index structure according to the query dimensions of the data object, this invention can significantly improve performance in high-frequency data operations and multi-dimensional query scenarios. Through the flexible configuration of hash indexes and sorted indexes, the optimal query method is selected for different query needs, avoiding the problem of traversing the entire dataset in existing technologies, thus greatly improving query speed and update efficiency.

[0053] 2. This invention supports the dynamic addition of new indexes when new query dimensions are added through a dynamic index expansion mechanism. It defines the key extraction function and type of the new index and constructs the new index in the main data storage area. This flexible index expansion allows the system to easily adapt to changes in business requirements, avoiding the refactoring problems of existing technologies and significantly improving the system's flexibility and scalability.

[0054] 3. This invention establishes index relationships and automatically triggers linked index updates when data objects are modified, ensuring data consistency across all indexes. By maintaining indexes and their relationships through a mapping table, modifications are automatically synchronized to the relevant indexes, avoiding data inconsistencies caused by manually maintaining multiple containers and improving data integrity and consistency.

[0055] 4. This invention supports batch data addition operations, and if the operation fails midway, it can be rolled back to the state before the operation. This mechanism effectively avoids the data inconsistency problem in the prior art, ensuring the reliability and efficiency of batch operations.

[0056] 5. This invention provides support for custom destructors in data object deletion operations, allowing users to define resource cleanup logic, such as closing network connections or releasing file handles. After deletion, an index compression mechanism is triggered to reclaim unused memory space, ensuring efficient use of system resources and avoiding memory fragmentation and resource waste.

[0057] 6. This invention employs an intelligent query algorithm. For range queries, it dynamically selects between linear scanning and binary search based on the size of the data set, improving query efficiency. Furthermore, the query results support secondary filtering, filtering the identifier set based on additional conditions, further enhancing query accuracy and performance. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0060] Figure 2 The memory layout diagram for the ValueContainer;

[0061] Figure 3 A memory layout diagram for the TableIndex table index;

[0062] Figure 4 This is a memory layout diagram for the Table;

[0063] Figure 5 A flowchart for adding an index to a table;

[0064] Figure 6 A flowchart for adding data objects to a table;

[0065] Figure 7 A flowchart for initiating a query for a table;

[0066] Figure 8 Flowchart for initiating a range query for a table;

[0067] Figure 9 A flowchart for initiating modifications to a Table;

[0068] Figure 10 A flowchart for initiating a deletion of a table. Detailed Implementation

[0069] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0070] Please see Figures 1-3 This invention provides a method for managing complex data sets in digital twin interactive applications. The specific embodiments of the invention are described in detail below with reference to the accompanying drawings. This embodiment aims to fully disclose the technical solution of the invention, enabling those skilled in the art to understand and implement it. The core of the method lies in achieving efficient data management through multi-dimensional indexing, linked indexes, and configurable storage strategies. The method includes steps such as creating master data storage, configuring the index structure, establishing index relationships, adding data objects, querying data objects, modifying data objects, deleting data objects, and dynamically expanding the index.

[0071] Step 1: Create the primary data store:

[0072] Initialize the main data storage area in memory to store data objects. Data objects are entities in digital twin applications, such as smart devices, containing multiple attributes including device ID, type, status, and location. The underlying structure of the main data storage area is chosen based on application requirements: either an array or a hash table. Array structures are suitable for traversal-intensive scenarios, providing contiguous memory access; hash table structures are suitable for scenarios with frequent additions and deletions, providing fast key-value access. The selection criteria include data operation frequency and query patterns: if the application needs to frequently traverse all data objects (such as counting all devices in the system), an array structure is chosen; if fast lookup, addition, or deletion based on a unique key (such as device ID) is required, a hash table structure is chosen. Array structures are implemented using contiguous memory allocation, with the initial memory block size set to accommodate the expected number of data objects, and support dynamic expansion: when the array is full, a larger memory block is reallocated, and the original data is copied to the new memory block. Hash table structures are implemented using an open-chaining method to resolve hash collisions; each hash bucket is a linked list storing entries with the same hash value. The load factor of the hash table (the ratio of the number of elements to the number of buckets) is monitored in real time. When the load factor exceeds a preset threshold (such as 0.75), the hash table automatically adjusts its size: the number of new buckets is twice the number of original buckets, and all entries are rehashed to maintain query efficiency.

[0073] The allocation of unique identifiers is implemented through a global counter. The global counter is a 32-bit unsigned integer variable, initialized to 0 at system startup. Each time a new data object is added, the counter is incremented, and the current value is assigned to that object as its identifier. This ensures the uniqueness and incrementing nature of the identifiers, facilitating internal referencing and management. Using identifiers instead of pointers to store data object references reduces memory footprint (32-bit integers are generally smaller than pointers) and improves caching efficiency because identifiers can be stored contiguously.

[0074] The specific memory layout of the main data storage area depends on the chosen structure. In an array structure, data objects are stored contiguously in the order they are added, with the identifier serving as the array index. In a hash table structure, data objects are stored in hash buckets, with the identifier as the key and the data object as the value. The hash function calculates the bucket index based on the identifier, for example using modulo operation: bucket index = identifier % number of buckets. To reduce collisions, the hash function can be combined with multiplicative hashing or circular shift.

[0075] Step 2: Configure the index structure:

[0076] Multiple indexes are created in the main data storage area, each corresponding to a query dimension (such as device ID, status, location, etc.). The indexes extract key-value pairs from data objects using key extraction functions. These key extraction functions are user-defined functions that take data objects as input and output key-value pairs. The function implementation depends on the programming language: in C++, it can be a function pointer or a function object; in Java, it can be a lambda expression or a method reference. For example, for the device status index, the key extraction function returns the status attribute value of the data object.

[0077] Each index is independently configured as either a hash index or a sorted index. Hash indexes use hash tables to store the mapping between key-value pairs and data object identifiers. The internal structure of a hash table is similar to the hash table in the main data store, but the keys are the extracted key-value pairs (such as status strings), and the values ​​are sets of identifiers (such as arrays or linked lists). Sorted indexes use ordered arrays to store the mapping between key-value pairs and identifiers. The ordered array can be implemented based on a balanced binary search tree (such as a red-black tree) or a sorted list. Balanced trees support efficient insertion, deletion, and range queries, but have higher memory overhead; sorted lists are memory-efficient, but modifications require reordering. The choice depends on the query pattern: if range queries are frequent (such as building number between1 and 5), choose a sorted index; if queries are mainly single-point queries, choose a hash index.

[0078] Each index can have a modification function. The modification function is invoked when the key-value pair of a data object is modified, and it performs the actual key-value modification operation. The definition of a modification function includes: receiving the data object and the new key-value pair as input, modifying the value of the corresponding attribute in the data object, and returning the modification result (e.g., success or failure). Modification functions are registered with the index management unit via parameter passing when the index is created. The registration process involves storing a function pointer or function object in the index metadata. Modification functions support type safety checks: at compile time, the input parameter types are verified to match the data object type and key-value pair type using templates or generics, and the output type is consistent with expectations. For example, in C++, `std::function` and `static_assert` are used for compile-time checks.

[0079] The key extraction function supports multi-attribute composite keys. Composite keys are composed of multiple attribute values ​​extracted from a data object. Combination methods include string concatenation (e.g., concatenating building number and floor number to "1-5"), numerical encoding (e.g., left-shifting the building number by 16 bits and then bitwise ORing it with the floor number), or hash fusion (e.g., calculating the XOR of the hash values ​​of each attribute). For sorted indexes, the comparison rules for composite keys are based on lexicographical order (for strings) or a custom comparison function. Custom comparison functions allow users to define sorting logic, such as first sorting by building number in ascending order, then by floor number in descending order.

[0080] Step 3: Establish index relationships:

[0081] Define the relationships between indices to form a relationship graph. The relationship graph is a directed graph structure where nodes represent indices and edges represent the linkages between indices (e.g., a modification to index A triggers an update to index B). These relationships are maintained through a mapping table, which is a key-value pair structure. The key is an index identifier (e.g., an index name string), and the values ​​are the set of all other index identifiers associated with that index. The mapping table can be implemented using a hash table, such as `std::unordered_map` in C++.<std::string,std::unordered_setstd::string> .

[0082] The association graph is constructed based on business logic. For example, in an equipment management system, both the building number index and the floor number index affect the building-floor composite index; therefore, the composite index is associated with both the building number index and the floor number index. These associations can be defined via API during the system configuration phase. When a new index is added, the system parses the association requirements and updates the mapping table. For example, when adding a new index, "Gateway Index," if it is associated with the "Region Index," an entry is added to the mapping table.

[0083] During data modification operations, when the key value of an index changes, the system queries the mapping table to find all related indexes and performs batch updates. The update process is recursive: if index B is related to index A, and index C is related to index B, then modifying index A will trigger updates to indexes B and C. The recursion depth is limited by the system to avoid infinite loops.

[0084] Step 4: Adding data objects:

[0085] When a new data object is added to the main data storage area, the system performs the following sub-steps:

[0086] Generate a unique identifier: Increment the global counter and assign a new identifier.

[0087] Store data objects in the main data storage area: depending on the underlying structure, if it is an array, append the object to the end of the array; if it is a hash table, insert the identifier and the object into the hash table.

[0088] Iterate through all configured indexes: For each index, use the key extraction function to extract the key-value pair from the data object, and then insert the key-value pair along with an identifier into the index's mapping structure. For hash indexes, this means adding key-value pairs to a hash table; for sorted indexes, this means inserting new elements into a sorted array while maintaining the sort order.

[0089] Data validation is performed before adding data. Validation includes attribute integrity checks (e.g., all required attributes are not empty), key-value validity checks (e.g., device ID conforms to format rules), and uniqueness checks (e.g., device ID is not duplicated). If validation fails, the add operation is rejected, and an error code and message are returned.

[0090] The add operation supports batch processing. During batch processing, the system receives multiple data objects at once;

[0091] Step 5: Data object query operation:

[0092] The system receives query requests, specifying the target index and query conditions. Query conditions can be single-point queries (e.g., a key value equals a specific value) or range queries (e.g., a key value is within a range). For hash indexes, queries are implemented through hash lookups: the hash value of the query key is calculated, the hash bucket is located, elements within the bucket are traversed to match the key value, and a set of identifiers is returned. Hash lookups have a time complexity of O(1), but in the worst case (full collisions), it is O(n).

[0093] For sorted indexes, range queries intelligently select the algorithm based on the dataset size. Algorithm selection is based on a preset threshold: if the data volume is less than the threshold, a linear scan is used; otherwise, a binary search is used. The threshold is dynamically adjusted: the system monitors historical query performance and calculates the average response time for linear scans and binary searches. As the data volume increases, the linear scan time increases linearly, while the binary search time increases logarithmically. The threshold is initially set to 1000 but is adjusted periodically: if the average binary search time is lower than the linear scan time, the threshold is lowered; otherwise, it is raised. Monitoring data is stored in a circular buffer, and the threshold is recalculated every 1000 queries.

[0094] The implementation of the binary search algorithm includes:

[0095] Determine the upper and lower bounds of the query range using key values.

[0096] Use binary search to locate the lower bound: compare the key value of the middle element with the lower bound, adjust the search interval, and continue until the first element greater than or equal to the lower bound is found.

[0097] A linear traversal is performed from the upper bound position to the upper bound position, collecting all identifiers within the range. The query result returns a set of identifiers. The set can be implemented as an array or a collection data structure, supporting iterative access. The result also supports secondary filtering: the user provides a filtering function, and the system applies the function to filter the identifier set, for example, retaining only devices with the status "online".

[0098] Step 6: Data object modification operation:

[0099] When a data object's property is modified, the system executes:

[0100] Locate the identifier of the data object in the main data storage area: if the main storage is an array, access it directly through the identifier; if it is a hash table, look it up by hash using the identifier.

[0101] For all indexes affected by the attribute modification (based on relationships), the change function is invoked to update the attribute values ​​in the data object. The change function performs the actual modification, such as setting a new building number.

[0102] Remove the mapping between old key values ​​and identifiers from the index: For each index, delete the mapping entry using the old key value.

[0103] Insert a new key-value mapping to an identifier: Add a mapping entry using the new key-value.

[0104] Automatically trigger synchronized updates of all related indexes based on the relationship: find the related indexes by querying the mapping table and recursively execute the update.

[0105] Perform a validity check before making any changes. The check includes key-value type matching (e.g., the new building number must be an integer) and range constraints (e.g., the building number must be between 1 and 100).

[0106] Step 7: Data object deletion operation:

[0107] During a deletion operation, the system executes:

[0108] Remove data objects and their identifiers from the main data storage area: if it is an array, delete it by swapping the tail; if it is a hash table, delete the key-value pairs.

[0109] Iterate through all indices and remove the key-value entry corresponding to the identifier from the mapping structure of each index.

[0110] If the main data is stored in an array, deletion at the tail of the queue will cause the index to become invalid. Therefore, before deleting at the tail, the index of the deletion position and the index of the tail of the queue should be recorded. When traversing all indices, these two markers should be used to update the index once more.

[0111] If a custom destructor is configured, it is called to release resources. A custom destructor is a user-defined function used to clean up resources such as closing file handles or network connections.

[0112] Step 8: Dynamic Index Expansion;

[0113] When a new query dimension is added, a new index is dynamically added:

[0114] Define the key extraction function and type for the new index.

[0115] Iterate through all existing data objects in the main data storage area, extract key-value pairs, and construct a mapping structure for the new index. Incorporate the new index into relationship management: update the mapping table.

[0116] The new index adds support for hot deployment, meaning that system services do not need to be stopped. During hot deployment, data modification operations are paused (via a global lock or flag), but query operations are allowed to continue.

[0117] Example:

[0118] The intelligent device management system manages tens of thousands to hundreds of thousands of devices within the park, including cameras, sensors, and controllers. Each device has attributes such as: device ID (string), type (enumeration), status (online, offline, standby), building number (integer), floor number (integer), area index (string), and gateway ID (string). The system needs to support queries such as: exact search by device ID, search for all online devices by status, search by building number range, search by building-floor composite key, and search by gateway. High real-time performance is required; query response time must be in the microsecond range.

[0119] Create the main data storage area: An array structure is chosen as the underlying storage because the system needs to frequently traverse all devices (e.g., to generate reports). The initial array capacity is 100,000, with a dynamic expansion step of 50,000. The global counter is initialized to 0.

[0120] Configure the index structure: Create the following indexes:

[0121] Device ID index: hash index, key extraction function is:

[0122] [](constDevice&d){returnd.id;}.

[0123] State index: hash index, key retrieval function is:

[0124] [](constDevice&d){returnd.status;}.

[0125] Building number index: sorted index (using red-black tree), key extraction function is:

[0126] [](constDevice&d){returnd.buildingNum;}.

[0127] Floor number index: sorted index, key extraction function is:

[0128] [](constDevice&d){returnd.floorNum;}.

[0129] Building-floor composite index: sorted index, key extraction function is:

[0130] The function `[](constDevice&d){return std::make_tuple(d.buildingNum,d.floorNum);}` can return a tuple directly or a custom struct, allowing you to overload comparison operators for the struct. If it's a hash index, the custom struct needs to support a hash function.

[0131] Register a change function for each index. For example, the change function for the building number index is:

[0132] [](Device&d,intnewVal){d.buildingNum=newVal;}.

[0133] Building-floor composite index: sorted index, key change function is:

[0134] [](Device&d,conststd::tuple<int,int> &newKey){d.buildingNum=newKey.get <0> ;d.floorNum=newKey.get <1> ;}

[0135] Establish index relationships: Define the relationships: the building number index is linked to the building-floor composite index, and the floor number index is linked to the composite index. The mapping table content is as follows:

[0136] {"buildingIndex",{"buildingFloorIndex"}},{"floorIndex",{"buildingFloorIndex"}}.

[0137] Example of adding data:

[0138] Add a new device: Device ID="CAM001", Type="Camera", Status="Online", Building Number=1, Floor Number=5.

[0139] Generate identifier: Increment the global counter to 1000 and assign identifier 1000.

[0140] Data validation: Check the uniqueness of the device ID (query existing index), passed.

[0141] Add to main storage: Store the device object at array index 1000.

[0142] Update index:

[0143] Device ID Index: Key "CAM001" maps to identifier 1000.

[0144] Status index: The key "online" is mapped to 1000.

[0145] Building number index: Key 1 maps to 1000 (inserting into a red-black tree).

[0146] Floor number index: Key 5 maps to 1000.

[0147] Composite index: The key "std::tuple(1,5)" maps to 1000.

[0148] Added successfully.

[0149] Example of a query operation:

[0150] Single point query: Find devices with a status of "Online". Specify the status index, key value "Online". Hash indexes directly return a set of identifiers (e.g., [1000, 1001, ...]). Time taken: approximately 5 microseconds.

[0151] Range query: Find devices in buildings 1 to 5. Specify building index, range [1, 5]. The data volume is approximately 100,000, exceeding the threshold (1000), so a binary search is used. First, locate the lower bound of building number 1 (the smallest node in the red-black tree), then traverse to the upper bound of building number 5, returning the set of identifiers. The process takes approximately 10 microseconds.

[0152] Example of modification operation:

[0153] Change the building number of device CAM001 from 1 to 2.

[0154] Location identifier 1000.

[0155] Legality check: New building number 2 is within the valid range (1-100), passed.

[0156] Update building number index: Call the change function to modify the building number of the device object to 2. Remove the mapping for key 1 from the index and add the mapping for key 2.

[0157] Linked Update: Querying the mapping table reveals that the building number index is associated with the composite index. Update the composite index: Remove the mapping for key "std::tuple(1,5)" and add the mapping for key "std::tuple(2,5)". Modification complete, data consistency maintained.

[0158] Example of deletion operation:

[0159] Delete device CAM001.

[0160] Backup: Record a log entry {Identifier:1000, Time:"2023-10-0110:00:00", Operation:"delete"}.

[0161] Primary storage removal: Removes the device from array position 1000, marking the position as empty.

[0162] Index update: Remove the mapping for identifier 1000 from all indexes.

[0163] Calling a custom destructor: If the device has a network connection, call d.closeConnection().

[0164] Example of dynamic expansion:

[0165] Added a new query dimension: Manage devices by gateway. Added a gateway index: Hash index, with the key extraction function being [](constDevice&d){returned.gatewayId;}.

[0166] Hot deployment: Pause data modification operations.

[0167] Activate index: Update the mapping table to associate the gateway index with the relevant index (such as the region index).

[0168] Restore data modification operations.

[0169] It is understandable that:

[0170] In the C++ implementation, the modification function is encapsulated through a function pointer. Type safety checks are performed using template specialization and `static_assert`. For example:

[0171] cpp

[0172] template <typenameRemovePointerType,typename KeyExtractor , typenameChangeFunc , typename Key = std::decay_t<std::invoke_result_t<KeyExtractor,const RemovePointerType&> >> requires UF::is_convertible_to_function_pointer<KeyExtractor, Key,const RemovePointerType&> ::value && UF::is_convertible_to_function_pointer<ChangeFunc, void,RemovePointerType&,const Key&> ::value &&std::is_convertible <decltype(GetTypeHash(std::declval <key>())), int32_t>::value TableKit& AddIndex(IndexKeyType type, KeyExtractor extractor ,ChangeFunc changeFunc, IndexType indexType = IndexType::Hash , bool bUnique =false )

[0173] During compilation, if the function signature does not match (e.g., incorrect parameter types), `concept` will report an error before compilation, while `static_assert` will report an error during compilation. This prevents runtime type errors.

[0174] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0175] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / key>

Claims

1. A method for managing complex datasets in digital twin interactive applications, characterized in that, Includes the following steps: Step 1: Create the main data storage. Initialize the main data storage area in memory to store data objects, which are entities in the digital twin application. The underlying structure of the main data storage area is selected as an array or a hash table according to the application requirements, and a unique identifier is assigned to each data object. The unique identifier is a 32-bit unsigned integer. Step 2: Configure the index structure and create multiple indexes for the main data storage area. Each index corresponds to a query dimension. Extract key values ​​from data objects based on the key extraction function. The type of each index is independently configured as a hash index or a sorted index. The hash index uses a hash table to store the mapping between key values ​​and data object identifiers, and the sorted index uses an ordered array to store the mapping between key values ​​and data object identifiers. Step 3: Establish index relationships, define the relationships between indexes to form a relationship graph, and maintain each index and its associated index set through a mapping table. The mapping table is a key-value pair structure, where the key is the index identifier and the value is a set of associated index identifiers. Step 4: Data object addition operation. When a new data object is added to the main data storage area, a unique identifier is generated and the data object is stored in the main data storage area. All configured indexes are traversed. For each index, the key extraction function is used to extract the key value from the data object and the key value and data object identifier are inserted into the index's mapping structure. Step 5: Data object query operation. Receive the query request, specify the target index and query conditions, execute the query according to the index type. For a hash index, locate the identifier set corresponding to the key value through hash lookup. For a sorted index, select linear scan or binary search algorithm to determine the query range based on the data size, and return the identifier set. Step 6: Data object modification operation. When the attribute of a data object is modified, locate the identifier of the data object in the main data storage area. For all indexes affected by the attribute modification, call the change function to update the attribute value in the data object, remove the old key value and identifier mapping from the index, insert the new key value and identifier mapping, and automatically trigger the linkage update of all related indexes according to the association relationship. Step 7: Data object deletion operation. When deleting a data object, the data object and its identifier are removed from the main data storage area. All indices are traversed, and the key-value entry corresponding to the identifier is removed from the mapping structure of each index. If a custom destructor is configured, the custom destructor is called to release the resources occupied by the data object. Step 8: Dynamic index expansion. When a new query dimension is added, a new index is dynamically added. The key extraction function and type of the new index are defined. All existing data objects in the main data storage area are traversed to extract key values ​​and build the mapping structure of the new index. The new index is then incorporated into the relationship management.

2. The method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 2, the configuration index structure further includes: A change function is set for each index, which is called when the key value of the data object is modified to perform the actual key value modification operation; The definition of the change function includes: receiving a data object and a new key value as input, modifying the value of the corresponding attribute in the data object, and returning the modification result; When the index is created, the change function is registered to the index management unit via parameter passing to ensure that it is automatically called in the data object modification operation in step 6. The modified function supports type safety checks, which verify the type matching of input parameters and return values ​​at compile time to avoid runtime errors.

3. The method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 3, establishing the index association relationship further includes: The association graph uses a directed graph structure to represent the dependencies between indexes, where nodes are indexes and edges represent the linkage relationships between indexes; The maintenance of the mapping table includes: when a new index is added, checking the association requirements of existing indexes and dynamically updating the mapping table; In the data object modification operation in step 6, all related indexes are queried according to the mapping table, and update operations are performed in batches to ensure data consistency; The association supports multi-level nesting, meaning that a modification to one index can trigger recursive updates to multiple levels of associated indexes.

4. The method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 5, the data object query operation further includes: For range queries on sorted indexes, the query algorithm is intelligently selected based on the size of the dataset. When the data volume is less than a preset threshold, a linear scan is used, and when the data volume is greater than or equal to the preset threshold, a binary search is used. The preset threshold is dynamically adjusted based on historical query performance data and is obtained by monitoring query response time statistics. The implementation of the binary search algorithm includes: first, determining the upper and lower bound key values ​​of the query range, using binary search to locate the lower bound position, and then linearly traversing to the upper bound position to collect all identifiers within the range; After the query results are returned, secondary filtering is supported, allowing you to filter the set of identifiers based on additional conditions.

5. A method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 1, the creation of the master data storage further includes: The selection of the underlying structure of the main data storage area is based on data operation frequency and query mode. Among them, array structure is suitable for traversal-intensive scenarios, and hash table structure is suitable for addition and deletion-intensive scenarios. The array structure is implemented using contiguous memory allocation, supporting fast sequential access; The hash table structure is implemented using the open-chain method to resolve hash collisions and dynamically adjusts the hash bucket size to maintain the load factor. The allocation of unique identifiers is achieved through a global counter, ensuring the uniqueness and incrementing nature of the identifiers.

6. The method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 2, the configuration index structure further includes: The definition of the key extraction function supports multi-attribute composite keys, that is, extracting multiple attribute values ​​from a data object and combining them into a composite key; Composite keys can be generated in various ways, including tuples, custom structures, string concatenation, numerical encoding, or hash fusion. For sorted indexes, the comparison rules for composite keys are based on lexicographical order or a custom sorting function.

7. A method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 4, the data object addition operation further includes: Perform data validation before adding operations, including checking the integrity of data object attributes and the validity of key values; If verification fails, refuse to add and return an error message; The add operation supports batch processing, which means adding multiple data objects at once. First, the integrity and validity of all objects are checked, then the information needed to build the index is collected in a centralized manner, and finally the objects are added in a unified manner, which improves efficiency.

8. A method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 7, the data object deletion operation further includes: Calls to custom destructors support resource cleanup hooks, allowing users to define cleanup logic such as closing network connections or releasing file handles.

9. A method for managing complex datasets in a digital twin interactive application according to claim 1, characterized in that, In step 8, the dynamic index expansion further includes: The addition of new indexes supports hot deployment, meaning that index building can be completed without stopping system services; During hot deployment, data modification operations are suspended, and only query operations are allowed to ensure data consistency; Before activating a new index, a consistency check is performed, comparing the new index with existing data. Only after the check passes can the index be included in the query process.

Images