Virtual training data management system and method based on lvc simulation

Abstract

This invention discloses a virtual training data management system and method based on LVC simulation, relating to the field of LVC simulation. It includes a hierarchical hash index module, a Bloom filter module, and a streaming event processing module. The hierarchical hash index module performs unified collection and standardized mapping of raw state data from three types of data sources, ensuring cross-source data consistency and temporal accuracy. A multi-layered hash table is constructed based on the ODSLA metric set, dynamically determining the number of hash layers and optimizing object indexing and storage efficiency. Real-time monitoring of the load of each hash bucket, combined with high-load splitting and low-load merging mechanisms, achieves dynamic load balancing, improving resource utilization and processing performance. An countable mechanism is used for initial screening of object states, reducing unnecessary exact matching calculations. Combined with local caching to handle out-of-order events and rule-based exact matching, efficient and accurate event stream processing is achieved, providing reliable data support and rapid response capabilities for virtual training.

Description

Technical Field

[0001] This invention relates to the field of LVC simulation, specifically to a virtual training data management system and method based on LVC simulation. Background Technology

[0002] LVC (Live-Virtual-Constructive) simulation integrates a real system (Live), a computer-driven human-in-the-loop / human-out-of-the-loop virtual system (Virtual), and a model-based constructive system (Constructive) in the same scenario to support comprehensive training and evaluation involving large-scale, heterogeneous devices. With the rapid increase in the number of sensors, platforms, and business systems, the scale of objects, the frequency of events, and the dimensionality of data in training scenarios are increasing exponentially. Data management is shifting from "offline aggregation" to a streaming processing mode of "real-time acquisition," placing higher demands on throughput, latency, and consistency.

[0003] Existing LVC data management generally relies on standardized interfaces and simulation middleware (such as common entity status, event notifications, attribute updates, and other message paradigms) to achieve multi-source access. However, significant heterogeneity still exists at the data layer. In high-concurrency scenarios, standardization and validation themselves can become processing bottlenecks, and under network jitter, out-of-order delivery, and packet loss conditions, timing inconsistencies can be amplified, affecting the accuracy and repeatability of event triggering. Furthermore, when both the number of objects and the average update frequency of objects increase simultaneously, the index structure of static parameters is prone to hotspot buckets and uneven distribution.

[0004] Hash indexes are a mature technology in database and caching systems used for quickly locating objects or records. However, traditional hash indexes can cause some hash buckets to become overheated (hotspots) when objects are unevenly distributed or their spatial locations change dynamically, increasing query and update latency. Frequent state updates may cause frequent index rebuilds or conflicts, adding extra computation and latency.

[0005] Bloom filters are used to quickly determine the existence of elements and are widely used in caching, deduplication, and database acceleration. Standard Bloom filters have a false positive rate, which increases with the number of objects, leading to unnecessary exact match calculations and increased latency. During high-frequency updates, Bloom filters cannot delete already triggered objects, making them prone to repeated triggering or requiring additional structure management.

[0006] Existing technology, such as the invention patent with publication number CN118378802A, is an intelligent method for LVC training, including the following steps: Step 1, establishing an LVC intelligent training architecture; Step 2, establishing an intelligent method for LVC training. Step 2 includes: Step 2-1, establishing an intelligent training planning method; Step 2-2, establishing an intelligent director-led adjustment method; Step 2-3, an intelligent adjudication and evaluation method. This invention proposes an intelligent method for LVC training, constructs an LVC intelligent training architecture, and proposes methods such as intelligent training planning, intelligent director-led adjustment, and intelligent adjudication and evaluation, providing theoretical support for improving training efficiency and guiding the planning and development of future training systems.

[0007] As can be seen from the above solutions, existing technologies in the field of LVC simulation mainly focus on the intelligent architecture and methodology of LVC training. However, in terms of data management, they lack real-time processing of the underlying data stream. In practical applications, since LVC has multiple data sources, LVC simulation suffers from high concurrency latency due to multi-source heterogeneous data. Existing technologies lack optimization mechanisms at this level and are unable to support large-scale real-time training environments. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a virtual training data management system and method based on LVC simulation. To achieve the above objectives, this invention utilizes the following technical solution: a virtual training data management system based on LVC simulation, comprising:

[0009] The hierarchical hash index module is used to collect raw state data from three types of data sources (L, V, and C) and then map it into an internal object format. It constructs a multi-level hash table based on the ODSLA (hierarchical hash parameter) metric set, while monitoring the load status of each hash bucket in real time and dynamically adjusting the hash bucket. The object state located by the index enters the Bloom filter module.

[0010] The Bloom filter module is used to initially screen the object state, identify the matching objects, and remove the missing objects. Combined with the countable mechanism, it makes dynamic adjustments when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream.

[0011] The streaming event processing module is used for precise matching of event streams, manages out-of-order events through local caching, and outputs and triggers state update events for objects that are precisely matched.

[0012] As a preferred technical solution, the raw state data from three types of data sources (L, V, and C) are collected and uniformly mapped into an internal object format. The specific process is as follows:

[0013] The system collects raw state data from three types of data sources: L, V, and C. The raw state data includes unique object identifiers, spatial coordinate information, and state identifiers. The raw state data from the three types of data sources is mapped to a unified internal object format.

[0014] During the field mapping and unit unification process, coordinate transformation functions are used to uniformly map them to the internally defined global coordinate system.

[0015] The state identifiers of objects from different data sources are mapped to internally defined enumeration types through a state mapping table built into the local database.

[0016] By using time base mapping and time difference compensation algorithms, time information from various data sources is uniformly converted into internal standard timestamps.

[0017] As a preferred technical solution, a multi-layer hash table is constructed based on the ODSLA index set. The specific process is as follows:

[0018] The ODSLA metrics set is a set of parameters used for hierarchical hashing, including the number of objects, average object update frequency, spatial distribution, object density, and access pressure.

[0019] Extract the ODSLA verification set from the local database, including the number of verification objects, the average update frequency of verification objects, the spatial distribution of verification objects, the density of verification objects, and the access pressure of verification.

[0020] The data processing load characteristic value is obtained by comparing the parameters in the ODSLA indicator set and the ODSLA verification set one by one and then weighting and coupling them. The data processing load characteristic value is used to characterize the scale characteristics and dynamic activity of the data.

[0021] The data processing load characteristic value is input into the pre-stored mapping set of data processing load characteristic value-hash layer number in the local database, and the hash layer number is obtained by mapping and matching. Then, a multi-level hash table is constructed based on the hash layer number.

[0022] As a preferred technical solution, real-time monitoring of the load status of each hash bucket is implemented, specifically including:

[0023] The system scans each hash bucket in the multi-level hash table in real time, collects the load status parameters of each hash bucket, including the total number of objects in the bucket, the average update frequency of object status, and the average access latency. The data is stored in the local monitoring unit, and the pre-stored load status verification parameters in the local database are extracted, including the total number of objects in the verification bucket, the average update frequency of object status, and the average access latency. The load status parameters of each hash bucket are compared with the corresponding load status verification parameters one by one, and then weighted and coupled to obtain the load status characterization value of each hash bucket.

[0024] As a preferred technical solution, dynamically adjusting the hash bucket specifically includes:

[0025] The system extracts a preset load status upper limit threshold from the local database, compares the load status representation value of each hash bucket with the load status upper limit threshold, and records the hash bucket as a high-load hash bucket when the load status representation value of a certain hash bucket is greater than the load status upper limit threshold, thus triggering the splitting and adjustment of the high-load hash bucket.

[0026] Extract the preset load status limit threshold from the local database, compare the load status representation value of each hash bucket with the load status limit threshold, and when the load status representation value of a hash bucket is less than the load status limit threshold, the hash bucket is recorded as a low load hash bucket. Count the number of low load hash buckets, and when the number of low load hash buckets exceeds the preset low load hash bucket number threshold, trigger the low load hash bucket merging and adjustment.

[0027] As a preferred technical solution, triggering high-load hash bucket splitting and adjustment specifically includes:

[0028] The load status representation value of the high-load hash bucket is subtracted from the load status upper limit threshold to obtain the upper limit difference of the load status representation of the high-load hash bucket. This upper limit difference is then input into a pre-stored mapping set of upper limit difference of load status representation - number of hash bucket splits in the database for mapping and matching to obtain the number of hash bucket splits for the high-load hash bucket. Based on the number of hash bucket splits, the high-load hash bucket is split into multiple sub-hash buckets. The specific splitting methods include:

[0029] Based on the number of hash bucket splits, the range of hash function modulo is increased, and objects in high-load hash buckets are remapped to each sub-hash bucket.

[0030] As a preferred technical solution, triggering low-load hash bucket merging and adjustment specifically includes:

[0031] The maximum computing resource parameters of the merging unit are statistically analyzed, including memory resource parameters, CPU computing resource parameters, and cache resource parameters. Based on the maximum computing resource parameters of the merging unit, the maximum number of buckets in a single merge is determined. The maximum number of buckets in a single merge is used to limit the number of low-load hash buckets that can be merged in a single hash bucket merge.

[0032] If the number of low-load hash buckets is less than or equal to the maximum number of buckets in a single merge, then each low-load hash bucket is recorded as a single merge batch. Computational resources are pre-allocated to the merged hash buckets, and objects in each low-load hash bucket are migrated to the new hash bucket one by one. The index position of the objects in the hash table is updated. After the migration is completed, the computational resources occupied by each low-load hash bucket are released, and the metadata of the hash table is updated. The metadata includes the total number of buckets, the modulo range, and the load information.

[0033] If the number of low-load hash buckets is greater than the maximum number of buckets for a single merge, then the low-load hash buckets are sorted according to their hash values. In the sorted list of low-load hash buckets, low-load hash buckets that do not exceed the maximum number of buckets for a single merge are selected as a single merge batch and merged. The remaining unmerged low-load hash buckets are selected as the next batch and merged, until all low-load hash buckets are merged.

[0034] As a preferred technical solution, the system identifies the matched objects and removes the unmatched objects. Combined with a counting mechanism, dynamic adjustments are made when the object state is updated. Specific processing conditions are as follows:

[0035] Create countable Bloom filters for each pre-defined rule in the local database. The rule pointers define the triggering conditions and decision logic for changes in the object's state.

[0036] Obtain the historical trigger probability and the number of historical trigger objects for each rule, and configure the size of the bit array of the countable Bloom filter based on the historical trigger probability and the number of historical trigger objects for each rule.

[0037] When an object's state first meets the triggering condition of a rule, it is recorded as a hit object. The unique identifier of the hit object is inserted into the CBF and a counter is incremented to record its existence count. After the hit object's state is updated, if the rule is still met, the existence count of the hit object is incremented in the counter. If the rule is no longer met, the existence count of the hit object is decremented in the counter. When the counter is zero, the hit object is logically removed and recorded as a miss object.

[0038] Before each object state update, the CBF is queried. If an object matches, it enters the streaming event processing module for precise matching. If no object matches, the precise calculation is skipped.

[0039] As a preferred technical solution, the event stream is precisely matched, out-of-order events are handled through local cache management, and the state update events of objects that are precisely matched are output and triggered. The specific process includes:

[0040] Receive the event stream, which includes state update events for each object. Maintain a local cache for the event stream. The local cache is an independent data structure used to sort the state update events of each object in the event stream in a time sequence. The specific process includes extracting the timestamp of each state update event in the event stream and inserting it into the time sequence queue.

[0041] The system retrieves the object's unique identifier, spatial coordinates, and status identifier from each object's status update event. Based on logical operators, it performs precise matching between each object's status update event and the rules. If an object's status update event satisfies a rule, it is recorded as a successful match, immediately output, and the object's status update event is triggered. If an object's status update event does not satisfy a rule, it continues to match with the next rule until a successful match is achieved. If an object's status update event does not match any of the rules in the local database, it is recorded as a failed match, and the matching process ends.

[0042] Virtual training data management methods based on LVC simulation include:

[0043] After collecting raw state data from three types of data sources (L, V, and C), it is uniformly mapped into an internal object format. A multi-layer hash table is constructed based on the ODSLA indicator set. At the same time, the load status of each hash bucket is monitored in real time and the hash bucket is dynamically adjusted. The object state located by the index enters the Bloom filter module.

[0044] The object state is initially screened to identify the matching objects and remove the missing objects. Combined with the counting mechanism, the state is dynamically adjusted when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream.

[0045] Perform precise matching on the event stream, handle out-of-order events through local cache management, and output and trigger state update events for objects that are precisely matched.

[0046] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0047] (1) This invention provides a virtual training data management system based on LVC simulation, which realizes unified collection and mapping of raw state data from three types of data sources: L, V, and C, through a hierarchical hash index module. By standardizing the unique object identifier, spatial coordinate information, and state identifier into an internal unified object format, and performing field mapping, unit unification, and global coordinate system transformation, the system can eliminate data differences between different data sources and improve the accuracy and consistency of data fusion. In addition, the time base mapping and time difference compensation algorithm unifies the time information of each data source into a standard timestamp, ensuring the time synchronization of cross-source data, thereby laying the foundation for subsequent data processing and real-time analysis.

[0048] (2) This invention generates data processing load characteristic values ​​by comprehensively processing parameters such as the number of objects, average update frequency of objects, spatial distribution, object density and access pressure, characterizing the data scale and dynamic activity, determining the number of hash layers, realizing the efficient construction of multi-level hash tables, optimizing data index and storage structure, improving the speed and accuracy of object status positioning, and providing quantitative basis for real-time load monitoring and dynamic adjustment of hash buckets, thereby enhancing the performance and stability of the system in large-scale simulation data processing scenarios.

[0049] (3) This invention achieves dynamic load balancing by monitoring the hash bucket load status in real time and combining a high-load hash bucket splitting and a low-load hash bucket merging mechanism. High-load hash buckets are split into multiple sub-buckets according to the load upper limit threshold, expanding the hash function modulo range and remapping objects, thereby reducing the processing pressure of a single bucket; low-load hash buckets are merged in batches according to the system's computing resource carrying capacity, rationally allocating memory, CPU and cache resources, and optimizing system resource utilization. This dynamic adjustment mechanism can ensure that the hash table continues to operate efficiently under changes in data scale or sudden events, improving the overall throughput and response speed of the system.

[0050] (4) This invention uses a countable Bloom filter to perform initial screening of object states, reducing unnecessary exact matching calculations and lowering computational overhead. The system manages the count of hit objects and dynamically adjusts it when the object state is updated. Missing objects skip exact matching, further improving processing efficiency. By combining local cache management of out-of-order events, rule-based exact matching based on logical operators, and an instant output triggering mechanism, the system can ensure the accuracy and real-time performance of event processing, thereby supporting efficient management of high-frequency, multi-source, and large-scale simulation data and accurate execution of virtual training tasks, providing reliable data support and improving operational efficiency for LVC training.

[0051] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the system modules of the present invention.

[0053] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0054] Figure 3 This is a schematic diagram of the method logic flow of the present invention.

[0055] Figure 4 This is a schematic diagram of the module logic flow of the present invention.

[0056] Figure 5 This is a continuation of the schematic diagram of the module logic flow of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] In the description of this invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "around", etc., which indicate orientation or positional relationship, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention.

[0059] Please see Figure 1 As shown, this embodiment of the invention provides a virtual training data management system based on LVC simulation, specifically including:

[0060] like Figure 3 The diagram illustrates the logical flow of the method involved in this embodiment of the invention, describing the overall data processing logic of a virtual training data management system based on LVC simulation. It covers the entire process from raw data acquisition to event output, collecting raw state data from three types of data sources: L, V, and C, including unique object identifiers, spatial coordinates, and state information. The collected data undergoes field mapping, coordinate transformation, and time synchronization processing to unify it into an internal object format. The processed object data is input into a hierarchical hashing module. The system constructs a multi-layered hash table based on the ODSLA metric, while simultaneously monitoring the load status of each hash bucket in real time, including the number of objects, average update frequency, and access latency. When a hash bucket is overloaded, it triggers the splitting of high-load hash buckets; when a hash bucket is underloaded and its number exceeds a threshold, it triggers the merging of low-load hash buckets. The object states after hash indexing enter the Bloom filtering module, where a countable Bloom filter performs initial screening. The system determines whether an object is a match according to preset rules; only matched objects enter the precise event processing flow. The state update event of a matched object enters the streaming event processing module. This module maintains a local cache, sorts out-of-order events, and performs exact matching between events and rules. Successfully matched events are immediately output, while unmatched events are processed and then terminated. This reflects the system's complete closed loop from data collection, indexing, initial screening to exact matching, highlighting its capabilities in dynamic load balancing, filtering optimization, and real-time event processing.

[0061] like Figure 4 and Figure 5The diagram illustrates the module logic flow of an embodiment of the present invention, describing the module relationships of the virtual training data management system based on LVC simulation and demonstrating the flow and processing order of data in each functional unit. This facilitates understanding the data flow and dependencies between modules and provides a visual representation of the core logic of the entire virtual training data management system.

[0062] It should be explained that LVC (Live–Virtual–Constructive) simulation refers to the collaborative operation of a real system (Live), a virtual system (Virtual), and a model-based constructed system (Constructive) in the same scenario.

[0063] The hierarchical hash index module is used to collect raw state data from three types of data sources (L, V, and C) and then map it into an internal object format. It constructs a multi-level hash table based on the ODSLA metric set, while monitoring the load status of each hash bucket in real time and dynamically adjusting the hash bucket. The object state located by the index enters the Bloom filter module.

[0064] After collecting raw state data from three types of data sources (L, V, and C), it is uniformly mapped into an internal object format. The specific process is as follows:

[0065] The system collects raw state data from three types of data sources: L, V, and C. The raw state data includes object unique identifiers, spatial coordinate information, and state identifiers. The raw state data from the three types of data sources is mapped to a unified internal object format. In this embodiment of the invention, an adapter is deployed at the acquisition layer, and parsing plugins are installed for the three types of data sources (L, V, and C) to perform data parsing and mapping. The mapping process includes field name conversion (e.g., converting "Track unique identifier" from the L source to the internal "obj_unique identifier"). The system converts the data with different structures from the three types of data sources into a unified internal object format that can be directly processed by the system, making it conform to the internal adaptation logic.

[0066] During field mapping and unit unification, a coordinate transformation function is used to uniformly map them to an internally defined global coordinate system. It should be noted that different data sources use different coordinate systems; for example, the L source typically uses the GPS WGS-84 coordinate system, the V source typically uses a local simulated world coordinate system, and the C source typically uses a map grid coordinate system. Therefore, it is necessary to unify them to the global coordinate system. In this embodiment of the invention, a mapping transformation coordinate function is used. By unifying the coordinates, spatial calculations (such as distance, velocity, collision detection, and region determination) are ensured to be performed on a unified spatial basis, avoiding calculation errors caused by inconsistent coordinate systems.

[0067] The status identifiers of objects from different data sources are mapped to internally defined enumeration types through the built-in status mapping table of the local database. It should be noted that since the status identifiers of different data sources are encoded differently, they need to be mapped to a unified internally defined enumeration type through the status mapping table. This avoids writing different status judgment rules frequently in business logic and ensures that the rule engine can make judgments directly based on the unified status enumeration.

[0068] By employing a time base mapping and time difference compensation algorithm, time information from various data sources is uniformly converted into an internal standard timestamp. In this embodiment of the invention, the time base mapping and time difference compensation algorithm uses a multi-source clock synchronization algorithm, ensuring that cross-source events are comparable in time.

[0069] The process of constructing a multi-level hash table based on the ODSLA indicator set is as follows:

[0070] The ODSLA metrics set is a set of parameters used for hierarchical hashing, including the number of objects, average object update frequency, spatial distribution, object density, and access pressure.

[0071] It should be noted that the number of objects refers to the total number of objects currently active in the system and participating in index management. This is obtained by directly counting the number of unique object identifiers validly recorded in the index table through the fast query interface. The average object update frequency refers to the average number of times an object's state changes per unit time. This is obtained by the stream processing engine maintaining a cache of the average object update frequency and periodically outputting it. Spatial distribution refers to the distribution of objects in a spatial coordinate system, used to measure the degree of clustering or dispersion. It is obtained through fast region aggregation operations on a quadtree, specifically by dividing the global space into fixed-size grids, denoted as unit spaces, counting the number of objects in each grid, and calculating the distribution variance. Object density refers to the average number of objects per unit space, used to reflect local congestion. This is obtained by taking the global average value through the spatial distribution calculation process. Access pressure refers to the total number of query and update requests for each hash bucket per unit time. This is obtained by maintaining access counters for each hash bucket, obtaining the number of queries and updates, and then averaging them to obtain the access pressure.

[0072] Extract the ODSLA verification set from the local database, including the number of verification objects, the average update frequency of verification objects, the spatial distribution of verification objects, the density of verification objects, and the access pressure of verification.

[0073] The data processing load characteristic value is obtained by mapping and comparing the parameters of the ODSLA indicator set with those of the ODSLA validation set one by one, and then weighting and coupling them. The data processing load characteristic value is used to characterize the scale characteristics and dynamic activity of the data. The specific calculation process includes:

[0074]

[0075] in, For data processing load characteristic values, Let be the i-th parameter in the ODSLA metric set. In this embodiment of the invention, it represents the number of objects, average object update frequency, spatial distribution, object density, and access pressure. In practical applications, other parameters characterizing data processing load can be selected. For ODSLA verification set and The corresponding parameters in this embodiment of the invention are the number of verification objects, the average update frequency of verification objects, the spatial distribution of verification, the density of verification objects, and the access pressure of verification. Let i be the weighting factor of the i-th parameter in the database, satisfying In this embodiment of the invention, the weighting factors include the number of objects, the average update frequency of objects, the spatial distribution, the object density, and the access pressure. i is the number of the parameter in the ODSLA metric set, i=1,2,3,...,n, and n is the total number of parameters in the ODSLA metric set. In this embodiment of the invention, n is 5. x1 is the number of objects, x2 is the average update frequency of objects, x3 is the spatial distribution, x4 is the object density, and x5 is the access pressure.

[0076] It should be noted that, in the embodiments of the present invention, the selected object quantity weighting factor, object average update frequency weighting factor, spatial distribution weighting factor, object density weighting factor, and access pressure weighting factor are used to adjust the weights of the corresponding parameters in the calculation. They can be obtained by analyzing and processing actual parameter test data and historical system operation data based on machine learning algorithms.

[0077] It should also be noted that in this embodiment of the invention, the five parameters selected—number of objects, average object update frequency, spatial distribution, object density, and access pressure—are correlated. The number of objects is a fundamental indicator; an increase in the total number of objects directly increases the load on system index queries, state updates, and event processing, and may also affect the performance of spatial distribution and object density. The average object update frequency indicates how frequently each object changes its state per unit of time. Frequently updated objects increase access pressure and create hotspots in hash buckets or caches, thus jointly determining the probability of index conflicts along with the number and density of objects. Spatial distribution reflects the degree of aggregation or uniformity of objects in space. Highly aggregated areas may form hotspots, increasing access pressure on certain hash buckets or caches. Object density is a local quantitative indicator of spatial distribution; the more objects in a unit of space, the higher the density, which also increases local access pressure and index update pressure. Access pressure directly reflects the frequency with which objects are accessed or matched; it is usually highly correlated with the number of objects, update frequency, and dense areas. High-density, high-update-frequency objects often generate higher access pressure.

[0078] The data processing load characteristic value is input into a pre-stored mapping set of data processing load characteristic value - hash layer number in the local database. After mapping and matching, the hash layer number is obtained. A multi-level hash table is then constructed based on the hash layer number, specifically including:

[0079] The mapping set of data processing load characteristics to the number of hash layers is derived from historical ODSLA metrics collected from the system. These metrics include the number of objects, average object update frequency, spatial distribution, object density, and access pressure. For each set of ODSLA metrics, the number of hash layers is recorded when constructing a multi-level hash table under that load condition. The collected historical ODSLA metrics are statistically analyzed to establish a relationship between data processing load characteristics and the optimal number of hash layers. Regression analysis is then used to map the multi-dimensional load characteristics to the recommended number of hash layers. The resulting mapping is stored in a local database.

[0080] Based on the hash layer number obtained from mapping and matching, the hierarchical structure of the hash table is determined. Each layer corresponds to an object index space of different granularities. In this embodiment of the invention, the top-level hash table is responsible for coarse-grained indexing, and the lower-level hash tables are responsible for fine-grained indexing. Hash buckets are created for each layer. The number and distribution of buckets are dynamically allocated according to load characteristic parameters such as the number of objects, update frequency, and spatial distribution of the layer to ensure index balance and avoid hotspots. When an object is inserted, the corresponding coarse-grained bucket is located through the top-level hash function, and then the fine-grained bucket is further located through the lower-level hash function. The object state information is stored in the cache of the corresponding bucket. When an object is updated or deleted, it is located according to the same layer and hash function to achieve efficient add, delete, and modify operations. The entire multi-level hash table structure also maintains metadata, including the number of layers, the number of buckets in each layer, the modulo range, and load information, to support subsequent hash bucket splitting, merging, and load balancing adjustments, thereby realizing multi-level index management that is dynamically optimized for different load characteristics.

[0081] Real-time monitoring of the load status of each hash bucket, specifically including:

[0082] The system scans each hash bucket in the multi-level hash table in real time, collecting load status parameters for each hash bucket, including the total number of objects in the bucket, the average update frequency of object status, and the average access latency. These parameters are collected in real time through the statistical tables inside the hash bucket.

[0083] The data is stored in the local monitoring unit, and the pre-stored load status verification parameters in the local database are extracted, including the total number of objects in the verification bucket, the average update frequency of the verification object status, and the average access latency. The load status parameters of each hash bucket are compared one by one with the corresponding load status verification parameters, and then weighted and coupled to obtain the load status characterization value of each hash bucket, specifically including:

[0084]

[0085] in, Let be the load status representation value of the y-th hash bucket. Let be the u-th load status parameter of the y-th hash bucket. In this embodiment of the invention, it includes the total number of objects in the bucket, the average update frequency of object status, and the average access latency. In practical applications, other parameters characterizing the load status of the hash bucket can be selected. The load status verification parameter corresponding to the u-th load status parameter of the y-th hash bucket, in this embodiment of the invention, includes the total number of objects in the verification bucket, the average update frequency of the verification object status, and the average access latency. The weighting factor for the u-th load state parameter of the y-th hash bucket, in this embodiment of the invention, includes a weighting factor for the total number of objects in the bucket, a weighting factor for the average update frequency of object state, and a weighting factor for the average access latency. y is the hash bucket number, y=1,2,3,...,p, p is the total number of hash buckets, u is the load state parameter number, u=1,2,3,...,m, m is the total number of load state parameters. In this embodiment of the invention, m is 3, m1 is the total number of objects in the bucket, m2 is the average update frequency of object state, and m3 is the average access latency.

[0086] It should be noted that the weighting factors for the total number of objects in the bucket, the average update frequency of object status, and the average access latency are used to adjust the weights of the corresponding parameters in the calculation. These factors can be obtained by analyzing and processing actual parameter test data and historical system operation data based on machine learning algorithms.

[0087] Dynamically adjusting hash buckets includes:

[0088] The system extracts a preset load status upper limit threshold from the local database. It then compares the load status representation value of each hash bucket with this threshold. If the load status representation value of a hash bucket exceeds the upper limit threshold, that hash bucket is designated as a high-load hash bucket, triggering a high-load hash bucket split and adjustment. Specifically, this includes:

[0089] The load status representation value of the high-load hash bucket is subtracted from the load status upper limit threshold to obtain the upper limit difference of the load status representation of the high-load hash bucket. This upper limit difference is then input into a pre-stored mapping set of upper limit difference of load status representation - number of hash bucket splits in the database for mapping and matching to obtain the number of hash bucket splits for the high-load hash bucket. Based on the number of hash bucket splits, the high-load hash bucket is split into multiple sub-hash buckets. The specific splitting methods include:

[0090] Based on the number of hash bucket splits, the range of hash function modulo is increased, and objects in high-load hash buckets are remapped to each sub-hash bucket.

[0091] In multiple sets of historical operational data, the difference between the load status representation value and the load upper limit threshold for each hash bucket is recorded, along with the optimal number of splits required when splitting the hash bucket under this difference. A discrete table is used to establish the relationship between the upper limit difference of the load status representation and the corresponding number of hash bucket splits, mapping different upper limit differences to recommended split numbers. The obtained mapping relationship is stored in a local database as a reference for the system to dynamically adjust high-load hash buckets. In actual operation, the system calculates the difference between the real-time load status representation value of each hash bucket and the upper limit threshold, and obtains the corresponding number of splits by calling the mapping set. This ensures that the splitting operation effectively alleviates high load while avoiding unnecessary over-splitting, thereby improving system performance and resource utilization.

[0092] The system extracts a preset load status threshold from the local database, compares the load status representation value of each hash bucket with the load status threshold, and records the hash bucket as a low-load hash bucket when its load status representation value is less than the load status threshold. The system then counts the number of low-load hash buckets. When the number of low-load hash buckets exceeds a preset low-load hash bucket count threshold, a low-load hash bucket merging and adjustment is triggered, specifically including:

[0093] The maximum computing resource parameters for the merging unit are calculated. The merging unit refers to the built-in data processing unit in this embodiment of the invention used to merge hash buckets. The maximum computing resource parameters include memory resource parameters, CPU computing resource parameters, and cache resource parameters. Based on the maximum computing resource parameters of the merging unit, the maximum number of buckets to merge in a single operation is determined. The specific process includes:

[0094] The maximum capacity computing resource parameters of the merging unit are input into the local database's built-in resource parameter-single merging bucket number mapping set, including the memory resource parameter-single merging bucket number mapping set, the CPU computing resource parameter-single merging bucket number mapping set, and the cache resource parameter-single merging bucket number mapping set. A one-to-one mapping match is performed to obtain the single merging bucket number corresponding to each maximum capacity computing resource parameter, and the minimum value is taken as the maximum number of buckets for a single merge.

[0095] The maximum number of buckets that a single hash bucket merge operation can safely handle under different memory, CPU, and cache resource configurations is extracted from historical records. Through statistical analysis of a large amount of historical data, mapping relationships are established between memory resource parameters and the number of buckets supported for a single merge, CPU resource parameters and the number of buckets supported for a single merge, and cache resource parameters and the number of buckets supported for a single merge. These mapping relationships are stored in a local database in the form of functions. In actual operation, the system obtains the corresponding number of buckets supported for a single merge by calling the mapping relationships based on the current resource status of each merge unit, and takes the minimum value among the three resource types as the maximum number of buckets for a single merge, thus ensuring that the merge operation is both efficient and safe, avoiding exceeding the computing resource limits.

[0096] It should be noted that memory resource parameters can be obtained by monitoring the available memory capacity of the merging unit under normal operating conditions, as well as the average amount of memory occupied by each hash bucket object, and then normalizing them into a percentage of memory resource utilization.

[0097] CPU computing resource parameters can be quantified by recording the CPU utilization and average execution time of the merge unit during hash calculations, object migrations, index updates, and other operations performed during the hash bucket merging process. This yields the maximum CPU requirement for a single merge operation and normalizes it to a percentage of CPU resources.

[0098] The cache resource parameter is calculated by measuring the cache used by the merge unit during the merge process, and combining the object access frequency and the cache hit rate of the migration operation to evaluate the maximum cache space occupied during the merge, and normalize it to a cache resource percentage.

[0099] The maximum number of buckets per merge is used to limit the number of low-load hash buckets that can be merged in a single hash bucket merge.

[0100] If the number of low-load hash buckets is less than or equal to the maximum number of buckets in a single merge, then each low-load hash bucket is recorded as a single merge batch. Computational resources are pre-allocated to the merged hash buckets, and objects in each low-load hash bucket are migrated to the new hash bucket one by one. The index position of the objects in the hash table is updated. After the migration is completed, the computational resources occupied by each low-load hash bucket are released, and the metadata of the hash table is updated. The metadata includes the total number of buckets, the modulo range, and the load information.

[0101] If the number of low-load hash buckets is greater than the maximum number of buckets for a single merge, then the low-load hash buckets are sorted according to their hash values. In the sorted list of low-load hash buckets, low-load hash buckets that do not exceed the maximum number of buckets for a single merge are selected as a single merge batch and merged. The remaining unmerged low-load hash buckets are selected as the next batch and merged, until all low-load hash buckets are merged.

[0102] The Bloom filter module is used to initially screen the object state, identify the matching objects, and remove the missing objects. Combined with the countable mechanism, it makes dynamic adjustments when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream.

[0103] Create countable Bloom filters for each pre-defined rule in the local database. The rule pointers set the trigger conditions and judgment logic for changes in the object's state, and are used to initially screen and determine whether the object needs further processing.

[0104] Obtain the historical trigger probability and the number of historical trigger objects for each rule, and configure the size of the bit array of the countable Bloom filter based on the historical trigger probability and the number of historical trigger objects for each rule. Specifically, this includes:

[0105] Historical trigger probability refers to the proportion of events that triggered the rule within a past time window out of all events; historical trigger object count refers to the number of independent objects that triggered the rule within the same time window. This data can be obtained from log systems, event databases, or cached statistics tables, and then aggregated within a time window.

[0106] The historical trigger probability and the number of historical triggering objects are combined to estimate the number of hash bits that each rule may require. The size of the CBF bit array directly affects the false positive rate and memory usage: rules with higher trigger probabilities and more triggering objects require more bits to reduce false positives; conversely, rules with low trigger probabilities or fewer objects can use fewer bits to save resources. The calculation method is as follows: ;in, t is the size of the bit array, t is the number of historically triggered objects, and w is the historical trigger probability.

[0107] When an object's state first meets the triggering condition of a rule, it is recorded as a hit object. The unique identifier of the hit object is inserted into the CBF (Countable Bloom Filter) and a counter is incremented to record its existence count. After the hit object's state is updated, if the rule is still met, the existence count of the hit object is incremented in the counter. If the rule is no longer met, the existence count of the hit object is decremented in the counter. When the counter is zero, the hit object is logically removed and recorded as a miss object.

[0108] Before each object state update, the CBF (Content Flow Framework) is queried. Matching objects proceed to the streaming event processing module for exact matching, while missing objects skip the exact calculation, significantly reducing unnecessary computational overhead. Skipping exact matching for missing objects avoids redundant calculations on a large number of objects irrelevant to the rules or not currently triggering them, thereby reducing CPU and memory load and improving overall system efficiency.

[0109] By dynamically maintaining the matched objects using a countable Bloom filter, the state changes of objects can be accurately tracked. A counter records the number of times each matched object exists, ensuring that even with frequent state changes, it can still correctly determine whether an object should participate in rule matching, avoiding missed or duplicate event triggers and improving the accuracy of event processing. Furthermore, this approach supports high concurrency and real-time processing, enabling the system to maintain millisecond-level response times when faced with state updates of thousands of objects. This ensures that the streaming event processing module can quickly obtain the objects that require precise matching, guaranteeing the timely triggering of critical events.

[0110] The streaming event processing module is used for precise matching of event streams, manages out-of-order events through local caching, and outputs and triggers state update events for objects that are precisely matched.

[0111] Receive the event stream, which includes state update events for each object. Maintain a local cache for the event stream. The local cache is an independent data structure used to sort the state update events of each object in the event stream in a time sequence. The specific process includes extracting the timestamp of each state update event in the event stream and inserting it into the time sequence queue.

[0112] By maintaining a local cache for the event stream and sorting it by timestamp, the timing of object state update events can be guaranteed. In LVC simulation, different data sources may have data delays or out-of-order issues. The local cache can temporarily store and sort out-of-order events, ensuring that events are processed in the correct chronological order before rule matching, thereby avoiding matching errors or logical inconsistencies caused by disordered event order.

[0113] The system retrieves the object's unique identifier, spatial coordinates, and status identifier from each object's status update event. Based on logical operators, it performs precise matching between each object's status update event and the rules. If an object's status update event satisfies a rule, it is recorded as a successful match, immediately output, and the object's status update event is triggered. If an object's status update event does not satisfy a rule, it continues to match with the next rule until a successful match is achieved. If an object's status update event does not match any of the rules in the local database, it is recorded as a failed match, and the matching process ends.

[0114] Using logical operators for precise matching of events and rules ensures that each rule is triggered only when the state of an object that meets the conditions is updated, avoiding false triggers. It also supports multi-rule matching logic, improving the accuracy of rule matching. Successfully matched events are immediately output and trigger responses, facilitating rapid response and processing of critical events.

[0115] In this embodiment, the present invention provides a virtual training data management method based on LVC simulation, comprising:

[0116] After collecting raw state data from three types of data sources (L, V, and C), it is uniformly mapped into an internal object format. A multi-layer hash table is constructed based on the ODSLA indicator set. At the same time, the load status of each hash bucket is monitored in real time and the hash bucket is dynamically adjusted. The object state located by the index enters the Bloom filter module.

[0117] The object state is initially screened to identify the matching objects and remove the missing objects. Combined with the counting mechanism, the state is dynamically adjusted when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream.

[0118] Perform precise matching on the event stream, handle out-of-order events through local cache management, and output and trigger state update events for objects that are precisely matched.

[0119] It should be noted that, in another embodiment of the invention, the above system is used as a subsystem of the LVC data integrated management system. The LVC data integrated management system collects LVC simulation training data in real time through multi-source heterogeneous interfaces, and combines artificial intelligence technology to perform intelligent annotation, feature extraction and knowledge mining on the raw data to achieve deep structured processing of the data.

[0120] Based on the underlying data processing flow of this invention, the LVC data management system in another embodiment supports high-concurrency access, ensuring efficient management and accuracy of data during LVC training. Simultaneously, a reinforcement learning-driven dynamic optimization engine is introduced, forming a closed-loop optimization mechanism that uses data feedback to drive strategy adjustments, thereby improving the effectiveness and adaptability of virtual training.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0122] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementation methods. Clearly, many modifications and variations can be made based on the content of this specification. The selection and detailed description of these embodiments in this specification are intended to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. Any modifications or variations that do not deviate from the structure of the invention or exceed the scope defined by the invention should fall within the protection scope of the invention.

Claims

1. A virtual training data management system based on LVC simulation, characterized in that, include: The hierarchical hash index module is used to collect raw state data from three types of data sources (L, V, and C) and map it into an internal object format. It constructs a multi-level hash table based on the ODSLA indicator set, while monitoring the load status of each hash bucket in real time and dynamically adjusting the hash bucket. The object state located by the index enters the Bloom filter module. The Bloom filter module is used to perform initial screening of object states, identify matching objects, and remove non-matching objects. Combined with a countable mechanism, it dynamically adjusts the state when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream. The streaming event processing module is used for precise matching of event streams, manages out-of-order events through local caching, and outputs and triggers state update events for objects that are precisely matched. The specific process of constructing a multi-layer hash table based on the ODSLA indicator set is as follows: The ODSLA metric set is a set of parameters used for hierarchical hashing, including the number of objects, average object update frequency, spatial distribution, object density, and access pressure. Extract the ODSLA verification set from the local database, including the number of verification objects, the average update frequency of verification objects, the spatial distribution of verification objects, the density of verification objects, and the access pressure of verification. The data processing load characteristic value is obtained by comparing the parameters in the ODSLA indicator set and the ODSLA verification set one by one and then weighted coupling. The data processing load characteristic value is used to characterize the scale characteristics and dynamic activity of the data. The data processing load characteristic value is input into the pre-stored mapping set of data processing load characteristic value-hash layer number in the local database, and the hash layer number is obtained by mapping and matching. Then, a multi-level hash table is constructed based on the hash layer number.

2. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The process of collecting raw state data from three types of data sources (L, V, and C) and then mapping it into an internal object format is as follows: The system collects raw state data from three types of data sources: L, V, and C. The raw state data includes a unique object identifier, spatial coordinate information, and state identifier. The raw state data from the three types of data sources is mapped to a unified internal object format. During the process of field mapping and unit unification, coordinate transformation functions are used to uniformly map them to the internally defined global coordinate system. The state identifiers of objects from different data sources are mapped to internally defined enumeration types through the state mapping table built into the local database; By using time base mapping and time difference compensation algorithms, time information from various data sources is uniformly converted into internal standard timestamps.

3. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The real-time monitoring of the load status of each hash bucket specifically includes: The system scans each hash bucket in the multi-level hash table in real time, collects the load status parameters of each hash bucket, including the total number of objects in the bucket, the average update frequency of object status, and the average access latency. The data is stored in the local monitoring unit, and the pre-stored load status verification parameters in the local database are extracted, including the total number of objects in the verification bucket, the average update frequency of object status, and the average access latency. The load status parameters of each hash bucket are compared with the corresponding load status verification parameters one by one, and then weighted and coupled to obtain the load status characterization value of each hash bucket.

4. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The dynamic adjustment of the hash bucket specifically includes: Extract the preset load status upper limit threshold from the local database, compare the load status representation value of each hash bucket with the load status upper limit threshold, and when the load status representation value of a hash bucket is greater than the load status upper limit threshold, the hash bucket is recorded as a high load hash bucket, triggering the high load hash bucket splitting and adjustment. Extract the preset load status limit threshold from the local database, compare the load status representation value of each hash bucket with the load status limit threshold, and when the load status representation value of a hash bucket is less than the load status limit threshold, the hash bucket is recorded as a low load hash bucket. Count the number of low load hash buckets, and when the number of low load hash buckets exceeds the preset low load hash bucket number threshold, trigger the low load hash bucket merging and adjustment.

5. The virtual training data management system based on LVC simulation according to claim 4, characterized in that: The triggering of high-load hash bucket splitting and adjustment specifically includes: The load status representation value of the high-load hash bucket is subtracted from the load status upper limit threshold to obtain the upper limit difference of the load status representation of the high-load hash bucket. This upper limit difference is then input into a pre-stored mapping set of upper limit difference of load status representation - number of hash bucket splits in the database for mapping and matching to obtain the number of hash bucket splits for the high-load hash bucket. Based on the number of hash bucket splits, the high-load hash bucket is split into multiple sub-hash buckets. The specific splitting methods include: Based on the number of hash bucket splits, the range of hash function modulo is increased, and objects in high-load hash buckets are remapped to each sub-hash bucket.

6. The virtual training data management system based on LVC simulation according to claim 4, characterized in that: The triggering of low-load hash bucket merging and adjustment specifically includes: The maximum computing resource parameters of the merging unit are statistically analyzed, including memory resource parameters, CPU computing resource parameters, and cache resource parameters. Based on the maximum computing resource parameters of the merging unit, the maximum number of buckets in a single merge is determined. The maximum number of buckets in a single merge is used to limit the number of low-load hash buckets that can be merged in a single hash bucket merge. If the number of low-load hash buckets is less than or equal to the maximum number of buckets in a single merge, then each low-load hash bucket is recorded as a single merge batch. Computational resources are pre-allocated to the merged hash buckets, and objects in each low-load hash bucket are migrated to the new hash bucket one by one. The index position of the objects in the hash table is updated. After the migration is completed, the computational resources occupied by each low-load hash bucket are released, and the metadata of the hash table is updated. The metadata includes the total number of buckets, the modulo range, and the load information. If the number of low-load hash buckets is greater than the maximum number of buckets for a single merge, then the low-load hash buckets are sorted according to their hash values. In the sorted list of low-load hash buckets, low-load hash buckets that do not exceed the maximum number of buckets for a single merge are selected as a single merge batch and merged. The remaining unmerged low-load hash buckets are selected as the next batch and merged, until all low-load hash buckets are merged.

7. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The process of identifying hit objects and removing miss objects, combined with a counting mechanism, involves dynamic adjustments during object state updates. Specific processing conditions are as follows: Create countable Bloom filters for each rule pre-defined in the local database, where each rule refers to the triggering conditions and judgment logic set for changes in the state of an object; Obtain the historical trigger probability and the number of historical trigger objects for each rule, and configure the size of the bit array of the countable Bloom filter based on the historical trigger probability and the number of historical trigger objects for each rule; When an object's state first meets the triggering condition of a rule, it is recorded as a hit object. The unique identifier of the hit object is inserted into a countable Bloom filter and a counter is incremented to record its existence count. After the hit object's state is updated, if the rule is still met, the existence count of the hit object is incremented in the counter. If the rule is no longer met, the existence count of the hit object is decremented in the counter. When the counter is zero, the hit object is logically removed and recorded as a miss object. Before each object state update, the countable Bloom filter is queried. If an object matches, it enters the streaming event processing module for exact matching. If no object matches, the exact calculation is skipped.

8. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The process of performing precise matching of the event stream, handling out-of-order events through local cache management, and outputting and triggering state update events for objects that have successfully matched precisely includes: Receive an event stream, which includes state update events for each object. Maintain a local cache for the event stream. The local cache is an independent data structure used to sort the state update events of each object in the event stream in a time sequence. The specific process includes extracting the timestamps of the state update events of each object in the event stream and inserting them into a time sequence queue. The system retrieves the object's unique identifier, spatial coordinates, and status identifier from each object's status update event. Based on logical operators, it performs precise matching between each object's status update event and the rules. If an object's status update event satisfies a rule, it is recorded as a successful match, immediately output, and the object's status update event is triggered. If an object's status update event does not satisfy a rule, it continues to match with the next rule until a successful match is achieved. If an object's status update event does not match any of the rules in the local database, it is recorded as a failed match, and the matching process ends.

9. A method applied to the virtual training data management system based on LVC simulation as described in any one of claims 1-8, characterized in that: After collecting raw state data from three types of data sources (L, V, and C), it is uniformly mapped into an internal object format. A multi-layer hash table is constructed based on the ODSLA indicator set. At the same time, the load status of each hash bucket is monitored in real time and the hash bucket is dynamically adjusted. The object state located by the index enters the Bloom filter module. The object state is initially screened to identify the hit objects and remove the missing objects. Combined with the counting mechanism, the state is dynamically adjusted when the object state is updated. The filtered object state update events are entered into the streaming event processing module as an event stream. Perform precise matching on the event stream, handle out-of-order events through local cache management, and output and trigger state update events for objects that are precisely matched.

Images