Virtual training data management system and method based on LVC simulation

Abstract

The invention discloses a virtual training data management system and method based on LVC simulation, and relates to the field of LVC simulation, the virtual training data management system comprises a hierarchical Hash index module, a Bloom filtering module and a streaming event processing module, the hierarchical Hash index module carries out unified collection and standardized mapping on original state data of three types of data sources, and the Bloom filtering module carries out standardized mapping on the original state data of the three types of data sources; and the cross-source data consistency and the time sequence accuracy are ensured. A multi-layer hash table is constructed based on an ODSLA index set, dynamic determination of the number of hash hierarchies is achieved, and object indexes and storage efficiency are optimized. The load of each hash bucket is monitored in real time, dynamic load balancing is realized by combining a high load splitting mechanism and a low load merging mechanism, and the resource utilization rate and the processing performance are improved. Object states are primarily screened through a countable mechanism, unnecessary accurate matching calculation is reduced, and efficient and accurate event stream processing is realized by combining local cache processing out-of-order events and rule accurate matching, so that reliable data support and quick response capability are provided for virtual training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of LVC simulation, in particular to a virtual training data management system and method based on LVC simulation. BACKGROUND

[0002] LVC (Live–Virtual–Constructive) simulation is a simulation technology that combines real systems (Live), computer-driven virtual systems with or without human-in-the-loop (Virtual), and model-based constructive systems (Constructive) in the same scene to support large-scale, heterogeneous device-involved integrated training and evaluation. With the rapid growth of sensors, platforms, and business systems, the scale of objects, the frequency of events, and the dimensionality of data in training scenarios are increasing exponentially, and the data management level is shifting from "offline aggregation" to "real-time collection" in a streaming processing mode, which puts higher requirements on throughput, latency, and consistency.

[0003] Existing LVC data management generally relies on standardized interfaces and simulation middleware (such as common entity state, event notification, attribute update, etc. message paradigm) to achieve multi-source access, but still faces significant heterogeneity at the data layer. In high-concurrency scenarios, standardization and verification itself can become a processing bottleneck, and under conditions of network jitter, out-of-order, and packet loss, it can amplify timing inconsistency issues, affecting the accuracy and repeatability of event triggering. At the same time, when the number of objects and the average update frequency of objects both increase, the index structure of static parameters is prone to hot buckets and uneven distribution.

[0004] Hash index is a very mature technology in database and cache systems, used for fast positioning of objects or records, but traditional hash index can cause some hash buckets to be overheated (hot spots) when the object distribution is highly uneven or the spatial location is highly dynamic, increasing query and update delays. High-frequency state updates can cause frequent index reconstruction or conflicts, increasing additional computation and delay.

[0005] Bloom filter is used to quickly determine whether an element exists and is also widely used in cache, deduplication, database acceleration. Standard Bloom filter has a false positive rate, and as the number of objects increases, the false positive probability increases, which can cause unnecessary precise matching calculations and increase latency. When updated frequently, Bloom filter cannot delete triggered objects, which can easily trigger repeatedly or require additional structure management.

[0006] The prior art such as the invention patent with the publication number CN118378802A is an intelligent method for LVC training, comprising the following steps: step 1, establishing an LVC intelligent training architecture; step 2, establishing an LVC training intelligent method. Step 2 includes: step 2-1, establishing an intelligent training planning method; step 2-2, establishing an intelligent director conditioning method; and step 2-3, an intelligent adjudication evaluation method. The present application proposes an intelligent method for LVC training, constructs an LVC intelligent training architecture, proposes intelligent training planning, intelligent director conditioning, intelligent adjudication evaluation and other methods, provides theoretical support for training effectiveness improvement, and leads the planning and development of future training system construction.

[0007] Based on the above scheme, it can be seen that the prior art in the field of LVC simulation mainly focuses on the intelligent architecture and methodology of LVC training, but lacks real-time processing of underlying data flow in data management. In actual application, since LVC has multiple data sources, LVC simulation has the problem of high concurrency and delay of multi-source heterogeneous data. The prior art lacks optimization mechanism at this level and is difficult to support large-scale real-time training environment. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a virtual training data management system and method based on LVC simulation. To achieve the above purpose, the present application is implemented by the following technical scheme: a virtual training data management system based on LVC simulation, comprising: A hierarchical hash index module is used to uniformly map the original state data from the L, V and C three types of data sources to an internal object format after collection, and a multi-layer hash table is constructed based on the ODSLA (hierarchical hash parameter) index set. Meanwhile, the load state of each hash bucket is monitored in real time and the hash bucket is dynamically adjusted. The object state positioned by the index enters the Bloom filter module.

[0009] A Bloom filter module is used to preliminarily screen the object state, identify the hit object, and eliminate the miss object. In combination with the countable mechanism, the object state is dynamically adjusted during the object state update, and the screened object state update event enters the stream event processing module as an event stream.

[0010] A stream event processing module is used to accurately match the event stream, process out-of-order events through local cache management, and output and trigger the object state update event of accurate matching success.

[0011] As a preferred technical scheme, the original state data from the L, V and C three types of data sources is uniformly mapped to an internal object format after collection, and the specific process is as follows: Real-time collection of original state data from L, V, C three types of data sources, original state data including object unique identifier, spatial coordinate information and state identifier, mapping original state data of three types of data sources to unified internal object format.

[0012] In the field mapping and unit unification process, it is unified to the internally defined global coordinate system through the coordinate conversion function.

[0013] The state identifier of the object of different data sources is mapped to the internally defined enumeration type through the state mapping table built in the local database.

[0014] Through the time reference mapping and time difference compensation algorithm, the time information of each data source is unified to the internal standard timestamp.

[0015] As a preferred technical solution, a multi-layer hash table is constructed based on the ODSLA index set, and the specific process is as follows: The ODSLA index set is a parameter set for layered hash, including object quantity, object average update frequency, spatial distribution, object density and access pressure.

[0016] The ODSLA check set is extracted from the local database, including check object quantity, check object average update frequency, check spatial distribution, check object density and check access pressure.

[0017] After corresponding and one-to-one comparison of the parameters in the ODSLA index set and the ODSLA check set, the data processing load characteristic value is obtained by weighted coupling, and the data processing load characteristic value is used to represent the scale characteristics and dynamic activity of the data.

[0018] The data processing load characteristic value is input into the mapping set of data processing load characteristic value-hash layer number pre-stored in the local database to obtain the hash layer number, and a multi-layer hash table is constructed based on the hash layer number.

[0019] As a preferred technical solution, the load state of each hash bucket is monitored in real time, specifically including: Real-time scanning of each hash bucket in the multi-layer hash table, collecting load state parameters of each hash bucket, including total number of objects in the bucket, average update frequency of object state and average access delay, storing the data into the local monitoring unit, extracting the pre-stored load state check parameters in the local database, including check total number of objects in the bucket, check average update frequency of object state and check average access delay, and one-to-one comparison and weighted coupling of the load state parameters of each hash bucket and the corresponding load state check parameters to obtain the load state representation value of each hash bucket.

[0020] As a preferred technical solution, the hash bucket is dynamically adjusted, specifically including: Extracting a preset upper limit threshold of load state from the local database, comparing the load state representation value of each hash bucket with the upper limit threshold of load state, when the load state representation value of a certain hash bucket is greater than the upper limit threshold of load state, the hash bucket is recorded as a high-load hash bucket, triggering high-load hash bucket splitting adjustment.

[0021] Extracting a preset lower limit threshold of load state from the local database, comparing the load state representation value of each hash bucket with the lower limit threshold of load state, when the load state representation value of a certain hash bucket is less than the lower limit threshold of load state, the hash bucket is recorded as a low-load hash bucket, and the number of low-load hash buckets is counted, when the number of low-load hash buckets exceeds a preset threshold of low-load hash bucket number, triggering low-load hash bucket merging adjustment.

[0022] As a preferred technical solution, triggering high-load hash bucket splitting adjustment, specifically including: The difference between the load state representation value of the high-load hash bucket and the upper limit threshold of load state is obtained to obtain the load state representation upper limit difference value of the high-load hash bucket, the load state representation upper limit difference value of the high-load hash bucket is input into the pre-stored load state representation upper limit difference value-hash bucket splitting number mapping set in the database for mapping matching to obtain the hash bucket splitting number of the high-load hash bucket, and the high-load hash bucket is split into multiple sub-hash buckets based on the hash bucket splitting number, and the specific splitting method includes: Based on the hash bucket splitting number, the modulo range of the hash function is increased, and the objects in the high-load hash bucket are remapped to each sub-hash bucket.

[0023] As a preferred technical solution, triggering low-load hash bucket merging adjustment, specifically including: The maximum carrying computing resource parameters of the merging unit are counted, including memory resource parameters, CPU computing resource parameters and cache resource parameters, based on the maximum carrying computing resource parameters of the merging unit, the maximum bucket number of single merging is determined, and the maximum bucket number of single merging is used to limit the number of low-load hash buckets allowed to be merged in single hash bucket merging.

[0024] If the number of low-load hash buckets is less than or equal to the maximum bucket number of single merging, each low-load hash bucket is recorded as a single merging batch, the computing resources of the merged hash bucket are pre-allocated, the objects in each low-load hash bucket are migrated to a new hash bucket one by one, and the index position of the object in the hash table is updated, after the migration is completed, the computing resources occupied by each low-load hash bucket are released, and the metadata of the hash table is updated, the metadata includes the total number of buckets, the modulo range and the load information.

[0025] If the number of low-load hash buckets is greater than the maximum number of buckets in a single merge, the low-load hash buckets are sorted in order of hash value, and from the sorted list of low-load hash buckets, low-load hash buckets that do not exceed the maximum number of buckets in a single merge are selected as a single merge batch, and hash bucket merging is performed, and the remaining low-load hash buckets that have not been merged are taken as the next batch, and hash bucket merging is performed until all low-load hash buckets are merged.

[0026] As a preferred technical solution, the hit objects are identified, and the non-hit objects are removed, and a countable mechanism is combined to dynamically adjust the object state update, and the specific processing conditions are: A countable Bloom filter is created for each rule in the local database, and the rule refers to the trigger condition and judgment logic set for the object state change.

[0027] The historical trigger probability and the historical number of triggered objects of each rule are obtained, and the size of the bit array of the countable Bloom filter is configured based on the historical trigger probability and the historical number of triggered objects of each rule.

[0028] When the object state first meets the trigger condition of a rule, it is recorded as a hit object, the unique identifier of the hit object is inserted into the CBF, and the counter is increased to record the number of times it exists, after the hit object state is updated, if it still meets the rule, the number of times the hit object exists in the counter is increased, if it no longer meets the rule, the number of times the hit object exists in the counter is reduced, when the counter is zero, the hit object is logically removed, and recorded as a non-hit object.

[0029] Before each object state update, the CBF is queried, the hit object enters the stream event processing module for accurate matching, and the non-hit object directly skips the accurate calculation.

[0030] As a preferred technical solution, the event stream is accurately matched, the out-of-order events are processed through local cache management, and the object state update event of accurate matching success is output and triggered, and the specific process includes: The event stream is received, the event stream includes object state update events, a local cache is maintained for the event stream, the local cache is an independent data structure, and is used for time sequence sorting of the object state update events in the event stream, and the specific process includes extracting the time stamp of each object state update event in the event stream and inserting it into the time sequence queue.

[0031] The object unique identifier, the space coordinate information and the state identifier in each object state update event are acquired, each object state update event is accurately matched with a rule based on a logical operator, if an object state update event meets a rule, the matching is recorded as successful, the object state update event is immediately output and triggered, if an object state update event does not meet a rule, the matching with the next rule is continued until the matching is successful, if an object state update event does not match each rule in the local database, the matching is recorded as failed, and the matching process is ended.

[0032] The virtual training data management method based on LVC simulation comprises the following steps: After collecting original state data from three types of data sources L, V and C, the original state data is uniformly mapped into an internal object format, a multi-layer hash table is constructed based on an ODSLA index set, the load state of each hash bucket is monitored in real time and the hash bucket is dynamically adjusted, and the object state positioned by the index enters a Bloom filter module.

[0033] The object state is preliminarily screened, the hit object is identified, and the non-hit object is removed, the countable mechanism is combined, the object state is dynamically adjusted during the object state update, and the screened object state update event enters a stream event processing module as an event stream.

[0034] The event stream is accurately matched, the out-of-order event is processed through local cache management, and the object state update event accurately matched successfully is output and triggered.

[0035] Compared with the prior art, the embodiments of the present application have at least the following beneficial effects: (1) The virtual training data management system based on LVC simulation is provided, and the original state data from three types of data sources L, V and C is uniformly collected and mapped through a hierarchical hash index module. By standardizing the object unique identifier, the space coordinate information and the state identifier into an internal uniform object format, and performing field mapping, unit unification and global coordinate system conversion, the system can eliminate the data differences between different data sources, improve the accuracy and consistency of data fusion. In addition, the time reference mapping and time difference compensation algorithm unifies the time information of each data source into a standard timestamp, ensures the time sequence synchronization of cross-source data, and lays a foundation for subsequent data processing and real-time analysis.

[0036] (2) The application generates data processing load characteristic values by comprehensively processing parameters such as the number of objects, the average update frequency of objects, spatial distribution, object density and access pressure, represents data size and dynamic activity, determines the number of hash table layers, realizes efficient construction of a multi-layer hash table, optimizes data index and storage structure, improves the speed and accuracy of object state positioning, and provides a quantitative basis for real-time load monitoring and dynamic adjustment of hash buckets, thereby enhancing the performance and stability of the system in large-scale simulation data processing scenarios.

[0037] (3) The application realizes dynamic load balancing by real-time monitoring of the load state of hash buckets, combining high-load hash bucket splitting and low-load hash bucket merging mechanisms. A high-load hash bucket is split into multiple sub-buckets according to the upper load threshold, the modulo range of the hash function is expanded and the objects are remapped, thereby reducing the processing pressure of a single bucket. A low-load hash bucket is batch-merged according to the system computing resource carrying capacity, reasonably allocating memory, CPU and cache resources, and optimizing system resource utilization. This dynamic adjustment mechanism can ensure that the hash table remains efficient in operation under changes in data size or event flow bursts, thereby improving the overall throughput and response speed of the system.

[0038] (4) The application realizes preliminary screening of object states through a countable Bloom filter, reduces unnecessary accurate matching calculations, and reduces computational overhead. The system manages the count of hit objects and dynamically adjusts when the object state is updated. Missed objects are directly skipped for accurate matching, further improving processing efficiency. Combined with local cache management of out-of-order events, rule accurate matching based on logical operators and instant output triggering mechanisms, the system can guarantee the accuracy and real-time performance of event processing, thereby supporting efficient management of high-frequency, multi-source and large-scale simulation data and accurate execution of virtual training tasks, providing reliable data support and operation efficiency improvement for LVC training.

[0039] Of course, any product implementing the present application does not necessarily need to achieve all the above advantages at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The figure is a schematic diagram of the system modules of the application.

[0041] Figure 2 The figure is a schematic diagram of the method flow of the application.

[0042] Figure 3 The figure is a schematic diagram of the method logic flow of the application.

[0043] Figure 4 The figure is a schematic diagram of the module logic flow of the application.

[0044] Figure 5 The figure is a schematic diagram of the module logic flow of the application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0046] In the description of the present application, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "periphery" and the like indicate the orientation or positional relationship, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the components or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0047] Please refer to Figure 1 The embodiment of the present application provides a virtual training data management system based on LVC simulation, and specifically comprises: As Figure 3 The method logic flowchart involved in the embodiment of the present application is shown, which describes the overall data processing logic of the virtual training data management system based on LVC simulation, covering the whole process from original data collection to event output. The original state data from L, V and C three types of data sources are collected, including object unique identification, spatial coordinates and state information. The collected data are unified into internal object format through field mapping, coordinate conversion and time synchronization processing. The processed object data are input into a layered hash module. The system constructs a multi-layered hash table based on ODSLA index, and simultaneously monitors the load state of each hash bucket, including object quantity, average update frequency and access delay. When the load of the hash bucket is too high, high-load hash bucket splitting is triggered; when the load of the hash bucket is too low and the number exceeds the threshold, low-load hash bucket merging is triggered. The object state after hash indexing enters the Bloom filter module, and the initial screening is realized through the countable Bloom filter. The system judges whether the object hits according to the preset rules, and only the hit object enters the accurate event processing flow. The hit object state update event enters the stream event processing module. The module maintains local cache, sorts out-of-order events, and accurately matches events with rules. The matched events are immediately triggered to output, and the unmatched events end the processing. It reflects the complete closed loop of the system from data collection, indexing, initial screening to accurate matching, and highlights the dynamic load adjustment, screening optimization and real-time event processing capabilities.

[0048] As Figure 4 And Figure 5The module logic flowchart related to the embodiment of the application is shown, the module relationship of the virtual training data management system based on LVC simulation is described, and the flow and processing order of data in each functional unit are shown. The data flow and dependency relationship between the modules are facilitated to understand, and the core logic of the entire virtual training data management system is visualized.

[0049] It should be explained that LVC (Live-Virtual-Constructive) simulation refers to the cooperative operation of real systems (Live), virtual systems (Virtual) and model-based constructive systems (Constructive) in the same scene.

[0050] The layered hash index module is used to uniformly map the original state data from the three types of data sources L, V and C into an internal object format after collection, construct a multi-layer hash table based on the ODSLA index set, simultaneously monitor the load state of each hash bucket and dynamically adjust the hash bucket, and the object state positioned by the index enters the Bloom filter module.

[0051] The original state data from the three types of data sources L, V and C is uniformly mapped into an internal object format after collection, and the specific process is as follows: The original state data from the three types of data sources L, V and C is collected in real time, and the original state data includes object unique identification, spatial coordinate information and state identification. The original state data of the three types of data sources is mapped into a unified internal object format. In the embodiment of the application, an adapter is deployed in the collection layer, and a parsing plug-in is installed for the three types of data sources L, V and C to perform data parsing and mapping. The mapping process includes field name conversion (such as converting the "Track unique identification" of the L source into the internal "obj_unique identification"), and converting the data of different structures of the three types of data sources into a unified internal object format that can be directly processed by the system, so as to make it conform to the internal adaptation logic.

[0052] In the field mapping and unit unification process, the coordinate conversion function is used to uniformly map it to the globally defined coordinate system. It should be noted that because different data sources use different coordinate systems, for example, the L source usually uses the GPS WGS-84 coordinate system, the V source usually uses the local simulation world coordinate system, and the C source usually uses the map grid coordinate system, so it needs to be unified into the global coordinate system. In the embodiment of the application, the coordinate conversion function uses a mapping conversion coordinate function. By unifying the coordinates, spatial calculations (such as distance, speed, collision detection, and area judgment) are ensured to be performed on a unified space, avoiding calculation errors caused by inconsistent coordinate systems.

[0053] The state identification of the objects of different data sources is mapped to an internally defined enumeration type through a state mapping table built in the local database, and it needs to be noted that, since the state identification of different data sources is different in coding, it needs to be mapped to a unified internally defined enumeration type through the state mapping table, so as to avoid frequently writing different state judgment rules in the business logic and to ensure that the rule engine can directly judge based on the unified state enumeration.

[0054] The time information of each data source is uniformly converted into an internal standard timestamp through a time reference mapping and time difference compensation algorithm. In the embodiment of the application, the time reference mapping and time difference compensation algorithm uses a multi-source clock synchronization algorithm. It is ensured that the cross-source events are comparable in time.

[0055] A multi-layer hash table is constructed based on the ODSLA index set, and the specific process is as follows: The ODSLA index set is a parameter set for layered hashing, including the number of objects, the average update frequency of objects, spatial distribution, object density and access pressure.

[0056] It needs to be noted that the number of objects refers to the total number of objects in the current system that are in an active state and participate in index management, which is obtained by directly counting the number of object unique identifiers in the index table in the fast query interface. The average update frequency of objects refers to the average number of times of object state changes per unit time, which is obtained by maintaining an object average update frequency cache by the stream processing engine and outputting it periodically. The spatial distribution refers to the distribution of objects in the spatial coordinate system, which is used to measure the degree of location aggregation or dispersion, and is obtained by fast regional aggregation operation on the quadtree, specifically including dividing the global space into fixed-size grids, denoted as unit spaces, counting the number of objects in each grid, and calculating the distribution variance. The object density refers to the average number of objects per unit space, which is used to reflect the local crowding degree, and is obtained by taking the global average value in the spatial distribution calculation process. The access pressure refers to the total number of query and update requests per unit time for the hash bucket, which is obtained by maintaining an access counter for each hash bucket, counting the number of queries and updates, and then performing mean processing to obtain the access pressure.

[0057] The ODSLA verification set is extracted from the local database, including the verification object quantity, the verification object average update frequency, the verification spatial distribution, the verification object density and the verification access pressure.

[0058] The parameters in the ODSLA index set and the ODSLA verification set are respectively corresponded and one-to-one compared, and then weighted and coupled to obtain a data processing load characteristic value, which is used to represent the scale characteristics and dynamic activity of data, and the specific calculation process includes:

[0059] wherein, is a data processing load characteristic value, is the i-th parameter in the ODSLA index set, in the embodiment of the present application, the object quantity, the object average update frequency, the spatial distribution, the object density and the access pressure, in actual application, other parameters representing the data processing load characteristics can be selected, is the parameter corresponding to in the ODSLA verification set, in the embodiment of the present application, the verification object quantity, the verification object average update frequency, the verification spatial distribution, the verification object density and the verification access pressure. is the weighting factor of the i-th parameter in the database, satisfying in the embodiment of the present application, including the object quantity weighting factor, the object average update frequency weighting factor, the spatial distribution weighting factor, the object density weighting factor and the access pressure weighting factor, i is the number of the parameter in the ODSLA index set, i = 1, 2, 3,..., n, n is the total number of the parameters in the ODSLA index set, in the embodiment of the present application, n is 5, x1 is the object quantity, x2 is the object average update frequency, x3 is the spatial distribution, x4 is the object density and x5 is the access pressure.

[0060] It should be noted that in the embodiment of the present application, the selected object quantity weighting factor, object average update frequency weighting factor, spatial distribution weighting factor, object density weighting factor and access pressure weighting factor are used to adjust the weight of the corresponding parameter in the calculation, which can be obtained by actual parameter test and historical system running data based on machine learning algorithm analysis and processing.

[0061] It should be further noted that in the embodiment of the present application, the selected object quantity, object average update frequency, spatial distribution, object density and access pressure have certain correlation between the five parameters, the object quantity is the basic index, the increase of the total object quantity will directly increase the load of system index query, state update and event processing, and may also affect the performance of spatial distribution and object density. The object average update frequency represents the frequency of state change of each object in unit time, the object with high update frequency will increase the access pressure and generate hot spots in the hash bucket or cache, and then determine the index conflict probability together with the object quantity and density. The spatial distribution reflects the aggregation or uniformity of objects in space, the highly aggregated area may form hot spots, so that the access pressure of some hash bucket or cache increases. The object density is a local quantitative index of spatial distribution, the more objects in unit space, the higher the density, which will also increase the local access pressure and index update pressure. The access pressure directly reflects the frequency of object access or matching, which is usually related to the object quantity, update frequency and dense area height, and the object with high density and high update frequency often generates higher access pressure.

[0062] The data processing load characteristic value is input into a pre-stored mapping set of data processing load characteristic value-hash layer number in a local database to obtain a hash layer number after mapping matching, and a multi-layer hash table is constructed based on the hash layer number, specifically including: The mapping set of data processing load characteristic value-hash layer number is a set of historical ODSLA indexes collected from the system, including indexes such as object quantity, object average update frequency, spatial distribution, object density and access pressure, and the hash layer number when the multi-layer hash table is constructed under the load condition corresponding to each set of ODSLA index set characteristics. The relationship between the data processing load characteristic value and the optimal hash layer number is established by statistically analyzing the collected set of historical ODSLA indexes. The mapping relationship obtained by analysis is stored in the local database.

[0063] According to the hash layer number obtained by mapping matching, the hierarchical structure of the hash table is determined, and each layer corresponds to an object index space of different granularity. In the embodiment of the application, the top layer hash table is responsible for coarse-grained indexing, and the lower layer hash table is responsible for fine-grained indexing. Hash buckets are created for each layer, and the number and distribution of the buckets are dynamically allocated according to load characteristic parameters such as the number of objects, update frequency and spatial distribution of the layer to ensure balanced indexing and avoid hotspots. When an object is inserted, the corresponding coarse-grained bucket is located through the top layer hash function, and the fine-grained bucket is further located through the lower layer hash function. The object state information is stored in the cache of the corresponding bucket. When an object is updated or deleted, the same hierarchical and hash function is used for positioning to achieve efficient add, delete and modify operations. The entire multi-layer hash table structure simultaneously maintains metadata, including the number of layers, the number of buckets in each layer, the modulo range and load information, which is used to support subsequent hash bucket splitting, merging and load balancing adjustment, thereby realizing dynamic optimization of multi-layer indexing management for different load characteristics.

[0064] The load state of each hash bucket is monitored in real time, specifically including: The load state parameters of each hash bucket are collected by real-time scanning of each hash bucket in the multi-layer hash table, including the total number of objects in the bucket, the average update frequency of object state and the average access delay. These parameters are collected in real time through the internal statistical table of the hash bucket.

[0065] The data is stored in the local monitoring unit, and the pre-stored load state verification parameters in the local database are extracted, including the verification of the total number of objects in the bucket, the verification of the average update frequency of object state and the verification of the average access delay. The load state parameters of each hash bucket are compared with the corresponding load state verification parameters to obtain the load state representation value of each hash bucket after weighted coupling, specifically including:

[0066] Wherein, a load state characterization value of the yth hash bucket, a u th load state parameter of the yth hash bucket, in the embodiment of the present application, including the total number of objects in the bucket, the average update frequency of object state, and the average access delay, in actual application, other parameters representing the load state of the hash bucket can be selected, a load state check parameter corresponding to the u th load state parameter of the yth hash bucket, in the embodiment of the present application, including the total number of objects in the check bucket, the average update frequency of the check object state, and the average access delay, a weighting factor of the u th load state parameter of the yth hash bucket, in the embodiment of the present application, including the total number of objects in the bucket weighting factor, the average update frequency of object state weighting factor, and the average access delay weighting factor, y is the hash bucket number, y = 1, 2, 3,..., p, p is the total number of hash buckets, u is the load state parameter number, u = 1, 2, 3,..., m, m is the total number of load state parameters, in the embodiment of the present application, m is 3, m1 is the total number of objects in the bucket, m2 is the average update frequency of object state, and m3 is the average access delay.

[0067] It should be noted that the total number of objects in the bucket weighting factor, the average update frequency of object state weighting factor, and the average access delay weighting factor are used to adjust the weight of the corresponding parameter in the calculation, which can be obtained by analyzing and processing the actual parameter test and historical system running data based on machine learning algorithm.

[0068] dynamically adjusting the hash bucket, specifically including: extracting a preset load state upper threshold from the local database, comparing the load state characterization value of each hash bucket with the load state upper threshold, when the load state characterization value of a certain hash bucket is greater than the load state upper threshold, the hash bucket is recorded as a high load hash bucket, triggering the high load hash bucket splitting adjustment, specifically including: subtracting the load state characterization value of the high load hash bucket from the load state upper threshold to obtain the load state characterization upper limit difference value of the high load hash bucket, inputting the load state characterization upper limit difference value of the high load hash bucket into the pre-stored mapping set of load state characterization upper limit difference value-hash bucket splitting number in the database to obtain the hash bucket splitting number of the high load hash bucket, and splitting the high load hash bucket into multiple sub-hash buckets based on the hash bucket splitting number, the specific splitting method includes: based on the hash bucket splitting number, increasing the modulo range of the hash function, and remapping the objects in the high load hash bucket to each sub-hash bucket.

[0069] In the plurality of historical operation data, the difference between the load state representation value of each hash bucket and the upper limit threshold of the load state, and the optimal split number required when the hash bucket is split at the difference, are recorded, and a discrete table is used to establish the relationship between the upper limit difference of the load state representation and the corresponding hash bucket split number, and different upper limit differences are mapped to the recommended split number. The mapping relationship obtained is stored in the local database as a reference for the system to dynamically adjust the high-load hash bucket. In actual operation, the system calculates the difference between the real-time load state representation value of each hash bucket and the upper limit threshold, and obtains the corresponding split number by calling the mapping set, so as to ensure that the split operation not only sufficiently relieves the high load, but also avoids unnecessary over-splitting, thereby improving the system performance and resource utilization.

[0070] The preset lower limit threshold of the load state is extracted from the local database, and the load state representation value of each hash bucket is compared with the lower limit threshold of the load state. When the load state representation value of a certain hash bucket is less than the lower limit threshold of the load state, the hash bucket is recorded as a low-load hash bucket, and the number of low-load hash buckets is counted. When the number of low-load hash buckets exceeds the preset threshold of the number of low-load hash buckets, low-load hash bucket merging adjustment is triggered, which specifically includes: The maximum bearing computing resource parameters of the merging unit are counted. The merging unit refers to the data processing unit built in the system in the embodiment of the application for merging hash buckets. The maximum bearing computing resource parameters include memory resource parameters, CPU computing resource parameters and cache resource parameters. Based on the maximum bearing computing resource parameters of the merging unit, the maximum number of buckets for single merging is determined, and the specific process includes: The maximum bearing computing resource parameters of the merging unit are respectively input into the resource parameter-single supported merging bucket number mapping set built in the local database, including the memory resource parameter-single supported merging bucket number mapping set, the CPU computing resource parameter-single supported merging bucket number mapping set and the cache resource parameter-single supported merging bucket number mapping set, and one-to-one mapping matching is performed to obtain the single supported merging bucket number corresponding to each maximum bearing computing resource parameter. The minimum value is taken as the maximum number of buckets for single merging.

[0071] The maximum number of buckets that can be safely processed by single hash bucket merging operation under different memory resource, CPU computing resource and cache resource configurations in the historical records is extracted respectively. Through statistical analysis of a large amount of historical data, the mapping relationship of memory resource parameter-single supported merging bucket number, CPU computing resource parameter-single supported merging bucket number and cache resource parameter-single supported merging bucket number is established. These mapping relationships are stored in the local database in the form of functions. In actual operation, the system obtains the corresponding single supported merging bucket number by calling the mapping relationship according to the current resource state of each merging unit, and takes the minimum value of the corresponding values of the three types of resources as the maximum number of buckets for single merging, so as to ensure that the merging operation is efficient and safe, and avoid exceeding the computing resource limit.

[0072] It should be noted that the memory resource parameter can be obtained by monitoring the available memory capacity of the merging unit in the normal running state, and the average memory amount occupied by each hash bucket object data, and normalized to the memory resource utilization percentage.

[0073] The CPU computing resource parameter can be quantified by recording the CPU occupancy rate and average execution time of the merging unit in the process of merging the hash bucket, performing hash calculation, object migration, index updating and other operations. The maximum CPU demand of a single merging operation is obtained, and normalized to the CPU resource percentage.

[0074] The cache resource parameter is measured by measuring the cache used by the merging unit during merging, combined with the cache hit rate of object access frequency and migration operation, to evaluate the maximum cache space occupied during merging, and normalized to the cache resource percentage.

[0075] The maximum number of buckets in a single merge is used to limit the number of low-load hash buckets allowed to merge in a single hash bucket merge.

[0076] If the number of low-load hash buckets is less than or equal to the maximum number of buckets in a single merge, each low-load hash bucket is recorded as a single merge batch, the computing resources for the merged hash bucket are pre-allocated, the objects in each low-load hash bucket are migrated to the new hash bucket one by one, and the index position of the object in the hash table is updated. After migration, release the computing resources occupied by each low-load hash bucket, and update the metadata of the hash table, including the total number of buckets, the modulo range and the load information.

[0077] If the number of low-load hash buckets is greater than the maximum number of buckets in a single merge, each low-load hash bucket is sorted in order of hash value, and in the sorted low-load hash bucket list, select no more than the maximum number of buckets in a single merge as a single merge batch for hash bucket merging. The remaining low-load hash buckets that have not been merged are used as the next batch for hash bucket merging until all low-load hash buckets are merged.

[0078] The Bloom filter module is used to initially screen the object state, identify the hit objects, and eliminate the non-hit objects. In combination with the countable mechanism, the object state is dynamically adjusted during the object state update, and the screened object state update event enters the stream event processing module as an event stream.

[0079] A countable Bloom filter is created for each rule preset in the local database. The rule refers to the trigger condition and judgment logic set for the object state change, which is used for initial screening to determine whether the object needs further processing.

[0080] The historical triggering probability and the historical triggering object quantity of each rule are obtained, and the bit array size of the countable Bloom filter is configured based on the historical triggering probability and the historical triggering object quantity of each rule, specifically comprising: The historical triggering probability refers to the proportion of the number of events that trigger the rule in the past time window to the total number of events; the historical triggering object quantity refers to the number of independent objects that trigger the rule in the same time window. These data can be obtained from the log system, event database or cache statistics table, and time window aggregation is performed.

[0081] The historical triggering probability and the historical triggering object quantity are combined to estimate the number of hash bits that each rule may need. The bit array size of the CBF directly affects the false positive rate and memory occupation: the higher the triggering probability and the more the triggering objects of a rule, the more bits are needed to reduce false positives; on the contrary, rules with low triggering probability or few objects can use fewer bits to save resources. The calculation method is specifically: ; wherein, is the bit array size, t is the historical triggering object quantity, and w is the historical triggering probability.

[0082] When the object state first satisfies the triggering condition of a rule, it is recorded as a hit object, and the unique identifier of the hit object is inserted into the CBF (countable Bloom filter) and the counter is increased to record the number of times it exists. After the hit object state is updated, if it still satisfies the rule, the number of times the hit object exists in the counter is increased, and if it no longer satisfies the rule, the number of times the hit object exists in the counter is decreased. When the counter is zero, the hit object is logically removed and recorded as a miss object.

[0083] Before each object state update, the CBF is queried, the hit object enters the stream event processing module for accurate matching, and the miss object directly skips the accurate calculation, which can significantly reduce unnecessary calculation overhead. The miss object directly skips the accurate matching, avoiding repeated calculation on a large number of objects that are not related to the rule or temporarily do not trigger the rule, thereby reducing CPU and memory load and improving overall system processing efficiency.

[0084] By dynamically maintaining the hit object through the countable Bloom filter, the state change of the object can be accurately tracked. The counter records the number of times each hit object exists, ensuring that in the case of frequent changes in the object state, it can still correctly determine whether it should participate in rule matching, avoiding missed or repeated triggering of events, and improving the accuracy of event processing. In addition, this method can support high concurrency and real-time processing, so that the system can still maintain millisecond-level response when facing the state update of thousands of objects, thereby ensuring that the stream event processing module can quickly obtain the objects that need accurate matching and ensure the timely triggering of key events.

[0085] The stream event processing module is used for accurate matching of the event stream, manages out-of-order events through local cache, and outputs and triggers the object state update event of accurate matching success.

[0086] The event stream is received, the event stream includes object state update events, a local cache is maintained for the event stream, the local cache is an independent data structure, and the object state update events in the event stream are time-sequentially sorted, and the specific process includes extracting the time stamp of each object state update event in the event stream and inserting the time stamp into a time-sequencing queue.

[0087] The time sequence of the object state update events can be ensured by maintaining the local cache for the event stream and sorting according to the time stamp. In LVC simulation, there may be data delay or out-of-order problems in different data sources, the local cache can temporarily store and sort the out-of-order events, ensure that the events are processed in the correct time sequence before rule matching, and thus avoid matching errors or logical inconsistencies caused by chaotic event order.

[0088] The object unique identifier, spatial coordinate information and state identifier in each object state update event are obtained, each object state update event is accurately matched with a rule based on a logical operator, if a certain object state update event meets a certain rule, it is recorded as matching success, and the object state update event is immediately output and triggered, if a certain object state update event does not meet a certain rule, it continues to be matched with the next rule until the matching success, and if the object state update event does not match each rule in the local database, it is recorded as matching failure, and the matching process is ended.

[0089] The accurate matching of the event and the rule based on the logical operator can ensure that each rule is triggered only when the object state update meets the condition, avoid false triggering, support the processing logic of multiple rules matched one by one, and improve the accuracy of rule matching. The matched event is immediately output and triggered, which is beneficial to the quick response and processing of the system to the key event.

[0090] In this embodiment, the present application provides a virtual training data management method based on LVC simulation, which comprises: After collecting the original state data from the L, V and C three types of data sources, the original state data is uniformly mapped into an internal object format, a multi-layer hash table is constructed based on the ODSLA index set, the load state of each hash bucket is monitored in real time, and the hash bucket is dynamically adjusted, and the object state positioned by the index enters the Bloom filter module.

[0091] The object state is preliminarily screened, the hit object is identified, and the non-hit object is removed, the countable mechanism is combined, the object state is dynamically adjusted when the object state is updated, and the screened object state update event enters the stream event processing module as the event stream.

[0092] The event stream is accurately matched, disordered events are managed through local cache management, and an object state update event of accurate matching success is output and triggered.

[0093] It should be noted that in another embodiment of the application, the above-mentioned system is applied as a subsystem of an LVC data integrated management system, the LVC data integrated management system collects LVC simulation training data in real time through a multi-source heterogeneous interface, intelligently labels, extracts features and mines knowledge from the original data in combination with artificial intelligence technology, and realizes deep structured processing of the data.

[0094] Based on the underlying data processing flow of the application, the LVC data integrated management system in another embodiment of the application supports high concurrency access, and guarantees efficient management and accuracy of data in LVC training.

[0095] It should be noted that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between or among the entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0096] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details of the application, nor limit the application to the specific embodiments. Obviously, many modifications and variations can be made to the application according to the content of the specification. The specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. Any modification or variation that does not deviate from the structure of the application or exceed the scope defined by the application should belong to the protection scope of the application.

Claims

1. A virtual training data management system based on LVC simulation, characterized in that: include: The hierarchical hash index module collects raw state data from three data sources (L, V, and C) and maps them uniformly into an internal object format. It then constructs a multi-layer hash table based on the ODSLA indicator set. It also monitors the load status of each hash bucket in real time and dynamically adjusts the hash buckets. The object state located by the index enters the Bloom filter module. The Bloom filter module is used to initially screen object states, identify hit objects, and eliminate miss objects. Combined with a countable mechanism, it dynamically adjusts object states when they are updated. The filtered object state update events are fed into the streaming event processing module as event streams. The streaming event processing module is used to accurately match event streams, handle out-of-order events through local cache management, and output and trigger object status update events that have successfully matched the event.

2. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The original state data from the three data sources L, V and C are collected and uniformly mapped into the internal object format. The specific process is as follows: Real-time collection of raw state data from three data sources: L, V, and C. The raw state data includes object unique identifiers, spatial coordinate information, and state identifiers, and mapping the raw state data from the three data sources into a unified internal object format. In the process of field mapping and unit unification, the coordinate conversion function is used to uniformly map them to the internally defined global coordinate system; The state identification of objects from different data sources uses the state mapping table built into the local database to map external state values ​​to internally defined enumeration types; Through time base mapping and time difference compensation algorithm, the time information of each data source is uniformly converted into an internal standard timestamp.

3. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The specific process of constructing a multi-layer hash table based on the ODSLA indicator set is as follows: The ODSLA indicator set is a parameter set for hierarchical hashing, including the number of objects, average object update frequency, spatial distribution, object density, and access pressure; Extract the ODSLA checksum from the local database, including the number of checksum objects, average update frequency of checksum objects, checksum spatial distribution, checksum object density, and checksum access pressure; The ODSLA indicator set and the parameters in the ODSLA verification set are respectively matched and compared one by one, and then weighted coupling is performed to obtain the data processing load characteristic value, which is used to characterize the scale characteristics and dynamic activity of the data; The data processing load characteristic value is input into the mapping of data processing load characteristic value-hash layer number pre-stored in the local database, and the hash layer number is obtained after mapping matching, and a multi-layer hash table is constructed based on the hash layer number.

4. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The real-time monitoring of the load status of each hash bucket specifically includes: Scan each hash bucket in the multi-layer hash table in real time, collect the load status parameters of each hash bucket, including the total number of objects in the bucket, the average update frequency of the object status and the average access delay, store the data in the local monitoring unit, extract the load status verification parameters pre-stored in the local database, including the total number of objects in the verification bucket, the average update frequency of the verification object status and the verification average access delay, compare the load status parameters of each hash bucket with the corresponding load status verification parameters one by one, and then perform weighted coupling to obtain the load status representation value of each hash bucket.

5. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The dynamic adjustment of the hash bucket specifically includes: Extract the preset load status upper limit threshold from the local database, compare the load status representation value of each hash bucket with the load status upper limit threshold, and when the load status representation value of a hash bucket is greater than the load status upper limit threshold, the hash bucket is marked as a high-load hash bucket, triggering the high-load hash bucket split adjustment; The preset load state lower limit threshold is extracted from the local database, and the load state representation value of each hash bucket is compared with the load state lower limit threshold. When the load state representation value of a hash bucket is less than the load state lower limit threshold, the hash bucket is recorded as a low-load hash bucket, and the number of low-load hash buckets is counted. When the number of low-load hash buckets exceeds the preset low-load hash bucket number threshold, the low-load hash bucket merger adjustment is triggered.

6. The virtual training data management system based on LVC simulation according to claim 5, characterized in that: The triggering of high-load hash bucket split adjustment specifically includes: The load state characterization value of the high-load hash bucket is subtracted from the load state upper limit threshold to obtain the load state characterization upper limit difference of the high-load hash bucket. The load state characterization upper limit difference of the high-load hash bucket is input into the mapping of load state characterization upper limit difference-hash bucket split number pre-stored in the database, and a mapping match is performed to obtain the hash bucket split number of the high-load hash bucket. The high-load hash bucket is split into multiple sub-hash buckets based on the hash bucket split number. The specific splitting method includes: Based on the number of hash bucket splits, the modulus range of the hash function is increased, and objects in the high-load hash bucket are remapped to each sub-hash bucket.

7. The virtual training data management system based on LVC simulation according to claim 5, characterized in that: The triggering of low-load hash bucket merging adjustment specifically includes: Counting the maximum load computing resource parameters of the merging unit, including memory resource parameters, CPU computing resource parameters, and cache resource parameters, and determining the maximum number of buckets for a single merge based on the maximum load computing resource parameters of the merging unit, wherein the maximum number of buckets for a single merge is used to limit the number of low-load hash buckets allowed to be merged in a single hash bucket merge; If the number of low-load hash buckets is less than or equal to the maximum number of buckets for a single merge, each low-load hash bucket is recorded as a single merge batch, computing resources are allocated to the merged hash buckets in advance, objects in each low-load hash bucket are migrated one by one to the new hash bucket, and the index position of the object in the hash table is updated. After the migration is completed, the computing resources occupied by each low-load hash bucket are released, and the metadata of the hash table is updated, which includes the total number of buckets, modulo range, and load information; If the number of low-load hash buckets is greater than the maximum number of buckets for a single merge, the low-load hash buckets are sorted in order of hash values. In the sorted low-load hash bucket list, the low-load hash buckets that do not exceed the maximum number of buckets for a single merge are selected as a single merge batch, and hash bucket merges are performed. The remaining unmerged low-load hash buckets are taken as the next batch, and hash bucket merges are performed until all low-load hash buckets are merged.

8. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The identification of hit objects and elimination of missed objects are combined with a counting mechanism to perform dynamic adjustments when the object status is updated. The specific processing conditions are: Creating a countable Bloom filter for each rule pre-set in the local database, wherein the rule refers to the trigger conditions and decision logic set for the object state change; Obtain the historical trigger probability and the number of historical trigger objects of each rule, and configure the bit array size of the countable Bloom filter based on the historical trigger probability and the number of historical trigger objects of each rule; When the state of an object meets the triggering condition of a rule for the first time, it is recorded as a hit object. The unique identifier of the hit object is inserted into the countable Bloom filter and the counter is increased to record its existence count. After the state of the hit object is updated, if it still meets the rule, the existence count of the hit object is increased in the counter. If it no longer meets the rule, the existence count of the hit object is reduced in the counter. When the counter reaches zero, the hit object is logically removed and recorded as a miss object. Before each object status update, the countable Bloom filter is queried, and the hit object enters the streaming event processing module for precise matching. If the object is not hit, the precise calculation is skipped directly.

9. The virtual training data management system based on LVC simulation according to claim 1, characterized in that: The process of performing precise matching on the event stream, processing out-of-order events through local cache management, and outputting and triggering the object status update event with successful precise matching includes: Receive an event stream, the event stream including each object state update event, maintain a local cache for the event stream, the local cache is an independent data structure used to time-sequence each object state update event in the event stream, and specifically extract the timestamp of each object state update event in the event stream and insert it into a time-sequence queue; Obtain the object unique identifier, spatial coordinate information and status identifier in each object status update event, and accurately match each object status update event with the rules based on logical operators. If an object status update event meets a certain rule, it is recorded as a successful match, and the object status update event is immediately output and triggered. If an object status update event does not meet a certain rule, it continues to match the next rule until a successful match is achieved. If an object status update event does not match any rules in the local database, it is recorded as a match failure and the matching process ends.

10. The method for managing virtual training data based on LVC simulation according to any one of claims 1 to 9, characterized in that: The original state data from the three data sources (L, V, and C) is collected and uniformly mapped into an internal object format. A multi-layer hash table is constructed based on the ODSLA indicator set. The load status of each hash bucket is monitored in real time and dynamically adjusted. The object status located by the index enters the Bloom filter module. Perform an initial screening of object status, identify hit objects, and eliminate miss objects. Combined with a countable mechanism, dynamic adjustments are made when object status is updated. The filtered object status update events enter the streaming event processing module as an event stream; Accurately match the event stream, handle out-of-order events through local cache management, and output and trigger object status update events that have been accurately matched.

Images