A high-concurrency time sequence data processing method and device for a counter-UAV system

Abstract

The application discloses a kind of high concurrency time series data processing method and device for anti-UAV system.The application uses lock-free hash table to cache time series data in cache space, lock-free cache design and CAS atomic operation reduce lock overhead, batch write throughput is high, conflict probability is low, support multi-thread high concurrency read-write, completely eliminate thread competition and blocking caused by traditional cache mechanism, improve concurrent processing capability;At the same time, batch write reduces the overhead of dynamic memory allocation, improves memory utilization;Through composite key fast positioning, hot data index structure priority query and lock-free reading mechanism, the query delay of hot data is extremely low, data preloading mechanism further reduces the delay of future related query, improves real-time response capability;Based on the dynamic elimination mechanism of elimination score, hot data / recent data can be preferentially retained, ensure resource utilization, significantly improve cache hit rate, greatly reduce access frequency to TSDB.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a high-concurrency time-series data processing method and device for a counter-unmanned aerial system. BACKGROUND

[0002] With the popularization of unmanned aerial vehicle technology, the importance of counter-unmanned aerial system (C-UAS) is increasingly prominent. The C-UAS system needs to collect, process and analyze high-concurrency time-series data (such as position, speed, height, signal strength, etc.) from multiple sensors such as radar, photoelectric, radio spectrum, etc. in real time to perform target detection, tracking, identification and threat assessment. These time-series data have the characteristics of large data volume, high concurrency, data immutability, high access frequency of recent data (hotspot characteristics), and complex query patterns.

[0003] Existing time-series data processing technologies, such as directly using time-series databases (TSDB) or traditional caching mechanisms, have obvious shortcomings when facing the high-concurrency write and complex query requirements of the C-UAS system:

[0004] 1. Concurrency performance bottleneck: Traditional caching mechanisms (such as lock-based hash tables) will cause severe thread contention and blocking in high-concurrency write scenarios, reducing system throughput and response speed;

[0005] 2. Cache data eviction policy is not intelligent enough to fully consider the hotspot and freshness characteristics of time-series data, resulting in low cache hit rate;

[0006] 3. Large memory management overhead: Frequent single data writes result in large dynamic memory allocation overhead, affecting performance;

[0007] 4. Lack of optimization for multi-dimensional composite keys and data preloading mechanisms, making it difficult to handle predictive queries and dynamic load changes.

[0008] These problems limit the efficiency and response speed of the system in large-scale urban C-UAS applications. Therefore, it is necessary to develop a new high-concurrency time-series data processing method and device to overcome the shortcomings of existing technologies and improve the overall performance of the C-UAS system. SUMMARY

[0009] The technical problem to be solved by the present application is to provide a high-concurrency time-series data processing method and device for a counter-unmanned aerial system, effectively addressing the challenges faced by existing technologies in the face of high-concurrency write and complex query requirements of the C-UAS system, and significantly improving the overall performance of the C-UAS system.

[0010] In a first aspect, a high-concurrency time-series data processing method for an anti-UAV system is provided, comprising the following steps: receiving high-concurrency time-series data from multiple sensors; generating a composite key for each time-series data, forming a key-value pair of the composite key of the time-series data and the value of the time-series data, and using a CAS atomic operation to batch-write the time-series data into a lock-free hash table; adding the composite key of the newly inserted time-series data of the lock-free hash table to a hot data index structure, and updating the hot data index structure; in response to a query request, using the hot data index structure and the lock-free hash table to perform data retrieval and return a query result; analyzing a query pattern of the query request, determining pre-fetch data to be loaded based on temporal locality and pattern correlation, and generating a pre-loading task; asynchronously executing the pre-loading task, loading the pre-fetch data and storing it into the lock-free hash table, and updating the hot data index structure; when the cache space is insufficient, dynamically removing low-priority data in the lock-free hash table based on an eviction score, the eviction score being calculated according to data access frequency and freshness.

[0011] In a second aspect, a high-concurrency time-series data processing device for an anti-UAV system is provided, comprising: a data receiving module for receiving high-concurrency time-series data from multiple sensors; a data writing module for generating a composite key for each time-series data, forming a key-value pair of the composite key of the time-series data and the value of the time-series data, and using a CAS atomic operation to batch-write the time-series data into a lock-free hash table; an index updating module for adding the composite key of the newly inserted time-series data of the lock-free hash table to a hot data index structure, and updating the hot data index structure; a data query module for, in response to a query request, using the hot data index structure and the lock-free hash table to perform data retrieval and return a query result; a pre-loading task generation module for analyzing a query pattern of the query request, determining pre-fetch data to be loaded based on temporal locality and pattern correlation, and generating a pre-loading task; a pre-loading task execution module for asynchronously executing the pre-loading task, loading the pre-fetch data and storing it into the lock-free hash table, and updating the hot data index structure; and a cache data eviction module for, when the cache space is insufficient, dynamically removing low-priority data in the lock-free hash table based on an eviction score, the eviction score being calculated according to data access frequency and freshness.

[0012] The high-concurrency time-series data processing method and device for the anti-UAV system described above uses a lock-free hash table to cache time-series data in the cache space, lock-free cache design and CAS atomic operation reduce lock overhead, batch write throughput is high, conflict probability is low, and multi-thread high-concurrency read and write are supported, thread competition and blocking caused by the traditional cache mechanism are completely eliminated, and the concurrent processing capability is improved; at the same time, batch write reduces the overhead of dynamic memory allocation, and improves the memory utilization; through the composite key fast positioning, hot data index structure priority query and lock-free reading mechanism, the query delay of hot data is extremely low, the data preloading mechanism further reduces the delay of future related queries, and the real-time response capability is improved; the dynamic elimination mechanism based on elimination score can preferentially retain hot / recent data, ensure resource utilization, significantly improve cache hit rate, greatly reduce access frequency to the TSDB, and improve the overall performance of the C-UAS system. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0014] Figure 1 is a flowchart of a high-concurrency time-series data processing method for an anti-UAV system in an embodiment of the present application;

[0015] Figure 2 is a flowchart of the high-concurrency time-series data processing method for the anti-UAV system in the embodiment of the present application; Figure 1 is a sub-flowchart of step S30 in

[0016] Figure 3 is a sub-flowchart of step S40 in Figure 1

[0017] Figure 4 is a sub-flowchart of step S50 in Figure 1

[0018] Figure 5 is a sub-flowchart of step S60 in Figure 1

[0019] Figure 6 is a sub-flowchart of step S70 in Figure 1

[0020] Figure 7 is a sub-flowchart of step S72 in Figure 6

[0021] Figure 8 ​​​​​is Figure 1 Another sub-flowchart of step S30 in

[0022] Figure 9 is a structural schematic diagram of a high-concurrency timing data processing device for an anti-UAV system in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0024] Embodiments of the present application provide a high-concurrency timing data processing method and device for an anti-UAV system, which are applied to a C-UAS system in a city scene. Through a lock-free cache design, batch processing, a data preloading mechanism and a dynamic elimination mechanism based on an elimination score, the C-UAS system significantly improves the concurrency capability, query speed and cache hit rate in processing massive timing data, and realizes high real-time and high-concurrency stream data processing. The present application will be described in detail through specific embodiments. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] Figure 1 One flowchart of the high-concurrency timing data processing method for an anti-UAV system provided by the embodiments of the present application is shown as follows, Figure 1 The high-concurrency timing data processing method for an anti-UAV system includes steps S10 to S80.

[0026] S10: Create a lock-free hash table of a specified size, a lock-free queue, a lock-free array and a hot data index structure in a cache space.

[0027] When the system is initialized, a lock-free hash table of a specified size (N slot positions), a lock-free queue, a lock-free array for storing metadata and a hot data index structure are created in a cache space.

[0028] S20: Receive high-concurrency timing data from multiple sensors.

[0029] The timing data stream of high-concurrency different sensors includes electromagnetic signal (radar, TDOA, UAV communication) data, perception signal (image, video, sound) data, environment signal (geography, weather) data and text data (task, scheme) and the like.

[0030] The received time series data contains multiple fields. In this embodiment, the time series data contains four fields (timestamp, device ID, measurement type, value); in other embodiments, the time series data may also contain five fields (timestamp, device ID, measurement type, accuracy level, value).

[0031] S30: Generate a composite key for each time series data, combine the composite key of the time series data with the value of the time series data to form a key-value pair, and use CAS atomic operation to write the time series data in batches into a lock-free hash table.

[0032] Lock-free hash tables store data in key-value pairs. This invention combines the composite key of time-series data with the value of time-series data to form key-value pairs and stores them in a lock-free hash table. When querying a lock-free hash table, the composite key is used as an index to quickly locate the corresponding time-series data.

[0033] like Figure 2 As shown, step S30 specifically includes steps S31 to S35:

[0034] S31: Push the received time-series data into a lock-free queue to buffer high-concurrency write requests.

[0035] Upon receiving the timing data, the timing data is immediately pushed into a lock-free queue for high-concurrency write buffering.

[0036] S32: Extracts timing data in batches from the lock-free queue and pre-allocates memory blocks for the extracted timing data.

[0037] The system monitors the lock-free queue. When the amount of data in the lock-free queue reaches the batch size B (default 100), it retrieves the time-series data in batches and writes the batch of time-series data into a pre-allocated memory block. By batch processing high-concurrency time-series data from multiple sensors, the overhead of dynamic memory allocation can be reduced.

[0038] S33: Generate a composite key for each time series data.

[0039] Based on the timestamp, device identifier, and measurement type of the time-series data, a composite key K is calculated and generated for each time-series data entry (K = hash(T | D | Q), where T, D, and Q represent the timestamp, device ID, and measurement type, respectively). The lock-free hash table stores data in key-value pairs. This step generates a composite key for each time-series data entry so that the composite key and the time-series data value can be combined to form a key-value pair, thereby writing the time-series data into the lock-free hash table. When querying the lock-free hash table, the composite key is used as an index to quickly locate the corresponding time-series data.

[0040] S34: The composite key of the time series data and the value of the time series data form a key-value pair, and the time series data is written in bulk to the lock-free hash table using the CAS atomic operation.

[0041] The time series data is written in bulk to the lock-free hash table through the CAS atomic operation, and the bulk write throughput is , where B is the bulk size, is the time overhead of a single CAS atomic operation, is the total time overhead of extracting data from the lock-free queue.

[0042] In the process of writing the time series data in bulk to the lock-free hash table, the composite key of the time series data and the value of the time series data form a key-value pair, and the time series data is inserted into the lock-free hash table slot in the form of a key-value pair using the CAS atomic operation. When a conflict occurs, retry, each lock-free hash table slot supports atomic update, and allows multi-thread concurrent read and write.

[0043] Conflict probability: , N is the number of slots, the larger the number of slots, the less the conflict. Therefore, by increasing the number of slots N, the conflict rate can be reduced, and the concurrent performance can be improved.

[0044] S35: Create metadata for the newly inserted time series data of the lock-free hash table, record the access time and access count of the time series data.

[0045] While writing the time series data in bulk to the lock-free hash table, create metadata for the newly inserted time series data of the lock-free hash table: record the access time and access count of the time series data, and store it in the lock-free array.

[0046] S40: Add the composite key of the newly inserted time series data of the lock-free hash table to the hot data index structure, and update the hot data index structure.

[0047] The metadata of each time series data in the lock-free hash table (including access time and access count) is stored in the lock-free array. Through the read operation, the access count of each time series data in the lock-free hash table can be read from the lock-free array, and the hot data index structure can be updated according to the read access count.

[0048] As shown in Figure 3 , the step S40 specifically includes the following steps:

[0049] S41: Add the composite key of the newly inserted time series data of the lock-free hash table to the hot data index structure;

[0050] S42: Read the access count of each composite key in the data index structure from the lock-free array, and adjust the order or priority of each composite key in the hot data index structure using the LRU or LFU strategy according to the read access count.

[0051] Through steps S41-S42, the composite keys of the newly inserted time-series data in the lock-free hash table are added to the hot data index structure. Based on the access count (access frequency) of each composite key in the hot data index structure, the order or priority of each composite key in the hot data index structure is adjusted using the LRU (Least Recently Used) or LFU (Least Frequently Used) strategy. Hot data is retained first, so that hot data (such as data from the most recent hour) is kept in the cache, reducing access to the TSDB (Time Series Database). The cache hit rate (H = number of hot data queries / total number of queries) can reach over 90%.

[0052] S50: In response to the query request, perform data retrieval using the hot data index structure and the lock-free hash table and return the query results.

[0053] like Figure 4 As shown, step S50 specifically includes steps S51 to S55:

[0054] S51: Receive query requests from C-UAS.

[0055] The query request specifies the time range, device ID, and measurement type. Here, the time range refers to the range of times when each time series data was generated.

[0056] S52: Based on the time range, device ID, and measurement type requested in the query request, search for composite keys that meet the query conditions in the hotspot data index structure.

[0057] Upon receiving a query request, the system first checks the hotspot data index structure. Specifically, it iterates through each composite key in the hotspot data index structure. If a composite key that meets the query conditions is found, it indicates a match; otherwise, it indicates a miss.

[0058] S53: If a match is found, the time-series data in the corresponding slot of the lock-free hash table is read directly without lock using the matched composite key and returned to C-UAS.

[0059] When a composite key that meets the query conditions is found in the hot data index structure, the time-series data in the corresponding slot of the lock-free hash table is read directly without locking using the matched composite key. All time-series data that meet the query conditions are collected and returned to C-UAS. The query time complexity is O(1), and the query latency is [not specified]. It's very small.

[0060] S54: If no match is found, query the time series database for time series data that meets the query conditions through the query request and return it to C-UAS.

[0061] If no composite key matching the query conditions is found in the hot data index structure, the query request is used to retrieve time series data matching the query conditions from the time series database and return it to C-UAS.

[0062] S55: Update the metadata of the queried time series data.

[0063] The metadata of time-series data includes access time and access count. Each time time-series data is accessed, the access count is incremented by 1, and the latest access time is recorded. The time-series data being queried includes time-series data already stored in a lock-free hash table. The original metadata of this part of the time-series data is already stored in a lock-free array. By incrementing the access count and recording the latest access time, the metadata of the queried time-series data can be updated. The time-series data being queried also includes time-series data that meets the query conditions and is retrieved from the time-series database. After this part of the time-series data is queried, it is added to the lock-free hash table. By recording the access count and access time of this time-series data in the lock-free array for the first time, the metadata of the queried time-series data can be updated.

[0064] S60: Analyze the query pattern of the query request, determine the prefetched data to be loaded based on temporal locality and pattern correlation, and generate a preloading task.

[0065] The goal of preloading is to analyze query patterns in query requests and preload potentially accessed data from the TSDB (Time Series Database) or lower-priority cache layers into high-hotspot cache layers (such as lock-free hash tables or hotspot data index structures). This reduces the number of direct accesses to the TSDB, thereby reducing future query latency and improving cache hit rate.

[0066] like Figure 5 As shown, step S60 specifically includes steps S61 to S63:

[0067] S61: Calculate the hotspot score of the time-series data corresponding to the query request based on the access frequency and access freshness of the composite key corresponding to the query request.

[0068] Specifically, upon receiving a query request, the query conditions are decomposed into a matching pattern of composite keys, the access count and access time of the composite key corresponding to the query request are recorded, and the access pattern of the composite key is updated. After the query is completed, the data preloading function is triggered, and the hotspot score S of the time-series data corresponding to the query request is calculated according to the formula S = α·F + β·R.

[0069] in, The access frequency of the composite key. For example, if a composite key is queried 100 times in one hour, and the time window is 3600 seconds, then... .

[0070] : Access freshness of composite keys, where, The timestamp of the current time; This is the timestamp of the most recent access time to the composite key. This is a normalization factor, usually the maximum time range (e.g., 1 day = 86400 seconds).

[0071] α and β are hyperparameters used to balance access frequency and access freshness, and α+β=1.

[0072] The calculated hotspot score is compared with a hotspot threshold (e.g., 0.03). If the hotspot score exceeds the hotspot threshold, step S62 is executed; otherwise, the data preloading function is terminated.

[0073] S62: If the hotspot score exceeds the hotspot threshold, then analyze the query pattern of the query request.

[0074] When the hotspot score exceeds the hotspot threshold, the query pattern of the query request is analyzed. In time-series data caching systems, query patterns typically exhibit temporal locality and pattern relevance:

[0075] * Temporal locality: User queries tend to focus on recent data (such as sensor data from the last hour) or continuous time periods.

[0076] * Pattern Relevance: Queries may involve a combination of specific device IDs, measurement types, or time ranges, and query patterns may repeat (e.g., periodically checking the speed of a drone).

[0077] Therefore, the analysis of the query pattern of the query request specifically includes:

[0078] (1) Identify the time range, device ID, and measurement type of the current query. This step is to identify the current query itself in order to analyze the query pattern later. The "time range of the current query" here refers to the range of the generation times of the various time series data involved in the current query.

[0079] (2) Check the access frequency of the device ID and measurement type in the current query. This step is a pattern correlation analysis of the current query, specifically a repeated query analysis of the combination of device ID and measurement type in the current query. If the access frequency of the device ID and measurement type in the current query reaches a preset value, data with the same device ID and measurement type as the current query but with different generation time will be identified as prefetch data to be loaded.

[0080] (3) Check whether the time series data of the current query is recent data. This step is a time locality analysis of the current query. Specifically, it checks whether the time series data of the current query is recent data. If the time series data of the current query is recent data, the user may continue to query the data of the next time period that is continuous with the current query time range. Then the data of the next time period that is continuous with the current query time range can be determined as the prefetch data to be loaded.

[0081] (4) Check the correlation between the device ID and the measurement type in the current query. This step is a pattern correlation analysis of the current query, specifically a correlation analysis between the device ID and the measurement type in the current query. By analyzing whether there is a correlation between the query history of the device ID and the measurement type in the current query, if there is a correlation between the query history of the device ID in the current query, then data with the same device ID but different measurement type as the current query is identified as prefetch data to be loaded; if there is a correlation between the query history of the measurement type in the current query, then data with the same measurement type but different device ID as the current query is identified as prefetch data to be loaded.

[0082] S63: Based on the analysis results, determine the prefetch data to be loaded and generate a preloading task.

[0083] When determining the prefetch data to be loaded, it can be determined based on temporal locality or pattern correlation.

[0084] The prefetch data to be loaded includes at least one of the following:

[0085] (1) Data with the same device ID and measurement type as the currently queried device, but generated at a different time;

[0086] (2) Data for the next time period that is continuous with the current query time range;

[0087] (3) Data with the same device ID as the currently queried device but a different measurement type;

[0088] (4) Data with the same measurement type as the current query but a different device ID.

[0089] S70: Execute the preloading task asynchronously, load the prefetched data and store it in the lock-free hash table, and update the hot data index structure.

[0090] like Figure 6 As shown, step S70 further includes:

[0091] S71: Check if system resources are sufficient.

[0092] S72: If system resources are sufficient, the preloading task will be executed asynchronously.

[0093] For steps S71-S72, the system resources are determined to be sufficient by judging whether the CPU utilization, memory utilization or other resource indicators exceed the threshold (e.g., <80%), so as to ensure that the preloaded task does not affect the system performance.

[0094] For example, the condition for sufficient resources is set as CPU utilization <80%. If the CPU utilization is below 80%, the CPU resources are considered sufficient, and the preloaded task is executed immediately, reserving 20% ​​margin to cope with sudden tasks (such as high-concurrency queries or writes) to avoid system overload.

[0095] For example, the condition for sufficient resources is set as memory utilization < 80%. If memory utilization is below 80%, then memory resources are considered sufficient and suitable for the execution of preloading tasks. Preloading tasks increase memory usage (e.g., lock-free hash tables and hot data index structures), and the 80% threshold ensures that there is enough space to store new data.

[0096] For example, if the resource sufficiency condition is set to CPU utilization < 80% and memory utilization < 80%, and the condition is not met (e.g., CPU utilization ≥ 80% or memory utilization ≥ 80%), then the preloaded task is delayed and pushed into a lock-free queue to wait for resource recovery.

[0097] like Figure 7 As shown, the asynchronous execution of the preloading task further includes:

[0098] S721: Generate composite keys in each time segment according to the prefetched data in batches;

[0099] S722: Based on the composite key in each time segment, load the corresponding data blocks from the time series database in batches to obtain the prefetched data;

[0100] S723: Use CAS operation to write the prefetched data in batches into a lock-free hash table;

[0101] S724: Create metadata for the prefetched data, and record the access time and access count of the prefetched data;

[0102] S725: Add the composite key of the prefetched data to the hot data index structure and update the hot data index structure.

[0103] Specifically, after adding the composite key of the prefetched data to the hot data index structure, the access count of each composite key in the hot data index structure is read from the lock-free array. Based on the read access count, the order or priority of each composite key in the hot data index structure is adjusted using an LRU or LFU strategy.

[0104] In one embodiment, step S70 further includes:

[0105] S73: Calculate the preload hit rate and adjust the hotspot threshold based on the preload hit rate.

[0106] In this step, according to the formula Calculate preload hit rate Then, the preload hit rate meets the preset conditions (such as...). When the hotspot threshold is lowered (e.g., to 0.01), the preload hit rate is lowered when the preload hit rate does not meet the preset conditions (e.g., When doing so, keep the current hotspot threshold unchanged. Adjusting the hotspot threshold can improve the cache hit rate.

[0107] S80: When the cache space is insufficient, low-priority data in the lock-free hash table is dynamically removed based on the eviction score, which is calculated based on the data access frequency and time freshness.

[0108] Dynamic eviction of low-priority data: When the cache is full, a dynamic eviction strategy based on access frequency and time freshness prioritizes retaining hot data, according to eviction scores (…). Remove low-priority data. The lower the score, the more likely the data is to be evicted; the higher the score, the more likely the data is to be retained. In the C-UAS system, It helps optimize cache hit rate, reduce access to TSDB, and improve query efficiency.

[0109] Data access frequency. Data accessed frequently (such as recent radar signals) is typically hot data and should be prioritized for caching to improve cache hit rate. For example, if a data item is accessed 100 times in one hour, and the time window is 3600 seconds, then... .

[0110] The time freshness of data reflects how close the time of data generation is to the current time. The timestamp of the current time; The timestamp represents the time the data was generated, reflecting how close the data generation time is to the current time; This is a normalization factor, typically the maximum time range (e.g., 1 day = 86400 seconds). The high hotspot characteristic of time-series data means that recent data (e.g., sensor data from the last hour) is more likely to be queried. Smaller (i.e., newer) data has higher priority; the most recent data (such as the current trajectory of a drone) is more valuable than historical data. This is used to evaluate which data should be prioritized for retention. For example, if the current time is 1625097600 seconds and the data was generated at 1625094000 seconds, ,but .

[0111] Weighted by access frequency, where access frequency is the most frequent factor. The weighting coefficients are adjusted to control their contribution to the elimination score. The importance of visit frequency in elimination decisions is also considered. The larger the value, the greater the impact of hot data, making it suitable for query-intensive scenarios. In C-UAS systems, query patterns may favor hot data (such as real-time data from specific devices), so a higher value can be set. (e.g., 0.6) to prioritize the retention of frequently accessed data.

[0112] Weighted by time freshness, and time freshness The weighting coefficients are used to control their contribution to the elimination score, thus adjusting the importance of time freshness in the elimination decision. The larger the value, the higher the priority of recent data, making it suitable for time-sensitive scenarios. In C-UAS systems, where high real-time requirements are needed (such as real-time drone monitoring), a higher value can be set. (e.g., 0.4) to prioritize retaining the latest data.

[0113] (Usually normalized) to ensure score comparability. Dynamically adjusted based on the C-UAS system's query pattern (e.g., real-time priority or hotspot priority). and By capturing the hotspot characteristics and time sensitivity of time-series data, and optimizing the cache eviction strategy, we ensure that highly popular and recent data remain in the cache, thereby improving the hit rate and query efficiency and adapting to the high concurrency requirements of the UAV C-UAS system.

[0114] The dynamic elimination mechanism (strategy) of this invention supports real-time or hotspot priority mode adjustments:

[0115] Real-time priority: increase (like ).

[0116] Hotspot Priority: Increase (like ).

[0117] Low-priority data in the lock-free hash table is dynamically evicted based on the eviction score. The data segment with the lowest score is removed, and cache space is freed up to store new data.

[0118] The high-concurrency time-series data processing method for anti-drone systems provided in this embodiment of the invention uses a lock-free hash table to cache time-series data in the cache space. The lock-free cache design and CAS atomic operation reduce lock overhead, resulting in high batch write throughput, low collision probability, and support for multi-threaded high-concurrency read and write. It completely eliminates thread contention and blocking caused by traditional caching mechanisms, thereby improving concurrent processing capabilities. At the same time, batch writing reduces the overhead of dynamic memory allocation and improves memory utilization. Through composite key fast location, priority query of hot data index structure, and lock-free reading mechanism, the query latency of hot data is extremely low. The data preloading mechanism further reduces the latency of future related queries and improves real-time response capabilities. The dynamic eviction mechanism based on eviction scores can prioritize the retention of hot / recent data, ensuring high resource utilization, significantly improving cache hit rate, greatly reducing the access frequency of TSDB, and improving the overall performance of the C-UAS system.

[0119] like Figure 8 As shown, in another embodiment provided in this application, step S30 specifically further includes steps S301 to S305:

[0120] S301: Push the received timing data into a lock-free queue to buffer high-concurrency write requests.

[0121] Upon receiving the time-series data, it is immediately pushed into a lock-free queue for high-concurrency write buffering. In this embodiment, the received time-series data contains five fields (timestamp, device ID, measurement type, accuracy level, and value).

[0122] S302: Extract time-series data in batches from the lock-free queue and segment the extracted time-series data according to the time range.

[0123] The system tracks the lock-free queue. When the amount of data in the lock-free queue reaches the batch size B (default 100), it retrieves the time-series data in batches and segments the extracted time-series data according to a time range (e.g., 1 hour). Where M is the number of segments, For the start time, This is the end time. It should be noted that the time range here refers to the range of times when each time series data point was generated.

[0124] S303: Generate a composite key for each time series data.

[0125] Based on the timestamp, device identifier, measurement type, and precision level of the time-series data, a composite key K is calculated and generated for each time-series data entry (K = hash(T | D | Q | P), where T, D, Q, and P represent the timestamp, device ID, measurement type, and precision level, respectively). The lock-free hash table stores data in key-value pairs. This step generates a composite key for each time-series data entry so that the composite key and the time-series data value can be combined to form a key-value pair, thus writing the time-series data into the lock-free hash table. When querying the lock-free hash table, the composite key is used as an index to quickly locate the corresponding time-series data.

[0126] S304: Allocate a cache segment for each time series data segment and record the metadata of each time series data segment, including access time and access count.

[0127] S305: Combine the composite key of the time series data with the value of the time series data to form key-value pairs, and use CAS atomic operations to write the time series data in batches into a lock-free hash table.

[0128] In the process of batch writing time-series data into a lock-free hash table, the composite key of the time-series data is first combined with the value of the time-series data to form a key-value pair. Then, the CAS atomic operation is used to insert the time-series data into the lock-free hash table slot in the form of key-value pairs. If there is a conflict, the process is retried. Each lock-free hash table slot supports atomic updates and allows multi-threaded concurrent read and write.

[0129] Through steps S301 to S305, the received time series data is segmented and stored according to time range, which can support efficient range query and exact match. The query time complexity of exact match is O(1), and the query time complexity of range query is O(k) (metadata filtering) + O(1) (exact match), where k is the number of matched entries.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] In one embodiment, a high-concurrency time-series data processing apparatus for an anti-drone system is provided. The modules of this apparatus correspond one-to-one with the steps of the high-concurrency time-series data processing method for an anti-drone system described in the above embodiments. For example... Figure 9 As shown, the high-concurrency time-series data processing device for anti-drone systems includes a system component creation module 10, a data receiving module 20, a data writing module 30, an index updating module 40, a data query module 50, a preload task generation module 60, a preload task execution module 70, and a cached data eviction module 80. Detailed descriptions of each functional module are as follows:

[0132] The system component creation module 10 is used to execute step S10 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to create a lock-free hash table of a specified size, a lock-free queue, a lock-free array, and a hot data index structure in the cache space.

[0133] The data receiving module 20 is used to execute step S20 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to receive high-concurrency time-series data from multiple sensors.

[0134] The data writing module 30 is used to execute step S30 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to generate a composite key for each time-series data, to form a key-value pair with the composite key of the time-series data and the value of the time-series data, and to use CAS atomic operations to write the time-series data in batches into a lock-free hash table.

[0135] The index update module 40 is used to execute step S40 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to add the composite key of the newly inserted time-series data in the lock-free hash table to the hot data index structure and update the hot data index structure.

[0136] The data query module 50 is used to execute step S50 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to respond to the query request, use the hot data index structure and the lock-free hash table to perform data retrieval and return the query results.

[0137] The preload task generation module 60 is used to execute step S60 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to analyze the query pattern of the query request, determine the prefetched data to be loaded based on time locality and pattern correlation, and generate a preload task.

[0138] The preloading task execution module 70 is used to execute step S70 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to asynchronously execute the preloading task, load the prefetched data and store it in the lock-free hash table, and update the hot data index structure.

[0139] The cached data eviction module 80 is used to execute step S80 in the high-concurrency time-series data processing method for anti-drone systems in the above embodiments, that is, to dynamically remove low-priority data in the lock-free hash table based on the eviction score when the cache space is insufficient. The eviction score is calculated based on the data access frequency and time freshness.

[0140] In one embodiment, the data writing module 30 is specifically used for:

[0141] The received time-series data is pushed into a lock-free queue to buffer high-concurrency write requests;

[0142] Extract time-series data in batches from the lock-free queue and pre-allocate memory blocks for the batch-extracted time-series data;

[0143] Generate a composite key for each time series data entry, the composite key being generated based on the timestamp, device identifier, and measurement type of the time series data;

[0144] The composite key of the time series data is combined with the value of the time series data to form key-value pairs, and the time series data is written in batches to a lock-free hash table using CAS atomic operations.

[0145] Create metadata for newly inserted time-series data in a lock-free hash table, recording the access time and access count of the time-series data.

[0146] In another embodiment provided in this application, the data writing module 30 is further specifically used for:

[0147] The received time-series data is pushed into a lock-free queue to buffer high-concurrency write requests;

[0148] Extract time-series data in batches from a lock-free queue, and segment the extracted time-series data according to time range;

[0149] Generate a composite key for each time series data point, the composite key being generated based on the timestamp, device identifier, measurement type, and accuracy level of the time series data;

[0150] Allocate a cache segment for each time series data segment and record the metadata of each time series data segment, including access time and access count;

[0151] The composite key of the time series data is combined with the value of the time series data to form key-value pairs, and the time series data is written in batches to a lock-free hash table using CAS atomic operations.

[0152] In one embodiment, the index update module 40 is specifically used for:

[0153] Add the composite key of the newly inserted time-series data to the hot data index structure in the lock-free hash table;

[0154] Read the access count of each composite key in the hot data index structure from the lock-free array, and adjust the order or priority of each composite key in the hot data index structure using the LRU or LFU strategy based on the read access count.

[0155] In one embodiment, the data query module 50 is specifically used for:

[0156] Receive query requests from C-UAS;

[0157] Based on the time range, device ID, and measurement type requested in the query request, search the hotspot data index structure for composite keys that meet the query conditions;

[0158] If a match is found, the time-series data in the corresponding slot of the lock-free hash table is read directly and without lock using the matched composite key, and returned to C-UAS;

[0159] If no match is found, the query request is used to retrieve time series data that meets the query conditions from the time series database and return it to C-UAS;

[0160] Update the metadata of the queried time series data.

[0161] In one embodiment, the preloaded task generation module 60 is specifically used for:

[0162] Based on the access frequency and access freshness of the composite key corresponding to the query request, the hotspot score of the time series data corresponding to the query request is calculated.

[0163] If the hotspot score exceeds the hotspot threshold, then analyze the query pattern of the query request;

[0164] Based on the analysis results, the prefetch data to be loaded is determined, and a preloading task is generated.

[0165] In one embodiment, the preloaded task execution module 70 is specifically used for:

[0166] Check if system resources are sufficient;

[0167] If system resources are sufficient, the preloading task will be executed asynchronously;

[0168] Calculate the preload hit rate and adjust the hotspot threshold based on the preload hit rate.

[0169] Asynchronous execution of the preloading task further includes:

[0170] Based on the various time segments of the prefetched data, generate composite keys in batches within each time segment;

[0171] Based on the composite key within each time segment, the corresponding data blocks are loaded in batches from the time series database to obtain the prefetched data;

[0172] The prefetched data is written in batches to a lock-free hash table using a CAS operation.

[0173] Metadata is created for the prefetched data to record the access time and access count of the prefetched data;

[0174] Add the composite key of the prefetched data to the hotspot data index structure and update the hotspot data index structure.

[0175] The high-concurrency time-series data processing device for anti-drone systems provided in this embodiment of the invention uses a lock-free hash table to cache time-series data in the cache space. The lock-free cache design and CAS atomic operation reduce lock overhead, resulting in high batch write throughput, low collision probability, and support for multi-threaded high-concurrency read and write. It completely eliminates thread contention and blocking caused by traditional caching mechanisms, thereby improving concurrent processing capabilities. At the same time, batch writing reduces the overhead of dynamic memory allocation and improves memory utilization. Through composite key fast location, priority query of hot data index structure, and lock-free reading mechanism, the query latency of hot data is extremely low. The data preloading mechanism further reduces the latency of future related queries and improves real-time response capabilities. The dynamic eviction mechanism based on eviction scores can prioritize the retention of hot / recent data, ensuring high resource utilization, significantly improving cache hit rate, greatly reducing the access frequency of TSDB, and improving the overall performance of the C-UAS system.

[0176] Specific limitations regarding the high-concurrency time-series data processing device for anti-drone systems can be found in the limitations of the high-concurrency time-series data processing method for anti-drone systems described above, and will not be repeated here. Each module in the aforementioned high-concurrency time-series data processing device for anti-drone systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0178] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A high-concurrency time-series data processing method for anti-drone systems, characterized in that, Includes the following steps: Receives high-concurrency time-series data from multiple sensors; Generate a composite key for each time series data, combine the composite key of the time series data with the value of the time series data to form a key-value pair, and use CAS atomic operation to write the time series data in batches into a lock-free hash table; Add the composite key of the newly inserted time-series data in the lock-free hash table to the hot data index structure, and update the hot data index structure. In response to a query request, the system uses the hotspot data index structure and the lock-free hash table to retrieve data and return the query results. Analyze the query pattern of the query request, determine the prefetched data to be loaded based on temporal locality and pattern correlation, and generate a preloading task; The preloading task is executed asynchronously to load the prefetched data and store it in the lock-free hash table, and to update the hot data index structure. When the cache space is insufficient, low-priority data in the lock-free hash table is dynamically removed based on the eviction score, which is calculated based on the data access frequency and freshness.

2. The high-concurrency time-series data processing method for anti-drone systems as described in claim 1, characterized in that, The process of generating a composite key for each piece of time-series data, forming a key-value pair by combining the composite key and the value of the time-series data, and using CAS atomic operations to write the time-series data in batches into a lock-free hash table includes the following steps: The received time-series data is pushed into a lock-free queue to buffer high-concurrency write requests; Extract time-series data in batches from the lock-free queue and pre-allocate memory blocks for the batch-extracted time-series data; Generate a composite key for each time series data entry, the composite key being generated based on the timestamp, device identifier, and measurement type of the time series data; The composite key of the time series data is combined with the value of the time series data to form key-value pairs, and the time series data is written in batches to a lock-free hash table using CAS atomic operations. Create metadata for newly inserted time-series data in a lock-free hash table, recording the access time and access count of the time-series data.

3. The high-concurrency time-series data processing method for anti-drone systems as described in claim 1, characterized in that, The step of adding the composite key of newly inserted time-series data into the hot data index structure and updating the hot data index structure includes the following steps: Add the composite key of the newly inserted time-series data to the hot data index structure in the lock-free hash table; Read the access count of each composite key in the hot data index structure from the lock-free array, and adjust the order or priority of each composite key in the hot data index structure using the LRU or LFU strategy based on the read access count.

4. The high-concurrency time-series data processing method for anti-drone systems as described in claim 1, characterized in that, The process of responding to a query request by retrieving data using the hotspot data index structure and the lock-free hash table and returning the query results includes the following steps: Receive query requests from C-UAS; Based on the time range, device ID, and measurement type requested in the query request, search the hotspot data index structure for composite keys that meet the query conditions; If a match is found, the time-series data in the corresponding slot of the lock-free hash table is read directly and without lock using the matched composite key, and returned to C-UAS; If no match is found, the query request is used to retrieve time series data that meets the query conditions from the time series database and return it to C-UAS; Update the metadata of the queried time series data.

5. The high-concurrency time-series data processing method for anti-drone systems as described in claim 1, characterized in that, The process of analyzing the query pattern of the query request, determining the prefetched data to be loaded based on temporal locality and pattern correlation, and generating a preloading task includes the following steps: Based on the access frequency and access freshness of the composite key corresponding to the query request, the hotspot score of the time series data corresponding to the query request is calculated. If the hotspot score exceeds the hotspot threshold, then analyze the query pattern of the query request; Based on the analysis results, the prefetch data to be loaded is determined, and a preloading task is generated.

6. The high-concurrency time-series data processing method for anti-drone systems as described in claim 5, characterized in that, The analysis of the query pattern of the query request further includes the following steps: Identify the time range, device ID, and measurement type of the current query; Check the access frequency of the currently queried device ID and measurement type; Check if the time-series data being queried is recent data; Check the device ID relevance and measurement type relevance of the current query.

7. The high-concurrency time-series data processing method for anti-drone systems as described in claim 6, characterized in that, The prefetched data includes at least one of the following: Data with the same device ID and measurement type as the currently queried device, but generated at a different time; Data for the next consecutive time period following the current query time range; Data with the same device ID as the currently queried device but a different measurement type; Data with the same measurement type as the current query but a different device ID.

8. The high-concurrency time-series data processing method for anti-drone systems as described in claim 1, characterized in that, The asynchronous execution of the preloading task, loading the prefetched data and storing it in the lock-free hash table, and updating the hot data index structure includes the following steps: Check if system resources are sufficient; If system resources are sufficient, the preloading task will be executed asynchronously; The asynchronous execution of the preloading task further includes: Based on the various time segments of the prefetched data, generate composite keys in batches within each time segment; Based on the composite key within each time segment, the corresponding data blocks are loaded in batches from the time series database to obtain the prefetched data; The prefetched data is written in batches to a lock-free hash table using a CAS operation. Metadata is created for the prefetched data to record the access time and access count of the prefetched data; Add the composite key of the prefetched data to the hotspot data index structure and update the hotspot data index structure.

9. The high-concurrency time-series data processing method for anti-drone systems as described in any one of claims 1-8, characterized in that, The process of generating a composite key for each piece of time-series data, forming a key-value pair by combining the composite key and the value of the time-series data, and using CAS atomic operations to write the time-series data in batches into a lock-free hash table includes the following steps: The received time-series data is pushed into a lock-free queue to buffer high-concurrency write requests; Extract time-series data in batches from a lock-free queue, and segment the extracted time-series data according to time range; Generate a composite key for each time series data point, the composite key being generated based on the timestamp, device identifier, measurement type, and accuracy level of the time series data; Allocate a cache segment for each time series data segment and record the metadata of each time series data segment, including access time and access count; The composite key of the time series data is combined with the value of the time series data to form key-value pairs, and the time series data is written in batches to a lock-free hash table using CAS atomic operations.

10. A high-concurrency time-series data processing device for anti-drone systems, characterized in that, include: The data receiving module is used to receive high-concurrency time-series data from multiple sensors; The data writing module is used to generate a composite key for each time series data, combine the composite key of the time series data with the value of the time series data to form a key-value pair, and use CAS atomic operation to write the time series data in batches into a lock-free hash table; The index update module is used to add the composite key of newly inserted time-series data into the hot data index structure of the lock-free hash table, and update the hot data index structure. The data query module is used to respond to query requests, perform data retrieval using the hot data index structure and the lock-free hash table, and return query results. The preloading task generation module is used to analyze the query pattern of the query request, determine the prefetched data to be loaded based on temporal locality and pattern correlation, and generate a preloading task. The preloading task execution module is used to asynchronously execute the preloading task, load the prefetched data and store it in the lock-free hash table and update the hot data index structure. The cached data eviction module is used to dynamically remove low-priority data from the lock-free hash table based on an eviction score when the cache space is insufficient. The eviction score is calculated based on the data access frequency and freshness.

Images