Kernel data collecting and processing method and device based on multi-layer cache optimization

Abstract

The invention relates to a kernel data collecting and processing method and device based on multilayer cache optimization, and the method comprises the steps: collecting kernel data from an operating system kernel, and carrying out the deduplication processing of the kernel data; controlling the switching frequency of AB double caches by adopting a time wheel, writing and reading the de-duplicated kernel data through a double-cache mechanism, performing batch processing on the de-duplicated kernel data, and merging to obtain operation data; performing batch processing queue classification and sorting on the operation data through the prefix tree to obtain a sorting result; according to a sorting result, writing the operation data into an annular buffer area with a corresponding priority in the shared memory; and on the basis of a set condition, target data are retrieved in the annular buffer area by adopting a skip list and are sorted, and the target data are exported to a corresponding process according to the service priority. According to the invention, the collection speed and the export speed of the kernel data can be matched, and high-performance and low-disturbance real-time operation data collection and processing are realized while multiple priorities are ensured.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of kernel data processing, and particularly relates to a kernel data collection and processing method and device based on multi-layer cache optimization, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In related technologies, kernel data collection and processing technology is a technology system that provides accurate data support for system performance optimization, security audit and fault diagnosis by using diversified means such as eBPF, performance counters, kernel trace points, to obtain and analyze runtime information such as memory allocation, access and leakage in real time. However, the overhead generated by the data collection behavior itself will significantly interfere with the real state of the observed system, and it is difficult to achieve both high-precision collection and low-performance overhead.

[0003] Therefore, there is an urgent need for a kernel data collection and processing method that can match the collection and export speed of kernel data, and realize real-time running data collection and processing with high performance and low disturbance. SUMMARY

[0004] To overcome the problems in related technologies, the present disclosure provides a kernel data collection and processing method and device based on multi-layer cache optimization, an electronic device and a computer readable storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a kernel data collection and processing method based on multi-layer cache optimization is provided, including: collecting kernel data from an operating system kernel, and performing deduplication processing on the kernel data; controlling the switching frequency of AB double buffers using a time wheel, writing and reading the deduplicated kernel data through a double buffer mechanism, and performing batch processing on the deduplicated kernel data to obtain operation data by merging; classifying and sorting the operation data through a prefix tree to obtain a sorting result; writing the operation data into a corresponding priority ring buffer in shared memory according to the sorting result; based on a set condition, searching for target data in the ring buffer using a skip list and sorting the target data, and exporting the target data to a corresponding process according to the business priority.

[0006] In some embodiments, the collecting kernel data from an operating system kernel and performing deduplication processing on the kernel data includes: collecting kernel data from an operating system kernel layer; performing deduplication on the collected kernel data using a Bloom filter, wherein if the Bloom filter returns all 1s, it indicates that the corresponding kernel data is duplicated, and the kernel data is discarded; if the Bloom filter returns 0, it indicates that the corresponding kernel data does not exist, and the kernel data is retained.

[0007] In some embodiments, the step of collecting kernel data from the operating system kernel and performing deduplication on the kernel data further includes: collecting kernel data from the operating system kernel level; converting the kernel data into a fixed-length hash value using a hash function; and performing deduplication by comparing the hash value of the key with the content using a hash table. Specifically, when inserting kernel data, if data already exists at the position corresponding to the hash value of the kernel data, the key content is compared. If the key content is the same, the corresponding kernel data is considered to exist repeatedly, and the kernel data is discarded; otherwise, the kernel data is retained.

[0008] In some embodiments, the use of a time wheel to control the switching frequency of the AB dual cache, writing and reading deduplicated kernel data through the dual cache mechanism, and batch processing the deduplicated kernel data to obtain operation data includes: dividing CPU time into fixed-length time slices based on a time wheel algorithm; setting one cache to a data writing state to write deduplicated kernel data according to the control of the time wheel, and setting the other cache to a data reading state to read the kernel data written in the cache; when the time slice is exhausted, triggering a cache pointer swap through a timer interrupt to adjust the state of the two caches; and batch processing the read kernel data through a batch processing queue to obtain operation data.

[0009] In some embodiments, the step of classifying and sorting the operation data in the batch processing queue using a prefix tree to obtain a sorting result includes: using a prefix tree to store the prefix relationship of keys in a tree structure, wherein the root node of the tree structure does not contain a character, each child node represents a character, and the characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node, and all child nodes of each node contain different characters, the prefix relationship includes IP address, URL path and log source; traversing and matching character by character from the root node to classify and sort the operation data in the batch processing queue to obtain a sorting result.

[0010] In some embodiments, writing the operation data into a circular buffer with a corresponding priority in shared memory according to the sorting result includes: determining the priority of the operation data according to the sorting result of the operation data; and writing the operation data into the corresponding circular buffer based on the priority of the operation data. Multiple circular buffers are provided, located in shared memory, and each circular buffer corresponds to a priority.

[0011] In some embodiments, the step of retrieving and sorting target data in the circular buffer using a skip list based on set conditions, and exporting the target data to the corresponding process according to business priority, includes: obtaining preset set conditions; constructing a multi-level index using a skip list based on the set conditions; retrieving the target data in the circular buffer; sorting the target data according to priority to obtain a priority queue; and exporting the target data to the corresponding process according to business priority.

[0012] According to a second aspect of the present disclosure, a kernel data acquisition and processing apparatus based on multi-layer cache optimization is provided, comprising: a kernel data deduplication module, used to acquire kernel data from the operating system kernel and perform deduplication processing on the kernel data; a kernel data dual-caching module, used to control the switching frequency of AB dual cache using a time wheel, write and read the deduplicated kernel data through a dual-caching mechanism, and perform batch processing on the deduplicated kernel data to merge it into operation data; an operation data classification module, used to classify and sort the operation data in a batch processing queue using a prefix tree; an operation data writing module, used to write the operation data into a circular buffer of corresponding priority located in shared memory according to the sorting result; and a target data retrieval module, used to retrieve target data from the circular buffer based on set conditions using a skip list and sort it, and export the target data to the corresponding process according to business priority.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the kernel data acquisition and processing method based on multi-layer cache optimization provided in the first aspect of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the kernel data acquisition and processing method based on multi-layer cache optimization provided in the first aspect of the present disclosure.

[0015] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: Kernel data is collected from the operating system kernel, and deduplication is performed on the kernel data to remove duplicate data, reducing the pressure on downstream processing units and improving overall processing efficiency; a time wheel is used to control the switching frequency of the AB dual buffer, and the deduplicated kernel data is written and read through the dual buffer mechanism, and the deduplicated kernel data is batch-processed and merged to obtain operation data. The dual buffer mechanism eliminates the waiting time between producers and consumers and avoids data conflicts; a prefix tree is used to classify and sort the operation data in the batch processing queue to obtain sorting results, reducing time and space overhead; according to the sorting results, the operation data is written to the corresponding priority circular buffer in shared memory to achieve efficient resource allocation and improve data transmission efficiency; based on the process setting conditions, the target data is retrieved from the circular buffer through a skip list and sorted, and the target data is exported to the corresponding process according to the business priority, ensuring multi-priority processing while improving system throughput and efficiency. This enables high-performance, low-disturbance real-time running status monitoring and data acquisition, matching the kernel data acquisition and export speeds, avoiding data loss and process starvation, and achieving high-precision acquisition of kernel data with low performance overhead.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a flowchart illustrating a kernel data acquisition and processing method based on multi-level cache optimization, according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating a kernel data deduplication method according to an exemplary embodiment.

[0020] Figure 3 This is a schematic diagram of the structure of a Bloom filter according to an exemplary embodiment.

[0021] Figure 4 This is a flowchart illustrating a kernel data deduplication method according to an exemplary embodiment.

[0022] Figure 5 This is a flowchart illustrating a kernel data batch processing method according to an exemplary embodiment.

[0023] Figure 6 This is a schematic diagram of the structure of a time wheel according to an exemplary embodiment.

[0024] Figure 7 This is a flowchart illustrating a batch processing queue classification and sorting method according to an exemplary embodiment.

[0025] Figure 8 This is a schematic diagram of the prefix tree structure according to an exemplary embodiment.

[0026] Figure 9 This is a flowchart illustrating an operational data writing method according to an exemplary embodiment.

[0027] Figure 10 This is a schematic diagram of the structure of a circular buffer according to an exemplary embodiment.

[0028] Figure 11 This is a flowchart illustrating a target data retrieval method according to an exemplary embodiment.

[0029] Figure 12 This is a schematic diagram of the structure of a skip table according to an exemplary embodiment.

[0030] Figure 13 This is an overall flowchart illustrating a kernel data acquisition and processing method based on multi-layer cache optimization, according to an exemplary embodiment.

[0031] Figure 14 This is a block diagram of a kernel data acquisition and processing device based on multi-level cache optimization, according to an exemplary embodiment.

[0032] Figure 15 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0033] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0034] It should be noted that the relevant embodiments and accompanying drawings are only for describing and illustrating exemplary embodiments provided by this disclosure, and not all embodiments of this disclosure, nor should this disclosure be understood to be limited to the relevant exemplary embodiments.

[0035] It should be noted that the terms "first," "second," etc., used in this disclosure are only used to distinguish different steps, devices, or modules. These terms do not represent any specific technical meaning, nor do they indicate any order or interdependence between them.

[0036] It should be noted that the terms “a,” “a plurality of,” and “at least one” used in this disclosure are illustrative rather than restrictive. Unless otherwise expressly indicated in the context, they should be understood as “one or more.”

[0037] It should be noted that the term "and / or" used in this disclosure is used to describe the relationship between related objects, and generally indicates that there are at least three relationships. For example, A and / or B can at least indicate: the existence of A alone, the existence of both A and B, and the existence of B alone.

[0038] It should be noted that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Unless otherwise specified, the scope of this disclosure is not limited by the order in which the steps are described in the relevant embodiments.

[0039] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0040] Exemplary method

[0041] Figure 1 This is a flowchart illustrating a kernel data acquisition and processing method based on multi-level cache optimization, according to an exemplary embodiment. Figure 1 As shown, the kernel data acquisition and processing method based on multi-level caching optimization is used in the kernel data acquisition and processing process, including the following steps:

[0042] In step S110, kernel data is collected from the operating system kernel and deduplication is performed on the kernel data.

[0043] In some embodiments, kernel data can be collected from the operating system kernel level by means of network packet capture, character data reading, hardware-level data capture, or function signal processing. The collected kernel data is then deduplicated to remove duplicate data, reduce the pressure on downstream processing units, and improve subsequent processing efficiency.

[0044] In step S120, a time wheel is used to control the switching frequency of the AB dual buffer. The deduplicated kernel data is written and read through the dual buffer mechanism, and the deduplicated kernel data is batch processed and merged to obtain the operation data.

[0045] In some embodiments, the deduplicated kernel data is passed into the AB dual buffer, and the switching frequency of the AB dual buffer is controlled by a time wheel, for example, once every 100 milliseconds. The data writing and data reading status of the two buffers, Buffer A and Buffer B, are controlled. The deduplicated kernel data is written and read through the dual buffer mechanism, and the fragmented kernel data after deduplication is batch processed and merged to obtain the operation data, which significantly reduces the number of system calls and context switches.

[0046] In step S130, the batch processing queue of the operation data is classified and sorted using a prefix tree to obtain the sorting result.

[0047] In some embodiments, a prefix tree is used to classify and sort the batch processing queue of the operation data to obtain a sorting result, so as to facilitate the writing of operation data according to the sorting result. Each node in the prefix tree stores a character and shares paths through common prefixes, which is suitable for fast string retrieval and can reduce time and space overhead by utilizing common prefixes between strings.

[0048] In addition, radix trees can be used to classify and sort the batch processing queue of operational data, and the number of nodes can be reduced by merging consecutive identical characters, thereby further compressing space.

[0049] In step S140, the operation data is written into the circular buffer of the corresponding priority located in the shared memory according to the sorting result.

[0050] In some embodiments, based on the sorting result of the prefix tree, the operation data is sequentially written into the circular buffers of corresponding priorities located in the shared memory. Different types of data are written into circular buffers of different priorities, enabling the circular buffers to support priority scheduling. High-priority buffers are consumed first, achieving efficient resource allocation and timely processing of latency-sensitive tasks. Furthermore, the circular buffers reduce data movement and memory management overhead by cyclically using a fixed-size memory region, thereby improving data transmission efficiency.

[0051] Meanwhile, multiple circular buffers are located in shared memory, enabling producer and consumer processes to share data efficiently with zero copying, avoiding the data copying overhead of inter-process communication.

[0052] In step S150, based on the set conditions, the target data is retrieved from the circular buffer using a skip list and sorted. The target data is then exported to the corresponding process according to the business priority.

[0053] In some embodiments, target data is retrieved from shared memory using a skip list in a circular buffer according to the set conditions required by the process, and the target data is sorted to support subsequent priority sorting.

[0054] Data processed by the skip list is sent to a priority queue to ensure that data is sorted according to business priority (not just data flow priority). The target data is then exported to the corresponding process, enabling the collaboration and pipelined implementation of multiple efficient data structures and end-to-end zero-copy processing. This improves system throughput and efficiency while ensuring multi-priority processing. For example, error logs are processed before message logs, and VIP user requests are prioritized over regular user requests.

[0055] In the above embodiments, kernel data is collected from the operating system kernel and deduplicated to remove duplicate data, reducing the pressure on downstream processing units and improving overall processing efficiency. A time wheel is used to control the switching frequency of the AB dual buffer. The deduplicated kernel data is written and read through the dual buffer mechanism, and the deduplicated kernel data is batch-processed and merged to obtain operation data. The dual buffer mechanism eliminates the waiting time between producers and consumers and avoids data conflicts. A prefix tree is used to classify and sort the batch processing queue of operation data to obtain sorting results, reducing time and space overhead. According to the sorting results, the operation data is written to the circular buffer of the corresponding priority in shared memory to achieve efficient resource allocation and improve data transmission efficiency. Based on the process setting conditions, the target data is retrieved from the circular buffer through a skip list and sorted. The target data is exported to the corresponding process according to the business priority, ensuring multi-priority processing while improving system throughput and efficiency. It can achieve high-performance, low-disturbance real-time running status monitoring and data acquisition, making the kernel data acquisition and export speed match, avoiding data loss and process starvation, and achieving high-precision acquisition of kernel data with low performance overhead.

[0056] Figure 2 This is a flowchart illustrating a kernel data deduplication method according to an exemplary embodiment, such as... Figure 2 As shown, the kernel data deduplication method is used in step S110, and includes the following steps:

[0057] In step S210, kernel data is collected from the operating system kernel level.

[0058] Specifically, when collecting kernel data, kernel data is collected directly from the operating system kernel level. For example, the Netfilter framework or raw sockets are used to directly process network packets for kernel data collection, or custom signal handling functions are called to collect real-time kernel data, or the terminal descriptor table or I / O APIC table is modified to jump to a custom function and read kernel data directly from the hardware port.

[0059] In step S220, a Bloom filter is used to deduplicate the acquired kernel data.

[0060] Specifically, if the Bloom filter returns all 1s, it means that the corresponding kernel data exists repeatedly, and the kernel data is discarded; if the Bloom filter returns 0, it means that the corresponding kernel data does not exist, and the kernel data is retained.

[0061] Specifically, the structure of a Bloom filter is as follows: Figure 3As shown, it is used to quickly determine whether a data item may already exist or definitely does not exist in the set, and to deduplicate the collected kernel data, thereby filtering out duplicate data with minimal memory overhead.

[0062] When performing data deduplication, if the Bloom filter returns all 1s, it is determined that the corresponding kernel data already exists, and the kernel data is discarded without proceeding to the next step; if the Bloom filter returns any 0s, it is determined that the corresponding kernel data does not exist, and the kernel data is retained for subsequent data processing.

[0063] In this embodiment, kernel data is collected from the operating system kernel level. The collected kernel data is deduplicated using a Bloom filter. When all Bloom filter return values ​​are 1, it indicates that the corresponding kernel data exists repeatedly, and the kernel data is discarded. Conversely, when the Bloom filter return value is 0, it indicates that the corresponding kernel data does not exist, and the kernel data is retained. This process can filter out most of the duplicate data, greatly reducing the pressure on downstream processing units and improving overall efficiency.

[0064] Figure 4 This is a flowchart illustrating a kernel data deduplication method according to an exemplary embodiment, such as... Figure 4 As shown, the kernel data deduplication method is used in step S110, and includes the following steps:

[0065] In step S310, kernel data is collected from the operating system kernel level.

[0066] Specifically, when collecting kernel data, kernel data is collected directly from the operating system kernel level. For example, the Netfilter framework or raw sockets are used to directly process network packets for kernel data collection, or custom signal handling functions are called to collect real-time kernel data, or the terminal descriptor table or I / O APIC table is modified to jump to a custom function and read kernel data directly from the hardware port.

[0067] In step S320, kernel data is converted into fixed-length hash values ​​using a hash function, and deduplication is performed by comparing the hash value of the key with the content using a hash table.

[0068] Specifically, when inserting kernel data, if data already exists at the position corresponding to the hash value of the kernel data, the key content is compared. If the key content is the same, it is determined that the corresponding kernel data exists repeatedly and the kernel data is discarded; otherwise, the kernel data is retained.

[0069] Specifically, after collecting kernel data, a hash table can be used for precise deduplication. The kernel data is converted into a fixed-length hash value using a hash function. The hash table compares the hash value of the key with the content to achieve deduplication. When inserting new kernel data, if data already exists at the position corresponding to the calculated hash value, the equals function is called to compare the key content. If the key content is the same, the corresponding kernel data is considered to exist repeatedly, and the kernel data is discarded or covered up. Otherwise, the kernel data is retained, thereby achieving precise deduplication. This is suitable for scenarios with high requirements for data integrity.

[0070] In this embodiment, kernel data is collected from the operating system kernel level. The kernel data is converted into a fixed-length hash value using a hash function. A hash table is used to compare the hash value of the key with the content for deduplication. When inserting kernel data, if data already exists at the position corresponding to the hash value of the kernel data, the key content is compared. If the key content is the same, the corresponding kernel data is considered to exist repeatedly and is discarded. Otherwise, the kernel data is retained, which improves the deduplication accuracy and computational efficiency. This method is suitable for scenarios with high requirements for data integrity.

[0071] Figure 5 This is a flowchart illustrating a kernel data batch processing method according to an exemplary embodiment, such as... Figure 5 As shown, the kernel data batch processing method is used in step S120, which includes the following steps:

[0072] In step S410, the CPU time is divided into time slices of fixed length based on the time wheel algorithm. According to the control of the time wheel, one cache is set to the data writing state to write the deduplicated kernel data, and the other cache is set to the data reading state to read the kernel data written in the cache.

[0073] Specifically, the time wheel algorithm is used to divide CPU time into time slices of fixed length. If the time slice length is too short, it will increase the context switching overhead, and if it is too long, it will reduce the real-time performance. The time slice length can be adjusted accordingly based on the system response requirements.

[0074] The structure of the time wheel is as follows Figure 6 As shown, under the control of the time wheel, the roles of the two buffers are switched. One buffer is set to the data writing state to write deduplicated kernel data, while the other buffer is set to the data reading state to read the kernel data written in the buffer. The two buffers run in parallel and the data is synchronized in real time. By combining the batch task processing capability of the time wheel with the asynchronous characteristics of the dual buffers, the response latency of the real-time system can be significantly reduced, and the coordination efficiency of timed tasks and data exchange can be improved.

[0075] In step S420, when the time slice expires, a timer interrupt triggers a buffer pointer swap to adjust the state of the two buffers.

[0076] Specifically, when the time slice of the time wheel expires, a timer interrupt is triggered to swap the pointers corresponding to the buffers, adjusting the states of the two buffers. The buffer in the write state is switched to the read state, and the buffer in the read state is switched to the write state, thereby achieving the switching of their roles.

[0077] In step S430, the read kernel data is batch-processed through a batch processing queue and merged to obtain operation data.

[0078] Specifically, the kernel data read from the circular buffer is batch-processed through a batch processing queue to obtain operation data. The timed batch processing strategy can merge scattered I / O kernel data to obtain operation data for larger sequential operations, significantly reducing the number of system calls and context switches.

[0079] In this embodiment, CPU time is divided into fixed-length time slices based on the time wheel algorithm. According to the control of the time wheel, one cache is set to the data writing state to write deduplicated kernel data, and the other cache is set to the data reading state to read the kernel data written in the cache. When the time slice is exhausted, the cache pointer is swapped through a timer interrupt to adjust the state of the two caches and realize the role switching between the two caches. A batch processing queue is used to batch process the read kernel data and merge it to obtain the operation data, thereby significantly reducing the number of system calls and context switches and improving data processing efficiency.

[0080] Figure 7 This is a flowchart illustrating a batch processing queue classification and sorting method according to an exemplary embodiment, such as... Figure 7 As shown, the batch processing queue classification and sorting method is used in step S130, and includes the following steps:

[0081] In step S510, a prefix tree is used to store the prefix relationship of the key through a tree structure. The root node of the tree structure does not contain a character, and each child node represents a character. The characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node. All child nodes of each node contain different characters.

[0082] The prefix relationship includes IP address, URL path, and log source.

[0083] Specifically, after batch processing, a prefix tree is used to classify the operation data. The structure of the prefix tree is as follows: Figure 8 As shown, a tree-like data structure for storing and retrieving string keys can efficiently classify and route data based on key prefix relationships, such as IP addresses, URL paths, and log sources.

[0084] Prefix trees store the prefix relationships of keys in a tree structure. The root node of the tree structure does not contain a character, and each child node represents a character. The characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node. All child nodes of each node contain different characters, which enables fast retrieval of the data through strings, which is beneficial for subsequent classification and sorting.

[0085] In step S520, the operation data in the batch processing queue is classified and sorted by character by character from the root node to obtain the sorting result.

[0086] Specifically, when classifying and sorting operation data using a prefix tree, it is necessary to traverse and match character by character from the root node to classify and sort the operation data in the batch processing queue. During sorting, the data can be sorted according to the priority of the data stream to obtain the sorting result. This allows the operation data to be stored in shared memory in sequence according to the sorting result, ensuring that high-priority requests are processed first and improving the overall system efficiency.

[0087] In this embodiment, a prefix tree is used to store the prefix relationship of keys through a tree structure, including IP address, URL path and log source. The root node of the prefix tree structure does not contain a character, and each child node represents a character. The characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node. All child nodes of each node contain different characters. The operation data in the batch processing queue is traversed and matched character by character from the root node to classify and sort the data, and obtain the sorting result to ensure that high-priority requests are processed first, thereby improving the overall system efficiency.

[0088] Figure 9 This is a flowchart illustrating an operational data writing method according to an exemplary embodiment, such as... Figure 9 As shown, the operation data writing method is used in step S140, and includes the following steps:

[0089] In step S610, the priority of the operation data is determined based on the sorting result of the operation data.

[0090] Specifically, since the operation data in the batch processing queue is sorted according to priority, the priority of the operation data can be obtained based on the sorting result, so that it can be written in the corresponding order according to the priority.

[0091] In step S620, the operation data is written into the corresponding circular buffer based on the priority of the operation data.

[0092] There are multiple circular buffers, which are located in shared memory, and each circular buffer has a corresponding priority.

[0093] Specifically, based on the priority of the obtained operation data, the operation data is written into circular buffers of different priorities. The circular buffers are shared memory, thereby enabling efficient data transmission and global resource access, resulting in flexible process relationships.

[0094] Shared memory is implemented using a circular buffer. The structure of the circular buffer is as follows: Figure 10 As shown, the circular queue of the circular buffer is connected end to end, which is conducive to achieving lock-free or low-lock concurrency, thereby providing extreme performance data exchange in shared memory. Multiple circular buffers are set up, each corresponding to a priority, so as to store operation data of different priorities.

[0095] In this embodiment, the priority of the operation data is determined according to the sorting result of the operation data, and the operation data is written into the corresponding priority ring buffer according to the priority level. Multiple ring buffers are set according to the priority level. Multiple ring buffers are located in shared memory, and each ring buffer corresponds to a priority, thereby enabling efficient data transmission and global resource access, and obtaining flexible process relationships.

[0096] Figure 11 This is a flowchart illustrating a target data retrieval method according to an exemplary embodiment, such as... Figure 11 As shown, the target data retrieval method is used in step S150, which includes the following steps:

[0097] In step S710, preset conditions are obtained, and based on these conditions, a multi-level index is constructed using a skip list. The target data is then retrieved by searching within a circular buffer.

[0098] Specifically, based on the corresponding set conditions obtained from the process, a multi-level indexed linked list is constructed using a skip list. Each node at each level contains a pointer to the next node at the same level and a down pointer to the corresponding node at the next lower level. The structure of the skip list is as follows: Figure 12 As shown, the target data is retrieved by searching in a circular buffer, realizing fast retrieval in shared memory, and supporting search, insertion and deletion operations with an average complexity of O(log n), making it suitable for range queries and retrieval of ordered data.

[0099] In step S720, the target data is sorted according to priority to obtain a priority queue, and the target data is exported to the corresponding process based on the business priority.

[0100] Specifically, the target data is sorted according to priority to obtain a priority queue. The target data is then exported to the corresponding process according to business priority, ensuring that the data is sorted and output according to priority. This enables the accurate capture of key performance indicators and anomalies in massive amounts of time, providing reliable data support for system bottleneck analysis and fault diagnosis, improving system throughput, and maintaining millimeter-level processing latency.

[0101] In this embodiment, by obtaining the set conditions of the process, a multi-level index is constructed based on the set conditions through a skip list, and the target data is obtained by searching in the circular buffer. The target data is sorted according to priority to obtain a priority queue, and the target data is exported to the corresponding process based on the business priority. This enables the accurate capture of key performance indicators and anomalies in massive amounts of time, improves system throughput, and maintains millimeter-level processing latency.

[0102] like Figure 13 As shown, based on the above embodiments, the overall operation flowchart of the kernel data acquisition and processing method based on multi-level caching optimization is obtained. In the Linux kernel, kernel data is acquired and filtered using a Bloom filter. The data exchange frequency of the AB caching mechanism is controlled by a time-wheel scheduler. The dual-caching mechanism smooths data flow, enabling synchronous kernel data writing and reading. A batch processing queue is used to read kernel data and merge it to obtain operation data. A prefix tree or radix tree classifier is used to efficiently classify and sort the operation data. Based on the sorting results, the operation data is written to shared memory and arranged according to priority. A skip list is used to retrieve the target data in shared memory. The target data is then sorted according to priority to obtain a priority queue. Based on business priority, the target data is exported to the user's dynamic process, thus realizing the kernel data acquisition and processing process. This method can be used for high-performance, low-disturbance real-time monitoring and data acquisition of large multi-process systems. Through multi-level buffering, deduplication, classification, and priority scheduling mechanisms, it ensures accurate capture of key performance indicators and anomalies in massive events, providing reliable data support for system bottleneck analysis and fault diagnosis.

[0103] Exemplary apparatus

[0104] Figure 14 This is a block diagram of a kernel data acquisition and processing apparatus based on multi-level caching optimization, according to an exemplary embodiment. (Refer to...) Figure 14 The device 800 includes a kernel data deduplication module 810, a kernel data dual cache module 820, an operation data classification module 830, an operation data writing module 840, and a target data retrieval module 850.

[0105] The kernel data deduplication module 810 is used to collect kernel data from the operating system kernel and perform deduplication processing on the kernel data.

[0106] The kernel data dual-buffer module 820 is used to control the switching frequency of the AB dual buffer using a time wheel, write and read deduplicated kernel data through the dual-buffer mechanism, and perform batch processing on the deduplicated kernel data to obtain operation data.

[0107] The operation data classification module 830 is used to classify and sort the operation data in the batch processing queue using a prefix tree;

[0108] The operation data writing module 840 is used to write operation data into a circular buffer of corresponding priority located in shared memory according to the sorting result;

[0109] The target data retrieval module 850 is used to retrieve target data from a circular buffer using a skip list based on set conditions, sort the data, and export the target data to the corresponding process according to business priority.

[0110] In one embodiment, the device further includes: a kernel data acquisition module for acquiring kernel data from the operating system kernel level; and a first deduplication module for deduplicating the acquired kernel data using a Bloom filter, wherein if all Bloom filter return values ​​are 1, it indicates that the corresponding kernel data exists repeatedly, and the kernel data is discarded; if the Bloom filter return value is 0, it indicates that the corresponding kernel data does not exist, and the kernel data is retained.

[0111] In one embodiment, the device further includes: a kernel data acquisition module for acquiring kernel data from the operating system kernel level; and a second deduplication module for converting the kernel data into a fixed-length hash value using a hash function, and performing deduplication by comparing the hash value of the key with the content using a hash table. Specifically, when inserting kernel data, if data already exists at the position corresponding to the hash value of the kernel data, the key content is compared. If the key content is the same, the corresponding kernel data is considered to exist repeatedly and is discarded; otherwise, the kernel data is retained.

[0112] In one embodiment, the device further includes: a time wheel control module, used to divide CPU time into fixed-length time slices based on a time wheel algorithm, and according to the control of the time wheel, set one cache to a data write state to write deduplicated kernel data, and set another cache to a data read state to read the kernel data written in the cache; a cache state adjustment module, used to trigger cache pointer swapping through a timer interrupt when the time slice is exhausted, and adjust the state of the two caches; and an operation data merging module, used to batch process the read kernel data through a batch processing queue and merge them to obtain operation data.

[0113] In one embodiment, the device further includes: a prefix relationship storage module, used to store the prefix relationship of keys in a tree structure using a prefix tree, wherein the root node of the tree structure does not contain a character, each child node represents a character, and the characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node, and all child nodes of each node contain different characters, and the prefix relationship includes IP address, URL path and log source; and a character traversal module, used to traverse and match characters one by one from the root node, classify and sort the operation data in the batch processing queue, and obtain the sorting result.

[0114] In one embodiment, the apparatus further includes: a priority determination module, configured to determine the priority of the operation data based on the sorting result of the operation data; and a data writing module, configured to write the operation data into a corresponding circular buffer based on the priority of the operation data, wherein multiple circular buffers are configured, the multiple circular buffers are located in shared memory, and each circular buffer corresponds to a priority.

[0115] In one embodiment, the device further includes: a retrieval module, configured to obtain preset conditions, construct a multi-level index through a skip list based on the conditions, and perform retrieval in a circular buffer to obtain target data; and a target data export module, configured to sort the target data according to priority to obtain a priority queue, and export the target data to the corresponding process based on business priority.

[0116] In the above embodiments, the operation of the device realizes a kernel data acquisition and processing method based on multi-layer cache optimization, ensuring that the device can implement all the contents of the kernel data acquisition and processing method based on multi-layer cache optimization.

[0117] Exemplary electronic device

[0118] Figure 15 This is a block diagram illustrating an electronic device 900 according to an exemplary embodiment. The electronic device 900 may be a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0119] Reference Figure 15 The electronic device 900 may include at least one processor 910 and a memory 920. The processor 910 can execute instructions stored in the memory 920. The processor 910 is communicatively connected to the memory 920 via a data bus. In addition to the memory 920, the processor 910 can also be communicatively connected to an input device 930, an output device 940, and a communication device 950 via the data bus.

[0120] Processor 910 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0121] The memory 920 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0122] In this embodiment of the present disclosure, the memory 920 stores executable instructions, and the processor 910 can read the executable instructions from the memory 920 and execute the instructions to implement all or part of the steps of the kernel data acquisition and processing method based on multi-level cache optimization as described in any of the exemplary embodiments above.

[0123] Exemplary computer-readable storage medium

[0124] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in any of the methods in the exemplary embodiments described above.

[0125] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0126] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0127] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0128] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A kernel data acquisition and processing method based on multi-layer cache optimization, characterized in that, include: Kernel data is collected from the operating system kernel, and the kernel data is deduplicated. A time wheel is used to control the switching frequency of the AB dual buffer. The deduplicated kernel data is written and read through the dual buffer mechanism, and the deduplicated kernel data is batch processed and merged to obtain the operation data. The operation data is classified and sorted in a batch processing queue using a prefix tree to obtain the sorting result; Based on the sorting result, the operation data is written into a circular buffer of the corresponding priority located in shared memory; Based on the set conditions, the target data is retrieved from the circular buffer using a skip list and sorted. The target data is then exported to the corresponding process according to the business priority.

2. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The process of collecting kernel data from the operating system kernel and performing deduplication on the kernel data includes: Kernel data is collected at the operating system kernel level; A Bloom filter is used to deduplicate the collected kernel data. If all Bloom filter values ​​are 1, it means that the corresponding kernel data exists repeatedly, and the kernel data is discarded. If any Bloom filter value is 0, it means that the corresponding kernel data does not exist, and the kernel data is retained.

3. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The step of collecting kernel data from the operating system kernel and performing deduplication on the kernel data further includes: Kernel data is collected at the operating system kernel level; The kernel data is converted into a fixed-length hash value using a hash function. A hash table is used to compare the hash value of the key with the content for deduplication. Specifically, when inserting kernel data, if data already exists at the position corresponding to the hash value of the kernel data, the key content is compared. If the key content is the same, the corresponding kernel data is considered to exist repeatedly and is discarded; otherwise, the kernel data is retained.

4. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The time-wheel control of the AB dual-buffer switching frequency is used to write and read deduplicated kernel data through the dual-buffer mechanism, and the deduplicated kernel data is batch-processed and merged to obtain operation data, including: The CPU time is divided into fixed-length time slices based on the time wheel algorithm. According to the control of the time wheel, one cache is set to the data writing state to write deduplicated kernel data, and the other cache is set to the data reading state to read the kernel data written in the cache. When the time slice expires, a timer interrupt is used to trigger a swap of the buffer pointers and adjust the state of the two buffers. The kernel data read is processed in batches using a batch processing queue, and then merged to obtain the operation data.

5. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The step of classifying and sorting the batch processing queue of the operation data using a prefix tree to obtain the sorting result includes: A prefix tree is used to store the prefix relationship of keys in a tree structure. The root node of the tree structure does not contain a character, and each child node represents a character. The characters on the path from the root node to the child node are concatenated to obtain the string corresponding to the node. All child nodes of each node contain different characters. The prefix relationship includes IP address, URL path and log source. The process involves traversing and matching characters one by one from the root node, classifying and sorting the operation data in the batch processing queue, and obtaining the sorted results.

6. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The step of writing the operation data into a circular buffer of corresponding priority in shared memory according to the sorting result includes: The priority of the operation data is determined based on the sorting result of the operation data; Based on the priority of the operation data, the operation data is written into the corresponding circular buffer. Multiple circular buffers are set up, and the multiple circular buffers are located in shared memory. Each circular buffer corresponds to a priority.

7. The kernel data acquisition and processing method based on multi-layer cache optimization according to claim 1, characterized in that, The process of retrieving and sorting target data in the circular buffer using a skip list based on set conditions, and then exporting the target data to the corresponding process according to business priority, includes: Obtain preset conditions, construct a multi-level index using a skip list based on the preset conditions, and retrieve the target data in the circular buffer. The target data is sorted according to priority to obtain a priority queue, and the target data is exported to the corresponding process based on business priority.

8. A kernel data acquisition and processing device based on multi-layer cache optimization, characterized in that, The kernel data acquisition and processing method based on multi-level cache optimization as described in any one of claims 1-7 includes: The kernel data deduplication module is used to collect kernel data from the operating system kernel and perform deduplication processing on the kernel data; The kernel data dual-buffer module is used to control the switching frequency of the AB dual buffer using a time wheel, write and read deduplicated kernel data through the dual-buffer mechanism, and perform batch processing on the deduplicated kernel data to obtain the operation data. The operation data classification module is used to classify and sort the operation data in the batch processing queue using a prefix tree; The operation data writing module is used to write the operation data into a circular buffer of corresponding priority located in shared memory according to the sorting result; The target data retrieval module is used to retrieve target data from the circular buffer based on set conditions using a skip list, sort the data, and export the target data to the corresponding process according to business priority.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the kernel data acquisition and processing method based on multi-level cache optimization as described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the kernel data acquisition and processing method based on multi-level cache optimization as described in any one of claims 1-7.

Images