Trace data stream type processing method and system for distributed block storage system

Abstract

The invention relates to the technical field of distributed storage, and discloses a Trace data stream processing method and system for a distributed block storage system, and the method comprises the steps: collecting original Trace data from a data source of each service process through a preset burying point; processing the original Trace data by adopting a streaming computing model, and analyzing to obtain a streaming data record; and storing the analyzed data in a local file and carrying out visual display. By means of the method, the Trace data stream processing system for the distributed block storage system is achieved, the total amount of data needing to be stored is greatly reduced through stream computation, aggregation and analysis logic of main events are achieved for the block storage system, the complexity of processing and problem positioning is simplified, and the processing efficiency is improved. Efficient Trace data acquisition, analysis, analysis and display are realized in the user mode, frequent switching between the user mode and the kernel mode is avoided, and the method has remarkable performance and flexibility advantages.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a Trace data stream processing method and system for a distributed block storage system. Background Art

[0002] A distributed storage system refers to a system that stores data in multiple physical nodes in a distributed manner, and has the characteristics of high availability, high fault tolerance, and scalability. With the development of big data and cloud computing technologies, distributed storage has been widely used in various applications, especially in large-scale enterprise-level storage, data backup, and big data analysis. By dividing data into multiple blocks and storing them in different nodes, distributed storage systems can provide elastic expansion and efficient data access capabilities. Distributed storage usually uses redundancy mechanisms (such as replicas, erasure codes, etc.) to ensure data reliability. Under this architecture, operations such as storage node management, data scheduling, and node failure recovery all need to be coordinated in a distributed environment. Although distributed storage systems can provide users with high performance and high reliability, they also face a series of challenges such as performance tuning, fault location, and system optimization during actual operation.

[0003] To solve the problem of how to effectively diagnose and locate problems in distributed storage systems, such as performance bottlenecks, node failures, data loss, etc. The characteristics of distributed storage systems, such as multi-node distribution, asynchronous operation and complex fault-tolerant mechanisms, make it particularly difficult to locate faults and performance problems. Specifically, the difficulty of locating problems in distributed storage is mainly reflected in the following aspects: (1) The complexity between multiple nodes. Since data is distributed on multiple physical nodes, faults or performance problems may come from any node, or even problems with the coordination between multiple nodes. This makes it very complicated to locate faults and performance bottlenecks; (2) Lack of a global view. In a distributed storage system, each node usually only cares about local storage data and operations, and does not have a global monitoring and view. The lack of visibility into global data flows and operational status makes it difficult to analyze and diagnose problems in real time. (3) Asynchronous operations and delays. Many distributed storage systems use asynchronous operations to improve performance. Although this can speed up the reading and writing of data, it also increases system delays and uncertainty in responses. When problems occur, it is difficult to accurately track and capture the timing relationship of operations, which is a challenge. (4) As a data storage system, especially a block storage system, the IOPS requirement far exceeds that of other types of distributed systems. Therefore, it is particularly sensitive to the performance consumption of the positioning method of a single request (IO). The scale of requests that need to be processed is also larger than that of other types of distributed systems.

[0004] To address the above issues, tracing has become an important means to solve distributed storage system fault location and performance optimization. Trace data refers to the detailed operation logs and event information recorded during the operation of the storage system, including the timestamp of the request, the execution node, the access status of the data block, etc. This data can help operation and maintenance personnel and system administrators to comprehensively monitor and deeply analyze the behavior of the storage system. By collecting and analyzing trace data, the operation of the storage system can be tracked in real time to quickly identify faults or performance bottlenecks. For example, when read and write delays occur, trace data can accurately reveal the node and operation where the bottleneck is located. Trace data can provide a global view of the storage system and help analyze the collaborative work between different nodes. By correlating and analyzing trace data across nodes, potential system bottlenecks or root causes of faults can be found. Trace data can provide high-precision time series information, so that problem location no longer depends on abstract log data, but is based on actual operation records. This refined positioning capability is crucial for complex distributed storage systems.

[0005] However, the typical process of the existing trace processing system is shown in the figure Figure 1 As shown in the figure, trace collection agents are deployed on each storage node. These agents monitor system events or application running status in the node in real time and generate trace data. The collected trace data is transmitted to a centralized database through the network for storage and management. Users can access the database through the query interface to retrieve the required trace data, thereby analyzing the operating performance and potential problems of the storage system. The existing distributed storage system generates a huge amount of trace data under large-scale data flow and high concurrent access, and it is necessary to collect, store and process this data efficiently. At the same time, in the existing trace system, data is often simply collected and stored, and then other data analysis tools are used to find effective information. The relationship between these data is complex, and it is necessary to design logically complex query statements to extract valuable data from them. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the above-mentioned prior art and provide a Trace data streaming processing method and system for a distributed block storage system, which greatly reduces the total amount of data that needs to be stored through streaming computing, implements the aggregation and analysis logic of major events for the block storage system, and simplifies the complexity of processing and problem location.

[0007] On the one hand, a Trace data stream processing method for a distributed block storage system is provided, comprising the following steps: S1: Collects raw trace data from the data sources of each service process through preset tracking points; S2: Processing the original Trace data using a streaming computing model, parsing and obtaining streaming data records; S3: Stores the parsed data in local files and displays them visually.

[0008] Furthermore, in step S1, the preset burying point further includes: Setting tracking points on the critical path of the target program, the critical path includes function entry, IO operation and system call location, each tracking point records a set of structured event data, including timestamp, event type and parameter information; Dynamically enable or disable tracking points through the Trace data collection session management function, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

[0009] Preferably, the collecting event data through the user-mode probe and writing it into a CTF format file further includes: The CTF format file is adapted to Trace data storage, and provides a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

[0010] Furthermore, in step S2, the adopting the stream computing model to process the original Trace data further includes: The streaming computing model framework sets a timed polling file directory to capture the generation of CTF format files, thereby achieving seamless connection between data processing and data collection; Read the CTF format file, extract the structured events according to its format specification, and convert the content of the entire CTF format file into a series of structured events by parsing the file header information and data blocks to form a streaming data record for further analysis and processing.

[0011] Preferably, the parsing to obtain streaming data records further comprises: Taking events as the smallest unit, after successfully parsing the structured events, the next step of processing is seamlessly transferred through the real-time push mechanism, and streaming data records are obtained through analysis, including statistical summary data and trace data after association alignment; Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize analyzers. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, dynamic loading and execution of user-written scripts are supported, allowing users to verify the processing logic in real time.

[0012] Preferably, the process of processing the original Trace data using the stream computing model also includes an automatic cleanup mechanism: After the data parsing is completed, the streaming computing model framework triggers the file cleaning operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed; Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

[0013] Furthermore, in step S3, storing the parsed data in a local file further includes: After the trace data is parsed and processed by the streaming computing model, all events of the entire life cycle of a specific IO in the storage system are cached in the memory; Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the Trace data that meets the configuration setting conditions is completely stored, and the remaining data is deleted to reduce disk usage.

[0014] Furthermore, in step S3, the visual display further includes: Draw statistical graphs and time series graphs based on the summarized data to display aggregated statistical data and time series data; Use the hardware acceleration function provided by the rendering tool to adapt to high-resolution icon rendering; It allows users to expand data display capabilities as needed, dynamically adjust the icon display range through human-computer interaction, and support clicking to view detailed information of events.

[0015] On the other hand, a trace data stream processing system for a distributed block storage system is provided, including: The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points; A data processing module, used to process the original Trace data using a streaming computing model, and parse and obtain streaming data records; The data storage and display module is used to store the parsed data in local files and display them visually.

[0016] In addition, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for processing Trace data streams for a distributed block storage system described in any one of the above is implemented.

[0017] At the same time, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned Trace data streaming processing methods for distributed block storage systems.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention significantly reduces disk storage pressure by adopting a streaming computing model framework and its automatic cleanup mechanism, and is suitable for long-term and large-scale data collection scenarios; The present invention implements parsing logic through user state, supports hot loading scripts, and allows users to instantly verify processing logic without restarting the system, thus avoiding the high overhead of kernel state switching, and can significantly reduce performance loss in high-frequency Trace data scenarios. Data display usually has built-in fixed logic and cannot be dynamically expanded. Data display modules mostly use SVG or Canvas, which has poor performance, especially when frequently updated, which will significantly slow down the interactive response. The present invention allows users to expand data display capabilities on demand without relying on the development team, thus enhancing system flexibility. The present invention stores Trace data with user-configured filtering rules, completely retains all important Trace data, and is very helpful in locating system performance problems. The present invention supports high-frequency data updates and large-scale data display by drawing statistical charts and time series charts. At the same time, the data display format can also be controlled by dynamically loaded plug-ins, which improves the scalability of the software, can meet more data display needs, and provides dynamic interactive functions. Users can browse, zoom and filter data charts in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a Trace data processing system architecture diagram in the prior art; Figure 2 This is a flow chart of a Trace data stream processing method for a distributed block storage system of the present invention; Figure 3 This is a structural block diagram of a Trace data stream processing system for a distributed block storage system of the present invention; Figure 4 This is a structural block diagram of a data acquisition module of the present invention; Figure 5 This is a structural block diagram of a data processing module of the present invention; Figure 6 This is a structural block diagram of a data storage display module of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The technical solution of the present invention realizes efficient Trace data collection, parsing, analysis and display in user mode, avoids frequent switching between user mode and kernel mode, and has significant performance and flexibility advantages.

[0022] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.

[0023] like Figure 2 As shown, a technical solution of a Trace data stream processing method for a distributed block storage system provided in this embodiment includes the following steps: S1: Collects raw trace data from the data sources of each service process through preset tracking points; S2: Processing the original Trace data using a streaming computing model, parsing and obtaining streaming data records; S3: Stores the parsed data in local files and displays them visually.

[0024] like Figure 4 As shown in the figure, the data acquisition module collects the required trace data through the preset tracking points during the running of the target program. The collected data is stored in the disk in CTF (Common Trace Format) format. To achieve efficient data management, the data acquisition module cooperates with the framework layer and adopts a streaming processing architecture. After completing the data analysis, it automatically cleans up the processed CTF files to ensure efficient use of disk space.

[0025] Specifically, tracking points are logic codes embedded in the program, which are used to collect specific events or indicator data during operation. Tracking points are set on the key paths of the target program, such as function entry, IO operation, system call and other locations. Each tracking point records a set of structured event data, including timestamp, event type and parameter information; Dynamically enable or disable tracking points through the Trace data collection session management function, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

[0026] Among them, in this embodiment, the timestamp is a high-precision timestamp used to analyze the time relationship of events; the event type is used to identify the type of event (such as function call, exception, system IO); the parameter information is the key data of the event context, such as input parameters or return values. Then, the trace data collection session management function is used to dynamically enable or disable the tracking point, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

[0027] The collected trace data is stored in a CTF format file. The main features of this format file are as follows: CTF is a standardized binary file format designed for high-performance trace data storage. It provides efficient timestamp indexing and event segmentation structure to facilitate subsequent parsing and querying. Therefore, the process of collecting event data through user-mode probes and writing them into CTF format files further includes: The CTF format file is adapted to Trace data storage, and provides a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

[0028] Afterwards, if Figure 5 As shown in the figure, the data acquisition module is seamlessly connected with the streaming computing framework to realize real-time processing of CTF files.

[0029] In this embodiment, after detecting the newly generated CTF file, the streaming computing framework immediately loads the file content and parses it into streaming data records. The parser can directly process the file header information and data blocks to extract structured events. The data parsing process uses events as the smallest unit, and each time an event is parsed, it is sent to the subsequent analysis module (such as statistics, association with other events). Some common built-in analyzers have been implemented in the system, and interfaces are reserved to support users to dynamically add custom analyzers. Users can implement scripts using the Python language. The script contains analysis logic, storage logic, and display logic. The data analysis module dynamically executes the script, parses the Trace data in real time, and converts it into structured information. The parsed data will be transmitted to the result storage and display module.

[0030] Specifically, in step S2, the adopting the stream computing model to process the original Trace data includes: The streaming computing model framework sets a timed polling file directory to capture the generation of CTF format files, thereby achieving seamless connection between data processing and data collection; Read the CTF format file, extract the structured events according to its format specification, and convert the content of the entire CTF format file into a series of structured events by parsing the file header information and data blocks to form a streaming data record for further analysis and processing.

[0031] The parsing to obtain streaming data records further comprises: Taking events as the smallest unit, after successfully parsing the structured events, the next step of processing is seamlessly transferred through the real-time push mechanism, and streaming data records are obtained through analysis, including statistical summary data and trace data after association alignment; Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize the analyzer. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, it supports dynamic loading and execution of user-written scripts. It also supports hot loading of scripts, allowing users to instantly verify the processing logic and avoid frequent system restarts.

[0032] This design significantly reduces system expansion costs while enhancing system flexibility.

[0033] In addition, the use of the stream computing model to process the original Trace data also includes an automatic cleanup mechanism: After the data parsing is completed, the streaming computing model framework triggers the file cleaning operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed; Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

[0034] When locating performance problems, most of the trace data is normal, and only a few trace data are worthy of attention. After passing through the data processing module, all events of the entire life cycle of a specific IO in the storage system are cached in the memory. The trace data filter only stores the trace data that meets the conditions completely through the user-configured filtering rules (such as latency threshold, node ID, disk ID, etc.) for problem analysis, and the rest of the data is discarded.

[0035] The advantage of doing this is that it can retain the most detailed trace data while reducing disk usage.

[0036] Specifically, Figure 6As shown, in step S3, storing the parsed summary data and the Trace data in a local file includes: After the trace data is parsed and processed by the streaming computing model, all events of the entire life cycle of a specific IO in the storage system are cached in the memory; Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the Trace data that meets the configuration setting conditions is completely stored, and the remaining data is deleted to reduce disk usage.

[0037] The visual display further includes: Draw statistical graphs and time series graphs based on the summary data to display aggregated statistical data and time series data, wherein the statistical graphs support scatter plots, line graphs, etc., for displaying aggregated statistical data, such as latency distribution and cache hit rate, and the sequence graphs display time series data, such as IOPS, bandwidth usage, latency trend, etc.; Use the hardware acceleration function provided by the rendering tool WebGL to adapt to high-resolution icon rendering; Allow users to expand data display capabilities as needed, dynamically adjust the icon display range through human-computer interaction, and support clicking to view event details. Users can dynamically adjust the chart display range by dragging and zooming with the mouse. Support clicking on an event to view detailed information, such as the number of specific requests within a certain time period.

[0038] In this embodiment, data display usually has built-in fixed logic and cannot be dynamically expanded. The data display module mostly uses SVG or Canvas, which has poor performance, especially when updated at high frequency, which will significantly slow down the interactive response. Allow users to expand data display capabilities on demand without relying on the development team, enhancing system flexibility. In addition, WebGL rendering achieves high-performance dynamic rendering through hardware acceleration and adapts to large-scale data scenarios, while Trace data storage uses a combination of streaming computing and filtering rules, which can not only collect critical and detailed Trace data, but also significantly reduce data processing delays and disk usage.

[0039] This embodiment also provides a trace data stream processing system for a distributed block storage system, including: The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points; A data processing module, used to process the original Trace data using a streaming computing model, and parse and obtain streaming data records; The data storage and display module is used to store the parsed data in local files and display them visually.

[0040] The functional implementation of each module and unit in the Trace data streaming processing system for the distributed block storage system corresponds to each step in the above-mentioned embodiment of the intelligent audio playback control method, see Figures 3 to 6 Its functions and implementation process will not be described here one by one.

[0041] In summary, the method and system of this embodiment are implemented through the following aspects: (1) Efficient trace data collection and processing based on user state. Trace data is collected through key path tracking technology to generate CTF format files, avoiding traditional methods that rely on kernel state. The streaming computing framework is used to complete real-time parsing of the CTF format, and the original data file is automatically deleted after processing to save disk space. The performance overhead caused by frequent switching between user state and kernel state is avoided, and the real-time and efficient data processing are improved. The automatic cleanup mechanism significantly reduces the occupancy of storage resources, allowing the system to run for a long time under limited resources, improving the real-time processing capability of trace data, and is particularly suitable for high-frequency collection and large data volume scenarios. After receiving an event indicating the end of the life cycle of an IO, the entire IO will be judged according to the set rules. If it is judged to be an abnormal IO, all its events and timestamps will be written to the disk, which can not only reduce disk usage, but also obtain the most accurate trace data, making it easier to locate the problem.

[0042] (2) Common analysis scripts related to key steps in the IO path of the distributed block storage system are pre-set to simplify the analysis process and provide a simple interface so that users can quickly write and load custom analysis scripts. This greatly reduces the threshold for developing analysis logic, enhances the flexibility of the system, can adapt to diverse analysis needs, and improves the ease of use and scalability of the system.

[0043] (3) Use WebGL technology to draw statistical charts and time series charts, support high-frequency data updates and large-scale data display. At the same time, the data display format can also be controlled by dynamically loaded plug-ins, which improves the scalability of the software, can meet more data display needs, and provide dynamic interactive functions. Users can browse, zoom and filter data charts in real time. In the case of insufficient traditional Canvas drawing performance, smooth dynamic display is achieved through WebGL, which significantly improves the user experience and meets the needs of real-time and interactivity. It is particularly suitable for high-frequency data update scenarios.

[0044] Finally, it should be noted that the above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0045] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A Trace data stream processing method for a distributed block storage system, characterized in that: The steps include: S1: Collects raw trace data from the data sources of each service process through preset tracking points; S2: Processing the original Trace data using a streaming computing model, parsing and obtaining streaming data records; S3: Stores the parsed data in local files and displays them visually.

2. The Trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: In step S1, the preset burying point further includes: Setting tracking points on the critical path of the target program, the critical path includes function entry, IO operation and system call location, and each tracking point records a set of structured event data, including timestamp, event type and parameter information; Dynamically enable or disable tracking points through the Trace data collection session management function, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

3. The Trace data stream processing method for a distributed block storage system according to claim 2, characterized in that: The collecting event data through the user-mode probe and writing it into the CTF format file further includes: The CTF format file is adapted to Trace data storage, and provides a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

4. The Trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: In step S2, the adopting the stream computing model to process the original Trace data further includes: The streaming computing model framework sets a timed polling file directory to capture the generation of CTF format files, thereby achieving seamless connection between data processing and data collection; Read the CTF format file, extract the structured events according to its format specification, and convert the content of the entire CTF format file into a series of structured events by parsing the file header information and data blocks to form a streaming data record for further analysis and processing.

5. The Trace data stream processing method for a distributed block storage system according to claim 4, characterized in that: The parsing to obtain streaming data records further comprises: Taking events as the smallest unit, after successfully parsing the structured events, the next step of processing is seamlessly transferred through the real-time push mechanism, and streaming data records are obtained through analysis, including statistical summary data and trace data after association alignment; Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize analyzers. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, dynamic loading and execution of user-written scripts are supported, allowing users to verify the processing logic in real time and avoid frequent system restarts.

6. The Trace data stream processing method for a distributed block storage system according to claim 5, characterized in that: The use of the stream computing model to process the original Trace data also includes an automatic cleanup mechanism: After the data parsing is completed, the streaming computing model framework triggers the file cleaning operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed; Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

7. The Trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: In step S3, storing the parsed data in a local file further includes: After the trace data is parsed and processed by the streaming computing model, all events of the entire life cycle of a specific IO in the storage system are cached in the memory; Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the Trace data that meets the configuration setting conditions is completely stored, and the remaining data is deleted to reduce disk usage.

8. The Trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: In step S3, the visual display further includes: Draw statistical graphs and time series graphs based on the summarized data to display aggregated statistical data and time series data; Use the hardware acceleration function provided by the rendering tool to adapt to high-resolution icon rendering; It allows users to expand data display capabilities as needed, dynamically adjust the icon display range through human-computer interaction, and support clicking to view detailed information of events.

9. A trace data stream processing system for a distributed block storage system, characterized in that: include: The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points; A data processing module, used to process the original Trace data using a streaming computing model, and parse and obtain streaming data records; The data storage and display module is used to store the parsed data in local files and display them visually.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the Trace data stream processing method for a distributed block storage system according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the Trace data streaming processing method for a distributed block storage system as described in any one of claims 1-8.

Images