Trace data streaming processing method and system for distributed block storage system

Abstract

The present invention relates to the field of distributed storage technology, and discloses a Trace data streaming processing method and system for a distributed block storage system, the method comprising: collecting raw Trace data from the data source of each service process through a preset tracking point; processing the raw Trace data using a streaming computing model, parsing to obtain streaming data records; storing the parsed data in a local file and visually displaying it. Through the above method, a Trace data streaming processing system for a distributed block storage system is realized, which greatly reduces the total amount of data that needs to be stored through streaming computing, implements aggregation and analysis logic for major events in the block storage system, simplifies the complexity of processing and problem location, and realizes efficient Trace data collection, parsing, analysis and display in the user state, avoiding frequent switching between the user state and the kernel state, and has significant performance and flexibility advantages.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a trace data stream processing method and system for a distributed block storage system. Background Art

[0002] Distributed storage systems store data across multiple physical nodes, offering high availability, fault tolerance, and scalability. With the development of big data and cloud computing technologies, distributed storage has gained widespread adoption in various applications, particularly in large-scale enterprise storage, data backup, and big data analytics. By partitioning data into multiple blocks and storing them across different nodes, distributed storage systems provide elastic scalability and efficient data access. Distributed storage typically employs redundancy mechanisms (such as replicas and erasure codes) to ensure data reliability. In this architecture, operations such as storage node management, data scheduling, and node failure recovery all require coordination within the distributed environment. While distributed storage systems offer high performance and reliability, they also present challenges in actual operation, including performance tuning, fault location, and system optimization.

[0003] To solve the problem of how to effectively diagnose and locate problems that occur in distributed storage systems, such as performance bottlenecks, node failures, data loss, etc. The characteristics of distributed storage systems, such as multi-node distribution, asynchronous operation and complex fault tolerance mechanisms, make it particularly difficult to locate faults and performance problems. Specifically, the difficulty of locating problems in distributed storage is mainly reflected in the following aspects: (1) The complexity between multiple nodes. Since data is distributed on multiple physical nodes, faults or performance problems may come from any node, or even problems with the coordination between multiple nodes. This makes locating faults and performance bottlenecks very complicated; (2) Lack of a global view. In a distributed storage system, each node usually only cares about local storage data and operations, and does not have a global monitoring and view. The lack of visibility into global data flows and operational status makes real-time analysis and diagnosis of problems difficult; (3) Asynchronous operations and delays. Many distributed storage systems use asynchronous operations to improve performance. While this can speed up data reading and writing, it also increases system delays and uncertainty in responses. When problems occur, it is difficult to accurately track and capture the timing relationship of operations, which is a challenge; (4) As a data storage system, especially a block storage system, the IOPS requirements far exceed those of other types of distributed systems. Therefore, it is particularly sensitive to the performance consumption of the positioning method of a single request (IO), and the scale of requests that need to be processed is also larger than that of other types of distributed systems.

[0004] To address these issues, tracing has become a crucial tool for fault location and performance optimization in distributed storage systems. Trace data refers to detailed operation logs and event information recorded during storage system operation, including request timestamps, execution nodes, and data block access details. This data helps operations personnel and system administrators comprehensively monitor and conduct in-depth analysis of storage system behavior. By collecting and analyzing trace data, storage system operations can be tracked in real time, allowing for rapid identification of faults or performance bottlenecks. For example, when read or write latency occurs, trace data can accurately reveal the bottleneck node and operation. Trace data provides a global view of the storage system, helping to analyze the interoperability of different nodes. Correlating trace data across nodes can identify potential system bottlenecks or root causes of faults. Trace data provides highly accurate time series information, enabling problem location based on actual operation records rather than abstract log data. This granular location capability is crucial for complex distributed storage systems.

[0005] However, the typical process of the existing trace processing system is shown in the attached figure Figure 1 As shown, trace collection agents are deployed on each storage node. These agents monitor system events or application status within the node in real time and generate trace data. The collected trace data is transmitted over the network to a centralized database for storage and management. Users can access the database through a query interface to retrieve the required trace data, thereby analyzing storage system performance and potential issues. Existing distributed storage systems generate enormous amounts of trace data due to large-scale data flows and high concurrent access, necessitating efficient collection, storage, and processing of this data. Furthermore, existing trace systems often simply collect and store data, then rely on other data analysis tools to identify useful insights. The complex relationships between these data require the design of logically complex queries to extract valuable data. Summary of the Invention

[0006] The purpose of the present invention is to address the shortcomings of the above-mentioned prior art and to provide a trace data streaming processing method and system for a distributed block storage system. Through streaming computing, the total amount of data that needs to be stored is greatly reduced, the aggregation and analysis logic of major events in the block storage system is implemented, and the complexity of processing and problem location is simplified.

[0007] On the one hand, a trace data stream processing method for a distributed block storage system is provided, comprising the following steps:

[0008] S1: Collects raw trace data from the data sources of each service process through preset tracking points;

[0009] S2: Processing the original trace data using a streaming computing model to parse and obtain streaming data records;

[0010] S3: Stores the parsed data in local files and displays them visually.

[0011] Furthermore, in step S1, the preset tracking points further include:

[0012] Set tracking points on the critical path of the target program, which includes function entry, IO operation and system call location. Each tracking point records a set of structured event data, including timestamp, event type and parameter information.

[0013] Dynamically enable or disable tracking points through the Trace data collection session management function, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

[0014] Preferably, the collecting event data through the user-mode probe and writing the data into a CTF format file further includes:

[0015] The CTF format file is adapted to Trace data storage, providing a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

[0016] Furthermore, in step S2, the processing of the original Trace data using the stream computing model further includes:

[0017] The streaming computing model framework sets a periodic polling file directory to capture the generation of CTF format files, thus achieving seamless connection between data processing and data collection;

[0018] Read CTF format files, extract structured events according to their format specifications, and convert the entire CTF format file content into a series of structured events by parsing file header information and data blocks, forming streaming data records for further analysis and processing.

[0019] Preferably, the parsing to obtain streaming data records further comprises:

[0020] Taking events as the smallest unit, after successfully parsing the structured events, they are seamlessly transferred to the next step of processing through a real-time push mechanism. Streaming data records are obtained through analysis, including statistical summary data and trace data after correlation and alignment.

[0021] Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize the analyzer. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, it supports dynamic loading and execution of user-written scripts, allowing users to verify the processing logic in real time.

[0022] More preferably, the process of processing the raw trace data using the stream computing model further includes an automatic cleanup mechanism:

[0023] After the data parsing is completed, the streaming computing model framework triggers the file cleanup operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed;

[0024] Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

[0025] Furthermore, in step S3, storing the parsed data in a local file further includes:

[0026] After the trace data is parsed and processed by the streaming computing model, all events of the entire life cycle of a specific IO in the storage system are cached in the memory;

[0027] Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the trace data that meets the configuration conditions is completely stored, and the remaining data is deleted to reduce disk usage.

[0028] Furthermore, in step S3, the visual display further includes:

[0029] Draw statistical graphs and time series graphs based on the summarized data to display aggregated statistical data and time series data;

[0030] Use the hardware acceleration function provided by the rendering tool to adapt to high-resolution icon rendering;

[0031] It allows users to expand data display capabilities as needed, dynamically adjust the icon display range through human-computer interaction, and support clicking to view detailed information of events.

[0032] In another aspect, a trace data stream processing system for a distributed block storage system is provided, comprising:

[0033] The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points;

[0034] A data processing module is used to process the original trace data using a streaming computing model and parse and obtain streaming data records;

[0035] The data storage and display module is used to store the parsed data in local files and display them visually.

[0036] In addition, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the program implements any of the above-mentioned trace data stream processing methods for a distributed block storage system.

[0037] At the same time, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned trace data streaming processing methods for a distributed block storage system.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention significantly reduces disk storage pressure by adopting a streaming computing model framework and its automatic cleanup mechanism, and is suitable for scenarios where large amounts of data are collected over a long period of time.

[0040] This invention implements parsing logic in user mode, supports hot script loading, and allows users to instantly verify processing logic without restarting the system, avoiding the high overhead of kernel mode switching. It can significantly reduce performance loss, especially in high-frequency trace data scenarios.

[0041] Data display usually has fixed built-in logic and cannot be dynamically expanded. Data display modules often use SVG or Canvas, which has poor performance and significantly slows down interactive responses, especially when frequently updated. This invention allows users to expand data display capabilities on demand without relying on the development team, thus enhancing system flexibility.

[0042] The present invention stores trace data with user-configured filtering rules, completely retaining all important trace data, which is very helpful in locating system performance problems.

[0043] The present invention supports high-frequency data updates and large-scale data display by drawing statistical charts and time series charts. At the same time, the data display format can also be controlled by dynamically loaded plug-ins, which improves the scalability of the software, can meet the needs of more data display, and provides dynamic interactive functions. Users can browse, zoom and filter data charts in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 This is a diagram of the architecture of a Trace data processing system in the prior art;

[0046] Figure 2 This is a flow chart of a Trace data stream processing method for a distributed block storage system according to the present invention;

[0047] Figure 3 This is a structural block diagram of a Trace data stream processing system for a distributed block storage system according to the present invention;

[0048] Figure 4 This is a structural block diagram of a data acquisition module of the present invention;

[0049] Figure 5 This is a structural block diagram of a data processing module of the present invention;

[0050] Figure 6 This is a structural block diagram of a data storage display module of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The technical solution of the present invention realizes efficient Trace data collection, parsing, analysis and display in user mode, avoids frequent switching between user mode and kernel mode, and has significant performance and flexibility advantages.

[0053] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.

[0054] like Figure 2 As shown in FIG, a technical solution for a trace data stream processing method for a distributed block storage system provided in this embodiment includes the following steps:

[0055] S1: Collects raw trace data from the data sources of each service process through preset tracking points;

[0056] S2: Processing the original trace data using a streaming computing model to parse and obtain streaming data records;

[0057] S3: Stores the parsed data in local files and displays them visually.

[0058] like Figure 4 As shown in the figure, the data acquisition module collects the required trace data using pre-set tracking points while the target program is running. The collected data is stored on disk in the Common Trace Format (CTF) format. To achieve efficient data management, the data acquisition module collaborates with the framework layer, using a streaming processing architecture. After data parsing, it automatically cleans up processed CTF files to ensure efficient use of disk space.

[0059] Specifically, tracking is logic code embedded in a program that collects specific events or metrics during execution. Tracking is set along the target program's critical paths, such as function entry points, I / O operations, and system calls. Each tracking record records a set of structured event data, including timestamps, event types, and parameter information.

[0060] Dynamically enable or disable tracking points through the Trace data collection session management function, support flexible configuration of the event range and granularity to be collected, collect event data through user-mode probes and write them into CTF format files.

[0061] In this embodiment, the timestamp is a high-precision timestamp used to analyze the temporal relationship of events; the event type identifies the type of event (such as function call, exception, or system I / O); and the parameter information is key data in the event context, such as input parameters or return values. Trace data collection session management is then used to dynamically enable or disable tracking points, supporting flexible configuration of the event scope and granularity to be collected. Event data is collected through user-mode probes and written to CTF files.

[0062] The collected trace data is stored as a CTF format file. The main features of this format file are as follows:

[0063] CTF is a standardized binary file format designed for high-performance trace data storage. It provides efficient timestamp indexing and event segmentation, facilitating subsequent parsing and querying. Therefore, the process of collecting event data through user-mode probes and writing it to a CTF-formatted file further includes:

[0064] The CTF format file is adapted to Trace data storage, providing a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

[0065] Afterwards, if Figure 5 As shown in Figure 1, the data acquisition module is seamlessly connected with the streaming computing framework to achieve real-time processing of CTF files.

[0066] In this embodiment, upon detecting a newly generated CTF file, the streaming computing framework immediately loads the file contents and parses them into streaming data records. The parser can directly process file header information and data blocks to extract structured events. The data parsing process uses events as the smallest unit, and each parsed event is sent to subsequent analysis modules (such as statistics and correlation with other events). The system has implemented some common built-in analyzers, and interfaces are reserved to support users to dynamically add custom analyzers. Users can use Python to implement scripts that contain analysis logic, storage logic, and display logic. The data analysis module dynamically executes the scripts, parsing trace data in real time and converting it into structured information. The parsed data is then transmitted to the result storage and display module.

[0067] Specifically, in step S2, the processing of the original Trace data using the stream computing model includes:

[0068] The streaming computing model framework sets a periodic polling file directory to capture the generation of CTF format files, thus achieving seamless connection between data processing and data collection;

[0069] Read CTF format files, extract structured events according to their format specifications, and convert the entire CTF format file content into a series of structured events by parsing file header information and data blocks, forming streaming data records for further analysis and processing.

[0070] The parsing to obtain streaming data records further includes:

[0071] Taking events as the smallest unit, after successfully parsing the structured events, they are seamlessly transferred to the next step of processing through a real-time push mechanism. Streaming data records are obtained through analysis, including statistical summary data and trace data after correlation and alignment.

[0072] Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize the analyzer. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, it supports dynamic loading and execution of user-written scripts. It also supports hot loading of scripts, allowing users to instantly verify the processing logic and avoid frequent system restarts.

[0073] This design significantly reduces system expansion costs while enhancing system flexibility.

[0074] In addition, the use of the stream computing model to process the raw trace data also includes an automatic cleanup mechanism:

[0075] After the data parsing is completed, the streaming computing model framework triggers the file cleanup operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed;

[0076] Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

[0077] When locating performance issues, the vast majority of trace data is normal, with only a small amount of trace data worthy of attention. After passing through the data processing module, all events in the entire lifecycle of a specific IO in the storage system are cached in memory. The trace data filter uses user-configured filtering rules (such as latency threshold, node ID, disk ID, etc.) to store only the trace data that meets the conditions in full for problem analysis, and the rest of the data is discarded.

[0078] The advantage of this is that it can retain the most detailed trace data while reducing disk usage.

[0079] Specifically, such as Figure 6 As shown, in step S3, storing the parsed summary data and the Trace data into a local file includes:

[0080] After the trace data is parsed and processed by the streaming computing model, all events of the entire life cycle of a specific IO in the storage system are cached in the memory;

[0081] Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the trace data that meets the configuration conditions is completely stored, and the remaining data is deleted to reduce disk usage.

[0082] The visual display further includes:

[0083] Draw statistical graphs and time series graphs based on the summarized data to display aggregated statistical data and time series data. Statistical graphs support scatter plots, line graphs, etc., for displaying aggregated statistical data such as latency distribution and cache hit rate, and sequence graphs for displaying time series data such as IOPS, bandwidth utilization, latency trends, etc.

[0084] Use the hardware acceleration function provided by the rendering tool WebGL to adapt to high-resolution icon rendering;

[0085] Allows users to expand data display capabilities on demand, dynamically adjust the chart display range through human-computer interaction, and support click-to-view event details. Users can dynamically adjust the chart display range by dragging and zooming with the mouse. Click on an event to view detailed information, such as the number of specific requests within a certain time period.

[0086] In this embodiment, data display typically has built-in fixed logic and cannot be dynamically expanded. Data display modules often use SVG or Canvas, which have poor performance and significantly slow interactive responses, especially during high-frequency updates. This allows users to expand data display capabilities on demand without relying on the development team, enhancing system flexibility. Furthermore, WebGL rendering achieves high-performance dynamic rendering through hardware acceleration, adapting to large-scale data scenarios. Trace data storage uses a combination of streaming computing and filtering rules to collect critical and detailed trace data while significantly reducing data processing latency and disk usage.

[0087] This embodiment further provides a trace data stream processing system for a distributed block storage system, including:

[0088] The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points;

[0089] A data processing module is used to process the original trace data using a streaming computing model and parse and obtain streaming data records;

[0090] The data storage and display module is used to store the parsed data in local files and display them visually.

[0091] The functional implementation of each module and unit in the Trace data stream processing system for the distributed block storage system corresponds to the steps in the above-mentioned embodiment of the intelligent audio playback control method, see Figures 3 to 6 Its functions and implementation processes will not be described here one by one.

[0092] In summary, the method and system of this embodiment are implemented through the following aspects:

[0093] (1) Efficient trace data collection and processing based on user state. Trace data is collected through key path tracking technology and CTF format files are generated, avoiding the traditional method of relying on kernel state. The streaming computing framework is used to complete the real-time parsing of the CTF format, and the original data file is automatically deleted after processing to save disk space. The performance overhead caused by frequent switching between user state and kernel state is avoided, and the real-time and efficient data processing is improved. The automatic cleanup mechanism significantly reduces the storage resource usage, allowing the system to run for a long time under limited resources, improving the real-time processing capability of trace data, and is particularly suitable for high-frequency collection and large data volume scenarios. After receiving an event indicating the end of the life cycle of an IO, the entire IO will be judged according to the set rules. If it is determined to be an abnormal IO, all its events and timestamps will be written to the disk, which can not only reduce disk usage but also obtain the most accurate trace data, making it easier to locate the problem.

[0094] (2) Common analysis scripts related to key steps in the IO path of distributed block storage systems are pre-installed to simplify the analysis process. A simple interface is provided so that users can quickly write and load customized analysis scripts. This significantly reduces the threshold for developing analysis logic, enhances the flexibility of the system, can adapt to diverse analysis needs, and improves the system's ease of use and scalability.

[0095] (3) Using WebGL technology to draw statistical charts and time series charts supports high-frequency data updates and large-scale data display. At the same time, the data display format can also be controlled by dynamically loaded plug-ins, which improves the scalability of the software and can meet the needs of more data display. It provides dynamic interactive functions, allowing users to browse, zoom, and filter data charts in real time. In the case of insufficient traditional Canvas drawing performance, WebGL achieves smooth dynamic display, significantly improving the user experience and meeting the needs of real-time and interactivity. It is particularly suitable for high-frequency data update scenarios.

[0096] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that improvements and modifications that do not depart from the principles of the present invention, which are apparent to those skilled in the art, should also be considered within the scope of protection of the present invention.

[0097] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A trace data stream processing method for a distributed block storage system, characterized in that: The steps include: S1: Collect raw trace data from the data source of each service process through preset tracking points. Tracking points are set on the critical path of the target program, including function entry, IO operation and system call location. Each tracking point records a set of structured event data, including timestamp, event type and parameter information. Tracking points are dynamically enabled or disabled through the trace data collection session management function, supporting flexible configuration of the event range and granularity to be collected. Event data is collected through user-mode probes and written to CTF format files, which provide timestamp indexes and event segmentation structures. S2: Use the streaming computing model to process the raw trace data, read the CTF format file, extract the structured events according to its format specifications, and convert the entire CTF format file content into a series of structured events by parsing the file header information and data blocks. Parse and obtain streaming data records. Through the built-in analyzer and custom analyzer of the streaming computing model framework, support dynamic loading and execution of user-written scripts, allowing users to instantly verify the processing logic; S3: Storing the parsed data in a local file and visually displaying it. Storing the parsed data in a local file includes: All events of the entire life cycle of a specific IO in the storage system are cached in memory. Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the trace data that meets the configured conditions is completely stored, and the remaining data is deleted to reduce disk usage.

2. The trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: The collecting event data through the user-mode probe and writing it into the CTF format file further includes: The CTF format file is adapted to Trace data storage, providing a timestamp index and an event segmentation structure. The CTF format file adopts segmented storage, and each acquisition session generates a series of independent CTF format files, and the file name includes a timestamp and a session ID.

3. The trace data stream processing method for a distributed block storage system according to claim 1, characterized in that: In step S2, the processing of the original Trace data using the stream computing model further includes: The streaming computing model framework sets a periodic polling file directory to capture the generation of CTF format files, thus achieving seamless connection between data processing and data collection; Read CTF format files, extract structured events according to their format specifications, and convert the entire CTF format file content into a series of structured events by parsing file header information and data blocks, forming streaming data records for further analysis and processing.

4. The trace data stream processing method for a distributed block storage system according to claim 3, characterized in that: The parsing to obtain streaming data records further includes: Taking events as the smallest unit, after successfully parsing the structured events, they are seamlessly transferred to the next step of processing through a real-time push mechanism. Streaming data records are obtained through analysis, including statistical summary data and trace data after correlation and alignment. Taking into account the diverse analysis needs of users in actual application scenarios, an interface is reserved for users to customize the analyzer. Through the built-in analyzer of the streaming computing model framework and the customized analyzer, it supports dynamic loading and execution of user-written scripts, allowing users to verify the processing logic in real time and avoid frequent system restarts.

5. The trace data stream processing method for a distributed block storage system according to claim 4, characterized in that: The use of a stream computing model to process the raw trace data also includes an automatic cleanup mechanism: After the data parsing is completed, the streaming computing model framework triggers the file cleanup operation through the callback mechanism, and deletes the CTF format file only after confirming that all events in the file have been parsed; Monitor disk usage and set a safety threshold. If the usage exceeds the threshold, the oldest CTF format file will be deleted first.

6. The trace data stream processing method for a distributed block storage system according to claim 4, characterized in that: In step S3, the visual display further includes: Draw statistical graphs and time series graphs based on the summarized data to display aggregated statistical data and time series data; Use the hardware acceleration function provided by the rendering tool to adapt to high-resolution icon rendering; It allows users to expand data display capabilities as needed, dynamically adjust the icon display range through human-computer interaction, and support clicking to view detailed information of events.

7. A trace data stream processing system for a distributed block storage system, characterized in that: include: The data collection module is used to collect raw trace data from the data sources of each service process through preset tracking points. Tracking points are set on the critical path of the target program, which includes function entry, IO operation and system call location. Each tracking point records a set of structured event data, including timestamp, event type and parameter information. Tracking points are dynamically enabled or disabled through the trace data collection session management function, supporting flexible configuration of the event range and granularity to be collected. Event data is collected through user-mode probes and written to CTF format files, which provide timestamp indexes and event segmentation structures. The data processing module is used to process the raw trace data using a streaming computing model, read CTF format files, extract structured events according to their format specifications, convert the entire CTF format file content into a series of structured events by parsing file header information and data blocks, and parse to obtain streaming data records. Through the built-in analyzer and custom analyzer of the streaming computing model framework, it supports dynamic loading and execution of user-written scripts, allowing users to instantly verify the processing logic; The data storage and display module is used to store the parsed data in a local file and display it visually. The storage of the parsed data in a local file includes: All events of the entire life cycle of a specific IO in the storage system are cached in memory. Through the user-configured filtering mechanism, all event data of the IO life cycle in the memory cache are traversed, and the trace data that meets the configured conditions is completely stored, and the remaining data is deleted to reduce disk usage.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the trace data stream processing method for a distributed block storage system is implemented according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the trace data streaming processing method for a distributed block storage system according to any one of claims 1 to 6.

Images