Method for determining a performance anomaly event and electronic device
Patent Information
- Application Number
- CN202611327515.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-10-09
AI Technical Summary
[0004]本申请提供了性能异常事件的确定方法和电子设备，以至少解决相关技术中无法对分布式存储系统IO请求进行全链路追踪与精准瓶颈定位的问题
[0009]本申请还提供了一种计算机程序产品，包括计算机程序，计算机程序被处理器执行时实现上述性能异常事件的确定方法的步骤。
Smart Images
Figure CN122884739A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed storage technology, and in particular to methods and electronic devices for determining performance anomaly events. Background Technology
[0002] Monitoring tools in related technologies (such as input / output statistics tools, or iostat) can only provide aggregated performance metrics at the disk level, such as disk read / write throughput and average input / output operations per second (IOPS). While these metrics can reflect the overall disk operating status, they cannot penetrate the long and complex IO path in a distributed storage system: from the file system receiving requests, to the block device layer processing data, the device driver completing instruction translation, and finally to the actual input / output (IO) operations on the physical disk, each step can be a potential source of performance bottlenecks.
[0003] There is still no effective solution to the problem that related technologies cannot perform full-link tracing and accurate bottleneck location of I / O requests in distributed storage systems. Summary of the Invention
[0004] This application provides a method and electronic device for determining performance anomaly events, in order to at least solve the problem in related technologies that it is impossible to perform full-link tracing and accurate bottleneck location of IO requests in distributed storage systems.
[0005] This application provides a method for determining performance anomaly events, comprising: deploying probes in each layer of a target cluster, and obtaining entry and exit events of at least one IO request at each layer based on the probes; constructing a full-link tracing path for each IO request in the target cluster based on multiple entry and exit events, wherein the full-link tracing path includes entry and exit timestamps of each IO request at each layer; calculating the layer latency of each IO request at each layer and the total latency of each IO request in the target cluster based on the full-link tracing path, and determining performance anomaly events in the target cluster based on the layer latency and the total latency, wherein the performance anomaly events include: a first anomaly event indicating the existence of an abnormal IO request in the target cluster, and a second anomaly event indicating the existence of a performance bottleneck layer in the target cluster.
[0006] This application also provides a device for determining performance anomaly events, comprising: an acquisition module, configured to deploy probes in each layer of a target cluster and acquire entry and exit events of at least one IO request at each layer based on the probes; a construction module, configured to construct a full-link tracing path for each IO request in the target cluster based on multiple entry and exit events, wherein the full-link tracing path includes an entry timestamp and an exit timestamp of each IO request at each layer; and a calculation module, configured to calculate the layer latency of each IO request at each layer and the total latency of each IO request in the target cluster based on the full-link tracing path, and determine performance anomaly events in the target cluster based on the layer latency and the total latency, wherein the performance anomaly events include: a first anomaly event indicating the existence of an abnormal IO request in the target cluster and a second anomaly event indicating the existence of a performance bottleneck layer in the target cluster.
[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing a method for determining performance exception events when executing the computer program.
[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for determining the above-described performance anomaly event.
[0009] This application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for determining the above-mentioned performance abnormality event.
[0010] According to this application, by deploying probes at each level of the target cluster, the entry and exit events of at least one IO request at each level can be obtained. Then, based on multiple entry and exit events, the hierarchical latency and total latency of each IO request in the target cluster, including the entry and exit timestamps of each IO request at each level, can be constructed to determine performance anomalies in the target cluster. In other words, this application solves the problem in related technologies that cannot perform full-link tracing and accurate bottleneck location for IO requests in distributed storage systems by obtaining multiple entry and exit events corresponding to at least one IO request through probes, and then determining the target cluster based on the full-link tracing path. Attached Figure Description
[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a method of determining performance abnormal events according to an embodiment of this application.
[0013] Figure 2 This is a flowchart of an optional method for determining performance anomaly events according to an embodiment of this application;
[0014] Figure 3 This is a flowchart of a distributed system IO performance monitoring method according to an optional embodiment of this application;
[0015] Figure 4 This is a structural block diagram of a device for determining a performance abnormality event according to an embodiment of this application. Detailed Implementation
[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a particular order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a list of steps or units is not necessarily limited to those steps or units expressly listed, but may include other steps or units not expressly listed or inherent to such processes, methods, products, or apparatus.
[0018] The methods and embodiments provided in this application can be run on a computer terminal. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of determining performance abnormal events according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MPU), or programmable logic device (PLD) processing system) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned computer terminal. For example, the camera device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.
[0019] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining performance anomaly events in this embodiment. The processor 102 runs various functional applications and data processing by executing the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a secure text network via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0020] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the camera equipment's communication provider. In one example, the transmission system 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.
[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0022] This embodiment provides a method for determining performance anomaly events, including but not limited to those applied to the aforementioned computer terminal. Figure 2 This is a flowchart of an optional method for determining performance anomaly events according to an embodiment of this application. The process includes the following steps:
[0023] Step S202: Deploy probes in each layer of the target cluster, and obtain at least one IO request entry event and exit event at each layer based on the probes;
[0024] The target cluster includes multiple layers, including but not limited to: file system layer, block device layer, device driver layer, etc.
[0025] Step S204: Construct a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events, wherein the full-link tracing path includes the ingress timestamp and egress timestamp of each IO request at each level;
[0026] Step S206: Calculate the layer latency of each IO request at each layer and the total latency of each IO request in the target cluster based on the end-to-end tracing path, and determine the performance anomaly events in the target cluster based on the layer latency and the total latency, wherein the performance anomaly events include: a first anomaly event indicating the existence of abnormal IO requests in the target cluster, and a second anomaly event indicating the existence of a performance bottleneck layer in the target cluster.
[0027] Based on the above steps, probes are deployed at each level of the target cluster. These probes can then be used to obtain the entry and exit events of at least one IO request at each level. Furthermore, based on multiple entry and exit events, the hierarchical latency and total latency of each IO request in the target cluster, including the entry and exit timestamps of each IO request at each level, are constructed to determine performance anomalies in the target cluster. In other words, this application solves the problem in related technologies that cannot perform full-link tracing and accurate bottleneck location for IO requests in distributed storage systems by using probes to obtain multiple entry and exit events corresponding to at least one IO request and constructing a full-link tracing path in the target cluster, and then determining the target cluster based on this path. This enables full-link tracing and accurate bottleneck location for IO requests in distributed storage systems.
[0028] Optionally, probes are deployed in each layer of the target cluster, and at least one entry and exit event of an IO request at each layer is obtained based on the probes. This includes: deploying probes at the entry and exit points of each layer; and, if any probe is determined to be triggered, obtaining event data corresponding to the target event that triggered the probe, and determining the entry and / or exit event of the IO request corresponding to the target event at the layer where the probe is located based on the event data. The event data includes at least one of the following: the event type corresponding to the target event, and the request identifier of the IO request corresponding to the target event.
[0029] Understandably, probes can be deployed at the entry and exit points of each level. For example, probes can be deployed in kernel functions such as read request entry and write request exit at the file system level; and probes can be deployed at block request initiation and block request completion at the block device level. When an IO request passes through the entry and / or exit point of each level, the corresponding probe can be triggered, and the triggered probe can capture the event data corresponding to the target event generated at that moment. This event data includes, but is not limited to: event type (e.g., entry event, exit event), IO request identifier, timestamp, device number, and data volume.
[0030] For example, when a read request (including I / O requests) is recorded at the file system layer entry point, the event data is: {Event Type: "Entry Point", Request Identifier: "io_0x7f2a1c8b", Timestamp: 1712345678901234567, Level: "ext4"}. When this read request is subsequently recorded at the block device layer exit point, the event data is: {Event Type: "Exit Point", Request Identifier: "io_0x7f2a1c8b", Timestamp: "1712345678901345678", Level: "blkdev"}. The request identifier of the I / O request is used to associate multiple event data points at different levels and times scattered across the target cluster.
[0031] In summary, by deploying probes at the entry and exit points of each level and linking event data with the request identifier of IO requests, end-to-end full-link monitoring of the distributed storage IO path can be achieved without intruding on business operations or affecting performance.
[0032] Optionally, constructing a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events includes: pairing the multiple ingress events and multiple egress events based on the request identifier corresponding to each IO request to generate event pairs for each IO request at at least one level and / or unpaired single events for each IO request at at least one level, wherein the event pairs include: ingress events and egress events for each IO request at each level; sorting the event pairs and / or the single events according to the timestamp of each event in the multiple ingress events and multiple egress events to generate an event sequence for each IO request in the target cluster; and constructing a full-link tracing path for each IO request in the target cluster based on the event sequence.
[0033] It is understandable that an I / O request originating from the application layer of the target cluster must pass through layers such as the file system, block device layer, and device driver layer, with each layer potentially generating multiple independent event records. Due to the existence of the kernel's concurrent processing mechanism, target events from different I / O requests may arrive at the acquisition system interleaved, and some requests may be split at the block device layer (for example, a logical write request may be split into two physical sector writes), resulting in an asymmetry in the number of events and a disordered order of events. Therefore, this application needs to match and reassemble the collected entry and exit events based on the request identifier of the IO request. For example, the request identifier of a read request is io_0x7f2a1c8b, and the following target events related to the read request are captured: File system entry: {Type: entry event, request ID: io_0x7f2a1c8b, timestamp: 1712345678901234567}; File system exit: {Type: exit event, request ID: io_0x7f2a1c8b, timestamp: 1712345678901300000}; At this time, for the file system layer, the events captured at the entry point and the events captured at the exit point can be matched into an event pair.
[0034] In addition, if an event is missing an exit (e.g., due to system crash or probe loss), this event is retained as a single point event and marked as: incomplete IO or missing exit, etc., for anomaly diagnosis and link integrity detection.
[0035] Then, the event pairs and / or single events are sorted based on the timestamps of the event pairs and single events, and then a full-link tracing path is generated based on the event sequence.
[0036] Based on the above embodiments, even in complex scenarios such as IO splitting, concurrent interleaving, and event loss, the actual flow path of a single IO request in the distributed cluster can still be accurately restored by sorting by request identifier and timestamp, avoiding misjudgment caused by event confusion.
[0037] Optionally, constructing a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events includes: if it is determined that the first IO request in the at least one IO request has been split into multiple sub-IO requests in a target layer across multiple levels, obtaining a first ingress event and a first egress event corresponding to each sub-IO request, wherein the multiple ingress events include the first ingress event, and the multiple egress events include the first egress event, and the target cluster includes multiple levels; associating the first ingress event with a request identifier corresponding to the first IO request to generate a first association between the first ingress event and the first IO request, and associating the first egress event with a request identifier corresponding to the first IO request to generate a second association between the first egress event and the first IO request; and constructing a full-link tracing path for the first IO request in the target cluster based on the first association and the second association.
[0038] It is understandable that the target layer mentioned above can be the block device layer.
[0039] When an I / O request reaches the target layer, it may be split into multiple physically independent sub-I / O requests due to factors such as block size limitations of the physical storage medium, Redundant Array of Independent Disks (RAID) striping strategies, or logical volume management mechanisms. For example, a 4KB logical write request might be mapped to two 2KB block device requests, writing to two physical sectors on different disks respectively. In this case, it is necessary to capture the entry and exit events of each sub-I / O request and ensure that the sub-I / O requests are semantically associated with the original I / O requests. For example, the following events: Block device layer entry: {Type: entry event, request ID: io_0x7f2a1c8b, timestamp: 1712345678901310000}; Block device layer exit (primary IO): {Type: exit event, request ID: io_0x7f2a1c8b, timestamp: 1712345678901400000}; Block device layer exit (sub-IO1): {Type: exit request, request ID: io_0x7f2a1c8b_sub1, timestamp: 1712345678901410000}; Block device layer exit (sub-IO2): {Type: exit, request ID: io_0x7f2a1c8b_sub2, timestamp: 1712345678901420000}.
[0040] Two sub-IOs were captured at the exit point. However, because the request identifiers of their parent request and the two sub-requests have an inheritance relationship, it can be determined that the two sub-IOs are derived from the parent request corresponding to the event captured at the block device layer entry point. At this point, a first association can be established between the entry event and the first sub-IO request, and a second association can be established between the exit event of the first sub-IO request and the first sub-IO request. Simultaneously, a first association can be established between the entry event and the second sub-IO request, and a second association can be established between the exit event of the second sub-IO request and the second sub-IO request.
[0041] Then, a full-link tracing path is constructed based on the first and second association relationships.
[0042] In summary, this application achieves complete restoration of complex I / O paths in distributed storage by establishing a correlation between sub-IO events and original requests.
[0043] Optionally, performance anomaly events in the target cluster can be determined based on the hierarchical latency and the total latency. The following are two optional methods for determining performance anomaly events based on hierarchical latency and total latency as exemplified in the embodiments of this application. In practical applications, other methods for determining performance anomaly events based on hierarchical latency and total latency are also within the protection scope of this application.
[0044] Method 1: Determine the ratio between the latency of the second IO request in the first layer of multiple layers and the total latency of the second IO request in the target cluster, wherein the target cluster includes multiple layers; if the ratio is determined to be greater than or equal to a preset ratio, the first layer is determined as the performance bottleneck layer, and the second abnormal event is determined to exist in the target cluster.
[0045] Understandably, Method 1 defines a technical solution based on hierarchical latency and total latency to determine a second anomalous event used to indicate the existence of a performance bottleneck layer in the target cluster. Specifically:
[0046] The ratio of the latency of the second I / O request at each layer to the total latency is determined, and then each layer is identified as a performance bottleneck based on this ratio. For example, if the total latency of the second I / O request is 1200 microseconds, the latency at each layer is as follows: File system layer: 1020 microseconds (85% of total latency); Block device layer: 120 microseconds (10% of total latency); Driver layer: 30 microseconds (2.5% of total latency); Hardware layer: 30 microseconds (2.5% of total latency). The default ratio is 75%.
[0047] Since the ratio of the time spent in the file system layer to the total time spent is greater than the preset ratio, the file system layer can be identified as the performance bottleneck layer, which means that a second abnormal event exists in the target cluster.
[0048] Method 2: Determine the relationship between the total latency of each IO request in the target cluster and the preset latency; if the relationship indicates that the total latency of the third IO request among the at least one IO requests in the target cluster is greater than the preset latency, determine the third IO request as the abnormal IO request, and determine that the first abnormal event exists in the target cluster.
[0049] Understandably, Method 2 defines a technical solution based on total latency to determine the first abnormal event used to indicate the presence of abnormal IO requests in the target cluster. Specifically:
[0050] It is possible to determine the relationship between the total latency of each IO request in the target cluster and the preset latency, and then determine whether there are abnormal IO requests in the target cluster based on this relationship. For example, if the total latency of an IO request in the target cluster is 500 microseconds, while the preset latency is only 300 microseconds, then this IO request can be determined to be an abnormal IO request, that is, the first abnormal event exists in the target cluster.
[0051] Furthermore, a global performance baseline (e.g., P99 latency) can be established to identify I / O requests whose total latency exceeds the normal range. For example, a P99 total latency baseline established based on historical data is 500 microseconds, indicating that 99% of I / O requests have a total latency of no more than 500 microseconds. In this case, if a new I / O request has a total latency of 1800 microseconds, significantly exceeding 500 microseconds, this new I / O request is determined to be an abnormal I / O request, thus identifying the first abnormal event in the target system.
[0052] Optionally, after determining the third IO request as the abnormal IO request and determining that the first abnormal event exists in the target cluster, the method further includes: obtaining an event set corresponding to the third IO request in the end-to-end tracing path, wherein the event set includes a second entry event and a second exit event corresponding to the third IO request; sorting multiple events in the event set according to the timestamp of each event in the event set and the level corresponding to each event in the event set, so as to reconstruct the call path corresponding to the third IO request based on the sorted multiple events, wherein the call path includes at least one of the following: the target level of the delay timeout event of the third IO request, and the triggering context corresponding to the third IO request in the target level; performing structured clustering of the call path and historical call paths to generate a common abnormal pattern between the call path and the historical call path; determining a target diagnostic rule matching the common abnormal pattern in a preset diagnostic rule, and diagnosing the abnormal cause corresponding to the first abnormal event according to the target diagnostic rule.
[0053] Understandably, upon confirming the existence of a first abnormal event, it is necessary to diagnose this event. Specifically, if an I / O request (e.g., the third I / O request) is marked as abnormal due to its total latency exceeding a preset latency, the complete event stream corresponding to this I / O request can be extracted from the end-to-end tracing data corresponding to this latency (the events contained in the complete event stream are the same as those in the event set). The complete event stream contains the entry and exit events of this I / O request at each level. Each event carries a timestamp, its level information, and key context fields, including but not limited to queue depth and file path. These events are then sorted based on their timestamps in the event set to generate the call path corresponding to this I / O request.
[0054] The call path is input into the analysis engine in the form of a structured vector. The output results include, but are not limited to, features such as hierarchical sequence, bottleneck level, bottleneck latency, and context labels.
[0055] Furthermore, the call path is clustered with historical call paths to identify common anomaly patterns between historical call paths and call path matches, and target diagnostic rules are determined based on these common anomaly patterns.
[0056] Optionally, after determining the performance anomaly events in the target cluster based on the hierarchical latency and the total latency, the method further includes: constructing a first visualization view corresponding to the target cluster based on the entry timestamp and exit timestamp of each IO request at each level and the hierarchical latency of each IO request at each level; and / or, determining the distribution status of the performance bottleneck layer in the target cluster based on the performance anomaly events, and constructing a second visualization view corresponding to the target cluster based on the distribution status; and / or, determining the number of occurrences of abnormal IO requests within each time window based on the performance anomaly events, and constructing a third visualization view corresponding to the target cluster based on the number of occurrences; and displaying the first visualization view, and / or the second visualization view, and / or the third visualization view on the display screen of the target device.
[0057] Understandably, after identifying the performance anomalies in the target cluster, a first, second, and third visualization view corresponding to the target cluster can be constructed. Specifically:
[0058] The first visualization view can be a breakdown diagram of the end-to-end latency of a single IO request. Based on the entry and exit timestamps of each IO request at key levels such as the file system, block device layer, device driver, and hardware controller, the independent time consumption of each level is calculated, and the complete execution trajectory of a single IO request is presented in the form of a horizontally stacked Gantt chart or flame chart.
[0059] For example, the time distribution of a write request is as follows: File system layer: 820μs; Block device layer: 110μs; Device driver layer: 40μs; Hardware controller layer: 230μs. A horizontal timeline is constructed based on the execution path of this request, displaying the time bars for each layer from left to right. The length of each bar is proportional to its time consumption, and different layers can be distinguished by different colors (e.g., red for the file system layer, blue for the hardware controller layer). Each bar is labeled with a specific value above and accompanied by contextual information below. Clicking on any path expands the kernel call stack for this I / O request, the number of interrupts triggered, and whether I / O splitting occurred, among other data.
[0060] The second visualization view can be a cluster heatmap of the performance bottleneck layer. It aggregates the performance bottleneck identification results reported by all nodes, counts the most frequently occurring bottleneck level on each node, and displays it in the form of a cluster topology heatmap. For example, a target cluster contains 32 storage nodes. The distribution of major bottlenecks for each node over the past hour is as follows: Rack A (nodes 1–8), the main bottleneck layer is the file system layer, with 7 bottleneck nodes; Rack B (nodes 9–16), the main bottleneck layer is the hardware controller, with 6 bottleneck nodes. On the cluster physical layout diagram, the status of each node can be marked with color gradients, for example: dark red: the file system layer is a high-frequency bottleneck; dark blue: the hardware controller layer is a high-frequency bottleneck.
[0061] The third visualization view can be a time-series trend chart of abnormal I / O requests. The number of abnormal I / O requests is counted at fixed time windows (e.g., every 5 minutes), generating a time-series line chart.
[0062] To better understand the architecture of the above-mentioned method for determining performance anomalies, the above-mentioned interconnected system will be further described below in conjunction with optional embodiments, but this is not intended to limit the technical solutions of the embodiments of this application.
[0063] In the operation and maintenance management of distributed storage systems, IO performance monitoring and root cause diagnosis are directly related to the stable operation and service quality of the system. With the continuous expansion of distributed storage scale and the increasing complexity of business workloads, existing monitoring methods are insufficient to meet the needs of refined operation and maintenance. Because existing monitoring methods lack the ability to measure each stage precisely, when high latency issues occur, maintenance personnel are forced to manually correlate and analyze massive and scattered component logs, such as file system logs, block device layer operation reports, device driver debugging information, and disk data. This not only consumes a significant amount of time to sort out data relationships but is also highly susceptible to diagnostic errors due to human error or information omissions. This cumbersome and inefficient troubleshooting method not only severely prolongs the problem-solving cycle but also greatly affects the maintainability of the distributed storage system, hindering the efficiency and effectiveness of system performance optimization.
[0064] To address the aforementioned issues, this application proposes an optional embodiment of a distributed system IO performance monitoring method and system, introducing bpftrace (a tracing tool) dynamic tracing technology. bpftrace's low-intrusiveness and high flexibility allow for the precise implantation of probes along critical paths without interrupting normal system operation. These probes can capture the entry and exit timestamps of each IO request at each processing layer in real time. Through precise calculation and analysis of these timestamps, fine-grained latency decomposition is achieved. Whether it's the time spent processing requests in the file system, the latency of data forwarding at the block device layer, or the time spent converting device driver instructions, all can be clearly quantified and presented. Furthermore, to make performance data easier to understand and utilize, this application's optional embodiment also integrates IO performance data collected across the entire chain, constructing a unified visualization interface. The visualization interface displays the time distribution of IO requests at each layer in an intuitive chart format. Operations personnel can quickly locate abnormal time-consuming stages without requiring specialized data interpretation skills, significantly lowering the barrier to performance analysis. Simultaneously, abnormal IO requests can be marked, and their entire call path from initiation to completion can be fully traced, clearly reconstructing the entire process of the problem's occurrence.
[0065] The system in an optional embodiment of this application includes the following architecture:
[0066] 1) Data acquisition layer, used for:
[0067] Probe deployment: A standardized set of bpftrace scripts is used to deploy probes in batches on all storage nodes in the cluster through automated tools, ensuring full coverage of data awareness capabilities for all key nodes from the file system to the device driver layer.
[0068] Data collection: Establish a lightweight data pipeline to capture raw performance events generated by probes at each node in real time, and transmit them to the back-end analysis system efficiently and reliably, providing a continuous data stream for subsequent processing.
[0069] 2) Data integration layer, used for:
[0070] First, the collected raw data undergoes an integrity check, filtering out invalid or abnormal records (such as out-of-order timestamps or missing fields in IO events). Then, the heterogeneous formats of data from different levels (such as pathnames in file system events and sector numbers in block-level requests) are parsed and mapped to unified, standardized fields. Based on this, using the request ID (i.e., request identifier) as a unique identifier, independent events generated by the same IO request at different levels are merged. If the request is split at the block device layer, the system will fully record the trajectory of all sub-IOs, ensuring that their association with the parent request is traceable, thereby guaranteeing the integrity and consistency of data throughout the entire chain.
[0071] 3) Data analysis layer, used for:
[0072] Benchmarking Analysis: Calculate the absolute latency of each I / O request at each processing layer (e.g., file system layer, block device layer, driver layer, etc.) and its relative proportion in the entire link, forming a quantifiable performance profile. Specifically: Layer-by-layer performance comparison and bottleneck identification: Compare the real-time calculated layer-by-layer latency and proportion data with a predefined normal I / O process benchmark model. When the latency or proportion of a certain layer significantly exceeds a threshold (e.g., a sudden increase in latency of 50% or a proportion deviating from the benchmark by more than 20%), the system automatically marks it as a potential performance bottleneck layer; Abnormal I / O detection and end-to-end tracing: Synchronously evaluate the overall I / O latency, and mark requests that exceed the normal range (e.g., the P99 baseline) as "abnormal I / O". For these abnormal requests, a full-link tracing mechanism is initiated. Based on their unique request ID and nanosecond-level timestamp sequence, the call path from initiation to termination is completely reconstructed to accurately locate the specific link of performance degradation. Correlation analysis and root cause inference: By combining cross-analysis of abnormal layers and abnormal I / O, common problems (i.e., common abnormal patterns, such as multiple requests timeouts caused by the same driver layer) are identified, and statistically significant root cause conclusions are output to provide a directional basis for performance optimization.
[0073] 4) Visualization layer, used for:
[0074] This system transforms complex I / O link data into intuitive, multi-dimensional visualizations. Based on integrated, standardized data, the visualization layer constructs a full-link latency decomposition diagram, dashboards, and diagnostic reports. It accurately displays the time distribution of requests at each level using flame charts or Gantt charts, presents cluster performance status using node topology heatmaps, and supports multi-granularity time-series trend analysis. It features drill-down capabilities from aggregated views to individual requests, supports dynamic filtering and real-time updates, enabling operations personnel to quickly identify bottlenecks, locate root causes of anomalies, and ultimately complete the entire closed-loop process from data collection and integration to analysis.
[0075] Figure 3 This is a flowchart of a distributed system IO performance monitoring method according to an optional embodiment of this application, applied to the above-mentioned system, such as... Figure 3 As shown:
[0076] Data acquisition layer execution:
[0077] Step S301, IO monitoring;
[0078] I / O monitoring via probes.
[0079] Step S302, Data Collection;
[0080] Data integration layer execution:
[0081] Step S303, data integration;
[0082] Step S304: Generate I / O links;
[0083] Data analysis layer execution:
[0084] Step S305, Data Analysis;
[0085] Step S306: Determine whether the single IO latency exceeds the threshold, and determine whether the average latency ratio of each layer exceeds the threshold.
[0086] If the single IO latency exceeds the threshold, proceed to step S307; if the average latency ratio of each layer exceeds the threshold, proceed to step S308.
[0087] If it is determined that the single IO request does not exceed the threshold and the average latency ratio of each layer does not exceed the threshold, then proceed to step S309.
[0088] Step S307, IO tracing;
[0089] Step S308: Mark the exception level;
[0090] Visualization layer execution:
[0091] Step S309, data display.
[0092] In summary, this application first utilizes bpftrace to achieve a technological leap from coarse-grained statistics to end-to-end nanosecond-level tracing; secondly, through a data integration mechanism, heterogeneous data is transformed into a unified and correlated performance profile; thirdly, through benchmark-based diagnostics, the transformation from manual experience to automated localization is achieved; and finally, with the help of flexible deployment and visualization solutions, a complete closed loop from data collection to decision support is formed.
[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to run the methods of the various embodiments of this application.
[0094] This embodiment also provides a device for determining performance anomaly events. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0095] Figure 4 This is a structural block diagram of a device for determining performance abnormality events according to an embodiment of this application; as shown... Figure 4 As shown, it includes:
[0096] The acquisition module 42 is used to deploy probes in each layer of the target cluster and acquire at least one IO request entry event and exit event in each layer based on the probes.
[0097] Module 44 is used to construct a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events, wherein the full-link tracing path includes the ingress timestamp and egress timestamp of each IO request at each level;
[0098] The calculation module 46 is used to calculate the layer latency of each IO request at each layer and the total latency of each IO request in the target cluster according to the end-to-end tracing path, and to determine the performance abnormal events in the target cluster according to the layer latency and the total latency, wherein the performance abnormal events include: a first abnormal event indicating the existence of abnormal IO requests in the target cluster and a second abnormal event indicating the existence of a performance bottleneck layer in the target cluster.
[0099] By deploying probes at each level of the target cluster using the aforementioned modules, the entry and exit events of at least one IO request at each level can be obtained. Then, based on multiple entry and exit events, the hierarchical latency and total latency of each IO request in the target cluster, including the entry and exit timestamps of each IO request at each level, can be constructed to determine performance anomalies in the target cluster. In other words, this application solves the problem of inability to perform full-link tracing and accurate bottleneck location for IO requests in distributed storage systems by obtaining multiple entry and exit events corresponding to at least one IO request using probes to construct a full-link tracing path in the target cluster, and then determining the target cluster based on the full-link tracing path. This enables full-link tracing and accurate bottleneck location for IO requests in distributed storage systems.
[0100] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the above-described embodiments of the method for determining performance anomalies.
[0101] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the embodiments of the method for determining any of the above-described performance anomaly events when it is run.
[0102] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0103] An embodiment of this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the above-described method for determining performance anomalies.
[0104] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method for determining performance anomalies.
[0105] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0106] The foregoing has provided a detailed description of the method and system for determining performance anomalies, as well as the storage medium and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A method for determining performance anomaly events, characterized in that, include: Deploy probes in each tier of the target cluster and obtain ingress and egress events for at least one IO request at each tier based on the probes; A full-link tracing path for each IO request in the target cluster is constructed based on multiple ingress events and multiple egress events, wherein the full-link tracing path includes the ingress timestamp and egress timestamp of each IO request at each level; The layer latency of each IO request at each layer and the total latency of each IO request in the target cluster are calculated based on the end-to-end tracing path. Performance anomaly events in the target cluster are determined based on the layer latency and the total latency. The performance anomaly events include: a first anomaly event indicating the existence of abnormal IO requests in the target cluster and a second anomaly event indicating the existence of a performance bottleneck layer in the target cluster.
2. The method for determining performance anomaly events according to claim 1, characterized in that, Deploy probes at each tier in the target cluster and obtain ingress and egress events for at least one IO request at each tier based on the probes, including: Probes are deployed at the entrance and exit of each level; If any probe is determined to be triggered, the event data corresponding to the target event that triggered the probe is obtained, and the entry event and / or exit event of the IO request corresponding to the target event at the level where the probe is located is determined according to the event data. The event data includes at least one of the following: the event type corresponding to the target event, and the request identifier of the IO request corresponding to the target event.
3. The method for determining performance anomaly events according to claim 1, characterized in that, Construct a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events, including: The plurality of ingress events and the plurality of egress events are paired based on the request identifier corresponding to each IO request to generate event pairs for each IO request at at least one level and / or unpaired single-point events for each IO request at at least one level, wherein the event pairs include: ingress events and egress events for each IO request at each level; The event pairs and / or the single-point events are sorted according to the timestamp of each of the multiple ingress events and the multiple egress events to generate an event sequence for each IO request in the target cluster; Construct a full-link tracing path for each IO request in the target cluster based on the event sequence.
4. The method for determining performance anomaly events according to claim 1, characterized in that, Construct a full-link tracing path for each IO request in the target cluster based on multiple ingress events and multiple egress events, including: If it is determined that the first IO request in the at least one IO request has been split into multiple sub-IO requests in the target layer of multiple layers, the first entry event and the first exit event corresponding to each sub-IO request are obtained, wherein the multiple entry events include the first entry event, and the multiple exit events include the first exit event, wherein the target cluster includes: multiple layers; Associating the first entry event with the request identifier corresponding to the first IO request to generate a first association between the first entry event and the first IO request; and associating the first exit event with the request identifier corresponding to the first IO request to generate a second association between the first exit event and the first IO request. Based on the first association and the second association, construct the full-link tracing path of the first IO request in the target cluster.
5. The method for determining performance anomaly events according to claim 1, characterized in that, Based on the hierarchical latency and the total latency, the performance anomaly events in the target cluster are determined, including: Determine the ratio between the tier latency of the second IO request in the at least one IO request at the first tier in a plurality of tiers and the total latency of the second IO request in the target cluster, wherein the target cluster includes: a plurality of tiers; If the ratio is determined to be greater than or equal to a preset ratio, the first level is determined to be the performance bottleneck layer, and the second abnormal event is determined to exist in the target cluster.
6. The method for determining performance anomaly events according to claim 1, characterized in that, Determining performance anomaly events in the target cluster based on the hierarchical latency and the total latency further includes: Determine the relationship between the total latency of each IO request in the target cluster and the preset latency; If the size relationship indicates that the total latency of the third IO request in the at least one IO request in the target cluster is greater than the preset latency, the third IO request is identified as the abnormal IO request, and it is determined that the first abnormal event exists in the target cluster.
7. The method for determining performance anomaly events according to claim 6, characterized in that, After identifying the third I / O request as the abnormal I / O request and determining that the first abnormal event exists in the target cluster, the method further includes: Obtain the event set corresponding to the third IO request in the full-link tracing path, wherein the event set includes the second entry event and the second exit event corresponding to the third IO request; The events in the event set are sorted according to the timestamp of each event in the event set and the level corresponding to each event in the event set, so as to reconstruct the call path corresponding to the third IO request based on the sorted events. The call path includes at least one of the following: the target level of the delay timeout event of the third IO request, and the triggering context corresponding to the third IO request in the target level. The target diagnostic rule corresponding to the third IO request is determined based on the call path, and the abnormal cause corresponding to the first abnormal event is diagnosed based on the target diagnostic rule.
8. The method for determining performance anomaly events according to claim 7, characterized in that, The target diagnostic rule corresponding to the third I / O request is determined based on the call path, including: The call paths and historical call paths are structured and clustered to generate common abnormal patterns between the call paths and the historical call paths; Determine the target diagnostic rule that matches the common abnormal pattern from the preset diagnostic rules.
9. The method for determining performance anomaly events according to claim 1, characterized in that, After determining the performance anomaly events in the target cluster based on the hierarchical latency and the total latency, the method further includes: Construct a first visualization view of the target cluster based on the entry timestamp, exit timestamp, and layer-level latency of each IO request at each layer; and / or, Based on the performance anomaly events, determine the distribution status of the performance bottleneck layer in the target cluster, and construct a second visualization view corresponding to the target cluster based on the distribution status; and / or, The number of abnormal IO requests occurring within each time window is determined based on the performance anomaly events, and a third visualization view corresponding to the target cluster is constructed based on the number of occurrences. The first visualization view, and / or the second visualization view, and / or the third visualization view are displayed on the screen of the target device.
10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for determining performance anomaly events as described in any one of claims 1 to 9 when executing the computer program.