A multi-dimensional analysis method and system for long tail problem of distributed system
By statistically analyzing the timing and time consumption information of the last response of each IO in a distributed system from the shard, the shard, node, or execution stage of long-tail problems can be quickly identified, solving the performance bottleneck caused by long-tail latency in existing technologies and improving analysis efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ELECTRONICS CLOUD DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-21
AI Technical Summary
Long-tail latency issues in distributed systems lead to performance bottlenecks, and existing analysis methods are cumbersome and difficult to efficiently locate the root cause.
By analyzing the timing and time consumption information of the slave shards in the final response of each IO by the master shard or client, the shards, nodes, or execution stages that cause long-tail problems can be quickly identified. The multi-dimensional analysis method eliminates the need for a full-link investigation node by node.
It enables efficient and accurate location of long-tail problems, improving the efficiency of performance troubleshooting and operation and maintenance of distributed systems.
Smart Images
Figure CN122431612A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of distributed system task execution technology, specifically relating to a multi-dimensional analysis method, system, computer-readable storage medium, and electronic device for the long-tail problem in distributed systems. Background Technology
[0002] In distributed systems, a single task typically needs to be distributed to multiple roles for collaborative execution. The task can only be considered complete once all roles have finished processing. Taking the multi-replica data writing mechanism of a distributed storage system as an example, the current industry-typical implementations mainly fall into the following two modes:
[0003] 1. Client-side distribution mode
[0004] like Figure 1 As shown, the client sends data to multiple shards in parallel, and each shard independently completes data persistence and returns a response to the client. The client determines that the write operation is complete after receiving confirmation responses from all shards.
[0005] 2. Server-side distribution mode
[0006] like Figure 2 As shown, among the multiple shards, there is a primary shard. The client sends the write request to the primary shard, which is responsible for forwarding the request to the other secondary shards. After each secondary shard completes data persistence, it replies to the primary shard. The primary shard, after successfully writing its own data and receiving responses from all secondary shards, finally returns an acknowledgment to the client.
[0007] The two mechanisms described above each have their own characteristics: In the client-side distribution mode, the server-side shards receive requests almost simultaneously, but the client needs to bear more message distribution and coordination overhead; In the server-side distribution mode, the client only needs to interact with the main shard, which simplifies the client logic, but the request needs to be forwarded through the main shard, there is a time delay in receiving the request from the shard, and the main shard bears a heavier message processing load.
[0008] However, despite their different implementations, both of these models face the same core problem—tail latency. When the number of shards is large (such as typical configurations like 4+2, 8+2, and 16+2 using erasure coding redundancy mechanisms), the response speeds of each shard inevitably differ because their runtime states are difficult to keep completely synchronized. According to the "barrel effect," the completion time of the entire write operation depends on the slowest-responding shard; the more shards there are, the more pronounced the tail latency problem becomes. For performance-sensitive distributed systems, tail latency has become a significant bottleneck restricting overall performance.
[0009] The causes of the long tail problem involve multiple factors:
[0010] Hardware and node-level anomalies: The storage medium corresponding to a specific shard has performance degradation (such as a slow disk), or the memory, CPU and other resources of the node it is located on are abnormal;
[0011] Software and path-level latency: All shards show no significant external anomalies, but response latency occurs at specific stages of the IO execution path or in external components they depend on, forming a hidden long tail.
[0012] Furthermore, locating and analyzing long-tail problems is technically challenging. This is because initial performance degradation in a system is often difficult to attribute directly to the long-tail effect. Conventional analysis methods require a cumbersome sampling process, collecting operational information node by node, segment by segment, and I / O segment by I / O, and then systematically analyzing the data to identify long-tail problems and pinpoint their root causes. Summary of the Invention
[0013] To overcome the aforementioned shortcomings in existing technologies and to accurately and efficiently identify and analyze long-tail problems, this application proposes a novel multi-dimensional analysis method for long-tail problems in distributed systems.
[0014] The following technical strategies are mainly adopted in this application:
[0015] 1. By analyzing the last response from the slave shard for each IO operation through the primary shard or client statistics, you can quickly determine whether a slow shard or node is causing long-tail problems by simply viewing the statistics of a single shard or node.
[0016] 2. By statistically analyzing the execution time of each stage of the slave shard in the final response of each IO through the primary shard or client, it is possible to quickly determine whether a slow IO stage is causing long-tail problems by simply viewing the statistics of a single shard or node.
[0017] Specifically, this application provides the following technical solutions:
[0018] The first aspect of this application provides a multi-dimensional analysis method for the long-tail problem in distributed systems, such as... Figure 3 As shown, it includes the following steps:
[0019] Obtain the response information of each task execution unit in the distributed system to the task. The response information includes the timing information of each task execution unit completing the task and / or the time consumption information of each stage of task execution by each task execution unit; wherein:
[0020] Based on the timing information, the frequency of each task execution unit as the last completed unit within a preset period is counted, and task execution units with long-tail problems are identified based on this frequency.
[0021] Based on the time consumption information, the frequency of each execution stage being identified as the slowest stage within a preset period is counted, and the execution stage that causes the long tail problem is identified based on this frequency.
[0022] Furthermore, in the method of this application, obtaining the response information of each task execution unit in the distributed system to the task includes:
[0023] In the server-side distribution mode, the main task execution unit collects the response information of each slave task execution unit to the task;
[0024] Alternatively, in client-distribution mode, the client collects the response information of each task execution unit to the task.
[0025] Furthermore, in the method of this application, the main task execution unit or client records the timing of the completion of tasks by each slave task execution unit or each task execution unit at the granularity of a single task, and determines the final completion unit of each task.
[0026] Furthermore, in the method of this application, the step of identifying the task execution unit with long-tail problem based on the frequency includes: sorting the frequency of each task execution unit by size; if the frequency of a certain task execution unit is significantly higher than that of other task execution units, then the task execution unit is determined to be the source of the long-tail problem.
[0027] The step of identifying the execution stage that causes the long-tail problem based on the frequency includes: sorting the frequency of each execution stage by size; if the frequency of a certain execution stage is significantly higher than that of other execution stages, then the execution stage is determined to be the bottleneck stage that causes the long-tail problem.
[0028] Furthermore, the method of this application also includes:
[0029] The frequencies of multiple task execution units on the same physical node are aggregated, and the frequency of each node as the final completion node is counted at the node level. Based on this frequency, nodes with long-tail problems are identified.
[0030] Furthermore, in the method of this application, the time consumption information of each stage of the task execution by each task execution unit is obtained in the following way:
[0031] The task execution process is broken down into multiple execution stages. Each task execution unit counts the time consumed in each stage during the task execution process and sends the statistical results to the information collection end through a response message.
[0032] Furthermore, in the method of this application, the execution phase includes: a message receiving phase, a queue scheduling phase, a data persistence phase, a metadata update phase, and a response reply phase.
[0033] Furthermore, the method of this application also includes:
[0034] The statistical results are output and displayed in a preset format, which includes at least one of log output, monitoring interface display, table output, and tree diagram output.
[0035] The second aspect of this application provides a multi-dimensional analysis system for the long-tail problem in distributed systems. The system, when running, implements the steps of the aforementioned multi-dimensional analysis method for the long-tail problem in distributed systems, such as... Figure 4 As shown, the system includes:
[0036] The information acquisition module is used to acquire the response information of each task execution unit in the distributed system to the task. The response information includes the timing information of each task execution unit completing the task and / or the time consumption information of each task execution unit in each stage of the task execution.
[0037] The timing information analysis module is used to count the frequency of each task execution unit as the last completed unit within a preset period based on the timing information, and to identify task execution units with long-tail problems based on the frequency.
[0038] The time consumption information analysis module is used to count the frequency of each execution stage as the slowest stage within a preset period based on the time consumption information, and to identify the execution stage that causes the long tail problem based on the frequency.
[0039] The output display module is used to output and display statistical results in a preset format, which includes at least one of log output, monitoring interface display, table output, and tree diagram output.
[0040] Furthermore, in this application system, the information acquisition module includes:
[0041] The server-side collection unit is used to collect the response information of each slave task execution unit to the task in the server-side distribution mode.
[0042] The client collection unit is used to collect the response information of each task execution unit to the task in the client distribution mode.
[0043] A third aspect of this application provides an electronic device, including: a memory and a processor;
[0044] Memory: Used to store computer programs;
[0045] Processor: Used to execute the computer program to implement the steps of the aforementioned multi-dimensional analysis method for the long-tail problem of distributed systems.
[0046] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned multi-dimensional analysis method for the long-tail problem of distributed systems.
[0047] In summary, this application proposes a universal, efficient, multi-dimensional, and rapid analysis method for long-tail problems commonly found in distributed systems. By statistically analyzing the completion sequence and time consumption of each stage of task execution units at a single point, the source of long-tail problems can be quickly located without needing to investigate the entire chain node by node. This method combines the advantages of high analysis efficiency and high diagnostic accuracy. It also supports multi-dimensional statistics at the sharding and node granularities, as well as bottleneck identification at the execution stage level, achieving accurate location of long-tail problems from macro to micro levels. This effectively improves the efficiency of troubleshooting performance problems and operational efficiency in distributed systems.
[0048] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description
[0049] To more clearly illustrate the background technology and the technical solutions of this application, the accompanying drawings involved in the description of this application will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.
[0050] Figure 1 This is a schematic diagram of the client IO distribution mode in the existing technology.
[0051] Figure 2 This is a schematic diagram of the server-side IO distribution mode in the existing technology.
[0052] Figure 3 This is a flowchart illustrating the overall implementation of the multi-dimensional analysis method for the long-tail problem in distributed systems presented in this application.
[0053] Figure 4 This is a structural diagram of the system composition for the multi-dimensional analysis of the long-tail problem in the distributed system proposed in this application.
[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0056] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.
[0057] Example: A multi-dimensional analysis method for the long-tail problem in distributed systems
[0058] This invention aims to provide an efficient, multi-dimensional method for analyzing long-tail problems in distributed systems. This embodiment uses the server-side master-shard distribution mode as an example for illustration.
[0059] 1. On the master shard S1, the slave shards that received the last response for each IO can be counted. Using each IO as the granularity, the number of times the last response of other slave shards is counted, and then sorted in descending order of the number of responses. Because the data shuffling algorithm of the distributed system can ensure that each shard will communicate with almost all other shards and act as master and slave to each other, it is only necessary to check the statistical results of one shard or one node to quickly identify whether a certain shard or a certain node is slow and causing long-tail problems, thereby improving the efficiency of long-tail problem location.
[0060]
[0061] Alternatively, data from multiple shards on the same node can be further aggregated and then statistically analyzed at the node level.
[0062]
[0063] If the statistical results shown in the table above appear, or if only S3 / Node3 appears in the statistics, it can be determined that the long tail is caused by the slowness of S3 / Node3. Based on the statistical information from multiple dimensions, it can be quickly determined whether the problem is with a certain shard or a certain node.
[0064] The client-side distribution model is similar, except that the information aggregation, statistics, and display are handled on the client side.
[0065] 2. Decompose the IO execution process on each shard into multiple stages, such as message reception step 1, queue scheduling step 2, data write to disk step 3, metadata update step 4, and message recovery step 5 (this is just an example; different systems can be divided according to their actual situations). Each shard counts the time spent in each stage during the IO execution process, and then transmits this statistical information to the main shard in the network message it replies to. In this way, the main shard can count the time spent in each stage of each IO in the last reply from the shard, and thus quickly identify whether the long tail is caused by the slow execution of a certain stage of the IO.
[0066]
[0067] Similarly, if the count in step 4 is much larger than that in other stages, or if only step 4 has statistics, it can be quickly determined that the long tail is caused by slow metadata updates, and then the reasons for the slow metadata updates can be further analyzed.
[0068] The statistical information above can be displayed in different ways depending on the implementation of different systems. For example, it can be printed to logs or displayed directly on the monitoring web interface, in the form of tables, tree diagrams, etc. Alternatively, the analysis results can be given directly through further automated data processing.
[0069] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.
[0070] like Figure 5 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned multi-dimensional analysis method for the long-tail problem of distributed systems.
[0071] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).
[0072] The operations described in this application can be implemented by writing computer program code using one or more programming languages or a combination thereof. The programming languages include, but are not limited to, the following types:
[0073] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.
[0074] Conventional procedural programming languages, such as "C" or similar programming languages.
[0075] The execution methods of program code include, but are not limited to:
[0076] It runs entirely on the user's computer;
[0077] Part of it executes on the user's computer, and part of it executes on a remote computer;
[0078] Execute as a standalone software package;
[0079] It is executed entirely on a remote computer or server.
[0080] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.
[0081] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the multi-dimensional analysis method for the long-tail problem of distributed systems disclosed in this application.
[0082] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:
[0083] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.
[0084] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.
[0085] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.
[0086] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.
[0087] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.
[0088] (6) Fiber optic storage devices: storage media based on fiber optic technology.
[0089] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.
[0090] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.
[0091] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.
[0092] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.
[0093] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.
[0094] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the multi-dimensional analysis method for long-tail problems in distributed systems disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.
[0095] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.
[0096] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A multi-dimensional analysis method for the long-tail problem in distributed systems, characterized in that, Includes the following steps: Obtain the response information of each task execution unit in the distributed system to the task. The response information includes the timing information of each task execution unit completing the task and / or the time consumption information of each stage of task execution by each task execution unit; wherein: Based on the timing information, the frequency of each task execution unit as the last completed unit within a preset period is counted, and task execution units with long-tail problems are identified based on this frequency. Based on the time consumption information, the frequency of each execution stage being identified as the slowest stage within a preset period is counted, and the execution stage that causes the long tail problem is identified based on this frequency.
2. The method according to claim 1, characterized in that, The process of obtaining the response information of each task execution unit in the distributed system to the task includes: In the server-side distribution mode, the main task execution unit collects the response information of each slave task execution unit to the task; Alternatively, in client-distributed mode, the client collects the response information of each task execution unit to the task.
3. The method according to claim 2, characterized in that, The main task execution unit or client records the timing of task completion by each slave task execution unit or each task execution unit at the granularity of a single task, and determines the final completion unit for each task.
4. The method according to claim 1, characterized in that, The step of identifying task execution units with long-tail problems based on the frequency includes: sorting the frequency of each task execution unit by size; if the frequency of a certain task execution unit is significantly higher than that of other task execution units, then the task execution unit is determined to be the source of the long-tail problem. The step of identifying the execution stage that causes the long-tail problem based on the frequency includes: sorting the frequency of each execution stage by size; if the frequency of a certain execution stage is significantly higher than that of other execution stages, then the execution stage is determined to be the bottleneck stage that causes the long-tail problem.
5. The method according to claim 1, characterized in that, The method further includes: The frequencies of multiple task execution units on the same physical node are aggregated, and the frequency of each node as the final completion node is counted at the node level. Based on this frequency, nodes with long-tail problems are identified.
6. The method according to claim 1, characterized in that, The time consumption information for each stage of the task execution by each task execution unit is obtained through the following methods: The task execution process is broken down into multiple execution stages. Each task execution unit counts the time consumed in each stage during the task execution process and sends the statistical results to the information collection end through a response message.
7. The method according to claim 6, characterized in that, The execution phases include: message receiving phase, queue scheduling phase, data persistence phase, metadata update phase, and response reply phase.
8. The method according to claim 1, characterized in that, The method further includes: The statistical results are output and displayed in a preset format, which includes at least one of the following: log output, monitoring interface display, table output, and tree diagram output.
9. A multi-dimensional analysis system for long-tail problems in distributed systems, characterized in that, The steps of implementing the multi-dimensional analysis method for the long-tail problem of distributed systems as described in any one of claims 1-8 during system runtime include: The information acquisition module is used to acquire the response information of each task execution unit in the distributed system to the task. The response information includes the timing information of each task execution unit completing the task and / or the time consumption information of each task execution unit in each stage of the task execution. The timing information analysis module is used to count the frequency of each task execution unit as the last completion unit within a preset period based on the timing information, and to identify task execution units with long-tail problems based on the frequency. The time consumption information analysis module is used to count the frequency of each execution stage as the slowest stage within a preset period based on the time consumption information, and to identify the execution stage that causes the long tail problem based on the frequency. The output display module is used to output and display statistical results in a preset format, which includes at least one of log output, monitoring interface display, table output, and tree diagram output.
10. The system according to claim 9, characterized in that, The information acquisition module includes: The server-side collection unit is used to collect the response information of each slave task execution unit to the task in the server-side distribution mode. The client collection unit is used to collect the response information of each task execution unit to the task in the client distribution mode.