Bottleneck diagnosis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a bottleneck diagnosis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining each module in a three-order association architecture which comprises a TCP module, a block equipment module and an NVME driving module, determining an event timestamp of each module, carrying out the clock calibration of the event timestamp of each module, and carrying out the clock calibration of the event timestamp of each module. According to the method, the delay of each module in the third-order association architecture is calculated based on the calibrated event timestamp of each module, the bottleneck in the third-order association architecture is determined from each delay, association is carried out based on the TCP module, the block device module and the NVME driving module, and time calibration is carried out, so that bottleneck diagnosis is more accurate.

Description

Technical Field

[0001] This invention relates to the field of bottleneck analysis, and more specifically, to a bottleneck diagnosis method, apparatus, electronic device, and storage medium. Background Technology

[0002] On the collaborative path of distributed storage and high-performance networks, the existing observation system exhibits a three-level fracture of "island-fault-blind zone", making it almost impossible to complete end-to-end performance diagnosis across TCP→BIO→NVMe.

[0003] Currently, the network layer TCP tracepoint, block layer BIO probe, and NVMe layer nvme-command probe operate independently; the lack of a unified transaction ID prevents BIO requests derived from a TCP stream from being automatically linked with NVMe commands.

[0004] Traditional tools cannot distinguish between bottleneck types such as network congestion, block device queuing, and NVMe firmware latency. Summary of the Invention

[0005] The purpose of this invention is to provide a bottleneck diagnosis method, device, electronic device, and storage medium that can accurately locate bottlenecks.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a bottleneck diagnosis method, the method comprising: Obtain the modules in the three-level association architecture, wherein the three-level association architecture includes a TCP module, a block device module, and an NVME driver module, and the TCP module, the block device module, and the NVME driver module are associated; Determine the event timestamps for each of the aforementioned modules; Clock calibration is performed on the event timestamps of each module; Based on the calibrated event timestamps of each module, the latency of each module in the third-order correlation architecture is calculated; From the aforementioned delays, the bottlenecks in the third-order correlation architecture are identified.

[0007] In an optional implementation, the step of clocking the event timestamps of each module includes: Obtain the earliest timestamp from each of the event timestamps; The module corresponding to the earliest timestamp is used as the base time. Based on the reference time, the event timestamps of other modules besides the module corresponding to the reference time are aligned.

[0008] In an optional implementation, the module corresponding to the reference time is a TCP module, and the step of aligning the event timestamps of other modules besides the module corresponding to the reference time based on the reference time includes: Calculate the first relative time offset between the event timestamp of the block device module and the reference time; Calculate the second time offset between the event timestamp of the NVME driver module and the reference time; Determine the clock drift value; The first relative time offset is compensated based on the clock drift value to achieve alignment of the event timestamps of the block device module; The second relative time offset is compensated based on the clock drift value to achieve alignment of the event timestamps of the NVME drive module.

[0009] In an optional implementation, the TCP module corresponds to a first mapping table, the block device module corresponds to a second mapping table, and the NVME driver module corresponds to a third mapping table; The first mapping table contains the correspondence between request chain IDs and the complete state of the entire I / O chain; The second mapping table contains the correspondence between the request chain ID and the BIO structure address; The third mapping table contains the correspondence between the BIO structure address and the NVMe command identifier.

[0010] In an optional implementation, the step of calculating the delay of each module in the third-order correlation architecture based on the calibrated event timestamps of each module includes: Based on the calibrated event timestamps of each module, the network layer latency, BIO queuing latency, and NVME processing latency are determined. Determine the completion time of the NVME driver module and the event timestamp of the TCP module, and calculate the total latency.

[0011] In an optional implementation, the step of determining the bottleneck in the third-order correlation architecture from the respective delays includes: Calculate the ratio of the network layer latency to the total latency, and use it as the bottleneck contribution of the TCP module; Calculate the ratio of the BIO queuing delay to the total delay, and use it as the bottleneck contribution of the block device module; The ratio of the NVME processing latency to the total latency is calculated and used as the bottleneck contribution of the NVME driver module. The bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module are compared with the latency threshold respectively. Obtain a target bottleneck contribution greater than the intermediate latency threshold from the bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module. The module corresponding to the target bottleneck contribution is taken as the bottleneck output.

[0012] In an optional implementation, the step of determining the bottleneck in the third-order correlation architecture from the respective delays includes: Compare the network layer latency with a network layer latency threshold; If the network layer latency is greater than the network layer latency threshold, the device parameters of the TCP module are determined, wherein the device parameters include at least one of the following: queue utilization, thread load, and device error conditions; The system determines whether the TCP module is a bottleneck based on the TCP device parameters.

[0013] Secondly, embodiments of this application provide a bottleneck diagnostic device, the device comprising: The acquisition module is used to acquire each module in the three-level association architecture, wherein the three-level association architecture includes a TCP module, a block device module, and an NVME driver module, and the TCP module, the block device module, and the NVME driver module are associated. A determination module is used to determine the event timestamps of each of the modules; A calibration module is used to perform clock calibration on the event timestamps of each of the modules. The calculation module is used to calculate the latency of each module in the third-order correlation architecture based on the calibrated event timestamps of each module. The determining module is further configured to determine the bottleneck in the third-order correlation architecture from each of the delays.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the bottleneck diagnosis method.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the bottleneck diagnosis method.

[0016] This application has the following beneficial effects: This application obtains the event timestamps of each module in a third-order correlation architecture, which includes a TCP module, a block device module, and an NVME driver module. It then performs clock calibration on the event timestamps of each module, calculates the latency of each module in the third-order correlation architecture based on the calibrated event timestamps, identifies bottlenecks in the third-order correlation architecture from the latency, and performs correlation and time calibration based on the TCP module, block device module, and NVME driver module, making bottleneck diagnosis more accurate. Attached Figure Description To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A block diagram of an electronic device provided in an embodiment of the present invention; Figure 2 This is one of the flowcharts illustrating a bottleneck diagnosis method provided in an embodiment of the present invention; Figure 3 This is a second schematic flowchart of a bottleneck diagnosis method provided in an embodiment of the present invention; Figure 4 The third schematic flowchart of a bottleneck diagnosis method provided in an embodiment of the present invention; Figure 5 The fourth schematic flowchart of a bottleneck diagnosis method provided in an embodiment of the present invention; Figure 6 The fifth schematic flowchart of a bottleneck diagnosis method provided in an embodiment of the present invention; Figure 7 A flowchart illustrating a bottleneck diagnosis method provided in an embodiment of the present invention is shown in Figure 6. Figure 8 This is a structural block diagram of a bottleneck diagnostic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0022] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0023] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] Through extensive research, the inventors discovered that existing technologies independently collect data from the BIO layer, NVMe layer, and TCP layer, lacking cross-layer correlation timing, thus failing to establish a complete causal chain of TCP stream -> BIO request -> NVMe command.

[0025] Furthermore, since events at different levels use different clock sources, it is impossible to accurately align the time series, making it difficult to distinguish between actual processing time and queuing time, resulting in inaccurate bottleneck analysis.

[0026] Furthermore, traditional tools cannot distinguish between bottleneck types such as network congestion, block device queuing, and NVMe firmware latency. Existing technologies lack quantitative indicators to determine the contribution of each layer to the total latency, resulting in poor accuracy in identifying bottlenecks.

[0027] In view of the above-mentioned problems, this embodiment provides a bottleneck diagnosis device, electronic device, and storage medium that can acquire the modules in a third-order correlation architecture, wherein the third-order correlation architecture includes a TCP module, a block device module, and an NVME driver module. It determines the event timestamps of each module, performs clock calibration on the event timestamps of each module, calculates the latency of each module in the third-order correlation architecture based on the calibrated event timestamps, identifies the bottleneck in the third-order correlation architecture from the latency, and performs correlation based on the TCP module, block device module, and NVME driver module, and calibrates the time, making the bottleneck diagnosis more accurate. The solution provided in this embodiment will be described in detail below.

[0028] This embodiment provides an electronic device capable of diagnosing bottlenecks. In one possible implementation, the electronic device can be a user terminal, such as, but not limited to, a server, smartphone, personal computer (PC), tablet computer, personal digital assistant (PDA), mobile internet device (MID), etc.

[0029] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application. The electronic device 100 may further include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0030] The electronic device 100 includes a bottleneck diagnostic device 110, a memory 120, and a processor 130.

[0031] The components of the memory 120 and processor 130 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The bottleneck diagnostic device 110 includes at least one software function module that can be stored in the memory 120 in the form of software or firmware or embedded in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the memory 120, such as the software function modules and computer programs included in the bottleneck diagnostic device 110.

[0032] The memory 120 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 120 is used to store programs, and the processor 130 executes the programs after receiving execution instructions.

[0033] Please refer to Figure 2 , Figure 2 For application Figure 1 The flowchart below shows a bottleneck diagnosis method for an electronic device 100, and the method includes a detailed description of each step.

[0034] S201: Obtain the modules in the third-order relational architecture.

[0035] The three-level associative architecture includes a TCP module, a block device module, and an NVME driver module, which are associated with each other.

[0036] S202: Determine the event timestamps for each module.

[0037] S203: Perform clock calibration on the event timestamps of each module.

[0038] S204: Calculate the latency of each module in the third-order correlation architecture based on the calibrated event timestamps of each module.

[0039] S205: Identify the bottleneck in the third-order correlation architecture from the various delays.

[0040] It should be noted that the TCP module corresponds to the first mapping table, the block device module corresponds to the second mapping table, and the NVME driver module corresponds to the third mapping table. The first mapping table contains the correspondence between the request chain ID and the complete state of the entire I / O chain, the second mapping table contains the correspondence between the request chain ID and the BIO structure address, and the third mapping table contains the correspondence between the BIO structure address and the NVMe command identifier.

[0041] Based on the first mapping table corresponding to the TCP module, the second mapping table corresponding to the block device module, and the third mapping table corresponding to the NVME driver module, the TCP module, block device module, and NVME driver module in the three-order association architecture can be interconnected, thus solving the observation island problem.

[0042] The three-tiered relational architecture, comprising the TCP module, block device module, and NVME driver module, is shown below: +---------------------+| Ceph OSD Process|+----------+----------+|+----------v----------+| TCP Layer Trace||- tcp_sendmsg| |- tcp_recvmsg+----------+----------+|+----------v----------+| BIO Layer Trace||- block_rq_issue| |- block_rq_complete| +----------+----------+|+----------v----------+| NVMe Driver Trace||- nvme_setup_cmd| |- nvme_complete| +----------+----------+|+----------v----------+| eBPF Correlation||Engine| |- Request chain construction||- Time axis alignment||- Loss quantization|+---------------------+ The TCP module interacts with the underlying block device module, and the block device module interacts with the underlying NVME driver module.

[0043] The TCP module corresponds to the first mapping table, the block device module corresponds to the second mapping table, and the NVME driver module corresponds to the third mapping table, all constructed in the following manner: When the TCP module sends a data packet, an initial request chain is created and the skb address is recorded in the first mapping table `chain_map`. When the block device module initiates a BIO request, the corresponding request chain is looked up by process / thread ID and associated with the BIO address. Simultaneously, a second mapping table `bio_chain_map` is defined, containing the mapping between BIO structure addresses and request chain IDs. A third mapping table `nvme_cmd_map` contains the mapping between BIO structure addresses and NVMe command identifiers, where the command identifier includes command ID + CPU. This provides the infrastructure for the subsequent NVMe driver module to complete the cross-layer association between events and TCP module requests, enabling end-to-end tracing of the storage I / O path.

[0044] The mapping table can be constructed using the following code: / / Mapping table definition (at the top of the file) struct {__uint(type, BPF_MAP_TYPE_HASH);__uint(max_entries, MAX_CHAINS);__type(key, __u64); / / Request chain ID __type(value,struct io_chain);} chain_map SEC(".maps");struct {__uint(type, BPF_MAP_TYPE_HASH);__uint(max_entries, MAX_CHAINS);__type(key, __u64); / / BIO address __type(value, __u64); / / Request chain ID} bio_chain_map SEC(".maps");struct nvme_cmd_map_key {__u64 cmd_id;__u32 cpu_id; / / Add CPU ID to avoid command ID conflicts between different CPUs};struct nvme_cmd_map_value {__u64 bio_addr;};struct {__uint(type, BPF_MAP_TYPE_HASH);__uint(max_entries, MAX_CMDS);__type(key, struct nvme_cmd_map_key);__type(value, struct nvme_cmd_map_value);} nvme_cmd_map SEC(".maps"); / / TCP module captures skbint trace_tcp_sendmsg(struct pt_regs *ctx) {struct io_chain chain ={};chain.tcp_skb = (__u64)PT_REGS_PARM2(ctx); / / Record skb address bpf_map_update_elem(&chain_map,&pid_tid,&chain); / / Store in mapping} / / Block device module association request int trace_block_rq_issue(struct pt_regs *ctx) {struct io_chain *chain = bpf_map_lookup_elem(&chain_map,&pid_tid); if (chain) {chain->bio_req = (__u64)PT_REGS_PARM1(ctx); / / Association bio request}}. To improve the accuracy of bottleneck diagnosis, clock calibration is required. There are several ways to implement clock calibration based on the event timestamps of each module. One implementation method is as follows: Figure 3 As shown, it includes the following steps: S301: Get the earliest timestamp from the event timestamps.

[0045] S302: Use the module corresponding to the earliest timestamp as the base time.

[0046] S303: Align event timestamps of modules other than those corresponding to the base time based on the base time.

[0047] The clocks of the TCP module, block device module, and NVMe driver module are unified. A reference time, or base time, is selected from the clocks of the TCP module, block device module, and NVMe driver module. This base time is then subtracted from the clocks of the other modules to obtain the offset time relative to this base. Based on this offset time, the clocks of the TCP module, block device module, and NVMe driver module are aligned.

[0048] Specifically, the reference time is selected from the clocks of the TCP module, the block device module, and the NVMe driver module by determining the event timestamps of each module, and the module corresponding to the earliest timestamp is used as the reference time.

[0049] The module corresponding to the base time is the TCP module. Based on the base time, there are multiple ways to align the event timestamps of modules other than the module corresponding to the base time. In one implementation, such as... Figure 4 As shown, it includes the following steps: S401: Calculate the first relative time offset between the event timestamp of the block device module and the base time.

[0050] S402: Calculate the second time offset between the event timestamp of the NVME driver module and the base time.

[0051] S403: Determine the clock drift value.

[0052] S404: Compensate for the first relative time offset based on the clock drift value to align the event timestamps of the block device module.

[0053] S405: Compensate for the second relative time offset based on the clock drift value to align the event timestamps of the NVME driver module.

[0054] In the module corresponding to the reference time, which is the TCP module, the first relative time offset between the event timestamp of the block device module and the reference time is calculated, and the second time offset between the event timestamp of the NVME driver module and the reference time is calculated. The first time offset is added to the clock drift value to complete the synchronization of the event timestamp of the block device module based on the reference time. The second time offset is added to the clock drift value to complete the synchronization of the event timestamp of the NVME driver module based on the reference time.

[0055] Specifically, the event timestamps of modules other than the module corresponding to the base time are aligned based on the following code.

[0056] / / Timeline alignment function static void align_timestamps(struct io_chain_value *chain){u64 base_ts = chain->tcp_send_ts; / / Calculate relative time offset (ensure all timestamps are based on the same baseline) chain->bio_issue_ts -= base_ts; chain->nvme_submit_ts -= base_ts; / / Clock drift compensation (based on CPU TSC calibration) u64 tsc_delta = (bpf_ktime_get_ns()base_ts) -(bpf_get_smp_processor_id() * TIME_CALIB_FACTOR); chain->bio_issue_ts += tsc_delta; chain->nvme_submit_ts += tsc_delta;} tsc_delta is the clock drift value.

[0057] bpf_ktime_get_ns() is the current system time, bpf_get_smp_processor_id() is the current kernel sequence number, and TIME_CALIB_FACTOR is the current scheduling time.

[0058] There are multiple ways to calculate the latency of each module in a third-order correlation architecture based on the calibrated event timestamps of each module. In one implementation, such as... Figure 5 As shown, it includes the following steps: S501: Based on the calibrated event timestamps of each module, determine the network layer latency, BIO queuing latency, and NVME processing latency.

[0059] S502: Determine the completion time of the NVME driver module and the event timestamp of the TCP module, and calculate the total delay.

[0060] Specifically, network layer latency is calculated as the difference between the event timestamp of the calibrated block device module and the time when the event enters the TCP module.

[0061] BIO queuing latency is calculated as the difference between the submission time of the NVME driver module and the event timestamp of the calibrated block device module.

[0062] NVMe processing latency is calculated as the difference between the completion time and the submission time of the NVMe driver module.

[0063] The total latency is calculated as the difference between the completion time of the NVME driver module and the time when the event enters the TCP module.

[0064] Network layer latency, BIO queuing latency, NVME processing latency, and total latency can be implemented using the following code: Network layer latency = bio_issue_tsTCP latency - tcp_send_ts; BIO queue delay = nvme_submit_ts - bio_issue_tsBIO; NVMe processing latency = nvme_complete_ts - nvme_submit_ts; Total latency = nvme_complete_ts - tcp_send_tsTCP.

[0065] There are multiple ways to identify bottlenecks in a third-order correlation architecture from various delays. In one implementation, such as... Figure 6 As shown, it includes the following steps: S601: Calculate the ratio of network layer latency to total latency, as the bottleneck contribution of the TCP module.

[0066] S602: Calculate the ratio of BIO queuing delay to total delay as the bottleneck contribution of the block device module.

[0067] S603: Calculate the ratio of NVME processing latency to total latency, as the bottleneck contribution of the NVME driver module.

[0068] S604: Compare the bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module with the latency threshold respectively.

[0069] S605: Obtain a target bottleneck contribution greater than the medium latency threshold from the bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module.

[0070] S606: The module corresponding to the target bottleneck contribution is used as the bottleneck output.

[0071] For example, the bottleneck contribution of the TCP module is 48%, the block device module is 51%, and the NVMe driver module is 42%. When the latency threshold is 50%, the bottleneck contribution of the TCP module (48%) is compared to the latency threshold. The TCP module's bottleneck contribution is less than the latency threshold of 50%. The block device module's bottleneck contribution (51%) is compared to the latency threshold. The block device module's bottleneck contribution is greater than the latency threshold of 50%. Therefore, the block device module's bottleneck contribution is taken as the target bottleneck contribution, and the block device module is diagnosed as the bottleneck. Finally, the NVMe driver module's bottleneck contribution is compared to the latency threshold. The NVMe driver module's bottleneck contribution is less than the latency threshold.

[0072] Another way to identify bottlenecks in a third-order associative architecture from various delays, such as... Figure 7 As shown, it includes the following steps: S701: Compare the network layer latency with the network layer latency threshold.

[0073] S702: Determine the device parameters of the TCP module when the network layer latency is greater than the network layer latency threshold.

[0074] The device parameters include at least one of the following: queue utilization, thread load, and device error conditions.

[0075] S703: Determine whether the TCP module is a bottleneck based on TCP device parameters.

[0076] Specifically, different thresholds can be preset for network layer latency, BIO queuing latency, and NVME processing latency, i.e., network layer latency corresponds to network layer latency threshold, BIO queuing latency corresponds to BIO queuing latency threshold, and NVME processing latency corresponds to NVME processing latency threshold.

[0077] For example, the network layer latency threshold can be set to 200 μs, the BIO queuing latency threshold can be set to 150 μs, and the NVME processing latency threshold can be set to 100 μs.

[0078] The settings for network layer latency threshold, BIO queuing latency threshold, and NVME processing latency threshold can be based on the actual system conditions, and this application embodiment does not impose specific restrictions on them.

[0079] For example, when determining whether network layer latency is a bottleneck, the network layer latency is compared with a network layer latency threshold. If the network layer latency is greater than the network layer latency threshold, the device parameters of the TCP module are determined. For example, the device parameter of the TCP module is determined to be queue utilization. The queue utilization is compared with a preset queue utilization. If the queue utilization of the TCP module is greater than the preset queue utilization, then the TCP module is determined to be a bottleneck.

[0080] For example, when determining whether BIO queuing latency is a bottleneck, the BIO queuing latency is compared with a BIO queuing latency threshold. If the BIO queuing latency is greater than the BIO queuing latency threshold, the device parameters of the block device module are determined. For example, if the device parameter of the block device module is determined to be thread load, the thread load is compared with a preset thread load threshold. If the thread load of the block device module is greater than the preset thread load threshold, then the block device module is determined to be a bottleneck.

[0081] For example, when determining whether NVME processing latency is a bottleneck, the NVME processing latency is compared with an NVME processing latency threshold. If the NVME processing latency is greater than the NVME processing latency threshold, the device parameters of the NVME driver module are determined. For example, if the device parameters of the NVME driver module are determined to be in a device error condition, and the device error condition indicates an error, then the NVME driver module is determined to be a bottleneck.

[0082] This application also provides a system architecture including a three-order correlation architecture, which divides the functionality into four core modules: a visualization analysis layer, an analysis and calculation layer, a correlation engine layer, and a data acquisition layer, which together support three-order correlation tracing analysis. The following is a detailed description of the system architecture: Deploying TCP / BIO / NVMe probes via eBPF is shown in the following deployment code: # Dynamically mount eBPF application bpftool prog load ebpf_corr.o / sys / fs / bpf / prog bpftool net attach xdp / sys / fs / bpf / prog eth0 # Activate tracepoint echo 1> / sys / kernel / debug / tracing / events / tcp / tcp_sendmsg / enable echo 1> / sys / kernel / debug / tracing / events / block / block_rq_issue / enable echo 1> / sys / kernel / debug / tracing / events / nvme / nvme_setup_cmd / enable The complete tracing chain from the NVMe storage driver layer completion event to the upper-layer block device request (BIO) is as follows: When the NVMe command completes, the corresponding BIO request is looked up by the command ID, then associated with the preset I / O request chain data structure, the completion timestamp is recorded, the complete request chain data is output to user space, and the three-level mapping table is cleaned up simultaneously. The resources (nvme_cmd_map->bio_chain_map->chain_map) are used to achieve end-to-end performance analysis of the storage I / O path. The specific implementation code is as follows: / / eBPF code: NVMe layer completion association int trace_nvme_complete(struct pt_regs *ctx) { / / 1. Get the completion event data structure struct nvme_completion *comp = (struct nvme_completion *)PT_REGS_PARM1(ctx);__u64 cmd_id = comp->command_id; / / Extract the NVMe command ID / / 2. Get the corresponding BIO request by command ID struct bio *bio = get_bio_from_cmd(cmd_id);if (!bio) {return 0; / / No associated BIO found, possibly an irrelevant command} / / 3. Find the corresponding request chain by BIO request struct io_chain *chain = find_chain_by_bio(bio);if (!chain){return 0; / / No associated request chain found} / / 4. Record the NVMe completion timestamp chain->nvme_complete_ts = bpf_ktime_get_ns(); / / 5. Output complete request chain data to user space bpf_perf_event_output(ctx,&events, BPF_F_CURRENT_CPU, chain, sizeof(*chain)); / / 6. Clean up resources cleanup_chain_resources(chain, bio); return 0;} / / Get the corresponding BIO request by NVMe command ID static struct bio *get_bio_from_cmd(__u64 cmd_id) { / / 1. Find the mapping from command ID to request address struct nvme_cmd_map_key cmd_key = {.cmd_id = cmd_id,.cpu_id = bpf_get_smp_processor_id()}; / / 2. Get the BIO address from the mapping table struct nvme_cmd_map_value*cmd_val = bpf_map_lookup_elem(&nvme_cmd_map,&cmd_key); if (!cmd_val) {return NULL; / / Mapping not found} / / 3.Convert the address to a BIO structure pointer return (struct bio *)cmd_val->bio_addr;} / / Find the corresponding request chain by BIO request static struct io_chain *find_chain_by_bio(struct bio *bio) { / / 1. Create a BIO address as the lookup key __u64 bio_addr = (__u64)bio; / / 2. Get the request chain ID from the mapping table __u64 *chain_id_ptr = bpf_map_lookup_elem(&bio_chain_map,&bio_addr); if (!chain_id_ptr) {return NULL; / / No associated request chain found} / / 3. Get the complete request chain by chain ID return bpf_map_lookup_elem(&chain_map,chain_id_ptr);} / / Clean up request chain related resources static void cleanup_chain_resources(struct io_chain *chain, struct bio *bio) { / / 1. 1. Clean up the BIO address mapping: `__u64 bio_addr = (__u64)bio;bpf_map_delete_elem(&bio_chain_map,&bio_addr);` / / 2. Clean up the NVMe command mapping: `struct nvme_cmd_map_key cmd_key = {.cmd_id = chain->nvme_cmd_id,.cpu_id = bpf_get_smp_processor_id()};bpf_map_delete_elem(&nvme_cmd_map,&cmd_key);` / / 3. Clean up the request chain mapping: `__u64 chain_id = (__u64)chain;bpf_map_delete_elem(&chain_map,&chain_id);` Bottleneck analysis uses latency data and system metrics at each stage of the I / O request chain to diagnose performance bottlenecks in the end-to-end storage path. By calculating network layer latency, BIO queuing latency, and NVME processing latency, high-latency components are identified based on preset thresholds. In-depth root cause analysis is performed using device parameters such as queue utilization, thread load, and device error conditions. Finally, the main bottleneck is determined based on the percentage contribution of latency at each stage, and a detailed diagnostic report is generated, including severity level, problem description, and optimization suggestions. The specific code is as follows: def analyze_bottleneck(io_chain, sys_metrics=None): """Analyze the performance bottleneck of the I / O request chain: param io_chain: The request chain object containing timestamps of each stage: param sys_metrics: System-level metrics (optional): return: (Main bottleneck, contribution percentage, detailed analysis results)"""# 1. Calculate the latency of each stage net_latency = io_chain.bio_issue_ts - io_chain.tcp_send_ts bio_queue = io_chain.nvme_submit_ts - io_chain.bio_issue_ts nvme_latency = io_chain.nvme_complete_ts - io_chain.nvme_submit_ts total = io_chain.nvme_complete_ts - io_chain.tcp_send_ts# 2. Calculate the basic contribution bottlenecks = {"network": net_latency,"bio_queue": bio_queue,"nvme_processing": nvme_latency}# 3. Advanced bottleneck diagnosis main_bottleneck = "unknown" detailed_analysis = {}# Network layer bottleneck diagnosis if net_latency>200:# More than 200μs is considered high network latency detailed_analysis["network"] = {"severity": "high","message": "Network processing time is too long","suggestions": ["Check network congestion", "Optimize TCP parameters"] }if sys_metrics and sys_metrics.get("tcp_retrans", 0)>5:detailed_analysis["network"]["subtype"]= "retransmission"detailed_analysis["network"]["suggestions"].append("Reduce TCP retransmission count")# BIO queuing bottleneck diagnosisif bio_queue>150:# Exceeding 150μs is considered high queuing latencydetailed_analysis["bio_queue"] = {"severity": "critical" if bio_queue>300 else "high","message": "High queuing latency at block device layer","suggestions": ["Increase NVMe queue depth", "Optimize OSD thread pool"]}# In-depth analysis combined with system metricsif sys_metrics:queue_depth = sys_metrics.get("nvme_queue_depth", 0)queue_util =sys_metrics.get("nvme_queue_util", 0)if `queue_util>90: detailed_analysis["bio_queue"]["subtype"]= "queue_saturation" detailed_analysis["bio_queue"]["suggestions"].append(f"Current queue utilization {queue_util}%, it is recommended to increase queue depth")` `elifsys_metrics.get("osd_thread_util", 0)>80: detailed_analysis["bio_queue"]["subtype"]= "thread_pool_bottleneck" detailed_analysis["bio_queue"]["suggestions"].`append("OSD thread pool overload, increase concurrent threads") # NVMe processing bottleneck diagnosis if nvme_latency>100: # Exceeding 100μs is considered high processing latency detailed_analysis["nvme_processing"] ={"severity": "high","message": "NVMe device processing time is too long","suggestions": ["Check SSD health status", "Upgrade NVMe driver"]} # Combine device metric analysis. if sys_metrics:if sys_metrics.get("nvme_media_errors", 0)>0:detailed_analysis["nvme_processing"]["subtype"]= "media_error"detailed_analysis["nvme_processing"]["suggestions"].append("Media error detected, consider replacing SSD")elif sys_metrics.get("nvme_util", 0)>95:detailed_analysis["nvme_processing"]["subtype"]= "device_saturation"detailed_analysis["nvme_processing"]["suggestions"].append(f"Device utilization {queue_util}%, consider adding OSD or upgrading SSD")# 4. Determine the main bottleneck (based on contribution and threshold)if total>0:# Calculate the contribution percentage contrib = {k: (v / total * 100) for k, v in bottlenecks.items()} # Main bottleneck determination logic if contrib["bio_queue"]>50: main_bottleneck = "bio_queue" elif contrib["nvme_processing"]>40: main_bottleneck = "nvme_processing" elif contrib["network"]>30: main_bottleneck = "network" elif contrib["bio_queue"] + contrib["nvme_processing"]>70: main_bottleneck = "storage_subsystem" else: # Select the maximum contribution point when there is no significant bottleneck main_bottleneck = max(contrib, key=contrib.get) return main_bottleneck,contrib[main_bottleneck], detailed_analysis # Abnormal case where total latency is 0 return "invalid", 0, {"error": "Zero total latency"}.

[0083] Please refer to Figure 8 This application embodiment also provides an application for Figure 1 The bottleneck diagnostic device 110 of the electronic device 100 includes: The acquisition module 111 is used to acquire each module in the three-level association architecture, wherein the three-level association architecture includes a TCP module, a block device module, and an NVME driver module; Module 112 is used to determine the event timestamps of each of the modules; The calibration module 113 is used to perform clock calibration on the event timestamps of each of the modules; The calculation module 114 is used to calculate the latency of each module in the third-order correlation architecture based on the calibrated event timestamps of each module. The determining module 112 is further configured to determine the bottleneck in the third-order correlation architecture from each of the delays.

[0084] This application also provides an electronic device 100, which includes a processor 130 and a memory 120. The memory 120 stores computer-executable instructions, which, when executed by the processor 130, implement the bottleneck diagnosis method.

[0085] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor 130, implements the bottleneck diagnosis method.

[0086] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0087] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A bottleneck diagnosis method, characterized in that, The method includes: Obtain the modules in the three-level association architecture, wherein the three-level association architecture includes a TCP module, a block device module, and an NVME driver module, and the TCP module, the block device module, and the NVME driver module are associated; Determine the event timestamps for each of the aforementioned modules; Clock calibration is performed on the event timestamps of each module; Based on the calibrated event timestamps of each module, the latency of each module in the third-order correlation architecture is calculated; From the aforementioned delays, the bottlenecks in the third-order correlation architecture are identified.

2. The method according to claim 1, characterized in that, The step of clocking the event timestamps of each module includes: Obtain the earliest timestamp from each of the event timestamps; The module corresponding to the earliest timestamp is used as the base time. Based on the reference time, the event timestamps of other modules besides the module corresponding to the reference time are aligned.

3. The method according to claim 2, characterized in that, The module corresponding to the reference time is the TCP module. The step of aligning the event timestamps of other modules besides the module corresponding to the reference time based on the reference time includes: Calculate the first relative time offset between the event timestamp of the block device module and the reference time; Calculate the second time offset between the event timestamp of the NVME driver module and the reference time; Determine the clock drift value; The first relative time offset is compensated based on the clock drift value to achieve alignment of the event timestamps of the block device module; The second relative time offset is compensated based on the clock drift value to achieve alignment of the event timestamps of the NVME drive module.

4. The method according to claim 1, characterized in that, The TCP module corresponds to the first mapping table, the block device module corresponds to the second mapping table, and the NVME driver module corresponds to the third mapping table; The first mapping table contains the correspondence between request chain IDs and the complete state of the entire I / O chain; The second mapping table contains the correspondence between the request chain ID and the BIO structure address; The third mapping table contains the correspondence between the BIO structure address and the NVMe command identifier.

5. The method according to claim 1, characterized in that, The step of calculating the latency of each module in the third-order correlation architecture based on the calibrated event timestamps of each module includes: Based on the calibrated event timestamps of each module, the network layer latency, BIO queuing latency, and NVME processing latency are determined. Determine the completion time of the NVME driver module and the event timestamp of the TCP module, and calculate the total latency.

6. The method according to claim 5, characterized in that, The step of determining the bottleneck in the third-order correlation architecture from the various delays includes: Calculate the ratio of the network layer latency to the total latency, and use it as the bottleneck contribution of the TCP module; Calculate the ratio of the BIO queuing delay to the total delay, and use it as the bottleneck contribution of the block device module; The ratio of the NVME processing latency to the total latency is calculated and used as the bottleneck contribution of the NVME driver module. The bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module are compared with the latency threshold respectively. Obtain a target bottleneck contribution greater than the intermediate latency threshold from the bottleneck contribution of the TCP module, the bottleneck contribution of the block device module, and the bottleneck contribution of the NVME driver module. The module corresponding to the target bottleneck contribution is taken as the bottleneck output.

7. The method according to claim 5, characterized in that, The step of determining the bottleneck in the third-order correlation architecture from the various delays includes: Compare the network layer latency with a network layer latency threshold; If the network layer latency is greater than the network layer latency threshold, the device parameters of the TCP module are determined, wherein the device parameters include at least one of the following: queue utilization, thread load, and device error conditions; The system determines whether the TCP module is a bottleneck based on the TCP device parameters.

8. A bottleneck diagnostic device, characterized in that, The device includes: The acquisition module is used to acquire each module in the three-level association architecture, wherein the three-level association architecture includes a TCP module, a block device module, and an NVME driver module; A determination module is used to determine the event timestamps of each of the modules; A calibration module is used to perform clock calibration on the event timestamps of each of the modules. The calculation module is used to calculate the latency of each module in the third-order correlation architecture based on the calibrated event timestamps of each module. The determining module is further configured to determine the bottleneck in the third-order correlation architecture from each of the delays.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-7.

Images