A method, system, electronic device and storage medium for analyzing abnormal business request delimitation

Abstract

The present invention discloses a method, system, electronic device and storage medium for business request anomaly delimitation analysis, including monitoring the kernel event list in the target business system through an eBPF probe to obtain kernel indicators; adding labels of processes and threads in a single request to each kernel indicator; triggering the corresponding process call function based on capturing kernel events; judging whether the URL information in the request object exists in the mapping, and if so, replacing the URL with the mapping; adding the mapping, process and thread in a single request to the filter mapping; associating the kernel indicators of the same label with the request object. The present invention uses non-intrusive monitoring means, so that enterprises can achieve non-interference monitoring of the business request processing flow, ensuring that the monitoring process itself does not have a negative impact on business performance, thereby promoting the refinement of business management, enhancing the ability to quickly identify and respond to potential faults, and improving the overall service quality and user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, electronic device and storage medium for analyzing abnormal delimitation of business requests. Background Art

[0002] With the development and popularization of cloud-native technologies, enterprises are accelerating the adoption of Kubernetes to achieve containerized management of workloads. This trend, accompanied by the full containerization of business systems, has aroused concerns about the complexity of operation and maintenance monitoring brought about by the increasingly large cluster scale. For cloud business scenarios, on the basis of identifying the running status of business applications, no additional standardized logs and traceId transformation processes are used to provide data support for establishing a unified business anomaly demarcation. The existing business anomaly troubleshooting capabilities are still limited to the application network status or the transformation of application code. The system running in the Kubernetes cluster cannot effectively associate kernel indicators and perform standardized business problem demarcation. The existing business of the cluster is transformed accordingly, and additional application probes and a large number of code transformation services are introduced to perform business anomaly demarcation analysis. At the same time, cumbersome configuration is required, the transformation cost and learning cost are high, and the intrusion to business applications is obvious, which affects the use of business applications.

[0003] There are generally three types of abnormal problems in existing businesses running on Kubernetes: The first is to integrate monitoring tools such as Prometheus and Grafana to collect and analyze cluster and application-level indicator data in real time, such as CPU usage, memory consumption, network I / O, and disk I / O, so as to promptly discover resource bottlenecks and potential failures, and define business-level problems through resource indicators. The second is to carefully adjust the application configuration, and introduce standardized log monitoring and code transformation services with necessary code monitoring to build a deep code-level monitoring framework to analyze and define the root causes of business anomalies. The third is the bypass monitoring mode, which saves data by acquiring network data packets and implements monitoring through the port mirroring function of network devices such as switches. In this mode, the monitoring device is connected to the designated mirror port of the switch, obtains network packets, analyzes network data anomalies, and analyzes and defines network and application problems.

[0004] Limitations of using cloud monitoring components

[0005] In the Kubernetes environment, traditional monitoring methods have exposed significant limitations, especially in monitoring the overall picture of business operations. Log monitoring outputs the execution process and status of the application by embedding points in the key methods of the application. It can be used for development debugging and application troubleshooting, but it brings the problem of standardized log format, high storage and resource requirements, and real-time query limitations, processing and retrieving large amounts of log data. At the same time, although application performance management (APM) tools can go deep into the application, their implementation process is cumbersome and requires the implantation of monitoring agents in the target system, which often requires the application to be restarted, thereby increasing the cost and complexity of deployment and maintenance. The previous method of relying on the mirroring function of network devices such as switches to capture traffic data is powerless when facing the cloud-native virtual network architecture because they fail to effectively identify and track complex cloud traffic patterns.

[0006] There are blind spots in defining business problems on the cloud

[0007] The solution indicators of the traditional market rely on experts and luck. Only the corresponding expert experience can quickly understand and apply the interpretation of professional indicators. A large number of problems are added by experts through experience. Indicators and logs have resulted in blind spots in monitoring not disappearing on their own, but still existing, and it is impossible to form a standardized indicator system process. For the generation of indicators, the cloud business system records logs where problems are expected, and adds indicators or event records where problems have occurred. At the same time, a large number of network indicators, application performance, and hardware indicators appear discretely, resulting in the inability to use indicators and troubleshoot problems, which further leads to the inability to form a systematic indicator governance in series.

[0008] The silo problem with monitoring tools

[0009] Information cannot be effectively shared between monitoring system modules, resulting in an incoherent overall monitoring view. Key information may be missed, and different tools may provide different monitoring views and alarm information, requiring more time and effort to analyze and compare, which prolongs the time to identify the problem. Summary of the invention

[0010] In view of the shortcomings existing in the above problems, the present invention provides a method, system, electronic device and storage medium for analyzing business request anomaly demarcation.

[0011] To achieve the above object, the present invention provides a method for analyzing abnormal business request demarcation, comprising:

[0012] Use eBPF probes to monitor the kernel event list in the target business system, capture kernel events in real time, and obtain kernel indicators;

[0013] Add a label of the process pid and the thread tid in a single request to each of the kernel indicators;

[0014] Based on capturing the kernel event, triggering the corresponding process system call function, and storing the corresponding time consumption, request things and kernel events;

[0015] Determine whether the URL information in the request transaction exists in the mapping, and if so, replace the URL with the mapping;

[0016] Add the mapping, the process pid and the thread tid in the single request to the filter map;

[0017] The kernel indicator with the same tag is associated with the request transaction to display the specific abnormal problem in the abnormal application request.

[0018] Preferably, the kernel indicators include kernel memory, CPU limit, swap usage, connection indicators, network packet loss, network retransmission, DNS resolution, disk read and write, other process IO distribution, lock time, and futex time indicator data.

[0019] Preferably, the kernel event list in the target service system monitored by the eBPF probe includes:

[0020] Define a kernel module initialization function;

[0021] The initialization function registers a kprobe to the specified kernel function __alloc_pages_nodemask;

[0022] Define the kprobe structure, and finally call the register_kprobe function to register kprobe;

[0023] When the memory allocation function is called, a log is printed to record the location of the memory allocation attempt; when the memory allocation function returns, a log is printed to record the result of the memory allocation; finally, a for loop is used to iterate over each SearchHit object in the search results;

[0024] Use conditional judgment and Map lookup to replace the URL; then put the URL into the filt map.

[0025] Preferably, the eBPF probe records the system call function from the start time starttime to the end time endtime starting from the request IO reading; obtains all kernel event functions occurring during the request through the eBPF probe, and adds labels of the process pid and the thread tid in a single request to all the kernel indicators; through the same label, the kernel indicators with the same label are linked in series and associated with the request to display the specific abnormal problem in the abnormal application request.

[0026] Preferably, the eBPF probe reads the system call function and associates the tid and pid steps include:

[0027] Obtain the kernel indicator, capture the system call function of the current process when triggered, and save the corresponding time consumption, request transactions and kernel events;

[0028] Determine the URL information of the request, map, replace and fill the filter map, and use conditional judgment and Map search to replace the URL;

[0029] Preserve URL, tip and pid field labels, and provide necessary information for subsequent filtering or processing operations by putting URL, tid and pid into the filt map;

[0030] Confirm the time period in which the requested event occurs, record the start time startTime and the end time endtime, and associate the log information and display the page through the time period in the acquired log.

[0031] The present application also provides a business request anomaly demarcation and analysis system, including:

[0032] The probe module is used to monitor the kernel event list in the target business system through the eBPF probe, capture kernel events in real time, and obtain kernel indicators;

[0033] Add a module for adding a label of the process pid and the thread tid in a single request to each of the kernel indicators;

[0034] A trigger module, for triggering a corresponding process system call function based on capturing the kernel event, and storing the corresponding time consumption, request transaction and kernel event;

[0035] A replacement module, used for determining whether the URL information in the request transaction exists in the mapping, and if so, replacing the URL with the mapping;

[0036] A mapping module, used for adding a mapping, the process pid and the thread tid in the single request to a filter map;

[0037] The association module is used to associate the kernel indicators of the same label with the request object to display the specific abnormal problems in the abnormal application request.

[0038] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0039] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention fills the blind spot of traditional monitoring methods in kernel indicators, and can also capture and analyze the behavior of business requests at the kernel level in real time and accurately, thereby realizing comprehensive monitoring of the health status of the business system and rapid fault location, significantly improving the stability and reliability of cloud services; through non-intrusive monitoring methods, enterprises can achieve non-interference monitoring of the business request processing flow, ensuring that the monitoring process itself has no negative impact on business performance, thereby promoting the refinement of business management, enhancing the ability to quickly identify and respond to potential faults, and improving the overall service quality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a flowchart of a method for analyzing business request anomaly demarcation. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Reference Figure 1 The present invention provides a method for analyzing abnormal business request delimitation, comprising:

[0045] Use eBPF probes to monitor the kernel event list in the target business system, capture kernel events in real time, and obtain kernel indicators; kernel indicators include kernel memory, CPU limit, swap usage, association indicators, network packet loss, network retransmission, DNS resolution, disk read and write, other process IO distribution, lock time, and futex time indicator data;

[0046] Add labels of process pid and thread tid in a single request to each kernel metric;

[0047] Based on the captured kernel events, the corresponding process system call function is triggered, and the corresponding time consumption, request transactions and kernel events are stored;

[0048] Determine whether the URL information in the request transaction exists in the mapping. If so, the mapping replaces the URL;

[0049] Add the mapping, process pid and thread tid in a single request to the filter map;

[0050] Correlate kernel metrics with the same label with request objects to display specific abnormal issues in abnormal application requests.

[0051] In this embodiment, a lightweight non-intrusive collection technology based on eBPF is used to capture the time-consuming information of business requests from the kernel level, and the obtained kernel indicators are used to perform abnormal delimitation analysis of business requests. The thread state can be divided into two states: On-CPU state and Off-CPU state. On-CPU means that the thread is executing code on the CPU, and Off-CPU means that the thread is not executing code on the CPU and is in a state of waiting for other resources, such as being blocked on I / O, locks, timers, paging / swapping. Determining the cause of CPU busyness usually requires in-depth analysis of the system stack. Generating a kernel-level flame graph and analyzing it by sampling at a fixed rate is a rough but extremely effective method that can help us intuitively identify which code paths are most active on the CPU. This process creates a timed interrupt to collect the current program counter, function address, and even complete stack backtrace information, and finally converts these complex data into easy-to-understand reports. With the help of a visual tool such as a flame graph, not only can the hot code path be quickly located, but the overall performance bottleneck of the system can also be grasped from a macro perspective, thereby providing strong support for optimizing system performance.

[0052] Specifically, kernel indicators are obtained through eBPF, such as CPU indicators: kernel memory, cpu limit, swap usage; kubernetes network performance indicators: connection indicators, network packet loss, network retransmission, DNS resolution; storage performance indicators: disk read and write, other process IO distribution; waiting time: lock time, futex time and other indicator data, to build an analysis of the time consumption of associated single business requests. The governance system can make up for the lack of observability of the existing monitoring for the network and operating system, such as: the inability to troubleshoot transaction timeouts caused by business retransmissions; application problems cannot be deeply demarcated from threads or kernel indicators, etc.

[0053] All interactions between applications and the kernel are carried out through system calls; the number of system calls is limited and can be categorized and organized; system calls can be intercepted by eBPF; all data is integrated through kernel indicators and the request system of a single transaction to achieve the complete business program execution process.

[0054] In this embodiment, the indicator event record of the entire execution process of the business system request is obtained based on the eBPF technology, so that the kernel indicator fault demarcation of the business system request at the system call layer in the kernel space can be achieved.

[0055] In kernel space, use the kernel probes kprobes and kretprobes in the kernel. Kprobes sets breakpoints at any location in kernel execution and collects execution information at these breakpoints. It inserts a custom eBPF program before kernel function execution, thereby collecting data before and after function execution. Kretprobes sets a breakpoint at the return point of the kernel function and executes the eBPF program when the function returns. For example, to track memory recovery operations in the kernel, use the kprobes kernel probe to track memory recovery operations in the kernel and track the __alloc_pages_nodemask function.

[0056] The specific process steps are as follows:

[0057] Step 1: Initialize kprobe registration. Define a kernel module initialization function, which registers a kprobe to the specified kernel function __alloc_pages_nodemask. First initialize the function, then define the kprobe structure, and finally call the register_kprobe function to register the kprobe. The code is as follows:

[0058] / / Register kprobe probe to __alloc_pages_nodemask function

[0059] static int __init my_init(void){

[0060] struct kprobe kp = {

[0061] .kp = (struct kstat_func *)&alloc_pages_entry,

[0062] .handler = alloc_pages_entry,

[0063] .exit_handler = alloc_pages_exit,

[0064] };

[0065] kp.kp.addr = (unsigned long)&__alloc_pages_nodemask;

[0066] register_kprobe(&kp);

[0067] return 0;

[0068] }

[0069] Step 2: Monitor memory allocation function calls. First, print a log when the memory allocation function is called to record the location of the memory allocation attempt; then print a log when the memory allocation function returns to record the result of the memory allocation; finally, use a for loop to traverse each SearchHit object in the search results. The code is as follows:

[0070] / / When the __alloc_pages_nodemask function is called, alloc_pages_entry is triggered and the calling location is recorded. When the function returns, alloc_pages_exit is triggered and the return value of the function is recorded.

[0071] for (SearchHit hit:sr.getHits().getHits()){

[0072] String url = object.getJSONObject(constants.DEFAULT_MESSAGE)

[0073] .getString(constants.URL);

[0074] }

[0075] static void alloc_pages_entry(struct kretprobe_instance *ki, structpt_regs *regs){

[0076] printk(KERN_INFO "Memory allocation attempt at %pS\n , antime", regs->ip);

[0077] }

[0078] static void alloc_pages_exit(struct kretprobe_instance *ki, structpt_regs *regs){

[0079] printk(KERN_INFO "Memory allocation result: %ld\n, antime ", regs->ax);

[0080] }

[0081] Step 3: URL mapping replacement and filling filter mapping. First, use conditional judgment and Map search to replace the URL; then put the URL into the filter mapping to provide necessary information for subsequent filtering or processing operations. The code is as follows:

[0082] / / Iterate over each hit in the search results, extract the URL field from the search results, replace the URL if it exists in the mapping, and add the URL to the filter mapping.

[0083] if(!urlMap.isEmpty() && urlMap.containsKey(url)){

[0084] url = urlMap.get(url);

[0085] }

[0086] filt.put(constants.URL,url);

[0087] Through the above steps, we can monitor the kernel memory allocation function and process the search results to ensure that the required kernel indicators can be extracted from the kernel space. Through deep tracing of system call functions in the kernel space, detailed monitoring of business system requests at the system call level in the kernel space can be achieved to solve the non-intrusive cloud kernel indicator delimited business request anomalies.

[0088] In this embodiment, the monitoring solution based on eBPF technology uses a non-intrusive bypass collection method, combined with the data docking of Kubernetes open interfaces, to comprehensively aggregate all-round data such as indicator measurement, link tracking, and operation status, complete the kernel operation and maintenance indicator data modeling in a single business request, and build an operation and maintenance indicator data governance system. Based on the observable probe eBPF, the sandbox program is run in the kernel without changing the kernel source code. By collecting, processing, and analyzing various events in the kernel, the business association of kernel indicators on the cloud is realized.

[0089] An observable probe based on eBPF technology. First, eBPF Trace records the system call function from the start time starttime to the end time endtime from the request IO reading. Then, all kernel event functions that occur during the request are obtained through eBPF, and the process pid and thread tid labels in a single request are added to all indicators. Finally, the kernel indicators with the same label are linked to the request through the same label to display the specific abnormal problems in the abnormal application request. At the same time, the start time is recorded at the beginning of the request, and the end time is recorded at the end of the request. The log information in these two time periods is queried in the log file and displayed on the page. In the time period when the abnormal application request is discovered, eBPF provides kernel event association, obtains the log data in the corresponding time period of the abnormal application for display, and restores the execution process of a single request on the faulty node according to the resource usage.

[0090] In a single request, the key steps to use the eBPF program to read the incoming system call function and associate the tid, pid, and other threads, processes, and logs are as follows:

[0091] Step 1: Get kernel space indicator information, capture the system call function of the current process when triggered, and save the corresponding time consumption, request transactions and kernel events.

[0092] Step 2: Determine the URL information of the transaction request, map, replace and fill the filter map, and use conditional judgment and Map search to replace the URL.

[0093] Step 3: Keep the URL, tip, and pid field labels, and provide necessary information for subsequent filtering or processing operations by putting URL, tid, and pid into the filt map.

[0094] Step 4: Confirm the time period in which the transaction request occurs, record the start time startTime and the end time endtime, and associate the log information and display the page through the time period in the obtained log. The code is as follows:

[0095] / / Extract the URL field from the search results, replace the URL if it exists in the map, add the URL, tid, and pid to the filter map.

[0096] if(!urlMap.isEmpty() && urlMap.containsKey(url)){

[0097] url = urlMap.get(url);

[0098] }

[0099] filt.put(constants.URL,url);

[0100] filt.put(constants.tid,object.getJSONObject(constants.DEFAULT_MESSAGE).getString(constants.tid));

[0101] filt.put(constants.pid,object.getJSONObject(constants.DEFAULT_MESSAGE).getString(constants.pid));

[0102] When the sys_open system call is triggered, the PID, thread TID, and process name of the current process are captured, and then the incoming file path is read, and this information is recorded in a BPF map, and the data is associated to achieve detailed monitoring of kernel operations.

[0103] Through the process PID, thread TID and event-related kernel events and log functions, the four-layer indicators of the business kernel space, including CPU usage, storage interface status, memory usage and network system call status, can be non-intrusive on-cloud kernel indicator demarcation of business request anomaly faults, making the monitoring of business request anomaly demarcation more refined and real-time.

[0104] The present application also provides a business request anomaly demarcation and analysis system, including:

[0105] The probe module is used to monitor the kernel event list in the target business system through the eBPF probe, capture kernel events in real time, and obtain kernel indicators;

[0106] Added module for adding labels of process pid and thread tid in a single request to each kernel metric;

[0107] The trigger module is used to trigger the corresponding process system call function based on the captured kernel event, and store the corresponding time consumption, request transaction and kernel event;

[0108] The replacement module is used to determine whether the URL information in the request transaction exists in the mapping, and if so, the mapping replaces the URL;

[0109] A mapping module, used to add mappings, process pids, and thread tids in a single request to the filter map;

[0110] The association module is used to associate kernel indicators with request objects of the same label to display specific abnormal problems in abnormal application requests.

[0111] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0112] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0113] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for analyzing business request anomaly demarcation, characterized in that: include: The kernel event list in the target business system is monitored by an eBPF probe, and the kernel events are captured in real time to obtain kernel indicators. The kernel event list in the target business system is monitored by an eBPF probe, including: defining an initialization function of a kernel module; the initialization function registers a kprobe to a specified kernel function __alloc_pages_nodemask; defining a kprobe structure, and finally calling the register_kprobe function to register the kprobe; printing a log when the kernel function is called for memory allocation to record the location of the memory allocation attempt; printing a log when the kernel function returns for memory allocation to record the result of the memory allocation; finally, using a for loop to traverse each SearchHit object in the search result; using conditional judgment and Map search to replace the URL; and then putting the URL into the filt map; Add a label of the process pid and the thread tid in a single request to each of the kernel indicators; Based on capturing the kernel event, triggering the corresponding process system call function, and storing the corresponding time consumption, request things and kernel events; Determine whether the URL information in the request transaction exists in the mapping, and if so, replace the URL with the mapping; Add the mapping, the process pid and the thread tid in the single request to the filter map; Associating the kernel indicator of the same label with the request thing to display the specific abnormal problem in the abnormal application request; The eBPF probe records the system call function from the start time starttime to the end time endtime from the request IO reading; obtains all kernel event functions that occur during the request through the eBPF probe, and adds the process pid and thread tid labels in a single request to all the kernel indicators; through the same label, the kernel indicators with the same label are linked in series and associated with the request to display the specific abnormal problems in the abnormal application request; The eBPF probe reads the system call function and associates the tid and pid steps include: Obtain the kernel indicator, capture the system call function of the current process when triggered, and save the corresponding time consumption, request transactions and kernel events; Determine the URL information of the request, map, replace and fill the filter map, and use conditional judgment and Map search to replace the URL; Preserve URL, tip and pid field labels, and provide necessary information for subsequent filtering or processing operations by putting URL, tid and pid into the filt map; Confirm the time period in which the requested event occurs, record the start time startTime and the end time endtime, and associate the log information and display the page through the time period in the acquired log.

2. The method for analyzing business request anomaly demarcation according to claim 1, characterized in that: The kernel indicators include kernel memory, CPU limit, swap usage, connection indicators, network packet loss, network retransmission, DNS resolution, disk read and write, other process IO distribution, lock time, and futex time indicator data.

3. A business request anomaly demarcation and analysis system, characterized in that: include: The probe module is used to monitor the kernel event list in the target business system through the eBPF probe, capture kernel events in real time, and obtain kernel indicators; The list of kernel events in the target business system monitored by the eBPF probe includes: defining an initialization function of a kernel module; the initialization function registers a kprobe to the specified kernel function __alloc_pages_nodemask; defining a kprobe structure, and finally calling the register_kprobe function to register the kprobe; printing a log when the kernel function of memory allocation is called to record the location of the memory allocation attempt; printing a log when the kernel function of memory allocation returns to record the result of memory allocation; finally, using a for loop to traverse each SearchHit object in the search result; using conditional judgment and Map search to replace the URL; and then putting the URL into the filt map; Add a module for adding a label of the process pid and the thread tid in a single request to each of the kernel indicators; A trigger module, for triggering a corresponding process system call function based on capturing the kernel event, and storing the corresponding time consumption, request transaction and kernel event; A replacement module, used for determining whether the URL information in the request transaction exists in the mapping, and if so, replacing the URL with the mapping; A mapping module, used for adding a mapping, the process pid and the thread tid in the single request to a filter map; An association module, used to associate the kernel indicator of the same label with the request object, and display the specific abnormal problem in the abnormal application request; The eBPF probe records the system call function from the start time starttime to the end time endtime from the request IO reading; obtains all kernel event functions that occur during the request through the eBPF probe, and adds the process pid and thread tid labels in a single request to all the kernel indicators; through the same label, the kernel indicators with the same label are linked in series and associated with the request to display the specific abnormal problems in the abnormal application request; The eBPF probe reads the system call function and associates the tid and pid steps include: Obtain the kernel indicator, capture the system call function of the current process when triggered, and save the corresponding time consumption, request transactions and kernel events; Determine the URL information of the request, map, replace and fill the filter map, and use conditional judgment and Map search to replace the URL; Preserve URL, tip and pid field labels, and provide necessary information for subsequent filtering or processing operations by putting URL, tid and pid into the filt map; Confirm the time period in which the requested event occurs, record the start time startTime and the end time endtime, and associate the log information and display the page through the time period in the acquired log.

4. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 2.

5. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method described in any one of claims 1 to 2.

Images