Host service observation method and electronic equipment

Abstract

The invention discloses a host service observation method and electronic equipment, and belongs to the technical field of service operation and maintenance. Loading an eBPF probe in the kernel space of the host; generating a unique call chain identifier for each application call request, and generating a unique span identifier for each function call or cross-process call in a call chain; associating index data of an operating system layer, a process layer, a database layer, an application component layer and an application calling layer, which are collected by utilizing a probe, with the generated calling chain identifier and span identifier by combining event context capture, process relation mapping and a service triggering link tracing technology; based on the associated multi-level index data, aggregating the index data associated with all the span identifiers according to the call chain identifier; and constructing a hierarchical structure of the call chain, and forming a complete resource call chain which is triggered from the initial application request and sequentially passes through the application component, the database, the process and the operating system. The method can realize observation of full services of the host.

Description

Technical Field

[0001] This application belongs to the field of business operation and maintenance technology, and specifically relates to a host business observation method and electronic equipment. Background Technology

[0002] In current host operation and monitoring practices, traditional methods for observable host services generally rely on installing agents on the host to collect data. However, this traditional approach has revealed many limitations in practical applications, severely impacting the efficiency and security of host service operation and maintenance. Specific problems include: (1) Highly invasive: The normal operation of the agent program requires installation and startup in the host operating system, which consumes critical system resources such as CPU, memory, and disk. In high-concurrency and high-load business scenarios, this resource consumption is even more prominent, which can easily interfere with the performance of business systems running on the host, and may lead to problems such as delayed business response and reduced processing capacity, thus affecting the normal operation of the business.

[0003] (2) Significant compatibility issues: Host operating systems are diverse, and even within the same type of operating system, there are different versions, such as Kylin OS and UnionTech OS, which have multiple version iterations. Different models and versions of host operating systems have significantly different compatibility requirements for agent programs. To ensure the normal operation of data collection, it is often necessary to develop and adapt corresponding agent program versions for different operating system versions. This not only increases the workload of the development team but also significantly increases subsequent maintenance costs, hindering large-scale promotion and application.

[0004] (3) Complex Deployment and Upgrades: In large-scale host cluster environments, the deployment, configuration, and subsequent version upgrades of the agent program require significant manpower and time. During deployment, operations must be performed on each host individually, and personalized settings must be made according to the configuration of different hosts. During upgrades, data collection may be interrupted due to version incompatibility. Once this happens, maintenance personnel will be unable to obtain host service operation data in a timely manner, making it difficult to effectively monitor and manage the service status.

[0005] (4) High security risks: To achieve comprehensive data collection, the agent program usually needs to obtain higher privileges on the host, which increases the host's security risks to some extent. If the agent program itself has security vulnerabilities and is exploited by hackers, it may lead to serious security problems such as illegal attacks on the host and leakage of sensitive data, posing a huge threat to the information security and business stability of the enterprise.

[0006] Given the aforementioned shortcomings of traditional proxy methods, there is an urgent need for a non-intrusive, low-resource-consumption, highly compatible, and secure method for full-service observability of hosts, in order to meet the pressing needs of hosts in service operation and maintenance and monitoring, and improve the stability and security of host service operation. Summary of the Invention

[0007] To address the aforementioned issues, this application provides a host service observation method and electronic device.

[0008] This application provides a host service observation method, including: Load eBPF probes into the host's kernel space to collect metrics data from the host's operating system layer, process layer, database layer, application component layer, and application call layer. A unique Trace ID is generated for each application call request, and a unique Span ID is generated for each function call or cross-process call in the call chain. The Span ID contains a Parent Span ID, which is used to represent the hierarchical relationship in the call chain. By combining event context capture, process relationship mapping, and business trigger link tracing technologies, the collected indicator data from each layer are associated with the generated call chain identifier Trace ID and span identifier Span ID; Based on the associated multi-level metric data, the metric data associated with all span identifiers (SpanIDs) are aggregated according to the call chain identifier (Trace ID). The hierarchical structure of the call chain is constructed according to the parent span identifier (Parent Span ID), forming a complete resource call chain that starts from the initial application request and passes through the application components, database, process, and operating system in sequence.

[0009] Furthermore, by combining event context capture, process relationship mapping, and business trigger link tracing technologies, the collected metric data from each layer are associated with the generated call chain identifier Trace ID and span identifier Span ID, including: By actively embedding data and passing context information, application call layer data is associated with the Trace ID and Span ID corresponding to function calls or cross-process calls. Based on the association between application components and application calls through protocols and interfaces, the application component layer metric data is associated with the call chain identifier Trace ID and span identifier Span ID extracted from protocol requests and interfaces; Based on the triggering relationship between SQL operations and application calls, database layer metric data is associated with the call chain identifier Trace ID and span identifier Span ID; Based on the hierarchical relationship between processes and application calls, process-level metric data involved in application calls are associated with the corresponding call chain identifier Trace ID and span identifier Span ID; The operating system-level metrics data generated by system calls, network connections, and disk I / O events related to application calls are associated with the Trace ID and Span ID of the corresponding process through the time of the event and the associated process information.

[0010] Furthermore, through proactive data entry and context passing, application call layer data is associated with the corresponding call chain identifier (Trace ID) and scheduling identifier (Span ID) for the function call or cross-process call, including: At the function call entry point of the application, the eBPF probe is triggered by the uprobe to generate or inherit the call chain identifier Trace ID and span identifier Span ID, and directly bind the function name, parameters, and timestamp information to the Trace ID and Span ID; For RPC calls, the eBPF probe writes the Trace ID and Span ID into the protocol header when serializing the request data; when the receiver parses the protocol header, it extracts the Trace ID and Span ID to ensure that the metric data at both ends of the cross-process call are associated with the same Trace ID and Span ID. If a function call is not actively captured, the function call stack of the current process is obtained through eBPF, the parent function of the nearest associated Span is found, and the uncaptured function call is used as the Trace ID to associate the child Span with the parent function.

[0011] Furthermore, based on the association between application components and the protocols and interfaces called by the application, the application component layer metric data is associated with the Trace ID and Span ID extracted from the protocol requests and interfaces, including: For web components using the HTTP protocol, the eBPF probe extracts the Trace ID and Span ID from the HTTP request headers during parsing, and associates the request processing metrics data of the web component with the extracted Trace ID and Span ID. For the caching component, when the application calls Redis through the SDK, the eBPF probe captures the Trace ID and Span ID of the current application call in the SDK's entry function and writes them as key prefixes to the Redis operation. The application component layer parses the Trace ID and Span ID in the key name and associates them with the cache operation metric data. When application components and business processes communicate via local sockets, the eBPF probe injects the Trace ID and Span ID into the sendmsg function of the sender, and extracts the Trace ID and Span ID in the recvmsg function of the receiver, thus realizing cross-process component indicator data association.

[0012] Furthermore, based on the triggering relationship between SQL operations and application calls, database layer metric data is associated with the call chain identifier Trace ID and span identifier Span ID, including: When an application establishes a database connection, the eBPF probe injects the Trace ID and Span ID of the current application call into the connection function, and stores the database connection ID-Trace ID mapping relationship in the kernel cache. All subsequent SQL operations on this connection are associated with the corresponding Trace ID and Span ID through the connection ID. For scenarios without an explicit connection ID, the business identifier is extracted from the SQL statement and directly matched with the Trace ID. If the business identifier does not exist in the SQL statement, the corresponding Trace ID and Span ID are obtained by associating the process identifier (PID) and timestamp of the SQL operation with the process identifier (PID) - call chain identifier (Trace ID) mapping table. The process identifier (PID) - call chain identifier (Trace ID) mapping table stores a multi-dimensional mapping relationship between the process identifier (PID), call chain identifier (Trace ID), span identifier (Span ID), and the time range of the span. By capturing the database audit log output through eBPF probes, the client IP and port, operation time, SQL statement contained in the logs are matched with the cross-process call information of the application call layer, and then the database layer indicator data in the logs are associated with the call chain identifier Trace ID and span identifier Span ID.

[0013] Furthermore, based on the hierarchical relationship between processes and application calls, the process-level metric data involved in the application call are associated with the corresponding call chain identifier (Trace ID) and span identifier (Span ID), including: When an application calls a process to trigger process creation or thread startup, the eBPF probe captures the parent process's Trace ID and Span ID in the process and thread's entry function and inherits them to the child process and thread. These are written into a maintained process identifier (PID) - Trace ID mapping table. All subsequent behavioral metrics of the child process are associated with the parent's Trace ID through this mapping table. The PID - Trace ID mapping table stores a multi-dimensional mapping relationship between the process identifier (PID), Trace ID, Span ID, and the time range of the span. For processes whose runtime exceeds the set duration, their resource usage is associated with the process's corresponding call chain identifier Trace ID and span identifier Span ID by time slice; If the file path of a process involves a business identifier, the Trace ID is directly matched; otherwise, the process identifier (PID) and timestamp of the file operation are combined, and the corresponding call chain identifier (Trace ID) and span identifier (Span ID) of the process are associated through the process identifier (PID) - call chain identifier (Trace ID) mapping table. The process identifier (PID) - Trace ID mapping table stores a multi-dimensional mapping relationship of process identifier (PID), call chain identifier (Trace ID), span identifier (Span ID), and time range of span.

[0014] Furthermore, operating system-level metrics data generated by system calls, network connections, and disk I / O events related to application calls are associated with the Trace ID and Span ID corresponding to the process, based on the event's occurrence time and associated process information. This includes: When generating the Trace ID and Span ID at the application call layer, the call start time is recorded synchronously. When the eBPF probe captures an operating system event, if the event time falls within the lifecycle of a certain Span and the process identifier PID associated with the event is consistent with the source process identifier PID of that Span, it is determined to be a system event triggered by that Span. The system resource indicators, kernel event indicators, and system state associated indicators corresponding to the operating system event are associated with the corresponding Trace ID and Span ID. When the application call layer's Span starts, the current process identifier (PID), call chain identifier (Trace ID), and span identifier (Span ID) are written into the maintained process identifier PID-call chain identifier (Trace ID) mapping table. When the current process identifier PID associated with an operating system event exists in the mapping table and the event time is within the Span's lifecycle, the system resource indicators, kernel event indicators, and system state-related indicators corresponding to the operating system event are directly associated with the call chain identifier (Trace ID) and span identifier (Span ID) corresponding to the process PID. The process identifier PID-Trace ID mapping table stores a multi-dimensional mapping relationship between the process identifier PID, call chain identifier (Trace ID), span identifier (Span ID), and the time range of the span. For network-related system calls, extract the network quadruple information of the event; using the maintained network quadruple-call identifier Trace ID mapping table, if the extracted network quadruple is consistent with the cross-process call network characteristics of a certain Span in the application call layer and the timestamp matches, then associate the operating system layer indicator data corresponding to the network-related system call with the call chain identifier Trace ID and span identifier Span ID of that Span. The network quadruple-call identifier Trace ID mapping table stores a multi-dimensional mapping relationship of network quadruple, call chain identifier Trace ID, span identifier Span ID, and the time range of the span.

[0015] This application also provides an electronic device, including: Memory, which stores execution instructions; and The processor executes the instructions stored in memory, causing the processor to perform the methods described above.

[0016] This application also provides a readable storage medium storing executable instructions, which are executed by a processor to implement the above-described method.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] Compared with the prior art, this application has the following advantages: 1. "Non-intrusive + Full-level" collaborative observation architecture: Breaking through the intrusive collection mode of traditional Agent proxy, it realizes non-intrusive data collection in kernel mode based on eBPF technology, covering five layers of indicator data from operating system to application call, solving the technical pain points of existing technologies that are "either highly intrusive or incompletely covered", and achieving the unity of "non-interference" and "full observation".

[0019] 2. Dynamic call chain connection mechanism: By dynamically generating Trace ID and Span ID and associating multi-level data, an end-to-end call chain system of "request triggering - full-link tracing - data aggregation" is constructed, which in particular solves the problem of call chain breaks between database and application components and realizes the traceability of the entire business link.

[0020] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a host service observation method according to an embodiment of this application is shown; Figure 2 The process and topology of constructing a call chain according to an embodiment of this application are illustrated; Figure 3 A flowchart illustrating the overall process of index data storage and analysis according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The host service observation method in this application embodiment is as follows: Figure 1 As shown, it includes the following steps: Step 101: Load the eBPF probe in the host's kernel space to collect indicator data from the host's operating system layer, process layer, database layer, application component layer, and application call layer. The following explains the indicator data collected in step 101: 1) Operating system layer indicator collection Operating system layer metrics are collected using eBPF probes attached to critical kernel functions, including system calls such as `sys_open`, `sys_read`, `sys_write`, and `sys_socket`, as well as kernel event functions such as `sched_switch` and `timer_expire`. The collected operating system layer metric data includes: System resource metrics: CPU utilization, which reflects the busyness of the host CPU; memory utilization, which reflects the usage of memory resources; disk I / O throughput, which shows the speed and amount of disk read and write data; network bandwidth utilization, which reflects the efficiency of network resource utilization; process scheduling latency, which reflects the timeliness of process scheduling on the CPU.

[0025] Kernel event metrics: System call count, recording the total number of system calls per unit time; Process creation / destruction count, reflecting the dynamic changes of host processes; File open / close count, reflecting the frequency of file operations; Network connection establishment / disconnection count, displaying the dynamic status of network connections.

[0026] System status indicators: Operating system version information, specifying the exact version of the operating system used by the host; kernel version information, reflecting the kernel's update status and functional characteristics; system uptime, showing the duration of continuous operation of the host; current active process count, reflecting the current process activity of the host.

[0027] 2) Process-level indicator collection Using eBPF probes (uprobe) attached to the entry, exit, and key functionalities (such as the main, fork, and exec functions) of user-mode processes, process-level metrics data are collected. The collected process-level metrics data covers basic process information and behavioral characteristics, including: Basic process metrics: Process ID (PID), used to uniquely identify a process; Parent Process ID (PPID), which identifies the parent process of this process; Process Name, which helps identify the purpose of the process; Process Start Time, which records when the process started running; CPU Time Used by the Process, which reflects the process's consumption of CPU resources; Memory Size Used by the Process, including virtual memory and physical memory, which reflects the process's memory resource usage.

[0028] Process behavior metrics: File operation behavior of the process, which records in detail the file paths opened by the process and the file read / write size, reflecting the process's file operation behavior; Network operation behavior of the process, which records the source IP, destination IP, source port, destination port and data sent / received in the network connection, showing the process's network communication status; Signal handling behavior of the process, which records the types of signals received by the process and the signal processing results, reflecting the process's response and processing capabilities to external signals.

[0029] 3) Database layer metric collection For various databases running on the host, eBPF probes attached to key functions of the database process are used to collect database-layer metrics data. These key functions include database connection handling functions, SQL statement parsing functions, and data query / insert / update / delete operation functions. The collected database-layer metrics data comprehensively reflect the database's operational status and business operations, including: Database connection metrics: Database connection count, including current connection count, maximum connection count, and idle connection count, reflecting the database connection resource usage; Connection establishment time, recording the duration from the start of connection establishment to successful establishment; Connection termination time, recording the moment from the start of connection termination to complete termination; Connection source IP, specifying the address that initiated the connection request.

[0030] SQL operation metrics: SQL statement type, distinguishing different operation types such as query, insert, update, and delete; SQL statement execution time, reflecting the efficiency of the SQL operation; SQL statement execution result, indicating whether the operation was successful or failed; number of rows affected, recording the number of data in the database table affected by the SQL operation; table names / field names involved in the SQL statement, specifying the specific data objects targeted by the SQL operation.

[0031] Database performance metrics include: database cache hit rate, reflecting the efficiency of database cache utilization; lock wait time, reflecting the lock contention situation in the database; transaction commit / rollback count, recording the completion status of transactions; and log write volume, showing the database log generation status.

[0032] 4) Collection of application component layer metrics For various application components running on the host, such as web server components, message queue components, and caching components, and these components must be compatible versions (e.g., compatible versions of Nginx, Redis, and RabbitMQ), eBPF probes are used to collect application component layer metrics data. These critical functions include request processing functions, response sending functions, cache read / write functions, and message sending / receiving functions. The collected application component layer metrics data comprehensively reflect the application component's runtime status and business processing capabilities, including: Component performance metrics: Component process ID, uniquely identifies the application component process; Component version information, specifies the application component version; Component startup time, records the moment the application component starts running; CPU / memory resources used by the component, reflecting the component's consumption of system resources; Component connection count, including the current number of connections and the maximum number of connections, reflects the component's connection processing capacity.

[0033] Component business metrics: For the Web server component, HTTP request count records the total number of requests received by the Web server per unit time; request response time reflects the efficiency of the Web server in processing requests; HTTP status codes, such as 200 (request successful), 404 (requested resource not found), and 500 (internal server error), indicate the processing result of the request; the request URL path specifies the specific resource targeted by the request. For the message queue component, message production rate reflects the speed of message generation; message consumption rate reflects the speed of message processing; message backlog shows the backlog of messages in the queue; message delivery success rate indicates the reliability of message delivery. For the caching component, cache hit count reflects the effective utilization of the cache; cache miss count reflects instances where the cache was not used; cache read / write rate displays the read / write performance of the cache.

[0034] 6) Application call layer metric collection By attaching eBPF probes to the application's function call entry points, exit points, and inter-process communication functions (such as socket communication functions and RPC call functions), application call layer metric data is collected. The collected application call layer metric data comprehensively reflects the application's internal function calls and inter-process communication, including: Function call metrics include: function name (identifying the called function); caller process ID (identifying the process initiating the function call); callee process ID (identifying the process receiving the function call); function call start and end times (recording the time range of the function call); function call duration (obtained by subtracting the start time from the end time, reflecting the efficiency of function execution); function call parameters (such as the service name, method name, and parameter values ​​of the RPC call, specifying the detailed information of the function call); and function return result (identifying the execution result of the function call).

[0035] Cross-process call metrics: Type of cross-process call, such as HTTP call, RPC call, Socket call, etc., distinguishing different cross-process communication methods; Source process information, including PID and process name, identifying the initiator of the call; Target process information, including PID, process name, IP address, and port number, identifying the receiver of the call; Call request data volume and call response data volume, reflecting the scale of data transmission during the cross-process call; Call response time, reflecting the processing efficiency of the cross-process call; Call status, indicating whether the call was successful or failed.

[0036] Step 102: Generate a unique Trace ID for each application call request, and generate a unique Span ID for each function call or cross-process call in the call chain; The span identifier Span ID includes the parent span identifier Parent Span ID, which is used to represent the hierarchical relationship in the call chain, thus clearly presenting the complete call chain structure.

[0037] Specifically, during the process of eBPF probes collecting application call layer metric data, when a new application call request, such as an HTTP request or RPC request, is detected, a corresponding Trace ID is automatically generated, and a Span ID is generated for the initial function call corresponding to the request. At this time, the Parent Span ID of this Span ID is empty, indicating that it is the starting node of the call chain. When a subsequent function call or cross-process call triggered by this call request is detected, the newly generated Span ID inherits the current Trace ID to ensure that all related operations of the same call request belong to the same call chain. At the same time, the Span ID corresponding to the parent call of the current call is used as the Parent Span ID of the new Span ID, thereby accurately constructing the hierarchical structure of the call chain.

[0038] Step 103: Combine event context capture, process relationship mapping, and business trigger link tracing technologies to associate the collected indicator data of the operating system layer, process layer, database layer, application component layer, and application call layer with the generated Trace ID or Span ID; The purpose of step 103 is to ensure that data at each level can be clearly attributed to a specific call chain and call node.

[0039] The specific association methods are explained below: 1. Association method of application call layer metrics The application call layer (function call, RPC call) is the starting point of the call chain. Through active instrumentation and context passing, the application call layer data is associated with the Trace ID and Span ID corresponding to the function call or cross-process call.

[0040] Function entry / exit capture: At the function call entry point of the application (such as the send function of the RPC client and the recv function of the server), the eBPF probe is triggered by the uprobe to generate or inherit the Trace ID and Span ID (if it is a sub-call, the Parent Span ID is set to the Span ID of the current call), and directly bind the function name, parameters, timestamp and other information to the Trace ID and Span ID.

[0041] Cross-process call protocol encapsulation: For RPC calls (such as Dubbo, gRPC), the eBPF probe writes the Trace ID and Span ID into the protocol header (such as the attachment field of Dubbo) when serializing the request data; when the receiver parses the protocol header, it extracts the Trace ID and Span ID to ensure that the indicator data at both ends of the cross-process call are associated with the same Trace ID and Span ID.

[0042] Abnormal call chain completion: If a function call is not actively captured (such as a third-party library function), the parent function of the nearest associated Span is found by backtracking the call stack (eBPF obtains the function call stack of the current process), and the unaware call is used as the Trace ID of the child Span associated with the parent function.

[0043] It should be noted that during application calls, in order to enrich the observed data and provide more analytical data for business analysis, logs are pre-set in the code to record information such as the transaction process and transaction steps. Although eBPF has a variety of methods, it lacks information from the business side, so logs and other data need to be used together for analysis.

[0044] 2. Association methods for application component layer metrics Based on the association between application components and application call protocols and interfaces, application component layer metric data is associated with Trace ID and Span ID extracted from protocol requests and interfaces.

[0045] Protocol layer identifier passing: For web components using the HTTP protocol (such as Nginx), the eBPF probe extracts custom call chain identifiers (such as X-Trace-ID:8f7e6d5c, X-Span-ID:1a2b3c4d) when parsing the HTTP request header, and directly associates Nginx's request processing metrics (response time, status code) with these Trace IDs and Span IDs.

[0046] Component client SDK instrumentation: For caching components (such as Redis), when the application calls Redis through the SDK (e.g., jedis.set()), the eBPF probe captures the Trace ID and Span ID of the current application call in the SDK's entry function and writes them as a "key prefix" to the Redis operation (e.g., SET trace_8f7e6d5c:user:1). The component layer parses the Trace ID in the key name and associates it with cache operation metric data (such as read / write time, hit status).

[0047] Inter-process communication (IPC) context inheritance: When application components and business processes communicate through local sockets (such as Unix Domain Sockets), the eBPF probe injects Trace ID and Span ID into the sendmsg function of the sender (business process) and extracts the Trace ID and Span ID in the recvmsg function of the receiver (component process), thereby realizing cross-process component indicator data association.

[0048] 3. Relationship methods for database-level indicator data Based on the triggering relationship between SQL operations and application calls, database layer metric data is associated with Trace ID and Span ID.

[0049] Database connection context passing: When an application establishes a database connection (e.g., by calling the `dm_connect` function), the eBPF probe inserts the Trace ID and Span ID of the current application call into the connection function (read from process memory or extracted from function parameters), and stores the database connection ID-Trace ID mapping table in the kernel cache. All subsequent SQL operations on this connection (e.g., `dm_execute_sql`) are associated with the corresponding Trace ID and Span ID through the connection ID. The database connection ID-Trace ID mapping table stores a multi-dimensional mapping relationship between the connection ID, Trace ID, Span ID, and Span's time range.

[0050] SQL statement feature extraction: For scenarios without an explicit connection ID (such as short connection operations), extract the business identifier in the SQL statement (e.g., select * from order where trace_id='8f7e6d5c', where 8f7e6d5c is the business identifier) ​​and directly match the Trace ID; if the SQL has no business identifier, then use the process PID (application process PID) and timestamp of the SQL execution process, combined with the PID-Trace ID mapping table, to associate with the corresponding Trace ID and Span ID. The PID-Trace ID mapping table stores a multi-dimensional mapping relationship of PID + Trace ID + Span ID + Span time range.

[0051] Database log association: The eBPF probe captures the audit log output of the database (such as the dm_audit.log write event of database A). The information contained in the log, such as "client IP and port, operation time, SQL statement", can be matched with the cross-process call information of the application call layer (source IP and port, call time). Then, the database layer indicator data in the log is associated with Trace ID and Span ID.

[0052] 4. Relationship methods between process-level metrics (basic process metrics, process behavior metrics) data Based on the hierarchical relationship between processes and application calls, process-level metric data involved in application calls are associated with the corresponding Trace ID and Span ID of the process.

[0053] Process call stack tracing: When an application calls a function that triggers process creation (such as fork) or thread startup, the eBPF probe captures the parent process's Trace ID and Span ID in the process and thread's entry function (such as the clone system call or pthread_create), and inherits them to the child process / thread, writing them into the PID-Trace ID mapping table. All subsequent behavioral indicators of the child process (such as opening files or CPU usage) are associated with the parent's call chain Trace ID through this mapping table.

[0054] Resource usage time segmentation: For long-running processes (such as database processes with a runtime exceeding a set duration), their resource usage needs to be associated with specific call chains based on time slices. For example, if the CPU usage of database process A (PID: 5678) during the period from 10:00:00 to 10:00:05 contains 3 associated database operation spans (corresponding to 3 trace IDs), then the CPU usage metric is split according to the execution time percentage of each span (e.g., Span1 accounts for 60%, Span2 accounts for 30%, and Span3 accounts for 10%), and associated with the corresponding trace ID and span ID respectively.

[0055] File path feature matching: If a process's file operations (such as opening / writing a log file) involve a business identifier (such as the log path containing / opt / app / logs / trace_8f7e6d5c.log), then the Trace ID fragment in the filename is directly matched and associated; if there is no explicit business identifier, then the process PID and timestamp of the file operation are combined, and the corresponding call chain identifier Trace ID and Span ID of the process are associated through the PID-Trace ID mapping table.

[0056] 5. Correlation methods for operating system layer indicator data The operating system metrics data generated by system calls, network connections, and disk I / O events related to application calls are associated with the Trace ID and Span ID of the corresponding process by the time the event occurred and the associated process information.

[0057] Specifically, because system calls, network connections, disk I / O, and other events at the operating system level do not directly contain Trace IDs and Span IDs, the relationships must be inferred from the event context, as follows: Timestamp anchoring: When generating Trace IDs and Span IDs at the application call layer, the call start time is recorded synchronously (accurate to the microsecond level). When the eBPF probe captures an operating system event (such as sys_socket sending network data), if the event time falls within the [start time, end time] interval of a certain Span, and the process identifier PID associated with the event is consistent with the source process identifier PID of that Span, it is determined to be a system event triggered by that Span. The system resource indicators (such as CPU utilization, disk I / O throughput, network bandwidth utilization, process scheduling latency), kernel event indicators (such as system call count, network connection establishment / disconnection count), and system status related indicators (such as the number of currently active processes, process scheduling latency) corresponding to the operating system event are associated with the corresponding Trace ID and Span ID.

[0058] Process-Call Binding: Maintains a real-time mapping table of "Process PID-Trace ID". When the application call layer's Span starts (e.g., when an RPC call begins), the current process identifier PID, Trace ID, and Span ID are written to the mapping table. When the PID associated with an operating system event (e.g., sys_write writes to disk) exists in the mapping table, and the event time is within the Span's lifecycle, the system resource metrics (e.g., CPU utilization, disk I / O throughput, network bandwidth utilization, process scheduling latency), kernel event metrics (e.g., number of system calls, number of network connection establishment / disconnection), and system status-related metrics (e.g., number of currently active processes, process scheduling latency) corresponding to the operating system event are directly associated with the Trace ID and Span ID corresponding to the process PID.

[0059] Network connection feature matching: For network-related system calls (such as connect, sendto), extract the event's four-tuple information (source IP, source port, destination IP, destination port) and event timestamp. A mapping table (actually containing four-tuples, TraceID, Span ID, and Span start and end times) is used to check if the extracted four-tuple exists in the network four-tuple-Trace ID mapping table, and if the event timestamp is within the lifecycle of the corresponding Span. If a match is successful, the corresponding operating system layer metrics data for that network system call (including network bandwidth utilization, number of network connection establishment / disconnection times, data transmission / reception volume, network connection latency, etc.) are directly associated with the matched Trace ID and Span ID. The network four-tuple-Trace ID mapping table stores a multi-dimensional mapping relationship between the network four-tuple, Trace ID, Span ID, and Span's time range.

[0060] The core safeguards of the aforementioned association mechanism are a dynamic mapping table and timestamp calibration. To ensure the accuracy of the association, two major support systems need to be built: Real-time mapping table: A dynamic mapping table such as "PID-Trace ID", "Connection ID-Trace ID", and "Network quadruple-Trace ID" is maintained in the kernel space and updated in real time by the eBPF probe (written at the start of the call and deleted at the end of the call), supporting millisecond-level queries.

[0061] Timestamp calibration: All levels of metric collection use the kernel monotonic clock (CLOCK_MONOTONIC) to avoid correlation deviations caused by system time synchronization errors and ensure the consistency of event timestamps.

[0062] The above methods enable precise correlation of metrics across the entire call chain, from application calls to the underlying operating system, providing complete data support for call chain analysis.

[0063] Step 104: Based on the associated multi-level indicator data, aggregate the indicator data associated with all Span IDs according to the Trace ID.

[0064] Step 105: Construct a hierarchical structure of the call chain based on the Parent Span ID, forming a complete resource call chain that starts from the initial application request and passes through the application components, database, process, and operating system in sequence.

[0065] To facilitate operations and maintenance personnel's intuitive understanding of the call chain, a visual interface is used to display the topology of the call chain, clearly presenting the call relationships between each link; it also displays the time consumed by each link, helping operations and maintenance personnel quickly identify performance bottlenecks; it displays the resource consumption of each link, allowing them to understand the system resource consumption of each link; and it displays the execution status of each link, clarifying whether each link is running normally. This achieves observability of the entire business chain, providing operations and maintenance personnel with intuitive and detailed evidence for troubleshooting and performance optimization.

[0066] To clearly demonstrate the construction process and topology of the call chain, as follows: Figure 2 As shown, it includes: Initial application request (e.g., HTTP / RPC request) → Generate Trace ID and initial Span ID (Parent SpanID is empty) → Trigger application component processing (generate new Span ID, Parent Span ID is the initial Span ID) → Trigger database operation (generate new Span ID, Parent Span ID is the application component Span ID) → Trigger process resource call (generate new Span ID, Parent Span ID is the database Span ID) → Trigger operating system resource call (generate new Span ID, Parent Span ID is the process Span ID) → Aggregate all Span data to form a complete call chain topology, and display the time consumption, resource consumption, and execution status of each step.

[0067] Furthermore, the multi-level indicator data collected in step 101 and the call chain data constructed in step 105 can be stored in a distributed time-series database. This type of database has the ability to efficiently store and quickly query massive amounts of data, and can meet the data storage and query needs in a large-scale host cluster environment.

[0068] Meanwhile, based on the stored metric data, functions such as real-time monitoring, anomaly alerts, and performance analysis are developed and provided.

[0069] Real-time monitoring function: It can display the changes of indicator data at all levels in real time, enabling operation and maintenance personnel to keep abreast of the operating status of host services; Anomaly alarm function: By setting reasonable indicator thresholds, such as excessively high CPU utilization, excessively long application call response time, excessive number of SQL execution failures, etc., when the indicator data exceeds the preset threshold, an alarm notification will be automatically triggered. The alarm information can be pushed to operation and maintenance personnel through various means such as email, SMS, and WeChat, ensuring that operation and maintenance personnel can be informed of anomalies in a timely manner; Performance analysis function: By statistically analyzing historical indicator data, it can uncover the performance patterns of host services, provide data support for optimizing service performance, help operation and maintenance personnel to discover and solve business problems in a timely manner, and ensure the stable and efficient operation of host services.

[0070] The overall flowchart of indicator data storage and analysis is as follows: Figure 3 As shown, the relationship between the functional modules includes: multi-level indicator data and call chain data → distributed time series database → data query and processing → anomaly alarm module (preset threshold → data comparison → trigger alarm → notify operation and maintenance personnel). Real-time monitoring module (displays indicator changes) → Anomaly alarm module (preset threshold → data comparison → trigger alarm → notify maintenance personnel).

[0071] To make the technical solution of this application clearer and easier to understand, the following detailed explanation of the non-intrusive method for achieving full service observability of the host is provided in conjunction with specific implementation cases.

[0072] The method described in this application enables full service observability of 100 domestically produced mainframes, and the following results were achieved after implementation: Low resource consumption: The average CPU utilization of the eBPF probe is 0.8%, and the average memory usage is 45MB, which has no impact on the performance of e-commerce transaction business (peak TPS: 5000); Improved operational efficiency: Probe deployment time has been reduced from 2 hours with traditional agents to 5 minutes, and troubleshooting time has been reduced from an average of 4 hours to 30 minutes (for example, in the case of a slow SQL execution issue, the missing database index was quickly located through the call chain, and the SQL execution time was reduced from 800ms to 100ms after optimization). Timely fault detection: Through the anomaly alarm mechanism, three potential faults were successfully captured (such as one sudden drop in Redis cache hit rate and two instances of Nginx connection limit exceeding the limit), and all of them were handled before the faults affected business operations; Good compatibility: The probe runs normally on all Kylin V5.0 hosts and is compatible with DM database and domestically adapted application components, with no compatibility issues.

[0073] Comparative analysis with existing technologies To clearly highlight the technical advantages of this application, this method is compared with current mainstream host observation techniques, as shown in the table below:

[0074] As can be seen from the comparison, this application is significantly superior to existing technologies in terms of invasiveness, compatibility, deployment efficiency, indicator coverage and security. In particular, it has formed a unique technical advantage in terms of adaptability and full-service observability in the domestic host environment.

[0075] This application achieves non-intrusive full-service observability of the host based on eBPF technology, which has the following significant advantages compared to traditional proxy methods: 1. Non-invasive and low resource consumption No agent program needs to be installed on the host machine; data collection can be completed simply by loading the eBPF probe into the kernel space, fundamentally avoiding the occupation of host CPU, memory, disk, and other resources by agent programs. Actual testing shows that the CPU utilization of the eBPF probe is less than 1% and memory usage is less than 50MB, with negligible impact on the performance of the host business system. It is particularly suitable for high-concurrency, high-load business scenarios, ensuring the stable operation of the business system.

[0076] 2. High compatibility and wide applicability. eBPF technology relies on the kernel's dynamic tracing mechanism, eliminating the need to develop adaptable versions for different operating system versions (such as Kylin V4 / V5, UnionTech UOS20 / 22, etc.). eBPF probes can run on different host versions using unified loading tools (such as bcc, bpftrace), and are compatible with mainstream databases and application components such as Nginx and Redis, significantly reducing development and maintenance costs and facilitating widespread application in large-scale host clusters.

[0077] 3. Easy deployment and upgrades, high operational efficiency Loading and upgrading eBPF probes can be completed in batches via remote commands or automated operation and maintenance tools (such as Ansible and SaltStack), without the need to log in to each host individually. In large-scale host cluster environments, probe deployment on all hosts can be completed within minutes simply by issuing probe loading commands through the operation and maintenance platform. During upgrades, only the eBPF bytecode file needs to be updated and reloaded, without stopping the business system, avoiding data collection interruptions that may occur during traditional agent upgrades, and significantly improving operation and maintenance efficiency.

[0078] 4. Safe and reliable, reducing security risks. eBPF probes run in kernel space and only require the minimum privileges needed for kernel tracing, unlike traditional agents which need to acquire high-level host privileges, thus reducing security risks at the source. Simultaneously, eBPF technology has a strict kernel security verification mechanism; the loaded eBPF bytecode must pass kernel security checks (such as sandbox verification and boundary checks) to prevent malicious code injection and ensure host kernel security. Furthermore, encrypted transmission and storage strategies are employed during data collection, effectively preventing the leakage of sensitive operational data and further enhancing overall security.

[0079] 5. Full business observability and precise problem localization It achieves full coverage of multi-level metrics from the operating system layer, process layer, database layer, application component layer to the application call layer, and completes the call chain connection through Trace ID and Span ID. Operations personnel can intuitively view the entire link operation status from the initial application request to the operating system resource call through a visual interface, including the time consumption, resource consumption and execution status of each link, and can quickly locate performance bottlenecks (such as slow SQL execution, application component response delay) and fault nodes (such as network connection failure, abnormal process exit), which greatly shortens the problem investigation time and improves the accuracy of operation and maintenance decisions.

[0080] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform any of the methods described above.

[0081] The hardware architecture of electronic devices / devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. A bus connects various circuits, including one or more processors, memories, and / or hardware modules. A bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc. Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be categorized as address buses, data buses, control buses, etc.

[0082] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0083] The specific implementation of each module in the above-mentioned device can be referred to the implementation process of the corresponding steps in the above-mentioned method implementation method of this disclosure, and will not be repeated here.

[0084] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0085] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0086] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A host service observation method, characterized in that, include: Load eBPF probes into the host's kernel space to collect metrics data from the host's operating system layer, process layer, database layer, application component layer, and application call layer. A unique Trace ID is generated for each application call request, and a unique Span ID is generated for each function call or cross-process call in the call chain. The Span ID contains a Parent Span ID, which is used to represent the hierarchical relationship in the call chain. By combining event context capture, process relationship mapping, and business trigger link tracing technologies, the collected indicator data from each layer are associated with the generated call chain identifier Trace ID and span identifier Span ID; Based on the associated multi-level metric data, the metric data associated with all span IDs are aggregated according to the Trace ID; the hierarchical structure of the call chain is constructed according to the Parent Span ID, forming a complete resource call chain that starts from the initial application request and passes through the application components, database, process and operating system in sequence.

2. The method according to claim 1, characterized in that, By combining event context capture, process relationship mapping, and business trigger link tracing technologies, the collected metric data from each layer are associated with the generated call chain identifier Trace ID and span identifier Span ID, including: By actively embedding data and passing context information, application call layer data is associated with the Trace ID and Span ID corresponding to function calls or cross-process calls. Based on the association between application components and application calls through protocols and interfaces, the application component layer metric data is associated with the call chain identifier Trace ID and span identifier Span ID extracted from protocol requests and interfaces; Based on the triggering relationship between SQL operations and application calls, the database layer metric data is associated with the call chain identifier TraceID and the span identifier Span ID; Based on the hierarchical relationship between processes and application calls, process-level metric data involved in application calls are associated with the corresponding call chain identifier Trace ID and span identifier Span ID; The operating system-level metrics data generated by system calls, network connections, and disk I / O events related to application calls are associated with the Trace ID and Span ID of the corresponding process through the time of the event and the associated process information.

3. The method according to claim 2, characterized in that, By proactively embedding data and passing context information, application call layer data is associated with the Trace ID and Span ID corresponding to the function call or cross-process call, including: At the function call entry point of the application, the eBPF probe is triggered by the uprobe to generate or inherit the call chain identifier TraceID and span identifier Span ID, and directly bind the function name, parameters, and timestamp information to the Trace ID and Span ID; For RPC calls, the eBPF probe writes the Trace ID and Span ID into the protocol header when serializing the request data; when the receiver parses the protocol header, it extracts the Trace ID and Span ID to ensure that the metric data at both ends of the cross-process call are associated with the same Trace ID and Span ID. If a function call is not actively captured, the function call stack of the current process is obtained through eBPF, the parent function of the nearest associated Span is found, and the uncaptured function call is used as the Trace ID to associate the child Span with the parent function.

4. The method according to claim 2, characterized in that, Based on the association between application components and the protocols and interfaces called by the application, the application component layer metric data is associated with the Trace ID and Span ID extracted from the protocol requests and interfaces, including: For HTTP protocol Web components, when parsing the HTTP request header, the eBPF probe extracts the call chain identifier TraceID and span identifier Span ID from it, and associates the request processing indicator data of the Web component with the extracted call chain identifier Trace ID and span identifier Span ID; For the caching component, when the application calls Redis through the SDK, the eBPF probe captures the Trace ID and Span ID of the current application call in the SDK's entry function and writes them as key prefixes to the Redis operation. The application component layer parses the Trace ID and Span ID in the key name and associates them with the cache operation metric data. When application components and business processes communicate via local sockets, the eBPF probe injects the Trace ID and Span ID into the sendmsg function of the sender, and extracts the Trace ID and Span ID in the recvmsg function of the receiver, thus realizing cross-process component indicator data association.

5. The method according to claim 2, characterized in that, Based on the triggering relationship between SQL operations and application calls, database layer metric data is associated with the call chain identifier Trace ID and span identifier Span ID, including: When an application establishes a database connection, the eBPF probe injects the Trace ID and Span ID of the current application call into the connection function, and stores the database connection ID-Trace ID mapping relationship in the kernel cache. All subsequent SQL operations on this connection are associated with the corresponding Trace ID and Span ID through the connection ID. For scenarios without an explicit connection ID, the business identifier is extracted from the SQL statement and directly matched with the Trace ID. If the business identifier does not exist in the SQL statement, the corresponding Trace ID and Span ID are obtained by associating the process identifier (PID) and timestamp of the SQL operation with the process identifier (PID) - call chain identifier (Trace ID) mapping table. The process identifier (PID) - call chain identifier (Trace ID) mapping table stores a multi-dimensional mapping relationship between the process identifier (PID), call chain identifier (Trace ID), span identifier (Span ID), and the time range of the span. By capturing the database audit log output through eBPF probes, the client IP and port, operation time, SQL statement contained in the logs are matched with the cross-process call information of the application call layer, and then the database layer indicator data in the logs are associated with the call chain identifier Trace ID and span identifier Span ID.

6. The method according to claim 2, characterized in that, Based on the hierarchical relationship between processes and application calls, process-level metric data involved in application calls are associated with the corresponding call chain identifier (Trace ID) and span identifier (Span ID), including: When an application calls a process to trigger process creation or thread startup, the eBPF probe captures the parent process's Trace ID and Span ID in the process and thread's entry function and inherits them to the child process and thread. These are written into a maintained process identifier (PID) - Trace ID mapping table. All subsequent behavioral metrics of the child process are associated with the parent's Trace ID through this mapping table. The PID - Trace ID mapping table stores a multi-dimensional mapping relationship between the process identifier (PID), Trace ID, Span ID, and the time range of the span. For processes whose runtime exceeds the set duration, their resource usage is associated with the process's corresponding call chain identifier Trace ID and span identifier Span ID by time slice; If the file path of a process involves a business identifier, the Trace ID is directly matched; otherwise, the process identifier (PID) and timestamp of the file operation are combined, and the corresponding call chain identifier (Trace ID) and span identifier (Span ID) of the process are associated through the process identifier (PID) - call chain identifier (Trace ID) mapping table. The process identifier (PID) - Trace ID mapping table stores a multi-dimensional mapping relationship of process identifier (PID), call chain identifier (Trace ID), span identifier (Span ID), and time range of span.

7. The method according to claim 2, characterized in that, Operating system-level metrics data generated by system calls, network connections, and disk I / O events related to application calls are linked to the process's corresponding call chain identifier (Trace ID) and span identifier (Span ID) based on the event's occurrence time and associated process information. This includes: When generating the Trace ID and Span ID at the application call layer, the call start time is recorded synchronously. When the eBPF probe captures an operating system event, if the event time falls within the lifecycle of a certain Span and the process identifier PID associated with the event is consistent with the source process identifier PID of that Span, it is determined to be a system event triggered by that Span. The system resource indicators, kernel event indicators, and system state associated indicators corresponding to the operating system event are associated with the corresponding Trace ID and Span ID. When the application call layer's Span starts, the current process identifier (PID), call chain identifier (Trace ID), and span identifier (Span ID) are written into the maintained process identifier PID-call chain identifier (Trace ID) mapping table. When the current process identifier PID associated with an operating system event exists in the mapping table and the event time is within the Span's lifecycle, the system resource indicators, kernel event indicators, and system state-related indicators corresponding to the operating system event are directly associated with the call chain identifier (Trace ID) and span identifier (Span ID) corresponding to the process PID. The process identifier PID-Trace ID mapping table stores a multi-dimensional mapping relationship between the process identifier PID, call chain identifier (Trace ID), span identifier (Span ID), and the time range of the span. For network-related system calls, extract the network quadruple information of the event; using the maintained network quadruple-call identifier Trace ID mapping table, if the extracted network quadruple is consistent with the cross-process call network characteristics of a certain Span in the application call layer and the timestamp matches, then associate the operating system layer indicator data corresponding to the network-related system call with the call chain identifier Trace ID and span identifier Span ID of that Span. The network quadruple-call identifier Trace ID mapping table stores a multi-dimensional mapping relationship of network quadruple, call chain identifier Trace ID, span identifier Span ID, and the time range of the span.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images