Methods, devices, equipment, and storage media for generating ledger data

Abstract

This application relates to a method, apparatus, device, and storage medium for generating ledger data. The method includes: attaching probes of an eBPF program to the entry and / or exit points of a target function in a target gateway, capturing data at the entry and / or exit points of the target function to obtain target captured data; performing statistical analysis on the target captured data to determine target indicators related to the target captured data; querying the target context in a preset gateway service based on the target gateway data of the target gateway; and generating ledger data for the target gateway based on the target captured data, target indicators, and target context. This method avoids the complex operation of decrypting encrypted data packets, fundamentally solving the problem of existing technologies being unable to generate ledgers due to the inability to decrypt data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating ledger data. Background Technology

[0002] With the acceleration of digitalization, various network services and applications are emerging in an endless stream, making network architecture increasingly complex. Against this backdrop, gateways, as key hubs in network communication, generate ledger data that is of paramount importance for many aspects, including network management, business analysis, and security monitoring.

[0003] In existing technologies, the common method for obtaining gateway ledger data is to capture data packets from the gateway. By capturing these data packets and using specific data analysis methods and tools, valuable information is extracted, and the required ledger data is then constructed.

[0004] However, since the packet capture program runs independently of the gateway, this method has significant limitations, only applicable to capturing unencrypted data packets. Once encrypted data packets are captured, the program lacks the decryption key and cannot extract valid data. This not only restricts the integrity of the data acquisition but also severely impacts the generation of subsequent ledger data. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for generating ledger data. By embedding probes at the entry and / or exit points of encryption and decryption functions, precise monitoring of key nodes in data flow can be achieved. Before data enters the encryption process or after decryption is completed, the probes can quickly capture the unencrypted data and then generate ledger data based on this data. Unlike traditional packet capture methods that are hampered by encryption, this probe-based approach bypasses the encryption barrier, avoids the complex operation of decrypting encrypted data packets, and fundamentally solves the problem of existing technologies being unable to generate ledgers due to the inability to decrypt.

[0006] Firstly, this application provides a method for generating ledger data, the method comprising: By attaching the probes of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; Perform statistical analysis on the target capture data to determine target indicators related to the target capture data; Based on the target gateway data of the target gateway, query the target context in the preset gateway service; Based on the target captured data, the target metrics, and the target context, the ledger data of the target gateway is generated.

[0007] Optionally, generating the ledger data of the target gateway based on the target captured data, the target metrics, and the target context includes: The request plaintext data and response plaintext data in the target captured data are matched to obtain composite transaction data; The composite transaction data is sampled using an intelligent sampling strategy to obtain sampled composite transaction data. Based on the sampled composite transaction data, the target metric, and the target context, the ledger data of the target gateway is generated.

[0008] Optionally, the step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Obtain the target status code corresponding to the response plaintext data in the composite transaction data; When the target status code in the response plaintext data is an abnormal status code, the composite transaction data is determined to be the sampled composite transaction data.

[0009] Optionally, the step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Obtain the API endpoint corresponding to the composite transaction data; When the API endpoint is the first API endpoint captured, the composite transaction data is determined to be the sampled composite transaction data.

[0010] Optionally, the step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Statistics on the API endpoints corresponding to all composite transaction data; Based on the API endpoint corresponding to each composite transaction data, determine the target API endpoint whose call frequency is greater than a preset value and has no anomalies. Using a preset first sampling frequency, the composite transaction data corresponding to the API endpoint is sampled to obtain the sampled composite transaction data.

[0011] Optionally, when the target function is a decryption function, the step of attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, and capturing data at the entry and / or exit of the target function to obtain target captured data includes: When the target gateway uses the decryption function to decrypt the received encrypted request, the exit probe of the decryption function is triggered to obtain the plaintext data of the request; The plaintext request data is copied to the specified area to obtain the target capture data.

[0012] Optionally, when the target function is an encryption function, the step of attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, and capturing data at the entry and / or exit of the target function to obtain target captured data includes: When the target gateway uses the encryption function to encrypt the response plaintext data, the exit probe of the encryption function is triggered to obtain the response plaintext data; The plaintext response data is copied to a specified area to obtain the target capture data.

[0013] Secondly, this application provides an apparatus for generating ledger data, the apparatus comprising: The capture unit is used to capture data at the entry and / or exit of the target function by attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, thereby obtaining target capture data. An analysis unit is used to perform statistical analysis on the target capture data and determine target indicators related to the target capture data. The query unit is used to query the target context in a preset gateway service based on the target gateway data of the target gateway; The generation unit is used to generate ledger data for the target gateway based on the target captured data, the target indicators, and the target context.

[0014] Optionally, the generation unit is used for: The request plaintext data and response plaintext data in the target captured data are matched to obtain composite transaction data; The composite transaction data is sampled using an intelligent sampling strategy to obtain sampled composite transaction data. Based on the sampled composite transaction data, the target metric, and the target context, the ledger data of the target gateway is generated.

[0015] Optionally, the generation unit is used for: Obtain the target status code corresponding to the response plaintext data in the composite transaction data; When the target status code in the response plaintext data is an abnormal status code, the composite transaction data is determined to be the sampled composite transaction data.

[0016] Optionally, the generation unit is used for: Obtain the API endpoint corresponding to the composite transaction data; When the API endpoint is the first API endpoint captured, the composite transaction data is determined to be the sampled composite transaction data.

[0017] Optionally, the generation unit is used for: Statistics on the API endpoints corresponding to all composite transaction data; Based on the API endpoint corresponding to each composite transaction data, determine the target API endpoint whose call frequency is greater than a preset value and has no anomalies. Using a preset first sampling frequency, the composite transaction data corresponding to the API endpoint is sampled to obtain the sampled composite transaction data.

[0018] Optionally, when the target function is a decryption function, the capturing unit is used to: When the target gateway uses the decryption function to decrypt the received encrypted request, the exit probe of the decryption function is triggered to obtain the plaintext data of the request; The plaintext request data is copied to the specified area to obtain the target capture data.

[0019] Optionally, when the target function is an encryption function, the capturing unit is used to: When the target gateway uses the encryption function to encrypt the response plaintext data, the exit probe of the encryption function is triggered to obtain the response plaintext data; The plaintext response data is copied to a specified area to obtain the target capture data.

[0020] Thirdly, this application provides a ledger data generation device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: By attaching the probes of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; Perform statistical analysis on the target capture data to determine target indicators related to the target capture data; Based on the target gateway data of the target gateway, query the target context in the preset gateway service; Based on the target captured data, the target metrics, and the target context, the ledger data of the target gateway is generated.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for generating ledger data.

[0022] Compared with the prior art, the technical solution provided in this application has the following advantages: By attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; statistical analysis is performed on the target captured data to determine the target indicators related to the target captured data; the target context is queried in the preset gateway service based on the target gateway data of the target gateway; and ledger data of the target gateway is generated based on the target captured data, the target indicators, and the target context. By implanting probes at the entry and / or exit of encryption and decryption functions, precise monitoring of key nodes in data flow can be achieved. Before data enters the encryption process or after the decryption operation is completed, the probe can quickly capture these unencrypted data, and then generate ledger data based on these data. Unlike traditional packet capture methods that are hampered by encryption, this probe-based technical path bypasses the encryption barrier, avoids the complex operation of decrypting encrypted data packets, and fundamentally solves the problem that existing technologies cannot generate ledgers due to the inability to decrypt. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0026] Figure 1 A flowchart illustrating a method for generating ledger data provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for generating ledger data provided in an embodiment of this application; Figure 3 A schematic diagram of a ledger data generation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of a ledger data generation device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0029] With the acceleration of digitalization, various network services and applications are emerging in an endless stream, making network architecture increasingly complex. Against this backdrop, gateways, as key hubs in network communication, generate ledger data that is crucial for network management, business analysis, and security monitoring. Currently, the common method for obtaining gateway ledger data is to capture data packets within the gateway. By capturing these packets and using specific data analysis methods and tools, valuable information is extracted to construct the required ledger data. However, packet capture programs run independently of the gateway, and this current method has significant limitations, only applicable to capturing unencrypted data packets. Once encrypted data packets are captured, the packet capture program lacks the decryption key and cannot extract valid data. This situation not only restricts the integrity of the data acquisition but also severely impacts the subsequent generation of ledger data.

[0030] In summary, this application provides a method for generating ledger data. This method can achieve precise monitoring of key nodes in data flow by implanting probes at the entry and / or exit points of encryption and decryption functions. Before data enters the encryption process or after decryption, the probes can quickly capture the unencrypted data and then generate ledger data based on this data. Unlike traditional packet capture methods that are hampered by encryption, this probe-based approach bypasses the encryption barrier, avoids the complex operation of decrypting encrypted data packets, and fundamentally solves the problem of existing technologies being unable to generate ledgers due to the inability to decrypt. Figure 1 As shown, the specific steps include: Step 101: By attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data.

[0031] The target functions are encryption and decryption functions. When the target function is encryption, the probe is set at the entry point of the encryption function. When the target function is decryption, the probe is set at the exit point of the decryption function. Specifically, the encryption function is SSL_read, and the decryption function is SSL_write.

[0032] The ledger generation system includes an eBPF program, which sets eBPF probes at the entry and / or exit points of the target functions. The specific setup steps are as follows: Since the gateway process address and the memory address of the encryption function are not fixed and cannot be hardcoded into a configuration file, automatic scanning is required to automatically mount the probes. No manual configuration is needed; the system automatically adapts to gateway restarts or address changes. When automatic probe mounting is required, the system scans the real-time information directory of each process on the host, i.e., the / proc directory, at a preset period (e.g., every 10 seconds). This directory contains all running processes on the host, each with a corresponding PID. By reading the / proc / [pid] / comm or / proc / [pid] / cmdline file, the system obtains the process name for each PID. These process names are then matched against a predefined gateway whitelist (e.g., gateway process names are worker process, apisix, kong) to identify the PIDs corresponding to all running gateway processes. For each identified PID, its memory-mapped file ` / proc / [pid] / maps` is parsed. This file records the memory mapping relationships of all files and libraries loaded into memory by the current process, i.e., which libraries are loaded into memory and their starting memory address (base address). Since the gateway's core encryption / decryption capabilities all rely on the `libssl.so` shared encryption library, the base address of the shared encryption library that the gateway process depends on is located by searching for keywords such as `libssl.so` in this file. After obtaining the base address of the shared encryption library, the ELF (Executable and Linkable Format) header information of the library file is parsed, and its dynamic symbol table (`.dynsym`) is read. This dynamic symbol table includes all function names in `libssl.so` (such as `SSL_read`, `SSL_write`, and `SSL_connect`), as well as the offset of each function within the library file. By traversing the symbol table, the offset of the target function (the `SSL_read` and `SSL_write` functions) relative to the base address is determined. Adding the obtained base memory address to the function offset yields the absolute address of the target function in the current process's virtual memory. Then, the `bpf()` system call is invoked to attach eBPF probes (the entry probe `uprobe` and the exit probe `uretprobe`) to these addresses. Because this process is dynamic, even if the gateway restarts or the library file address changes due to ASLR (Address Space Layout Randomization), it can be rediscovered and reattached.

[0033] In this step, after receiving the encryption request, the target gateway calls the decryption function to decrypt the request, obtaining the plaintext request data. At this time, the probe deployed at the decryption function's exit point is triggered, directly initiating the eBPF program to acquire the plaintext request data and store it in a designated area of ​​the kernel, thus obtaining the target captured data. Next, the target gateway generates a response based on the plaintext request data, obtaining the plaintext response data. Before calling the encryption function to encrypt this plaintext response data, the probe deployed at the encryption function's entry point is triggered, initiating the eBPF program to acquire the plaintext response data and store it in a designated area of ​​the kernel, thus obtaining the target captured data. Subsequently, the target gateway performs an encryption operation on the plaintext response data, generating response information and sending it to the requester.

[0034] After storing both request and response plaintext data in a preset location in the kernel, the eBPF program can mark and filter this data: it checks whether the data meets preset marking conditions; if it does, it is marked; otherwise, it is not marked. After marking, the eBPF program sends the marked data from the preset location to the raw-api-transactions topic (a topic in the message queue called the raw transaction data topic) of the ledger generation system at preset time intervals. The matching module in the ledger generation system subscribes to the raw-transactions-topic messages. Whenever new data is stored in the raw-transactions-topic, the matching module retrieves that data. If the data is response plaintext data, it pairs the data to obtain composite transaction data, and sends this composite transaction data to the ledger generation engine so that the ledger generation engine can generate ledger data based on the composite transaction data.

[0035] Based on the above, this application provides a detailed description of the composition of the ledger generation system: The ledger generation system includes an eBPF program, a data bus module, a matching module, and a ledger generation engine. The eBPF program is used for front-end collection of plaintext data and acquisition of indicator data. The data bus module stores the data uploaded by the eBPF program, acting as a "shock absorber" and "decoupling layer" for the system architecture, allowing the eBPF program and the ledger generation engine to work independently and scale at their own pace, greatly enhancing the resilience and stability of the entire system. The matching module is used to match the collected data, and the ledger generation engine is used to generate ledger data.

[0036] The aforementioned data bus module typically consists of a distributed message queue cluster, with Apache Kafka being the preferred technology due to its high throughput, persistence, partitioning, and high availability. The data bus module includes two topics: the raw-api-transactions topic and the aggregated-api-metrics topic. The raw-api-transactions topic stores plaintext data; since plaintext data is discarded after subsequent consumption, its retention period is relatively short. The aggregated-api-metrics topic stores subsequently obtained metrics, and because it contains a continuous historical record of system status, it can be used for trend analysis and backtracking; therefore, its data retention period can be set to a longer time.

[0037] Step 102: Perform statistical analysis on the target acquisition data to determine the target indicators related to the target acquisition data.

[0038] Among them, target metrics are key operational indicators obtained by analyzing target captured data. Target metrics can include data such as the number of response anomalies, the number of requests, the number of successful responses, and the frequency of error code distribution.

[0039] In this step, after acquiring the plaintext request and response data, the eBPF program performs deep analysis to extract and calculate target metrics. The specific process is as follows: An eBPF mapping table of type BPF_MAP_TYPE_HASH (named metrics_map) is initialized. This mapping table uses API endpoints as keys and structures containing multi-dimensional counters as values. Each time plaintext request data is captured, the corresponding API endpoint is used as the index key, and the request_count counter in the value corresponding to that key is atomically incremented to achieve real-time statistics of the request count. Each time plaintext response data is captured, the status code counter is atomically incremented based on the HTTP status code carried in the response plaintext data: if the status code is a 2xx success status code, status_2xx_count is incremented; if the status code is a 5xx server error status code, status_5xx_count is incremented, thus achieving categorized counting of different response results. When the probes at the exit of the decryption function and the entry of the encryption function are triggered, the bpf_ktime_get_ns() function is called to obtain the corresponding timestamps respectively. The processing latency of a single request is calculated by the difference between the two timestamps. At the same time, the latency data is updated to the eBPF mapping table of type BPF_MAP_TYPE_HISTOGRAM, and the latency bucketing statistics are completed directly in kernel mode.

[0040] After the eBPF program obtains the target metrics, it sends these metrics to another independent topic, such as the aggregated-api-metrics topic, via the data bus.

[0041] Step 103: Based on the target gateway data of the target gateway, query the target context in the preset gateway service.

[0042] In this step, the ledger generation system obtains the target gateway data and, based on this data, searches for the target context related to the target gateway in the gateway service. Specifically, the ledger generation system can interact with external metadata services to obtain richer context information about the target gateway, such as Pod name, service name, and namespace.

[0043] Step 104: Generate the target gateway's ledger data based on the target capture data, target metrics, and target context.

[0044] Among them, the ledger data is a standardized, structured monitoring and audit record that integrates captured data, indicator statistical results, and target context.

[0045] In this step, the ledger generation system summarizes the composite transaction data, target indicator set, and context data to generate ledger records corresponding to each transaction ID. The ledger records corresponding to these transaction IDs are then summarized to obtain the ledger data of the target gateway.

[0046] At the gateway host, located close to the data source, eBPF (Extended Berkeley Packet Filter) technology is used to "spy" on the behavior of the API gateway application at the operating system kernel level. By setting dynamic probes on encryption functions, plaintext data entering and leaving the gateway is securely captured in a "zero-contact" manner. Simultaneously, eBPF's efficient kernel-mode aggregation capabilities are used to perform real-time statistics on all API calls. An adaptive sampling engine, based on these real-time statistical metrics, filters only the raw packets containing the most valuable data for analysis (such as errors, slow calls, and new interfaces), while only recording aggregated metrics for the massive amounts of normal and recurring calls.

[0047] In the ledger generation engine, which is far removed from the production environment, the two types of heterogeneous data received (low-frequency, detailed sampled data and high-frequency, lightweight aggregated metrics) are fused and processed. It is responsible for performing heavy computational tasks such as stateful request / response correlation, and finally reconstructs a "panoramic API ledger" that has both a global macro view ("forest") and specific micro details ("trees").

[0048] In this embodiment, by attaching probes from an eBPF program to the entry and / or exit points of a target function in a target gateway, data is captured at the entry and / or exit points of the target function to obtain target captured data. Statistical analysis is performed on the target captured data to determine target indicators related to it. Based on the target gateway data of the target gateway, the target context is queried in a preset gateway service. Ledger data for the target gateway is generated based on the target captured data, the target indicators, and the target context. By implanting probes at the entry and / or exit points of encryption and decryption functions, precise monitoring of key nodes in data flow can be achieved. Before data enters the encryption process or after decryption is completed, the probes can quickly capture this unencrypted data, and then generate ledger data based on this data. Unlike traditional packet capture methods that are hampered by encryption, this probe-based approach bypasses the encryption barrier, avoids the complex operation of decrypting encrypted data packets, and fundamentally solves the problem of existing technologies being unable to generate ledgers due to the inability to decrypt.

[0049] In this embodiment, to reduce the transmission and storage burden of the data upload link, the plaintext request data and plaintext response data in the target captured data are correlated and matched to construct complete composite transaction data; an intelligent sampling strategy is used to filter and sample the composite transaction data to obtain composite transaction sample data that meets the analysis requirements; finally, the sampled composite transaction data, target indicators, and target context are combined to generate the ledger data of the target gateway. Therefore, this embodiment provides a ledger data generation method, which is as follows: Figure 2 As shown, the specific steps include: Step 201: Match the request plaintext data and response plaintext data in the target captured data to obtain composite transaction data.

[0050] In this step, when the eBPF program captures plaintext request data, a composite transaction ID is generated based on preset rules and written into the plaintext request data. When the gateway generates plaintext response data, this transaction ID is reused and written into the plaintext response data. Both the plaintext request data and the plaintext response data carry a transaction ID, which is used to uniquely identify the business relationship between the request and the response. When processing the target captured data, the transaction ID carried in the plaintext response data is first extracted, and then a matching query is performed in the plaintext request data based on this transaction ID. If plaintext request data carrying the same transaction ID is found, the plaintext request data corresponding to that transaction ID is associated with the plaintext response data, and combined to form the composite transaction data corresponding to that transaction ID.

[0051] The composite transaction ID includes the current process ID, thread ID, TCP four-tuple, and / or HTTP / 2 stream ID. It covers three levels: process, network connection, and application layer protocol, and is a unique identifier combination with high entropy. Its specific generation rules are as follows: To achieve precise matching of requests and responses, the eBPF program calls `pf_get_current_pid_tgid()` to obtain the current process ID and thread ID; simultaneously, it captures the TCP connection four-tuple (source IP, source port, destination IP, destination port) associated with the current socket by hooking the kernel network function (such as the `tcp_sendmsg` function) using `kprobe` (kernel-mode probe). For HTTP / 2 protocol requests, the application layer stream ID is additionally extracted; for HTTP / 1.1 protocol requests, the stream ID dimension can be omitted. The above information is combined to generate the composite transaction ID.

[0052] It should be noted that in practice, due to memory overflow or database connection crashes during business logic execution, the gateway process may be forcibly terminated and unable to assemble response data. This means that each request plaintext data may not have a corresponding response plaintext data, but the response plaintext data will definitely have a corresponding request plaintext data. Therefore, the corresponding request plaintext data can be found by using the transaction ID in the response plaintext data.

[0053] Step 202: Use an intelligent sampling strategy to sample the composite transaction data to obtain the sampled composite transaction data.

[0054] In practice, a gateway often has multiple API endpoints, which are standardized interface access points exposed by the gateway. Therefore, it's possible to acquire composite transaction data corresponding to each API endpoint on the target gateway. When the plaintext response data in a composite transaction includes an error code or the corresponding latency exceeds a threshold, this composite transaction must be captured and marked for use as target capture data. Similarly, when the API endpoint corresponding to a composite transaction is a new API endpoint, this composite transaction must also be captured and marked for use as target capture data. Furthermore, sampling can be performed based on a preset sampling frequency. To reduce the sampling burden, if the call frequency of an API endpoint exceeds a preset value, and the API endpoint shows no anomalies, it indicates a high call frequency but no anomalies, and the data corresponding to that API endpoint can be sampled at a lower sampling frequency.

[0055] The call frequency corresponding to the API endpoint refers to the cumulative number of requests received by the API endpoint within a preset statistical period.

[0056] Step 203: Generate the ledger data of the target gateway based on the sampled composite transaction data, target indicators, and target context.

[0057] In this embodiment, to ensure the complete capture of abnormal transaction data and enhance the fault tracing value of ledger data, sampling is triggered based on the abnormal status code of the response plaintext data. This enables targeted locking of all abnormal transactions during the gateway's request processing, avoiding the problem of missing abnormal data due to probability in conventional random sampling. Therefore, this application provides a data sampling method, including: obtaining the target status code corresponding to the response plaintext data in the composite transaction data; when the target status code in the response plaintext data is an abnormal status code, determining the composite transaction data as the sampled composite transaction data.

[0058] In this step, the target status code is extracted from the plaintext response data of the composite transaction data. This target status code includes two categories: standard protocol status codes (such as HTTP 4xx client error codes and 5xx server error codes) and gateway-defined exception status codes (such as gateway permission verification failure codes, data decryption failure codes, and backend service call timeout codes). An exception status code whitelist is pre-configured to clearly define the range of exception status codes that need to trigger sampling. The extracted target status code is compared with the exception status code whitelist. If the target status code belongs to an exception status code within the whitelist, the composite transaction data is directly identified as the sampled composite transaction data; if the target status code is a normal status code, no sampling is performed, and the composite transaction data is directly released to avoid redundant data consuming resources.

[0059] In this application embodiment, during the initial stage of a new API endpoint's launch, its operational stability, compatibility, and response logic are all in the verification phase. The first / initial batch of composite transaction data is the core basis reflecting its initial operational status. This application ensures that no such data is missed through full sampling, providing developers / operations personnel with complete initial information about the new API endpoint and preventing hidden problems from going undetected in the early stages of the new API endpoint's launch due to sampling omissions. Therefore, this application embodiment provides a sampling method, the specific steps of which include: obtaining the API endpoint corresponding to the composite transaction data; when the API endpoint is the first captured API endpoint, determining the composite transaction data as the sampled composite transaction data.

[0060] In this step, a list of captured API endpoints is maintained in advance. This list is a dynamically updated dataset that records all API endpoints that have been captured during the gateway's operation, along with their first capture timestamp. After acquiring an API endpoint with composite transaction data, it is compared with the list of captured API endpoints. If no matching endpoint is found in the list, the API endpoint is determined to be a newly captured API endpoint. Simultaneously, a list update mechanism is triggered to add the identifier and current timestamp of the new API endpoint to the list, preventing subsequent endpoints of the same type from being repeatedly identified as newly captured. Once an API endpoint is determined to be a newly captured API endpoint, its corresponding composite transaction data is immediately fully sampled, meaning that the first / initial batch of composite transaction data for that API endpoint is completely retained without omissions or filtering.

[0061] In this embodiment, high-frequency, anomaly-free API endpoints are the core business endpoints of the gateway. They have high call frequency and large data volume; full sampling would consume significant storage and computing resources. This application uses a first sampling frequency for non-full targeted sampling, avoiding the excessive gateway load caused by full sampling while retaining the operational characteristics of these API endpoints through reasonable sampling. This achieves the dual goals of minimizing resource consumption and ensuring data integrity. Therefore, this embodiment provides a data sampling method, specifically including: statistically analyzing the API endpoints corresponding to all composite transaction data; determining target API endpoints with call frequencies greater than a preset threshold and no anomalies based on the API endpoints corresponding to each composite transaction data; and sampling the composite transaction data corresponding to the API endpoints using a preset first sampling frequency to obtain the sampled composite transaction data.

[0062] The first sampling frequency is a low-frequency sampling frequency, such as sampling composite transaction data at a ratio of 1:100, that is, sampling 1 data item for every 100 data items, or sampling at fixed time intervals. The preset threshold is a comprehensive setting of the call frequency threshold based on the target gateway's historical operating data, peak business traffic, and operational experience.

[0063] In this step, the API endpoint identifiers (such as interface path, interface number, service name + interface name combination identifier, etc.) carried in each composite transaction data are extracted to determine the API endpoint corresponding to each composite transaction data. The number of calls to each API endpoint is counted within a preset statistical time window to obtain the call frequency of each API endpoint. The call frequency of each API endpoint is compared with a preset threshold to filter out candidate API endpoints whose call frequency is greater than a preset value. Finally, the candidate API endpoints are checked for abnormal states. The verification criteria include: all response status codes of the composite transaction data corresponding to the API endpoint are normal status codes (such as HTTP2xx series status codes), the response time is within a preset normal range, and there are no abnormal records such as data format errors or link interruptions. Then, candidate API endpoints with any abnormal records within the time window are removed, and the target API endpoints that meet the "high frequency and no abnormalities" condition are finally determined. For the selected target API endpoints, a preset first sampling frequency is used to perform targeted sampling.

[0064] In this embodiment, when the target function is a decryption function, a probe can be placed at the exit of the decryption function. Thus, upon receiving an encryption request from the client, the decryption function is used to decrypt the request, obtaining the plaintext request data. This triggers the probe at the decryption function's exit, which then copies the plaintext request data to a designated area, obtaining the target capture data. Therefore, this embodiment provides a method for determining target capture data, with specific steps including: When the target gateway uses the decryption function to decrypt the received encrypted request, the exit probe of the decryption function is triggered to obtain the plaintext data of the request; the plaintext data of the request is copied to the specified area to obtain the target captured data.

[0065] In this step, to ensure data transmission security, the client encrypts the request information before sending it to the target gateway. The target gateway is constantly listening, waiting to receive encrypted requests from the client. When the gateway successfully receives the encrypted request, it calls a pre-defined decryption function to parse the encrypted request, restoring the encrypted information to its original readable form, thus obtaining the plaintext request data. At the moment the decryption function completes its decryption operation and outputs the plaintext request data, its exit probe is simultaneously triggered. The exit probe is similar to a signal trigger, designed to mark the end of the decryption function's execution flow. Afterward, the eBPF program copies the plaintext request data to a designated area. This designated area is a pre-planned storage space for storing specific data; it could be a segment of memory address space, a specific database table, or a file directory, depending on the architecture design of the ledger system.

[0066] In this embodiment, when the target function is an encryption function, the target captured data is obtained by attaching the probe of the eBPF program to the entry point of the target function in the target gateway and capturing data at the entry point of the target function. Therefore, this embodiment provides a method for determining target captured data, the specific steps of which include: when the target gateway uses an encryption function to encrypt the plaintext response data, triggering the exit probe of the encryption function to obtain the plaintext response data; and copying the plaintext response data to a specified area to obtain the target captured data.

[0067] like Figure 3 As shown, this application embodiment provides a ledger data generation device, which corresponds to the method embodiment, and specifically includes: The capture unit 301 is used to capture data at the entry and / or exit of the target function by attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, thereby obtaining target capture data. Analysis unit 302 is used to perform statistical analysis on the target capture data and determine target indicators related to the target capture data; The query unit 303 is used to query the target context in a preset gateway service based on the target gateway data of the target gateway. The generation unit 304 is used to generate the ledger data of the target gateway based on the target captured data, the target indicators and the target context.

[0068] Optionally, the generation unit 304 is used for: The request plaintext data and response plaintext data in the target captured data are matched to obtain composite transaction data; The composite transaction data is sampled using an intelligent sampling strategy to obtain sampled composite transaction data. Based on the sampled composite transaction data, the target metric, and the target context, the ledger data of the target gateway is generated.

[0069] Optionally, the generation unit 304 is used for: Obtain the target status code corresponding to the response plaintext data in the composite transaction data; When the target status code in the response plaintext data is an abnormal status code, the composite transaction data is determined to be the sampled composite transaction data.

[0070] Optionally, the generation unit 304 is used for: Obtain the API endpoint corresponding to the composite transaction data; When the API endpoint is the first API endpoint captured, the composite transaction data is determined to be the sampled composite transaction data.

[0071] Optionally, the generation unit 304 is used for: Statistics on the API endpoints corresponding to all composite transaction data; Based on the API endpoint corresponding to each composite transaction data, determine the target API endpoint whose call frequency is greater than a preset value and has no anomalies. Using a preset first sampling frequency, the composite transaction data corresponding to the API endpoint is sampled to obtain the sampled composite transaction data.

[0072] Optionally, when the target function is a decryption function, the capture unit 301 is used to: When the target gateway uses the decryption function to decrypt the received encrypted request, the exit probe of the decryption function is triggered to obtain the plaintext data of the request; The plaintext request data is copied to the specified area to obtain the target capture data.

[0073] Optionally, when the target function is an encryption function, the capturing unit 304 is used to: When the target gateway uses the encryption function to encrypt the response plaintext data, the exit probe of the encryption function is triggered to obtain the response plaintext data; The plaintext response data is copied to a specified area to obtain the target capture data.

[0074] like Figure 4 As shown in the figure, this application embodiment provides a ledger data generation device, including a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. Memory 403 is used to store computer programs; In one embodiment of this application, when the processor 401 executes the program stored in the memory 403, it implements the method for generating ledger data provided in any of the foregoing method embodiments, including: By attaching the probes of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; Perform statistical analysis on the target capture data to determine target indicators related to the target capture data; Based on the target gateway data of the target gateway, query the target context in the preset gateway service; Based on the target captured data, the target metrics, and the target context, the ledger data of the target gateway is generated.

[0075] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the ledger data generation method provided in any of the foregoing method embodiments.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0079] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating ledger data, characterized in that, The method includes: By attaching the probes of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; Perform statistical analysis on the target capture data to determine target indicators related to the target capture data; Based on the target gateway data of the target gateway, query the target context in the preset gateway service; Based on the target captured data, the target metrics, and the target context, the ledger data of the target gateway is generated.

2. The method according to claim 1, characterized in that, The step of generating the ledger data of the target gateway based on the target captured data, the target metrics, and the target context includes: The request plaintext data and response plaintext data in the target captured data are matched to obtain composite transaction data; The composite transaction data is sampled using an intelligent sampling strategy to obtain sampled composite transaction data. Based on the sampled composite transaction data, the target metric, and the target context, the ledger data of the target gateway is generated.

3. The method according to claim 2, characterized in that, The step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Obtain the target status code corresponding to the response plaintext data in the composite transaction data; When the target status code in the response plaintext data is an abnormal status code, the composite transaction data is determined to be the sampled composite transaction data.

4. The method according to claim 2, characterized in that, The step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Obtain the API endpoint corresponding to the composite transaction data; When the API endpoint is the first API endpoint captured, the composite transaction data is determined to be the sampled composite transaction data.

5. The method according to claim 2, characterized in that, The step of using an intelligent sampling strategy to sample the composite transaction data to obtain sampled composite transaction data includes: Statistics on the API endpoints corresponding to all composite transaction data; Based on the API endpoint corresponding to each composite transaction data, determine the target API endpoint whose call frequency is greater than a preset value and has no anomalies. Using a preset first sampling frequency, the composite transaction data corresponding to the API endpoint is sampled to obtain the sampled composite transaction data.

6. The method according to claim 1, characterized in that, When the target function is a decryption function, the process involves attaching the eBPF program's probe to the entry and / or exit of the target function in the target gateway, and capturing data at the entry and / or exit of the target function to obtain target captured data, including: When the target gateway uses the decryption function to decrypt the received encrypted request, the exit probe of the decryption function is triggered to obtain the plaintext data of the request; The plaintext request data is copied to the specified area to obtain the target capture data.

7. The method according to claim 1, characterized in that, When the target function is an encryption function, the step of attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, and capturing data at the entry and / or exit of the target function to obtain target captured data includes: When the target gateway uses the encryption function to encrypt the response plaintext data, the exit probe of the encryption function is triggered to obtain the response plaintext data; The plaintext response data is copied to a specified area to obtain the target capture data.

8. A device for generating ledger data, characterized in that, The device includes: The capture unit is used to capture data at the entry and / or exit of the target function by attaching the probe of the eBPF program to the entry and / or exit of the target function in the target gateway, thereby obtaining target capture data. An analysis unit is used to perform statistical analysis on the target capture data and determine target indicators related to the target capture data. The query unit is used to query the target context in a preset gateway service based on the target gateway data of the target gateway; The generation unit is used to generate ledger data for the target gateway based on the target captured data, the target indicators, and the target context.

9. A device for generating ledger data, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: By attaching the probes of the eBPF program to the entry and / or exit of the target function in the target gateway, data is captured at the entry and / or exit of the target function to obtain target captured data; Perform statistical analysis on the target capture data to determine target indicators related to the target capture data; Based on the target gateway data of the target gateway, query the target context in the preset gateway service; Based on the target captured data, the target metrics, and the target context, the ledger data of the target gateway is generated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for generating ledger data according to any one of claims 1 to 7.

Images