Span-based call chain log sampling method and device and medium

Abstract

The invention discloses a Span-based call chain log sampling method and device and a medium, and the method comprises the steps: obtaining log call chain data, carrying out the data mining of the obtained log call chain data, obtaining the call topological relation between the Spans of each Trace, carrying out the preliminary grouping of the Spans in each Trace, and carrying out the initial grouping of the Spans in each Trace; extracting the SQL of the Span in each preliminary group, constructing a statement sequence according to an execution time sequence, executing sliding grouping according to a preset time window rule, performing semantic recognition on the SQL in each sliding group, splitting and / or combining the sliding groups under the same preliminary group to obtain refined groups, and obtaining refined groups; according to an information maximization principle, a corresponding log sampling strategy is formulated and executed for each refined group, so that sampling resources are focused on a Span set with consistent service logic, and the service correlation and accuracy of sampling data and the universality and practicability of a sampling scheme are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer and log processing technology, and in particular to a Span-based method, device and medium for sampling call chain logs. Background Technology

[0002] In modern distributed systems, a user request typically requires the collaborative processing of multiple microservices, resulting in a large number of call trace logs containing multiple spans. These spans record the execution details of the request in each service. As the system scales up, especially in data-intensive application scenarios, a single trace may contain tens of thousands of spans, causing the storage and processing costs of the logs to rise sharply.

[0003] Current mainstream call chain log sampling techniques are mainly divided into two categories: header sampling and tail sampling. Header sampling determines whether to sample the entire trace at the beginning of the request. Typical implementations include fixed-ratio sampling, adaptive sampling that dynamically adjusts the sampling rate according to system load, and rule-based sampling based on attributes such as service name and user ID. Tail sampling, on the other hand, determines whether to retain the trace after it has been fully generated, based on the characteristics of the entire trace. Typical implementations cover sampling methods based on latency, error rate, and specific business rules.

[0004] However, existing sampling techniques make sampling decisions based on the entire Trace, failing to distinguish the importance of different spans within a Trace. This results in coarse information granularity and reduces the accuracy of the sampled data. The retained Trace often contains a large number of redundant or irrelevant spans, leading to low storage efficiency. Furthermore, there is a phenomenon where an entire Trace is discarded due to a few abnormal spans, resulting in the loss of valid information carried by other spans. This causes deviations in the statistical values ​​of the remaining spans, leading to inaccurate anomaly detection and reducing the versatility and practicality of the sampling scheme. Summary of the Invention

[0005] This invention provides a Span-based call chain log sampling method, device, and medium to solve the problems of low accuracy, low storage efficiency, low versatility, and low practicality of log sampling.

[0006] According to one aspect of the present invention, a Span-based call chain log sampling method is provided, comprising:

[0007] The system acquires log call chain data and performs data mining on it to obtain the call topology relationships between spans of each trace. Based on the call topology relationships, the span data within each trace is grouped to obtain at least one preliminary group corresponding to each trace. The SQL statements of each preliminary group are extracted, and statement sequences corresponding to each preliminary group are constructed according to the execution sequence of the SQL statements corresponding to the preliminary groups within each trace. Each statement sequence is then slid-grouped according to a preset time window rule to obtain at least one sliding group corresponding to each statement sequence. Semantic recognition is performed on the SQL statements in each sliding group, and based on the semantic recognition results, sliding groups under the same preliminary group are split and / or merged to obtain refined groups corresponding to each trace. Based on the principle of maximizing information, the log sampling strategy corresponding to each refined group is determined, and log sampling is performed by executing the log sampling strategy.

[0008] According to another aspect of the present invention, a Span-based call chain log sampling device is provided, comprising:

[0009] The data mining module acquires log call chain data and performs data mining on it to obtain the call topology relationship between spans of each trace in the log call chain data. The sequence construction module groups the span data within each trace according to the call topology relationship to obtain at least one preliminary group corresponding to each trace, extracts the SQL of each span in each preliminary group, and constructs the statement sequence corresponding to each preliminary group according to the execution time sequence of the SQL of the corresponding preliminary group within each trace. The sliding grouping module slides each statement sequence according to a preset time window rule to obtain at least one sliding group corresponding to each statement sequence. The fine grouping module performs semantic recognition on the SQL in each sliding group and splits and / or merges the sliding groups under the same preliminary group according to the semantic recognition results to obtain fine groupings corresponding to each trace. The log sampling module determines the log sampling strategy corresponding to each fine group according to the principle of maximizing information and performs log sampling by executing the log sampling strategy.

[0010] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0011] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the Span-based call chain log sampling method according to any embodiment of the present invention.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the Span-based call chain log sampling method according to any embodiment of the present invention.

[0013] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0014] The technical solution of this invention involves data mining of the acquired log call chain data to obtain the call topology relationship between the spans of each call chain trace. Based on this, the spans within each trace are initially grouped. The SQL of each span in the initial group is extracted and a statement sequence is constructed according to the execution time order. Then, sliding grouping is executed according to a preset time window rule. Furthermore, semantic recognition is performed on the SQL in each sliding group, and sliding groups under the same initial group are split and / or merged to obtain refined groups. Based on the principle of maximizing information, a corresponding log sampling strategy is formulated and executed for each refined group. Through a hierarchical and progressive grouping mechanism driven by business semantics, sampling resources are focused on a set of spans with consistent business logic characteristics, improving the business relevance and accuracy of the sampled data, avoiding invalid sampling and redundant data storage, reducing the cost of the entire log processing chain, improving storage efficiency, and preserving the complete logical context of business operations. This provides structured and interpretable data for subsequent call chain analysis and fault diagnosis. At the same time, with dynamically generated refined grouping and adaptive sampling strategies, it flexibly adapts to the needs of diverse business scenarios, improving the versatility and practicality of the sampling scheme.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a call chain log sampling method based on Span according to Embodiment 1 of the present invention;

[0018] Figure 2 This is a flowchart of another Span-based call chain log sampling method provided in Embodiment 2 of the present invention;

[0019] Figure 3 This is a flowchart of another Span-based call chain log sampling method provided in Embodiment 3 of the present invention;

[0020] Figure 4 This is a schematic diagram of a Span-based call chain log sampling device according to Embodiment 4 of the present invention;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the Span-based call chain log sampling method according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example 1

[0025] Figure 1 This is a flowchart of a Span-based call chain log sampling method provided in Embodiment 1 of the present invention. This embodiment is applicable to log sampling situations. This method can be executed by a Span-based call chain log sampling device, which can be implemented in hardware and / or software and is generally configured in an electronic device. Figure 1 As shown, the method includes:

[0026] S110. Obtain log call chain data and perform data mining on the log call chain data to obtain the call topology relationship between the spans of each Trace in the log call chain data.

[0027] In this embodiment of the invention, the log call chain data can be specifically understood as: a data set recording the end-to-end flow of user requests in a distributed system, composed of multiple traces, each trace containing several spans, including execution information of the request in each service or component (such as start and end times, relationships, and operation attributes). Data mining can be specifically understood as: during offline analysis, extracting the correlation features between spans by sorting out the temporal execution patterns of spans in all traces and mining frequently co-occurring span combinations.

[0028] A trace can be understood as the complete path of a request within an application, composed of multiple spans arranged hierarchically or in parallel, with all spans sharing the same trace identifier. A span can be understood as the basic unit of work that constitutes a trace, representing a single operation or task, containing a unique span identifier, its parent trace identifier, a parent span identifier, and start and end timestamps, among other core information. Spans can be nested or exist in parallel. The root span serves as the starting point of the trace, and its parent span identifier is usually null or a preset null value. All spans other than the root span (including nested and parallel spans) must be associated with their corresponding parent span identifiers to form an end-to-end request flow view.

[0029] The topology relationship can be understood as follows: based on the execution order and high-frequency co-occurrence characteristics of Spans, the business relationships and hierarchical structure between Spans are sorted out, thereby determining which Spans belong to the same business process and have a fixed execution order or collaborative relationship.

[0030] Specifically, after obtaining the log call chain data that records the request flow in the distributed system, offline data mining methods are used to analyze this data. First, each Span and its attributes (including trace identifier, span identifier, parent Span identifier, and start and end timestamps) in all traces are extracted.

[0031] Sequence pattern mining algorithms (such as frequent sequence mining) can be used to analyze the temporal execution of Spans in all traces. Support thresholds (e.g., appearing in over 80% of related traces) can be set to filter out frequently co-occurring Span temporal combinations. Simultaneously, Spans can be grouped according to business logic. This can be done by calculating the similarity of Span operation names, analyzing the correlation between Span parameter characteristics and execution results, and combining this with clustering algorithms (such as hierarchical clustering based on similarity) to identify Spans that consistently appear together (e.g., the percentage of co-occurrences exceeds a preset threshold) and belong to the same business process, and grouping them together. Spans within the same business logic group are functionally related and execute collaboratively.

[0032] Finally, by integrating the temporal execution relationship, high-frequency co-occurrence characteristics, and business grouping results of the Spans, and using graph structure modeling (with the Spans as nodes and the temporal relationship and co-occurrence association as edges), the call topology relationship between the Spans within each Trace is constructed to represent the associated logic, execution structure, and grouping affiliation of the Spans in the business process.

[0033] Optionally, based on the above embodiments, data mining can be performed on the log call chain data to obtain the call topology relationship between the spans of each trace in the log call chain data, which may include:

[0034] Static analysis is performed on the application source code in the log call chain data to obtain the call topology relationship between the spans of each trace in the log call chain data; and / or data mining is performed on the trace data collected in the simulation stress test environment or the actual online environment in the log call chain data using machine learning algorithms to obtain the call topology relationship between the spans of each trace in the log call chain data.

[0035] In this embodiment of the invention, static analysis can be specifically understood as: an analysis method that extracts potential execution flows and component associations by parsing static information such as the syntax structure, function call relationships, and class dependencies of the code without running the application source code. The simulation load testing environment can be specifically understood as: a test environment that simulates the load and business scenarios of the online system, used to collect trace data close to real business data, avoiding direct reliance on online data for analysis. The actual online environment can be specifically understood as: the production environment where the application system is running normally, where the collected trace data reflects the execution status of real business requests.

[0036] Specifically, data mining of log call chain data to obtain the call topology relationships between spans of each trace can be achieved in two independent or combined ways. One is to use static code analysis tools to perform static analysis on the application source code corresponding to the log call chain data, analyze the function call relationships, component dependency structures, and potential execution flows in the code, and directly deduce the hierarchical relationships and business logic relationships between spans to form the call topology relationships.

[0037] In a specific example, static code analysis tools are used to perform static analysis on the application source code corresponding to the log call chain data. This generates an abstract syntax tree to extract the call hierarchy of functions or methods, as well as the dependencies between classes and interfaces. Simultaneously, the dependency structure of components such as remote procedure calls and database access between microservices is analyzed. Then, combined with the Span event tracking rules of the system's distributed tracing framework (such as automatically generating Spans when a specific function is executed and custom annotations marking Spans), the function call hierarchy in the source code is mapped to the parent-child hierarchy of Spans (e.g., the main function generates the root Span, and the called child function generates the child Span), and the interaction flow between components is mapped to the upstream and downstream execution relationships of Spans. Based on the business comments and functional module divisions in the code (e.g., order module, inventory module, and payment module), the business functions corresponding to different functions or components are analyzed. The code logic implementing the same business process (e.g., order creation or logistics query) is associated with the corresponding Span set, determining the business logic collaboration relationships between Spans. Through graph structure modeling (using Spans as nodes and hierarchical associations and business logic relationships as edges), all mapping and analysis results are integrated to directly deduce the call topology relationships between the Spans of each trace.

[0038] Secondly, trace data is collected from simulated stress testing environments or actual online environments. Offline data mining is then performed using machine learning algorithms. The Apriori frequent itemset mining algorithm can be used to count the sets of frequently co-occurring Spans (for example, in a certain type of business trace, s3, s4, and s5 always appear together, which is determined to be a strongly related call Span). Combined with the Louvain graph clustering algorithm, closely related Spans are clustered into business groups. At the same time, the temporal execution and hierarchical dependency relationships of Spans are analyzed, and finally, the call topology relationship between Spans of each trace is constructed.

[0039] In a specific example, a large amount of trace data from a certain business scenario is collected. The Span sequences in each trace are extracted (arranged in execution order, e.g., t1:[s1,s3,s4,s5,s7], t2:[s1,s2,s3,s4,s5,s9,s10], and t3:[s1,s2,s4,s3,s4,s5,s2,s4,s10]). All Span sequences are used as the input dataset for the Apriori algorithm. Then, a support threshold is set (e.g., defining strongly related Span calls). To ensure co-occurrence in over 80% of the target business traces, the dataset is iteratively scanned using the Apriori algorithm: identifying individual Spans (1-itemsets) appearing in a large number of traces, generating all possible combinations of two Spans (candidate 2-itemsets) based on the 1-itemsets, filtering out high-frequency 2-itemsets that meet the support threshold, and then gradually generating 3-itemsets and 4-itemsets, etc., until no new high-frequency itemsets can be generated. Finally, high-frequency co-occurring Span sets such as s3, s4, and s5 are identified and determined to be strongly associated call Spans.

[0040] Using each Span as a node, the co-occurrence frequency (or normalized association strength, with higher co-occurrence frequency resulting in higher weight) among Spans in the high-frequency co-occurrence Span set is used as the edge weight to construct a Span association graph (e.g., the edge weights between s3 and s4, s3 and s5, and s4 and s5 are significantly higher than their association weights with other Spans). This association graph is then input into the Louvain graph clustering algorithm. The algorithm iteratively optimizes the modularity index (higher modularity indicates tighter associations within a community and sparser associations between communities), continuously adjusting the community division of the Spans. Iteration stops when the modularity reaches its maximum value; at this point, each community represents a business group. Ultimately, this achieves the goal of clustering closely associated Spans (such as s3, s4, and s5) into the same business group. Understandably, offline data cannot cover all business grouping scenarios. In the online Trace data processing stage, new Span combination patterns not covered by offline data can be captured in real time and added to the existing business grouping scheme for continuous iterative optimization.

[0041] By applying static source code analysis to perform data mining on log call chain data to obtain the call topology relationships between spans of each trace, and / or by providing machine learning algorithms based on simulated load testing or actual online environment trace data to perform data mining on log call chain data to obtain the call topology relationships between spans of each trace, static analysis can parse pre-defined logic such as function calls and component dependencies at the system design level, deriving the span hierarchy and business relationships based on design specifications. Machine learning algorithms can capture dynamic features such as the actual span execution sequence and high-frequency co-occurrence patterns during system operation. The two methods, used independently or in combination, cover the topology relationships expected in the design and supplement the actual relationships in operation, avoiding the limitations of a single analysis method. This solution addresses missing topologies by constructing a more comprehensive and complete call topology. Simultaneously, the deterministic Span hierarchy derived from static analysis serves as a baseline framework for topological relationships. Machine learning algorithms can validate and correct theoretical biases in static analysis using extensive real-world trace data, capturing Span combinations with strong business logic relationships. The two mutually reinforce and complement each other, reducing the probability of misjudgments and improving the accuracy and reliability of topological relationships. Furthermore, this solution is adaptable to diverse system scenarios. Newly launched systems can quickly build basic topologies through static analysis, mature systems can use algorithms to discover topologies that fit actual business traffic and dynamically adapt to business iterations, and complex distributed systems can combine both approaches to balance design logic and operational realities, enhancing the flexibility and applicability of topology construction.

[0042] S120. Based on the call topology, group the Span data in each Trace to obtain at least one preliminary group corresponding to each Trace, and extract the structured query language of the Span in each preliminary group. According to the execution sequence of the SQL of the corresponding preliminary group in each Trace, construct the statement sequence corresponding to each preliminary group.

[0043] In this embodiment of the invention, the preliminary grouping can be understood as: a set formed by preliminary clustering of Spans within a single Trace based on the call topology (the clearly defined hierarchical dependencies, business associations, and execution sequence relationships between Spans). Each group corresponds to a business association scenario (such as core business association groups and independent association groups), which is the basis for subsequent refined grouping. The Structured Query Language (SQL) can be understood as: structured statements recorded in the leaf node Spans, used for data querying or manipulation, reflecting the specific data processing behavior corresponding to the Span. Specifically, the leaf node Span can be understood as: the lowest-level Span without lower-level child Spans in the call topology, usually directly corresponding to specific data operations (such as SQL execution), and is the core object for extracting SQL. The execution sequence can be understood as: the actual execution order of SQL in the business process corresponding to the preliminary group, which is the basis for constructing the statement sequence. The statement sequence can be understood as: an ordered set formed by arranging the SQL within a single preliminary group according to the execution sequence, preserving the data operation logic of the business process.

[0044] Specifically, based on the established call topology, the Span data within each Trace is grouped according to the strong correlation characteristics of the Spans. Strongly correlated Spans are clustered into core business correlation groups, while independently existing Spans or those with low correlation are grouped into independent correlation groups. For example, for t1:[s1,s3,s4,s5,s7], the strongly correlated {s3,s4,s5} are clustered into the core business correlation group, while the less correlated {s1} and {s7} are grouped into independent correlation groups (or merged into {s1,s7} based on correlation strength), resulting in at least one preliminary grouping for each Trace (each preliminary grouping contains at least one Span). The SQL corresponding to the leaf node Span in each preliminary group is extracted, and the SQL is arranged in order according to the execution sequence of the SQL in the corresponding Trace for that preliminary group, constructing an independent statement sequence for each preliminary group, thus fully preserving the data operation logic of the preliminary group.

[0045] S130. Perform sliding grouping on each statement sequence according to the preset time window rules to obtain at least one sliding group corresponding to each statement sequence.

[0046] In this embodiment of the invention, the preset time window rule can be specifically understood as: the basis for dividing the sliding group can be divided into two categories: division by duration and division by the number of spans, which needs to be selected according to the business scenario. When the SQL execution intervals vary greatly and it is necessary to focus on business operations and business processes within a specific time segment that are strongly correlated with time (such as scheduled batch tasks), it is possible to choose to divide by duration, such as splitting the window into 500-millisecond intervals to capture intensive SQL operations within a certain time period. When the SQL execution frequency is uniform and it is necessary to control the processing scale of the sliding group, it is possible to choose to divide by the number of spans.

[0047] Sliding grouping can be understood as: a subset formed by sliding the splitting of a statement sequence through a preset time window. It is the smallest processing unit for semantic recognition, and each statement sequence corresponds to at least one sliding grouping.

[0048] S140. Perform semantic recognition on the SQL in each sliding group, and based on the semantic recognition results, split and / or merge the sliding groups under the same initial group to obtain refined groups corresponding to each Trace.

[0049] In this embodiment of the invention, semantic recognition can be specifically understood as: the process of parsing the operation type (query, insert, and update, etc.), business meaning (such as order query or inventory deduction), and execution characteristics (such as batch or single execution) of SQL in sliding groups, with the aim of extracting the business attributes and access patterns of the SQL. Refined grouping can be specifically understood as: the final grouping formed by splitting or merging sliding groups based on the semantic recognition results under the same initial grouping, requiring that the business stage and access pattern of the spans in each group be consistent, serving as the basis for subsequent sampling strategy formulation. Business stage can be specifically understood as: the business function positioning corresponding to the SQL (such as initialization, data collection, and calculation, etc.). Access pattern can be specifically understood as: the execution characteristic pattern of the SQL (such as single query, batch update, and loop execution, etc.).

[0050] Specifically, for each initial group of statement sequences, sliding grouping is performed according to a preset time window rule. By continuously sliding the window across the statement sequence (with adjustable overlap ratios to avoid losing context during splitting), the ordered SQL sequence is divided into multiple sliding groups (which can be overlapping or non-overlapping), resulting in at least one sliding group for each statement sequence. Subsequently, semantic recognition is performed on the SQL statements in each sliding group, extracting the operation type, syntax structure (such as batch keywords, calculation syntax (aggregate functions), parameter sequences, and repetition features) and business metadata (related table names, field names, and business-related keywords) for each SQL statement in each sliding group. Through preset business stage mapping rules, combined with the SQL execution context (such as the business stage of the refined group to which it belongs, the operation markers of the associated spans, and the execution sequence position, etc.), the business stage classification (such as initialization, data collection, and calculation) is completed.

[0051] Next, batch updates are determined by syntactic structure (e.g., containing multiple update statements, batch keywords, or calls to batch processing stored procedures). Single queries are determined by the independence and non-repetition of individual SQL statements (e.g., a single query statement, no batch keywords, and executed only once). Loop execution is determined by the template consistency, parameter differences, and execution time density of SQL statements within the sliding group (e.g., the same query statement appears a preset number of times within a preset time). Finally, the identification results of all SQL statements within the sliding group are aggregated and verified. If most SQL statements within the same sliding group belong to the same business stage and have the same access pattern, the business stage and access pattern of the sliding group are determined. If multiple business stages or access patterns exist, they are marked as mixed types, providing an accurate basis for the subsequent splitting and merging of sliding groups under the same initial group. After parsing the corresponding business stages (such as initialization, data collection and calculation) and access patterns (such as single query, batch update or loop execution), the sliding groups under the same initial group are split and / or merged according to the semantic recognition results. If the business stages and access patterns of multiple sliding groups are completely consistent, they are merged into a refined group. If a single sliding group contains multiple business stages or access patterns, it is split into multiple refined groups. Finally, at least one refined group corresponding to each Trace is obtained, and each refined group is a set of Spans that uniquely matches the business stage and access pattern.

[0052] S150. Based on the principle of maximizing information, determine the log sampling strategy corresponding to each refined group, and perform log sampling by executing the log sampling strategy.

[0053] In this embodiment of the invention, the principle of maximizing information can be specifically understood as follows: during the log sampling process, with the goal of retaining the most valuable information with the minimum amount of data, priority is given to retaining log data that is of key significance for fault diagnosis, performance monitoring and business analysis, while filtering redundant and low-value data, and balancing storage costs and data availability.

[0054] Specifically, based on the business stage (such as initialization, data collection, and core computation) and access mode (such as single query, batch update, and loop execution) of each refined group, the importance of the business scenario corresponding to the group can also be included (such as order payment related groups having higher priority than ordinary query groups). The value weight of log data within the group is evaluated. For example, the value weight of groups corresponding to core computation and critical access modes (such as batch updates, which are prone to performance issues) is higher than that of groups corresponding to ordinary stages (such as non-critical initialization), normal states, and repetitive and redundant access modes (such as loop queries without anomalies). Differentiated log sampling strategies are formulated based on value weights. For example, for high-value groups, a high-proportion random sampling method is adopted, retaining all anomalies (e.g., retaining 100% of the anomaly spans and 90% of the normal spans of the core computation group). For access pattern groups prone to problems, such as batch updates, key spans at the beginning and end are additionally retained to restore the execution context. For medium-value groups, a medium-proportion sampling method is adopted (e.g., retaining 50% of the single query spans during the data collection phase). For low-value redundant groups, a low-proportion sampling method or targeted filtering is adopted (e.g., retaining only 10% of the non-anomaly loop execution spans, or directly discarding ordinary initialization spans that exceed the threshold for repetition). Finally, the log collection system executes the formulated sampling strategy, filtering the span logs one by one according to the fine-grained grouping, retaining key data that meets the strategy, and filtering redundant low-value data to complete the log sampling. While controlling the log storage and processing costs, it ensures that sufficient valuable log information can be obtained for subsequent fault diagnosis, performance monitoring, and other scenarios, achieving a balance between minimizing data volume and maximizing information value.

[0055] The technical solution of this invention involves data mining of the acquired log call chain data to obtain the call topology relationship between the spans of each call chain trace. Based on this, the spans within each trace are initially grouped. The SQL of each span in the initial group is extracted and a statement sequence is constructed according to the execution time order. Then, sliding grouping is executed according to a preset time window rule. Furthermore, semantic recognition is performed on the SQL in each sliding group, and sliding groups under the same initial group are split and / or merged to obtain refined groups. Based on the principle of maximizing information, a corresponding log sampling strategy is formulated and executed for each refined group. Through a hierarchical and progressive grouping mechanism driven by business semantics, sampling resources are focused on a set of spans with consistent business logic characteristics, improving the business relevance and accuracy of the sampled data, avoiding invalid sampling and redundant data storage, reducing the cost of the entire log processing chain, improving storage efficiency, and preserving the complete logical context of business operations. This provides structured and interpretable data for subsequent call chain analysis and fault diagnosis. At the same time, with dynamically generated refined grouping and adaptive sampling strategies, it flexibly adapts to the needs of diverse business scenarios, improving the versatility and practicality of the sampling scheme.

[0056] Example 2

[0057] Figure 2 This is a flowchart of another Span-based call chain log sampling method provided in Embodiment 2 of the present invention. This embodiment is a refinement of the above embodiment's "semantic recognition of SQL in each sliding group, and splitting and / or merging sliding groups under the same initial group according to the semantic recognition results to obtain refined groups corresponding to each Trace." Figure 2 As shown, the method includes:

[0058] S210. Obtain log call chain data and perform data mining on the log call chain data to obtain the call topology relationship between the spans of each Trace in the log call chain data.

[0059] S220. Based on the call topology, group the Span data in each Trace to obtain at least one preliminary group corresponding to each Trace, and extract the SQL of the Span in each preliminary group. According to the execution time sequence of the SQL of the corresponding preliminary group in each Trace, construct the statement sequence corresponding to each preliminary group respectively.

[0060] S230. Perform sliding grouping on each statement sequence according to the preset time window rules to obtain at least one sliding group corresponding to each statement sequence.

[0061] S240. Calculate the operation feature value of SQL in each sliding group, and combine the position of each sliding group in the corresponding statement sequence to classify each sliding group into business stages, and obtain the result of business stage grouping as the semantic recognition result.

[0062] In this embodiment of the invention, the operation feature value can be specifically understood as: quantitative features extracted based on the SQL syntax structure, operation type, and business keywords (such as whether it contains aggregate functions, whether it is a batch operation, and the matching degree of business keywords, etc.), used for business stage classification. Specifically, business keywords can be understood as: words strongly related to specific business processes, functional modules, and data operation goals (specifically including: core word roots in table names and field names, and business scenario terms, etc.). They are semantic identifiers for identifying SQL business attributes and distinguishing business stages, and can directly map the functional positioning of SQL in the business process. For example, config corresponds to the initialization stage, order_raw corresponds to the data collection stage, and sum corresponds to the core calculation stage.

[0063] Specifically, the operational feature values ​​of the SQL statements in each sliding group are calculated. For example, whether an SQL statement contains aggregate functions or batch keywords is converted into binary feature values ​​(e.g., 1 for containing aggregate functions and 0 for not containing batch keywords). The business keyword matching degree is converted into a continuous feature value between 0 and 1 using a keyword matching algorithm. Then, the average (or weighted average) of the feature values ​​of all SQL statements within the group is taken to obtain the overall operational feature value of the sliding group. Next, the temporal position of each sliding group in the corresponding statement sequence is determined (e.g., the front of the sequence tends to be the initialization stage, the middle tends to be the data collection stage, and the back end tends to be the calculation stage). Based on preset classification rules (e.g., if the proportion of aggregate functions in the operational feature values ​​is high and it is located at the back of the sequence, it is determined to be the core calculation stage; if the business keyword matches the collection category and it is located in the middle of the sequence, it is determined to be the data collection stage; if the business keyword matches the configuration category and it is located at the front of the sequence, it is determined to be the initialization stage), the sliding group is classified into the corresponding business stage. Sliding groups in the same business stage form a business stage group.

[0064] Optionally, based on the above embodiments, the operation feature value of the SQL in each sliding group is calculated, and combined with the position of each sliding group in the corresponding statement sequence, the business stage classification is performed for each sliding group, and the result of business stage grouping is used as the semantic recognition result, which may include:

[0065] Calculate the operation feature value of the SQL in each sliding group; the operation feature value includes: read operation ratio, write operation ratio, cache operation ratio, aggregation operation complexity, multi-table join operation ratio, and parameter configuration table read feature value; based on the operation feature value and the position feature of the sliding group in the corresponding statement sequence, mark the operation semantic feature of each sliding group; the operation semantic feature includes: initialization feature, data acquisition feature, data preprocessing feature, core calculation feature, result persistence feature, and notification feature; classify each sliding group into a business stage group that matches the marked operation semantic feature, and use the result of the business stage group as the semantic recognition result.

[0066] In this embodiment of the invention, the read operation ratio can be specifically understood as: the proportion of SQL statements within a sliding group that are read operations (such as SELECT (query)) to the total number of SQL statements in that group, reflecting the degree to which the group is primarily for data reading. The write operation ratio can be specifically understood as: the proportion of SQL statements within a sliding group that are write operations (such as INSERT (insert), UPDATE (update), and DELETE (delete)) to the total number of SQL statements in that group, reflecting the degree to which the group is primarily for data writing. The cache operation ratio can be specifically understood as: the proportion of SQL statements or related operations involving cache operations (such as GET (read key-value) and SET (set key-value) commands in Redis (remote dictionary server) to the total number of operations in that group, reflecting the frequency of cache usage. The aggregation operation complexity can be specifically understood as: a quantified indicator (which can be divided into low, medium, and high levels) of the frequency and complexity of aggregate functions (such as SUM (summation), COUNT (counting), and GROUPBY (grouping) statements) in SQL, reflecting the complexity of data statistical calculations. The percentage of multi-table join operations can be specifically understood as the proportion of SQL statements involving multi-table JOIN operations within a sliding group to the total number of SQL statements in that group, reflecting the frequency of cross-table data processing.

[0067] The parameter configuration table read feature values ​​can be understood as follows: feature values ​​quantified through semantic matching (searching for table names, column names, and externally called table names and message queue names containing keywords such as config, param, setting, and env, etc.) (which can be divided into three levels: low, medium, and high), reflecting the degree to which group reads configuration data. The positional features in the statement sequence can be understood as the proportion of the sliding group's execution time within the corresponding Trace, i.e., the proportion of the group's execution time to the total execution time of the entire Trace, used to locate the group's temporal position in the business process. Operational semantic features can be understood as feature labels characterizing the sliding group's business function based on operation feature values ​​and positional features, including initialization features, data acquisition features, data preprocessing features, core calculation features, result persistence features, and notification features.

[0068] Specifically, the SQL statements of the leaf nodes (Span) of the Trace are extracted to form a sequence. After sliding grouping by time window, the operation feature values ​​of each sliding group are calculated: the proportion of read, write, and cache operations is obtained by statistically analyzing the ratio of the number of corresponding types of operations to the total number of operations in the group; the complexity of aggregation operations is quantified into corresponding levels based on the number of times the aggregation function is used, according to a preset grading threshold; the proportion of multi-table join operations is calculated by statistically analyzing the ratio of the number of multi-table JOIN SQL operations to the total number of SQL operations in the group; the parameter configuration table read feature values ​​are obtained by performing semantic matching (searching for keywords such as config, param, setting, and env) on the table names, column names, externally called table names, and message queue names of the SQL, and quantifying into corresponding levels based on the number of matched keywords according to a preset grading threshold. The positional feature of each sliding group is determined, that is, the proportion of its appearance in the corresponding Trace time period. The operation feature values ​​and positional features are matched according to preset rules to mark the operation semantic features of the sliding group.

[0069] Features meeting the following criteria are marked as initialization features: "Being in the first 10% of the trace period, high parameter configuration table read characteristic value, read operation percentage greater than the preset read operation percentage threshold, and multi-table join operation percentage less than the preset multi-table join operation percentage threshold." Typically, features meeting the following criteria are marked as initialization features: "Being in the first 10% of the trace period, high parameter configuration table read characteristic value, read operation percentage > 0.8, and multi-table join operation percentage < 0.3." For example... (Indicates querying the system configuration table) (Indicates querying the application configuration table) and (This indicates that the data was read using the Redis GET command with the specified string "") (Cache key-value pairs prefixed with "").

[0070] Data collection features are defined as those that meet the following criteria: the percentage of operations occurring before the Trace period reaches a preset threshold; the percentage of read operations exceeds a preset threshold; the percentage of multi-table join operations exceeds a preset threshold; and the percentage of cached operations is less than a preset threshold. Typically, data collection features are defined as those that meet the following criteria: the percentage of operations occurring within the first 30% of the Trace period is greater than 0.9; the percentage of multi-table join operations exceeds 0.5; and the percentage of cached operations is less than 0.3. For example... (Indicates querying the device status table) and (This indicates a query of the original message table.)

[0071] Data preprocessing features are defined as those that meet the following criteria: "within the Trace's preset time period, read and write operations are close in percentage, cache operations are greater than the preset cache operation percentage threshold, and aggregation operation complexity is medium." Typically, data preprocessing features include those that meet the following criteria: "within the Trace's 20% to 40% time period, read and write operations are close in percentage (difference less than the preset threshold), cache operations are greater than 0.4%, and aggregation operation complexity is medium." Examples of preprocessing features include SELECT+GROUP BY (a query containing a GROUP BY clause, often used with aggregate functions like SUM or COUNT to perform basic statistical grouping on the collected raw data), temporary table INSERT / UPDATE (inserting or updating data in a temporary table), and Redis SETEX (caching intermediate results using the Redis SETEX command, which sets a key-value pair with an expiration time).

[0072] The following are marked as core computational characteristics: "Being within the pre-defined Trace time period, high aggregation operation complexity, cache operation percentage greater than the pre-defined cache operation percentage threshold, and read operation percentage higher than write operation percentage". Typically, the following are marked as core computational characteristics: "Being within the Trace 30% to 70% time period, high aggregation operation complexity, cache operation percentage > 0.5, and read operation percentage higher than write operation percentage". Examples include complex JOIN queries (representing complex join queries involving multiple tables), a large number of Redis GET / MGET (representing batch execution of Redis GET commands or MGET (batch read) commands to retrieve multiple key values), and SQL statements containing computational functions (representing SQL statements with computational functions such as SUM, AVG, or MAX).

[0073] Results meeting the following criteria are marked as persistent characteristics: "Percentage of operations occurring after a Trace period exceeding a preset threshold, write operations exceeding a preset write operation percentage threshold, multi-table join operations exceeding a preset multi-table join operation percentage threshold, and low operation error rate." Typically, results meeting the following criteria are marked as persistent characteristics: "Percentage of operations occurring after a Trace period exceeding 30%, write operations exceeding 0.6%, multi-table join operations exceeding 0.4%, and low operation error rate." For example... (Indicates to " Insert data into the result table with " as the prefix ( (For storing business results) (Indicates an update using " State table with " as prefix ( (Storing business status data) and (This indicates that the final result is cached using the Redis SET command (in the form of ""). (with " as a prefix)).

[0074] Notification features are defined as those that meet the following criteria: "occurring within a preset time period after Trace, having a moderate percentage of write operations including specific table patterns, or involving external service calls." Typically, notification features are defined as those that meet the following criteria: "occurring within 10% of the time period after Trace, having a moderate percentage of write operations, including operation notification tables or message queue lists, or involving external service calls." Examples include INSERTINTO notification (indicating inserting data into the notification table (which stores notification message data), INSERTINTO message_queue (indicating inserting a message into the message queue (where message_queue is the message queue storage table or queue name), and HTTP POST to webhook (indicating calling an external webhook interface via an HTTP (HyperText Transfer Protocol) POST request).

[0075] Sliding groups with the same semantic features are categorized to form business stage groups corresponding to the initialization feature INIT, data acquisition feature COLLECT, data preprocessing feature PREPROCESS, core computation feature COMPUTE, result persistence feature PERSIST, and notification feature NOTIFY. This grouping result, serving as a semantic recognition outcome, can accurately identify business stages involving repetitive SQL-like spans, preventing the accidental deletion of critical spans. It also adapts to scenarios with numerous repetitive SQL calls in data-intensive systems, compensating for the inability to accurately identify business logic relationships solely based on call topology (feature parameters can be fitted and adjusted using supervised machine learning algorithms during actual runtime).

[0076] By quantifying the proportion of read operations, write operations, cache operations, aggregation operation complexity, multi-table join operations, and parameter configuration table read characteristics in each sliding group, and combining the positional characteristics of the sliding group in the corresponding statement sequence, the semantic features of operations such as initialization, data acquisition, data preprocessing, core calculation, result persistence, or notification are marked for each sliding group and categorized to the corresponding business stage. This not only enables precise anchoring of the business semantics of SQL operations, solving the problem of distinguishing the value of repetitive SQL in data-intensive systems, and avoiding the inability to distinguish between repetitive and critical SQL and repetitive and redundant SQL based solely on SQL syntax or call topology, but also provides a basis for... The solution provides precise data for risk control at key business stages, focusing on high-risk aspects (such as computation time in the core computing stage and data writing errors in the result persistence stage) to reduce the probability of core process failures. Furthermore, the solution covers all stages of the business process and supports flexible adjustment of quantitative rules and judgment thresholds. It is adaptable to data-intensive system scenarios and meets the differences in business processes across industries, exhibiting strong versatility and scalability. It also provides core semantic basis for subsequent sliding group splitting or merging, ensuring that the final grouping conforms to business logic and accurately reflects the functional relationships of SQL, avoiding analytical biases caused by group confusion.

[0077] S250. From the log call chain data, obtain the execution time data and execution timestamp data of the SQL execution of the corresponding Span for each sliding group, and calculate the execution interval data of two adjacent SQL based on the execution timestamp data.

[0078] In this embodiment of the invention, execution time data can be specifically understood as: the time span from the start to the end of SQL execution, independently stored in the original full Trace data, without relying on log sampling logic. Execution timestamp data can be specifically understood as: the time record of when SQL execution begins, used to calculate the execution interval between adjacent SQL statements. Execution interval data can be specifically understood as: the timestamp difference between two adjacent SQL statements arranged in execution order within the same sliding group, reflecting the density or rhythm of SQL execution.

[0079] Specifically, the SQL execution time for each Span is calculated and stored independently. Understandably, an anomaly detection step can be included after the time calculation. The time calculation and detection are independent of the log sampling logic, employing any lightweight anomaly detection algorithm (such as a dynamic threshold method based on quantiles or a sliding window statistical method) to ensure the data accurately reflects the actual business scenario: First, based on the trace data in the original full log call chain data, the SQL execution time data for each sliding group corresponding to the Span is extracted from the independently stored time data (this can also be calculated based on the execution start and end timestamps in the SQL information). Simultaneously, the execution timestamp data of the corresponding SQL is extracted from the log call chain data. Based on the execution sequence of the SQL within each sliding group, the timestamp difference between two adjacent SQL statements is calculated to obtain the execution interval data for each adjacent statement. A lightweight anomaly detection algorithm is used to detect anomalies in execution time data or execution interval data, determining whether the time or interval of each SQL statement is normal or abnormal, and generating anomaly labels (including the degree of anomaly). Finally, the SQL statement, execution time data, execution interval data and anomaly labels are bound and stored independently. (This anomaly detection step is independent of the log sampling logic and is calculated based on the full data to ensure the accuracy of subsequent analysis.)

[0080] S260. Vectorize the SQL in each sliding group to obtain the semantic vector corresponding to each sliding group.

[0081] Specifically, for each SQL statement within a sliding group, parameterization is performed to extract a unified pattern. If the logs are from the JDBC (Java Database Connectivity) library in the APM (Application Performance Monitoring) call chain, the automatically converted PreparedStatement format (with parameter values ​​replaced by "?") is directly used as the unified pattern. If the logs are from other output methods, log pattern algorithms such as Drain (a lightweight log pattern extraction algorithm) are used to cluster similar logs, replacing variable parts with placeholders (such as "?") to extract a fixed format as the unified pattern. The extracted SQL unified pattern is then vectorized using text encoding methods such as TF-IDF (Term Frequency-Inverse Document Frequency) or Word2Vec (word vectors) to convert the text-based SQL pattern into a fixed-length numerical vector. Each dimension of the vector represents the weight or semantic feature of keywords (such as SELECT and WHERE) in the SQL pattern.

[0082] S270. For all sliding groups under the same initial grouping, calculate the semantic vector similarity between any two sliding groups.

[0083] Specifically, for all sliding groups under the same initial group, the semantic vector similarity between any two sliding groups under the same initial group is calculated using cosine similarity or Euclidean distance algorithms (the numerical range is 0 to 1, where 1 indicates that they are completely identical), and a similarity threshold (such as 0.8) is set.

[0084] S280. Based on semantic vector similarity, execution time data and execution interval data of SQL in the sliding group, identify the access pattern of each sliding group, and obtain the access pattern identification result as the semantic recognition result.

[0085] Specifically, if the semantic vector similarity of two sliding groups is greater than or equal to the similarity threshold, they are determined to be the same type of operation. Combining the results of similar operations based on semantic vector similarity, the execution time data and execution interval data of SQL in the sliding group, the access pattern of each sliding group is identified: If the execution time of SQL of the same type of operation is concentrated (e.g., the proportion of the average execution time within the preset standard deviation range is greater than or equal to the preset proportion value) and the execution interval is uniform (e.g., the coefficient of variation of the interval difference is less than or equal to the preset threshold), it is determined to be a loop execution; if the SQL in the sliding group is of the same type of operation, the execution time is long (e.g., greater than the preset multiple of the normal execution time threshold) and there is no consecutive adjacent execution interval (or the interval is greater than the execution time itself), it is determined to be a single batch operation; if the SQL in the sliding group has no repeated similar operations (no objects meet the semantic vector similarity standard), the execution time is short (less than the preset threshold) and the execution interval is irregular, it is determined to be a single query; for similar operations with abnormal execution time, the execution interval is further used to verify whether it is an abnormal access pattern. If the abnormal execution time or the interval fluctuates greatly (the execution time exceeds the range based on the average execution time of similar operations and the preset standard deviation multiple threshold, or the coefficient of variation of the execution interval is greater than the preset coefficient of variation threshold), it is determined to be an abnormal access pattern. Finally, the access pattern recognition results of each sliding group are obtained, which together with the business stage grouping results constitute the semantic recognition results.

[0086] Optionally, based on the above embodiments, the access patterns of each sliding group are identified according to semantic vector similarity, execution time data and execution interval data of SQL in the sliding group, and the identification results of the access patterns are used as semantic recognition results, which may include:

[0087] Sliding groups with semantic vector similarity greater than a preset similarity threshold are selected to form a target sliding group set. Based on the execution time and interval data of all SQL statements in each target sliding group, the execution time variance and execution interval variance of the SQL statements within the target sliding group are calculated. If the execution time variance of the target sliding group is within a preset low variance range, the access mode of the target sliding group is marked as batch operation mode; if the execution interval variance of the target sliding group is within a preset low variance range, the access mode of the target sliding group is marked as loop operation mode; if the target... If the variance of the execution time of the sliding group is within a preset low variance range, and the variance of the execution interval is also within a preset low variance range, then the execution frequency of the SQL of the target sliding group within a preset time is calculated. If the execution frequency is higher than a preset frequency threshold, it is marked as a batch operation mode; otherwise, it is marked as a loop operation mode. For sliding groups not included in the target sliding group set, their access mode is marked as a normal operation mode. Through the above operations, the access mode identification of each sliding group is completed, and the identification results of the access modes including batch operation mode, loop operation mode, and normal operation mode are obtained as semantic recognition results.

[0088] In this embodiment of the invention, the target sliding group set can be specifically understood as: a set consisting of two or more sliding groups with semantic vector similarity greater than a preset similarity threshold, used for execution feature analysis of semantically consistent operation groups. Execution time variance can be specifically understood as: a statistical indicator measuring the dispersion of execution time for all SQL statements within the target sliding group; the smaller the variance, the more stable and consistent the execution time, used to determine batch operation modes. Execution interval variance can be specifically understood as: a statistical indicator measuring the dispersion of execution intervals between adjacent SQL statements within the target sliding group; the smaller the variance, the more stable and consistent the interval, used to determine loop operation modes. The preset low variance interval can be specifically understood as: a pre-set variance threshold range; execution time or execution interval is considered stable and consistent if it falls within this interval (e.g., variance ≤ preset threshold indicates a low variance interval). Execution frequency can be specifically understood as: the number of times the SQL statements in the target sliding group are executed within a preset time period, used to distinguish access modes that simultaneously satisfy low variance in execution time and interval (high frequency indicates batch, low frequency indicates loop). The normal operation mode can be understood as the access mode corresponding to the sliding group that is not included in the target sliding group set (there are no other groups with similar semantics), which represents the normal SQL execution behavior without batch and loop characteristics.

[0089] Specifically, for each SQL statement within a sliding group, parameterization is performed to extract a unified pattern. This SQL pattern is then vectorized to obtain the semantic vector for each sliding group. After calculating the semantic vector similarity between groups, sliding groups with semantic vector similarity greater than a preset similarity threshold are selected to form the target sliding group set. For each target sliding group within the target sliding group set, the execution time and execution interval data for all SQL statements are extracted. The variance of execution time (representing time stability) and the variance of execution interval (representing interval stability) are calculated. Based on the variance results, the access mode is matched: if only the execution time variance of the target sliding group is within a preset low variance range (consistent and stable execution time), it is marked as a batch operation mode; if only the execution interval variance is within a preset low variance range (consistent and stable interval), it is marked as a loop operation mode; if both are within the preset low variance range, the SQL execution frequency within a preset time period needs to be calculated. If the frequency exceeds a preset frequency threshold, it is marked as a batch operation mode; otherwise, it is marked as a loop operation mode. Sliding groups not included in the target sliding group set are directly marked as normal operation modes. Understandably, if there are actual business needs, other specific classification modes related to transaction operations can be added. By completing the access pattern recognition of all sliding groups through the above steps, we obtain recognition results including batch operation mode, loop operation mode, and normal operation mode. This result, together with the business stage classification result, constitutes a complete semantic recognition result, providing an accurate basis for the subsequent splitting or merging of sliding groups.

[0090] By filtering semantically consistent target sliding groups using semantic vector similarity, and combining quantitative analysis of execution time variance, execution interval variance, and execution frequency, accurate identification of batch, loop, and normal access modes is achieved. This not only overcomes the limitations of relying solely on semantic similarity to determine group association, but also allows for more refined differentiation of semantically consistent sliding groups through execution behavior characteristics (time and interval stability and execution frequency). For example, groups with similar semantics but stable time are marked as batch mode, and groups with stable intervals are marked as loop mode. This solves the problem of not being able to effectively distinguish groups with similar semantics but large differences in execution behavior, ensuring that subsequent group splitting or merging is more in line with actual execution logic. Furthermore, it provides a basis for the management of different access modes. For batch operation mode, the focus is on monitoring data processing efficiency and throughput; for loop operation mode, the focus is on preventing excessive resource consumption caused by repeated execution; and for normal operation mode, the conventional strategy is used for management, avoiding resource waste or risk omissions caused by indiscriminate management.

[0091] S290. Based on the identification results of the business stage grouping and access mode of each sliding group, the sliding groups under the same initial group are split and / or merged to obtain the refined grouping corresponding to each Trace.

[0092] Specifically, based on the semantic recognition result, sliding groups under the same initial group are split or merged. If multiple sliding groups have the same business stage and access mode, they are merged into a refined group; if a single sliding group contains multiple business stages or access modes, it is split into multiple refined groups, ultimately resulting in refined groups for each trace, where the business stage and access mode of the span within the group are consistent.

[0093] S2100. Based on the principle of maximizing information, determine the log sampling strategy corresponding to each refined group, and perform log sampling by executing the log sampling strategy.

[0094] The technical solution of this invention involves data mining of the acquired log call chain data to obtain the call topology relationship between the spans of each call chain trace. Based on this, the spans within each trace are initially grouped. The SQL statements of each span in the initial group are extracted and a statement sequence is constructed according to the execution time order. Then, sliding grouping is executed according to a preset time window rule. By calculating the operation feature value of the SQL in each sliding group and combining it with the sequence position, business stage grouping is completed. The SQL execution time and timestamp data are obtained from the log call chain data, and the execution interval is calculated. After vectorizing the SQL, the semantic vector similarity between sliding groups under the same initial group is calculated. Then, the access pattern is identified by combining the semantic vector similarity, execution time and interval data. Finally, the sliding groups are split and / or merged based on the identification results of business stage grouping and access pattern. By obtaining fine-grained groupings and recognizing business semantics and operational patterns, the importance of different spans within a trace can be accurately distinguished, solving the problem of coarse information granularity. It also focuses on high-value span sets, avoiding the ineffective retention of redundant and low-value spans, thus improving log storage and processing efficiency. Simultaneously, it ensures the accuracy of anomaly detection, preventing the loss of effective information due to discarding an entire trace due to a few abnormal spans, and preventing anomalies from being masked by a large number of normal, repetitive spans. This provides reliable data support for root cause analysis of faults. Furthermore, the fine-grained groupings generated based on multi-dimensional dynamic features can fit the characteristics of different business scenarios, providing a flexible foundation for subsequent personalized processing and adapting to the business needs of diverse distributed systems. Based on the principle of maximizing information, corresponding log sampling strategies are formulated and executed for each fine-grained group, improving the universality and practicality of the sampling scheme.

[0095] Example 3

[0096] Figure 3 This is a flowchart of another Span-based call chain log sampling method provided in Embodiment 3 of the present invention. This embodiment is a refinement of the Span-based call chain log sampling method in the above embodiments. Figure 3 As shown, the method includes:

[0097] S310. Obtain log call chain data and perform data mining on the log call chain data to obtain the call topology relationship between the spans of each Trace in the log call chain data.

[0098] S320. Based on the call topology, group the Span data in each Trace to obtain at least one preliminary group corresponding to each Trace, and extract the SQL of the Span in each preliminary group. According to the execution time sequence of the SQL of the corresponding preliminary group in each Trace, construct the statement sequence corresponding to each preliminary group respectively.

[0099] S330. Perform sliding grouping on each statement sequence according to the preset time window rules to obtain at least one sliding group corresponding to each statement sequence.

[0100] S340. Perform semantic recognition on the SQL in each sliding group, and based on the semantic recognition results, split and / or merge the sliding groups under the same initial group to obtain refined groups corresponding to each Trace.

[0101] S350. Based on the principle of maximizing information, determine the log sampling strategy corresponding to each refined group, and perform log sampling by executing the log sampling strategy.

[0102] Optionally, based on the above embodiments, the semantic recognition result may include: the recognition result of the access mode; the access mode may include: batch operation mode, loop operation mode and normal operation mode;

[0103] Accordingly, based on the above embodiments, and according to the principle of maximizing information, the log sampling strategy corresponding to each refined group can be determined as follows:

[0104] Set a total sampling quota for a single Trace, and allocate a basic sampling quota to each fine-grained group based on the proportion of the number of spans in each fine-grained group to the total number of spans in the corresponding Trace. For the basic sampling quota of each fine-grained group, select spans to be retained using general rules as the initial selection result. For fine-grained groups with access mode of normal operation, directly use the initial selection result as the log sampling strategy. For fine-grained groups with access mode of batch operation or loop operation, perform a secondary selection on the fine-grained group using matching special rules to obtain a secondary selection result, and use the combination of the initial selection result and the secondary selection result as the log sampling strategy. The general rules include: calculating the time consumption anomaly score of each span in the fine-grained group, retaining anomaly spans whose time consumption anomaly score exceeds a preset dynamic threshold; if the number of anomaly spans exceeds the basic sampling quota of the fine-grained group... The Spans are sorted from highest to lowest time consumption anomalies, and the number of Spans before sorting is taken as the basic sampling quota. If the number of anomaly Spans is less than or equal to the basic sampling quota of the refined group, after retaining all anomaly Spans, the Span with the fewest sampling times in the most recent preset time window of the remaining quota is retained. The special rules corresponding to the batch operation mode include: sorting the Spans of the refined group according to the execution time sequence, sampling the first and last preset number of Spans of the sequence according to the first preset ratio, and sampling the Spans in the middle part of the sequence according to the second preset ratio. The first preset ratio is higher than the second preset ratio. The special rules corresponding to the loop operation mode include: marking the Spans corresponding to the same loop in the refined group as a whole, selecting the Spans of the entire loop at preset intervals, performing a full retention operation on the Spans of the selected loops, and performing a full discard operation on the Spans of the unselected loops.

[0105] In this embodiment of the invention, the total sampling quota can be specifically understood as: a preset maximum number of Spans allowed to be retained per Trace (e.g., 300), used to control the amount of sampled data and avoid log redundancy. The basic sampling quota can be specifically understood as: the initial sampling quota allocated according to the proportion of Spans in the refined group to the total number of Spans in the Trace, with each group retaining at least one Span (ensuring group integrity). The general rule can be specifically understood as: a basic sampling rule applicable to all refined groups, the core of which is to retain abnormal Spans and low-sampling-frequency Spans, balancing anomaly monitoring and data diversity. The time consumption anomaly score can be specifically understood as: a statistical indicator measuring the degree to which the execution time of a Span deviates from the normal range, used to quantify the degree of abnormality in Span execution time. The preset dynamic threshold can be specifically understood as: a quantile threshold calculated based on historical data (e.g., P90, meaning 90% of Spans have execution times below this value), Spans with a time consumption anomaly score exceeding this threshold are determined to be abnormal Spans. The specific rules can be understood as: supplementary sampling rules for batch or cyclic operation grouping, adapted to their execution characteristics (such as the criticality of the first and last nodes of batch operations and the correlation of rounds in cyclic operations). The first preset ratio and the second preset ratio can be understood as: the segmented sampling ratios for the first and last parts and the middle part of the batch operation grouping (the sampling ratio for the first and last parts is higher than that for the middle part), ensuring that no critical nodes are missed. Here, "first" and "second" are only used to distinguish different preset ratios and have no order of precedence.

[0106] Specifically, after completing the business stage classification and access pattern recognition (the semantic recognition results can contain batch, loop, or normal operation mode labels), a differentiated log sampling strategy is formulated for each refined group based on the principle of maximizing information: A total sampling quota for a single Trace is set, the proportion of the number of spans in each refined group to the total number of spans in the corresponding Trace is calculated, and a basic sampling quota is allocated according to this proportion, with each group's basic sampling quota being greater than or equal to 1 (ensuring that each group retains at least 1 span). Then, a general rule filtering is performed on all refined groups to obtain initial filtering results: the time consumption anomaly score (which can be a Z-score) of each span within the group is calculated, and anomaly spans whose time consumption anomaly scores exceed a preset dynamic threshold (P90) are retained; if the number of anomaly spans exceeds the group's basic sampling quota, they are sorted from high to low according to their time consumption anomaly scores, and the number of spans within the first basic sampling quota is truncated; if the number of anomaly spans is less than or equal to the basic sampling quota, after retaining all anomaly spans, the remaining quota is used to retain the span that has been sampled the least in the most recent preset time window (ensuring sampling diversity), forming the initial filtering results.

[0107] Targeted secondary filtering is performed based on the access mode of the group: For groups with a normal operation mode, the initial filtering results are directly used as the final log sampling strategy; for groups with a batch operation mode, secondary filtering is performed according to special rules: the spans within the group are sorted according to the execution time sequence, and the first and last preset number (e.g., 50) of spans in the sequence are sampled at a first preset ratio (e.g., 50%), and the spans in the middle of the sequence are sampled at a second preset ratio (e.g., 1%), with the first preset ratio being higher than the second preset ratio. The initial filtering results and the secondary filtering results are combined as the final strategy, that is, the two types of results are merged first and duplicate spans are removed to form a candidate list. If the basic sampling quota is not exceeded, all are retained. If the quota is exceeded, the exception priority > batch feature priority is used to sort and truncate (the exception spans of the initial screening are retained first, and the first and last batch spans of the secondary screening are retained) as the final strategy. For groups with a cyclic operation access mode, a secondary filtering is performed according to specific rules: Spans corresponding to the same cycle within the group are marked as a whole. Spans from the entire cycle are selected at preset intervals (all selected cycles' spans are retained, while unselected cycles' spans are discarded, i.e., interval sampling). The initial filtering results are combined with the secondary filtering results as the final strategy. This means merging the results, removing duplicates, and ensuring the integrity of selected cycle's spans. If the quota is not exceeded, all spans are retained; if the quota is exceeded, abnormal spans are retained first, followed by key selected cycle's spans in chronological order. This final strategy maximizes the retention of high-value log information while controlling the total sampling quota, and adapts to the execution characteristics of different access modes.

[0108] By allocating basic sampling quotas according to the proportion of spans in fine-grained groups and ensuring that each group retains at least one span, the problem of excessive concentration of sampling resources on high-frequency groups while ignoring niche groups is avoided. This ensures the completeness of sampling coverage across various business semantic dimensions. Furthermore, controlling the total sampling quota effectively reduces log redundancy and lowers storage and transmission costs. A general rule prioritizes retaining abnormal spans with time-consuming anomalies exceeding a preset dynamic threshold, and supplements the retention of low-frequency spans when quotas are sufficient. This ensures the core requirement of problem investigation (no abnormal data is missed) while avoiding excessive duplication of similar data through diverse sampling, achieving a basic sampling goal that balances anomaly focus and diversity. For batch processing... By grouping and adding dedicated rules for loop operation modes and combining initial screening results, the sampling strategy can accurately adapt to the execution characteristics of different access modes. The batch operation mode samples at a high ratio at the beginning and end and a low ratio in the middle to ensure that core execution nodes such as batch start and end are not missed. The loop operation mode selects samples at whole round intervals, which not only preserves the integrity of the loop execution logic, but also avoids sampling redundancy caused by repeated rounds. This solves the problem that traditional unified sampling cannot adapt to diverse execution behaviors. Through the combination of general rules and dedicated rules, the sampling data is simplified and comprehensive under the principle of maximizing information, which improves the efficiency of subsequent trace data analysis and problem localization, and improves the reliability and consistency of sampling results.

[0109] S360: While performing log sampling by executing the log sampling strategy, it supplements the time consumption information for the Spans that are discarded by sampling in each fine-grained group using a preset method.

[0110] S370. Based on the completed call chain time consumption information, assist in locating time-consuming abnormal links.

[0111] The preset methods include: when performing log sampling processing, generating a virtual Span for each refined group and recording the total time consumption of the corresponding refined group through the virtual Span; or calculating and updating the historical mean and variance of the time consumption of each refined group according to a preset period, and reporting the historical mean and variance as the time consumption statistics of the refined group to the monitoring system, and supplementing the time consumption characteristics of the discarded Span through statistical indicators.

[0112] In this embodiment of the invention, the virtual Span can be specifically understood as: a virtual call chain node generated for each refined group, used to record the total time consumption of the group, indirectly representing the overall time consumption impact of the discarded Span, and ensuring the integrity of the call chain time consumption data.

[0113] Specifically, when sampling logs according to a differentiated log sampling strategy, some spans within a refined group may be discarded due to sampling quota limitations. If the time consumption data of these discarded spans is missing, it may cause deviations in the root cause analysis performed by the monitoring system based on call chain time consumption data. Therefore, it is necessary to simultaneously supplement the time consumption information of these spans through preset methods. Ultimately, based on the complete call chain time consumption information after supplementation, it can assist in accurately locating the time consumption anomalies. There are two preset methods: The first is to generate a virtual span for each refined group during sampling. This virtual span does not correspond to actual business operations but is only used to record the total time consumption of the current group (i.e., the sum of the execution times of all spans within the group). The total time consumption of the group directly represents the contribution of the discarded spans to the overall trace time consumption, ensuring the integrity of the call chain time consumption data. The second is to calculate and update the historical mean and variance of the time consumption of each refined group in real time at preset periods (such as minutes or hours), and directly report these two statistical indicators to the monitoring system backend, using statistical data to replace the specific time consumption information of the discarded spans and supplement their time consumption characteristics. By using any of the above preset methods, information gaps caused by sample discarding can be avoided, ensuring the accuracy of root cause analysis in the monitoring system and providing reliable data support for locating time-consuming abnormal links.

[0114] This invention proposes a Span-based call chain log sampling method. It achieves business semantic awareness through semantic recognition and operation classification, ensuring the integrity of the business process after sampling. Specifically designed for data-intensive scenarios, it optimizes sampling adaptability in such scenarios. Based on historical indicators, it achieves dynamic quota allocation and anomaly detection, possessing flexible adaptive adjustment capabilities. Simultaneously, the quota system design ensures a constant amount of sampled data in the call chain system, preventing system crashes even during peak periods such as major sales events. Furthermore, by using time-completion methods such as grouped total time storage, it ensures that the original monitoring and fault location functions are largely unaffected by sampling, comprehensively guaranteeing system stability.

[0115] The technical solution of this invention involves data mining of the acquired log call chain data to obtain the call topology relationship between the spans of each call chain Trace. Based on this, the spans within each Trace are initially grouped. The SQL of each span in the initial group is extracted and a statement sequence is constructed according to the execution time order. Then, sliding grouping is executed according to a preset time window rule. Furthermore, semantic recognition is performed on the SQL in each sliding group, and the sliding groups under the same initial group are split and / or merged to obtain refined groups. Based on the principle of maximizing information, a corresponding log sampling strategy is formulated and executed for each refined group, thereby improving the business relevance and accuracy of the sampled data, increasing storage efficiency, and enhancing the versatility and practicality of the sampling scheme. While executing the log sampling strategy, for the Spans that were sampled and discarded in each refined group, the time consumption information is supplemented either by generating a virtual Span to record the total time consumption of the group during the sampling process, or by calculating and updating the historical mean and variance of the group's time consumption at a preset period and reporting it to the monitoring system. This supplemented time consumption information can help locate abnormal time consumption points. Sampling and discarding some Spans can easily lead to incomplete time consumption data in the call chain. The two supplementation methods fill the time consumption data gaps by intuitively representing the overall time consumption contribution of the discarded Spans and reflecting the time consumption distribution characteristics of the discarded Spans, respectively, thus avoiding factor-related errors. This method addresses the misjudgment of time consumption anomalies caused by missing data. The completed time consumption information retains accurate data for the key sampled spans and supplements the time consumption correlation characteristics of the discarded spans. This allows time consumption anomaly analysis to cover the entire business process of the trace, helping to accurately locate the specific group and business stage of abnormal time consumption and improving the efficiency and accuracy of root cause analysis. Moreover, this completion method does not require additional collection of original data. It only dynamically generates or calculates statistical indicators based on the grouping information during the sampling process. Without increasing the system's storage and processing burden, it ensures that the sampling strategy reduces redundant data while not affecting the demand for complete time consumption data in subsequent monitoring, alarm, performance optimization, and other scenarios.

[0116] Example 4

[0117] Figure 4 This is a schematic diagram of a Span-based call chain log sampling device provided in Embodiment 4 of the present invention. Figure 4 As shown, the device includes: a data mining module 410, a sequence construction module 420, a sliding grouping module 430, a fine grouping module 440, and a log sampling module 450, wherein:

[0118] The data mining module 410 is used to acquire log call chain data and perform data mining on the log call chain data to obtain the call topology relationship between the spans of each Trace in the log call chain data.

[0119] The sequence construction module 420 is used to group the Span data in each Trace according to the call topology relationship, obtain at least one preliminary group corresponding to each Trace, extract the SQL of the Span in each preliminary group, and construct the statement sequence corresponding to each preliminary group according to the execution time sequence of the SQL of the corresponding preliminary group in each Trace.

[0120] The sliding grouping module 430 is used to slide group each statement sequence according to a preset time window rule to obtain at least one sliding group corresponding to each statement sequence.

[0121] The fine-grained grouping module 440 is used to perform semantic recognition on the SQL in each sliding group, and to split and / or merge the sliding groups under the same initial group according to the semantic recognition results to obtain fine-grained groups corresponding to each Trace.

[0122] The log sampling module 450 is used to determine the log sampling strategy corresponding to each fine-grained group according to the principle of maximizing information, and to perform log sampling by executing the log sampling strategy.

[0123] The technical solution of this invention involves data mining of the acquired log call chain data to obtain the call topology relationship between the spans of each call chain trace. Based on this, the spans within each trace are initially grouped. The SQL of each span in the initial group is extracted and a statement sequence is constructed according to the execution time order. Then, sliding grouping is executed according to a preset time window rule. Furthermore, semantic recognition is performed on the SQL in each sliding group, and sliding groups under the same initial group are split and / or merged to obtain refined groups. Based on the principle of maximizing information, a corresponding log sampling strategy is formulated and executed for each refined group. Through a hierarchical and progressive grouping mechanism driven by business semantics, sampling resources are focused on a set of spans with consistent business logic characteristics, improving the business relevance and accuracy of the sampled data, avoiding invalid sampling and redundant data storage, reducing the cost of the entire log processing chain, improving storage efficiency, and preserving the complete logical context of business operations. This provides structured and interpretable data for subsequent call chain analysis and fault diagnosis. At the same time, with dynamically generated refined grouping and adaptive sampling strategies, it flexibly adapts to the needs of diverse business scenarios, improving the versatility and practicality of the sampling scheme.

[0124] Based on the above embodiments, the data mining module 410 is specifically used for:

[0125] Static analysis is performed on the application source code in the log call chain data to obtain the call topology relationship between the spans of each trace in the log call chain data; and / or data mining is performed on the trace data collected in the simulation stress test environment or the actual online environment in the log call chain data using machine learning algorithms to obtain the call topology relationship between the spans of each trace in the log call chain data.

[0126] Based on the above embodiments, the fine grouping module 440 is specifically used for:

[0127] Calculate the operation feature value of the SQL in each sliding group, and combine the position of each sliding group in the corresponding statement sequence to classify each sliding group into business stages, and obtain the business stage grouping result as the semantic recognition result; obtain the execution time data and execution timestamp data of the SQL execution of the Span corresponding to each sliding group from the log call chain data, and calculate the execution interval data of two adjacent SQL based on the execution timestamp data; vectorize the SQL in each sliding group to obtain the semantic vector corresponding to each sliding group; calculate the semantic vector similarity between any two sliding groups for all sliding groups under the same initial group; identify the access pattern of each sliding group based on the semantic vector similarity, the execution time data and execution interval data of the SQL in the sliding group, and obtain the access pattern identification result as the semantic recognition result; based on the business stage grouping and access pattern identification results of each sliding group, split and / or merge the sliding groups under the same initial group to obtain the refined grouping corresponding to each Trace.

[0128] Based on the above embodiments, the fine grouping module 440 is further used for:

[0129] Calculate the operation feature value of the SQL in each sliding group; the operation feature value includes: read operation ratio, write operation ratio, cache operation ratio, aggregation operation complexity, multi-table join operation ratio, and parameter configuration table read feature value; based on the operation feature value and the position feature of the sliding group in the corresponding statement sequence, mark the operation semantic feature of each sliding group; the operation semantic feature includes: initialization feature, data acquisition feature, data preprocessing feature, core calculation feature, result persistence feature, and notification feature; classify each sliding group into a business stage group that matches the marked operation semantic feature, and use the result of the business stage group as the semantic recognition result.

[0130] Based on the above embodiments, the fine grouping module 440 is further used for:

[0131] Sliding groups with semantic vector similarity greater than a preset similarity threshold are selected to form a target sliding group set. Based on the execution time and interval data of all SQL statements in each target sliding group, the execution time variance and execution interval variance of the SQL statements within the target sliding group are calculated. If the execution time variance of the target sliding group is within a preset low variance range, the access mode of the target sliding group is marked as batch operation mode; if the execution interval variance of the target sliding group is within a preset low variance range, the access mode of the target sliding group is marked as loop operation mode; if the target... If the variance of the execution time of the sliding group is within a preset low variance range, and the variance of the execution interval is also within a preset low variance range, then the execution frequency of the SQL of the target sliding group within a preset time is calculated. If the execution frequency is higher than a preset frequency threshold, it is marked as a batch operation mode; otherwise, it is marked as a loop operation mode. For sliding groups not included in the target sliding group set, their access mode is marked as a normal operation mode. Through the above operations, the access mode identification of each sliding group is completed, and the identification results of the access modes including batch operation mode, loop operation mode, and normal operation mode are obtained as semantic recognition results.

[0132] Based on the above embodiments, the semantic recognition results include: the recognition results of the access patterns; the access patterns include: batch operation mode, loop operation mode and normal operation mode;

[0133] Accordingly, based on the above embodiments, the log sampling module 450 is specifically used for:

[0134] Set a total sampling quota for a single Trace, and allocate a basic sampling quota to each fine-grained group based on the proportion of the number of spans in each fine-grained group to the total number of spans in the corresponding Trace. For the basic sampling quota of each fine-grained group, select spans to be retained using general rules as the initial selection result. For fine-grained groups with access mode of normal operation, directly use the initial selection result as the log sampling strategy. For fine-grained groups with access mode of batch operation or loop operation, perform a secondary selection on the fine-grained group using matching special rules to obtain a secondary selection result, and use the combination of the initial selection result and the secondary selection result as the log sampling strategy. The general rules include: calculating the time consumption anomaly score of each span in the fine-grained group, retaining anomaly spans whose time consumption anomaly score exceeds a preset dynamic threshold; if the number of anomaly spans exceeds the basic sampling quota of the fine-grained group... The Spans are sorted from highest to lowest time consumption anomalies, and the number of Spans before sorting is taken as the basic sampling quota. If the number of anomaly Spans is less than or equal to the basic sampling quota of the refined group, after retaining all anomaly Spans, the Span with the fewest sampling times in the most recent preset time window of the remaining quota is retained. The special rules corresponding to the batch operation mode include: sorting the Spans of the refined group according to the execution time sequence, sampling the first and last preset number of Spans of the sequence according to the first preset ratio, and sampling the Spans in the middle part of the sequence according to the second preset ratio. The first preset ratio is higher than the second preset ratio. The special rules corresponding to the loop operation mode include: marking the Spans corresponding to the same loop in the refined group as a whole, selecting the Spans of the entire loop at preset intervals, performing a full retention operation on the Spans of the selected loops, and performing a full discard operation on the Spans of the unselected loops.

[0135] Furthermore, based on the above embodiments, the Span-based call chain log sampling device may further include: a completion time module and an auxiliary positioning module, wherein:

[0136] The time completion module is used to complete the time completion information for the Spans that are discarded by sampling in each fine-grained group by executing the log sampling strategy, and in accordance with the preset method.

[0137] The auxiliary positioning module is used to help locate abnormal time-consuming links based on the completed call chain time-consuming information. The preset methods include: generating a virtual Span for each fine-grained group during log sampling processing, and recording the total time of the corresponding fine-grained group through the virtual Span; or calculating and updating the historical mean and variance of the time consumption of each fine-grained group at a preset period, and reporting the historical mean and variance as time consumption statistics of the fine-grained group to the monitoring system, and completing the time consumption characteristics of the discarded Span through statistical indicators.

[0138] The Span-based call chain log sampling device provided in this embodiment of the invention can execute the Span-based call chain log sampling method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] In the technical solution disclosed herein, the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0141] In the technical solution disclosed herein, if automated decision-making is involved, a corresponding operation entry will be provided to the user, allowing the user to choose to agree to or reject the automated decision result; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0142] Example 5

[0143] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0144] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the Span-based call chain log sampling method, i.e.:

[0147] The system acquires log call chain data and performs data mining on it to obtain the call topology relationships between spans of each trace. Based on the call topology relationships, the span data within each trace is grouped to obtain at least one preliminary group corresponding to each trace. The SQL statements of each preliminary group are extracted, and statement sequences corresponding to each preliminary group are constructed according to the execution sequence of the SQL statements corresponding to the preliminary groups within each trace. Each statement sequence is then slid-grouped according to a preset time window rule to obtain at least one sliding group corresponding to each statement sequence. Semantic recognition is performed on the SQL statements in each sliding group, and based on the semantic recognition results, sliding groups under the same preliminary group are split and / or merged to obtain refined groups corresponding to each trace. Based on the principle of maximizing information, the log sampling strategy corresponding to each refined group is determined, and log sampling is performed by executing the log sampling strategy.

[0148] In some embodiments, the Span-based call chain log sampling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the Span-based call chain log sampling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the Span-based call chain log sampling method by any other suitable means (e.g., by means of firmware).

[0149] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0154] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0155] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for sampling call chain logs based on span, characterized in that, include: Obtain log call chain data and perform data mining on the log call chain data to obtain the call topology relationship between the spans of each call chain Trace in the log call chain data; Based on the call topology, the Span data in each Trace is grouped to obtain at least one preliminary group corresponding to each Trace. The Structured Query Language (SQL) of the Span in each preliminary group is extracted. According to the execution sequence of the SQL of the corresponding preliminary group in each Trace, the statement sequence corresponding to each preliminary group is constructed respectively. Each statement sequence is grouped into sliding groups according to a preset time window rule to obtain at least one sliding group corresponding to each statement sequence; Semantic recognition is performed on the SQL in each sliding group, and based on the semantic recognition results, the sliding groups under the same initial group are split and / or merged to obtain refined groups corresponding to each Trace; Based on the principle of maximizing information, the log sampling strategy corresponding to each refined group is determined, and log sampling is performed by executing the log sampling strategy.

2. The method according to claim 1, characterized in that, Data mining was performed on the log call chain data to obtain the call topology relationships between the spans of each trace in the log call chain data, including: Static analysis of the application source code within the log call chain data reveals the call topology relationships between the spans of each trace in the log call chain data; and / or For Trace data collected in simulated load testing environments or actual online environments in the log call chain data, data mining is performed using machine learning algorithms to obtain the call topology relationship between the spans of each Trace in the log call chain data.

3. The method according to claim 1, characterized in that, Semantic recognition is performed on the SQL in each sliding group, and based on the semantic recognition results, sliding groups under the same initial group are split and / or merged to obtain refined groups corresponding to each Trace, including: Calculate the operation feature value of the SQL in each sliding group, and combine the position of each sliding group in the corresponding statement sequence to classify each sliding group into business stages. The result of business stage grouping is used as the semantic recognition result. From the log call chain data, obtain the execution time and timestamp data of the SQL execution of the corresponding Span for each sliding group, and calculate the execution interval data of two adjacent SQL based on the execution timestamp data; The SQL in each sliding group is vectorized to obtain a semantic vector corresponding to each sliding group; For all sliding groups under the same initial grouping, calculate the semantic vector similarity between any two sliding groups; Based on semantic vector similarity, execution time data and execution interval data of SQL in sliding groups, the access patterns of each sliding group are identified, and the access pattern identification results are used as semantic recognition results. Based on the identification results of business stage groups and access patterns of each sliding group, sliding groups under the same initial group are split and / or merged to obtain refined groups corresponding to each Trace.

4. The method according to claim 3, characterized in that, Calculate the operation feature value of the SQL in each sliding group, and combine it with the position of each sliding group in the corresponding statement sequence to classify each sliding group into business stages. The result of business stage grouping is used as the semantic recognition result, including: Calculate the operation characteristic value of SQL in each sliding group; where the operation characteristic value includes: read operation ratio, write operation ratio, cache operation ratio, aggregation operation complexity, multi-table join operation ratio, and parameter configuration table read characteristic value; Based on the operation feature value and the position feature of the sliding group in the corresponding statement sequence, an operation semantic feature is marked for each sliding group; among which, the operation semantic features include: initialization features, data acquisition features, data preprocessing features, core calculation features, result persistence features, and notification features; Each sliding group is classified into a business stage group that matches the marked operational semantic features, and the result of the business stage group is used as the semantic recognition result.

5. The method according to claim 3, characterized in that, Based on semantic vector similarity, execution time data and execution interval data of SQL in sliding groups, the access patterns of each sliding group are identified, and the identification results of the access patterns are used as semantic recognition results, including: Select sliding groups whose semantic vector similarity is greater than a preset similarity threshold to form a target sliding group set; Based on the execution time and execution interval data of all SQL statements in each target sliding group in the target sliding group set, calculate the variance of execution time and the variance of execution interval of SQL statements within the target sliding group; If the execution time variance of the target sliding group is within the preset low variance range, then the access mode of the target sliding group is marked as batch operation mode. If the execution interval variance of the target sliding group is within the preset low variance range, then the access mode of the target sliding group is marked as the loop operation mode. If the execution time variance of the target sliding group is within the preset low variance range, and the execution interval variance is within the preset low variance range, then calculate the execution frequency of the SQL of the target sliding group within the preset time. If the execution frequency is higher than the preset frequency threshold, it is marked as batch operation mode; otherwise, it is marked as loop operation mode. For sliding groups that are not included in the target sliding group set, their access mode is marked as normal operation mode; The above operations are used to identify the access patterns of each sliding group, and the identification results of the access patterns, including batch operation mode, loop operation mode and normal operation mode, are obtained as semantic recognition results.

6. The method according to any one of claims 1-5, characterized in that, The semantic recognition results include: the recognition results of access patterns; the access patterns include: batch operation mode, loop operation mode, and normal operation mode; Accordingly, based on the principle of maximizing information, the log sampling strategy corresponding to each refined group is determined, including: Set the total sampling quota for a single Trace, and allocate a basic sampling quota to each refined group according to the proportion of the number of spans in each refined group to the total number of spans in the corresponding Trace; For each refined group's basic sampling quota, the Spans to be retained are selected using general rules as the initial selection results; For fine-grained groups with access mode of normal operation mode, the initial filtering results are directly used as the log sampling strategy; For fine-grained groups with access modes of batch operation mode or loop operation mode, a second filtering is performed on the fine-grained groups through matching special rules to obtain the second filtering results. The combination of the initial filtering results and the second filtering results is used as the log sampling strategy. The general rules include: calculating the time-consuming anomaly score of each Span within the refined group, and retaining the abnormal Span whose time-consuming anomaly score exceeds the preset dynamic threshold. If the number of abnormal Spans exceeds the basic sampling quota for fine-grained grouping, sort them from high to low according to the time-consuming abnormalities, and extract the basic sampling quota number of Spans before sorting. If the number of abnormal spans is less than or equal to the basic sampling quota for fine-grained grouping, after retaining all abnormal spans, the span with the fewest sampling times in the most recent preset time window of the remaining quota will be retained. The special rules corresponding to the batch operation mode include: sorting the finely grouped Spans according to the execution time sequence, sampling a preset number of Spans at the beginning and end of the sequence according to a first preset ratio, and sampling the Spans in the middle part of the sequence according to a second preset ratio; wherein, the first preset ratio is higher than the second preset ratio. The specific rules corresponding to the loop operation mode include: marking the Span corresponding to the same loop in the refined group as a whole, selecting the Span of the whole loop at preset intervals, performing a full retention operation on the Span of the selected loop, and performing a full discard operation on the Span of the unselected loop.

7. The method according to claim 1, characterized in that, In addition to sampling logs by executing a log sampling strategy, the process also includes: For the Spans that were discarded in each refined group, the time consumption information is supplemented using a preset method; Based on the completed call chain time consumption information, we can help locate abnormal time consumption points; The preset methods include: During log sampling processing, a virtual span is generated for each fine-grained group, and the total time spent on the corresponding fine-grained group is recorded through the virtual span; or The historical mean and variance of the time consumed by each refined group are calculated and updated according to a preset period. The historical mean and variance are then reported to the monitoring system as statistical indicators of the time consumed by the refined group. The time consumption characteristics of the discarded Span are supplemented by statistical indicators.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the Span-based call chain log sampling method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the Span-based call chain log sampling method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the Span-based call chain log sampling method according to any one of claims 1-7.

Images